Abnormality processing system, method, device, and storage medium

Abstract

Embodiments of the present application provide an exception handling system, method, device and storage medium. In the embodiments of the present application, a metadata node can predict an application data node that will experience service interruption before the service interruption, and store an identification of the predicted abnormal application data node that will experience the service interruption. The user terminal and the metadata node interact through the identification of the abnormal application data node, so that the user terminal can actively avoid the abnormal application data node during access to the distributed storage system, and switch the access traffic to other normal application data nodes without the user's awareness, thereby reducing the risk of service interruption, even without service interruption, so as to ensure the service quality of the distributed storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud services, and in particular to an exception processing system, method, device and storage medium. BACKGROUND

[0002] A data center is a powerful support for the development of digital economy as an information infrastructure. The data center may sometimes have temperature and humidity abnormalities, power outages or network outages, etc., which may cause the upper layer services of the data center to be unavailable. Services are deployed across data centers, and in the case of an exception in a data center, the services can be switched to other data centers, and the services can be continued to be provided by the other data centers.

[0003] However, the existing data center exception processing method is a passive switching method, and it takes a long time to determine the data center exception. During the determination of whether the data center is abnormal, service interruption may occur, which affects the service quality. SUMMARY

[0004] Aspects of the present application provide an exception processing system, method, device and storage medium to reduce the risk of service interruption.

[0005] In a first aspect, an exception processing system is provided, comprising: a distributed storage system, a user terminal and an operation and maintenance node of the distributed storage system; the distributed storage system comprises: a storage cluster; the storage cluster comprises: a plurality of storage units; the plurality of storage units are deployed in a plurality of physical spaces; a storage unit corresponds to a physical space one by one; the storage unit comprises: a metadata node and an application data node; the metadata nodes in the plurality of storage units are backed up to each other and used to store metadata of the storage cluster; the application data nodes in the plurality of storage units are backed up to each other and used to store application data;

[0006] The operation and maintenance node is configured to send exception prompt information to a target metadata node in the plurality of physical spaces in response to service interruption warning information for a target physical space in the plurality of physical spaces; the exception prompt information comprises an identifier of the target physical space;

[0007] The target metadata node is configured to mark an application data node in the target physical space as an exception application data node in response to the exception prompt information; and store an identifier of the exception application data node;

[0008] The user terminal is configured to obtain the identifier of the exception application data node from the target metadata node; and send an access request to an application data node other than the exception application data node.

[0009] Secondly, this application also provides an exception handling method applicable to the operation and maintenance nodes of a distributed storage system. The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: metadata nodes and application data nodes; the metadata nodes in the multiple storage units back each other up and are used to store the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up and are used to store application data.

[0010] The method includes:

[0011] Obtain service interruption warning information for the target physical space among the multiple physical spaces;

[0012] In response to the service interruption warning information, an abnormality prompt information is sent to the target metadata nodes in the multiple physical spaces. In response to the abnormality prompt information, the target metadata nodes mark the application data nodes in the target physical space as abnormal application data nodes and store the identifier of the abnormal application data nodes. This allows the user terminal to send access requests to other application data nodes other than the application data node corresponding to the identifier of the abnormal application data node.

[0013] Thirdly, this application also provides an exception handling method applicable to a target metadata node in a distributed storage system. The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: a metadata node and an application data node; the metadata nodes in the multiple storage units back each other up for storing the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up for storing application data.

[0014] The method includes:

[0015] Obtain the abnormal prompt information sent by the operation and maintenance node of the distributed storage system; the abnormal prompt information is generated by the operation and maintenance node in response to the service interruption warning information for the target physical space among the multiple physical spaces, and the abnormal prompt information includes the identifier of the target physical space;

[0016] In response to the abnormal prompt information, the application data node in the target physical space is marked as an abnormal application data node;

[0017] store the identification of the abnormal application data node, so that the user terminal acquires the identification of the abnormal application data node and sends an access request to other application data nodes except the application data node corresponding to the identification of the abnormal application data node.

[0018] In a fourth aspect, the embodiments of the present application further provide an exception handling method, which is suitable for a user terminal and a distributed storage system including a storage cluster and the storage cluster including a plurality of storage units deployed in a plurality of physical spaces, the storage units corresponding to the physical spaces one by one, the storage units including metadata nodes and application data nodes, the metadata nodes in the plurality of storage units being backed up to each other and used for storing metadata of the storage cluster, and the application data nodes in the plurality of storage units being backed up to each other and used for storing application data.

[0019] The method includes the following steps.

[0020] acquiring an identification of an abnormal application data node from a target metadata node in the plurality of physical spaces, the abnormal application data node being an application data node in a target physical space warned by the service interruption early warning information, and the target physical space being part of the plurality of physical spaces;

[0021] sending an access request to other application data nodes except the application data node corresponding to the identification of the abnormal application data node.

[0022] In a fifth aspect, the embodiments of the present application further provide a computing device including a memory, a processor and a communication component, the memory being used for storing a computer program.

[0023] The processor is coupled to the memory and the communication component and is used for executing the computer program to perform the steps in the exception handling method provided in the second aspect, the third aspect and / or the fourth aspect.

[0024] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, the one or more processors perform the steps in the exception handling method provided in the second aspect, the third aspect and / or the fourth aspect.

[0025] In the embodiments of the present application, the metadata node can predict the application data node that will experience service interruption before the service interruption, and store the identification of the abnormal application data node that will experience service interruption. The user terminal and the metadata node interact through the identification of the abnormal application data node, so that the user terminal can actively avoid the abnormal application data node during access to the distributed storage system, and switch the access traffic to other normal application data nodes without the user's awareness, thereby reducing the risk of service interruption, even without service interruption, thereby ensuring the service quality of the distributed storage system. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0027] Figure 1 A structural schematic diagram of an abnormal processing system provided by the embodiments of the present application is shown in the figure;

[0028] Figure 2 A structural schematic diagram of a physical space provided by the embodiments of the present application is shown in the figure;

[0029] Figures 3-5 A flowchart of an abnormal processing method provided by the embodiments of the present application is shown in the figure;

[0030] Figure 6 A structural schematic diagram of a computing device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in conjunction with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0032] A distributed storage system stores data in a distributed manner. The data can be file data, image data, or a data table, etc. The distributed storage system can implement distributed storage of data based on a block storage service. For example, data in the distributed storage system is stored in the form of data blocks (Chunks) in different disks (such as disks D0-Dn) in a distributed manner. For most distributed storage systems, metadata and application data are usually stored independently, that is, the control flow and the data flow are separated, so as to obtain higher policy scalability and Input / Output (I / O) concurrency. Based on this, the distributed storage system is generally divided into two types of service nodes, such as a metadata (Meta) node and an application data (Data) node.

[0033] Among them, the metadata node is used to store the metadata of the distributed storage system, and the metadata includes an address mapping relationship of the application data, that is, address information of the application data node where the application data is located. The application data node is used to store the application data, that is, the service data. The metadata node can control the position of the application data written to the application data node, and inform the user end of the position of the application data node where the data to be read is located when reading the data.

[0034] When the distributed storage system is deployed across data centers, the metadata node can automatically identify the application data node of each data center, so that the metadata node can establish a heartbeat connection with each application data node. When the application data node in a certain data center is abnormal, the metadata node can determine that the application data node is abnormal by detecting multiple heartbeat abnormalities of the application data node. If the number of abnormal application data nodes in the data center exceeds a set number threshold, it is determined that the data center is abnormal. From then on, subsequent data reading and writing are switched to other normal data centers and are no longer diverted to the abnormal data center, thereby achieving the purpose of avoiding the abnormal data center.

[0035] The above-mentioned data center abnormality processing method needs multiple heartbeat abnormalities to determine the abnormality of the data node, and this process needs to consume minutes to complete, which consumes a long time. During the determination of whether the data center is abnormal, service interruption may occur, affecting the service quality. Furthermore, if the data center has temperature and humidity abnormalities or unexpected disasters (such as fire, flood, etc.), not all application data nodes are abnormal at the same time, and the above-mentioned method determines the abnormality of the entire data center for a longer time, which causes the data reading and writing to continue to be diverted to the abnormal data center, affecting the service quality.

[0036] In order to reduce the service interruption risk and improve the service quality, in some embodiments of the present application, the metadata node can predict the application data node that will cause the service interruption before the service interruption, and store the identification of the abnormal application data node that will cause the service interruption. The user terminal and the metadata node interact with the identification of the abnormal application data node, so that the user terminal can actively avoid the abnormal application data node during accessing the distributed storage system, and switch the access traffic to other normal application data nodes without the user's awareness, thereby reducing the risk of service interruption, even without service interruption, so as to ensure the service quality of the distributed storage system.

[0037] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0038] It should be noted that the same reference numerals represent the same objects in the following drawings and embodiments, so that once an object is defined in one drawing or embodiment, it does not need to be further discussed in subsequent drawings and embodiments.

[0039] Figure 1 The structure schematic diagram of the abnormal processing system provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the abnormal processing system includes a distributed storage system 10, an operation and maintenance node 20 of the distributed storage system, and a user terminal 30. Figure 1

[0040] As shown in FIG. 2, the distributed storage system 10 can include a storage cluster 101. The storage cluster can be one or more. More means two or more. The distributed storage system 10 can use the storage cluster 101 to store data in a distributed manner. Figure 1

[0041] Among them, the storage cluster 101 includes a plurality of storage units 102. In order to increase the security of the data, the plurality of storage units 102 are deployed in a plurality of physical spaces 40, and the storage unit 102 corresponds to the physical space 40 one by one. That is, one physical space 40 includes one storage unit 102 in the same storage cluster 101. In the embodiments of the present application, more means two or more.

[0042] In the embodiments of the present application, the physical space 40 refers to the actual carrier space for storing servers and other electronic equipment, and the physical space 40 can provide power and network resources for the servers. The distributed storage system 10 is deployed across the physical space 40. The distributed storage system 10 deployed across the physical space 40 can connect the servers in multiple physical spaces 40 to provide services. When a problem occurs in one physical space, the servers in other physical spaces can take over the services in the abnormal physical space to continue to ensure the normal operation of the services.

[0043] ​​The implementation form of the physical space 40 is determined by the distributed storage granularity of the distributed storage system 10. For example, the distributed storage system 10 is deployed across data centers, and the physical space 40 can be a data center. One data center can serve as one physical space. For another example, the distributed storage system 10 is deployed across machine rooms, and the physical space 40 can be a machine room. One or more machine rooms can also constitute one physical space. For another example, the distributed storage system 10 is deployed across racks, and the physical space 40 can be a rack. One or more racks can constitute one physical space. Among them, as shown in Figure 2 a data center can include one or more machine rooms; and one machine room can include multiple racks. The rack, as the infrastructure of the data center, can serve as a carrier for storing servers, and can also provide resources such as power and communication for the servers.

[0044] In this embodiment, the distributed storage system 10 stores both metadata and application data independently, that is, separates the control flow and the data flow. Accordingly, as shown in Figure 1 the storage unit 102 can include a metadata node 103 and an application data node 104.

[0045] In this embodiment, the metadata nodes 103 in the multiple storage units 102 in the same storage cluster 101 are backed up to each other for storing metadata of the storage cluster. The application data nodes 104 in the multiple storage units 102 in the same storage cluster 101 are backed up to each other for storing application data. Among them, the metadata includes an address mapping relationship of the application data in the storage cluster 101, that is, address information of the application data node where the application data is located. The application data node is used to store application data, that is, service data. The metadata node can control the position of the application data written to the application data node, and inform the user terminal 30 of the position of the application data node where the data to be read is located when reading data.

[0046] In this embodiment, the user terminal 30 refers to a computer device occupied by a user and having functions such as computing, surfing the Internet, and communication required by the user, which can be a mobile phone, a tablet computer, a personal computer, or a wearable device, etc. The user terminal 30 can obtain address information of an application data node to be accessed from the metadata node 103, and access the application data node based on the address information. Among them, the address information of the application data node can be address information required by a network protocol followed by an access request. For example, the network protocol followed by the access request is the Transmission Control Protocol (TCP), and the address information of the application data node can be an Internet Protocol (IP) address and a port number.

[0047] Operation and maintenance node 20 is the server-side device where the operation and maintenance system of the distributed storage system resides. It generally has the capability to undertake and guarantee services. The server-side device can be a single server device, a cloud-based server array, or a virtual machine (VM) running in a cloud-based server array. In addition, server-side device can also refer to other computing devices with corresponding service capabilities, such as computer terminal devices (running service programs), etc.

[0048] In this embodiment, as Figure 1 As shown in step 1, the operation and maintenance node 20 obtains the service interruption warning information of the physical space 40. In this embodiment, the cause of the service interruption of the physical space 40 is not limited. An anomaly in the physical space 40 will lead to a service interruption of the physical space 40. Anomalies in the physical space 40 are generally predictable. For example, the air parameters (temperature and / or humidity, etc.) of the physical space 40 change gradually, rather than changing abruptly. An anomaly in the physical space 40 may be an anomaly in the air parameters of the physical space 40, such as an anomaly in the temperature and / or humidity of the physical space 40. An anomaly in the air parameters of the physical space 40 may lead to an anomaly in the server corresponding to the storage unit in the physical space 40, thereby causing a service interruption of the physical space 40. In this embodiment, an air parameter monitoring device (not shown in the accompanying drawings) can be set in the physical space 40. The air parameter monitoring device can monitor the air parameters of the physical space 40 and issue an air parameter anomaly warning (such as an abnormal temperature warning and / or an abnormal humidity warning, etc.) when the detected air parameters exceed the set normal range, and send the air parameter anomaly warning information to the operation and maintenance node 20. Operation and maintenance node 20 obtains abnormal air parameters warning information for this physical space as service interruption warning information.

[0049] For example, an anomaly in physical space 40 could be an anomaly in its power resources, such as a power outage expected within a certain timeframe. A power outage in physical space 40 would also lead to service interruption. In this embodiment, the power company typically notifies the maintenance personnel of physical space 40 in advance that a power outage will occur in one or more physical spaces due to insufficient power or other reasons. Maintenance node 20 can also obtain power outage warning information before the power outage in physical space 40. Accordingly, maintenance node 20 can obtain power outage warning information for physical space 40 as service interruption warning information.

[0050] For example, an anomaly in physical space 40 could be a network anomaly, such as a network outage in physical space 40 over a certain period of time. A network outage in physical space 40 will also lead to service interruption. In this embodiment, the network operations and maintenance department will typically notify the maintenance personnel of physical space 40 in advance of the time and location when a particular physical space or several physical spaces 40 will be disconnected from the network. Maintenance node 20 can also obtain network outage warning information before the power outage in physical space 40. Accordingly, maintenance node 20 can obtain network outage warning information for physical space 40 as service interruption warning information.

[0051] The reasons for service interruption caused by physical space anomalies illustrated in the above embodiments are merely illustrative and do not constitute a limitation. In summary, the operation and maintenance node 20 can obtain anomaly warning information for the physical space 40 as service interruption warning information. This anomaly warning information may include: the cause of the anomaly and the time information when the anomaly will occur, etc.

[0052] In some embodiments of this application, maintenance of physical space 40 may also lead to service interruption of physical space 40. Maintenance of servers in physical space 40 is also predictable in advance; maintenance personnel will announce in advance the time when maintenance will be performed on physical space 40. Maintenance of physical space 40 includes, but is not limited to: service upgrades for devices in physical space 40, and / or network switching of physical space 40 (such as changing network operators). Accordingly, maintenance node 20 can obtain maintenance forecast information for physical space 40 as service interruption warning information. The maintenance forecast information may include: maintenance items and maintenance time, etc.

[0053] Since service interruption in physical space 40 is predictable, in this embodiment, intervention can be implemented in advance based on the predicted service interruption to reduce the risk of service interruption in the distributed storage system. The solution is described in detail below. In this embodiment, for ease of description and distinction, the physical space warned by the service interruption warning information is defined as the target physical space, i.e., the target physical space will experience service interruption. The service interruption warning information may include: the identifier of the warned target physical space and the time information of the service interruption, etc.

[0054] like Figure 1 As shown in step 2, upon receiving a service interruption warning for the target physical space 40, the operation and maintenance node 20, in response to the warning, sends an anomaly alert to the target metadata nodes 103 in multiple physical spaces 40. This anomaly alert indicates that a service interruption will occur in the target physical space 40 and triggers the target metadata nodes 103 to respond accordingly. The anomaly alert may include the identifier of the target physical space.

[0055] In the embodiments of the present application, the target metadata node 103 can be all metadata nodes in each storage cluster 101. Since the metadata nodes 103 in the same storage cluster 101 are backed up to each other, in some embodiments, a leader node can be selected from the metadata nodes 103 in the same storage cluster 101 by a leader election method, and the leader node interacts with the operation and maintenance node 20. Accordingly, the target metadata node can also be the leader node generated by the leader election of the metadata nodes 103 in each storage cluster 101. In which, one storage cluster 101 has one leader node. Preferably, the target metadata node 103 is the leader node generated by the leader election of the metadata nodes in the storage cluster 101.

[0056] In the embodiments of the present application, the leader election method of the metadata nodes in the storage cluster 101 is not limited. In some embodiments, the metadata nodes in the storage cluster 101 can use a distributed consistency algorithm, such as the Raft algorithm, to elect a leader. Specifically, the Raft algorithm divides time into terms, and performs leader election at the beginning of each term. In which, the Raft algorithm uses a heartbeat to trigger the leader election operation. When the metadata nodes in the storage cluster 101 are started, they are initialized as follower nodes. The leader node periodically sends heartbeat information to other follower nodes. The follower nodes confirm that the node sending the heartbeat information is the leader node when they receive the heartbeat information.

[0057] For the target metadata node 103, in response to the abnormal prompt information, the application data node in the target physical space can be marked as an abnormal application data node, and the identification of the abnormal application data node is stored (corresponding to step 3). Figure 1 Optionally, the identification of the abnormal application data node can be cached in the memory of the target metadata node 103, and of course, the identification of the abnormal application data node can also be stored in the persistent storage medium (such as a disk, a hard disk, etc.) of the target metadata node 103.

[0058] In the embodiments, the identification of the abnormal application data node can be information for uniquely identifying an application data node, such as address information, a number or an identity document (ID) of the application data node, etc.

[0059] For example, Figure 1As shown in step 4, the user terminal 30 can obtain the identifier of the abnormal application data node from the target metadata node 103. Optionally, the user terminal 30 can periodically obtain the identifier of the abnormal application data node from the target metadata node 103. Specifically, the user terminal 30 can periodically send query requests to the target metadata node 103. In some embodiments, the user terminal 30 can install the software development kit (SDK) corresponding to the distributed storage system 10 and call the storage service through the SDK. Accordingly, the user terminal 30 can periodically send query requests to the target metadata node 103 through the SDK. This query request is used to request the identification of the abnormal application data node stored in the target metadata node 103.

[0060] Accordingly, the target metadata node 103 can respond to the query request and return the identifier of the stored abnormal application data node to the user terminal 30. The user terminal 30 receives the identifier of the abnormal application data node returned by the target metadata node 103. Specifically, the target metadata node 103 can respond to the query request and return the identifier of the stored abnormal application data node to the SDK on the user terminal 30 side. The SDK on the user terminal 30 side receives the identifier of the abnormal application data node. Further, as... Figure 1 As shown in step 5, the user terminal 30 (specifically, the SDK) can send access requests to the distributed storage system to application data nodes other than the identifier of the abnormal application data node. These other application data nodes belong to the same storage cluster as the abnormal application data node. Since the SDK runs as a background thread on the user terminal 30, the user is unaware that the user terminal 30 is resending access requests to application data nodes other than the abnormal application data node.

[0061] In this embodiment, the metadata node can predict the application data nodes that will experience service interruptions before service is interrupted, and store the identifiers of the predicted abnormal application data nodes. By exchanging these identifiers with the metadata node, the user terminal can proactively avoid abnormal application data nodes when accessing the distributed storage system. Access traffic is switched to other normal application data nodes without the user's awareness, reducing the risk of service interruption, or even eliminating it altogether.

[0062] On the other hand, there is a certain delay for the user terminal to obtain the latest identification of the abnormal application data node from the metadata node. During the period when the user terminal has not updated to the latest abnormal application data node, the access traffic is still diverted to the application data node marked as abnormal. In the embodiment, the metadata node only marks the application data node of the target physical space as abnormal, and does not stop the service of the application data node. Therefore, before the access traffic of the user terminal is completely diverted to other application data nodes which are not marked as abnormal, if the application data node accessed by the user terminal includes the application data node marked as abnormal, the application data node marked as abnormal can still provide service, and the quality of service of the distributed storage system can be ensured.

[0063] In the embodiment of the present application, the specific implementation that the user terminal 30 sends the access request to the application data node other than the application data node corresponding to the identification of the abnormal application data node is not limited. In some embodiments, the user terminal 30 can detect whether the current to-be-accessed application data node includes the abnormal application data node according to the obtained identification of the abnormal application data node.

[0064] Specifically, when accessing the distributed storage system 10, the user terminal 30 can obtain the identification of the to-be-accessed application data node, such as the address information of the to-be-accessed application data node, from the metadata node 103. Further, the user terminal 30 can match the obtained identification of the abnormal application data node in the identification of the current to-be-accessed application data node; if the identification of the current to-be-accessed application data node includes the identification of the abnormal application data node, it is determined that the current to-be-accessed application data node includes the abnormal application data node.

[0065] Further, in the case where the current to-be-accessed application data node includes the abnormal application data node, the user terminal 30 stops sending the access request to the abnormal application data node; and determines a new to-be-accessed application data node from other application data nodes belonging to the same storage cluster as the abnormal application data node. In the embodiment of the present application, in order to facilitate description and distinction, the original to-be-accessed application data node of the user terminal 30 is defined as a first application data node; and the determined new to-be-accessed application data node is defined as a second application data node.

[0066] The access operation of the access request is different, and the implementation of determining the new application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node is also different. In some embodiments, the access operation of the access request is a write operation, and the access request is a write request. When the user terminal 30 determines the new application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node, the user terminal 30 can re-request the application data node to be written from the target metadata node 103.

[0067] Correspondingly, the target metadata node 103 can determine the new application data node to be written from the other application data nodes belonging to the same storage cluster as the abnormal application data node as the second application data node in response to the request of the user terminal 30 for the application data node to be written.

[0068] In the embodiments of the present application, the specific implementation of the target metadata node 103 determining the new application data node to be written from the other application data nodes belonging to the same storage cluster as the abnormal application data node is not limited.

[0069] In some embodiments, the data amount of the data to be written in the write request has a requirement for the free storage resource of the application data node, that is, the free storage resource amount of the application data node should meet the storage resource requirement of the write request, such as the free storage resource amount of the application data node being greater than the storage resource amount required by the write request. Correspondingly, the target metadata node 103 can determine the application data node whose free storage resource amount meets the storage resource requirement of the write request from the other application data nodes belonging to the same storage cluster as the abnormal application data node. If the application data node whose free storage resource amount meets the storage resource requirement of the write request is one, the application data node whose free storage resource amount meets the storage resource requirement of the write request is determined as the second application data node.

[0070] If there are multiple application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request, the target metadata node 103 can further select, as the second application data node, an application data node whose distance to the user terminal 30 satisfies a set distance condition, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request. The set distance condition can be that the distance is less than or equal to a set distance threshold. Accordingly, the target metadata node 103 selects, as the second application data node, an application data node whose distance to the user terminal 30 satisfies the condition of being less than or equal to the set distance threshold, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request. Alternatively, the set distance condition can be that the distance is the shortest. Accordingly, the target metadata node 103 selects, as the second application data node, an application data node whose distance to the user terminal 30 is the shortest, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request, and so on.

[0071] In some other embodiments, the target metadata node 103 can select, as the second application data node, an application data node whose storage water level satisfies a set water level condition, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request, according to the storage water levels of other application data nodes belonging to the same storage cluster as the abnormal application data node.

[0072] The storage water level of an application data node 104 can be the water level of the occupied storage resource in the application data node 104, i.e., the proportion of the occupied storage resource in the application data node 104 to the total storage resource of the application data node 104. In some embodiments, to minimize storage resource fragmentation, the target metadata node 103 can select, as the second application data node, an application data node with the largest storage water level, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request. In some other embodiments, to balance the load, the target metadata node 103 can select, as the second application data node, an application data node with the smallest storage water level, from the application data nodes whose free storage resource amount satisfies the storage resource requirement of the write request.

[0073] Alternatively, the storage water level of the application data node 104 can also be the water level of the unoccupied storage resources (i.e. free storage resources) in the application data node 104, i.e. the proportion of the unoccupied storage resources (i.e. free storage resources) in the application data node 104 in the total storage resources of the application data node 104. Accordingly, in some embodiments, in order to realize storage resource fragmentation minimization, the application data node with the minimum storage water level can be selected as the second application data node from the application data nodes whose free storage resources meet the storage resource requirement of the write request. In other embodiments, in order to realize load balancing, the application data node with the maximum storage water level can be selected as the second application data node from the application data nodes whose free storage resources meet the storage resource requirement of the write request.

[0074] The determination manner of the second application data node shown in the above embodiments is only exemplary and does not constitute a limitation. After the identity of the second application data node is determined, the target metadata node 103 can return the identity of the second application data node to the user terminal 30. Accordingly, the user terminal 30 can receive the identity of the application data node returned by the target metadata node 103, and determine the application data node corresponding to the identity of the application data node returned by the target metadata node 103 as the second application data node.

[0075] Further, the user terminal 30 can send the write request to the second application data node, and the second application data node writes the data to be written carried by the write request to the second application data node in response to the write request.

[0076] In other embodiments, the access operation of the access request is a read operation, and the access request is a read request. When determining the second application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node, the user terminal 30 can determine the application data node belonging to the above-mentioned first application data node originally to be read as the second application data node from the other application data nodes belonging to the same storage cluster as the abnormal application data node. That is, for the read request, the user terminal 30 only reads data from the other application data nodes except the abnormal application data node in the application data node to be read by the read request, thereby avoiding diverting the read request to the abnormal application data node.

[0077] The above embodiments exemplarily illustrate the implementation manner of determining the second application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node with the read request or the write request as an example, but do not constitute a limitation.

[0078] After determining the second application data node to be accessed by the user terminal 30, the user terminal 30 can send an access request to the second application data node. The second application data node can respond to the access request and perform an access operation according to the access request. For example, the access request is a write request, and the second application data node writes data to be written carried by the write request to the second application data node in response to the write request. For another example, the access request is a read request, and the second application data node reads data requested to be read by the read request from the second application data node in response to the read request. Further, the second application data node returns the data requested to be read by the read request to the user terminal 30.

[0079] In the embodiments of the present application, the service of the target physical space where the service interruption occurs is also restored. For the embodiment where the service interruption of the target physical space 40 is caused by the abnormality of the target physical space 40, the target physical space 40 restores the service after the abnormality of the target physical space 40 is repaired. For example, the target physical space 40 restores the service after the air parameters of the target physical space 40 return to normal. For another example, the target physical space 40 restores the service after the target physical space 40 is re-powered. For the embodiment where the service interruption of the target physical space 40 is caused by the operation and maintenance of the target physical space 40, the target physical space 40 restores the service after the operation and maintenance of the target physical space 40 is completed. For example, the target physical space 40 restores the service after the device upgrade of the target physical space 40 is completed. For another example, the target physical space 40 restores the service after the network change of the target physical space 40 is completed.

[0080] In the case where the target physical space restores the service, the embodiments of the present application can also redirect the access traffic to the application data node of the target physical space. Specifically, the operation and maintenance node 20 can send a service restoration prompt information to the target metadata node 103 in the case where the target physical space restores the service, and the service restoration prompt information includes the identifier of the target physical space. Correspondingly, the target metadata node 103 can delete the identifier of the application data node in the target physical space from the stored identifiers of the abnormal application data nodes in response to the service restoration prompt information. In this way, the access request of the user terminal can be redirected to the application data node in the target physical space in subsequent application data node access by the user terminal.

[0081] For example, when the user terminal 30 generates a new access request to the distributed storage system 10, the identifier of the application data node in the target physical space no longer exists in the identifiers of the abnormal application data nodes obtained from the target metadata node 103, and the new access request can be sent to the application data node in the target physical space to restore the access of the target physical space.

[0082] For a write request, the user terminal 30 can request the application data node to be written from the target metadata node 103 when there is new data to be written; the target metadata node 103 can exclude the application data node corresponding to the identifier of the stored abnormal application data node from the application data nodes of the storage cluster to which the target metadata node 103 belongs, to obtain normal application data nodes, and determine the application data node to be written from the normal application data nodes. The specific implementation of the target metadata node 103 determining the application data node to be written from the normal application data nodes can refer to the related content of the target metadata node 103 determining the second application data node from the other application data nodes belonging to the same storage cluster as the abnormal application data node, which will not be described here. Further, the target metadata node 103 can return the identifier of the application data node to be written to the user terminal 30. The user terminal 30 sends the write request to the application data node to be written based on the identifier of the application data node to be written.

[0083] Since the identifier of the abnormal application data node stored by the target metadata node 103 has deleted the identifier of the application data node in the target physical space for service recovery, the target metadata node 103 can list the application data node in the target physical space in the selection range when determining the application data node to be written, and can normally direct the write request to the target physical space for service recovery to recover the write operation of the target physical space.

[0084] For a read request, the user terminal 30 can request the application data node to be read from the target metadata node 103 when reading data; the target metadata node 103 can exclude the application data node corresponding to the identifier of the stored abnormal application data node from the application data nodes of the storage cluster to which the target metadata node 103 belongs, to obtain normal application data nodes, and determine the application data node where the data to be read corresponding to the read request is located from the normal application data nodes. Further, the target metadata node 103 can return the identifier of the application data node where the data to be read corresponding to the read request is located to the user terminal 30. The user terminal 30 sends the read request to the application data node where the data to be read corresponding to the read request is located based on the identifier of the application data node where the data to be read corresponding to the read request is located. The application data node where the data to be read corresponding to the read request is located can execute the read request to read the data to be read, and send the read data to the user terminal 30.

[0085] Since the target metadata node 103 stores the identification of the abnormal application data node, the identification of the application data node in the target physical space that has been deleted for service recovery, when the target metadata node 103 determines the application data node where the data to be read corresponding to the read request is located, the application data node in the target physical space can be listed in the selection range, and the read request can be normally diverted to the target physical space for service recovery, so that the read operation of the target physical space is recovered.

[0086] In addition to the system embodiments described above, the embodiments of the present application also provide an abnormal processing method, and the abnormal recovery method provided by the embodiments of the present application is exemplarily described below.

[0087] Figure 3 A flowchart of the abnormal processing method provided by the embodiments of the present application. The abnormal processing method is applicable to an operation and maintenance node of a distributed storage system. As shown in the figure, Figure 3 the abnormal processing method comprises the following steps.

[0088] 301. Obtain service interruption warning information for a target physical space in a plurality of physical spaces.

[0089] 302. In response to the service interruption warning information, send abnormal prompt information to a target metadata node in the plurality of physical spaces, so that the target metadata node marks an application data node in the target physical space as an abnormal application data node in response to the abnormal prompt information, and stores the identification of the abnormal application data node, so that a user terminal sends an access request to other application data nodes except the application data node corresponding to the identification of the abnormal application data node.

[0090] Figure 4 A flowchart of another abnormal processing method provided by the embodiments of the present application. The abnormal processing method is applicable to a target metadata node in a distributed storage system. As shown in the figure, Figure 4 the abnormal processing method comprises the following steps.

[0091] 401. Obtain abnormal prompt information sent by an operation and maintenance node of a distributed storage system; the abnormal prompt information is generated by the operation and maintenance node in response to service interruption warning information for a target physical space in a plurality of physical spaces, and the abnormal prompt information comprises the identification of the target physical space.

[0092] 402. In response to the abnormal prompt information, mark an application data node in the target physical space as an abnormal application data node.

[0093] 403. Store the identification of the abnormal application data node, so that a user terminal obtains the identification of the abnormal application data node, and sends an access request to other application data nodes except the application data node corresponding to the identification of the abnormal application data node.

[0094] Figure 5 A flowchart of another abnormality processing method provided by embodiments of the present application. The method is mainly applicable to a user terminal. The user terminal can access a distributed storage system. As shown in the figure, the abnormality processing method includes: Figure 5

[0095] 501. Obtain an identifier of an abnormal application data node from a target metadata node in a plurality of physical spaces; the abnormal application data node is an application data node in a target physical space warned by service interruption warning information; the target physical space is part of the plurality of physical spaces.

[0096] 502. Send an access request to other application data nodes except the application data node corresponding to the identifier of the abnormal application data node.

[0097] In embodiments of the present application, the distributed storage system includes a storage cluster. The storage cluster can be one or more. More means two or more. The distributed storage system can store data in a distributed manner using the storage cluster.

[0098] The storage cluster includes a plurality of storage units. In order to increase the security of data, the plurality of storage units are deployed in a plurality of physical spaces, and the storage units correspond one-to-one to the physical spaces. That is, one physical space includes one storage unit in the same storage cluster. In embodiments of the present application, more means two or more.

[0099] The implementation form of the physical space is determined by the distributed storage granularity of the distributed storage system. For example, the distributed storage system is deployed across data centers, and the physical space can be a data center. One data center can serve as one physical space. For another example, the distributed storage system is deployed across machine rooms, and the physical space can be a machine room. One or more machine rooms can also constitute a physical space. For another example, the distributed storage system is deployed across cabinets, and the physical space can be a cabinet. One or more cabinets can constitute a physical space.

[0100] In this embodiment, the storage unit can include a metadata node and an application data node. The metadata nodes in the plurality of storage units in the same storage cluster are backed up to each other and used to store metadata of the storage cluster. The application data nodes in the plurality of storage units in the same storage cluster are backed up to each other and used to store application data.

[0101] In this embodiment, for the operation and maintenance node, in step 301, service interruption warning information for a target physical space in a plurality of physical spaces can be obtained. In some embodiments, abnormal warning information for the target physical space can be obtained as the service interruption warning information. The abnormal warning information can include abnormal reasons and time information when the abnormality will occur, etc.​

[0102] In some embodiments, the operation and maintenance information of the physical space can be acquired as the service interruption warning information. The description of the abnormal cause and operation and maintenance of the target physical space can refer to the related content of the system embodiments described above, and will not be repeated here.

[0103] Since the service interruption of the physical space can be predicted in advance, in the embodiments of the present application, the service interruption risk of the distributed storage system can be reduced by intervening in advance based on the predicted service interruption. The scheme will be described in detail below. In the embodiments of the present application, in order to facilitate description and distinction, the physical space warned by the service interruption warning information is defined as the target physical space, that is, the target physical space will have a service interruption. The service interruption warning information can include the identification of the target physical space warned and the time information of the service interruption, etc.

[0104] In the case where the operation and maintenance node acquires the service interruption warning information of the target physical space, in step 302, the operation and maintenance node can send abnormal prompt information to the target metadata node in the plurality of physical spaces in response to the service interruption warning information. The abnormal prompt information is used to prompt that the target physical space will have a service interruption, and trigger the target metadata node to respond accordingly. The abnormal prompt information can include the identification of the target physical space.

[0105] In the embodiments of the present application, the target metadata node can be all metadata nodes in each storage cluster. Since the metadata nodes in the same storage cluster are backed up to each other, in some embodiments, a leader node (Leader) can be selected from the plurality of metadata nodes in the same storage cluster by a leader election method, and the leader node interacts with the operation and maintenance node. Accordingly, the target metadata node can also be the leader node generated by the leader election of the metadata nodes in each storage cluster. There is one leader node in one storage cluster. Preferably, the target metadata node is the leader node generated by the leader election of the metadata nodes in the storage cluster. The specific implementation of the leader election of the plurality of metadata nodes can refer to the related content of the system embodiments described above, and will not be repeated here.

[0106] For the target metadata node, in step 401, the abnormal prompt information can be acquired, and in step 402, the application data node in the target physical space can be marked as an abnormal application data node in response to the abnormal prompt information. Further, in step 403, the identification of the abnormal application data node can be stored for the user terminal to acquire the identification of the abnormal application data node and send an access request to other application data nodes except the application data node corresponding to the identification of the abnormal application data node. Alternatively, the identification of the abnormal application data node can be cached in the memory of the target metadata node, and of course the identification of the abnormal application data node can also be stored in the persistent storage medium (such as a disk, a hard disk, etc.) of the target metadata node.

[0107] In this embodiment, the identification of the abnormal application data node can be information for uniquely identifying an application data node, such as address information, a number, or an ID of the application data node, etc.

[0108] Correspondingly, for the user terminal, in step 501, the identification of the abnormal application data node can be acquired from the target metadata node. Alternatively, the identification of the abnormal application data node can be periodically acquired from the target metadata node. Specifically, a query request can be periodically sent to the target metadata node. The query request is used to request to acquire the identification of the abnormal application data node stored by the target metadata node.

[0109] Correspondingly, the target metadata node can respond to the query request and return the stored identification of the abnormal application data node to the user terminal. The user terminal receives the identification of the abnormal application data node returned by the target metadata node.

[0110] Further, for the user terminal, in step 502, an access request to the distributed storage system can be sent to other application data nodes except the identification of the abnormal application data node. The other application data nodes are application data nodes belonging to the same storage cluster as the abnormal application data node.

[0111] In this embodiment, the metadata node can predict the application data node that will occur service interruption before the service interruption, and store the identification of the predicted abnormal application data node that will occur service interruption. The user terminal and the metadata node interact through the identification of the abnormal application data node, so that the user terminal can actively avoid the abnormal application data node in accessing the distributed storage system, and switch the access traffic to other normal application data nodes without user awareness, thereby reducing the risk of service interruption, and even without service interruption.

[0112] On the other hand, there is a certain delay in that the user terminal acquires the latest identification of the abnormal application data node from the metadata node. During the period when the user terminal has not updated to the latest abnormal application data node, the access traffic is still diverted to the application data node marked as abnormal. In this embodiment, the metadata node only marks the application data node of the target physical space as abnormal, and does not stop the service of the application data node. Therefore, before the access traffic of the user terminal is completely diverted to other application data nodes that are not marked as abnormal, if the application data node accessed by the user terminal includes the application data node marked as abnormal, the application data node marked as abnormal can still provide service, which can ensure the service quality of the distributed storage system.

[0113] In the embodiments of the present application, the specific implementation of step 502 of sending the access request to the application data node other than the application data node corresponding to the identification of the abnormal application data node is not limited. In some embodiments, the current application data node to be accessed can be detected according to the acquired identification of the abnormal application data node.

[0114] Specifically, when accessing the distributed storage system, the user terminal can acquire the identification of the application data node to be accessed from the metadata node, such as address information of the application data node to be accessed. Further, the user terminal can match the acquired identification of the abnormal application data node in the identification of the current application data node to be accessed; if the identification of the current application data node to be accessed includes the identification of the abnormal application data node, it is determined that the current application data node to be accessed includes the abnormal application data node.

[0115] Further, in the case where the current application data node to be accessed includes the abnormal application data node, the sending of the access request to the abnormal application data node is stopped; and a new application data node to be accessed is determined from other application data nodes belonging to the same storage cluster as the abnormal application data node. In the embodiments of the present application, in order to facilitate description and distinction, the original application data node to be accessed is defined as a first application data node; and the determined new application data node to be accessed is defined as a second application data node.

[0116] The access operation of the access request is different, and the implementation of determining the new application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node is also different. In some embodiments, the access operation of the access request is a write operation, and the access request is a write request. The above determining the new application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node can be implemented as follows: the target metadata node can be requested again for the application data node to be written; the identification of the application data node returned by the target metadata node is received; and the application data node corresponding to the identification of the application data node returned by the target metadata node is determined as the second application data node.

[0117] Correspondingly, the target metadata node can determine the new application data node to be written from the other application data nodes belonging to the same storage cluster as the abnormal application data node as the second application data node in response to the request of the user terminal for the application data node to be written.

[0118] In the embodiments of the present application, the specific implementation of determining the new application data node to be written from the other application data nodes belonging to the same storage cluster as the abnormal application data node is not limited.

[0119] In some embodiments, the data amount of the data to be written in the write request has a requirement for the idle storage resource of the application data node, that is, the idle storage resource amount of the application data node should meet the storage resource requirement of the write request, such as the idle storage resource amount of the application data node being greater than the storage resource amount required by the write request. Correspondingly, the application data node with the idle storage resource amount meeting the storage resource requirement of the write request can be determined from the other application data nodes belonging to the same storage cluster as the abnormal application data node. If the application data node with the idle storage resource amount meeting the storage resource requirement of the write request is one, the application data node with the idle storage resource amount meeting the storage resource requirement of the write request is determined as the second application data node.

[0120] If the application data nodes whose idle storage resource amounts satisfy the storage resource demand of the write request are multiple, the application data node whose distance to the user terminal satisfies a set distance condition can be selected from the application data nodes whose idle storage resource amounts satisfy the storage resource demand of the write request as the second application data node according to the distance between the application data nodes and the user terminal. The set distance condition can be that the distance is less than or equal to a set distance threshold. Accordingly, the application data node whose distance to the user terminal satisfies the condition of being less than or equal to the set distance threshold can be selected from the application data nodes whose idle storage resource amounts satisfy the storage resource demand of the write request as the second application data node. Alternatively, the set distance condition is the shortest distance. Accordingly, the application data node whose distance to the user terminal is the shortest can be selected from the application data nodes whose idle storage resource amounts satisfy the storage resource demand of the write request as the second application data node, and so on.

[0121] In some other embodiments, the application data node whose storage water level satisfies a set water level condition can be selected from the application data nodes whose idle storage resource amounts satisfy the storage resource demand of the write request as the second application data node according to the storage water levels of the other application data nodes belonging to the same storage cluster as the abnormal application data node.

[0122] The determination manner of the second application data node shown in the above embodiments is only exemplary and does not constitute a limitation. After the identity of the second application data node is determined, the target metadata node can return the identity of the second application data node to the user terminal. Accordingly, the user terminal can receive the identity of the application data node returned by the target metadata node, and determine that the application data node corresponding to the identity of the application data node returned by the target metadata node is the second application data node.

[0123] Further, the write request can be sent to the second application data node for the user terminal, and the second application data node writes the to-be-written data carried by the write request to the second application data node in response to the write request.

[0124] In some other embodiments, the access operation of the access request is a read operation, and the access request is a read request. When determining the second application data node to be accessed from the other application data nodes belonging to the same storage cluster as the abnormal application data node, the application data node belonging to the above-mentioned originally to-be-read first application data node can be determined from the other application data nodes belonging to the same storage cluster as the abnormal application data node as the second application data node. That is, for the read request, the user terminal reads data from the other application data nodes except the abnormal application data node in the application data nodes to be read by the read request, thereby avoiding diverting the read request to the abnormal application data node.

[0125] The above embodiments take the access request as an example of a read request or a write request, and exemplarily describe the implementation of determining the second application data node to be accessed from other application data nodes belonging to the same storage cluster as the abnormal application data node, but do not constitute a limitation.

[0126] After determining the second application data node to be accessed by the user terminal, the user terminal can send the access request to the second application data node. The second application data node can respond to the access request and perform an access operation according to the access request.

[0127] In the embodiments of the present application, the service of the target physical space where the service interruption occurs is also restored. In the case where the service of the target physical space is restored, the embodiments of the present application can also re-divert the access traffic to the application data node in the target physical space. Specifically, for the operation and maintenance node, the service recovery prompt information can be sent to the target metadata node in the case where the service of the target physical space is restored; the service recovery prompt information includes the identifier of the target physical space. Correspondingly, the target metadata node can delete the identifier of the application data node in the target physical space from the stored identifier of the abnormal application data node in response to the service recovery prompt information. In this way, when the user terminal performs the application data node access subsequently, the access request of the user terminal can be diverted to the application data node in the target physical space, and the service of the target physical space is restored.

[0128] For example, when the user terminal generates a new access request to the distributed storage system, the identifier of the application data node in the target physical space no longer exists in the identifier of the abnormal application data node obtained from the target metadata node, the new access request can be sent to the application data node in the target physical space, and the access of the target physical space is restored.

[0129] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 401 and 402 can be device A; for another example, the execution subject of step 401 can be device A, and the execution subject of step 402 can be device B; and the like.

[0130] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or performed in parallel without following the order in which they appear in this text. The serial numbers of the operations, such as 401, 402, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0131] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the exception handling methods provided in the above embodiments.

[0132] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 6 As shown, the computing device includes a memory 60a, a processor 60b, and a communication component 60c. The memory 60a is used to store computer programs.

[0133] The processor 60b is coupled to the memory 60a and the communication component 60c, and is used to execute a computer program to perform the steps in the exception handling methods provided in the above embodiments. Detailed descriptions of the method steps can be found in the foregoing embodiments, and will not be repeated here.

[0134] In some alternative implementations, such as Figure 6 As shown, the computing device may also include optional components such as a power supply component 60d, a display component 60e, and an audio component 60f. Figure 6 The diagram only shows some components and does not mean that the computing device must contain them. Figure 6 The inclusion of all components does not imply that a computing device can only include... Figure 6 The components shown.

[0135] in addition, Figure 6 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the form factor of the electronic device. The computing device in this embodiment can be a desktop computer, laptop computer, mobile phone, or IoT device; it can also be a traditional server, cloud server, or server cluster, or other server equipment.

[0136] In the embodiments of the present application, the memory is used to store the computer program and can be configured to store other various data to support the operation on the device where it is located. Among them, the processor can execute the computer program stored in the memory to realize the corresponding control logic. The memory can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0137] In the embodiments of the present application, the processor can be any hardware processing device that can execute the above method logic. Alternatively, the processor can be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU) or a Microcontroller Unit (MCU); it can also be a programmable device such as a Field-Programmable Gate Array (FPGA), a Programmable Array Logic (PAL), a General Array Logic (GAL), a Complex Programmable Logic Device (CPLD), etc.; or an Advanced RISC Machines (ARM) or a System on Chip (SoC), etc., but not limited to this.

[0138] In embodiments of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access wireless networks based on communication standards, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can also be implemented based on a Near Field Communication (NFC) technology, a Radio Frequency Identification (RFID) technology, an Infrared Data Association (IrDA) technology, an Ultra Wide Band (UWB) technology, a Bluetooth (BT) technology, or other technologies.

[0139] In embodiments of the present application, the display component can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the display component includes a touch panel, the display component can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect duration and pressure related to the touch or slide operation.

[0140] In embodiments of the present application, the power component is configured to provide power to various components of the device in which it is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0141] In embodiments of the present application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, for a device with a language interaction function, voice interaction with a user can be implemented through the audio component, etc.

[0142] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, occupation and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0143] It should also be noted that the "first", "second", and the like in the present document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit the "first" and "second" to be different types.

[0144] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, system, or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0145] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system), and computer program product of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0146] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction apparatus, which implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0147] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0148] In one typical configuration, the computing device includes one or more processors (CPUs, etc.), input / output interfaces, network interfaces, and memory.

[0149] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, etc. The memory can also include non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), flash memory, solid-state memory device, etc. The memory can be used for storing various data used during the execution of operating systems or other instructions.

[0150] The storage media of the computer is a readable storage medium, also known as a readable medium. The readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of storage media of the computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, the computer readable medium does not include transitory computer readable media, such as modulated data signals and carriers.

[0151] ​​It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0152] The above embodiments of the present application are only used to illustrate the technical solutions of the present application, and not intended to limit the present application. The present application can have various modifications and variations, and therefore, should not be limited to the above embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An exception handling system, characterized in that, include: Distributed storage system, user terminal, and operation and maintenance node of the distributed storage system; The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: metadata nodes and application data nodes; the metadata nodes in the multiple storage units back each other up and are used to store the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up and are used to store application data. The operation and maintenance node is used to respond to service interruption warning information for a target physical space in the plurality of physical spaces by sending an anomaly prompt message to the target metadata node in the plurality of physical spaces; the anomaly prompt message includes the identifier of the target physical space; The target metadata node is used to mark the application data node in the target physical space as an abnormal application data node in response to the abnormal prompt information; and to store the identifier of the abnormal application data node. The user terminal is configured to obtain the identifier of the abnormal application data node from the target metadata node; and send the access request to other application data nodes other than the application data node corresponding to the identifier of the abnormal application data node.

2. The system according to claim 1, characterized in that, The physical space refers to a data center, server room, or server rack.

3. An exception handling method, applicable to the operation and maintenance nodes of a distributed storage system, characterized in that, The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: metadata nodes and application data nodes; the metadata nodes in the multiple storage units back each other up and are used to store the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up and are used to store application data. The method includes: Obtain service interruption warning information for the target physical space among the multiple physical spaces; In response to the service interruption warning information, an abnormality prompt information is sent to the target metadata nodes in the multiple physical spaces. In response to the abnormality prompt information, the target metadata nodes mark the application data nodes in the target physical space as abnormal application data nodes and store the identifier of the abnormal application data nodes. This allows the user terminal to send access requests to other application data nodes other than the application data node corresponding to the identifier of the abnormal application data node.

4. The method according to claim 3, characterized in that, The service interruption warning information includes: Anomaly warning information for the target physical space, and / or, Operation and maintenance forecast information for the target physical space.

5. The method according to claim 3, characterized in that, Also includes: In the event that the target physical space service is restored, a service restoration notification message is sent to the target metadata node; The service recovery notification includes the identifier of the target physical space, so that the identifier of the application data node in the target physical space can be removed from the identifier of the abnormal application data node.

6. An exception handling method, applicable to target metadata nodes in a distributed storage system, characterized in that, The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: metadata nodes and application data nodes; the metadata nodes in the multiple storage units back each other up and are used to store the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up and are used to store application data. The method includes: Obtain the abnormal prompt information sent by the operation and maintenance node of the distributed storage system; the abnormal prompt information is generated by the operation and maintenance node in response to the service interruption warning information for the target physical space among the multiple physical spaces, and the abnormal prompt information includes the identifier of the target physical space; In response to the abnormal prompt information, the application data node in the target physical space is marked as an abnormal application data node; The identifier of the abnormal application data node is stored so that the user terminal can obtain the identifier of the abnormal application data node and send the access request to other application data nodes other than the application data node corresponding to the identifier of the abnormal application data node.

7. The method according to claim 6, characterized in that, The access request is a write request; the method further includes: In response to the user terminal's request for the application data node to be written, a second application data node is determined from the other application data nodes; The identifier of the second application data node is returned to the user terminal so that the user terminal can send the access request to the second application data node.

8. The method according to claim 7, characterized in that, The step of determining the second application data node from the other application data nodes includes: From the other application data nodes, determine the application data node whose free storage resources meet the storage resource requirements of the write request; and from the application data nodes that meet the storage resource requirements of the write request, determine the application data node whose distance from the user terminal meets the set distance condition, and use it as the second application data node. or, From the other application data nodes, determine the application data node whose free storage resources meet the storage resource requirements of the write request; from the application data nodes whose free storage resources meet the storage resource requirements of the write request, determine the application data node whose storage level meets the set level conditions, and designate it as the second application data node.

9. The method according to any one of claims 6-8, characterized in that, Also includes: Obtain the service recovery notification information sent by the operation and maintenance node; The service recovery notification is issued by the operation and maintenance node when the service is restored in the target physical space; In response to the service recovery prompt, the identifier of the application data node in the target physical space is removed from the identifier of the abnormal application data node, so that when the user terminal accesses the application data node, the access request of the user terminal is redirected to the application data node in the target physical space.

10. An exception handling method, applicable to a user terminal, characterized in that, The distributed storage system includes: a storage cluster; the storage cluster includes: multiple storage units; the multiple storage units are deployed in multiple physical spaces; each storage unit corresponds one-to-one with a physical space; each storage unit includes: metadata nodes and application data nodes; the metadata nodes in the multiple storage units back each other up and are used to store the metadata of the storage cluster; the application data nodes in the multiple storage units back each other up and are used to store application data. The method includes: The identifier of the abnormal application data node is obtained from the target metadata node in the plurality of physical spaces; the abnormal application data node is the application data node in the target physical space warned by the service interruption warning information; the target physical space is a portion of the plurality of physical spaces; The access request is sent to other application data nodes besides the application data node corresponding to the identifier of the abnormal application data node.

11. The method according to claim 10, characterized in that, Sending the access request to other application data nodes besides the application data node corresponding to the identifier of the abnormal application data node includes: Based on the identifier of the abnormal application data node, detect whether the first application data node to be accessed currently contains the abnormal application data node; If the first application data node contains the abnormal application data node, stop sending access requests to the abnormal application data node; The second application data node is determined from the other application data nodes, and the access request is sent to the second application data node.

12. The method according to claim 11, characterized in that, The access request is a write request; determining the second application data node from the other application data nodes includes: The system re-requests the application data node to be written from the target metadata node; and receives the identifier of the application data node returned by the target metadata node; the application data node is determined by the target metadata node from the other application data nodes. The application data node corresponding to the identifier of the application data node returned by the target metadata node is determined to be the second application data node.

13. The method according to claim 11, characterized in that, The access request is a read request; determining the second application data node from the other application data nodes includes: From the other application data nodes, determine the application data node that belongs to the first application data node, and use it as the second application data node.

14. A computing device, characterized in that, include: Memory, processor, and communication components; The memory is used to store computer programs; The processor is coupled to the memory and the communication component for executing the computer program to perform the steps of the method according to any one of claims 3-13.

15. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 3-13.

Images