Fault processing method and device for distributed storage system, equipment and storage medium

Abstract

The invention relates to a fault processing method and device for a distributed storage system, equipment and a storage medium, and belongs to the technical field of distributed storage, the method comprises the steps of detecting the state of a target placement group corresponding to a received read-write request, and according to the state of the target placement group corresponding to the read-write request, determining the fault of the target placement group; determining a running state of the distributed storage system; when it is detected that the distributed storage system is in a fault state, based on a message synchronization mechanism, performing message marking on the target placement group, synchronizing a generated first marking message to a first terminal and a second terminal, and meanwhile, processing the fault of the distributed storage system, the second terminal being used for sending a read-write request to the first terminal; and when it is detected that the fault of the distributed storage system is recovered, sending a second mark message to the second terminal by using the first terminal, and responding to a read-write request sent by the second terminal. The stability of the distributed storage system can be improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a fault handling method, apparatus, device and storage medium for a distributed storage system. Background Art

[0002] In the process of providing storage services to the outside world, the distributed storage system will inevitably enter a faulty state due to software or hardware failures. When a distributed storage system fails, some member OSDs (Object Storage Devices) of some PGs (Placement Groups) may go offline during the process of writing or reading data. At this time, such PGs or an object in the PG may become inaccessible. For the client, there is no reply that the PG or an object in the PG is inaccessible. Therefore, the client will continue to send such read and write requests to the corresponding OSD. The OSD limits the total amount of resources consumed by incoming messages. When the number of messages or message bytes received by the OSD exceeds a certain threshold, it will no longer read new messages from the network socket, thereby causing network back pressure. In addition, after the OSD puts these requests into the queue, the consumption of OSD memory by these read and write requests may cause the OSD memory to exceed its own limit. All of the above may have a more serious impact on the distributed storage system.

[0003] Therefore, there is an urgent need to propose a distributed storage system fault handling method, device, equipment and storage medium that can avoid continuous invalid sending of client read and write requests and excessive consumption of OSD's own memory to improve the stability of the distributed storage system. Summary of the invention

[0004] Based on this, it is necessary to provide a fault handling method, device, equipment and storage medium for a distributed storage system that can avoid continuous invalid sending of client read and write requests and excessive consumption of OSD's own memory to improve the stability of the distributed storage system in response to the above technical problems.

[0005] In a first aspect, a method for handling a fault in a distributed storage system is provided, the method comprising:

[0006] In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0007] In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, the target placement group is message-marked, a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0008] In response to detecting that the failure of the distributed storage system has been recovered, the first terminal is used to send a second marking message to the second terminal, and respond to the read and write request issued by the second terminal.

[0009] Optionally, before receiving the read / write request, the method further includes:

[0010] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request;

[0011] Based on the hash function, calculate and determine the target key value corresponding to the target descriptor;

[0012] Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group;

[0013] Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process;

[0014] The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal;

[0015] After receiving the read / write request, the method further includes:

[0016] In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request;

[0017] The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

[0018] Optionally, the method for detecting the state of the target placement group corresponding to the read / write request includes:

[0019] In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state;

[0020] In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

[0021] Optionally, determining the operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request includes:

[0022] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state;

[0023] In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

[0024] Optionally, in response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, marking the target placement group with a message, and synchronizing the generated first marking message to the first terminal and the second terminal comprises:

[0025] In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type;

[0026] Sending the first marking message to the second terminal;

[0027] When receiving the first marking message, the second terminal generates first record information.

[0028] Optionally, after synchronizing the generated first marking message to the first terminal and the second terminal, the method further includes:

[0029] Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal;

[0030] When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

[0031] Optionally, in response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read / write request issued by the second terminal includes:

[0032] In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type;

[0033] Sending the second marking message to the second terminal;

[0034] When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

[0035] In a second aspect, a fault handling device for a distributed storage system is provided, the device comprising:

[0036] A state determination module, configured to detect the state of a target placement group corresponding to a read / write request upon receiving the read / write request, so as to determine the operation state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0037] a fault processing module, configured to, when detecting that the distributed storage system is in a fault state, mark the target placement group with a message based on a message synchronization mechanism, synchronize the generated first marking message to the first terminal and the second terminal, and process the fault of the distributed storage system at the same time, wherein the second terminal is configured to send the read / write request to the first terminal;

[0038] The request response module is used to send a second marking message to the second terminal by using the first terminal when it is detected that the failure of the distributed storage system has been restored, and respond to the read and write request issued by the second terminal.

[0039] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0040] In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0041] In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, the target placement group is message-marked, a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0042] In response to detecting that the failure of the distributed storage system has been recovered, the first terminal is used to send a second marking message to the second terminal, and respond to the read and write request issued by the second terminal.

[0043] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0044] In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0045] In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, the target placement group is message-marked, a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0046] In response to detecting that the failure of the distributed storage system has been recovered, the first terminal is used to send a second marking message to the second terminal, and respond to the read and write request issued by the second terminal.

[0047] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0048] In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0049] In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, the target placement group is message-marked, a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0050] In response to detecting that the failure of the distributed storage system has been recovered, the first terminal is used to send a second marking message to the second terminal, and respond to the read and write request issued by the second terminal.

[0051] The above-mentioned fault handling method, device, equipment and storage medium of the distributed storage system, the method includes: in response to receiving a read / write request, detecting the state of the target placement group corresponding to the read / write request, so as to determine the operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request; in response to detecting that the distributed storage system is in a faulty state, based on a message synchronization mechanism, marking the target placement group with a message, synchronizing the generated first marking message to a first terminal and a second terminal, and at the same time, processing the fault of the distributed storage system, wherein the second terminal is used to send the read / write request to the first terminal; in response to detecting that the fault of the distributed storage system has been restored, using the first terminal to send a second marking message to the second terminal, and responding to the read / write request issued by the second terminal. The present application can avoid the network back pressure and additional memory usage caused by the continuous sending of client requests when the placement group or an object in the placement group is in an inaccessible state, thereby avoiding the problem of continuous invalid sending of client requests and excessive memory consumption of the OSD itself, thereby improving the stability of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 An application environment diagram of a fault handling method for a distributed storage system in one embodiment;

[0053] Figure 2 A schematic diagram of a flow chart of a method for handling a fault in a distributed storage system according to an embodiment;

[0054] Figure 3 A schematic diagram of a process flow of actively setting a delayed trigger of a method for handling a fault of a distributed storage system in one embodiment;

[0055] Figure 4 A schematic diagram of a process of actively setting delayed recovery in a method for handling a fault in a distributed storage system in one embodiment;

[0056] Figure 5 is a structural block diagram of a fault handling device of a distributed storage system in one embodiment;

[0057] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0060] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0061] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0062] With the continuous development of information technology, data as an important part of information technology has received more and more attention. Analyzing and processing the collected data can obtain a lot of important information, thereby tapping into huge economic value. This has also promoted the rapid development of big data and high-performance computing. How to make storage devices more secure and reliable in data storage and provide efficient data access to the outside world has become one of the key issues of storage devices, and also one of the key issues of distributed storage. According to the background technology, in the process of providing storage services to the outside world, due to software or hardware failures, the distributed storage system will inevitably enter a fault state. At this time, due to component abnormalities, the storage pool that provides storage services to the outside world may fail. It will enter a degraded or inaccessible state. During the data reading process, the PG in the storage pool that is in an inaccessible state needs to go through a recovery mechanism after fault processing and wait for the PG to become accessible before providing services to the outside world. Among them, in a distributed storage device, after an external client accesses the storage device, the storage end is finally stored in the storage pool in the form of an object. The storage pool is divided into several PGs for object storage. During the storage process, the object is mapped once, so that objects with the same feature value are mapped to the same PG. The PG calculates the mapping through an algorithm and stores all objects of the PG in the OSD after the mapping is completed. In a distributed storage device, multiple copies or corrections are generally used. The data redundancy mechanism makes data more secure and reliable. Therefore, there may be multiple OSD members corresponding to the PG mapping, and they back up each other to ensure the security and reliability of the data. During the data writing or reading process, if the distributed storage system fails, some member OSDs of some PGs may go offline. At this time, such PG or an object in the PG may become inaccessible. The distributed storage system needs to handle the fault so that it can return to a normal state where it can provide services to the outside world and the data is accessible. During this period, the healthy member OSD of such PG will not be able to process the request from the client belonging to the PG or an object in the PG, and the OSD will add it to a queue. In the column, it waits for the PG status where the OSD is located to recover to an accessible state before processing these requests again. However, the client has not received a reply that the PG or an object in the PG is inaccessible. Therefore, the client will continue to send such requests to the corresponding OSD. The OSD limits the total amount of resources consumed by incoming messages. When the number of messages or message bytes received by the OSD exceeds a certain threshold, it will no longer read new messages from the network socket, causing network back pressure. In addition, after the OSD puts these requests into the queue, the consumption of OSD memory by these requests may cause the OSD memory to exceed its own limit. All of the above may have a more serious impact on the distributed storage system.

[0063] In order to solve the above technical problems, the present application provides a fault handling method, apparatus, device and storage medium for a distributed storage system, based on a fault handling method of actively setting delays, thereby reducing the network back pressure and additional memory consumption caused by the OSD receiving read and write requests that cannot be processed temporarily, thereby enhancing the stability of the distributed storage system.

[0064] The distributed storage system fault handling method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the data processing platform set on the server 104 through the network, wherein the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0065] In one embodiment, Figure 2 As shown, a method for handling faults in a distributed storage system is provided. Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0066] S1: In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of a distributed storage system according to the state of the target placement group corresponding to the read / write request.

[0067] It should be noted that the read and write requests are issued by the client (i.e., the second terminal) and received by the cluster end. The cluster end (i.e., the first terminal) refers to the distributed storage system; the placement group refers to PG (PlacementGroups), which is a virtual concept used to store objects. The status of the placement group includes accessible status and inaccessible status; the operating status of the distributed storage system includes fault status and normal status.

[0068] In some specific implementations, before receiving the read / write request, the method further includes:

[0069] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request, wherein the target identifier refers to the name of the object operated by the read / write request;

[0070] Based on a hash function, a target key value corresponding to the target descriptor is calculated and determined, wherein a hash function refers to a function that can compress an input of any length into an output of a fixed length through a hash algorithm, and the specific process of calculating the key value is a common method and will not be repeated here;

[0071] Based on the mapping relationship between the key value and the placement group, determining the target placement group corresponding to the target key value, wherein the mapping relationship between the key value and the placement group is preset and stored in a mapping table. In this step, the target key value is compared with the data in the mapping table one by one. When the comparison is successful, the placement group corresponding to the key value obtained by the successful comparison is determined as the target placement group;

[0072] Based on the mapping relationship between the placement group and the object storage daemon, determine the target object storage daemon corresponding to the target placement group, wherein the object storage daemon refers to OSD (Object Storage Device). Similarly, the mapping relationship between the placement group and the object storage daemon is preset and stored in a mapping table. In this step, the target placement group is compared with the data in the mapping table one by one. When the comparison is successful, the object storage daemon corresponding to the placement group obtained by the successful comparison is determined as the target object storage daemon;

[0073] The target object storage daemon based on the first terminal receives the read and write request issued by the second terminal, that is, receives the read and write request issued by the client through the cluster-side OSD;

[0074] After receiving the read / write request, the method further includes:

[0075] In response to receiving the read / write request, detecting the state of the target placement group corresponding to the read / write request, that is, after the cluster-side OSD receives the read / write request sent by the client, detecting whether the target PG is in an accessible state;

[0076] The running state of the distributed storage system is determined according to the state of the target placement group corresponding to the read and write request, that is, according to the state of the target PG, it is determined whether the running state of the distributed storage system is normal or faulty.

[0077] In some specific implementations, the method for detecting the state of the target placement group corresponding to the read / write request includes:

[0078] In response to detecting that there is an object storage daemon in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is in an inaccessible state, exemplarily, a placement group may include multiple object storage daemons, including a primary object storage daemon among the multiple object storage daemons, when the primary object storage daemon fails, one object storage daemon is selected from the remaining object storage daemons as the primary object storage daemon, at this time, defining the state of the target placement group corresponding to the read / write request as an inaccessible state, wherein the state detection and judgment is performed through the cluster-side OSD, and the target placement group in an inaccessible state may include the target PG or an object in the target PG;

[0079] In response to detecting that there is no object storage daemon in a faulty state among the object storage daemons corresponding to the target placement group, it is determined that the state of the target placement group corresponding to the read and write request is an accessible state. When the target placement group is in an accessible state, it is determined that the operating state of the distributed storage system is a normal state.

[0080] In some specific implementations, determining the operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request includes:

[0081] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the running state of the distributed storage system is in a fault state, that is, when the distributed storage system provides data writing or reading services, when the distributed storage system fails due to software or hardware failure, some PGs in the storage pool may enter an inaccessible state. Therefore, when the state of the target placement group is in an inaccessible state, it is determined that the running state of the distributed storage system is in a fault state;

[0082] In response to detecting that the state of the target placement group corresponding to the read / write request is accessible, determining that the operating state of the distributed storage system is normal. When the operating state of the distributed storage system is normal, there is no need to perform the following operations.

[0083] In the above implementation, the cluster-side OSD detects whether the current PG is in an accessible state, thereby determining the operating status of the distributed storage system based on the PG status, so as to quickly handle faults in the distributed storage system. This can also avoid network back pressure and additional memory usage caused by continuous sending of client requests when the PG is inaccessible, thereby enhancing the stability of the distributed storage system.

[0084] S2: In response to detecting that the distributed storage system is in a faulty state, based on a message synchronization mechanism, the target placement group is message-marked, and the generated first marking message is synchronized to the first terminal and the second terminal. At the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal.

[0085] It should be noted that the message synchronization mechanism refers to synchronizing the mark information to the first terminal and the second terminal, that is, synchronizing the mark information to the cluster end and the client; the first mark information refers to the MOSDBackoff message with a message type of block. The MOSDBackoff message is a special message sent by the OSD to the client in the Ceph (distributed file system) storage system. When the OSD knows or expects that a PG or an object in the PG will be unavailable for a period of time, in order to avoid consuming memory resources, the OSD will send a MOSDBackoff message to the client.

[0086] In some specific embodiments, Figure 3 As shown, in response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, message marking is performed on the target placement group, and the generated first marking message is synchronized to the first terminal and the second terminal, including:

[0087] In response to detecting that the distributed storage system is in a faulty state, based on the first terminal, the target placement group is message-marked to generate a first marking message of a first target type. Specifically, when the cluster-side OSD receives a read / write request from the client and determines that the target PG is in an inaccessible state, that is, the distributed storage system is in a faulty state, a MOSDBackoff message is initialized and the message type is set to block to prevent the client from continuing to send messages of the same type.

[0088] Sending the first marking message to the second terminal, that is, sending a MOSDBackoff message with a message type of block to the client;

[0089] When the second terminal receives the first marking message, it generates the first record information. Specifically, after receiving the MOSDBackoff message of the block type, the client forms a record to mark itself, so that the client stops sending read and write requests for the PG (i.e., the target placement group) or an object in the PG to the cluster end.

[0090] In some specific implementations, after synchronizing the generated first marking message to the first terminal and the second terminal, the method further includes:

[0091] Constructing a third marking message of the second target type at the second terminal, and sending the third marking message to the first terminal, wherein the third marking message of the second target type refers to a MOSDBackoff message of the ack-block (blocking reply) type. Specifically, when the client receives the MOSDBackoff message of the block type and forms a record, the client initializes a MOSDBackoff message of the ack-block type and replies to the OSD corresponding to the cluster end;

[0092] When the first terminal receives the third marking message, it generates second record information and adds the read and write request to the waiting queue. Specifically, after receiving the MOSDBackoff message of the ack-block type from the client, the cluster-side OSD forms a record as a delayed recovery backup and adds the read and write request to the waiting queue to wait for the PG status of the OSD to be restored to an accessible state before processing these read and write requests again.

[0093] Furthermore, the fault handling method for handling the fault of the distributed storage system is a common method, such as analyzing the log data to determine the fault point, and then using the corresponding fault recovery method to handle the fault. The specific analysis and processing method is a common method and will not be repeated here.

[0094] In the above implementation, a fault handling method of actively setting a delay is added. That is, when the cluster-side OSD determines that its own PG or an object in the PG is inaccessible because it is in fault recovery, it will actively report a block-type MOSDBackoff request to the client. After receiving this type of request, the client will mark itself and no longer issue read and write requests for the PG or an object in the PG. By actively setting and canceling delays in the form of messages between the OSD and the client, network back pressure caused by continuous sending of client requests and additional memory usage of the OSD when the PG is inaccessible is avoided, thereby enhancing the stability of the distributed storage system.

[0095] In some specific implementations, a method for determining the number of read and write requests that can be added to the waiting queue includes:

[0096] Obtain the resource usage of the distributed storage system at different time nodes within a target time period, where the target time period can be set according to actual needs, such as one month;

[0097] In response to detecting that the resource occupancy of the target time node is less than or equal to a first preset threshold, marking the target time node, wherein the first preset threshold can be set according to actual needs, such as 50%;

[0098] In response to detecting that the target time node is marked more than a second preset threshold value within the target time period, based on the difference between the resource occupancy of the target time node and the preset occupancy value, the number of read and write requests that can be added to the waiting queue is determined, wherein the second preset threshold value can be set according to actual needs, such as 20 times, etc. The calculation method of the number that can be added includes: k=(D 1 -D 2 ) / 2C, where D 1 Indicates the preset occupancy, D 2 represents the current resource usage, C represents the maximum resource usage corresponding to different read and write requests, and k represents the number of resources that can be added;

[0099] The mapping relationship between the target time node and the number of read and write requests that can be added in the waiting queue is stored in a preset mapping table, wherein the number that can be added is a positive number;

[0100] Get the time node within the next time period, and determine the number of read and write requests that can be added in the waiting queue corresponding to the time node based on the time node and the preset mapping table. At this time, if the distributed storage system is in a faulty state, add the number of read and write requests that can be added in the waiting queue to the first marking information sent to the client. When the client receives the first marking information, if it contains the information on the number of addable quantities, it can continue to send read and write requests to the cluster end until the number of read and write requests sent reaches the number of read and write requests that can be added in the waiting queue. If the first marking information sent to the client does not contain the number of read and write requests that can be added in the waiting queue, the client will directly stop sending read and write requests for the corresponding PG to the cluster end when it receives the first marking information.

[0101] In the above implementation, by dynamically setting the number of read and write requests that can be added to the waiting queue, the read and write requests are added to the waiting queue in advance while ensuring the operating stability of the distributed storage system, so that the corresponding read and write requests can be quickly processed when the distributed storage system fails and recovers, thereby improving the processing efficiency of read and write requests.

[0102] S3: In response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request issued by the second terminal.

[0103] It should be noted that the second marking message refers to a MOSDBackoff message whose message type is unblock (cancel blocking).

[0104] In some specific embodiments, Figure 4As shown, in response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request issued by the second terminal includes:

[0105] In response to detecting that the fault of the distributed storage system has been processed and the state of the target placement group is accessible, traverse the record information corresponding to the target placement group in the first terminal, and generate second marking information of the third target type. Specifically, when the cluster-side OSD detects that the state of the PG or an object in the PG becomes accessible, traverse its own backoff record and initialize a MOSDBackoff message of the unblock type, wherein the backoff record refers to the record information generated by the cluster-side OSD after receiving the MOSDBackoff message of the ack-block type from the client;

[0106] Sending the second marking message to the second terminal, that is, sending a MOSDBackoff message of the unblock type to the corresponding client;

[0107] When the second terminal receives the second marking message, the second terminal continues to send read and write requests to the first terminal. At the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests. Specifically, after receiving the MOSDBackoff message of the unblock type, the client restarts the sending of read and write requests for the PG or the object in the PG to send the read and write requests to the OSD on the cluster side. If there are read and write requests in the waiting queue, the newly sent read and write requests are added to the waiting queue. The cluster-side OSD processes the read and write requests in the waiting queue in sequence. If there are no read and write requests in the waiting queue, the cluster-side OSD processes the read and write requests newly sent by the client, and returns the processing results to the client after the read and write request processing is completed.

[0108] In the above implementation, when the PG becomes accessible, the cluster-side OSD will resend an unblock-type MOSDBackoff request to the client. After receiving the request, the client restarts the sending of the PG or the object request within the PG. By actively setting and canceling delays in the form of messages between the cluster-side OSD and the client, continuous invalid sending of client requests and excessive consumption of OSD's own memory are avoided, further improving the stability of the distributed storage system.

[0109] In the above-mentioned fault handling method of the distributed storage system, the method includes: in response to receiving a read / write request, detecting the state of the target placement group corresponding to the read / write request, so as to determine the operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request; in response to detecting that the distributed storage system is in a faulty state, based on a message synchronization mechanism, marking the target placement group with a message, synchronizing the generated first marking message to a first terminal and a second terminal, and at the same time, processing the fault of the distributed storage system, wherein the second terminal is used to send the read / write request to the first terminal; in response to detecting that the fault of the distributed storage system has been restored, using the first terminal to send a second marking message to the second terminal, and responding to the read / write request issued by the second terminal. In the present application, in the fault scenario of the distributed storage system, by actively setting a delay method, when the placement group or an object in the placement group is in an inaccessible state, the network back pressure and additional memory occupation caused by the continuous sending of client requests can be avoided, thereby avoiding the problem of continuous invalid sending of client requests and excessive memory consumption of the OSD itself, thereby improving the stability of the distributed storage system.

[0110] It should be understood that although Figure 2-Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0111] In one embodiment, Figure 5 As shown, a fault handling device for a distributed storage system is provided, comprising: a state determination module, a fault handling module and a request response module, wherein:

[0112] A state determination module, configured to detect the state of a target placement group corresponding to a read / write request upon receiving the read / write request, so as to determine the operation state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0113] a fault processing module, configured to, when detecting that the distributed storage system is in a fault state, mark the target placement group with a message based on a message synchronization mechanism, synchronize the generated first marking message to the first terminal and the second terminal, and process the fault of the distributed storage system at the same time, wherein the second terminal is configured to send the read / write request to the first terminal;

[0114] The request response module is used to send a second marking message to the second terminal by using the first terminal when it is detected that the failure of the distributed storage system has been restored, and respond to the read and write request issued by the second terminal.

[0115] As a preferred implementation, in the embodiment of the present invention, the state determination module is specifically used for:

[0116] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request;

[0117] Based on the hash function, calculate and determine the target key value corresponding to the target descriptor;

[0118] Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group;

[0119] Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process;

[0120] The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal;

[0121] After receiving the read / write request, the method further includes:

[0122] In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request;

[0123] The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

[0124] As a preferred implementation, in the embodiment of the present invention, the state determination module is further used to:

[0125] In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state;

[0126] In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

[0127] As a preferred implementation, in the embodiment of the present invention, the state determination module is further used to:

[0128] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state;

[0129] In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

[0130] As a preferred implementation manner, in the embodiment of the present invention, the fault processing module is specifically used for:

[0131] In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type;

[0132] Sending the first marking message to the second terminal;

[0133] When receiving the first marking message, the second terminal generates first record information.

[0134] As a preferred implementation manner, in the embodiment of the present invention, the fault processing module is further specifically used for:

[0135] Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal;

[0136] When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

[0137] As a preferred implementation, in the embodiment of the present invention, the request response module is specifically used for:

[0138] In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type;

[0139] Sending the second marking message to the second terminal;

[0140] When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

[0141] For the specific definition of the fault handling device of the distributed storage system, please refer to the definition of the fault handling method of the distributed storage system above, which will not be repeated here. Each module in the above-mentioned fault handling device of the distributed storage system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0142] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fault handling method for a distributed storage system is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0143] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0145] S1: In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of a distributed storage system according to the state of the target placement group corresponding to the read / write request.

[0146] S2: In response to detecting that the distributed storage system is in a faulty state, based on a message synchronization mechanism, the target placement group is message-marked, and the generated first marking message is synchronized to the first terminal and the second terminal. At the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal.

[0147] S3: In response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request issued by the second terminal.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request;

[0150] Based on the hash function, calculate and determine the target key value corresponding to the target descriptor;

[0151] Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group;

[0152] Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process;

[0153] The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal;

[0154] After receiving the read / write request, the method further includes:

[0155] In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request;

[0156] The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state;

[0159] In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state;

[0162] In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type;

[0165] Sending the first marking message to the second terminal;

[0166] When receiving the first marking message, the second terminal generates first record information.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal;

[0169] When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

[0170] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0171] In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type;

[0172] Sending the second marking message to the second terminal;

[0173] When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0175] S1: in response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0176] S2: In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, message marking is performed on the target placement group, and a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0177] S3: In response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request issued by the second terminal.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request;

[0180] Based on the hash function, calculate and determine the target key value corresponding to the target descriptor;

[0181] Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group;

[0182] Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process;

[0183] The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal;

[0184] After receiving the read / write request, the method further includes:

[0185] In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request;

[0186] The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0188] In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state;

[0189] In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state;

[0192] In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type;

[0195] Sending the first marking message to the second terminal;

[0196] When receiving the first marking message, the second terminal generates first record information.

[0197] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0198] Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal;

[0199] When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0201] In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type;

[0202] Sending the second marking message to the second terminal;

[0203] When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

[0204] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program when executed by a processor implements the following steps:

[0205] S1: in response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request;

[0206] S2: In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, message marking is performed on the target placement group, and a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal;

[0207] S3: In response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request issued by the second terminal.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request;

[0210] Based on the hash function, calculate and determine the target key value corresponding to the target descriptor;

[0211] Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group;

[0212] Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process;

[0213] The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal;

[0214] After receiving the read / write request, the method further includes:

[0215] In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request;

[0216] The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

[0217] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0218] In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state;

[0219] In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

[0220] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0221] In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state;

[0222] In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0224] In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type;

[0225] Sending the first marking message to the second terminal;

[0226] When receiving the first marking message, the second terminal generates first record information.

[0227] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0228] Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal;

[0229] When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

[0230] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0231] In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type;

[0232] Sending the second marking message to the second terminal;

[0233] When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

[0234] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0235] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0236] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for handling a fault in a distributed storage system, characterized in that: The method comprises: In response to receiving a read / write request, detecting a state of a target placement group corresponding to the read / write request, so as to determine an operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request; In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, the target placement group is message-marked, a generated first marking message is synchronized to a first terminal and a second terminal, and at the same time, the fault of the distributed storage system is processed, wherein the second terminal is used to send the read and write request to the first terminal; In response to detecting that the failure of the distributed storage system has been recovered, the first terminal is used to send a second marking message to the second terminal, and respond to the read and write request issued by the second terminal.

2. The fault handling method of the distributed storage system according to claim 1, characterized in that: Before receiving the read / write request, the method further includes: In response to detecting the read / write request sent by the second terminal, acquiring a target descriptor corresponding to the read / write request; Based on the hash function, calculate and determine the target key value corresponding to the target descriptor; Determine a target placement group corresponding to the target key value based on a mapping relationship between the key value and the placement group; Determine a target object storage daemon process corresponding to the target placement group based on a mapping relationship between the placement group and the object storage daemon process; The target object storage daemon process based on the first terminal receives the read and write request sent by the second terminal; After receiving the read / write request, the method further includes: In response to receiving the read / write request, detecting a state of a target placement group corresponding to the read / write request; The operating state of the distributed storage system is determined according to the state of the target placement group corresponding to the read / write request.

3. The fault handling method of the distributed storage system according to claim 1 or 2, characterized in that: The method for detecting the state of the target placement group corresponding to the read / write request includes: In response to detecting that there is an object storage daemon process in a faulty state among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an inaccessible state; In response to detecting that no object storage daemon in a faulty state exists among the object storage daemons corresponding to the target placement group, determining that the state of the target placement group corresponding to the read / write request is an accessible state.

4. The fault handling method of the distributed storage system according to claim 3, characterized in that: Determining the operating state of the distributed storage system according to the state of the target placement group corresponding to the read / write request includes: In response to detecting that the state of the target placement group corresponding to the read / write request is in an inaccessible state, determining that the operation state of the distributed storage system is a fault state; In response to detecting that the state of the target placement group corresponding to the read / write request is in an accessible state, determining that the operating state of the distributed storage system is in a normal state.

5. The fault handling method of the distributed storage system according to claim 1, characterized in that: In response to detecting that the distributed storage system is in a fault state, based on a message synchronization mechanism, marking the target placement group with a message, and synchronizing the generated first marking message to the first terminal and the second terminal includes: In response to detecting that the distributed storage system is in a fault state, based on the first terminal, message marking the target placement group to generate a first marking message of a first target type; Sending the first marking message to the second terminal; When receiving the first marking message, the second terminal generates first record information.

6. The method for handling a fault in a distributed storage system according to claim 5, characterized in that: After synchronizing the generated first marking message to the first terminal and the second terminal, the method further includes: Constructing a third marking message of a second target type at the second terminal, and sending the third marking message to the first terminal; When receiving the third marking message, the first terminal generates second record information and adds the read / write request to a waiting queue.

7. The fault handling method of a distributed storage system according to claim 1, characterized in that: In response to detecting that the failure of the distributed storage system has been recovered, using the first terminal to send a second marking message to the second terminal, and responding to the read and write request sent by the second terminal includes: In response to detecting that the failure of the distributed storage system has been processed and the state of the target placement group is accessible, traversing the record information corresponding to the target placement group in the first terminal, and generating second marking information of a third target type; Sending the second marking message to the second terminal; When receiving the second marking message, the second terminal continues to send read and write requests to the first terminal, and at the same time, the first terminal starts to process the read and write requests in the waiting queue and / or the newly received read and write requests.

8. A fault handling device for a distributed storage system, characterized in that: The device comprises: A state determination module, configured to detect the state of a target placement group corresponding to a read / write request upon receiving the read / write request, so as to determine the operation state of the distributed storage system according to the state of the target placement group corresponding to the read / write request; a fault processing module, configured to, when detecting that the distributed storage system is in a fault state, mark the target placement group with a message based on a message synchronization mechanism, synchronize the generated first marking message to the first terminal and the second terminal, and process the fault of the distributed storage system at the same time, wherein the second terminal is configured to send the read / write request to the first terminal; The request response module is used to send a second marking message to the second terminal by using the first terminal when it is detected that the failure of the distributed storage system has been restored, and respond to the read and write request issued by the second terminal.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images