An Adaptive Defense Method for Distributed Storage Systems

Abstract

The application discloses a kind of self-adapting defense methods of distributed storage system, which is by increasing defense detection in each core node in distributed storage system, the core path of data flow, with the way of active rejection, quickly returns response to upper layer;When not satisfying defense detection, return Backoff, help upper layer further decision;When processing Backoff, adopt Reprocess mechanism, start adaptive method, ensure request success rate, user experience, while avoiding invalid retry, quickly process request when system exception, reduce system load;The method is not only suitable for the external service provided by system, but also suitable for the interaction between internal core modules of system, defense detection is carried out first when module interaction, data reading and writing, ensure that the request call in system is carried out smoothly, the request into system can be handled as soon as possible, fully consider the call between internal core modules of distributed storage system, the circulation path of core data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software engineering, and in particular to a method for adaptive defense of a distributed storage system. BACKGROUND

[0002] With the explosive growth of mobile and industrial internet in recent years, the booming development of cloud computing, big data, AI and other technologies cannot be separated from massive data and distributed storage systems. The advantages of distributed storage systems are strong parallel processing capability, high scalability, high system availability and high data durability. Massive data storage makes the size of distributed storage systems larger and larger. Large systems involve thousands of servers, and the number of hardwares such as disks is in the tens of thousands. Moreover, they are deployed across multiple regions. When such a large system is running normally, server downtime, hardware damage, process crashes and power outages in the computer room may occur, which are result-deterministic because once they occur, the fault point is determined and the service cannot continue to be provided.

[0003] Usually, the system will consider fault tolerance for such failures when it is designed and implemented at the beginning, so that when the problem occurs, the impact on system availability and service quality is small, or even no impact. However, in the long-term high-load operation of a large and complex distributed storage system, various problems may occur, which cannot be predicted and classified as result-deterministic faults at the design time. Such exceptions are collectively referred to as non-deterministic exceptions in the present application, for example: a server connection number increases sharply, a network delay in a computer room in a certain region increases significantly, some disks overheat and read and write become slow, and the number of threads in a process increases sharply. The occurrence of these situations is usually a cumulative process, and the system can still operate normally and provide services to the outside at a time. The exception point does not directly die or refuse to serve, but only increases the processing request delay. In the early stage, it only affects the local range of the system. However, after a period of time, the situation may worsen and spread because of occupying a large amount of system resources, gradually producing request accumulation, and seriously reducing the availability of the entire system. In addition, the timeout retry mechanism commonly used by the system can usually improve the request success rate and system availability, but when the above situation occurs, the timeout retry is not only useless, but also increases the system pressure.

[0004] The communication between the existing modules in the distributed storage system is realized in the form of interface call. If the above non-deterministic exception occurs in a certain module, it will inevitably affect other modules that have a calling relationship with it, and gradually affect the entire system. Therefore, there is an urgent need for a defense detection method that fully considers the calls between the core modules of the distributed storage system and the flow of core data to solve the above technical problems. SUMMARY

[0005] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] Read, write, and delete processes without defense detection (existing technology) such as Figure 1 As shown, in this process, write requests are written to the disk of the storage node through the compute node, and a success message is returned to the client only after the metadata is successfully recorded. Read and delete requests, after passing through the compute node, first access the metadata node to obtain the data's metadata, and finally access the disk of the storage node. If a failure occurs at a node, a Fail message is returned to the upper layer and a retry is performed. This approach not only fails to handle requests successfully when faced with non-deterministic exceptions, but also further increases the system burden due to retries.

[0007] Therefore, to solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for adaptive defense of a distributed storage system, which adds defense modules to metadata nodes and storage nodes, and adds defense modules and adaptive modules to computing nodes, so that when a request arrives at a computing node, metadata node, or storage node, a defense test is first performed. Only if the defense test passes will the request continue to be processed. Requests that fail the defense test are returned with a Backoff, indicating that the node is currently not accepting new requests and is in a self-defense state. The response is quickly returned to the upper layer in an active rejection manner, and the upper layer starts an adaptive method for further processing.

[0008] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, if only one computing node experiences an uncertain anomaly, other computing nodes process requests normally, and the system provides services to the outside world normally.

[0009] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the client retryes after receiving a backoff. If it is only a local system anomaly, the retry may be handled by a normal computing node.

[0010] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the core interfaces provided by the distributed storage system are read, write, and delete interfaces, and the specific processes for these three interfaces are as follows:

[0011] Step 1: Before the request is processed by the compute node, a defense check is performed. If the check fails, a backoff is returned directly to the client, and the client receives a 5xx response code, which serves as an external defense mechanism.

[0012] Step 2: When the request to the metadata node fails due to defense detection, a Backoff is returned to the upper-layer computing node, and the Reprocess process is initiated, starting the adaptive method.

[0013] Step 3: When the read / write request of the storage node fails the defense detection, it returns a Backoff to the upper-layer compute node, enters the Reprocess process, and starts the adaptive method;

[0014] Step 4: Steps 2 and 3 enter the Reprocess process. After starting the adaptive method, the system processes the request again based on the request type, system status, etc.

[0015] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, each computing node, metadata node, and storage node in the distributed storage cluster, when receiving a read / write request, first obtains the latest threshold from the dynamic threshold control module, and then determines whether the current node connection count, bandwidth, and disk I / O latency are greater than the current threshold. If any of these detection items exceeds the threshold, the defense detection is not passed.

[0016] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the general formula for calculating the dynamic threshold is:

[0017] Threshold=Min(PropMax*Quota1, PropCommit*Quota2)

[0018] Among them, PropMax is the maximum pressure that the system can withstand;

[0019] Quota1 takes a value between [0,1] and is a parameter with dynamic adjustment capabilities;

[0020] The dynamic value of Quota2 is between [1, N];

[0021] PropCommit represents the pressure committed by the system.

[0022] The Quota value is dynamically updated and adjusted periodically.

[0023] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the hardware performance factors affecting PropMax include network card performance and disk processing capability.

[0024] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, PropCommit is primarily affected by hardware performance.

[0025] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the request entering the Reprocess process is reprocessed by starting an adaptive method based on the request type and current state-related data.

[0026] As a preferred embodiment of the adaptive defense method for the distributed storage system described in this invention, the adaptive method results in one of three outcomes: request failure, asynchronous reprocessing, or recalculation and retry. The purpose of this adaptive approach is to maximize the success rate of requests.

[0027] The beneficial effects of this invention are:

[0028] 1. This method addresses the issue of reduced system availability caused by nondeterministic anomalies by adding defensive detection to each core node and the core path through which data flows in the distributed storage system, and by actively rejecting responses to quickly return them to the upper layer.

[0029] 2. Returning backoff when the defense detection is not met helps the upper layer make further decisions. When handling backoff, a reprocessing mechanism is adopted to start an adaptive method, which ensures the success rate of requests and user experience while avoiding invalid retries. In the event of system anomalies, requests are processed quickly to reduce system load and protect the system.

[0030] 3. This method is not only applicable to the external services provided by the system, which can limit the rate and defend against external attacks in a timely manner, but also applicable to the interaction between core modules within the system. When modules interact or read and write data, defense detection is performed first to ensure that the internal requests and calls of the system can proceed smoothly and that requests entering the system can be processed as quickly and properly as possible. It fully considers the calls between core modules within the distributed storage system and the flow path of core data. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0032] Figure 1 This is a flowchart of the read, write, and delete process that is not protected against detection in existing technologies.

[0033] Figure 2 This is a system module diagram of the present invention.

[0034] Figure 3 This is a system flowchart of the present invention.

[0035] Figure 4This is a flowchart of the defense detection process of the present invention.

[0036] Figure 5 This is a flowchart of the adaptive method of the present invention.

[0037] Figure 6 The flowchart for avoiding multiple invalid retries within the system is provided in this invention.

[0038] Figure 7 The flowchart of this invention aims to prevent local system problems from affecting the overall system. Detailed Implementation

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0042] Reference Figures 2-7 As an embodiment of the present invention, an adaptive defense method for a distributed storage system is provided. The distributed storage system has three core nodes with different roles: compute nodes, metadata nodes, and storage nodes. These nodes are all deployed in a distributed and redundant manner. The compute nodes are responsible for receiving client requests, performing calculations to store metadata in the metadata nodes, and storing data in the storage nodes. The aforementioned data refers to user data, which is eventually persisted to the disk of the storage nodes according to the redundancy algorithm. The metadata is data that identifies user data and is the core data of the storage system. It is ultimately written to the distributed database deployed in the metadata nodes.

[0043] The core interface provided to users by the distributed storage system is the read, write, and delete interface. This solution designs an adaptive defense method for the distributed storage system for the above three interfaces. The core idea of ​​this method is to first perform a defense check when a request arrives at the compute node, metadata node, and storage node. Only if the defense check passes will the request continue to be processed. Requests that fail the defense check will return a Backoff, which means that this node is currently not accepting new requests and is in a self-defense state. The response is quickly returned to the upper layer in an active rejection manner, and the upper layer starts the adaptive method for further processing.

[0044] The specific steps of the adaptive defense method for distributed storage systems are as follows:

[0045] Step 1: Before a request is processed by a compute node, a defense check is performed. As the entry point of the distributed storage system, if the check fails, the compute node directly returns a Backoff to the client, rejecting the request. If the system's processing capacity is exceeded, all compute nodes are in a defensive state of not accepting new requests to avoid system crashes due to excessive pressure. If only one compute node experiences an unpredictable anomaly, other compute nodes process the request normally, and the system continues to provide services. After receiving the Backoff, the client can retry. If it is only a localized system anomaly, the retry may be processed by a normal compute node.

[0046] Step 2: When the request to the metadata node fails due to defense detection, a Backoff is returned to the upper-layer computing node, and the Reprocess process is initiated, starting the adaptive method.

[0047] Step 3: When the read / write request of the storage node fails the defense detection, it returns a Backoff to the upper-layer compute node, enters the Reprocess process, and starts the adaptive method;

[0048] The defense detection process mentioned in the above steps is as follows: Figure 4 As shown;

[0049] Step 4: When each compute node, metadata node, and storage node in the distributed storage cluster receives a read / write request, it first obtains the latest threshold from the dynamic threshold control module, and then determines whether the current node connection count, bandwidth, and disk I / O latency are greater than the current threshold. If any of these detection items exceeds the threshold, the defense detection fails.

[0050] The dynamic threshold can be adjusted at any time, and the general calculation formula is as follows:

[0051] Threshold=Min(PropMax*Quota1, PropCommit*Quota2)

[0052] Among them, PropMax is the maximum pressure that the system can withstand, which is mainly affected by hardware performance, such as network card performance and disk processing capacity.

[0053] Quota1 takes values ​​between [0,1] and is a parameter that can be dynamically adjusted;

[0054] PropCommit represents the pressure committed by the system, primarily influenced by the quality of service promised by the system to the outside world. For example, the system may promise that 95% of request latency will be less than 100ms.

[0055] The dynamic value of Quota2 is between [1, N];

[0056] The Quota value is dynamically updated and adjusted periodically.

[0057] Step 5: For requests entering the Reprocess process, an adaptive method is initiated for reprocessing based on the request type, current status, etc. The outcome of the adaptive method is one of three: request failure, asynchronous reprocessing, or recalculation and retry. Specifically, as follows... Figure 5 As shown;

[0058] Specifically:

[0059] Example 1: When a metadata node is under high load and the processing response is slow, the request fails the node's defense detection and is not allowed to be retried. The adaptive result is that the request fails, which means that the client's request has failed.

[0060] Example 2: Due to disk issues, the read / write latency of a certain storage node increased, and the request failed the defense detection. After the compute node's adaptive method determined that the data met the minimum redundancy requirement, the system directly returned a success message to the user, and the system processed the data asynchronously in the background.

[0061] Application Example 1: Avoiding multiple invalid retries within the system (e.g.) Figure 6 (As shown)

[0062] The specific process is as follows:

[0063] 1. Client-side GET request;

[0064] 2. Request to pass the compute node defense detection;

[0065] 3. Request defense detection through metadata node;

[0066] 4. Data in the storage system is divided into fragmented redundant storage, and it is planned to read fragmented data from storage nodes 1, 2, and 3;

[0067] 5. Data shards 1 and 2 were read through the defense detection of storage nodes 1 and 2;

[0068] 6. Storage node 3 is in a defensive state due to excessive pressure and returns to Backoff. The compute node enters Reprocess, starts the adaptive method, and tries again on storage node 4 to avoid multiple invalid retries on storage node 3.

[0069] 7. Data shard 4 was read through the defense detection of storage node 4;

[0070] 8. The compute node processes the read data shards 1, 2, and 4, returns the data stream to the client, and the read is successful.

[0071] The method proposed in this solution enables compute nodes to quickly recognize that storage node 3 is in a defensive state, initiate an adaptive method, and recalculate the retry target; it ensures efficient processing of requests between systems, avoids invalid timeout retries by compute nodes on storage node 3, and reduces system load.

[0072] Application Example 2: Avoiding local system problems from affecting the overall system (e.g.) Figure 7 (As shown)

[0073] The specific process is as follows:

[0074] 1. The client sends a GET request to read from multiple locations;

[0075] 2. Request to pass the compute node defense detection;

[0076] 3. The distributed storage system is deployed across regions and shares a single metadata system to achieve inter-regional communication; the metadata clusters in regions A, B, and C logically constitute a unified global distributed database; data and metadata are usually stored in one location. For example, if a user wants to get data from region A, they need to access the metadata cluster and storage node cluster in region A.

[0077] 4. When GETting data from locations A and B, the metadata is successfully accessed after passing the metadata defense detection of locations A and B. After passing the storage node defense detection, the data is read and returned to the client, indicating a successful read.

[0078] 5. GET data at location C. At this time, location C experiences network abnormalities, resulting in high read / write latency. The request for metadata from location C fails the defense detection and returns a Backoff. The compute node enters the Reprocess process, starts the adaptive method, and directly returns a request failure to the client.

[0079] The method proposed in this invention enables the system to put location C in a defensive state, actively rejecting requests and quickly returning failure, thus avoiding network failures in location C from affecting the reading and writing of metadata in locations A and B, ensuring the normal functioning of locations A and B, and the stability of the overall function. In contrast, previous systems frequently retried in location C, causing request backlogs that dragged down locations A and B, ultimately leading to a decrease in the availability of the entire system.

[0080] In summary, most current system rate limiting methods limit rates between the client and the server. This functional proposes a defense detection method within the distributed system, which implements rate limiting defense at the core module and data flow path, effectively improving the availability and robustness of the distributed system.

[0081] This invention proposes an adaptive method for rate limiting defense. When the upper-layer module learns that the lower-layer calling module is in a defensive state, this adaptive method is activated to recalculate and process requests, ensuring request success rate and user experience while avoiding invalid retries.

[0082] This invention is more suitable for distributed systems, especially for distributed storage systems with complex situations such as cross-regional deployment and the use of domestically produced hardware products. The defense detection proposed in this invention is a more granular and precise defense rate limiting method, restricting the core modules and core data paths of the distributed system. This invention improves the efficiency of the system in handling requests during anomalies, solves the problem of reduced system availability caused by nondeterministic anomalies, and enhances system robustness. This invention has a wide range of applications, not only applicable to distributed storage systems, but also to various large-scale systems and distributed systems.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for adaptive defense of a distributed storage system, the method comprising: In the metadata node, the storage node increases the defense module, and the computing node increases the defense module and the adaptive module, so that when the request reaches the computing node, the metadata node and the storage node, the defense detection is first made, and only after passing the defense detection will the request continue to be processed. The request that does not pass the defense detection returns Backoff, which represents that the node does not accept new requests at present and is in a self-defense state. In the active rejection manner, the response is quickly returned to the upper layer, and the adaptive method is started for further processing; ​ The core interface provided by the distributed storage system is the read-write-delete interface. The specific process of the three interfaces is as follows: Step one: Before the request is processed by the computing node, the defense detection is first performed. When the detection fails, Backoff is directly returned to the client, and the client receives a 5xx response code, which plays a role in external defense; Step two: When the request of the metadata node fails the defense detection, Backoff is returned to the upper computing node, and the Reprocess process is entered to start the adaptive method; Step three: When the read-write request of the storage node fails the defense detection, Backoff is returned to the upper computing node, and the Reprocess process is entered to start the adaptive method; Step four: After the Reprocess process in steps two and three starts the adaptive method, the request is processed again according to the request type, system state, etc. in the system.

2. The method for distributed storage system adaptive defense of claim 1, wherein: If only a certain computing node has an uncertain abnormality, the other computing nodes normally process the request, and the system normally provides services to the outside.

3. The method for distributed storage system adaptive defense of claim 1, wherein: After the client receives Backoff, it retries. If only the system has a local exception, there is a possibility that the retry is processed by the normal computing node.

4. The method for distributed storage system adaptive defense of claim 1, wherein: When the computing node, the metadata node and the storage node in the distributed storage cluster accept the read-write request, the latest threshold value is first obtained from the dynamic threshold control module, and then it is judged whether the current node connection number, bandwidth and disk IO delay are greater than the current threshold value. As long as any of the detection items exceeds the threshold value, it is not passed the defense detection.

5. The method for distributed storage system adaptive defense of claim 1, wherein: The general calculation formula of the dynamic threshold value is: wherein, Pmax is the maximum pressure the system can withstand; Quota1 takes value between is a parameter with dynamic adjustment function; The dynamic value of Quota2 is between [1, N]; Pressure to commit to the system; The Quota value is dynamically updated and adjusted every certain period of time.

6. The method for distributed storage system adaptive defense of claim 5, wherein: The hardware performance influencing factors include network card performance, disk processing capacity.

7. The method for distributed storage system adaptive defense of claim 5, wherein: Mainly affected by hardware performance.

8. The method for distributed storage system adaptive defense of claim 1, wherein: The request entering the Reprocess process is processed again according to the request type, current state related data, and the adaptive method is started.

9. The method for distributed storage system adaptive defense of claim 1, wherein: The processing result of the adaptive method is one of the three results including request failure, asynchronous reprocessing and re-calculation retry. The purpose of the adaptive method is to ensure the success rate of the request as much as possible.

Images