Metadata request retry method and device, computer equipment, medium and product

Abstract

The invention relates to a metadata request retry method and device, computer equipment, a medium and a product. The method comprises the following steps: when request return information sent by a server for a metadata request is received, obtaining request retry times of the metadata request in a preset sending period; determining an initial retry time interval and a punishment time interval of the metadata request based on the request retry times if it is determined that the metadata request meets a punishment retry condition according to the request retry times; the initial retry time interval is in positive correlation with the number of times of request retry; determining a target retry time interval of the metadata request according to the initial retry time interval and the penalty time interval; and sending the metadata request to the server again according to the target retry time interval. By adopting the method, the pressure of the metadata on the server in the retry process can be reduced.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a metadata request retry method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of information technology, big data and cloud computing technologies have gradually penetrated into every corner of society. The explosive growth of data has brought unprecedented challenges to distributed storage systems. In distributed storage systems, metadata management and operation are the key to ensuring the stable operation of the system, and its processing capacity directly determines the overall performance of the system. However, in the actual operation process, the server will inevitably encounter various abnormal situations when processing metadata requests, such as communication delays, server busyness, resource contention, concurrent lock conflicts, software design defects, network failures, storage device failures, etc. These abnormalities often lead to request failures, which in turn affect user experience and system stability.

[0003] The existing metadata request retry mechanism is often dominated by the server and implemented in conjunction with methods such as queues. That is, when the server detects a request failure, it puts the failed metadata request into a retry queue and tries to process it again after waiting for a certain period of time. Although this retry mechanism can achieve order preservation and priority control of requests to a certain extent, the server also needs to consume additional computing resources and memory space to maintain the retry mechanism, which increases the pressure on the server and affects the performance of metadata operations. Summary of the invention

[0004] Based on this, it is necessary to provide a metadata request retry method, apparatus, computer device, computer-readable storage medium and computer program product that can reduce the pressure on the server caused by metadata during the retry process in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a metadata request retry method, the method comprising:

[0006] Upon receiving the request response information sent by the server in response to the metadata request, obtaining the number of request retries of the metadata request within a preset sending cycle;

[0007] If it is determined that the metadata request satisfies the retry condition with penalty according to the request retry times, then an initial retry time interval and a penalty time interval of the metadata request are determined based on the request retry times; the initial retry time interval is positively correlated with the request retry times;

[0008] Determining a target retry time interval for the metadata request according to the initial retry time interval and the penalty time interval;

[0009] Resend the metadata request to the server according to the target retry time interval.

[0010] In one embodiment, the method further comprises:

[0011] Obtaining the total number of requests for sending metadata requests to the server within the preset sending period;

[0012] Determine the ratio of the number of request retries to the total number of requests as the request abnormality ratio;

[0013] When the request exception ratio is greater than or equal to the penalty ratio threshold, it is determined that the metadata request satisfies the penalty-retry condition.

[0014] In one embodiment, determining the initial retry time interval and the penalty time interval of the metadata request based on the number of request retries includes:

[0015] Determine a random jitter value and a basic unit time for calculating a time interval according to the number of request retries;

[0016] Based on the request retry count, the random jitter value, the basic unit time, and the preset interval growth amplitude value, an improved exponential backoff algorithm is used to determine an initial retry time interval of the metadata request;

[0017] A penalty time interval for the metadata request is determined according to the request abnormality ratio.

[0018] In one of the embodiments, determining the random jitter value and the basic unit time for calculating the time interval according to the number of request retries includes:

[0019] Randomly select an initial jitter value from a preset random jitter value range;

[0020] According to the parity of the request retry times, assigning a positive or negative direction to the initial jitter value to obtain a random jitter value for time interval calculation;

[0021] Based on the number of request retries, a preset interval increase value, and a maximum retry time, a basic unit time for calculating the time interval is determined.

[0022] In one embodiment, determining the penalty time interval for the metadata request according to the request abnormality ratio includes:

[0023] Get the preset penalty factor of the retry penalty mechanism;

[0024] The product of the preset penalty factor and the request abnormality ratio is determined as the penalty time interval of the metadata request.

[0025] In one embodiment, the method further comprises:

[0026] When the request exception ratio is less than the penalty ratio threshold, determining that the metadata request does not satisfy the penalty-bearing retry condition;

[0027] An initial retry time interval of the metadata request determined based on the number of request retries is determined as a target retry time interval of the metadata request.

[0028] In a second aspect, the present application provides a metadata request retry device, the device comprising:

[0029] A request retry count acquisition module, configured to acquire the request retry count of the metadata request within a preset sending cycle upon receiving request return information sent by the server in response to the metadata request;

[0030] a condition judgment module, configured to determine an initial retry time interval and a penalty time interval of the metadata request based on the number of request retries if it is determined that the metadata request satisfies a retry condition with a penalty according to the number of request retries; the initial retry time interval is positively correlated with the number of request retries;

[0031] a target retry time interval determination module, configured to determine a target retry time interval of the metadata request according to the initial retry time interval and the penalty time interval;

[0032] The request retry module is used to resend the metadata request to the server according to the target retry time interval.

[0033] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0036] In the above-mentioned metadata request retry method, device, computer equipment, storage medium and computer program product, when the client receives the request return information sent by the server for the metadata request, it obtains the request retry number of the metadata request within the preset sending cycle. If it is determined that the metadata request meets the penalty retry condition based on the request retry number, the initial retry time interval and the penalty time interval of the metadata request are determined based on the request retry number, and the initial retry time interval is positively correlated with the request retry number of the metadata request. According to the initial retry time interval and the penalty time interval, the target retry time interval of the metadata request is determined, and the metadata request is resent to the server according to the target retry time interval. By converting the execution subject of the request retry mechanism from the server to the client, the occupancy rate of the computing resources and memory space of the server when executing the retry mechanism can be effectively reduced. At the same time, by matching the metadata request with the retry time interval that meets the request retry number of the metadata request, and applying the retry penalty time, the invalid request pressure on the server by the client during the metadata request retry process can be further reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A diagram of an application environment of a metadata request retry method in some embodiments;

[0038] Figure 2 A flowchart of a metadata request retry method in some embodiments;

[0039] Figure 3 A schematic diagram of a flow chart of a metadata request retry method in some other embodiments;

[0040] Figure 4 A schematic diagram of a process for determining an initial retry time interval and a penalty time interval of a metadata request based on the number of request retries in some embodiments;

[0041] Figure 5 A schematic diagram of a flow chart of determining a random jitter value and a basic unit time for time interval calculation according to the number of request retries in some embodiments;

[0042] Figure 6 A schematic diagram of a flow chart of a metadata request retry method in some other embodiments;

[0043] Figure 7 A schematic diagram of a range of values ​​for a retry time interval of 10 retries in some embodiments;

[0044] Figure 8 A structural block diagram of a metadata request retry device in some embodiments;

[0045] Fig. 9 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The metadata request retry method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the client 102 communicates with the server 104 through the network. The data storage system can store the data that the client 102 needs to process. The data storage system can be integrated on the client 102, or it can be placed on the cloud or other network servers. When the client 102 receives the request return information sent by the server 104 for the metadata request, it obtains the request retry number of the metadata request within the preset sending cycle. If it is determined that the metadata request meets the retry condition with penalty according to the request retry number, the initial retry time interval and the penalty time interval of the metadata request are determined based on the request retry number. The initial retry time interval is positively correlated with the request retry number of the metadata request. The client 102 determines the target retry time interval of the metadata request according to the initial retry time interval and the penalty time interval, and resends the metadata request to the server 104 according to the target retry time interval.

[0048] Among them, the client 102 is the interface for the user to interact with the distributed storage system, which is responsible for initiating metadata operation requests, accepting request return information, and calculating the request retry mechanism. The client 102 can communicate with the server through a dedicated API or protocol, such as the TCP protocol. It can be understood that the client 102 is provided with a metadata request processing module and a retry mechanism calculation module. The metadata request processing module can be used to initiate metadata operation requests and accept request return information, and the retry mechanism calculation module can be used to calculate the request retry mechanism.

[0049] The server 104 is mainly responsible for data storage, management and processing of client requests in the distributed storage system, and may include two main parts: a metadata server and a data node. It is understandable that one server 104 may be connected to multiple clients 102. The server 104 may also be provided with a metadata request processing module, which is responsible for processing metadata operation requests.

[0050] In some embodiments, Figure 2 As shown, a metadata request retry method is provided, which is applied to Figure 1 Taking the client 102 in the example as an example, the following steps are included:

[0051] S202: upon receiving the request return information sent by the server in response to the metadata request, obtaining the number of request retries of the metadata request within a preset sending cycle.

[0052] Among them, metadata refers to data that describes data, and metadata request is the request information generated when the client requests the server to access or query metadata.

[0053] The request return information is notification information generated and sent by the server when it determines that the metadata request has failed. When the client receives the request return information sent by the server in response to the metadata request, it can determine that the sent metadata request has failed.

[0054] Among them, the preset sending cycle is a preset time period for counting the number of request retries. In order to reduce statistical complexity and improve statistical consistency, the designer can set the preset sending cycle in the client in advance. When determining the retry time interval for the request retry mechanism, the metadata request sending situation of the client can be determined according to the number of request retries counted within the preset sending cycle, and then the metadata request is matched with the retry time interval that meets the sending request. It can be understood that the specific length of the preset sending cycle can be determined by the designer according to the actual statistical needs, for example, the preset sending cycle can be one hour or one day.

[0055] The number of request retries refers to the number of times the client sends metadata requests to the server using the request retry mechanism within a preset sending cycle. The numerical value of the number of request retries can represent the current sending status of the client's metadata request. For example, it can provide feedback on whether there are abnormal situations in the current client, such as communication delays, busy servers, resource contention, concurrent lock conflicts, software design defects, network failures, storage device failures, etc., which may affect the success rate of metadata requests.

[0056] Specifically, when the server determines that the metadata request sent by the client fails, the server can generate request return information for the metadata request and send the request return information to the client corresponding to the metadata request. When the client receives the request return information sent by the server for the metadata request, it obtains the number of request retries of the metadata request within a preset sending period.

[0057] In some of the embodiments, a request retry count module is provided in the client, which can count the number of retries of the metadata request to determine the number of request retries of the metadata request within a preset sending cycle.

[0058] S204: If it is determined according to the request retry times that the metadata request satisfies the retry condition with penalty, an initial retry time interval and a penalty time interval of the metadata request are determined based on the request retry times.

[0059] Among them, the retry condition with penalty is a preset judgment condition used to determine whether it is necessary to add a penalty time interval based on the initial retry time interval when the client performs a request retry operation on the metadata request. The retry condition with penalty can be set in advance by the designer according to the abnormal reasons that cause the failure of the metadata request. For example, the designer can determine abnormal reasons such as software design defects, storage device failure, network failure, etc. as retry reasons with penalty. When the abnormal reason that causes the failure of the client's metadata request is at least one of the above-mentioned retry reasons with penalty, it can be determined that the metadata request meets the retry condition with penalty.

[0060] In some of these embodiments, the client may determine the abnormal cause of the failure of the client metadata request based on the number of request retries, and determine that the metadata request meets the penalty retry condition when the abnormal cause matches at least one of the penalty retry causes.

[0061] Among them, the initial retry time interval refers to the retry time interval that the client needs to wait when executing the request retry mechanism when the metadata request does not meet the penalty retry conditions. The existing client execution request retry mechanism method usually sets the maximum number of retries or the longest retry time, and then initiates the retry request in a time interval manner. Under this request retry mechanism, if the retry time interval is too short, it will not only cause frequent returns to the previous abnormal problems, such as concurrent resource contention, network failure, storage hardware failure, etc., but also cause frequent request pressure on the server, and even affect other metadata operation requests that could be executed normally; if the retry time interval is too long, it will cause the client to respond too slowly, affecting the performance of the application layer; at the same time, retrying at a time interval will also cause multiple retry tasks of multiple clients to be sent to the server at the same time in some scenarios, which may cause an avalanche effect due to excessive load pressure, such as multiple scheduled tasks set by multiple clients, server failures causing all requests within a period of time to fail, network congestion causing a large number of requests to arrive at the server at the same time, etc.

[0062] In order to reduce the pressure of invalid requests from the client to the server during the retry process, the client can determine the initial retry time interval that matches the number of request retries when executing the request retry mechanism. The initial retry time interval is positively correlated with the number of request retries, that is, the more request retries, the longer the initial retry time interval. In this way, even if the metadata request does not meet the retry conditions with penalty, the client needs to resend the metadata request to the server according to the initial retry time interval that matches the number of request retries, effectively reducing the number of invalid retries of metadata requests that are prone to failure on the server, thereby reducing the pressure of invalid requests from the client to the server during the retry process.

[0063] In some of the embodiments, the initial retry time interval can be determined by the client looking up the preset correspondence between the retry number and the retry time interval based on the request retry number. Specifically, the client can look up the preset correspondence between the retry number and the retry time interval based on the request retry number, and determine the retry time interval that matches the request retry number as the initial retry time interval.

[0064] In some of the embodiments, the initial retry time interval may be calculated by the client according to the number of request retries and a preset time interval retry algorithm. For example, the client may calculate the initial retry time interval based on an exponential backoff algorithm.

[0065] The penalty time interval is the additional waiting time interval that the client needs to wait for to resend the metadata request based on the initial retry time interval.

[0066] In some of the embodiments, the penalty time interval can be determined by the client by searching the preset correspondence between the number of retry attempts and the penalty time interval based on the number of request retries. Specifically, the client can search the preset correspondence between the number of retry attempts and the penalty time interval based on the number of request retries, and determine the penalty time interval that matches the number of request retries.

[0067] In some of the embodiments, a calculation function of the penalty time interval is preset in the client, and the client can determine the penalty time interval according to the number of request retries and the preset calculation function of the penalty time interval.

[0068] Specifically, after obtaining the number of request retries of the metadata request within a preset sending cycle, the client can determine whether the metadata request satisfies the retry condition with penalty according to the number of request retries. If it is determined that the metadata request satisfies the retry condition with penalty according to the number of request retries, the initial retry time interval and penalty time interval of the metadata request are determined based on the number of request retries.

[0069] S206: Determine a target retry time interval for the metadata request according to the initial retry time interval and the penalty time interval.

[0070] Specifically, the client may determine the sum of the initial retry time interval and the penalty time interval as the target retry time interval for the metadata request.

[0071] S208: Resend the metadata request to the server according to the target retry time interval.

[0072] Specifically, after determining the target retry time interval of the metadata request, the client may resend the metadata request to the server according to the target retry time interval to perform the metadata request retry operation.

[0073] In the above metadata request retry method, when the client receives the request return information sent by the server for the metadata request, it obtains the request retry times of the metadata request within the preset sending cycle. If it is determined that the metadata request meets the penalty retry condition based on the request retry times, the initial retry time interval and the penalty time interval of the metadata request are determined based on the request retry times. The initial retry time interval is positively correlated with the request retry times of the metadata request. According to the initial retry time interval and the penalty time interval, the target retry time interval of the metadata request is determined, and the metadata request is resent to the server according to the target retry time interval. By converting the execution subject of the request retry mechanism from the server to the client, the occupancy rate of the computing resources and memory space of the server when executing the retry mechanism can be effectively reduced. At the same time, by matching the metadata request with the retry time interval that meets the requirements of the metadata request through the request retry times of the metadata request, and applying the retry penalty time, the invalid request pressure on the server by the client during the metadata request retry process can be further reduced.

[0074] In some embodiments, Figure 3 As shown, the metadata request retry method further includes the following steps:

[0075] S302, obtaining the total number of requests for sending metadata requests to the server within a preset sending period.

[0076] The total number of requests refers to the total number of metadata requests sent by the client to the server within the preset sending period, including the number of times the request is successful and the number of retries due to request failure. The total number of requests can be obtained by the client's built-in request number statistics module.

[0077] Specifically, after obtaining the request retry count of the metadata request, the client may obtain the total request count of metadata requests sent by the client to the server within a preset sending cycle.

[0078] S304: Determine the ratio of the number of request retries to the total number of requests as the request abnormality ratio.

[0079] Specifically, the client may calculate the ratio of the number of request retries to the total number of requests, and determine the ratio as the request anomaly ratio. The request anomaly ratio may represent the overall anomaly situation in which an abnormality occurs when the client sends a metadata request.

[0080] S306: When the request abnormality ratio is greater than or equal to the penalty ratio threshold, determine that the metadata request satisfies the penalty-retry condition.

[0081] Among them, the penalty ratio threshold is a preset threshold parameter used to determine whether the metadata request meets the penalty retry condition. The penalty ratio threshold can be determined by the designer based on experimental data or empirical data. If the request abnormality ratio is greater than or equal to the penalty ratio threshold, it means that the proportion of client request abnormalities is relatively high. The current distributed storage system has certain faults in metadata request processing, which may be caused by the client, server or network factors. That is, the abnormal reason for the failure of the metadata request may be at least one of software design defects, storage device failure, and network failure. In this case, if the request is still sent at the normal frequency and abnormal retries are made, a large number of metadata operation requests will be retried, resulting in a large amount of resource occupation and waste. Therefore, it is necessary to impose a certain penalty time on the client's request retry until the overall system returns to normal and the number of abnormal requests is significantly reduced. At this time, the client can determine that the metadata request meets the penalty retry condition.

[0082] Specifically, after calculating the request exception ratio, the client can compare the request exception ratio with the penalty ratio threshold. When the request exception ratio is greater than or equal to the penalty ratio threshold, it can be determined that the metadata request meets the penalty retry condition.

[0083] In the above embodiment, by calculating the request abnormality ratio of the number of request retries to the total number of requests, the proportion of request abnormalities in the current client can be accurately reflected. The preset penalty ratio threshold can quickly determine whether the proportion of request abnormalities is too high. When it is determined that the proportion of request abnormalities is too high, it is determined that the metadata request meets the retry condition with penalty, which effectively improves the accuracy and efficiency of the judgment of the retry condition with penalty.

[0084] In some of the embodiments, when the request exception ratio is less than a penalty ratio threshold, it is determined that the metadata request does not satisfy the retry with penalty condition.

[0085] In some embodiments, Figure 4 As shown, determining the initial retry time interval and penalty time interval of the metadata request based on the number of request retries in S204 includes:

[0086] S402: Determine a random jitter value and a basic unit time for time interval calculation according to the number of request retries.

[0087] Random jitter refers to the operation of introducing a random factor based on the waiting time to change the retry time interval, and the random jitter value is the factor value of the introduced random factor. By introducing the random jitter value when calculating the initial retry time interval, the avalanche effect caused by the backlog of requests can be prevented to a certain extent. That is, by introducing random jitter, the peak of the retry request can be dispersed to a relatively constant rate, reducing the burst traffic in the network and improving the overall performance of the system.

[0088] The basic unit time is the basic waiting time interval in the exponential backoff algorithm, and represents the basic unit time of the client request retry mechanism.

[0089] Specifically, after obtaining the number of request retries, the client can determine the random jitter value and basic unit time for time interval calculation according to the number of request retries, thereby providing a data basis for subsequent execution of the improved exponential backoff algorithm.

[0090] S404: Based on the number of request retries, the random jitter value, the basic unit time, and the preset interval growth amplitude value, an improved exponential backoff algorithm is used to determine an initial retry time interval of the metadata request.

[0091] Among them, the exponential backoff algorithm is an error recovery and retry strategy used in network requests or data transmission. When network transmission errors or congestion occur, the algorithm reduces conflicts and retry loads by gradually increasing the time interval between retry requests, thereby improving the stability and efficiency of the system. The improved exponential backoff algorithm incorporates the advantages of equal time interval and exponential backoff algorithms, and introduces an exponential backoff algorithm with random jitter.

[0092] The interval growth amplitude value is the base of the exponential backoff algorithm, which determines the interval growth amplitude. It can represent the increase multiple of the waiting time before the next request relative to the previous one after each request failure, usually based on an exponential growth of a fixed base. The interval growth amplitude value can be preset by the designer. For example, the designer can determine the interval growth amplitude value from the interval growth amplitude selection range, and the selection range can be (1,2].

[0093] Specifically, the client may use an improved exponential backoff algorithm to determine an initial retry time interval for the metadata request based on the obtained request retry times, random jitter value, basic unit time, and a preset interval growth amplitude value.

[0094] In some embodiments, the formula for the improved exponential backoff algorithm is as follows:

[0095]

[0096] Among them, T represents the initial retry time interval, N represents the number of requests for metadata requests to be retried. When the number of metadata requests due to abnormal failures continues to accumulate, the retry time interval will increase exponentially, which can effectively reduce the request pressure on the server. d is the interval growth amplitude value, which is the base of the exponential backoff algorithm and determines the amplitude of the interval growth. The value range is usually (1,2]. r is the random jitter value. Adding random jitter can prevent the avalanche effect and ensure that the time interval between each retry increases. τ represents the basic unit time of the request retry mechanism.

[0097] S406: Determine a penalty time interval for metadata requests according to the request abnormality ratio.

[0098] Specifically, the client may determine the penalty time interval for metadata requests according to the request abnormality ratio.

[0099] In some of the embodiments, the client may determine the penalty time interval corresponding to the request abnormality ratio by searching a preset mapping relationship.

[0100] In the above embodiment, the client determines the initial retry time interval of the metadata request by using an improved exponential backoff algorithm, which can prevent the avalanche effect while ensuring that each retry time interval is increasing, effectively reducing the number of impacts of invalid requests on the service and reducing the pressure on the server.

[0101] In some embodiments, Figure 5 As shown, S402, determining the random jitter value and the basic unit time for time interval calculation according to the number of request retries, includes:

[0102] S502: Randomly select an initial jitter value from a preset random jitter value range.

[0103] The initial jitter value refers to a value randomly selected from a preset random jitter value range, and the randomness of the random jitter value can be ensured by random selection. The preset random jitter value range can be set by the designer based on experimental data or empirical data. For example, (0, 0.2] can be determined as the preset random jitter value range.

[0104] Specifically, the client may randomly select a value from a preset random jitter value range as the initial jitter value.

[0105] S504, assigning a positive or negative direction to the initial jitter value according to the parity of the request retry times, to obtain a random jitter value for time interval calculation.

[0106] The parity of the request retry count is used to characterize whether the request retry count is an odd number or an even number. The initial jitter value is given a positive or negative direction by the parity of the request retry count, which means that when the request retry count is an odd number, the initial jitter value is determined to be a positive value, and when the request retry count is an even number, the initial jitter value is determined to be a negative value.

[0107] Specifically, the client may assign a positive or negative direction to the initial jitter value according to the parity of the number of request retries, and obtain a random jitter value for time interval calculation.

[0108] Taking the preset random jitter value range of (0, 0.2] as an example, when the request is retried for the first time, the corresponding request retry number is an odd number, and the client can take a random value between [80%, 100%) as the random jitter value. When the request is retried for the second time, the corresponding request retry number is an even number, and the client can take a random value between (100%, 120%) as the random jitter value. When the request is retried for the third time, the corresponding request retry number is an odd number again, and the client can take a random value between [80%, 100%) as the random jitter value, and so on, alternating in sequence.

[0109] S506: Determine a basic unit time for calculating the time interval based on the number of request retries, a preset interval increase value, and a maximum retry time.

[0110] The maximum retry time is a preset waiting time limit for metadata request retries. Setting the maximum retry time can effectively prevent excessive waiting time, avoid slow client response, and affect the performance of the application layer.

[0111] Specifically, the client may determine a basic unit time for calculating the time interval based on the number of request retries, a preset interval growth amplitude value, and a maximum retry time.

[0112] In some embodiments, the calculation formula of the basic unit time is as follows:

[0113]

[0114] Among them, τ represents the basic unit time of the request retry mechanism, d represents the interval growth amplitude value, and t represents the maximum retry time.

[0115] In the above embodiment, the initial jitter value is assigned a positive or negative direction according to the parity of the number of request retries, which can prevent the hidden dangers that may be caused by the same basic waiting time for adjacent retries, further avoid the avalanche effect, and ensure that the waiting time interval of each retry request is incremental, and the basic waiting time for the odd-numbered and next even-numbered retries is the same, which is similar to a retry mechanism with equal time intervals. It can improve the possibility of successful client retry requests to a certain extent, and for scenarios where there will be no continuous failures occasionally, it can ensure the client's request response speed. At the same time, the basic unit time is calculated in combination with the geometric series summation formula. When the retry time interval is calculated based on the improved exponential backoff result, the maximum number of retries can be completed just within the longest retry time according to the average expectation.

[0116] In some embodiments, S406, determining the penalty time interval for metadata requests based on the request anomaly ratio includes: obtaining a preset penalty factor for the retry penalty mechanism, and multiplying the preset penalty factor by the request anomaly ratio to determine the penalty time interval for metadata requests.

[0117] Among them, the preset penalty factor is a preset parameter used to quantify and determine the degree of penalty, which can be pre-formulated by the designer according to specific client requirements and configured in the client.

[0118] Specifically, the client may obtain a preset penalty factor of the retry penalty mechanism, and determine the product of the preset penalty factor and the request abnormality ratio as the penalty time interval for the metadata request.

[0119] In the above embodiment, the preset penalty factor stipulates the penalty baseline value, and then determines the penalty time interval that matches the request abnormality of the current client based on the request abnormality ratio, which effectively improves the matching degree between the penalty time interval and the request abnormality of the current client, thereby improving the accuracy of determining the penalty time interval.

[0120] In some embodiments, the metadata request retry method further includes: when the request exception ratio is less than a penalty ratio threshold, determining that the metadata request does not meet the retry condition with penalty, and determining an initial retry time interval of the metadata request determined based on the number of request retries as a target retry time interval of the metadata request.

[0121] Specifically, if the request exception ratio is less than the penalty ratio threshold, it means that the proportion of client request exceptions is not high. The abnormal reasons for the failure of the metadata request may be server busy, resource competition, concurrent lock conflict, etc. The abnormality caused in this case can usually be recovered in a short period of time and does not require human intervention, but the specific recovery time is uncertain. Therefore, there is no need to impose a certain penalty time on the client's request retry. The client can determine the initial retry time interval of the metadata request based on the number of request retries, and determine the initial retry time interval as the target retry time interval of the metadata request. It can be understood that the determination step of the initial retry time interval is basically the same as the determination step in the above embodiment, and will not be repeated here.

[0122] In the above embodiment, when the request exception ratio is less than the penalty ratio threshold, directly determining the initial retry time interval as the target retry time interval for the metadata request can reduce the invalid request pressure on the server by the client during the metadata request retry process.

[0123] In the prior art, usually, a server will process metadata requests from multiple clients. If a retry mechanism led by the server is adopted, it is usually necessary to use structures such as queues. The server needs to consume additional computing resources and memory space to maintain the retry mechanism, which increases the pressure on the server. The retry method led by the client is not only simple and easy to implement, but also can effectively disperse the operational pressure and resource consumption caused by request retries, especially in scenarios such as one write and multiple reads, asymmetric distributed storage systems, etc., which can effectively reduce the load pressure on the server, thereby ensuring that the server provides efficient and reliable metadata processing performance, and has strong robustness in abnormal scenarios.

[0124] The metadata request processing module in the client is responsible for both the generation and sending of requests, as well as the reception and parsing of request return messages. Generally speaking, there are two situations in which metadata requests need to be retried: one is that the request return information parses the request processing failure; the other is that the request processing is considered to have timed out if the request return message is not received after the expected waiting time has passed after the request was sent. In a distributed storage system, it is inevitable to encounter problems such as communication delays, busy servers, resource contention, concurrent lock conflicts, software design defects, network failures, and storage device failures. These problems may lead to the above two situations and cause metadata operation requests to be retried. At the same time, in large-scale distributed storage systems, metadata operation request retry is a common and frequent scenario.

[0125] However, in a retry mechanism led by the client, it is challenging to calculate the timing of request retries (or the time interval between request retries) when a request is abnormal. From the client's perspective, it is impossible to fully understand the status of the server and it is difficult to control whether the current server can correctly handle the retry request.

[0126] At present, the retry mechanism dominated by the client usually sets the maximum number of retries or the longest retry time, and then initiates retry requests in equal time intervals. Under this retry mechanism, if the retry time interval is too short, it will not only cause frequent returns to previous abnormal problems, such as concurrent resource contention, network failure, storage hardware failure, etc., but also cause frequent request pressure on the server, and even affect other metadata operation requests that could have been executed normally; if the retry time interval is too long, it will cause the client to respond too slowly, affecting the performance of the application layer; at the same time, retrying at equal time intervals will also cause multiple retry tasks of multiple clients to be sent to the server at the same time in certain scenarios, which may cause an avalanche effect due to excessive load pressure, such as scheduled tasks set by multiple clients, server failures causing all requests within a period of time to fail, network congestion causing a large number of requests to arrive at the server at the same time, etc.

[0127] Another common method is to use an exponential backoff algorithm. The waiting time for the nth request is randomly selected from 0 to 2^(n-1) basic waiting time intervals (τ). The exponential backoff algorithm aims to increase the retry time interval with the number of retries to reduce the number of invalid retries of requests that are prone to failure on the server. At the same time, the introduction of random factors can prevent the avalanche effect caused by request backlogs to a certain extent. However, the random factors in the calculation results of the algorithm are too large and uncontrollable. It is possible that the waiting time for the next request is shorter than the previous request, or that the waiting time for the request retry is too long, resulting in poor client performance. On the other hand, the exponential backoff algorithm uses an important parameter - the basic waiting time interval. If a large number of requests arrive in a short time and fail to retry due to network congestion and other reasons, these requests are likely to be retried again at the same time at the τth, 2τth or 3τth... time point in the future, which poses certain risks.

[0128] In order to solve the above problems, in some embodiments, Figure 6 As shown, a metadata request retry method is provided, and the method specifically includes the following steps:

[0129] First, when a metadata request fails, the client determines the reason for the metadata request failure. If the failure reason is that the request timed out and did not return, it means that the client has been waiting for a long time and the abnormal reason is unclear. It is very likely caused by a network problem. The client should immediately initiate a retry request without calculating the retry waiting time interval.

[0130] If the failure reason is not due to request timeout, the client can obtain the number of request retries N of the metadata request within the preset sending cycle, and the total number of requests n that the client sends to the server for metadata requests. t , the total number of requests n t The ratio of the request anomaly ratio to the number of request retries N is determined as the request anomaly ratio. The request anomaly ratio is compared with the penalty ratio threshold. When the request anomaly ratio is greater than or equal to the penalty ratio threshold, it means that the proportion of client request anomalies is relatively high. The client can determine that the cause of the metadata request failure is a software design defect, storage device failure, network failure, etc. In this case, the overall system has some faults and it is difficult to recover in a short time without human intervention. If requests are still sent at a regular frequency and retried abnormally, a large number of metadata operation requests will be retried, resulting in a large amount of resource occupation and waste. Therefore, it is necessary to impose a certain penalty time on the client's request retries until the overall system returns to normal and the proportion of abnormal requests is significantly reduced.

[0131] Specifically, the client can obtain the preset interval growth amplitude value d and the maximum retry time t, randomly select the initial jitter value from the preset random jitter value range, assign positive and negative directions to the initial jitter value according to the parity of the request retry number N, and obtain the random jitter value r. The basic unit time τ is calculated based on the request retry number N, the maximum retry time t and the interval growth amplitude value d. The client can use the improved exponential backoff algorithm to calculate the initial retry time interval T of the metadata request based on the request retry number N, the interval growth amplitude value d, the random jitter value r and the basic unit time τ.

[0132] The improved exponential backoff algorithm proposed in this embodiment has the following characteristics:

[0133] First, the basic waiting time for the odd-numbered request retry and the next even-numbered request retry is the same, which is similar to the retry mechanism with equal time intervals. It can improve the possibility of successful client retry requests to a certain extent, and can ensure the client's request response speed for scenarios where there will be occasional continuous failures.

[0134] Second, the introduced random jitter has a negative value for odd-numbered retries and a positive value for even-numbered retries, which not only prevents the hidden dangers that may be caused by the same basic waiting time for adjacent retries, further avoiding the avalanche effect, but also ensures that the waiting time interval for each retry request is incremental.

[0135] Third, overall, the retry time of a request increases exponentially with the number of retries. This can effectively reduce the number of impacts of invalid requests on the service when requests are unlikely to succeed in certain failure scenarios.

[0136] For example, the longest retry time t is 10 seconds, the number of request retries N is 10 times, the base d of the exponential backoff algorithm is 1.6, and the calculated basic unit time τ of the retry mechanism is about 0.316 seconds. The initial retry time interval is shown in the following table:

[0137]

[0138] As shown in the table above, according to the improved exponential backoff algorithm proposed in this embodiment, the cumulative expected average time taken is about 10 seconds. In order to more intuitively reflect the value range of each retry calculated, the retry time interval range of these 10 retries is as follows: Figure 7 shown.

[0139] After obtaining the initial retry time interval T, the client can obtain the preset penalty factor P and determine the penalty time interval T' by multiplying the request abnormality ratio by the penalty factor P, that is:

[0140] ;

[0141] After obtaining the initial retry time interval T and the penalty time interval T', the client can determine the sum of the initial retry time interval T and the penalty time interval T' as the target retry time interval T p .

[0142] If the request exception ratio is less than the penalty ratio threshold, it means that the reason for the failure of the metadata request is that the server is busy, resource competition, concurrent lock conflict, etc. The exceptions caused by such situations can usually be recovered in a short period of time and do not require human intervention, but the specific recovery time is uncertain. Therefore, the client can obtain the preset interval growth amplitude value d and the maximum retry time t, randomly select the initial jitter value from the preset random jitter value range, and assign positive and negative directions to the initial jitter value according to the parity of the request retry number N to obtain the random jitter value r. The basic unit time τ is calculated based on the request retry number N, the maximum retry time t and the interval growth amplitude value d. Using the improved exponential backoff algorithm, the initial retry time interval T of the metadata request is calculated based on the request retry number N, the interval growth amplitude value d, the random jitter value r and the basic unit time τ, and the initial retry time interval T is determined as the target retry time interval T p .

[0143] The above embodiment proposes a metadata request retry method led by the client, which is simple and easy to implement. Compared with the existing common server-side retry method relying on queues, it does not require complex data structures and software logic, reduces the consumption of computing resources and memory space on the server, and enables the server to provide more efficient and reliable metadata management and operation performance. When performing the initial retry time interval calculation, an improved exponential backoff algorithm is used, and random jitter is introduced, which can not only ensure that the retry time interval increases exponentially with the increase in the number of request retries, but also avoid the avalanche effect caused by a large number of request pressures at the same time. At the same time, the proportion of request anomalies is counted in the client, and the retry penalty time is applied accordingly, so that the occupation and waste of resources can be reduced in the event of a system failure.

[0144] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a metadata request retry device for implementing the metadata request retry method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more metadata request retry device embodiments provided below can refer to the limitations of the metadata request retry method above, and will not be repeated here.

[0146] In some embodiments, Figure 8 As shown, a metadata request retry device 800 is provided, comprising: a request retry times acquisition module 801, a condition judgment module 802, a target retry time interval determination module 803 and a request retry module 804, wherein:

[0147] The request retry times acquisition module 801 is used to acquire the request retry times of the metadata request within a preset sending cycle when receiving the request return information sent by the server in response to the metadata request.

[0148] The condition judgment module 802 is used to determine the initial retry time interval and the penalty time interval of the metadata request based on the number of request retries if it is determined that the metadata request meets the penalty retry condition based on the number of request retries; the initial retry time interval is positively correlated with the number of request retries.

[0149] The target retry time interval determination module 803 is used to determine the target retry time interval of the metadata request according to the initial retry time interval and the penalty time interval.

[0150] The request retry module 804 is used to resend the metadata request to the server according to the target retry time interval.

[0151] In some embodiments, the metadata request retry means further comprises:

[0152] The total request times acquisition module is used to acquire the total request times of sending metadata requests to the server within a preset sending cycle.

[0153] The request anomaly ratio determination module is used to determine the request anomaly ratio by taking the ratio of the number of request retries to the total number of requests.

[0154] The threshold comparison module is used to determine that the metadata request meets the retry condition with penalty when the request abnormality ratio is greater than or equal to the penalty ratio threshold.

[0155] In some embodiments, the conditional judgment module 802 is used to: determine the random jitter value and basic unit time for time interval calculation based on the number of request retries; determine the initial retry time interval of the metadata request using an improved exponential backoff algorithm based on the number of request retries, the random jitter value, the basic unit time and a preset interval growth amplitude value; determine the penalty time interval of the metadata request based on the request abnormality ratio.

[0156] In some embodiments, the conditional judgment module 802 is used to: randomly select an initial jitter value from a preset random jitter value range; assign a positive or negative direction to the initial jitter value according to the parity of the number of requested retries, and obtain a random jitter value for time interval calculation; based on the number of requested retries, the preset interval growth amplitude value and the longest retry time, determine the basic unit time for time interval calculation.

[0157] In some embodiments, the condition determination module 802 is used to: obtain a preset penalty factor of the retry penalty mechanism; and determine the product of the preset penalty factor and the request abnormality ratio as the penalty time interval of the metadata request.

[0158] In some embodiments, the target retry time interval determination module is further used to: determine that the metadata request does not meet the retry condition with penalty when the request exception ratio is less than the penalty ratio threshold; and determine the initial retry time interval of the metadata request determined based on the number of request retries as the target retry time interval of the metadata request.

[0159] Each module in the metadata request retry device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0160] In some embodiments, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as request return information, request retry times, retry conditions with penalties, initial retry time interval, penalty time interval, target retry time interval, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a metadata request retry method is implemented.

[0161] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the specific steps of the above-mentioned metadata request retry method embodiment are implemented.

[0163] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the specific steps of the above-mentioned metadata request retry method embodiment are implemented.

[0164] In some embodiments, a computer program product is provided, including a computer program, which implements the specific steps of the above-mentioned metadata request retry method embodiment when executed by a processor.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. In addition, the acquisition, storage, processing, and transmission of data are in compliance with relevant laws and regulations.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A metadata request retry method, characterized in that: The method comprises: Upon receiving the request response information sent by the server in response to the metadata request, obtaining the number of request retries of the metadata request within a preset sending cycle; If it is determined that the metadata request satisfies the retry condition with penalty according to the request retry times, then an initial retry time interval and a penalty time interval of the metadata request are determined based on the request retry times; the initial retry time interval is positively correlated with the request retry times; Determining a target retry time interval for the metadata request according to the initial retry time interval and the penalty time interval; Resend the metadata request to the server according to the target retry time interval.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining the total number of requests for sending metadata requests to the server within the preset sending period; Determine the ratio of the number of request retries to the total number of requests as the request abnormality ratio; When the request exception ratio is greater than or equal to the penalty ratio threshold, it is determined that the metadata request satisfies the penalty-retry condition.

3. The method according to claim 2, characterized in that The determining, based on the number of request retries, an initial retry time interval and a penalty time interval of the metadata request includes: Determine a random jitter value and a basic unit time for calculating a time interval according to the number of request retries; Based on the request retry count, the random jitter value, the basic unit time, and the preset interval growth amplitude value, an improved exponential backoff algorithm is used to determine an initial retry time interval of the metadata request; A penalty time interval for the metadata request is determined according to the request abnormality ratio.

4. The method according to claim 3, characterized in that The step of determining the random jitter value and the basic unit time for calculating the time interval according to the number of request retries includes: Randomly select an initial jitter value from a preset random jitter value range; According to the parity of the request retry times, assigning a positive or negative direction to the initial jitter value to obtain a random jitter value for time interval calculation; Based on the number of request retries, a preset interval increase value, and a maximum retry time, a basic unit time for calculating the time interval is determined.

5. The method according to claim 3, characterized in that: The step of determining the penalty time interval for the metadata request according to the request abnormality ratio includes: Get the preset penalty factor of the retry penalty mechanism; The product of the preset penalty factor and the request abnormality ratio is determined as the penalty time interval of the metadata request.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: When the request exception ratio is less than the penalty ratio threshold, determining that the metadata request does not satisfy the penalty-bearing retry condition; An initial retry time interval of the metadata request determined based on the number of request retries is determined as a target retry time interval of the metadata request.

7. A metadata request retry device, characterized in that: The device comprises: A request retry count acquisition module, configured to acquire the request retry count of the metadata request within a preset sending cycle upon receiving request return information sent by the server in response to the metadata request; a condition judgment module, configured to determine an initial retry time interval and a penalty time interval of the metadata request based on the number of request retries if it is determined that the metadata request satisfies a retry condition with a penalty according to the number of request retries; the initial retry time interval is positively correlated with the number of request retries; a target retry time interval determination module, configured to determine a target retry time interval of the metadata request according to the initial retry time interval and the penalty time interval; The request retry module is used to resend the metadata request to the server according to the target retry time interval.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images