Data traffic control method and device, electronic equipment, medium and program product

Abstract

The invention discloses a data traffic control method and device, electronic equipment, a medium and a program product, and relates to the technical field of computer networks and distributed storage, and the method comprises the steps: predicting the remaining time and pause time corresponding to a buffer area of a first storage node through the change rate of the buffer area of the first storage node; and under the condition that the remaining time is smaller than the time threshold value, selecting a target sending end from a plurality of sending ends corresponding to the source distribution through the source distribution of the data traffic of the first storage node, and indicating the target sending end to suspend sending the data traffic to the first storage node based on the pause time. According to the invention, the problem of inaccurate processing of the network congestion of the distributed storage system in the prior art is solved, and the effect of predicting and accurately processing the network congestion in the distributed storage system in advance is realized.

Description

Technical Field

[0001] The present application relates to the field of computer networks and distributed storage technology, and in particular to a data flow control method, device, electronic device, medium and program product. Background Art

[0002] In distributed storage systems, Remote Direct Memory Access (RDMA) technology enables low-latency, high-throughput data transmission through RDMA over Converged Ethernet (RoCE) networks. RoCE networks collaborate to achieve lossless transmission through Layer 2 Priority Flow Control (PFC) and Layer 3 Data Center Quantized Congestion Notification (DCQCN), but existing mechanisms have significant limitations.

[0003] PFC, a Layer 2 congestion control mechanism, relies on passively triggering pauses based on fixed thresholds and cannot predict congestion in advance. This results in tail latency as high as 820μs in 16:1 many-to-one injection (Incast) scenarios, and also presents head-of-line blocking and traffic unfairness. DCQCN relies on Explicit Congestion Notification (ECN) marking and explicit rate control, but feedback delays lead to a lengthy control loop and prevent timely congestion suppression. Inadequate coordination between these two mechanisms results in distributed storage networks experiencing buffer peaks reaching 100% in Incast scenarios, with tail latency fluctuations exceeding ±50%. More precise congestion control mechanisms are urgently needed.

[0004] Regarding the related art, the problem of inaccurate handling of network congestion of distributed storage systems has not yet been effectively solved. Summary of the Invention

[0005] The present application provides a data flow control method, device, electronic device, medium and program product to at least solve the problem of inaccurate handling of network congestion of distributed storage systems in related technologies.

[0006] The present application provides a method for controlling data traffic, including: predicting the remaining time and pause time corresponding to the buffer of the first storage node through the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than a time threshold; when the remaining time is less than the time threshold, selecting a target sending end from multiple sending ends corresponding to the source distribution of the data traffic of the first storage node through the source distribution, and instructing the target sending end to pause sending data traffic to the first storage node based on the pause time.

[0007] The present application also provides a data traffic control device, comprising: a prediction module, configured to predict the remaining time and pause time respectively corresponding to the buffer of the first storage node through the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than a time threshold; an indication module, configured to select a target sending end from a plurality of sending ends corresponding to the source distribution of the data traffic of the first storage node through the source distribution when the remaining time is less than the time threshold, and instruct the target sending end to pause sending data traffic to the first storage node based on the pause time.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data flow control methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data flow control methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data flow control methods when executed by a processor.

[0011] Through this application, the remaining time and pause time corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than the time threshold; when the remaining time is less than the time threshold, the target sending end is selected from the multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, and the target sending end is instructed to pause sending data traffic to the first storage node based on the pause time. Therefore, the technical problem of inaccurate processing of network congestion of distributed storage systems in related technologies can be solved, and the technical effect of pre-predicting and accurately processing network congestion in distributed storage systems can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a hardware structure block diagram of a computer terminal for a data flow control method according to an embodiment of the present application;

[0014] Figure 2 is a flow chart of a method for controlling data flow according to an embodiment of the present application;

[0015] Figure 3 is an architectural diagram of a distributed storage system according to an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of module interaction according to an embodiment of the present application;

[0017] Figure 5 It is a structural block diagram of a data flow control device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data flow control method depends, the specific application environment architecture or specific hardware architecture is described here.

[0022] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal of a data flow control method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0023] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data flow control method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0024] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0025] Figure 2 This is a flow chart of a method for controlling data flow according to an embodiment of the present application, which is applied to a storage node of a distributed storage system. Figure 2 As shown, the process includes the following steps:

[0026] Step S202: predicting the remaining time and pause time corresponding to the buffer of the first storage node respectively based on the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic if the remaining time is less than a time threshold;

[0027] Step S204, when the remaining time is less than the time threshold, select a target sending end from multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, and instruct the target sending end to pause sending data traffic to the first storage node based on the pause time.

[0028] The first storage node is an object storage device (OSD).

[0029] For example, when the remaining time is less than the time threshold of 5ms, it is determined that there is a potential congestion risk in the buffer of the first storage node and intervention is required. The intervention means is to instruct the target sending end to pause sending data traffic to the first storage node based on the pause time.

[0030] Through the above steps, the remaining time and pause time corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than the time threshold; when the remaining time is less than the time threshold, the target sending end is selected from the multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, and the target sending end is instructed to pause sending data traffic to the first storage node based on the pause time. Therefore, the technical problem of inaccurate processing of network congestion of distributed storage systems in related technologies can be solved, and the technical effect of pre-predicting and accurately processing network congestion in distributed storage systems can be achieved.

[0031] An embodiment of the present application provides a method for controlling data traffic, and the method is described in detail in conjunction with the execution flow of the method for controlling data traffic.

[0032] In an exemplary embodiment, the remaining time and pause time respectively corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node, and the method further includes: continuously sampling the occupancy size of the buffer according to a sampling time interval to obtain a plurality of sampling data pairs, wherein each sampling data pair includes: the occupancy size of the buffer at two moments before and after the sampling time interval; and determining the buffer change rate through the multiple sampling data pairs.

[0033] Furthermore, the buffer change rate is determined by the multiple sampling data pairs, including: calculating the target buffer change rate corresponding to each sampling data pair through each sampling data pair and the sampling time interval; inputting multiple consecutive target buffer change rates into a sliding window to obtain the buffer change rate.

[0034] Target buffer change rate The calculation formula is:

[0035] .

[0036] in, is the sampling time interval, which can be 10ms; is the size of the buffer occupied at time t, The size of the buffer occupied after the sampling interval at time t. Generally, the sampling is continuous, for example, sampling at t=10ms, t=20ms, and t=30ms.

[0037] Buffer change rate The calculation formula is:

[0038] .

[0039] in, Indicates the number of sampled data pairs (i.e., the number of input target buffer change rates), for example, Represents a sliding window of 3 sampling time points.

[0040] The above embodiment, by periodically sampling the buffer occupancy and calculating the rate of change, can promptly capture the growth trend of buffer usage over time. Using a sliding window algorithm to smooth the sampled data can filter out instantaneous fluctuations and obtain a more stable and reliable buffer change rate. This helps to identify potential network congestion in advance, allowing preventive measures to be taken before congestion occurs.

[0041] Furthermore, after determining the buffer change rate through the multiple sampling data pairs, the method also includes: determining whether the buffer is in an overflow state after a first target time through the buffer change rate; when the buffer is in an overflow state after the first target time, instructing the multiple sending ends to reduce the speed of sending data traffic at the notification time, wherein the notification time is separated from the current time by a second target time, and the second target time is less than 1 / 2 of the first target time.

[0042] For example, if the calculated buffer change rate is 0.67MB, and this rate of change is maintained, the buffer will increase by 0.67 × 100 = 67MB after the first target time of 100ms. This, combined with the current 45MB, will cause an overflow (assuming a total capacity of 100MB). In this case, an action can be triggered in advance (for example, notifying the sender to reduce the speed at notification time t = 40ms) to prevent an actual overflow.

[0043] In an exemplary embodiment, the remaining time and pause time respectively corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node, including: predicting the remaining time by the buffer change rate and the round-trip delay corresponding to the first storage node; and predicting the pause time by the buffer change rate and the safety factor.

[0044] Furthermore, the remaining time is predicted by the buffer change rate and the round-trip delay corresponding to the first storage node, including: determining the difference between the total size of the buffer and the occupied size of the buffer at the current moment, and determining the ratio of the difference to the buffer change rate; predicting the remaining time by the ratio and the round-trip delay.

[0045] Specifically, the remaining time The calculation formula is:

[0046] .

[0047] in, Indicates the total size of the buffer, Indicates the current occupied size. Indicates the round-trip delay, where the round-trip delay (RTT) is the round-trip time.

[0048] By monitoring the buffer change rate in real time and combining it with the round-trip delay of the storage node, the system can calculate the remaining time before the buffer reaches full capacity in advance, thereby predicting when congestion will occur.

[0049] Furthermore, predicting the pause time by the buffer change rate and the safety factor includes: determining the difference between the total size of the buffer and the occupied size of the buffer at the current moment, and determining the ratio of the difference to the buffer change rate; predicting the pause time by the ratio and the safety factor.

[0050] Furthermore, predicting the pause time using the ratio and the safety factor includes: determining a historical adjustment factor using the previous k historical adjustment data at the current moment, wherein the i-th historical adjustment data in the previous k historical adjustment data includes: the i-th predicted pause time and the actual time when the first storage node is uncongested for the i-th time, wherein k is a positive integer, i=1,2,3...k; and, calculating a target pause time using the ratio and the safety factor; and adjusting the target pause time using the historical adjustment factor to obtain the pause time.

[0051] Specifically, the target pause time The calculation formula is:

[0052] .

[0053] in, is the safety factor, the default value is 0.2.

[0054] Optional, taking k=5 as an example, historical adjustment factor The calculation method is:

[0055] .

[0056] in, is the pause time obtained by the i-th prediction, is the actual time when the first storage node is relieved of congestion for the i-th time.

[0057] Thus, the target pause time is adjusted by the historical adjustment factor to obtain the pause time, including: ;in, Indicates pause time.

[0058] Based on the buffer change rate and safety factor, the system dynamically predicts the optimal pause time, ensuring that data inflow is appropriately reduced before the buffer reaches a critical value, thus preventing buffer overflow. This prediction mechanism allows the system to make more precise and timely adjustments to network traffic, avoiding the traditional global passive response approach, minimizing the impact on innocent traffic, and improving the accuracy of flow control and the fairness of network transmission. This reduces the possibility of buffer overflows, reduces tail latency, improves data transmission efficiency and network resource utilization, and enhances the overall performance and reliability of the distributed storage system.

[0059] In an exemplary embodiment, a target sending end is selected from multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, including: determining the total source amount of the data traffic of the first storage node, and determining each source amount corresponding to each sending end; determining the traffic proportion of each sending end among the multiple sending ends by the ratio of each source amount to the total source amount, wherein the traffic proportion is used to indicate the source distribution; selecting a target traffic proportion from the multiple traffic proportions sorted in the target order, and using the sending end corresponding to the target traffic proportion as the target sending end.

[0060] It is understandable that the traffic share The calculation formula is: ;in, Indicates the amount of each source corresponding to each sender, Indicates the total source volume of data traffic from the first storage node. m is the total number of source OSDs sending traffic to the target OSD. Multiple senders include m source OSDs.

[0061] Arrange m source OSDs in descending order of traffic proportion according to the target order; select source OSDs from high to low until the cumulative selected traffic proportion exceeds 75%, and use the selected source OSD as the target sending end.

[0062] Specifically, by performing a detailed analysis of the data traffic received by the first storage node (OSD node), we can determine the total contribution of all senders to the target node's data traffic and identify the traffic source of each sender. By calculating the ratio of each sender's traffic source to the total source volume, i.e., the traffic share, we can gain insight into the distribution of data traffic, which helps identify the data flows that have the greatest impact on network congestion.

[0063] Furthermore, senders are ranked in target order based on their traffic share. The senders with the highest traffic share are selected as "target senders," and traffic control measures, such as suspension or rate limiting, are applied to them. Other senders are exempt from control. This approach avoids the traditional one-size-fits-all suspension of network-wide traffic, reduces interference with normal data transmission, and ensures the rational allocation and efficient use of network resources.

[0064] In an exemplary embodiment, the target sending end is instructed to suspend sending data traffic to the first storage node based on the pause time, including: a sending step: sending a preset data frame to the target sending end, wherein the preset data frame is used to instruct the target sending end to suspend sending data traffic to the first storage node; an updating step: calculating the fairness adjustment factor corresponding to each sending end in the target sending end, and updating the traffic proportion of each sending end in the multiple sending ends by the fairness adjustment factor, and updating the target sending end by the updated traffic proportion; the above-mentioned sending step and updating step are looped until the preset condition is met, and it is determined that the instruction to the target sending end to suspend sending data traffic to the first storage node based on the pause time has been completed, wherein the preset condition includes one of the following: the pause time ends, and it is detected that the remaining time is greater than or equal to the time threshold.

[0065] Furthermore, the fairness adjustment factor corresponding to each of the target sending ends is calculated, including: determining the total time that each sending end is suspended in the most recent statistical period; and determining the fairness adjustment factor by the ratio of the total time to the cycle time corresponding to the statistical period.

[0066] Among them, the preset data frame is the PFC pause frame, and the fairness adjustment factor The calculation formula is:

[0067] .

[0068] in, The total time that each sender (such as the i-th source OSD) is suspended in the most recent statistical period (such as the past minute); the statistical period .

[0069] Traffic share The adjustment formula is: .

[0070] Through the above-mentioned embodiments, dynamic, fair and efficient network traffic control is achieved, especially when facing network congestion in a distributed storage system. By sending a preset data frame to the target sender to indicate a pause, the system can respond immediately to the buffer congestion condition and avoid packet drops and increased transmission delays caused by buffer overflow. At the same time, the introduction of a fairness adjustment factor dynamically adjusts the traffic share of the sender, ensuring that in the long term, each sender will not suffer from unfair data transmission restrictions due to frequent pauses, thereby avoiding traffic starvation and enhancing the fairness of network traffic control. The sending and updating steps are executed cyclically until the preset conditions are met. This mechanism ensures the real-time and effectiveness of network congestion control. At the same time, by gradually relaxing the traffic restrictions, the system is allowed to quickly resume data transmission after the buffer returns to normal, thereby improving the response speed and overall performance of the distributed storage network.

[0071] In an exemplary embodiment, after instructing the target sending end to pause sending data traffic to the first storage node based on the pause time, the method further includes: in a case where there is data traffic to be stored in a client communicating with a distributed storage system including the first storage node, respectively calculating the buffer occupancy, network delay and load factor of the first storage node; performing a weighted summation of the buffer occupancy, the network delay and the load factor to determine the network congestion factor of the first storage node, and instructing the client to select a second storage node corresponding to the data traffic to be stored in the distributed storage system based on the network congestion factor.

[0072] Among them, the network congestion factor The calculation formula is as follows:

[0073] .

[0074] in, Indicates buffer occupancy, Indicates network delay, Indicates the load factor. Optional. , , . is the baseline round trip delay.

[0075] For example, a storage node with a network congestion factor less than 0.6 may be selected as the second storage node.

[0076] Furthermore, after performing a weighted summation of the buffer occupancy, the network delay and the load factor to determine the network congestion factor of the first storage node, the method also includes at least one of the following: instructing a monitoring node in a distributed storage system including the first storage node to perform shard migration on the first storage node when the network congestion factor is higher than a preset threshold; instructing the monitoring node to calculate the congestion status of multiple storage nodes in the distributed storage system through a congestion prediction model and multiple network congestion factors, and instructing the monitoring node to perform shard migration on the first storage node according to the congestion status.

[0077] The congestion prediction model is:

[0078] .

[0079] in, It is an indicator function to determine the congestion factor of a single OSD Whether it exceeds 0.8 (which can be understood as "the OSD is in a high-risk congestion state"), if the condition is met, it returns 1, otherwise it returns 0; is the total number of OSD nodes in the cluster, when the congestion probability When the congestion warning state is reached, rebalancing is triggered in advance, and the rebalancing period is dynamically adjusted: 5 minutes in normal state, 30 seconds in high congestion warning state, and 10 seconds in congestion state.

[0080] In an exemplary embodiment, the remaining time and pause time respectively corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node. The method also includes: determining the load status of the first storage node, wherein the load status includes: central processing unit usage rate; adjusting the total size of the buffer according to the load status, and determining the response strategy of the first storage node to the data requests of the multiple sending ends according to the load status.

[0081] In an exemplary embodiment, the total size of the buffer is adjusted according to the load status, including: when the load status is in a first adjustment interval, determining the total size of the buffer to be a first value; when the load status is in a second adjustment interval, determining the total size of the buffer to be a second value; when the load status is in a third adjustment interval, determining the total size of the buffer to be a third value; wherein the first value is greater than the second value, and the second value is greater than the third value; the maximum value in the first adjustment interval is less than the minimum value in the second adjustment interval, and the maximum value in the second adjustment interval is less than the minimum value in the third adjustment interval.

[0082] Among them, the Central Processing Unit (CPU).

[0083] That is to say, the first storage node dynamically adjusts the total size of the receiving buffer according to its own load status: the load status in the first adjustment interval is a low load state (CPU usage <30%): the buffer size is adjusted to 16MB; the load status in the second adjustment interval is a medium load state (CPU usage 30%-70%): the total size of the buffer is adjusted to 12MB; the load status in the third adjustment interval is a high load state (CPU usage >70%): the total size of the buffer is adjusted to 8MB.

[0084] In an exemplary embodiment, the response strategy of the first storage node to the data requests of the multiple senders is determined by the load status, including: when the load status is in a first response interval, determining that the data requests of the multiple senders are allowed to be received; when the load status is in a second response interval, determining that the remaining time and pause time corresponding to the buffers are allowed to be predicted; when the load status is in a third response interval, determining that a temporary unavailable state is returned to the multiple senders, and refusing to receive the data requests of the multiple senders.

[0085] That is to say, a buffer water level warning mechanism is introduced, and the buffer usage is divided into three response intervals: the first response interval is the safe zone (0%-70%): data is received normally; the second response interval is the warning zone (70%-90%): the prediction of the remaining time and pause time corresponding to the buffer is started, and the pause time is calculated according to the buffer derivative; the third response interval is the danger zone (90%-100%): new write requests are rejected and a temporary unavailable state is returned.

[0086] In order to better understand the process of the above-mentioned data flow control method, the implementation process of the above-mentioned data flow control method is described below in combination with optional embodiments, but it is not used to limit the technical solution of the embodiments of this application.

[0087] The technological development of distributed storage systems stems from the explosive growth in demand for massive data storage and processing. With the rise of technologies such as cloud computing, big data, and artificial intelligence (AI), traditional centralized storage systems, due to poor scalability and significant performance bottlenecks, are unable to cope with exabyte-level data volumes and high concurrent access. Distributed storage achieves linear scalability of storage capacity and performance by distributing data across multiple nodes and leveraging replica redundancy, load balancing, and distributed consensus mechanisms. The prevalence of high-speed network technologies such as RDMA has accelerated their application in low-latency scenarios such as financial transactions and real-time analytics. However, distributed storage systems still face challenges such as network congestion caused by incast traffic and maintaining consistency among multiple replicas.

[0088] In distributed storage systems, Remote Direct Memory Access (RDMA) technology enables low-latency, high-throughput data transmission over RoCE networks. However, "many-to-one" incast traffic patterns (such as data replication and cluster re-establishment) often cause switch buffer overflows. PFC, a Layer 2 congestion control mechanism, relies on fixed thresholds to passively trigger pauses and cannot predict congestion in advance. This results in tail latency as high as 820μs in 16:1 incast scenarios, and also presents head-of-line blocking and traffic unfairness. While end-to-end DCQCN mitigates PFC's shortcomings, its slow convergence still causes switch queues to accumulate during congestion, creating a vicious cycle of "delay and congestion," making it difficult to meet the dual requirements of distributed storage for lossless networks and low latency.

[0089] In other words, RoCE networks achieve lossless transmission through the collaboration of Layer 2 PFC and Layer 3 DCQCN, but the existing mechanism has significant limitations. PFC passively responds to switch buffer thresholds, pausing all traffic on the same input port, easily affecting innocent flows. Its coarse-grained control fails to distinguish the primary source of congestion, often blocking low-volume services in mixed traffic scenarios (such as the coexistence of small metadata input / output (IO) and large data block IO). DCQCN relies on ECN marking and explicit rate control, but feedback delays result in a long control loop and an inability to promptly suppress sudden congestion. This lack of coordination results in distributed storage networks experiencing buffer peaks reaching 100% in Incast scenarios, with tail latency fluctuating by more than ±50%, necessitating a more precise congestion control mechanism.

[0090] Specifically, to solve the network congestion problem between OSDs in distributed storage systems, the relevant technologies and solutions related to PFC and DCQCN mainly include solutions based on traffic scheduling and path optimization, including:

[0091] 1) Real-time identification mechanism for congested OSD nodes.

[0092] Multi-dimensional monitoring indicators. Buffer water level monitoring: The receive buffer occupancy rate is collected in real time through the OSD node kernel buffer (BufferPool) interface, and an alert is triggered when the threshold (such as 70%) is exceeded for three consecutive sampling cycles (1ms / cycle). Network delay detection: Using active detection packets (echo requests of the Internet Control Message Protocol (ICMP) are sent every 50ms). If the RTT between OSDs exceeds 150% (i.e. 3ms) of the baseline value (such as 2ms) for five consecutive times, it is marked as a suspected congestion node. Traffic anomaly detection: Analyze the traffic pattern between OSDs. When the inbound traffic rate of an OSD surges by more than 200% of the historical average within 10ms, and is accompanied by a PFC pause frame reception frequency ≥ 5 times / second, it is confirmed as a congested node.

[0093] 2) Dynamic path rerouting based on the Controlled Replication Under Scalable Hashing (CRUSH) algorithm.

[0094] Topology awareness and traffic redistribution. CRUSH map update: After the MON (Monitor) node obtains the list of congested OSDs, it triggers the CRUSH algorithm's incremental rebalancing (not full rebuild), remapping the replication stream originally destined for the congested OSD to the non-congested OSD. Shard migration strategy: Migrate at the data shard granularity, prioritizing unfinished replication tasks, and synchronizing new path information to all relevant OSDs through the osd map update command; Traffic scheduling priority: Set the priority of rerouted traffic lower than that of normal business traffic to avoid bandwidth preemption, and limit the migration bandwidth to 30% of the link capacity.

[0095] 3) Multi-level triggering and execution of load balancing.

[0096] Three-level trigger mechanism. Primary trigger: When the length of a single OSD receiving queue exceeds 1000 data packets (approximately 1MB), local traffic scheduling is initiated, adjusting the diversion ratio of other OSDs in the same rack. Intermediate trigger: When more than three OSDs meet the primary conditions simultaneously, cross-rack path rerouting is triggered, and traffic is distributed to OSDs in other racks through CRUSH. Advanced trigger: When more than 10% of OSDs in the entire network are congested, global load balancing is initiated, recalculating the storage location of all data shards, and enabling incremental replication mode when the migration volume exceeds 5% of the total data volume.

[0097] 4) Feedback and verification of congestion relief.

[0098] Real-time status feedback: Congested OSDs report buffer occupancy and RTT change curves to the MON node every 100ms. When these indicators return to within 120% of the baseline value for 10 consecutive cycles, congestion is marked as relieved. Path validity verification: Rerouted traffic must pass a 30-second stability verification. If the RTT fluctuation of the new path exceeds 20% or there are more than three PFC pauses, the original path is reverted and a new target OSD is selected.

[0099] However, solutions based on traffic scheduling and path optimization have the following drawbacks:

[0100] 1) Quantitative impact of scheduling delay: The minimum CRUSH rebalancing period is 100ms, which cannot cope with congestion bursts of 10ms (for example, when the incast traffic peak lasts less than 50ms, congestion occurs before scheduling is complete). 2) Specific manifestations of metadata overhead: Each incremental rebalancing requires updating the mapping table of all OSDs in the entire network. When the cluster size exceeds 500 OSDs, the control plane traffic for a single update can reach 20MB, occupying 10% of the management network bandwidth. 3) Typical scenarios for local optimization: In clusters where cross-rack traffic accounts for more than 40%, path adjustment alone cannot resolve congestion bottlenecks on core switches. For example, in a 16:1 incast scenario, rerouting may still cause queues to accumulate at the aggregation switch egress.

[0101] In order to solve the above-mentioned defects, the optional embodiment of the present application realizes the precise prevention and control of distributed storage network congestion by constructing a predictive flow control mechanism based on the RoCE network. The node-level buffer prediction model is innovatively introduced, and the buffer occupancy derivative is combined with the network round-trip delay to perceive the congestion risk 2-3ms in advance, which is much earlier than the traditional mechanism. The congestion source is accurately located based on the CRUSH mapping, and the suspended source OSD is dynamically selected according to the traffic proportion, which effectively avoids the innocent flow from being mistakenly suspended and ensures traffic fairness. At the same time, according to the characteristics of the storage business, the pause time is dynamically adjusted, and key processes such as data replication and cluster reconstruction are deeply integrated to ensure the continuity and stability of the storage business during network congestion, and comprehensively improve the overall performance of the distributed storage system in the RoCE network environment.

[0102] Specifically, to address the issues of traditional PFC fixed-threshold passive response leading to the inadvertent suspension of innocent flows and DCQCN feedback delays causing congestion spread, an optional embodiment of the present application constructs a node-level buffer derivative prediction model, leveraging the buffer occupancy change rate and network round-trip latency to perceive congestion risks 2-3ms in advance, responding more than 80% earlier than traditional mechanisms. It innovatively combines CRUSH mapping with traffic share analysis, dynamically selecting a pause source OSD based on 75% of the cumulative traffic share, and using a fairness adjustment factor to avoid long-term pauses in a single node, achieving precise traffic control. Through a four-layer architecture collaboration (client-layer CRUSH optimization selection, control plane predictive rebalancing, data plane adaptive buffer management, and network congestion control layer dynamic pauses), the system reduces tail latency from approximately 800μs to approximately 200μs in a 16:1 Incast scenario, increasing throughput by 23% while ensuring over 98% traffic fairness. This solves the problem of synergizing lossless transmission and low latency in distributed storage networks.

[0103] like Figure 3 As shown in the figure, the distributed storage system adopts a decentralized and distributed architecture. The core components include clients, monitoring nodes (Monitor), metadata servers (MDS), and OSDs. Each module works together through a high-speed network (such as Ethernet and InfiniBand) to achieve data storage and management. The architecture layer is described as follows:

[0104] 1) Client layer (Client).

[0105] Positioning: The entry point for users to interact with distributed storage, providing three interfaces: block, object, and file.

[0106] Core components include:

[0107] librado library: used to encapsulate the underlying Application Programming Interface (API) and is responsible for interacting with the cluster.

[0108] CRUSH Client Module: This module caches the CRUSH map locally and calculates data storage locations. Inputs include the object name and storage pool rules (such as the number of replicas and fault domain hierarchy). The processing steps are: hashing the object name to generate a random number. It then traverses the CRUSH bucket hierarchy (e.g., host -> cabinet -> data center) according to the storage pool rules and selects OSDs based on their weights. This ensures balanced data distribution and meets redundancy policies (e.g., deploying replicas across cabinets).

[0109] 2) Control plane (Monitor / MDS).

[0110] Monitor: Maintains cluster metadata (OSD Map, CRUSH Map, authentication information) and ensures multi-node consistency through the Paxos algorithm.

[0111] Interaction with the client: The client pulls the latest cluster map when it first connects, and then updates it through heartbeat synchronization.

[0112] MDS (File Scenario): Manages file system metadata (directory tree, permissions), and caches hot metadata to accelerate access.

[0113] 3) Data plane (OSD cluster).

[0114] OSD nodes: Hardware components include CPU, memory, disk (Hard Disk Drive (HDD) / Solid State Drive (SSD)), and network interface (Gigabit / 10Gbit network card). They are responsible for data storage, replication, erasure coding, and consistency maintenance, and perceive cluster topology through the OSD map.

[0115] That is to say, based on the traditional distributed storage system architecture, combined with the RoCE (RDMA over ConvergedEthernet) network and predictive flow control mechanism, the optional embodiment of this application redefines the data flow path and processing logic between the client and the OSD cluster, optimizes the IO process of the distributed storage system, and improves the high performance and reliability of the distributed storage system through low-latency RDMA transmission and precise congestion control. Figure 4 The overall process, key steps and algorithm model of the method are further elaborated in detail.

[0116] like Figure 4 As shown, the IO process of the optional embodiment of the present application is based on a four-layer architecture design, including the client layer, control plane layer, data plane layer and network congestion control layer. Each layer achieves low-latency communication through the RoCE network. The overall process can be divided into the following stages:

[0117] The first stage is the client request processing stage: the client application converts the IO request into an object operation through the librado library, and calculates the data storage location in combination with the CRUSH Map cache. The CRUSH algorithm is extended to introduce the network congestion factor. As the basis for OSD selection. The calculation formula is detailed in the following steps. During client initialization, the librados library automatically identifies RoCE-capable devices using the DCBx (Data Center Bridging Exchange) protocol, establishes an RDMA connection pool, and configures parameters such as the number of queue pairs (QPs) and send / receive queue depth for each connection, laying the foundation for low-latency transmission. Furthermore, the local CRUSH Map cache uses a predictive update strategy, dynamically adjusting the cache update cycle based on the cluster's historical update frequency to reduce interaction overhead with the control plane.

[0118] The second phase, the control plane interaction phase, involves clients interacting with the monitor cluster to obtain the latest cluster status and with the MDS (in file system scenarios) to obtain metadata. Upon initial client access, the client pulls the complete CRUSH map and cluster status information from the monitor. Subsequent updates are synchronized via the heartbeat mechanism, which is also transmitted over the RoCE network. In file system scenarios, the MDS uses RDMA read operations to retrieve metadata from the client. The MDS also features metadata prefetching, preloading relevant metadata based on client access patterns and dynamically adjusting metadata caching strategies based on network congestion.

[0119] The third phase, data plane transmission, involves clients transmitting data directly to OSD nodes via the RoCE network. After the primary OSD completes local storage, it replicates the data to the replica OSD over the RoCE network. Data transmission utilizes zero-copy RDMA operations. When copying data from the primary OSD to the replica OSD, a batch copy strategy is employed to reduce operational overhead. In data recovery scenarios, the recovery bandwidth is dynamically adjusted based on a predictive flow control mechanism, and a progressive recovery strategy is employed to prevent recovery traffic from impacting normal services.

[0120] The fourth stage, predictive congestion control, monitors network congestion status in real time throughout data transmission. Using a buffer derivative prediction model, it detects congestion risks 2-3ms in advance and dynamically adjusts the PFC pause time and target port to achieve precise congestion control. The predictive flow control module continuously collects data such as buffer status and network latency, performing real-time analysis and calculations. Once congestion risk is determined, it immediately generates a PFC pause frame, determines the target port, and sends it. Simultaneously, it monitors the pause effect and adjusts subsequent control strategies based on actual conditions, forming a closed-loop control system.

[0121] In summary, the optional embodiments of the present application achieve efficient collaboration of the entire process from client requests to data storage, transmission, and congestion control through optimized design of each stage, significantly improving the performance and reliability of the distributed storage system in the RoCE network environment.

[0122] The above four stages are explained in further detail below.

[0123] Furthermore, the client layer request processing and RoCE connection establishment process includes:

[0124] Step 1: Initialize the RoCE network and librados library.

[0125] When the distributed storage client application starts, the librado library automatically discovers RoCE-capable devices on the network through the DCBx protocol. It then establishes an RDMA connection pool. Each connection configuration parameter includes: Number of Queues (QPs): 16 by default, supporting concurrent RDMA operations; Send Queue Depth: 1024; Receive Queue Depth: 1024; Maximum RDMA Read Request Size: 64KB; Maximum RDMA Write Request Size: 4MB.

[0126] Step 2: Local CRUSH Map cache and predictive update.

[0127] When a client first connects to a cluster, it retrieves the complete CRUSH Map from the monitor and caches it in local memory. A predictive update mechanism is introduced, dynamically adjusting the request cycle based on historical update frequency: when the cluster is running stably (less than one update per hour), the update cycle is 10 minutes; when the cluster is actively changing (more than three updates per hour), the update cycle is shortened to two minutes. The local CRUSH Map cache size is limited to 10MB, and a least recently used (LRU) strategy is used to eliminate old entries.

[0128] Step 3: Network congestion-aware OSD selection.

[0129] Expanding the traditional CRUSH algorithm, because in distributed storage, the "congestion state" of OSD is affected by buffer occupancy, network delay, and load pressure, the optional embodiment of this application introduces a network congestion factor Important parameters for OSD selection:

[0130] ;

[0131] Among them: Buffer occupancy Directly reflects the "real-time pressure" of OSD data reception. When the buffer is close to full, new data writing is prone to trigger packet loss or delay. Network delay : Reflects the transmission efficiency from the client to the OSD. High latency means that the round trip of data takes a long time, and superimposed write operations can easily aggravate congestion. Load factor IOPS: This is a comprehensive measure of the number of read and write operations per second (IOPS) and bandwidth usage, reflecting the "busyness" of the OSD's processing capabilities. Excessive load will indirectly lead to network queue accumulation. , represents the buffer zone occupancy weight; , represents the network delay weight; The load factor weights prioritize buffer security (highest weight) to avoid direct packet loss due to buffer overflow. Secondly, network latency is considered to ensure transmission efficiency, and finally, load balancing is used to prevent local overload from spreading. This meets the distributed storage requirements of "preventing data loss first, ensuring transmission efficiency second, and ultimately global balance"; is the baseline round trip delay, the default is 2ms; is the average load factor of the cluster OSD, calculated as:

[0132] ;

[0133] Client-preferred congestion factor The OSD of the target storage node is used. When the congestion factors of all candidate OSDs exceed 0.6, the Monitor rebalancing process is triggered, that is, data migration scheduling occurs: according to the new CRUSH Map, "shard migration" is started between OSDs, and the primary OSD gradually transfers the copies of the overloaded shards to the low-congestion nodes. During the process, predictive flow control is used to avoid aggravating congestion caused by migration traffic (such as dynamically limiting migration bandwidth to prioritize business IO).

[0134] Compared with the related art CRUSH algorithm that relies on "static weight" to select OSD, and rebalancing is only triggered when OSD fails; the optional embodiment of this application is to Real-time congestion perception and proactive rebalancing are enabled, upgrading "passive fault recovery" to "active congestion prevention" to avoid global performance avalanche caused by the spread of local congestion.

[0135] Furthermore, control plane interaction and RoCE acceleration include:

[0136] In step 1, RDMA-accelerated monitor cluster communication is implemented. Monitor nodes communicate via RDMA over the RoCE network, reducing Paxos protocol message transmission latency from 50 μs on traditional Ethernet to 10 μs. A batch confirmation mechanism is introduced, collecting and confirming every 100 Prepare / Accept messages, reducing message exchanges and improving Paxos protocol efficiency.

[0137] Step 2: Predictive cluster status maintenance Monitor monitors the network congestion status of OSD nodes in real time and builds a congestion prediction model:

[0138] ;

[0139] in: It is an indicator function to determine the congestion factor of a single OSD Whether it exceeds 0.8 (which can be understood as "the OSD is in a 'high risk congestion' state"), if the condition is met, it returns 1, otherwise it returns 0; is the total number of OSD nodes in the cluster, when the congestion probability When the system is running, CRUSH Map rebalancing is triggered in advance, and the rebalancing period is dynamically adjusted: 5 minutes in normal state, 30 seconds in high congestion warning state, and 10 seconds in congestion state. Assume that the cluster has 100 OSDs: If 8 OSDs ,but (less than 0.1), rebalancing is not triggered and the check cycle is maintained for 5 minutes. ,but (greater than 0.1), CRUSH Map rebalancing is triggered immediately, and the subsequent check period is shortened to 30 seconds until the congestion is relieved.

[0140] Step 3: Integrate MDS with RoCE (file system scenario). RDMA read operations are implemented between MDS and clients over the RoCE network, reducing metadata read latency from 100 μs in traditional Ethernet to 20 μs.

[0141] A metadata prefetching mechanism is introduced. When a client accesses a directory ( / home / user), the metadata of its subdirectory ( / home / user / data) is automatically prefetched.

[0142] Network congestion-aware metadata caching strategy:

[0143] Normal network status: The metadata cache hit rate target is 80%.

[0144] Highly congested network conditions: The metadata cache hit rate target is increased to 90%.

[0145] The Least Frequently Used (LFU) cache replacement strategy is adopted to prioritize metadata with high access frequency.

[0146] Dynamic cache size adjustment: The base size is 2GB. When network congestion increases, the cache size increases by 100MB every 5 seconds, up to a maximum of 4GB.

[0147] Furthermore, data plane transmission and RoCE optimization include:

[0148] Step 1: The core goal of integrating the RoCE protocol stack with predictive flow control is to predict buffer congestion trends in advance during RoCE network communications on OSD nodes, avoiding packet loss and latency spikes caused by buffer overflows, and ensuring more stable data transmission. The OSD nodes deploy an enhanced RoCE protocol stack that adds a predictive flow control module to the traditional RoCE v2. The predictive flow control module monitors the receive buffer change rate in real time. , the calculation formula is:

[0149] ;

[0150] in: , is the sampling time interval; is the buffer occupancy size at time t; a sliding window averaging mechanism is introduced to reduce the impact of instantaneous fluctuations (such as misjudgment caused by sudden small traffic):

[0151] ;

[0152] in, , represents a sliding window of 3 sampling points. A positive rate of change indicates that the buffer is "continuously increasing in occupancy" (possibly congested); a negative rate of change indicates that the buffer is "releasing space" (the state is easing).

[0153] For example, assuming the total capacity of the OSD receive buffer is 100MB, the sampling interval is , sliding window :

[0154] First sampling (t=10ms): , the last time (t=0) was 25MB, then ;

[0155] Second sampling (t=20ms): , the last time (t=10) was 30MB, then ;

[0156] The third sampling (t=30ms): , the last time (t=20) was 38MB, then .

[0157] After sliding average: .

[0158] If this rate of change continues, the buffer will increase by 0.67 × 100 = 67MB after 100ms. Adding this to the current 45MB will cause an overflow (assuming a total capacity of 100MB). At this point, the predictive flow control module can trigger an action in advance (for example, notifying the sender to slow down at t = 40ms) to prevent an actual overflow. Essentially, this quantifies the "buffer change trend" through mathematical modeling, enabling OSD nodes to predict congestion. This is particularly well-suited for RoCE networks, which offer low latency, high bandwidth, and sensitivity to bursty traffic. It is a key component of distributed storage data plane optimization.

[0159] Step 2: Adaptive buffer management OSD process. The node dynamically adjusts the receiving buffer size according to its own load status: low load state (CPU usage <30%): buffer size is 16MB; medium load state (CPU usage 30%-70%): buffer size is 12MB; high load state (CPU usage >70%): buffer size is 8MB; introduce a buffer water level warning mechanism, and divide the buffer usage into three areas: safe zone (0%-70%): normal data reception; warning zone (70%-90%): start predictive flow control, calculate the pause time according to the buffer derivative; danger zone (90%-100%): reject new write requests and return to a temporarily unavailable state.

[0160] Step 3: Data replication and recovery between OSDs based on RoCE. RDMA-accelerated data replication: When the primary OSD sends data to a replica OSD, zero-copy RDMA writes are performed over the RoCE network, reducing data replication latency from 200μs on traditional Ethernet to 50μs. A batch replication strategy is used, replicating 16 objects (approximately 64MB of data) per batch, reducing RDMA operation overhead.

[0161] Furthermore, predictive congestion control mechanisms include:

[0162] Step 1: The buffer derivative prediction model builds a congestion prediction model based on the buffer occupancy rate and its change rate to calculate the remaining time. : in: is the total size of the buffer, The current buffer size. is the sliding window average buffer, and the rate of change RTT is the round-trip delay, which is measured by periodically sending probe packets. When a buffer overflows, it identifies it as a potential congestion risk and triggers predictive flow control. This approach calculates how long it will take for the buffer to fill up and then subtracts the impact of network latency. This allows the storage system to proactively intervene before the buffer overflows (for example, by instructing the client to send data more slowly), thus avoiding actual packet loss and latency spikes.

[0163] Step 2: Dynamic pause time calculation Calculate the PFC pause time based on the prediction model : ;in: Is the safety factor, the default value is 0.2, and the historical adjustment factor is introduced , dynamically adjusted based on the results of the previous 5 pauses: ;

[0164] in: is the i-th predicted congestion relief time, is the actual congestion relief time of the i-th time, and the final pause time is corrected to: .

[0165] Step 3: Accurately locate the congestion source and selectively pause the process. Based on the CRUSH mapping relationship, analyze the traffic source distribution and calculate the traffic share of each source OSD. :

[0166] Where: m is the total number of source OSDs sending traffic to the target OSD. A selective pause strategy is used to prioritize pausing source OSDs with a high traffic share: when a pause is required, source OSDs are sorted in descending order by traffic share; OSDs are selected from high to low until the cumulative traffic share exceeds 75%; PFC pause frames are sent to the selected source OSDs, and unselected source OSDs continue to transmit data, introducing a fairness adjustment factor , to avoid long-term suspension of the same source OSD:

[0167] ;

[0168] in: The total time that source OSD i was suspended in the past minute. , is the statistical period, and the final weight is selected Calculation formula: The fairness adjustment factor The calculation must meet the following conditions: When the source OSD has been paused within the past minute If it exceeds 30 seconds, Automatically increase it to above 0.5, lower the pause priority of the OSD, and avoid traffic starvation caused by long-term preemption.

[0169] By deeply integrating the low latency and high bandwidth characteristics of the RoCE network with the predictive flow control mechanism, the optional embodiment of the present application achieves precise prevention and control of distributed storage network congestion, significantly improving the performance and reliability of the system in high concurrency and high load scenarios, and providing a more efficient storage solution for large-scale data centers.

[0170] In summary, the RoCE-based predictive flow control method proposed in the optional embodiment of this application (equivalent to the data flow control method in the above embodiment) significantly improves the performance and stability of distributed storage networks through innovative congestion prediction and precise flow control mechanisms. Compared to traditional PFC and DCQCN mechanisms, this optional embodiment uses node-level buffer status prediction to detect congestion risks 2-3ms in advance, reducing tail latency in a 16:1 Incast scenario from 830μs to 274μs, improving response speed by over 80%, and effectively solving the buffer backlog problem caused by the passive response of traditional mechanisms.

[0171] In terms of traffic fairness, CRUSH mapping accurately locates congestion sources and dynamically selects pause targets based on traffic proportion, avoiding the impact of traditional global pauses on innocent traffic. This has resulted in measured traffic fairness exceeding 98%. Furthermore, through dynamic adjustment of pause times and adaptive buffer management, network resource utilization has increased from 60%-70% to 75%-85%, and throughput has increased by 23%, achieving a balance between lossless transmission and high bandwidth utilization.

[0172] Targeting distributed storage services, this solution deeply integrates processes such as data replication and cluster reconstruction, ensuring core business continuity during congestion through progressive bandwidth control and priority scheduling. In a 500-node cluster, performance fluctuations during fault recovery were less than 20%, and metadata update overhead was reduced by 50%, effectively improving system stability in large-scale deployments. This solution, through a four-layer architecture collaboration, overcomes the challenges of balancing low latency, lossless transmission, and fairness, providing an efficient congestion control solution for exabyte-scale data centers.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0174] This embodiment also provides a data flow control device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0175] Figure 5 is a structural block diagram of a data flow control device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0176] a prediction module 52 configured to predict a remaining time and a pause time corresponding to the buffer of the first storage node, respectively, based on a change rate of the buffer of the first storage node, wherein the remaining time indicates a time interval from a current moment to a moment when the buffer is filled, and the pause time indicates a time during which the first storage node needs to pause receiving data traffic if the remaining time is less than a time threshold;

[0177] The indication module 54 is used to select a target sending end from multiple sending ends corresponding to the source distribution of the data traffic of the first storage node when the remaining time is less than the time threshold, and instruct the target sending end to pause sending data traffic to the first storage node based on the pause time.

[0178] Through the above-mentioned device, the remaining time and pause time corresponding to the buffer of the first storage node are predicted by the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than the time threshold; when the remaining time is less than the time threshold, the target sending end is selected from the multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, and the target sending end is instructed to pause sending data traffic to the first storage node based on the pause time. Therefore, the technical problem of inaccurate processing of network congestion of distributed storage systems in related technologies can be solved, and the technical effect of pre-predicting and accurately processing network congestion in distributed storage systems can be achieved.

[0179] In an exemplary embodiment, the device also includes a determination module for predicting the remaining time and pause time corresponding to the buffer of the first storage node respectively through the buffer change rate of the first storage node, and continuously sampling the occupancy size of the buffer according to a sampling time interval to obtain multiple sampling data pairs, wherein each sampling data pair includes: the occupancy size of the buffer at two moments before and after the sampling time interval; and determining the buffer change rate through the multiple sampling data pairs.

[0180] In an exemplary embodiment, the determination module is further used to calculate the target buffer change rate corresponding to each sampling data pair through each sampling data pair and the sampling time interval; and input multiple consecutive target buffer change rates into a sliding window to obtain the buffer change rate.

[0181] In an exemplary embodiment, the device also includes a notification module for determining the buffer change rate through the multiple sampling data pairs, and then determining whether the buffer is in an overflow state after a first target time through the buffer change rate; if the buffer is in an overflow state after the first target time, instructing the multiple sending ends to reduce the speed of sending data traffic at the notification time, wherein the notification time is separated from the current time by a second target time, and the second target time is less than 1 / 2 of the first target time.

[0182] In an exemplary embodiment, the prediction module is further configured to predict the remaining time by using the buffer change rate and the round-trip delay corresponding to the first storage node; and to predict the pause time by using the buffer change rate and a safety factor.

[0183] In an exemplary embodiment, the prediction module is further used to determine the difference between the total size of the buffer and the occupied size of the buffer at the current moment, and determine the ratio of the difference to the buffer change rate; and predict the remaining time through the ratio and the round-trip delay.

[0184] In an exemplary embodiment, the prediction module is further used to determine the difference between the total size of the buffer and the occupied size of the buffer at the current moment, and determine the ratio of the difference to the buffer change rate; and predict the pause time through the ratio and the safety factor.

[0185] In an exemplary embodiment, the prediction module is further used to determine a historical adjustment factor based on the previous k historical adjustment data at the current moment, wherein the i-th historical adjustment data in the previous k historical adjustment data includes: the pause time obtained by the i-th prediction, and the actual time when the first storage node is uncongested for the i-th time, wherein k is a positive integer, i=1,2,3...k; and, calculate the target pause time through the ratio and the safety factor; and adjust the target pause time through the historical adjustment factor to obtain the pause time.

[0186] In an exemplary embodiment, the indication module is also used to determine the total source amount of data traffic of the first storage node, and determine each source amount corresponding to each sending end; determine the traffic proportion of each sending end among the multiple sending ends by the ratio of each source amount to the total source amount, wherein the traffic proportion is used to indicate the source distribution; select a target traffic proportion from the multiple traffic proportions sorted in the target order, and use the sending end corresponding to the target traffic proportion as the target sending end.

[0187] In an exemplary embodiment, the indication module is also used for a sending step: sending a preset data frame to the target sending end, wherein the preset data frame is used to instruct the target sending end to suspend sending data traffic to the first storage node; an updating step: calculating a fairness adjustment factor corresponding to each of the target sending ends, and updating the traffic proportion of each sending end among the multiple sending ends by the fairness adjustment factor, and updating the target sending end by the updated traffic proportion; looping the above-mentioned sending step and updating step until the preset condition is met, and determining that the target sending end has been instructed to suspend sending data traffic to the first storage node based on the pause time, wherein the preset condition includes one of the following: the pause time ends, and it is detected that the remaining time is greater than or equal to the time threshold.

[0188] In an exemplary embodiment, the indication module is further configured to determine the total time each transmitting end is suspended in a most recent statistical period; and determine the fairness adjustment factor by the ratio of the total time to the period time corresponding to the statistical period.

[0189] In an exemplary embodiment, the device also includes a congestion factor determination module, which is used to calculate the buffer occupancy, network delay and load factor of the first storage node respectively when there is data traffic to be stored in a client communicating with a distributed storage system including the first storage node; perform weighted summation of the buffer occupancy, the network delay and the load factor to determine the network congestion factor of the first storage node, and instruct the client to select a second storage node corresponding to the data traffic to be stored in the distributed storage system based on the network congestion factor.

[0190] In an exemplary embodiment, the congestion factor determination module is also used to instruct the monitoring node in the distributed storage system including the first storage node to perform shard migration on the first storage node when the network congestion factor is higher than a preset threshold; instruct the monitoring node to calculate the congestion status of multiple storage nodes in the distributed storage system through a congestion prediction model and multiple network congestion factors, and instruct the monitoring node to perform shard migration on the first storage node according to the congestion status.

[0191] In an exemplary embodiment, the device also includes a response strategy determination module, which is used to determine the load status of the first storage node by predicting the remaining time and pause time corresponding to the buffer of the first storage node through the buffer change rate of the first storage node, wherein the load status includes: central processing unit usage rate; adjusting the total size of the buffer according to the load status, and determining the response strategy of the first storage node to the data requests of the multiple sending ends according to the load status.

[0192] In an exemplary embodiment, the response strategy determination module is also used to determine that the total size of the buffer is a first value when the load state is in a first adjustment interval; determine that the total size of the buffer is a second value when the load state is in a second adjustment interval; and determine that the total size of the buffer is a third value when the load state is in a third adjustment interval; wherein the first value is greater than the second value, and the second value is greater than the third value; the maximum value in the first adjustment interval is less than the minimum value in the second adjustment interval, and the maximum value in the second adjustment interval is less than the minimum value in the third adjustment interval.

[0193] In an exemplary embodiment, the response strategy determination module is also used to determine, when the load state is in a first response interval, that data requests from the multiple senders are allowed to be received; when the load state is in a second response interval, that prediction of the remaining time and pause time corresponding to the buffers is allowed; and when the load state is in a third response interval, that a temporary unavailable state is returned to the multiple senders, and that the data requests from the multiple senders are refused to be received.

[0194] For the description of the features in the embodiment corresponding to the data flow control device, please refer to the relevant description of the embodiment corresponding to the data flow control method, and no further details will be given here.

[0195] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data flow control method embodiments.

[0196] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data flow control method embodiments when running.

[0197] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0198] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned data flow control method embodiments are implemented.

[0199] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned data flow control method embodiments.

[0200] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0201] The above is a detailed introduction to a data flow control method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for controlling data flow, characterized in that: include: Predicting the remaining time and pause time corresponding to the buffer of the first storage node respectively according to the buffer change rate of the first storage node, wherein the remaining time is used to indicate the time interval from the current moment to the moment when the buffer is filled, and the pause time is used to indicate the time when the first storage node needs to pause receiving data traffic when the remaining time is less than a time threshold; When the remaining time is less than the time threshold, a target sending end is selected from multiple sending ends corresponding to the source distribution of the data traffic of the first storage node, and the target sending end is instructed to pause sending data traffic to the first storage node based on the pause time.

2. The data flow control method according to claim 1, characterized in that: Predicting the remaining time and the pause time respectively corresponding to the buffer of the first storage node by using the buffer change rate of the first storage node, and the method further includes: Continuously sampling the occupied size of the buffer at a sampling time interval to obtain a plurality of sampling data pairs, wherein each sampling data pair includes: the occupied size of the buffer at two moments before and after the sampling time interval; The buffer change rate is determined by using the plurality of sample data pairs.

3. The data flow control method according to claim 2, characterized in that: Determining the buffer change rate using the plurality of sampled data pairs includes: Calculating the target buffer change rate corresponding to each sampling data pair according to each sampling data pair and the sampling time interval; A plurality of consecutive target buffer change rates are input into a sliding window to obtain the buffer change rate.

4. The data flow control method according to claim 2, characterized in that: After determining the buffer change rate using the plurality of sampled data pairs, the method further includes: determining whether the buffer is in an overflow state after a first target time by using the buffer change rate; When the buffer is in an overflow state after the first target time, the multiple sending ends are instructed to reduce the speed of sending data traffic at the notification time, wherein the notification time is separated from the current time by a second target time, and the second target time is less than 1 / 2 of the first target time.

5. The data flow control method according to claim 1, characterized in that: Predicting the remaining time and the pause time respectively corresponding to the buffer of the first storage node according to the buffer change rate of the first storage node includes: Predicting the remaining time by using the buffer change rate and the round-trip delay corresponding to the first storage node; and The pause time is predicted by the buffer zone change rate and a safety factor.

6. The data flow control method according to claim 5, characterized in that: Predicting the remaining time by using the buffer change rate and the round-trip delay corresponding to the first storage node includes: Determining a difference between a total size of the buffer and an occupied size of the buffer at the current moment, and determining a ratio of the difference to a rate of change of the buffer; The remaining time is predicted by the ratio and the round-trip delay.

7. The data flow control method according to claim 5, characterized in that: Predicting the pause time by using the buffer zone change rate and the safety factor includes: Determining a difference between a total size of the buffer and an occupied size of the buffer at the current moment, and determining a ratio of the difference to a rate of change of the buffer; The pause time is predicted by the ratio and the safety factor.

8. The data flow control method according to claim 7, characterized in that: Predicting the pause time by using the ratio and the safety factor includes: determining a historical adjustment factor based on k previous historical adjustment data at the current moment, wherein the i-th historical adjustment data among the k previous historical adjustment data includes: the i-th predicted pause time and the i-th actual time when the first storage node is decongested, wherein k is a positive integer, i=1, 2, 3, ..., k; and Calculating a target pause time using the ratio and the safety factor; The target pause time is adjusted using the historical adjustment factor to obtain the pause time.

9. The data flow control method according to claim 1, characterized in that: Selecting a target sending end from a plurality of sending ends corresponding to the source distribution of the data traffic of the first storage node comprises: Determine the total source amount of data traffic of the first storage node, and determine the amount of each source corresponding to each sending end; Determine the traffic proportion of each sending end among the multiple sending ends by the ratio of each source amount to the total source amount, wherein the traffic proportion is used to indicate the source distribution; A target traffic proportion is selected from the multiple traffic proportions sorted according to the target order, and the sending end corresponding to the target traffic proportion is used as the target sending end.

10. The data flow control method according to claim 1, characterized in that: Instructing the target sending end to suspend sending data traffic to the first storage node based on the pause time includes: Sending step: sending a preset data frame to the target sending end, wherein the preset data frame is used to instruct the target sending end to suspend sending data traffic to the first storage node; Updating step: calculating a fairness adjustment factor corresponding to each of the target sending ends, updating the traffic proportion of each sending end among the multiple sending ends by using the fairness adjustment factor, and updating the target sending end by using the updated traffic proportion; The above-mentioned sending step and updating step are executed in a loop until a preset condition is met, and it is determined that the target sending end has been instructed to suspend sending data traffic to the first storage node based on the pause time, wherein the preset condition includes one of the following: the pause time ends, and it is detected that the remaining time is greater than or equal to the time threshold.

11. The data flow control method according to claim 10, characterized in that: Calculating a fairness adjustment factor corresponding to each of the target sending ends includes: Determine the total time each sending end is suspended in a most recent statistical period; The fairness adjustment factor is determined by the ratio of the total time to the period time corresponding to the statistical period.

12. The data flow control method according to claim 1, characterized in that: After instructing the target sending end to suspend sending data traffic to the first storage node based on the pause time, the method further includes: In a case where there is data traffic to be stored in a client communicating with the distributed storage system including the first storage node, respectively calculating a buffer occupancy, a network delay, and a load factor of the first storage node; Perform a weighted summation of the buffer occupancy, the network delay and the load factor to determine the network congestion factor of the first storage node, and instruct the client to select the second storage node corresponding to the data traffic to be stored in the distributed storage system based on the network congestion factor.

13. The data flow control method according to claim 12, characterized in that: After performing weighted summation of the buffer occupancy, the network delay, and the load factor to determine a network congestion factor of the first storage node, the method further includes at least one of the following: Instructing a monitoring node in a distributed storage system including the first storage node to perform shard migration on the first storage node when the network congestion factor is higher than a preset threshold; Instruct the monitoring node to calculate the congestion status of multiple storage nodes in the distributed storage system through a congestion prediction model and multiple network congestion factors, and instruct the monitoring node to perform shard migration on the first storage node according to the congestion status.

14. The data flow control method according to claim 1, characterized in that: Predicting the remaining time and the pause time respectively corresponding to the buffer of the first storage node by using the buffer change rate of the first storage node, and the method further includes: Determining a load status of the first storage node, wherein the load status includes: a central processing unit usage rate; The total size of the buffer is adjusted according to the load status, and the response strategy of the first storage node to the data requests of the multiple sending ends is determined according to the load status.

15. The data flow control method according to claim 14, characterized in that: Adjusting the total size of the buffer according to the load status includes: When the load state is in a first adjustment interval, determining the total size of the buffer zone to be a first value; When the load state is in a second adjustment interval, determining the total size of the buffer zone to be a second value; When the load state is in a third adjustment interval, determining the total size of the buffer zone to be a third value; The first value is greater than the second value, the second value is greater than the third value; the maximum value in the first adjustment interval is less than the minimum value in the second adjustment interval, and the maximum value in the second adjustment interval is less than the minimum value in the third adjustment interval.

16. The data flow control method according to claim 14, characterized in that: Determining, based on the load status, a response strategy of the first storage node to the data requests of the multiple sending ends, includes: When the load state is in a first response interval, determining to allow receiving data requests from the multiple sending ends; When the load state is in the second response interval, determining to allow prediction of the remaining time and pause time respectively corresponding to the buffer areas; When the load status is in the third response interval, it is determined to return a temporary unavailable status to the multiple sending ends, and to refuse to receive the data requests from the multiple sending ends.

17. A data flow control device, characterized in that: include: a prediction module, configured to predict, based on a buffer change rate of the first storage node, a remaining time and a pause time respectively corresponding to the buffer of the first storage node, wherein the remaining time is used to indicate a time interval from a current moment to a moment when the buffer is filled, and the pause time is used to indicate a time during which the first storage node needs to pause receiving data traffic if the remaining time is less than a time threshold; An indication module is used to select a target sending end from multiple sending ends corresponding to the source distribution of data traffic of the first storage node when the remaining time is less than the time threshold, and to instruct the target sending end to pause sending data traffic to the first storage node based on the pause time.

18. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data flow control method according to any one of claims 1 to 16 when executing the computer program.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data flow control method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data flow control method according to any one of claims 1 to 16 are implemented.

Images