Credit value-based congestion control method, device and equipment

Abstract

The invention provides a congestion control method, a congestion control device and congestion control equipment based on a credit value, which are used for solving the technical problem that the effectiveness of a credit value released by a UET protocol CBRC algorithm is poor. According to the invention, the ECN mechanism supported by the switch equipment is utilized, and the end side equipment is utilized to evaluate the link RTT, so that the receiver judges the link quality based on the two factors, and when the link quality is poor, the credit value is dynamically adjusted according to the RTT or the packet loss rate so as to reduce the credit value issued by the receiver to the sender on the link.

Description

Technical Field

[0001] The present invention relates to the technical field of communications and cloud computing, and in particular to a congestion control method, device and equipment based on credit value. Background Art

[0002] Artificial intelligence and high-performance computing bring new challenges to networks, such as the need for greater scale, higher bandwidth density, multipathing, rapid response to congestion, and interdependence of individual data flow execution (where tail latency is a key consideration). Existing protocols may address some aspects of the above issues, but because they are designed for general-purpose networks, they lack support for features that are critical to artificial intelligence and high-performance computing, such as multipathing and easy configuration.

[0003] The Ultra Ethernet Consortium (UEC) is a new organization hosted by the Linux Foundation and its Joint Development Foundation Initiative. The goal of UEC is to go beyond existing Ethernet features, such as Remote Direct Memory Access (RDMA) and RDMA over ConvergedEthernet (RoCE) technology to provide a high-performance, distributed, and lossless transport layer optimized for high-performance computing and artificial intelligence. Compared with some of the problems with existing Ethernet features mentioned above, the design of the UEC specification will fill these gaps and provide the larger-scale networking required for these workloads. The goal of UEC is to provide a complete communication stack that solves technical problems across multiple protocol layers and provides easy-to-configure and manage features to achieve higher network utilization and lower tail latency, both of which are critical to reducing the completion time of Artificial Intelligence (AI) and High Performance Computing (HPC) jobs. Specifically, the goals that UEC wants to achieve in the future are:

[0004] 1) Open protocol specifications based on existing IP and Ethernet protocols.

[0005] 2) Multi-path, packet-spraying transmission fully utilizes the AI ​​network without causing congestion or head-of-line blocking, and does not require a centralized load balancing algorithm and routing controller.

[0006] 3) Incast management mechanism to control the fan-in on the final link to the target host with minimal packet loss.

[0007] 4) An efficient rate control algorithm allows transmission to quickly increase to line rate without causing performance loss to competing flows.

[0008] 5) API for out-of-order packet delivery, supporting message completion in both out-of-order and in-order modes, maximizing network and application concurrency and minimizing message latency.

[0009] 6) Scalable future networks to support millions of endpoints.

[0010] 7) Performance and optimal network utilization without the need for network and workload specific congestion algorithm parameter tuning.

[0011] 8) Aims to achieve line-speed performance of 800G, 1.6T and future faster Ethernet on commodity hardware.

[0012] The transmission protocols used in traditional high-performance computing / AI scenarios include TCP, Data Center TCP, RDMA, etc., but the congestion control algorithms involved in these protocols can only guide the control loop of the receiving end based on the interactive information generated by the process of adjusting the data flows of different protocols based on the switch queue. This may lead to flow conflicts (unreasonable load balancing), unreasonable bandwidth allocation, incast congestion (referring to the incast congestion caused by multiple working nodes sending message groups to the same aggregation node at the same time), high latency and other problems.

[0013] In order to achieve an efficient incast management mechanism, the Ultra Ethernet Transport (UET) protocol supports the use of a credit-based rate control (CBRC) algorithm (also known as a credit-based congestion control algorithm). The principle of this algorithm is mainly based on the edge-queued datagram service (EQDS), that is, by transferring queue-related operations and adjustment processes from the data center network to the host side. The core idea of ​​CBRC is to set a reasonable allocation strategy by distributing credits through the UET host receiving end to ensure that different traffic can share bandwidth fairly when the traffic on this side is not overloaded. This method of achieving end-to-end congestion control without the participation of switches can effectively optimize bandwidth allocation, reduce latency, and effectively achieve the purpose of traffic load balancing.

[0014] However, the current CBRC algorithm only relies on the receiver to publish credit values ​​based on its own receiving capabilities to drive the sender, and is unable to perceive link congestion or overload (overscription). The fixed credit value publishing coefficient used in the credit value sending timer will cause it to blindly publish credit values, further exacerbating the deterioration of link quality. Summary of the invention

[0015] In view of this, the present invention provides a congestion control method, apparatus and device based on credit value, which are used to solve the technical problem that the validity of credit value published by CBRC algorithm of UET protocol is poor.

[0016] According to one aspect of an embodiment of the present invention, the present invention provides a congestion control method based on a credit value, the method being applied to a receiver device of a UET protocol for ultra-Ethernet transmission, the method comprising:

[0017] The receiving packet transport sublayer PDS receives the message sent by the first sender and measures the current one-way delay OTT from the first sender to the receiver. cur (i);

[0018] The receiver is based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

[0019] Furthermore, the method for dynamically adjusting the credit value issued to the first sender according to the network congestion condition is:

[0020] When the target one-way delay is OTT obj (i) When the ratio of the target one-way delay to the current one-way delay is greater than 1 and no ECN-marked message is received, the link status is determined to be good. The receiver increases the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay:

[0021] When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and no ECN-marked packets are received, it is determined that packets are accumulated in the link. The receiver reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay:

[0022] When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and an ECN-marked message is received, the link is considered to be severely congested. The receiver reduces the credit value MTU increment in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages;

[0023] The target one-way delay is the inherent round-trip delay RTT between the first sender and the receiver. fix (i) The product of the one-way delay OTT gain coefficient α and divided by 2.

[0024] Furthermore, while dynamically adjusting the credit value MTU increment part, the timing interval of the credit value release timer is synchronously modified to the currently calculated credit value release quantity.

[0025] Furthermore, when it is determined that the link state is good or packets are accumulated in the link, the receiver dynamically adjusts the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay as follows:

[0026] When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value of the first value and the second value; the first value is the credit value most recently requested by the sender to the receiver; the second value is the credit value most recently sent by the receiver to the sender, the sum of the product of the ratio of the target one-way delay to the current one-way delay and the maximum transmission unit MTU;

[0027] When the first sender is in the idle sender queue ISL, the credit value published by the receiver to the first sender is: the credit value most recently sent by the receiver to the sender, the ratio of the target one-way delay to the current one-way delay, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU.

[0028] Furthermore, when it is determined that the link is severely congested, the receiver reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages as follows:

[0029] When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value between the first value and the third value; the first value is the credit value most recently requested by the sender to the receiver; the third value is the credit value most recently sent by the receiver to the sender, the sum of the products of the absolute value of the ratio of the target one-way delay to the current one-way delay and the difference of the ECN message ratio multiplied by the maximum transmission unit MTU;

[0030] When the first sender is in the idle sender queue ISL, the credit value published by the receiver to the first sender is: the credit value most recently sent by the receiver to the sender, the absolute value of the difference between the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU;

[0031] The ECN message ratio is the ratio of messages with ECN mark received by the receiver within the most recent inherent round-trip delay of the sender to the total messages received within the inherent round-trip delay.

[0032] Furthermore, the first sender and the receiver use a precise time synchronization protocol to perform clock synchronization.

[0033] Based on one aspect of an embodiment of the present invention, the present invention further provides a congestion control device based on a credit value, the device being applied to a receiver device of a UET protocol for ultra-Ethernet transmission, the device comprising:

[0034] The receiving and measuring module is used to receive the message sent by the first sender at the receiving side packet transmission sublayer PDS and measure the current one-way delay OTT from the first sender to the receiver. cur (i);

[0035] Credit value publishing module, used by the receiver based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

[0036] Furthermore, the credit value issuing module includes:

[0037] Congestion status judgment unit, used to determine the congestion status based on the current one-way delay OTT cur (i) The network congestion status is determined by the ECN message; when the target one-way delay OTT obj (i) When the ratio of the target one-way delay to the current one-way delay is greater than 1 and no ECN-marked message is received, the link is judged to be in good condition; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and no ECN-marked message is received, the message accumulation in the link is judged to be; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and an ECN-marked message is received, the link is judged to be severely congested;

[0038] The first adjustment unit is used to increase the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the link status is good:

[0039] The second adjustment unit is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the message is accumulated in the link:

[0040] The third adjustment unit is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages when the link is severely congested;

[0041] The target one-way delay is the inherent round-trip delay RTT between the first sender and the receiver. fix (i) The product of the one-way delay OTT gain coefficient α and divided by 2.

[0042] Furthermore, the first adjustment unit, the second adjustment unit and the third adjustment unit are also used to synchronously modify the timing interval of the credit value release timer to the currently calculated credit value release quantity while dynamically adjusting the credit value MTU increment part.

[0043] Further, the method in which the first adjustment unit and the second adjustment unit dynamically adjust the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay is:

[0044] When the first sender is in the active sender queue ASL, the credit value issued to the first sender is the minimum value of the first value and the second value; the first value is the credit value that the sender requested from the receiver the most recently; the second value is the credit value that the receiver sent to the sender the most recently, the sum of the product of the ratio of the target one-way delay to the current one-way delay and the maximum transmission unit MTU;

[0045] When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the ratio of the target one-way delay to the current one-way delay, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU;

[0046] The method in which the third adjustment unit reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages is:

[0047] When the first sender is in the active sender queue ASL, the credit value published to the first sender is the minimum value between the first value and the third value; the first value is the credit value that the sender requested from the receiver the most recently; the third value is the credit value that the receiver sent to the sender the most recently, the sum of the products of the absolute value of the ratio of the target one-way delay to the current one-way delay and the difference of the ECN message ratio multiplied by the maximum transmission unit MTU;

[0048] When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the absolute value of the difference between the ratio of the target one-way delay to the current one-way delay and the ECN message ratio, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU;

[0049] The ECN message ratio is the ratio of messages with ECN mark received by the receiver within the most recent inherent round-trip delay of the sender to the total messages received within the inherent round-trip delay.

[0050] The device provided by the present invention can be implemented in software, hardware or a combination of software and hardware. When implemented in a software module, the program code of the software module is loaded into the storage medium of the device, and the processor reads the program code in the storage medium and executes it.

[0051] The present invention utilizes the ECN mechanism supported by the switch device and utilizes the evaluation of the link RTT by the end-side device, so that the receiver can judge the link quality based on these two factors, and when the link quality is poor, dynamically adjust the credit value according to the RTT or the packet loss rate to reduce the credit value issued by the receiver to the sender on such a link. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.

[0053] Figure 1 This is a schematic diagram of the overall architecture of the UET protocol;

[0054] Figure 2 A flow chart of the CBRC algorithm controlling the release of credit values ​​in the UET draft;

[0055] Figure 3 A schematic diagram of the steps of a congestion control method based on credit value provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of a congestion control device based on credit value provided in one embodiment of the present invention;

[0057] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present invention for implementing the credit value-based congestion control method provided in the present invention. DETAILED DESCRIPTION

[0058] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. Although the embodiments of the present invention may be described in a first, second, or third manner, this description is only used to distinguish similar information, entities, or steps, rather than to describe a specific order or sequence. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. For another example, in some scenarios, the first information may refer to one information or to multiple information of the same type. In addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination". The "and / or" in the present invention is only a kind of association relationship describing the associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B may be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0059] The transmission protocols used in traditional high-performance computing / AI scenarios include TCP, DCTCP, RDMA, etc., but the congestion control algorithms involved in these protocols can only guide the control loop of the receiving end based on the interactive information generated in the process of adjusting different protocol data flows by the switch queue. This may lead to flow conflicts (unreasonable load balancing), unreasonable bandwidth allocation, incast congestion, high latency and other problems.

[0060] In order to achieve an efficient incast management mechanism, UET (Ultra Ethernet Transport) supports the use of a credit-based rate control (CBRC) congestion control algorithm, i.e., credit-based rate control. The principle of this algorithm is mainly based on EQDS (edge-queued datagram service), that is, by transferring queue-related operations and adjustment processes from the data center network to the host side. The core idea of ​​CBRC is to distribute credit values ​​through the UET host receiving end and set a reasonable allocation strategy to ensure that different traffic can share bandwidth fairly when the traffic on this side is not overloaded. This end-to-end congestion control without the participation of switches can effectively optimize bandwidth allocation, reduce latency, and effectively balance traffic load.

[0061] The following first explains the basic operation process of the CBRC algorithm.

[0062] Figure 1The diagram of the overall architecture of the UET protocol is shown in Figure 1. CBRC belongs to the congestion management module in the overall architecture of the UET, and the congestion management module is located in the packet delivery sublayer (PDS). The application layer will first send the message (msg) to the semantics sublayer (SES) through libfabrics (a common low-level network API). SES will split the received complete message msg into packets (packets for short) and hand them over to PDS. On the UET sender side, when the message reaches the PDS and needs to be sent, the PDS layer message sending module will ask the congestion management module whether it can send the packet. CBRC will make a judgment based on the sender's current credit value (number of bytes). If the credit value is sufficient to send the current size of the message, the congestion management module will reply with a signal that it can send to notify the PDS that it can send the packet; if the credit value is not enough, the PDS is not allowed to send the packet, and at the same time notify the PDS to send a smaller request to send (RTS) control message to the UET receiver to request the receiver to publish a larger credit value to itself (the sender). From this perspective, the key to the design of CBRC lies in the credit value publishing process of the receiver. The ideal credit value publishing strategy should not only reduce the probability of incast congestion, but also reasonably allocate bandwidth to multiple senders. Incast congestion is a specific type of network congestion phenomenon that usually occurs in data center networks or distributed computing environments. It is characterized by multiple senders (source nodes) sending large amounts of data to the same receiver (destination node) almost simultaneously, causing the network interface or switch buffer at the receiver to be quickly filled up and causing serious network performance problems and data packet loss.

[0063] Figure 2 This is a flow chart of the CBRC algorithm in the UET draft to control the release of credit values. The credit value distribution process is as follows:

[0064] The receiver maintains two queues, namely the Active Sender List (ASL) and the Idle Sender List (ISL). The ASL queue stores all senders whose requested credit values ​​have not been met, and the ISL queue stores all senders whose requested credit values ​​have been met. When the credit value requested by the sender in the ASL is met, it will be transferred to the ISL; when the sender in the ISL sends a new credit value to the receiver, it will be transferred to the ASL.

[0065] In the UET draft, a credit value sending timer is set to control the release of credit values. The timer releases a specific credit value to the sender in the ASL or ISL at a timed interval, thereby continuously updating the sender's available sending window. The method for setting the timer interval of the credit value sending timer is as follows:

[0066] Credit value sending timer interval = k*MTU / linkspeed (unit: seconds)

[0067] Where k is the credit value publishing factor, MTU is the maximum transmission unit of the link (usually 1500 bytes), and linkspeed is the line speed of the port on the receiving device.

[0068] Analysis of the above timing interval setting method shows that the size of the k value determines the message burst or congestion (from the sender) on the receiver side link. The smaller the k value, the smaller the interval for issuing the credit value, and the less likely the receiver side link is to have a burst. However, due to the smaller k value, the credit value sending timer timeout is shorter, so it will frequently trigger the sending of control messages carrying credit values, resulting in additional bandwidth occupation. The larger the k value, the more likely it is to have a burst, but it is not easy to cause additional bandwidth occupation. The UET protocol draft stipulates that k is a fixed value, and recommends a smaller value (such as k = 4).

[0069] After the k value is determined and the credit value sending timer is started, the timer will be triggered every k*MTU / linkspeed seconds. After the triggering timer first checks whether the ASL is empty.

[0070] If ASL is not empty, it means that the credit value requested by the sender has not been met. At this time, the receiver will give priority to sending credit value to the sender at the head of the ASL. If the sender's request is met after sending the credit value this time, the sender will be removed from the ASL and placed at the end of the ISL; if it is still not met, it will be placed at the end of the ASL. After completing the above operations, the timer will exit the current process and be set to the timing interval again, waiting for the next trigger.

[0071] If ASL is empty, it means that the credits requested by all senders at the current moment have been met. At this time, if ISL is not empty, the receiver will send a credit value of k*MTU to the sender at the head of the ISL. If the credit value sent by the receiver does not exceed the credit value upper limit (the credit value requested by the sender plus the maximum sending window max_cwnd), the sender will be put back to the end of the ISL; if it exceeds the credit value upper limit, the sender will be removed from the ISL (but will not be put into the ASL, which is equivalent to being in a free state).

[0072] The CBRC credit value release process in the above UET draft only relies on the receiver to release the credit value according to its own receiving capability to drive the sender to send packets. The receiver cannot perceive the link congestion or overload (overscription) situation, and the credit value sending timer uses a fixed credit value release coefficient, which easily causes blind credit value release, thereby further aggravating the deterioration of link quality.

[0073] In order to solve the defects of the original CBRC rate control algorithm in publishing credit values, the present invention proposes a congestion control solution based on credit values. The present invention utilizes the ECN mechanism supported by the switching device and utilizes the evaluation of the link RTT by the end-side device, so that the receiver can judge the link quality based on these two factors, and when the link quality is poor, dynamically adjust the credit value according to the RTT or the packet loss rate to reduce the credit value published by the receiver to the sender on such a link, thereby more reasonably controlling the release of credit values, avoiding link congestion, and optimizing link quality.

[0074] The specific implementation process of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the step processes shown in the accompanying drawings and embodiments can be executed in a computer system such as a set of computer executable instructions, and one or more of the exemplified method steps can be single or multiple or aggregated together and run in the computer system in the form of a software module or a virtual device. Moreover, although the exemplified step processes have time and sequential logic, the present invention is not limited to these steps in certain cases. They can be executed in an order different from that here. Under the guidance of the purpose of the present invention, simple changes or combinations of the execution order of these steps should also fall within the scope of the disclosure of the embodiments of the present invention.

[0075] Figure 3 A schematic flow chart of the steps of a credit value-based congestion control method provided in one embodiment of the present invention. Figure 3 The example method steps are applied to the UET protocol receiver device. UET involves a complete communication stack. The present invention only optimizes the packet transmission sublayer PDS of the UET protocol on the network card side. Before executing the method, a UET protocol connection needs to be established between multiple end-side devices supporting the UET protocol. In order to enable the receiver to accurately measure the one-way delay OTT between the i-th sender and the receiver cur (i) The sender and the receiver also need to use a precise network time protocol to synchronize the time between the terminals, such as the Precision Time Protocol (PTP).

[0076] The credit value issuing method provided by the present invention dynamically adjusts the credit value issued to the sender in the credit value issuing timer based on the one-way delay detected between the sender and the receiver and the situation that the message carries the congestion notification ECN information. The method comprises:

[0077] S301. The receiver PDS receives the message sent by the first sender and measures the current one-way delay OTT from the first sender to the receiver. cur (i);

[0078] After receiving the message delivered by the semantic sublayer SES, the sender PDS sends the message to the receiver's PDS. When sending the message, the sender PDS will carry a sending timestamp in the message.

[0079] Here, i represents any one of the multiple senders that establish a connection relationship with the receiver, so the first sender in the present invention refers to any one of the multiple senders that establish a connection relationship with the receiver.

[0080] S302. The receiver is based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

[0081] Currently, most switching devices support the ECN (Explicit Congestion Notification) marking function. The principle of ECN is: when the interactive device on the link senses link congestion, the switching device will add an ECN mark to the IP header of the message when forwarding the message. When the receiver receives the message with the ECN mark, it will know that a certain link in the current link is at risk of congestion.

[0082] Round Trip Time (RTT) is also an important signal for sensing link congestion. When the RTT of a link suddenly increases from a stable state, it means that a certain section of the link is likely to have message accumulation or even packet loss. However, the measurement of RTT is often limited to the sender, because the sender itself uses the same clock and can sense the sending timestamp of each message and the receiving timestamp of the corresponding ACK. As long as the receiving timestamp is subtracted from the sending timestamp, and then the ACK delay consumed by the receiver to process the message, the RTT value can be obtained.

[0083] In one embodiment of the present invention, the receiving PDS detects the current one-way delay OTT. cur (i) and the case where the message carries the explicit congestion notification ECN, the network congestion status is divided into the following three cases:

[0084] [Link status is good]

[0085] The conditions for determining a good link status are: target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur (i) The ratio is greater than 1 and no ECN-marked message is received;

[0086] Among them, the target one-way delay OTT obj The calculation method of (i) is:

[0087]

[0088] RTT in formula (1) fix (i) is the inherent round-trip delay between the i-th sender and receiver, which is usually measured by sending a probe message after the network topology is determined. α is the one-way delay OTT gain coefficient, which is a system configurable parameter. The recommended value range can be set to [1.0, 1.2]. The target one-way delay is actually equivalent to the fixed one-way delay adjusted by the gain coefficient. When the ratio of the target one-way delay to the current one-way delay is greater than 1, it means that the currently measured one-way delay is less than the target one-way delay, indicating that the current link state is in a good state and the link is unobstructed.

[0089] When the link status is good, the receiver increases the credit value published to the sender in a certain proportion as follows:

[0090] S311. When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value between the first value and the second value; the first value is the credit value most recently requested by the sender to the receiver; the second value is the credit value most recently sent by the receiver to the sender, and the target one-way delay OTT is obj (i) Compared with the current one-way delay OTT cur The sum of the ratio of (i) and the product of the maximum transmission unit MTU. The formula is as follows:

[0091]

[0092] Among them, sender(i).last_pull on the right side of the equal sign is the credit value that the receiver sent to the sender the last time, and sender(i).cur_pull on the left side of the equal sign is the credit value that the receiver wants to send to the sender this time. MTU is the maximum transmission unit (usually set to 1500 bytes). OTT cur (i) is the current one-way delay from the i-th sender to the receiver measured in real time. OTT obj(i) is the target one-way delay. sender(i).pull_target is the credit value requested by the sender to the receiver in the most recent request. min is a function that takes a smaller value. The reason for taking the minimum value is that the credit value desired by each sender is different, and the difference between them may be very large. In this case, the credit value can only be published according to the receiver's own receiving capacity.

[0093] The present invention refers to the MTU increment part added on the basis of the credit value sender(i).last_pull sent to the sender in the formula for calculating the credit value released to the sender based on the CBRC algorithm in the UET protocol as the credit value MTU increment part. By comparing the credit value released by the sender in the ASL in the original UET protocol draft, it can be seen that the credit value MTU increment part in the original protocol draft is 1 MTU, while the present invention uses the ratio of the target one-way delay and the current one-way delay to dynamically adjust the credit value MTU increment part. Since this ratio is usually greater than 1 when the link status is good, the increment part must be greater than one MTU.

[0094] S312. When the first sender is in the idle sender queue ISL, the credit value issued by the receiver to the first sender is: the credit value of the receiver's most recent transmission to the sender, the target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur The sum of the ratio of (i) and the product of the credit value publishing coefficient k and the maximum transmission unit MTU. The formula is as follows:

[0095]

[0096] Among them, k is the credit value publishing coefficient specified in the original UET protocol draft, which determines the amount of data that can be triggered by the sender (at most) after the receiver releases the credit value to the sender this time. The smaller the k value, the less data (sent by the sender) can be triggered by a single (receiver-issued) credit value message, so the link pressure on the receiver side is relatively smaller. However, since the k value also determines the timer triggering frequency, a smaller k value will cause the timer to be triggered frequently, that is, the receiver needs to send credit value messages more frequently than a larger k value, so the selection of the k value needs to consider the above situation at the same time. The protocol recommends k=4.

[0097] Furthermore, in one embodiment of the present invention, when the link status is good, the timing interval of the credit value release timer is dynamically adjusted by synchronously adjusting the ratio of the target one-way delay to the current one-way delay, that is, the timing interval of the credit value release timer specified in the original UET protocol draft is modified to the currently calculated credit value release number, namely sender(i).cur_pull.

[0098] In one embodiment of the present invention, the current one-way delay OTT from the i-th sender to the receiver is measured in real time. cur The measurement method of (i) is as follows: the receiver reads the message sending timestamp from the message sent by the i-th sender that is most recently received, and subtracts the message sending timestamp from the clock synchronized with the local precise time protocol to obtain the current one-way delay OTT. cur (i).

[0099] [Message accumulation in the link]

[0100] The judgment condition for the accumulation of packets in the link is: target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur (i) The ratio is less than or equal to 1 and no ECN marked messages are received.

[0101] When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1, it means that the currently measured one-way delay is greater than or equal to the target one-way delay, and the failure to receive the message with the ECN mark indicates that the switching device in the link has not detected severe congestion, in which case the message accumulation may have occurred in the queue of the forwarding device. In this case, the present invention steadily reduces the credit value issued to the sender according to a certain ratio, but it cannot be reduced too quickly or too much, otherwise it will cause a sharp decrease in the sending rate of the sender.

[0102] In one embodiment of the present invention, when messages are piled up in a link, the receiver reduces the credit value issued to the sender by a certain ratio in the following manner:

[0103] S321. When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value between the first value and the second value; the first value is the credit value most recently requested by the sender to the receiver; the second value is the credit value most recently sent by the receiver to the sender, and the target one-way delay OTT is obj (i) Compared with the current one-way delay OTT cur The sum of the ratio of (i) and the product of the maximum transmission unit MTU. That is, the formula used to publish the credit value is still the same as formula (2).

[0104] S322. When the first sender is in the idle sender queue ISL, the credit value issued by the receiver to the first sender is: the credit value of the receiver's most recent transmission to the sender, the target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur The ratio of (i) to the product of the credit value publishing coefficient k and the maximum transmission unit MTU. That is, the formula used to publish the credit value is still the same as formula (3).

[0105] Although messages accumulate in the link, the receiver still uses the same credit value calculation method as when the link is in good condition. However, since the ratio of the target one-way delay to the current one-way delay is less than 1, the published credit value MTU increment is less than one MTU, thereby achieving the purpose of reducing the credit value published to the sender by a certain proportion, thereby reducing the degree of message accumulation in the link.

[0106] Furthermore, in one embodiment of the present invention, when messages accumulate in a link, the timing interval of the credit value release timer is dynamically adjusted by synchronously adjusting the ratio of the target one-way delay to the current one-way delay, that is, the credit value release coefficient k specified in the original UET protocol draft is replaced by the product of the ratio of the target one-way delay to the current one-way delay and k.

[0107] [Situation of severe link congestion]

[0108] The criteria for determining severe link congestion are: target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur (i) The ratio is less than or equal to 1 and the ECN marked message is received.

[0109] When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1, it means that the currently measured one-way delay is greater than or equal to the target one-way delay, and the receipt of the message marked with ECN indicates that the switching device in the link has detected serious link congestion, and packet loss may occur. In this case, the present invention quickly reduces the credit value issued to the sender according to a certain ratio.

[0110] In one embodiment of the present invention, when the link is severely congested, the receiver reduces the credit value issued to the sender in the following manner:

[0111] S331. When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value between the first value and the third value; the first value is the credit value most recently requested by the sender to the receiver; the third value is the credit value most recently sent by the receiver to the sender, and the target one-way delay OTT is obj (i) Compared with the current one-way delay OTT cur The sum of the products of the absolute value of the ratio of (i) and the ECN message ratio loss_rate(i) multiplied by the maximum transmission unit MTU. The formula is as follows:

[0112] sender(i).cur_pull=

[0113]

[0114] Among them, the ECN message ratio loss_rate(i) is the receiver’s most recent inherent round-trip delay RTT at the sender. fix (i) is the ratio of the packets with ECN mark received within the inherent round-trip delay to the total number of packets received within the inherent round-trip delay. The meanings of other parameters and variables are the same as those in formula (2).

[0115] By comparing the credit value published by the sender in the ASL in the original UET protocol draft, it can be seen that the MTU increment of the credit value MTU increment part in the original protocol draft is 1 MTU, and the present invention uses the ratio of the target one-way delay to the current one-way delay minus the absolute value of the ECN message proportion loss_rate(i) to dynamically adjust the credit value MTU increment part. In the case of severe link congestion, the closer the ratio of the target one-way delay to the current one-way delay is to the ECN message proportion, the more serious the link congestion is, and therefore the published credit value MTU increment part will be less. As the approach speed is faster, the credit value decreases faster. In this way, the credit value MTU increment part published to the sender can be quickly reduced.

[0116] S332. When the first sender is in the idle sender queue ISL, the credit value issued by the receiver to the first sender is: the credit value of the receiver's most recent transmission to the sender, the target one-way delay OTT obj (i) Compared with the current one-way delay OTT cur The absolute value of the difference between the ratio of (i) and the ECN message ratio loss_rate(i) is then multiplied by the credit value publishing coefficient k and the maximum transmission unit MTU. The formula is as follows:

[0117]

[0118] The meanings of the parameters and variables are explained in equations (2) and (4).

[0119] Furthermore, in one embodiment of the present invention, when the link status is severely congested, the timing interval of the credit value release timer is dynamically adjusted by synchronously adjusting the ratio of the target one-way delay to the current one-way delay and the ECN message share, that is, the timing interval of the credit value release timer specified in the original UET protocol draft is modified to the currently calculated credit value release number, namely sender(i).cur_pull.

[0120] Figure 4 This is a structural diagram of a congestion control device based on credit value provided in one embodiment of the present invention. The device 400 is applied to a receiver device of the Ultra Ethernet Transmission UET protocol. The device 400 includes: a receiving and measuring module 410 and a credit value issuing module 420.

[0121] The receiving and measuring module 410 is used to receive the message sent by the first sender at the receiving side packet transmission sublayer PDS, and measure the current one-way delay OTT from the first sender to the receiver. cur (i);

[0122] Credit value publishing module 420 is used by the receiver to calculate the credit value based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

[0123] Furthermore, the credit value issuing module 420 includes:

[0124] The congestion status judgment unit 421 is used to determine the congestion status based on the current one-way delay OTT. cur (i) The network congestion status is determined by the ECN message; when the target one-way delay OTT obj (i) When the ratio of the target one-way delay to the current one-way delay is greater than 1 and no ECN-marked message is received, the link is judged to be in good condition; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and no ECN-marked message is received, the message accumulation in the link is judged to be; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and an ECN-marked message is received, the link is judged to be severely congested;

[0125] The first adjustment unit 422 is used to increase the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the link status is good:

[0126] The second adjustment unit 423 is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the message is accumulated in the link:

[0127] The third adjustment unit 424 is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages when the link is severely congested;

[0128] The target one-way delay is the inherent round-trip delay RTT between the first sender and the receiver. fix (i) The product of the one-way delay OTT gain coefficient α and divided by 2.

[0129] Furthermore, the first adjustment unit 422, the second adjustment unit 423 and the third adjustment unit 424 are also used to adjust the timing interval of the credit value issuance timer in the same manner while dynamically adjusting the credit value MTU increment part.

[0130] Further, the method for the first adjustment unit 422 and the second adjustment unit 423 to dynamically adjust the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay is:

[0131] When the first sender is in the active sender queue ASL, the credit value issued to the first sender is the minimum value of the first value and the second value; the first value is the credit value that the sender requested from the receiver the most recently; the second value is the credit value that the receiver sent to the sender the most recently, the sum of the product of the ratio of the target one-way delay to the current one-way delay and the maximum transmission unit MTU;

[0132] When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the ratio of the target one-way delay to the current one-way delay, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU;

[0133] The method by which the third adjustment unit 424 reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages is:

[0134] When the first sender is in the active sender queue ASL, the credit value published to the first sender is the minimum value between the first value and the third value; the first value is the credit value that the sender requested from the receiver the most recently; the third value is the credit value that the receiver sent to the sender the most recently, the sum of the products of the absolute value of the ratio of the target one-way delay to the current one-way delay and the difference of the ECN message ratio multiplied by the maximum transmission unit MTU;

[0135] When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the absolute value of the difference between the ratio of the target one-way delay to the current one-way delay and the ECN message ratio, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU;

[0136] The ECN message ratio is the ratio of messages with ECN mark received by the receiver within the most recent inherent round-trip delay of the sender to the total messages received within the inherent round-trip delay.

[0137] Figure 5A schematic diagram of the structure of an electronic device for implementing the credit-based congestion control method provided by the present invention is provided in one embodiment of the present invention. The device 500 includes: a processor 510 such as a central processing unit (CPU), a communication bus 520, a communication interface 540, and a memory 530. The processor 510 and the memory 530 can communicate with each other through the communication bus 520. The memory 530 stores a computer program, and when the computer program is executed by the processor 510, the function of one or more steps in the credit-based congestion control method provided by the present invention can be implemented.

[0138] Memory refers to a device based on a certain storage medium for storing computer programs and / or data. It can be a volatile memory (VM, often called memory) or a non-volatile memory (NVM). Memory refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It is used as a storage medium for temporary data of the operating system and other running programs. Memory can be synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), etc. Non-volatile memory refers to a memory that uses a persistent storage medium. It has the characteristics of large capacity and persistent data storage. It can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. SCM is the industry's general term for new storage media between memory and flash memory. It is a composite storage technology that combines persistent storage characteristics and memory characteristics. Its access speed is slower than DRAM and faster than SSD hard disks.

[0139] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium configured in this way causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0141] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0142] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A congestion control method based on credit value, characterized in that: The method is applied to a UET protocol receiver device for ultra-Ethernet transmission, and the method comprises: The receiving packet transport sublayer PDS receives the message sent by the first sender and measures the current one-way delay OTT from the first sender to the receiver. cur (i); The receiver is based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

2. The method according to claim 1, characterized in that: The method for dynamically adjusting the credit value issued to the first sender according to the network congestion condition is: When the target one-way delay is OTT obj (i) When the ratio of the target one-way delay to the current one-way delay is greater than 1 and no ECN-marked message is received, the link status is determined to be good. The receiver increases the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay: When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and no ECN-marked packets are received, it is determined that packets are accumulated in the link. The receiver reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay: When the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and an ECN-marked message is received, the link is considered to be severely congested. The receiver reduces the credit value MTU increment in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages; The target one-way delay is the inherent round-trip delay RTT between the first sender and the receiver. fix (i) The product of the one-way delay OTT gain coefficient α and divided by 2.

3. The method according to claim 2, characterized in that While dynamically adjusting the credit value MTU increment part, the timing interval of the credit value release timer is synchronously modified to the currently calculated credit value release quantity.

4. The method according to claim 2, characterized in that: When the link is in good condition or packets are accumulated in the link, the receiver dynamically adjusts the credit value MTU increment in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay as follows: When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value of the first value and the second value; the first value is the credit value most recently requested by the sender to the receiver; the second value is the credit value most recently sent by the receiver to the sender, the sum of the product of the ratio of the target one-way delay to the current one-way delay and the maximum transmission unit MTU; When the first sender is in the idle sender queue ISL, the credit value published by the receiver to the first sender is: the credit value most recently sent by the receiver to the sender, the ratio of the target one-way delay to the current one-way delay, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU.

5. The method according to claim 2, characterized in that: When it is determined that the link is severely congested, the receiver reduces the credit value MTU increment in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages as follows: When the first sender is in the active sender queue ASL, the credit value issued by the receiver to the first sender is the minimum value between the first value and the third value; the first value is the credit value most recently requested by the sender to the receiver; the third value is the credit value most recently sent by the receiver to the sender, the sum of the products of the absolute value of the ratio of the target one-way delay to the current one-way delay and the difference of the ECN message ratio multiplied by the maximum transmission unit MTU; When the first sender is in the idle sender queue ISL, the credit value published by the receiver to the first sender is: the credit value most recently sent by the receiver to the sender, the absolute value of the difference between the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU; The ECN message ratio is the ratio of messages with ECN mark received by the receiver within the most recent inherent round-trip delay of the sender to the total messages received within the inherent round-trip delay.

6. The method according to claim 1, characterized in that The first sender and the receiver use a precise time synchronization protocol to perform clock synchronization.

7. A congestion control device based on credit value, characterized in that: The device is applied to a receiver device of a UET protocol for ultra-Ethernet transmission, and the device comprises: The receiving and measuring module is used to receive the message sent by the first sender at the receiving side packet transmission sublayer PDS and measure the current one-way delay OTT from the first sender to the receiver. cur (i); Credit value publishing module, used by the receiver based on the current one-way delay OTT cur (i) judging the network congestion status according to the ECN carried in the message, and dynamically adjusting the credit value issued to the first sender according to the network congestion status.

8. The device according to claim 7, characterized in that The credit value publishing module includes: Congestion status judgment unit, used to determine the congestion status based on the current one-way delay OTT cur (i) The network congestion status is determined by the ECN message; when the target one-way delay OTT obj (i) When the ratio of the target one-way delay to the current one-way delay is greater than 1 and no ECN-marked message is received, the link is judged to be in good condition; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and no ECN-marked message is received, the message accumulation in the link is judged to be; when the ratio of the target one-way delay to the current one-way delay is less than or equal to 1 and an ECN-marked message is received, the link is judged to be severely congested; The first adjustment unit is used to increase the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the link status is good: The second adjustment unit is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay when the message is accumulated in the link: The third adjustment unit is used to reduce the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages when the link is severely congested; The target one-way delay is the inherent round-trip delay RTT between the first sender and the receiver. fix (i) The product of the one-way delay OTT gain coefficient α and divided by 2.

9. The device according to claim 8, characterized in that The first adjustment unit, the second adjustment unit and the third adjustment unit are also used to dynamically adjust the credit value MTU increment part and synchronously modify the timing interval of the credit value release timer to the currently calculated credit value release quantity.

10. The device according to claim 8, characterized in that The method in which the first adjustment unit and the second adjustment unit dynamically adjust the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay is: When the first sender is in the active sender queue ASL, the credit value issued to the first sender is the minimum value of the first value and the second value; the first value is the credit value that the sender requested from the receiver the most recently; the second value is the credit value that the receiver sent to the sender the most recently, the sum of the product of the ratio of the target one-way delay to the current one-way delay and the maximum transmission unit MTU; When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the ratio of the target one-way delay to the current one-way delay, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU; The method in which the third adjustment unit reduces the credit value MTU increment part in the credit value calculation method according to the ratio of the target one-way delay to the current one-way delay and the proportion of ECN messages is: When the first sender is in the active sender queue ASL, the credit value published to the first sender is the minimum value between the first value and the third value; the first value is the credit value that the sender requested from the receiver the most recently; the third value is the credit value that the receiver sent to the sender the most recently, the sum of the products of the absolute value of the ratio of the target one-way delay to the current one-way delay and the difference of the ECN message ratio multiplied by the maximum transmission unit MTU; When the first sender is in the idle sender queue ISL, the credit value published to the first sender is: the credit value most recently sent by the receiver to the sender, the absolute value of the difference between the ratio of the target one-way delay to the current one-way delay and the ECN message ratio, and the product of the credit value publishing coefficient k and the maximum transmission unit MTU; The ECN message ratio is the ratio of messages with ECN mark received by the receiver within the most recent inherent round-trip delay of the sender to the total messages received within the inherent round-trip delay.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium communicate with each other via the communication bus; Storage medium for storing computer programs; A processor, for implementing the method according to any one of claims 1 to 6 when executing a computer program stored on a storage medium.

Images