PCIe congestion judgment and flow control method and device

Abstract

The invention belongs to the field of switch chips, and relates to a PCIe congestion judgment and flow control method and device, during PCIe communication, a PCIe data requester sends a read request data packet to a data finishing party, and the read data request packet carries a Request ID and tag information; after a data finishing party receives a read request data packet, a data message initial address required by a data requesting party is read and sent according to a carried Request ID, a tag field is used for performing data order-preserving processing, meanwhile, the field size of the tag corresponds to the data size of the read request, and the data packet is sent to the data finishing party according to the read request data packet. Meanwhile, it is guaranteed that a data requester prepares a storage space with the corresponding size to store data to be returned, and flow control over a data completing party is achieved in the mode of adjusting the size of a tag field. According to the PCIe data transmission characteristics, the network congestion state is monitored synchronously when the request is read, and traffic control is performed by using a tag size adjustment mode, so that network congestion is avoided.

Description

Technical Field

[0001] The present invention relates to a PCIe congestion judgment and flow control method and device, belonging to the technical field of switching chips. Background Art

[0002] The peripheral high-speed interconnect interface is a widely used interface. Two PCIe devices with PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) interfaces can be interconnected through the PCIe interface.

[0003] In the PCIe transaction processing process, the application layer is responsible for generating the header information of the transport layer TLP (Transaction Layer Packet) message, in which the Requester ID field and the Tag field are used to form the transaction ID, where: Requester ID: used to identify the device address that initiated this transaction request. It is the unique BDF identifier (Bus, Device, Function) corresponding to the request initiator. Tag: used to match requests and responses to distinguish different transactions. Therefore, each TLP sent is given a unique tag, which can help PCIe track and manage data transmission, such as parallel processing, flow control or out-of-order processing.

[0004] For non-forwarded transactions (NP), all transaction requests include that all recipients need to return a completion message after receiving the request message. The completion message may or may not carry data.

[0005] With the increasing demand for intelligent computing, the number of PCIe devices in a single system is increasing. Due to task requirements, there are more and more scenarios where a device simultaneously communicates data with multiple devices. PCIe devices use the BDF field to identify device nodes. When multiple PCIe device nodes communicate data, congestion may occur, which may affect other data communications. Figure 1 As shown in: Devices 0, 1, 2, 3, 4, 5, 6, and 7 are all devices that support the standard PCIe communication protocol and are called PCIe devices.

[0006] When PCIe device 1 transmits data to PCIe devices 0, 2, and 5, congestion occurs because PCIe device 5 simultaneously receives data from PCIe devices 1, 4, and 6, which affects the data transmission of PCIe device 1. On the one hand, it affects the data communication from PCIe device 1 to PCIe devices 0 and 2, and on the other hand, it also affects the data transmission from PCIe device 2 to PCIe device 4.

[0007] Therefore, how to sense congestion at the PCIe device terminal and adjust the flow size in time is of great significance to improving data transmission efficiency.

[0008] In the prior art, the method for a data sender to perform congestion control management is generally as follows: In Ethernet data transmission, flow control is mostly performed in an end-to-end manner to avoid congestion during network transmission. The data sender mainly detects network congestion and adjusts the data transmission rate in time to avoid greater congestion. To this end, the data sender must first detect the current congestion status and then perform flow control for data transmission.

[0009] There are many traditional congestion detection methods on the sender side, which mainly include two categories: 1. Determine the congestion status on the path based on the change in the round-trip time (RTT) of the data packet; Second, the congestion status on the path is determined based on the switch queue length on the path and the congestion information reflected by the received ECN (Explicit Congestion Notification) flag.

[0010] When congestion is detected during the transmission process, flow control is performed by adjusting the size of the data sending window according to certain rules.

[0011] The existing technology performs congestion detection and judgment at the data sending end. Both methods judge the congestion on the round-trip path and cannot truly locate the congestion on the data transmission data, which easily leads to misjudgment of the congestion status.

[0012] Based on this, the present invention is proposed. Summary of the invention

[0013] The technical problems to be solved by the present invention are as follows: In non-forwarding NP mode PCIe communication, the requester sends a TLP request data packet to the completer, and the completer returns a TLP data packet to the requester. The requester can estimate congestion by receiving the characteristics of the data completion packet, and adjust the size and frequency of sending data request packets in time to control the rate of sending completion packets to avoid causing greater congestion in the network.

[0014] The present invention determines the path congestion state by monitoring received data at the PCIe device end, and performs flow control by adjusting the data transmission flow size of the sender, thereby reducing the occurrence of congestion.

[0015] The specific technical solutions of the present invention are as follows: The present invention utilizes the characteristics of PCIe data transmission, performs congestion assessment on the data request side, and notifies the data sender to perform flow control by adjusting the size of the tag in the read request, thereby avoiding congestion: 1. On the data request side, the current network congestion status is evaluated by monitoring the return time of the read completion data packet; 2. The data requester adjusts the tag value of the next read data request packet based on the congestion status assessment result; 3. The data completer determines the amount of returned data based on the tag value in the received data read request message.

[0016] More specifically, on the one hand, a PCIe congestion judgment and flow control method, when performing PCIe communication, the data requester of PCIe sends a read request data packet to the data completer, and the read data request packet carries the Requester ID and tag information; after the data completer receives the read request data packet, it reads and sends the starting address of the data message required by the data requester according to the carried Requester ID, and the tag field is used to perform data order preservation processing. At the same time, the tag field size also corresponds to the data size of this read request. At the same time, the data requester ensures that storage space of corresponding size is prepared to store the data to be returned, and the flow control of the data completer is achieved by adjusting the size of the tag field.

[0017] As a further improvement, the processing flow of the data requester and the data completer in the PCIe communication process includes the following steps: Step 1: The data requester initiates a data read request from the application layer, sends a read request data packet through a read request sending module, and uses the read request data packet to send the read request information to the data completer; Step 2: After receiving the read request data packet, the data completion party parses the data return address and the starting address of the data to be read through the read request receiving module, and transmits the tag value of this read request to the flow control module; the flow control module determines the amount of data that can be returned each time through the tag value; and sends the CplD data packet carrying the returned data to the data requester through the data packet sending module; Step 3: After receiving the returned CplD data packet through the data packet receiving module, the data requester obtains the requested data packet information; and records the time interval from sending the read request to obtaining the returned CplD data packet, so as to obtain the arrival time interval and disorder of the returned CplD data packet after this data read request, and sends it to the congestion assessment module; Step 4: The data requester uses the information recorded in step 3 to perform congestion assessment through the congestion assessment module. If congestion is predicted, the flow control module is used to reduce the tag size in the read request data packet of the next read request. Step 5: The data requester then sends a new read request data packet; the data completer decides how much data to send next time based on the tag size in the received read request data packet.

[0018] As a further improvement, the congestion assessment method comprises the following steps: The data requester records the time of sending each read request data packet. The sending time of the i-1th group of read request data packets is t1. i-1 ; The time to receive the first returned read completion data packet is t2 i-1 ; The time to receive all returned data packets is t3 i -1 ; T1 i-1 =t2 i-1 -t1 i-1 ; T2 i-1 =t3 i-1 -t2 i-1 ; tag=n, the tag value is n, the returned data is sequenced and the maximum amount of returned data is n; The sending time of the i-th group read request packet is t1 i ; The time to receive the first returned completion data packet is t2 i ; The time to receive all returned data packets is t3 i ; T1 i =t2 i -t1 i ; T2 i =t3i -t2 i ; tag=m, the tag value is m, the returned data is sequenced and the maximum amount of returned data is m; The congestion assessment is performed using the information of the i-1th and ith groups, and the tag value is adjusted as the parameter of the new read request data packet of the i+1th group.

[0019] For further improvement, congestion assessment is performed based on the timing information of the i-1th and ith times, and the tag value is adjusted as follows: 1) When T1 i ≤T1 i-1 And T2 i / m≤T2 i-1 / n, increase the tag value size in the data request packet; 2) When T1 i >T1 i-1 And T2 i / m>T2 i-1 / n, reduce the tag value size in the data request packet; 3) When T1 i ≤T1 i-1 And T2 i / m>T2 i-1 / n, keep the previous tag value; or, When T1 i >T1 i-1 And T2 i / m≤T2 i-1 / n, keep the previous tag value; 4) The data completer receives the data read request packet and determines the amount of data to be sent based on the tag size in the read request packet.

[0020] On the other hand, a PCIe congestion judgment and flow control device implements the PCIe congestion judgment and flow control method when running.

[0021] Beneficial effects of the present invention: When the read request node detects possible transmission congestion, the flow rate of the read data sent back by the data completion party next time is reduced by reducing the tag value size of the next group of read request data; if the current network status is detected to be good, the tag value size of the next group of read request data can be increased to increase the flow rate of the read data sent back by the data completion party next time. The present invention uses the transmission characteristics of PCIe data to synchronously monitor the network congestion status during the read request, and uses the tag size adjustment method to perform flow control to avoid network congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of PCIe device communication congestion in the background technology; Figure 2 This is an overall schematic diagram of the PCIe congestion judgment and flow control processing process; Figure 3 This is a schematic diagram of the data read request process; Figure 4 For congestion assessment and tag adjustment process; Figure 5 An example diagram of the process of performing flow control for read requests. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Example 1

[0024] The present invention can avoid network congestion caused by high data transmission pressure. By utilizing the characteristics of the PCIe non-forwarding communication mode, the data requester monitors the process of receiving the read completion data packet to evaluate the current network congestion state and perform corresponding data flow control. When the data requester receives the read completion data packet, it detects that the current data path may be congested, and then adjusts the number of data requests each time to notify the data completion party to reduce the size and frequency of data packet transmission, thereby alleviating the congestion in the current network.

[0025] The present invention utilizes the read and write characteristics of the NP data packet of PCIe to perform end-to-end flow control. When performing data communication, the data requester of PCIe sends a Requester (read request data packet) to the data completion party, and the Requester (read data request packet) carries information such as the Requester ID and tag; after the data completion party receives the read request data packet, it reads and sends the starting address of the data message required by the data requester according to the carried Requester ID, and the tag field is used to perform data order preservation processing. At the same time, the field size of the tag also corresponds to the data size of this read request. At the same time, the data requester needs to ensure that a storage space of a corresponding size is prepared to store the data to be returned, so the flow control of the data completion party can be achieved by adjusting the size of the tag field.

[0026] like Figure 2 As shown in , according to the requirements of the present invention, the specific processing flow of the data requester and the data completer in the PCIe communication process is as follows: Step 1: The data requester initiates a data read request from the application layer, sends a read request data packet through a read request sending module, and uses the read request data packet to send the read request information to the data completer; Step 2: After receiving the read request data packet, the data completion party parses the data return address and the starting address of the data to be read through the read request receiving module, and transmits the tag value of this read request to the flow control module; the flow control module determines the amount of data that can be returned each time through the tag value; and sends the CplD data packet (read completion data packet) carrying the returned data to the data requester through the data packet sending module. Step 3: After receiving the returned CplD data packet through the data packet receiving module, the data requester obtains the requested data packet information; and records the time interval from sending the read request to obtaining the returned CplD data packet, so as to obtain the arrival time interval and disorder of the returned CplD data packet after this data read request, and sends it to the congestion assessment module; Step 4: The data requester uses the information recorded in step 3 to perform congestion assessment through the congestion assessment module. If congestion is predicted, the flow control module is used to reduce the tag size in the read request data packet of the next read request to reduce the data volume of the next read request, so as to avoid congestion on the data transmission path. Step 5: The data requester then sends a new read request data packet; the data completer decides how much data to send next time based on the tag size in the received read request data packet.

[0027] The present invention utilizes the received data situation in each data reading cycle on the end side to evaluate the current network congestion status, and adjusts the amount of data sent back by the data completion party each time by adjusting the number of tags in the read request to avoid network congestion. Example 2

[0028] Network congestion status assessment (congestion assessment): like Figure 3 As shown, the data requester records the time of sending the read request data each time, such as: The sending time of the read request data packet of the i-1th group (i is an integer greater than 1) is t1 i-1 ; The time to receive the first returned read completion data packet is t2 i-1 ; The time to receive all returned data packets is t3 i-1 .

[0029] The calculation results are: T1 i-1 =t2 i-1 -t1 i-1 ; T2 i-1 =t3 i-1 -t2 i-1 ; tag=n, the tag value is n, the returned data is sequenced and the maximum amount of returned data is n; The sending time of the i-th group read request packet is t1 i ; The time to receive the first returned completion data packet is t2 i ; The time to receive all returned data packets is t3 i .

[0030] The calculation results are: T1 i =t2 i -t1 i ; T2 i =t3 i -t2 i ; tag=m, the tag value is m, the returned data is sequenced and the maximum amount of returned data is m; The congestion assessment is performed using the information of the i-1th and ith groups, and the tag value is adjusted as the parameter of the new read request data packet of the i+1th group.

[0031] exist Figure 3 In the data request, Addr indicates the data address information that needs to be read.

[0032] like Figure 4 As shown in the figure, congestion assessment is performed based on the timing information of the i-1th and ith times, and the tag value is adjusted: 1) When T1 i ≤T1 i-1 And T2 i / m≤T2 i-1 / n, it means that the transmission path status of group i is good and the transmission performance is improved, so the tag value size in the data request packet is increased; 2) When T1 i >T1 i-1 And T2 i / m>T2 i-1 / n, it means that the transmission path status of the i-th group has deteriorated and the transmission performance has decreased, so the tag value size in the data request packet is reduced; 3) In other cases, such as: When T1 i ≤T1 i-1 And T2 i / m>T2 i-1 / n; or, When T1 i >T1 i-1 And T2 i / m≤T2 i-1 / n; At this time, it indicates that the network congestion status is unstable and the previous tag value is maintained; 4) The data completer receives the data read request packet and determines the amount of data to be sent based on the tag size in the read request packet. Example 3

[0033] A specific application example: like Figure 5 As shown, it can be seen that the data requesting party PCIe device node A (referred to as node A) requests the data completing party PCIe device node B (referred to as node B) to read data. The processing process example is as follows: 1) Node A sends a read request data packet to node B, which reaches node B through the transmission network; 2) After receiving the read request data packet, Node B determines the data starting address and data size of the read completion data packet based on the read data starting address and Tag value in the read request data packet; 3) After receiving the read completion data packet, node A evaluates the current congestion status of the transmission path and adjusts the new Tag value according to the congestion status.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A PCIe congestion judgment and flow control method, characterized in that: During PCIe communication, the data requester of PCIe sends a read request data packet to the data completer. The read data request packet carries the Requester ID and tag information. After receiving the read request data packet, the data completer reads and sends the starting address of the data message required by the data requester according to the carried Requester ID. The tag field is used to preserve the data order. The size of the tag field also corresponds to the data size of this read request. At the same time, the data requester ensures that storage space of corresponding size is prepared to store the data to be returned. The flow control of the data completer is achieved by adjusting the size of the tag field.

2. A PCIe congestion judgment and flow control method according to claim 1, characterized in that: The processing flow of the data requester and the data completer during the PCIe communication process includes the following steps: Step 1: The data requester initiates a data read request from the application layer, sends a read request data packet through a read request sending module, and uses the read request data packet to send the read request information to the data completer; Step 2: After receiving the read request data packet, the data completion party parses the data return address and the starting address of the data to be read through the read request receiving module, and transmits the tag value of this read request to the flow control module; the flow control module determines the amount of data that can be returned each time through the tag value; and sends the CplD data packet carrying the returned data to the data requester through the data packet sending module; Step 3: After receiving the returned CplD data packet through the data packet receiving module, the data requester obtains the requested data packet information; and records the time interval from sending the read request to obtaining the returned CplD data packet, so as to obtain the arrival time interval and disorder of the returned CplD data packet after this data read request, and sends it to the congestion assessment module; Step 4: The data requester uses the information recorded in step 3 to perform congestion assessment through the congestion assessment module. If congestion is predicted, the flow control module is used to reduce the tag size in the read request data packet of the next read request. Step 5: The data requester then sends a new read request data packet; the data completer decides how much data to send next time based on the tag size in the received read request data packet.

3. A PCIe congestion judgment and flow control method according to claim 2, characterized in that: The congestion assessment method comprises the following steps: The data requester records the time of sending each read request data packet. The sending time of the i-1th group of read request data packets is t1. i-1 ; The time to receive the first returned read completion data packet is t2 i-1 ; The time to receive all returned data packets is t3 i-1 ; T1 i-1 =t2 i-1 -t1 i-1 ; <h2 style=";text-align:left;direction:ltr">T2<h2 style=";text-align:left;direction:ltr"> i-1 <h2 style=";text-align:left;direction:ltr"> =t3<h2 style=";text-align:left;direction:ltr"> i-1 <h2 style=";text-align:left;direction:ltr"> -t2<h2 style=";text-align:left;direction:ltr"> i-1 <h2 style=";text-align:left;direction:ltr"> ; tag=n, the tag value is n, the returned data is sequenced and the maximum amount of returned data is n; The sending time of the i-th group read request packet is t1 i ; The time to receive the first returned completion data packet is t2 i ; The time to receive all returned data packets is t3 i ; T1 i =t2 i -t1 i ; <h2 style=";text-align:left;direction:ltr">T2<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =t3<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> -t2<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ; tag=m, the tag value is m, the returned data is sequenced and the maximum amount of returned data is m; The congestion assessment is performed using the information of the i-1th and ith groups, and the tag value is adjusted as the parameter of the new read request data packet of the i+1th group.

4. A PCIe congestion judgment and flow control method according to claim 3, characterized in that: Congestion assessment is performed based on the i-1th and ith timing information, and the tag value is adjusted as follows: 1) When T1 i ≤T1 i-1 And T2 i / m≤T2 i-1 / n, increase the tag value size in the data request packet; 2) When T1 i >T1 i-1 And T2 i / m>T2 i-1 / n, reduce the tag value size in the data request packet; 3) When T1 i ≤T1 i-1 And T2 i / m>T2 i-1 / n, keep the previous tag value; or, When T1 i >T1 i-1 And T2 i / m≤T2 i-1 / n, keep the previous tag value; 4) The data completer receives the data read request packet and determines the amount of data to be sent based on the tag size in the read request packet.

5. A PCIe congestion judgment and flow control device, characterized in that: The PCIe congestion judgment and flow control device implements a PCIe congestion judgment and flow control method as described in any one of claims 1-4 when running.

Images