A read rate regulation method of a PCIe switching chip

Abstract

The application discloses a read rate regulation method of a PCIe switching chip, wherein an input port of the switching chip receives data packets from an external PCIe link respectively, one data packet corresponds to one descriptor used for managing data packet sending, the data packet and the descriptor are stored in the form of a queue, and the input port submits a forwarding request of each queue to a switching switch in the switching chip; the switching switch arbitrates each forwarding request according to a PCIe ordering rule; a queue of a port which obtains a forwarding opportunity through arbitration forwards a data packet at the head and descriptor information of the data packet to an output port on the switching chip through the switching switch; the output port stores the data packet and the descriptor information in a queue and sends the data packet saved in each queue to the external PCIe link. The application can regulate the read request rate of each port of the PCIe switching chip, so that congestion of the switching chip caused by DMA reading is eliminated.

Description

Technical Field

[0001] This invention belongs to the field of PCIe technology, and specifically relates to a method for controlling the read rate of a PCIe switching chip. Background Technology

[0002] Fan-out is the most common application scenario for PCIe switches, widely used in DMA I / O models for workstations and servers. The DMA controller at the I / O device endpoint both writes data blocks to and reads data from host memory. The upstream port of a PCIe switch chip, acting as an aggregation point connecting hosts, typically has a wider bandwidth than any downstream port, but it is not necessarily equal to the sum of the bandwidths of all downstream ports. If the upstream port bandwidth is insufficient, congestion and bandwidth sharing become critical issues.

[0003] The PCIe specification prohibits endpoint devices from performing flow control on completion packets. Endpoint devices must pre-define buffers for all read data they request so that they can receive the data at line rate when it returns through the PCIe link. This mechanism avoids queue congestion in the interconnect link when the source (host) and receiver (endpoint) bandwidths are the same, but it does not address the more common scenario where the source (host) bandwidth and connection speed are often much higher than those of the endpoint device.

[0004] The potential problems are further exacerbated by the following characteristics of the root complex (RC) and the endpoint devices themselves:

[0005] 1. The root complex (RC) typically processes read requests in a first-in, first-out (FIFO) order, without round-robin scheduling of queued requests from different devices to avoid blocking devices.

[0006] 2. Endpoint devices are often designed following the traditional PCI approach. In order to secure a fair share of bandwidth, they typically employ an aggressive read-ahead strategy, sending consecutive memory read requests to the host. Each memory read requests a large block of data (e.g., 4096 bytes).

[0007] Typical problem scenarios such as Figure 1 As shown, the switching chip is connected to the central processing unit (CPU) and memory through the root complex (RC). The link bandwidth between the switching chip and the root complex is ×8, and the link bandwidth between the switching chip and endpoints A and B is ×4.

[0008] Endpoint A sends 16 memory read operations to the host, each requesting 4KB of data. Endpoint B sends 2 memory read operations to the host, each requesting 1500 bytes of data. It then waits for the data it needs to read to arrive before sending other memory read operations. The bandwidth of the data returned by the host to endpoint A reaching the switching chip is ×8, far exceeding the data reception rate of port A (the port corresponding to endpoint A), which has a link width of ×4. This causes congestion on downstream port A. This congestion blocks the forwarding of read return data that has already reached the upstream port of the PCIe switch to downstream port B (the port corresponding to endpoint B). Endpoint B, connected to downstream port B, triggers a timeout retransmission mechanism because it cannot receive the read data in time, and sends another read request to the host. The read request is then forwarded through the downstream port to the upstream port, and finally to the host. The host reads the data required by endpoint B from memory again and forwards the read data to the upstream port of the PCIe switch chip. Due to congestion, the returned read data will eventually fill the cache of the upstream port of the PCIe switch chip. The PCIe switch chip will eventually put back pressure on the host, causing the host's performance to degrade. At the same time, the devices connected to the downstream port of the switch chip will not receive the returned read data for a long time, which will seriously affect the performance of the connected devices.

[0009] To alleviate downstream congestion, users can try configuring the read behavior of endpoint devices. Unfortunately:

[0010] 1. The PCIe architecture does not have traffic shaping or rate limiting mechanisms, and the required critical control interfaces are usually not available.

[0011] 2. The only feasible solution is to reduce the maximum read request size, but this can only partially alleviate the problem.

[0012] To avoid the situation described above, it is necessary to regulate the rate of read requests issued by each port. However, there is currently a lack of a mechanism or method to regulate the rate of read requests issued by each port in existing technologies, and there is also a lack of relevant reports in existing literature. Summary of the Invention

[0013] To address the problems existing in the prior art, this invention provides a PCIe switching chip read rate control mechanism to regulate the rate at which each port issues read requests, thereby alleviating congestion.

[0014] To solve the above-mentioned technical problems and achieve the above-mentioned technical effects, the present invention is implemented through the following technical solution:

[0015] A method for controlling the read rate of a PCIe switching chip includes the following steps:

[0016] One or more ports on the PCIe switching chip receive data packets from the external PCIe link. After classifying the received data packets, the data packets are stored in the corresponding input data packet queue, and the information related to the data packets is stored in the corresponding input descriptor queue.

[0017] According to the "first-in, first-out" rule, each input port that receives a data packet extracts the head descriptor of each input descriptor queue, and submits a forwarding request for the head data packet of its respective input data packet queue to the switching switch in the PCIe switching chip based on the extracted head descriptor information.

[0018] The switching switch follows the PCIe sorting rules to arbitrate each forwarding request and determine the queue of input packets that can obtain a forwarding opportunity among all the input ports that have submitted the forwarding request;

[0019] The input data packet queue that has obtained a forwarding opportunity through arbitration sends the head data packet and its descriptor of the received input data packet queue to the switching switch in sequence, and then forwards them to one or more output ports on the PCIe switching chip in sequence through the switching switch.

[0020] The output port stores the received data packets in the corresponding output data packet queue and the information related to the data packets in the corresponding output descriptor queue. According to the "first-in, first-out" rule, the head descriptor of each output descriptor queue is extracted. Based on the extracted head descriptor information and the PCIe sorting rules, one of the output data packet queues is determined to have a forwarding opportunity. The output data packet queue that has obtained the forwarding opportunity through arbitration sends the received head data packet to the external PCIe link.

[0021] Furthermore, the PCIe switching chip contains at least three ports, each containing an input module and an output module. Each port can use its internal input module as an input port for receiving data and its internal output module as an output port for sending data.

[0022] Furthermore, when a data packet enters the input port from an external PCIe link, it is taken over by the input module within the input port. The input module consists of an input data packet management module, at least one input data packet queue, and at least one input descriptor queue. The input data packet management module first receives data packets from the external PCIe link, establishes an input queue, and then submits a forwarding request for the head of the input queue to the switching switch. After the switching switch agrees to send, it finally sends the received data packet and its descriptor to the switching switch.

[0023] Data packets received from external PCIe links are stored in the input data packet queue, and information related to data packets received from external PCIe links is stored in the input descriptor queue. One descriptor corresponds to one data packet. The head data of each input descriptor queue is extracted and a forwarding request is submitted to the switching switch. The switching switch arbitrates the forwarding request based on the data packet information provided by the data packet descriptor. The input descriptor queue and the data packet queue corresponding to the descriptor queue selected by the arbitration are forwarded.

[0024] Furthermore, the input data packet management module includes an input queue management submodule, a packet size estimation submodule, and a first read interval control submodule;

[0025] The input data packet management module manages the storage and forwarding of data packets received from the external PCIe link by calling the input queue management submodule. Specifically, it first allocates the received data packets to the corresponding input data packet queue and allocates the relevant data packet information to the corresponding input descriptor queue.

[0026] Then, a forwarding request for the head data of the input descriptor queue is submitted to the switching switch. The head descriptor in the input descriptor queue that has obtained the forwarding opportunity and its corresponding head data packet of the input data packet queue are respectively sent to the switching switch.

[0027] When the received data packet is a memory read data packet, the input data packet management module will call the completion packet size estimation submodule to first estimate the completion packet size that the received memory read data packet should return. The completion packet size estimation value is one of the relevant information of the memory read data packet and is saved as the data packet descriptor.

[0028] The input data packet management module then calls the first read interval control submodule to control the forwarding of the received memory read data packets. Only after the first read interval control submodule agrees to forward the received memory read data packets can the input queue management submodule send a memory read data packet forwarding request to the switching switch.

[0029] After the switch agrees to forward the request, the input data packet management module then calls the input queue management submodule to send the memory read data packet and the corresponding estimated size of the completion packet to the switch, and the switch then sends the memory read data packet and the corresponding estimated size of the completion packet to the output module of the output port.

[0030] Furthermore, the first read interval control submodule contains a first timer and a first memory read forwarding enable register;

[0031] When the received memory read data packet is at the head of a certain input queue, if the value of the first memory read forwarding enable register in the first read spacing control submodule is 1, the input queue management submodule will send a forwarding request for the received memory read data packet to the switching switch; if the value of the first memory read forwarding enable register in the first read spacing control submodule is 0, the input queue management submodule will not send a forwarding request for the received memory read data packet to the switching switch.

[0032] When a memory read request packet and its corresponding descriptor are sent to the switching switch, the estimated size of the completed packet contained in the corresponding memory descriptor is simultaneously sent to the first read spacing control submodule. The first timer in the first read spacing control submodule is started, and the value of the first memory read forwarding enable register is cleared from 1 to 0. After loading an initial value, a countdown begins. During the countdown, the first timer is decremented by a subtrahend in each clock cycle. When the value of the first timer is 0 or less than the subtrahend, the first timer stops working, and the value of the first memory read forwarding enable register is set from 0 to 1.

[0033] Furthermore, the initial value loaded when the first timer starts working is equal to the estimated size of the received completed packet, and the subtrahend is equal to the number of bytes that the port can forward per clock cycle when forwarding data packets at the current rate; the multi-rate port contains multiple subtrahend registers, and the value of each subtrahend register is configured by the chip administrator. When the switching chip is working, each port can automatically select the value of the subtrahend register corresponding to the current rate as the subtrahend value.

[0034] Furthermore, the completion packet size estimation submodule estimates the completion packet size corresponding to the data packet read from the memory based on the read data length information extracted from the data packet read from the memory:

[0035] If the memory read packet initiates a zero-length read, the estimated completion packet size is the standard overhead of one completion packet supported by the system. If the read packet initiates a non-zero-length read, but the read length is less than or equal to the read completion boundary (RCB) supported by the system, the estimated completion packet size is the memory packet read length plus the standard overhead of one completion packet.

[0036] If the read length of a memory read packet exceeds the system's supported read completion boundary, a memory read packet will return one or more completion packets. In this case, the estimated size of the completion packet is the read length of the memory read packet plus the standard overhead of all possible completion packets. The standard overhead of all completion packets is equal to the double-word read length of the memory read packet shifted right by the overhead factor bits and then multiplied by the standard overhead of one completion packet. The overhead factor is configured by the system administrator through the overhead factor register of the PCIe switch chip and is a positive integer; the overhead factor is related to the system's supported read completion boundary.

[0037] The standard overhead of a completion packet consists of physical layer overhead, data link layer overhead, and the completion packet header.

[0038] Furthermore, after the switch sends the received data packet and its related information to the output port, the output module within the output port takes over. The output module consists of an output data packet management module, at least one output data packet queue, and at least one output descriptor queue. The output data packet management module first receives data packets from the switch, establishes an output queue, and finally sends the received data packets to the external PCIe link.

[0039] Data packets received from the switching switch are stored in the output data packet queue, and information related to the data packets received from the switching switch is stored in the output descriptor queue. One descriptor corresponds to one data packet, and the output descriptor queue is used to manage and schedule the transmission of the output data packet queue.

[0040] Furthermore, the output data packet management module includes an output queue management submodule and a second read interval control submodule;

[0041] The output data packet management module calls the output queue management submodule to store the data packets and related information received from the switching switch, and forwards the data packets. Specifically, the received data packets are first allocated to the corresponding output data packet queue of the output queue, and the related information corresponding to the received data packets is saved to the corresponding output descriptor queue.

[0042] Then, following the "first-in, first-out" rule, the head descriptor of each output descriptor queue is extracted. Then, according to the PCIe sorting rules, a descriptor is selected from one or more descriptors that are eligible to participate in forwarding scheduling, and the data packet corresponding to the selected descriptor is sent to the external PCIe link.

[0043] When the data packet to be forwarded is a memory read data packet, the output data packet management module controls the forwarding of the memory read data packet by calling the second read interval control submodule. Only after the second read interval control submodule allows the output data packet management module to send a forwarding request for the memory read data packet can the corresponding memory read data packet descriptor be eligible to participate in the forwarding scheduling.

[0044] Furthermore, the second read spacing control submodule contains a second timer and a second memory read forwarding enable register;

[0045] If the value of the second memory read forwarding enable register is 1, the output queue management submodule allows the output data packet management module to send a memory read data packet forwarding request to the output data packet management module. If the value of the second memory read forwarding enable register is 0, the output queue management submodule does not send a memory read data packet forwarding request to the output data packet management module.

[0046] If the memory read data packet forwarding request issued by the output queue management submodule is accepted by the output data packet management module and forwarded by the output queue management submodule, then the memory read data packet leaves the corresponding output port and is sent to the external PCIe link.

[0047] Simultaneously, the output data packet management module extracts the estimated completion packet size corresponding to the memory read data packet being sent from the descriptor and forwards the corresponding estimated completion packet size to the second memory read forwarding enable register. The second memory read forwarding enable register loads the number of bytes corresponding to the received estimated completion packet size into the second timer. The second timer starts and begins counting down. The value of the second memory read forwarding enable register is cleared from 1 to 0. When the value of the second timer is 0 or less than the subtrahend, the second timer stops working and the value of the second memory read forwarding enable register is set from 0 to 1.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] The read rate control mechanism of this invention can control the rate of read requests issued by each port of the PCIe switching chip, thereby effectively alleviating congestion downstream of the PCIe switching chip, avoiding triggering the timeout retransmission mechanism, and preventing the returned read data from filling the cache of the upstream port of the PCIe switching chip. This avoids the problem of host performance degradation caused by the PCIe switching chip ultimately putting back pressure on the host. At the same time, it can also prevent the devices connected to the downstream ports of the PCIe switching chip from being severely affected by the failure to receive returned read data.

[0050] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0052] Figure 1 This is a schematic diagram of a typical scenario of memory read congestion in existing PCIe switching chips;

[0053] Figure 2 This is a simplified structural diagram of a three-port PCIe switching chip in an embodiment of the present invention;

[0054] Figure 3 This is a simplified internal structure diagram of the PCIe switching chip port input module in an embodiment of the present invention;

[0055] Figure 4 This is a simplified internal structure diagram of the input data packet management module of the input module in this embodiment of the invention;

[0056] Figure 5 This is a simplified internal structure diagram of the PCIe switching chip port output module in an embodiment of the present invention;

[0057] Figure 6 This is a simplified internal structure diagram of the output data packet management module in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the link bandwidth of the PCIe switching chip in an embodiment of the present invention. Detailed Implementation

[0059] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the invention's purpose, features, and advantages. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the invention's technical solution.

[0060] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0061] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0062] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0063] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to mean “and / or” unless otherwise expressly stated herein.

[0064] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0065] This invention provides a method for read rate control of a PCIe switching chip. Typically, a PCIe switching chip has n ports, where n is a natural number not less than three. Each port receives data packets from an external PCIe link. The received data packets are forwarded to other ports via a switch within the PCIe switching chip, and then further sent to the external PCIe link. The forwarded PCIe data packets are classified into three transaction types: Posted, No-Posted, and Complete.

[0066] See Figure 7 As shown, Figure 7This is a simplified application scenario for a PCIe switch chip. The switch chip contains three ports: two downstream ports (Downstream Port A and Downstream Port B) and one upstream port (Upstream Port C). Downstream Port A is connected to Terminal A, Downstream Port B is connected to Terminal B, and Upstream Port C is directly connected to the root complex. There are four PCIe links between Downstream Port A and Terminal A, two PCIe links between Downstream Port B and Terminal B, and six PCIe links between Upstream Port C and the root complex.

[0067] See Figure 2 As shown, Figure 2 This is a simplified internal structure diagram of a PCIe switching chip according to an embodiment of the present invention. The PCIe switching chip internally includes three ports and a switching switch 1. The three ports are downstream port A, downstream port B, and upstream port C. Each port contains an input module 2 and an output module 3. Each port can use its internal input module 2 as an input port for receiving data and its internal output module 3 as an output port for sending data.

[0068] See Figure 3 and Figure 4 As shown, Figure 3 This is a simplified internal structure diagram of the input modules of each port of the PCIe switching chip in this embodiment of the invention. Figure 4 This is a simplified internal structural diagram of the input data packet management module of the input module in this embodiment of the invention. The input module 2 consists of an input data packet management module 21 and M input queues 22, where M equals 3. The input data packet management module 21 includes an input queue management submodule 211, a packet size estimation submodule 212, and a first read interval control submodule 213. The first read interval control submodule 213 contains a first timer 2131 and a first memory read forwarding enable register 2132. The input queues 22 include an input data packet queue and an input descriptor queue. The input data packet queue stores data packets, and the input descriptor queue stores data packet-related information. One input data packet queue corresponds to one input descriptor queue.

[0069] See Figure 5 and Figure 6 As shown, Figure 5 This is a simplified internal structure diagram of the output modules of each port of the PCIe switching chip in an embodiment of the present invention. Figure 6This is a simplified internal structure diagram of the output data packet management module in an embodiment of the present invention. The output module 3 consists of an output data packet management module 31 and N output queues 32, where N equals 3. The output data packet management module 31 includes an output queue management submodule 311 and a second read interval control submodule 312. The second read interval control submodule 312 contains a second timer and a second memory read forwarding enable register. The output queues 32 include an output data packet queue and an output descriptor queue. The output data packet queue stores data packets, and the output descriptor queue stores data packet-related information. One output data packet queue corresponds to one output descriptor queue.

[0070] When the chip is powered on, the first memory read forwarding enable register contained in the first read spacing control submodule 213 inside the input queue management module 211 of each port is set to 1.

[0071] by Figure 7 For example, after the chip powers on, the system, including the PCIe switching chip, first completes enumeration. At a certain moment, terminal A initiates 5 Direct Memory Access (DMA) read requests to main memory, with a read length of 4096 bytes for each of these 5 DMA read requests. Terminal B initiates 2 DMA read requests to main memory, with each read request being 1500 bytes in length.

[0072] After receiving a memory read request packet from terminal A, the input queue management submodule 211 of the input data packet management module 21 of its internal input module 2 parses the packet header and determines that it is a memory read request packet with a transaction type of Non-posted, intended to be sent to the root complex via port C. Therefore, it sends the read length information of the memory read request packet to the completion packet size estimation submodule 212. The completion packet size estimation submodule 212 estimates the completion packet size returned by the memory read request packet and allocates the memory read request packet to the input data packet queue of the input queue 22 of the input module 2 of the downlink port A. It also allocates the packet descriptor information of the memory read request packet, including the completion packet size estimation value, the packet transaction type Posted, and the output port C, to the input descriptor queue of the input queue 22 of the input module 2 of the downlink port A. Similar to the working principle of downlink port A, after receiving the memory read request packet sent by terminal B, downlink port B parses the packet header to determine that it is a memory read request packet with a transaction type of No-posted and intends to send it to the root complex through port C. It then allocates the memory read request packet to the input packet queue of input queue 22 of input module 2 of downlink port B, and allocates the packet descriptor information, including the estimated packet size, the transaction type Posted, and the output port C, to the input descriptor queue of input queue 22 of input module 2 of downlink port B.

[0073] The completion packet size estimation submodule 212 estimates the completion packet size corresponding to the memory read data packet based on the read data length information extracted from the memory read data packet: if the memory read data packet initiates a zero-length read, the estimated completion packet size is the standard overhead of one completion packet supported by the system; if the read data packet initiates a non-zero-length read, but the read length is less than or equal to the read completion boundary supported by the system... (RBC) Boundary: The estimated size of the completion packet is the memory data packet read length plus the standard overhead of one completion packet. If the read length of the memory data packet exceeds the system's supported read completion boundary, a memory data packet will return one or more completion packets. In this case, the estimated size of the completion packet is the memory data packet read length plus the standard overhead of all possible completion packets. The standard overhead of all completion packets is equal to the memory data packet read length in double-word units shifted right by the overhead factor bits and then multiplied by the standard overhead of one completion packet. The overhead factor is configured by the system administrator through the overhead factor register of the PCIe switch chip and is a positive integer. The overhead factor is related to the system's supported read completion boundary. When the read completion boundary is 64 bytes, the overhead factor is equal to 4; when the read completion boundary is 128 bytes, the overhead factor is equal to 5.

[0074] The overhead of a completion packet typically includes physical layer overhead, data link layer overhead, and a header. For systems supporting PCIe prefixes, the completion packet overhead may also include an optional prefix field: Process Address Space ID (PASID). In Gen2, the physical layer overhead consists of a 1-byte "Start" frame character and a 1-byte "End" frame character, totaling 2 bytes. In Gen3 and above, the physical layer overhead is a 4-byte "Start" frame character, without an "End" frame character. The Gen2 data link layer overhead includes a 2-byte sequence number ID and a 4-byte Link Cyclic Redundancy Check (LCRC), totaling 6 bytes. The completion packet header is a 3-double word and may also include an optional 4-byte end-to-end Cyclic Redundancy Check (ECRC). Without considering the prefix field, the overhead of a completion packet is physical layer overhead + data link layer overhead + header + optional ECRC. Without ECRC, the completion packet overhead is 20 bytes; with ECRC, the completion packet overhead is 24 bytes. For packets with a PASID prefix, without ECRC, the completion packet overhead is 24 bytes; with ECRC, the completion packet overhead is 28 bytes.

[0075] In many systems, the RCB is 64 bytes, each read data completion packet is 64 bytes, which is 16DW, and the overhead factor is 4.

[0076] If the standard overhead for each completion packet is 28 bytes, then:

[0077] Total cost of all completed packages = (Length / 16) * 28 = (Length >> 4) * 28.

[0078] If the cost of each completion packet is 24 bytes, then:

[0079] Total cost of all completed packages = (Length / 16) * 24 = (Length >> 4) * 24.

[0080] If the cost of each completion packet is 20 bytes, then:

[0081] Total cost of all completed packages = (Length / 16) * 20 = (Length >> 4) * 20.

[0082] For a system, the standard overhead bytes for each completion packet are fixed and configured through registers. The following example, using a completion packet with a standard overhead of 24 bytes and an RCB of 64 bytes, illustrates the method for estimating the total size of the completion packet.

[0083] For a memory read length of 4096 bytes, or 1024DW, the total overhead of all completed packets returned is (1024>>4)*24, which is 1536 bytes. The estimated total size of all completed packets is 4096+1536, which is 5632 bytes.

[0084] For a memory read length of 1500 bytes, or 375DW, the total overhead of all completed packets returned is (375>>4)*24, which is approximately 563 bytes. The estimated total size of all completed packets is 1500+563, or 2063 bytes.

[0085] When the switching chip is operating in Gen3, 8 bytes are transmitted per clock cycle for a ×2 link, 16 bytes per clock cycle for a ×4 link, and 24 bytes per clock cycle for a ×6 link.

[0086] For downlink port A, when the descriptor of the memory read request packet is at the head of its corresponding input descriptor queue, its corresponding input queue management submodule 211 knows that it corresponds to a memory read request packet and wants to output through port C. It will check the memory read forwarding enable register of the first read interval controller 213 of downlink port A. If the value of the memory read forwarding enable register is 1, it will send a forwarding request for the memory read request packet to the switch 1. If the value of the memory read forwarding enable register is 0, it will not send a forwarding request for the memory read request packet to the switch 1. Only when the value of the memory read forwarding enable register is 1 can the input queue management submodule 211 of downlink port A send a forwarding request for the memory read request packet to the switch 1.

[0087] Similarly, for downlink port B, when the descriptor of the memory read request packet is at the head of its corresponding input descriptor queue, its corresponding input queue management submodule 211 knows that it corresponds to a memory read request packet and wants to output through port C. It will check the memory read forwarding enable register of the first read interval controller 213 of downlink port B. If the value of the memory read forwarding enable register is 1, it will send a forwarding request for the memory read request packet descriptor to the switching switch 1. If the value of the memory read forwarding enable register is 0, it will not send a forwarding request for the memory read request packet to the switching switch 1. Only when the value of the memory read forwarding enable register is 1 can the input queue management submodule 211 of downlink port B send a forwarding request for the memory read request packet to the switching switch.

[0088] If at the same time, switch 1 receives memory read requests from downlink port A and downlink port B and forwards them through port C, switch 1 will arbitrate the requests. Based on the arbitration mechanism adopted by switch 1, assume switch 1 chooses to forward the packet descriptor of downlink port A and the memory read request packet.

[0089] After receiving the arbitration result from switch 1, the input queue management submodule 211 of downlink port A forwards the descriptor of the head of its corresponding input descriptor queue and the memory read request packet of the head of the input data packet queue corresponding to that descriptor to switch 1. At the same time, it starts the first timer of the first read interval controller 213 in its input data packet management module 21 to start the countdown, and clears the value of the memory read forwarding enable register to 0.

[0090] Before the countdown begins, the first timer loads an initial value equal to the estimated size of the received completion packet, derived from the descriptor of the corresponding memory read request packet. During the countdown, the first timer is subtracted by a subtrahend each clock cycle. This subtrahend equals the number of bytes that the input port can forward per clock cycle when forwarding data packets at the current rate. When the value of the first timer is 0 or less than the subtrahend, the first timer stops working, and the value of the first memory read forwarding enable register is set from 0 to 1. For downlink port A with 4 channels, since the read length of the memory read request is 4096 bytes, the estimated total size of all corresponding completion packets is 5632 bytes, according to the method described above. Downlink port A operates in Gen3 and can transmit 16 bytes per clock cycle. Therefore, its initial countdown value is 5632, and the countdown timer subtracts 16 each clock cycle.

[0091] The multi-rate port contains multiple subtraction registers. The value of each subtraction register is configured by the chip administrator. When the switching chip is working, each input port can automatically select the value of the subtraction register corresponding to the current rate as the subtraction value.

[0092] During the countdown period, even if the memory read request packet and related descriptor of downlink port A have been forwarded, and there are still memory read request packets and related descriptors waiting to be sent in the head of the input data packet queue and the head of the input descriptor queue of downlink port A, the input queue management submodule 211 of downlink port A will not send a memory read request packet forwarding request to the switching switch 1 because the value of the corresponding memory read forwarding enable register is cleared to 0.

[0093] After the first data packet descriptor and its corresponding memory read request packet are forwarded on downlink port A, and no more memory read request packet forwarding requests can be issued, the input queue management submodule 211 of downlink port B keeps issuing memory read request packet forwarding requests. Switch 1 selects downlink port B to send the descriptor of the memory read request packet and the memory read request packet corresponding to the descriptor.

[0094] Similarly, after receiving the arbitration result from switch 1, the input queue management submodule 211 of downlink port B forwards its corresponding input descriptor queue head descriptor and the memory read request packet corresponding to that head descriptor to switch 1. Simultaneously, it starts the first timer of the first read interval controller 213 within its input data packet management module 21, begins the countdown, and clears the value of the memory read forwarding enable register to 0. For downlink port B with two channels, since the read length of the memory read request is 1500 bytes, according to the previously described method, the estimated total size of all completed packets is 2063 bytes. Downlink port B operates in Gen3, capable of transmitting 8 bytes per clock cycle. Therefore, its initial countdown value is 2063, and the countdown timer decrements by 8 each clock cycle.

[0095] During the countdown period, even if the memory read request packet and related descriptor of downlink port B have been forwarded, there are still memory read request packets and related descriptors waiting to be sent in the head of the input data packet queue and the head of the input descriptor queue of downlink port B. However, since the value of the corresponding memory read forwarding enable register is cleared to 0, the input queue management submodule 211 of downlink port B will not issue a memory read request packet forwarding request.

[0096] Similarly, when a memory read data packet with a read length of 4096 bytes arrives at uplink port C via switch 1, the output queue management submodule 311 of the output data packet management module 31 inside uplink port C allocates the memory read request packet to the output data packet queue of its corresponding output queue 32, and allocates the descriptor associated with the memory read request packet to the output descriptor queue of its corresponding output queue 32. Since it is the first arriving memory read data packet, the second memory read forwarding enable register of the second read interval control submodule 312 within the corresponding output data packet management module 31 is set to 1. Therefore, it can immediately send a forwarding request to the output data packet management module 31. If the forwarding request is accepted by the output data packet management module 31, the corresponding memory read request packet will be forwarded to the output link of uplink port C, i.e., the PCIe link.

[0097] Simultaneously, the second timer of the second read interval control submodule 312 on uplink port C begins its countdown, and the value of its corresponding second memory read forwarding enable register is cleared to 0. The initial countdown value is 5632 bytes, and the second countdown timer decrements by 24 every clock cycle. During the countdown of the second timer, even if the output module 3 of uplink port C receives a memory read request data packet, as long as the second countdown timer is not 0 or is less than 24, the value of the corresponding second memory read forwarding enable register will remain 0, and no forwarding request will be issued; therefore, the memory read request packet will not be forwarded.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for regulating read rate of a PCIe switch chip, characterized in that, Includes the following steps: One or more ports on the PCIe switching chip receive data packets from the external PCIe link. After classifying the received data packets, the data packets are stored in the corresponding input data packet queue, and the information related to the data packets is stored in the corresponding input descriptor queue. According to the "first-in, first-out" rule, each input port that receives a data packet extracts the head descriptor of each input descriptor queue, and submits a forwarding request for the head data packet of its respective input data packet queue to the switching switch in the PCIe switching chip based on the extracted head descriptor information. The switching switch follows the PCIe sorting rules to arbitrate each forwarding request and determine the queue of input packets that can obtain a forwarding opportunity among all the input ports that have submitted the forwarding request; The input data packet queue that has obtained a forwarding opportunity through arbitration sends the head data packet and its descriptor of the received input data packet queue to the switching switch in sequence, and then forwards them to one or more output ports on the PCIe switching chip in sequence through the switching switch. The output port stores the received data packets in the corresponding output data packet queue and the information related to the data packets in the corresponding output descriptor queue. According to the "first-in, first-out" rule, the head descriptor of each output descriptor queue is extracted. Based on the extracted head descriptor information and the PCIe sorting rules, one of the output data packet queues is determined to have a forwarding opportunity. The output data packet queue that has obtained the forwarding opportunity through arbitration sends the received head data packet to the external PCIe link.

2. The read rate control method for a PCIe switching chip according to claim 1, characterized in that, The PCIe switching chip has at least three ports, each containing an input module and an output module. Each port can use its internal input module as an input port for receiving data and its internal output module as an output port for sending data.

3. The read rate control method for a PCIe switching chip according to claim 2, characterized in that, When a data packet enters the input port from an external PCIe link, it is taken over by the input module within the input port. The input module consists of an input data packet management module, at least one input data packet queue, and at least one input descriptor queue. The input data packet management module first receives data packets from the external PCIe link, establishes an input queue, and then submits a forwarding request for the head of the input queue to the switching switch. After the switching switch agrees to send, it finally sends the received data packet and its descriptor to the switching switch. Data packets received from external PCIe links are stored in the input data packet queue, and information related to data packets received from external PCIe links is stored in the input descriptor queue. One descriptor corresponds to one data packet. The head descriptor of each input descriptor queue is extracted and a forwarding request is submitted to the switching switch. The switching switch arbitrates the forwarding request based on the data packet information provided by the data packet descriptor. The input descriptor and the data packet corresponding to the descriptor selected by the arbitration are forwarded.

4. The read rate control method for a PCIe switching chip according to claim 3, characterized in that, The input data packet management module includes an input queue management submodule, a packet size estimation submodule, and a first read interval control submodule; The input data packet management module manages the storage and forwarding of data packets received from the external PCIe link by calling the input queue management submodule. Specifically, it first allocates the received data packets to the corresponding input data packet queue and allocates the relevant data packet information to the corresponding input descriptor queue. Then, a forwarding request for the head descriptor of the input descriptor queue is submitted to the switching switch. The head descriptor in the input descriptor queue that has obtained the forwarding opportunity and its corresponding head data packet in the input data packet queue are respectively sent to the switching switch. When the received data packet is a memory read data packet, the input data packet management module will first estimate the size of the completed packet that should be returned by the received memory read data packet by calling the completed packet size estimation submodule. The estimated completed packet size is one of the relevant information of the memory read data packet and is saved as the data packet descriptor. The input data packet management module then calls the first read interval control submodule to control the forwarding of the received memory read data packets. Only after the first read interval control submodule agrees to forward the received memory read data packets can the input queue management submodule submit the descriptor forwarding request corresponding to the memory read data packets to the switching switch. After the switch agrees to forward the descriptor corresponding to the memory read data packet, the input data packet management module then calls the input queue management submodule to send the corresponding data packet descriptor and the memory read data packet corresponding to the data packet descriptor to the switch. The data packet descriptor contains the corresponding estimated value of the completed packet size. The switch then sends the data packet descriptor and the memory read data packet corresponding to the data packet descriptor to the output module of the output port.

5. The read rate control method for a PCIe switching chip according to claim 4, characterized in that, The first read interval control submodule contains a first timer and a first memory read forwarding enable register; When the received memory read data packet is at the head of a certain input queue, if the value of the first memory read forwarding enable register in the first read spacing control submodule is 1, the input queue management submodule will send a forwarding request for the descriptor corresponding to the received memory read data packet to the switching switch; if the value of the first memory read forwarding enable register in the first read spacing control submodule is 0, the input queue management submodule will not send a forwarding request for the descriptor corresponding to the received memory read data packet to the switching switch. When the memory read request data packet and the corresponding descriptor are sent to the switching switch, the estimated size of the completed packet contained in the corresponding memory descriptor is simultaneously sent to the first read interval control submodule. The first timer in the first read interval control submodule is started, and the value of the first memory read forwarding enable register is cleared from 1 to 0. After loading an initial value, the countdown begins. During the countdown, the first timer is subtracted by a subtrahend in each clock cycle. When the value of the first timer is 0 or less than the subtrahend, the first timer stops working and the value of the first memory read forwarding enable register is set from 0 to 1.

6. The read rate control method for a PCIe switching chip according to claim 5, characterized in that, The initial value loaded when the first timer starts working is equal to the estimated size of the received completed packet. The subtrahend is equal to the number of bytes that the input port can forward per clock cycle when forwarding data packets at the current rate. The multi-rate port contains multiple subtrahend registers. The value of each subtrahend register is configured by the chip administrator. When the switching chip is working, each input port can automatically select the value of the subtrahend register corresponding to the current rate as the subtrahend value.

7. The read rate control method for a PCIe switching chip according to claim 4, characterized in that, The complete packet size estimation submodule estimates the complete packet size corresponding to the data packet read from the memory based on the read data length information extracted from the data packet read from the memory: If the memory read packet initiates a zero-length read, the estimated completion packet size is the standard overhead of one completion packet supported by the system; if the read packet initiates a non-zero-length read, but the read length is less than or equal to the read completion boundary supported by the system, the estimated completion packet size is the memory packet read length plus the standard overhead of one completion packet. If the read length of a memory read packet exceeds the system's supported read completion boundary, a memory read packet will return one or more completion packets. In this case, the estimated size of the completion packet is the read length of the memory read packet plus the standard overhead of all possible completion packets. The standard overhead of all completion packets is equal to the double-word read length of the memory read packet shifted right by the overhead factor bits and then multiplied by the standard overhead of one completion packet. The overhead factor is configured by the system administrator through the overhead factor register of the PCIe switch chip and is a positive integer; the overhead factor is related to the system's supported read completion boundary. The standard overhead of a completion packet consists of physical layer overhead, data link layer overhead, and the completion packet header.

8. The read rate control method for a PCIe switching chip according to claim 2, characterized in that, After the switch sends the received data packet and its related information to the output port, the output module in the output port takes over. The output module consists of an output data packet management module, at least one output data packet queue and at least one output descriptor queue. The output data packet management module first receives data packets from the switch, establishes an output queue, and finally sends the received data packets to the external PCIe link. The data packets received from the switching switch are stored in the output data packet queue, and the data packet descriptor information received from the switching switch is stored in the output descriptor queue. One descriptor corresponds to one data packet. The output descriptor queue is used to manage and schedule the transmission of the output data packet queue.

9. The read rate control method for a PCIe switching chip according to claim 8, characterized in that, The output data packet management module includes an output queue management submodule and a second read interval control submodule; The output data packet management module calls the output queue management submodule to store the data packets and their descriptor information received from the switching switch, and forwards the data packets. Specifically, the received data packets are first allocated to the corresponding output data packet queue of the output queue, and the descriptor information corresponding to the received data packets is saved to the corresponding output descriptor queue. Then, following the "first-in, first-out" rule, the head descriptor of each output descriptor queue is extracted. Then, according to the PCIe sorting rules, a descriptor is selected from one or more descriptors that are eligible to participate in forwarding scheduling, and the data packet corresponding to the selected descriptor is sent to the external PCIe link. When the data packet to be forwarded is a memory read data packet, the output data packet management module controls the forwarding of the memory read data packet by calling the second read interval control submodule. Only after the second read interval control submodule allows the output data packet management module to send a forwarding request for the memory read data packet can the corresponding memory read data packet descriptor be eligible to participate in the forwarding scheduling.

10. The read rate control method for a PCIe switching chip according to claim 9, characterized in that, The second read interval control submodule contains a second timer and a second memory read forwarding enable register; If the value of the second memory read forwarding enable register is 1, the output queue management submodule allows the output data packet management module to send a memory read data packet forwarding request to the output data packet management module. If the value of the second memory read forwarding enable register is 0, the output queue management submodule does not send a memory read data packet forwarding request to the output data packet management module. If the memory read data packet forwarding request issued by the output queue management submodule is accepted by the output data packet management module and forwarded by the output queue management submodule, then the memory read data packet leaves the corresponding output port and is sent to the external PCIe link. Simultaneously, the output data packet management module obtains the corresponding estimated completion packet size from the descriptor of the memory read data packet being sent, and forwards the corresponding estimated completion packet size to the second memory read forwarding enable register. The second memory read forwarding enable register loads the number of bytes corresponding to the received estimated completion packet size into the second timer. At the same time, the value of the second memory read forwarding enable register is cleared from 0 to 1, the second timer starts and begins counting down. When the value of the second timer is 0 or the value of the second timer is less than the subtrahend, the second timer stops working and the value of the second memory read forwarding enable register is set from 0 to 1.

Images