A method for writing data into a memory and a network element
By adopting the method of recycled resource use in the asynchronous input and output queuing system, the data discarding and stagnation caused by host memory filling is solved, and efficient data processing and resource utilization are achieved in high-speed network communication.
Patent Information
- Application Number
- CN202010419130.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-05-20
- Filing Date
- 2020-05-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-05-18
AI Technical Summary
Existing asynchronous input and output queuing systems In high-speed network communication, host memory is easily filled, resulting in data discarding or stagnation, and solutions such as adding buffers or increasing CPU load are costly.
The allocated resources are used in a circular manner, and the network elements continuously receive data and overwrite the oldest packets in the memory to maintain the most recent data while reducing the consumption of memory and CPU load.
It effectively reduces memory consumption and CPU load, avoids data discarding and network congestion, and improves resource utilization efficiency.
Smart Images

Figure CN111970213B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates generally to input-output queuing systems and, in particular but not exclusively, to asynchronous input-output queuing systems. Background Art
[0002] It is known that network elements such as switches or network interface controllers (NICs) communicate with external devices / hosts via asynchronous input and output queuing systems, for example, via PCI or PCI-e interfaces. Summary of the invention
[0003] The present invention, in certain embodiments thereof, is directed to providing an improved input-output queuing system.
[0004] The inventors of the present invention have found that in existing asynchronous input and output queuing systems, particularly those used with network elements (such as switches or network interface controllers (NICs)), the asynchronous queuing system requires external devices / hosts (these terms are used interchangeably herein; the term "devices external to the network element" is also used herein) that communicate with the network element to allocate memory for receiving and sending data. Furthermore, in addition to memory allocation for data, the external device generally needs to allocate memory for messages.
[0005] The external device may configure different queues for different purposes so that each queue holds data relevant to a given purpose; such purposes may include, for example, monitoring, IP management, errors, tunnel management, etc. Typically, the host tells the network where to read from and where to write to by maintaining queues whose entries each include a pointer (address) indicating the appropriate location in the internal device memory from which data is to be read or to which data is to be written.
[0006] In some scenarios, a portion of the network traffic generates events to be sent to the host; it will be appreciated that, therefore, especially if the network element implements a high-speed network, the host memory consumption is high and the allocated memory on the host fills up quickly. Once the allocated memory on the host is full, in order to receive more data from the network element, the host (which may be a processor packaged with the network element, or may be a processor located outside the network element and communicating with the network element through an appropriate communication mechanism, such as PCI-e, by way of non-limiting example) needs to allocate more memory for receiving further data and for issuing new memory and control descriptors (i.e., a memory range needs to be allocated for new queue entries).
[0007] In a situation where the network element does not pass data to the host, if there is no free memory in the host and new queue entries pointing to the buffer in the host memory are not refreshed by the host software in a timely manner, the data stored in the buffer in the host memory may be outdated and therefore irrelevant, while the most relevant data may be discarded or stagnant by the network element due to lack of appropriate resources.
[0008] The inventors of the present invention believe that there are two simple options to reduce but not solve the above problems. The first solution is to use more / larger buffers and thereby increase the amount of data that can be received by the host. The second option is to refresh the host memory more frequently at the expense of higher CPU load. In each case, a large cost (more memory, higher CPU load) is required.
[0009] The following is an explanation of a specific implementation of the current method described above. Software running on the host allocates memory for received packets using descriptors called work queue entries (WQEs) maintained in a received data queue (RDQ). Each WQE includes an address in physical memory in the host device to which data is to be written or read.
[0010] When the network element has data to send to the host, the network element "consumes" a WQE from the appropriate RDQ and sends the data to the allocated memory indicated in the WQE through the appropriate interface, such as a PCI-e interface, for non-limiting example. In the case where there is no available WQE, the network element will operate according to the selected mechanism:
[0011] Lossy – the network element discards (abandons) new information (packets, data from packets).
[0012] Lossless - the network element stalls the receive path (from the device to the host) until a new WQE is available; as known in the art, such stalling can lead to network congestion that may propagate throughout the network.
[0013] As mentioned above, the host is the master of the interface: if a WQE is not assigned, the host will stop receiving data from the network element (on a specific RDQ).
[0014] In certain exemplary embodiments of the present invention, the above-mentioned problems of consistent resource allocation by the host and / or the need to pre-allocate very large amounts of resources are solved. The allocated resources are used in a round-robin manner; resources are allocated by the host and then the network elements use these resources in a round-robin manner, thereby reducing host intervention / overhead while continuously receiving data from the network elements. It should be understood that in this exemplary embodiment, the most recent (newest) packet will generally overwrite the oldest packet in the host's memory. This allows the most recent (generally most relevant) data to be kept in memory, while consuming less memory and reducing CPU load.
[0015] Furthermore, in certain exemplary embodiments of the present invention, prior to initiating the use of the circular buffer just described above, a "standard" RDQ may be used so that the first data received by the host is stored as usual; the circular RDQ described above is used only when the "standard" RDQ is full (no further WQE entries are available therein). In a further exemplary embodiment, prior to using the circular RDQ described above, multiple "standard" RDQs may be used one after another. In a further exemplary embodiment, multiple "standard" RDQs may be used one after another without using the circular RDQ described above. In any of these approaches (whether in the case of a single standard RDQ followed by a circular RDQ or in both cases of multiple standard RDQs mentioned above), in addition to keeping the most recent (latest) packet received (generally in the case of using a circular buffer), the first (oldest) packet is also kept.
[0016] Therefore, according to an exemplary embodiment of the present invention, a method is provided, comprising: providing a network element, which includes a buffer address control circuit and an output circuit; receiving a packet containing data from outside the network element; reading a given entry from a first queue maintained in a memory of a device outside the network element by the buffer address control circuit, the first queue having at least a first entry and a last entry, and the given entry including a destination address in the memory; writing the data to the destination address in the memory according to the given entry by the output circuit; specifying a next entry by the buffer address control circuit in the following manner: when the given entry is not the last entry in the first queue, specifying the next entry as an entry after the given entry in the first queue; and when the given entry is the last entry in the first queue, specifying the next entry as the first entry in the first queue; and using the next entry as the given entry and using another packet received from outside the network element and containing data, performing the writing and the specifying again.
[0017] Further in accordance with an exemplary embodiment of the present invention, the first queue comprises a received data queue (RDQ), and each entry in the RDQ in the first queue comprises a work queue entry (WQE).
[0018] According to an exemplary embodiment of the present invention, the method further includes: performing the following items before reading the given entry from the first queue: reading, by the buffer address control circuit, a second queue given entry from a second queue maintained in the memory of the device outside the network element, the second queue having at least a first second queue entry and a last second queue entry, the second queue given entry including a destination address in the memory; writing, by the output circuit, data to the destination address in the memory according to the second queue given entry; specifying, by the buffer address control circuit, a next second queue entry in the following manner: when the second queue given entry is not the last entry in the second queue, specifying the next second queue entry as an entry after the given entry in the second queue, and using the next entry as the given entry and using another packet received from outside the network element and containing data to perform the writing according to the second queue given entry again, and the specifying of the next second queue entry; and when the second queue given entry is the last entry in the second queue, continuing, by the buffer address control circuit, the reading of the given entry from the first queue using another packet received from outside the network element and containing data.
[0019] Furthermore, according to an exemplary embodiment of the present invention, the second queue comprises a receive data queue (RDQ), and each entry in the RDQ in the second queue comprises a work queue entry (WQE).
[0020] In addition, according to an exemplary embodiment of the present invention, the method further provides a plurality of queues; selecting a queue from the plurality of queues and, for the selected queue from the plurality of queues, performing the following items before reading the given entry from the first queue: reading a selected queue given entry from the selected queue maintained in the memory of the device outside the network element by the buffer address control circuit, the selected queue having at least a first selected queue entry and a last selected queue entry, the selected queue given entry including a destination address in the memory; writing data to the destination address in the memory according to the selected queue given entry by the output circuit; specifying a next selected queue entry by the buffer address control circuit in the following manner: when the selected queue given entry is not the last entry in the selected queue, specifying the next selected queue entry as the last entry in the selected queue; an entry after the given entry, and using the next entry as the given entry and using another packet received from outside the network element and containing data to perform again the writing according to the selected queue given entry, and the designating the next selected queue entry; and when the selected queue given entry is the last entry in the selected queue, perform the following: when any queue among the multiple queues has not been selected, select a different queue from the multiple queues, and using another packet received from outside the network element and containing data to perform again the reading of the selected queue given entry, the writing according to the selected queue given entry, and the designating the next selected queue entry; and when all queues among the multiple queues have been selected, using another packet received from outside the network element and containing data and continuing the reading of the given entry from the first queue by the buffer address control circuit.
[0021] Further in accordance with an exemplary embodiment of the present invention, each queue of the plurality of queues comprises a receive data queue (RDQ), and each entry in each RDQ of the plurality of queues comprises a work queue entry (WQE).
[0022] Further according to an exemplary embodiment of the present invention, the packet comprises a plurality of packets each comprising data, and the method further comprises, before continuing with the reading of the first given entry from the first queue: the network element discarding at least one packet of the plurality of packets.
[0023] Further in accordance with an exemplary embodiment of the present invention, the packet comprises a plurality of packets each comprising data, and the method further comprises, before proceeding with the reading of the first given entry from the first queue, the network element storing at least one packet of the plurality of packets.
[0024] Further in accordance with an exemplary embodiment of the present invention, the network element includes a network interface controller (NIC).
[0025] Furthermore, according to an exemplary embodiment of the present invention, the network element includes a switch.
[0026] According to another exemplary embodiment of the present invention, a method is also provided, comprising: providing a network element, which comprises a buffer address control circuit and an output circuit; receiving a packet containing data from outside the network element; providing a plurality of queues; and selecting a queue from the plurality of queues, and performing the following items for the selected queue from the plurality of queues: reading a selected queue given entry from the selected queue maintained in a memory of the device outside the network element by the buffer address control circuit, the selected queue having at least a first selected queue entry and a last selected queue entry, the selected queue given entry comprising a destination address in the memory; writing data to the destination address in the memory by the output circuit according to the selected queue given entry; and performing the following items by the buffer address control circuit. The zone address control circuit specifies the next selected queue entry in the following manner: when the given entry of the selected queue is not the last entry in the selected queue, specifying the next selected queue entry as the entry after the given entry in the selected queue, and using the next entry as the given entry and using another packet received from outside the network element and containing data to again perform the writing according to the given entry of the selected queue, and the specifying of the next selected queue entry; and when the given entry of the selected queue is the last entry in the selected queue, selecting a different queue from the multiple queues, and again performing the reading of the given entry of the selected queue, the writing according to the given entry of the selected queue, and the specifying of the next selected queue entry.
[0027] Further in accordance with an exemplary embodiment of the present invention, the network element includes a network interface controller (NIC).
[0028] Further in accordance with an exemplary embodiment of the present invention, the network element comprises a switch.
[0029] According to another exemplary embodiment of the present invention, a network element is also provided, including: a buffer address control circuit, which is configured to read a given entry from a first queue maintained in a memory of a device outside the network element, the first queue having at least a first entry and a last entry, and the given entry includes a destination address in the memory; an output circuit, which is configured to write data to the destination address in the memory according to the given entry, the data being included in a packet received from outside the network element; and a next entry designation circuit, which is configured to designate a next entry in the following manner: when the given entry is not the last entry in the first queue, designating the next entry as an entry after the given entry in the first queue; and when the given entry is the last entry in the first queue, designating the next entry as the first entry in the first queue.
[0030] Further in accordance with an exemplary embodiment of the present invention, the first queue comprises a receive data queue (RDQ), and each entry in the RDQ in the first queue comprises a work queue entry (WQE).
[0031] Further according to an exemplary embodiment of the present invention, the buffer address control circuit is also configured to, before reading the given entry from the first queue, read a second queue given entry from a second queue maintained in the memory of the device outside the network element, the second queue having at least a first second queue entry and a last second queue entry, the second queue given entry including a destination address in the memory, and the output circuit is also configured to write data to the destination address in the second queue given entry, and the buffer address control circuit is also configured to specify a next second queue entry in the following manner: when the second queue given entry is not the last entry in the second queue, specifying the next second queue entry as an entry after the given entry in the second queue; and when the second queue given entry is the last entry in the second queue, reading the given entry from the first queue.
[0032] Further in accordance with an exemplary embodiment of the present invention, the second queue comprises a receive data queue (RDQ), and each entry in the RDQ in the second queue comprises a work queue entry (WQE).
[0033] Further according to an exemplary embodiment of the present invention, the buffer address control circuit is also configured to, before reading the given entry from the first queue, read a selected queue given entry from the selected queue maintained in the memory of the device outside the network element for each selected queue from a plurality of queues, the selected queue having at least a first selected queue entry and a last selected queue entry, the selected queue given entry including a destination address in the memory, and the output circuit is also configured to write data to the destination address in the selected queue given entry, and the buffer address control circuit is also configured to specify a next selected queue entry in the following manner: when the selected queue given entry is not the last entry in the selected queue, specifying the next selected queue entry as an entry after the given entry in the selected queue; and when the selected queue given entry is the last entry in the selected queue and each of the plurality of queues has been processed as a selected queue, reading a given entry from the first queue.
[0034] Furthermore, according to an exemplary embodiment of the present invention, the network element comprises a network interface controller (NIC).
[0035] In addition, according to an exemplary embodiment of the present invention, the network element includes a switch.
[0036] According to another exemplary embodiment of the present invention, a network element is also provided, comprising: a buffer address control circuit, which is configured to read a selected queue given entry from the selected queue maintained in a memory of a device outside the network element for each selected queue from a plurality of queues, the selected queue having at least a first selected queue entry and a last selected queue entry, the selected queue given entry including a destination address in the memory; and an output circuit, which is configured to write data to the destination address in the memory according to the given entry, the data being included in a packet received from outside the network element, wherein the buffer address control circuit is further configured to specify a next selected queue entry in the following manner: when the selected queue given entry is not the last entry in the selected queue, specifying the next selected queue entry as an entry after the given entry in the selected queue; and when the selected queue given entry is the last entry in the selected queue, selecting a different queue from the plurality of queues and using the different queue as the selected queue.
[0037] Further in accordance with an exemplary embodiment of the present invention, the network element includes a network interface controller (NIC).
[0038] Further in accordance with an exemplary embodiment of the present invention, the network element comprises a switch.
[0039] Furthermore, according to an exemplary embodiment of the present invention, each of the plurality of queues comprises a receive data queue (RDQ), and each entry in each RDQ of the plurality of queues comprises a work queue entry (WQE).
[0040] In addition, according to an exemplary embodiment of the present invention, the packet includes a plurality of packets, each packet containing data, and the network element is further configured to discard at least one packet of the plurality of packets before the next entry designation circuit designates the next entry as the first entry in the first queue.
[0041] Further according to an exemplary embodiment of the present invention, the packet includes multiple packets, each packet contains data, and the network element is further configured to discard at least one packet of the multiple packets before the next entry designation circuit designates the next entry as the first entry in the first queue. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The present invention will be more fully understood and appreciated through the following detailed description taken in conjunction with the accompanying drawings, in which:
[0043] Figure 1 is a simplified block diagram illustration of an input-output queuing system constructed and operated in accordance with an exemplary embodiment of the present invention;
[0044] Figure 2 is a simplified block diagram illustration of an input-output queuing system constructed and operated in accordance with another exemplary embodiment of the present invention;
[0045] Figure 3 yes Figure 2 A simplified block diagram illustration of an exemplary implementation of a system;
[0046] Figure 4 yes Figure 2 A simplified flowchart illustration of an exemplary method of operation of the system; and
[0047] Figure 5 yes Figure 2 A simplified flowchart illustration of another exemplary method of operation of a system. DETAILED DESCRIPTION
[0048] Reference now Figure 1 , which is a simplified block diagram illustration of an input-output queuing system constructed and operated in accordance with an exemplary embodiment of the present invention. Figure 1 The system (generally designated 101) includes the following:
[0049] a host memory 103 contained in a host device (not shown); the host device may be, for example, a suitable processor packaged with the network element, or may be a suitable processor external to the network element and communicating therewith via a suitable communication mechanism (such as PCI-e, by way of non-limiting example); and
[0050] The network element 105, for example, may include a switch (for example, by way of non-limiting example, a suitable switch based on a Spectrum-2 ASIC, such switches (a specific example of such a switch is the SN2700 switch) commercially available from Mellanox Technologies Ltd.) or a network interface controller (NIC) (which may be any suitable NIC, such as, for example, a ConnectX-5 NIC commercially available from Mellanox Technologies Ltd., by way of a specific non-limiting example).
[0051] The host memory 103 stores a plurality of work queue entries (WQEs). Figure 1 107, WQE1 109, WQE2 111, WQE3 113, and (other WQEs not shown until) WQEn 115. It should be understood that Figure 1 The specific number of WQEs shown in is not meant to be limiting, and in some cases, by way of non-limiting example, there may be hundreds or thousands of WQEs.
[0052] Multiple WQEs are maintained in a received data queue (RDQ) 120. It should be appreciated that, for simplicity of illustration, multiple WQEs are depicted as being in a single RDQ 120; in some exemplary embodiments, there may be multiple RDQs rather than a single RDQ.
[0053] Each of the plurality of WQEs contains a host memory address; Figure 1 In a simplified description of:
[0054] WQE0 107 stores WQE0 host memory address 122;
[0055] WQE1 109 stores WQE1 host memory address 124;
[0056] WQE2 111 stores WQE2 host memory address 126;
[0057] WQE3 113 stores WQE3 host memory address 128; and
[0058] WQEn 115 stores the WQEn host memory address 130 .
[0059] Each of host memory addresses 122 , 124 , 126 , 128 , and 130 may be considered a pointer to a location in host memory 103 .
[0060] Now briefly describe Figure 1 105. A plurality of incoming packets are received at the network element 105. To simplify the depiction and description, Figure 1 In FIG, multiple incoming packets are shown as:
[0061] Group 0 132;
[0062] Group 1 134;
[0063] Group 2 136;
[0064] Group 3 138; and
[0065] (Other packets not shown, up to) packet n 140.
[0066] It will be appreciated that in practice a much larger number of packets may be received.
[0067] When a given packet, such as Packet 0 132, is received at the network element 105, the network element 105 reads the next WQE in the RDQ 120; in the specific example of Packet 0 132, the next WQE is the first WQE, WQE0 107. The network element 105 then determines (in the specific non-limiting example of WQE0 107) the host memory address 122 stored in WQE0 107 and stores the data of Packet 0 132 (generally including all of its data, but possibly only a portion thereof) at the specified address location of the host memory 103; Figure 1 , the location for storage of data from packet 0 based on host memory address 122 is indicated by reference numeral 142 .
[0068] When the next packet, Packet 1 134, arrives, the next WQE, WQE1 109, is accessed by the network element 105, and the data of Packet 1 134 is then stored in the specified address location of the host memory 103 based on the host memory address 124 in WQE1 109. Figure 1 In FIG. 1 , the location for storage of data from packet 1 is indicated by reference numeral 144 .
[0069] Similarly, further incoming packets ( Figure 1The data (depicted as packet 2 136, packet 3 138, and packet n 140) is stored at a specified address location ( Figure 1 denoted by reference numerals 146, 148 and 150).
[0070] like Figure 1 As depicted in FIG. 1 , it should be understood that the order of host memory addresses used for packet data storage is not necessarily the same as the order of WQEs; for example, in Figure 1 , host memory address 148 associated with WQE3 113 is shown as being between host memory address 142 associated with WQE0 107 and host memory address 144 associated with WQE1 109 .
[0071] As mentioned above, it should be understood that Figure 1 In the exemplary embodiment of FIG. 1 , this may be the case—particularly if the network element 105 implements a method in which a portion of the network traffic generates an event (in Figure 1 In an exemplary embodiment of the present invention, a high-speed network (corresponding to packets 132, 134, 136, 138 and 140) - the event (which, by way of non-limiting example, may include: packets with errors; a certain fixed percentage of received packets; etc.) can be sent to a host (not shown) at a high rate for storage in host memory 103.
[0072] In the described case of a high rate of incoming packets, it will be appreciated that memory consumption in the host memory 103 is high and therefore the memory allocated for received data ( Figure 1 142, 144, 146, 148 and 150 in the host memory 103 may fill up quickly. Once the memory allocated for received data in the host memory 103 is full, additional WQEs in the RDQ 120 and additional memory allocated for received data will be allocated by the host (not shown) to allow for the reception of additional packets. In such a case, if the additional WQEs in the RDQ 120 and additional memory allocated for received data are not provided quickly enough ("fast enough" in terms of the rate of received packets), the network element 105 will generally be unable to write further data to the host memory 103, so that the incoming packets will be lost due to being discarded by the network element 105. Alternatively, the network element 105 may prevent packet loss by storing as much as possible until a WQE becomes available, but due to the limited number of packets that can be stored in the network element 105, such a scenario may cause "back pressure", which may lead to widespread network congestion, as is known in the art under the "back pressure" scenario.
[0073] Reference now Figure 2, which is a simplified block diagram illustration of an input-output queuing system constructed and operated in accordance with another exemplary embodiment of the present invention.
[0074] Figure 2 The system, generally denoted 201, includes the following:
[0075] Host memory 203, which is contained in a host device (not shown); the host device can be similar to the host device described above with reference to Figure 1 The host device; and
[0076] The network element 205, which may include, for example, a switch or a network interface controller (NIC), may be similar to the one described above with reference to Figure 1 Those switches or NICs mentioned.
[0077] The host memory 203 stores a plurality of work queue entries (WQEs), Figure 2 207, WQE1 209, WQE2 211, WQE3 213, and (other WQEs not shown until) WQEn 215. It should be understood that Figure 2 The specific number of WQEs shown in is not meant to be limiting, and in some cases, by way of non-limiting example, there may be hundreds or thousands of WQEs.
[0078] Multiple WQEs are held in a receive data queue (RDQ) 220. It should be appreciated that, for simplicity of illustration, multiple WQEs are depicted as being in a single RDQ 220; in certain exemplary embodiments, there may be multiple RDQs rather than a single RDQ.
[0079] Each of the plurality of WQEs contains a host memory address; Figure 2 In a simplified description of:
[0080] WQE0 207 stores WQE0 host memory address 222;
[0081] WQE1 209 stores WQE1 host memory address 224;
[0082] WQE2 211 stores WQE2 host memory address 226;
[0083] WQE3 213 stores WQE3 host memory address 228; and
[0084] WQEn 215 stores the WQEn host memory address 230 .
[0085] Each of host memory addresses 222 , 224 , 226 , 228 , and 230 may be considered a pointer to a location in host memory 203 .
[0086] Now briefly describe Figure 2 1. A plurality of incoming packets are received at network element 205. To simplify the depiction and description, Figure 2 In FIG, multiple incoming packets are shown as:
[0087] Group 0 232;
[0088] Group 1 234;
[0089] Group 2 236;
[0090] Group 3 238;
[0091] (other packets not shown, up to) packet n 240; and
[0092] Group n+1 252.
[0093] It will be appreciated that in practice a much larger number of packets may be received.
[0094] When a given packet, such as Packet 0 232, is received at the network element 205, the network element 205 accesses the next WQE in the RDQ 220; in the specific example of Packet 0 232, the next WQE is the first WQE, WQE0 207. The network element 205 then determines (in the specific non-limiting example of WQE0 207) the host memory address 222 stored in WQE0 207 and stores the data of Packet 0 232 (similar to the above reference to RDQ 220). Figure 1 The mechanism described above) at a specified address location of the host memory 203; at Figure 2 , the location for storage of data from group 0 based on host memory address 222 is indicated by reference numeral 242 (as explained in more detail below, to simplify the depiction and description, host memory address 242 is shown as if host memory address 242 is "outside" host memory 203, when in fact host memory address 242 is contained within host memory 203).
[0095] When the next packet, packet 1 234, arrives, the next WQE, namely WQE1 209, is accessed by network element 205, and the data of packet 1 234 is then stored in the specified address location of host memory 203 based on the host memory address 224 in WQE1 209. Figure 2 , the location of storage of data for packet 1 is indicated by reference numeral 244 .
[0096] Similarly, further incoming packets ( Figure 2The data (depicted as packet 2 236, packet 3 238, and packet n 240) are stored at designated address locations ( Figure 2 denoted by reference numerals 246, 248 and 250).
[0097] like Figure 2 As depicted in FIG. 1 , it should be understood that the order of the host memory addresses for storage of the data portion of the packet is not necessarily the same as the order of the WQEs; for example, in Figure 2 , host memory address 244 associated with WQE1 209 is shown as being between host memory address 248 associated with WQE3 213 and host memory address 246 associated with WQE2 211 .
[0098] As mentioned above, it should be understood that Figure 2 In the exemplary embodiment of FIG. 1 , this may be the case—particularly if the network element 205 implements a method in which a portion of the network traffic generates an event (in Figure 2 In the exemplary embodiment of FIG. 2 , corresponding to packets 232, 234, 236, 238, and 240) of FIG. 2 , the high speed network may send the event to a host (not shown) at a high rate for storage in host memory 203. In the described case of high rate incoming packets, it should be understood that the rate of memory consumption in host memory 203 is high, and therefore, the memory allocated for received data ( Figure 2 Once the memory allocated for received data in host memory 203 is full and an additional packet, such as packet n+1 252, is received, network element 205 accesses RDQ 220 in a "round-robin" manner, whereby after WQEn 215 has been accessed, the next WQE accessed for packet n+1 252 is WQE0 207, causing the data portion of packet n+1 252 to be stored at host memory address 254 (which is actually the same as host memory address 242), thereby replacing the data previously stored at that location (at Figure 2 In the exemplary embodiment, the data originally stored at this location is the data of group 0 232).
[0099] It should be understood that the "round-robin" access to the WQEs in the RDQ 220 can continue indefinitely, wherein the WQEs are repeatedly (indefinitely) reused and the locations in the host memory 203 for data storage are repeatedly (indefinitely) reused. In this way, the above reference to Figure 1The problem described above is that the network element 105 will be unable to write further data to the host memory 103, causing incoming packets to be lost (or causing network congestion to occur), despite paying the "price" of overwriting older data stored in the host memory 103. Figure 2 In the exemplary embodiment of FIG. 1 , it will be appreciated that the most recent (newest) packet will generally overwrite the oldest packet in the host's memory. This can allow the most recent (generally, most relevant) data to be kept in memory while consuming less memory than if a very large amount of memory were to be allocated to handle a large number of incoming packets, and reduce CPU load relative to a situation where more and more WQEs and more and more memory locations are allocated to handle a large number of incoming packets.
[0100] In other exemplary embodiments of the present invention, the same method as described above can be first performed. Figure 1 The operation is similar until all WQEs in RDQ 120 have been used; and then the WQEs can be used in a "round-robin" manner. Figure 2 The WQE of RDQ 220 is carried out in the same manner as above. Figure 2 In this way, in addition to retaining data from the most recent (latest) packet received, data from the first (oldest) packet received may also be retained. In further exemplary embodiments, more than one RDQ may be provided, for example Figure 1 120 of RDQs, wherein the above reference is performed once for each RDQ Figure 1 The operation described above; and can then be used in a "loop" manner Figure 2 The WQE of RDQ 220 is carried out in the same manner as above. Figure 2 The operations described above are similar to the operations described above.
[0101] In further exemplary embodiments, more than one RDQ may be provided, e.g. Figure 1 120 of RDQs, wherein the above reference is performed once for each RDQ Figure 1 In this exemplary embodiment, if a sufficient number of RDQs are provided, even if the RDQs are not used in a "round-robin" manner (e.g. Figure 2 RDQ 220), similar to the Figure 2 The advantages of the system are explained.
[0102] Reference now Figure 3 , which is Figure 2 A simplified block diagram illustration of an exemplary implementation of a system.
[0103] Figure 3 An exemplary implementation of includes the following:
[0104] Network element 305, which may be as described above Figure 2 as stated; and
[0105] The external device 310 includes a memory 315, both of which may be as described above with reference to Figure 2 described.
[0106] exist Figure 3 The network element 305 is depicted as including the following elements, and it should be understood that other elements (not shown, which may include conventional elements of a conventional network element) may also be included in the network element 305:
[0107] Buffer address control circuit 320;
[0108] Output circuit 325; and
[0109] The next entry specifies circuit 330 .
[0110] It should be understood that although the buffer address control circuit 320, the output circuit 325 and the next entry designation circuit 330 are shown as separate, they can be combined in various ways in actual implementation; by way of non-limiting example, the buffer address control circuit 320 and the next entry designation circuit 330 can be combined into a single element.
[0111] Now briefly describe Figure 3 Example operating modes of an example implementation of.
[0112] A packet is received at the network element 305 from a source external thereto (shown as a single packet 335 for simplicity, it should be understood that as described above with reference to Figure 2 As described above, a large number of packets can be processed).
[0113] The buffer address control circuit 320 and the next entry designation circuit 330 are configured together to access one or more RDQs ( Figure 3 ) in WQE, as described above with reference to Figure 1 and Figure 2 For example, the buffer address control circuit 320 may be configured to access a given WQE in the RDQ and to supply the memory address contained in the WQE to the output circuit 325. The next entry designation circuit 330 may be configured to select the next WQE (described above with reference to Figure 1 The method described above, or Figure 2 The circulation method described above).
[0114] When accessing RDQ, you can refer to the above Figure 1 Access zero, one or more RDQs in the manner described, then follow the instructions above. Figure 2Alternatively, you can use the "loop" method described above to access one or more RDQs. Figure 1 access multiple RDQs in the manner described above, without following the Figure 2 The "round robin" approach described visits any RDQ.
[0115] Output circuit 325 is configured to output the WQE in the RDQ (neither in Figure 3 ), writes data from an incoming packet (eg, packet 335) into memory 315 at an address in FIG. 314 ; the address is supplied by the buffer address control circuit as described above.
[0116] Reference now Figure 4 , which is Figure 2 A simplified flow chart illustration of an exemplary method of operation of a system. Figure 4 The method may include the following steps:
[0117] A network element is provided, which includes at least a buffer address control circuit and an output circuit (step 405).
[0118] A packet containing data is received from outside the network element (step 410).
[0119] The buffer address control circuit reads a given entry from a (first) queue maintained in a memory of a device external to the network element. The queue has at least a first entry and a last entry. It should be understood that whenever a queue is indicated herein as having a first entry and a last entry, the queue may alternatively have only one entry, which would be both the first entry and the last entry in the queue; thus, the description of the "first entry" and the "last entry" in a queue is not limiting, and such a queue may have only one entry. The given entry includes a destination address in the memory (step 415).
[0120] The output circuit writes data to the destination address in the memory according to the given entry (step 420).
[0121] The next entry is designated by the buffer address control circuit as follows: when the given entry is not the last entry in the (first) queue, the next entry is designated as the entry after the given entry in the (first) queue; when the given entry is the last entry in the (first) queue, the next entry is designated as the first entry in the (first) queue (step 425).
[0122] The next entry (as specified in step 425) is used as the given entry (step 430). Processing then continues with step 420.
[0123] Reference now Figure 5 , which is Figure 2A simplified flowchart illustration of another exemplary method of operation of a system. Figure 5 The method may include the following steps:
[0124] A network element is provided, which includes at least a buffer address control circuit and an output circuit (step 505).
[0125] A packet containing data is received from outside the network element (step 510).
[0126] A queue is selected from the provided plurality of queues, and the buffer address control circuit reads a given entry from the selected queue maintained in the memory of the device external to the network element. The selected queue has at least a first entry and a last entry. The given entry includes a destination address in the memory (step 515).
[0127] The output circuit writes data to the destination address in the memory according to the given entry (step 520).
[0128] The next entry is designated by the buffer address control circuit as follows: when the given entry is not the last entry in the given queue, the next entry is designated as the entry after the given entry in the given queue; when the given entry is the last entry in the given queue, another queue in the plurality of queues is selected as the given queue, and the next entry is designated as the first entry in the (new) given queue (steps 525 and 530). Processing then continues with step 520.
[0129] It will be appreciated that the software components of the present invention may be implemented in the form of a ROM (read only memory) if desired. If desired, the software components may generally be implemented in hardware using conventional techniques. It will also be appreciated that the software components may be instantiated, for example, as a computer program product or on a tangible medium. In some cases, it may be possible to instantiate the software components as signals that can be interpreted by a suitable computer, although such instantiation may be excluded in certain embodiments of the present invention.
[0130] It should be understood that, for the sake of clarity, the various features of the present invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity, the various features of the present invention described in the context of a single embodiment may also be provided separately or in any suitable sub-combination form.
[0131] It should be understood by those skilled in the art that the present invention is not limited by the specific representations and descriptions above. Instead, the scope of the invention is determined by the appended claims and their equivalents.
Claims
1. A method for writing data into a memory, comprising: Providing a network element, which includes a buffer address control circuit and an output circuit; receiving a packet containing data from outside the network element; Reading a given work queue entry from a first receive data queue held in the memory by the buffer address control circuit, wherein the memory is arranged in a device outside the network element, the first receive data queue has a plurality of work queue entries, each work data queue entry includes a destination address in the memory, and the plurality of work data queue entries include at least a first work queue entry and a last work queue entry; designating the first work queue entry as a given work queue entry; and writing the data to the destination address in the memory by the output circuit according to the given work queue entry; The buffer address control circuit specifies the next work queue entry in the following manner: When the given work queue entry is not the last work queue entry in the first received data queue, specifying the next work queue entry as a work queue entry after the given work queue entry in the first received data queue; as well as When the given work queue entry is the last work queue entry in the first work receive data queue, designating the next work queue entry as the first work queue entry in the first receive data queue again, thereby reusing the first work queue entry in the first receive data queue and allowing the destination address included in the first work queue entry to be written again; as well as The writing and the designating are performed again using the next work queue entry as the given work queue entry and using another packet including data and received from outside the network element and containing data.
2. The method according to claim 1, and further comprising: Before reading the given work queue entry from the first receive data queue, performing the following: reading, by the buffer address control circuit, a second queue given work queue entry from a second queue maintained in the memory of the device outside the network element, the second queue having at least a first second work queue entry and a last second work queue entry, the second queue given work queue entry including a destination address in the memory; Writing data to the destination address in the memory by the output circuit according to a given work queue entry of the second queue; The buffer address control circuit specifies the next second work queue entry in the following manner: When the second-queue given work queue entry is not the last entry in the second received data queue, designating the next second-queue work queue entry as the work queue entry after the second-queue given work queue entry in the second received data queue, and using the next second-queue work queue entry as the given work queue entry and using another packet received from outside the network element and containing data to perform again: the writing according to the second-queue given work queue entry, and the designating the next second-queue work queue entry; and When the given work queue entry of the second queue is the last work queue entry in the second queue, the buffer address control circuit continues reading the first work queue entry from the first receive data queue using another packet received from outside the network element and containing data.
3. The method according to claim 1, and further comprising: Provide multiple queues; Selecting a queue from the plurality of queues and for the selected queue from the plurality of queues, performing the following before reading the given work queue entry from the first receive data queue: reading, by the buffer address control circuit, a selected queue given work queue entry from the selected queue maintained in the memory of the device external to the network element, the selected queue having at least a first selected work queue entry and a last selected work queue entry, the selected queue given work queue entry including a destination address in the memory; The output circuit writes data to the destination address in the memory according to a given work queue entry of the selected queue; and The buffer address control circuit specifies the next selected queue work queue entry in the following manner: When the selected queue given work queue entry is not the last work queue entry in the selected queue, designating the next selected work queue entry as the work queue entry after the given work queue entry in the selected queue, and using the next selected queue work queue entry as the given work queue entry and using another packet received from outside the network element and containing data to perform again: the writing according to the selected queue given work queue entry, and the designating the next selected work queue entry; and When the selected queue given work queue entry is the last entry in the selected queue, performing the following: When any queue of the plurality of queues has not been selected, selecting a different queue from the plurality of queues, and performing again the reading of the selected queue given work queue entry, the writing according to the selected queue given work queue entry, and the designating of the next selected work queue entry using another packet received from outside the network element and containing data; and When all queues of the plurality of queues have been selected, another packet received from outside the network element and containing data is used by the buffer address control circuit and the reading of the first working queue entry from the first receiving data queue is continued.
4. The method of claim 1, and wherein the packet comprises a plurality of packets each comprising data, and the method further comprises: Before continuing to read the first given work queue entry from the first receive data queue: the network element discards at least one packet from the plurality of packets.
5. The method of claim 1, and wherein the packet comprises a plurality of packets each comprising data, and the method further comprises: Before proceeding with reading the given work queue entry from the first receive data queue: the network element stores at least one packet of the plurality of packets.
6. The method of claim 1, and wherein the network element comprises a network interface controller (NIC).
7. The method of claim 1, and wherein the network element comprises a switch.
8. A network element, comprising: a buffer address control circuit configured to read a given work queue entry from a first receive data queue maintained in a memory of a device external to the network element, the first receive data queue having a plurality of work queue entries, the plurality of work queue entries including at least a first work queue entry and a last work queue entry, the given work queue entry including a destination address in the memory; an output circuit configured to write data to the destination address in the memory according to the given work queue entry, the data being contained in a packet received from outside the network element; as well as A next entry designation circuit is configured to designate a next work queue entry by: When the given work queue entry is not the last work queue entry in the first receive data queue, specifying the next work queue entry as a work queue entry after the given work queue entry in the first receive queue; as well as When the given work queue entry is the last work queue entry in the first receive data queue, the next work queue entry is again designated as the first work entry in the first receive data queue, thereby reusing the first work queue entry in the first receive data queue.
9. The network element according to claim 8, and wherein the buffer address control circuit is further configured to, before reading the given work queue entry from the first receive data queue, read a second queue given work queue entry from a second queue maintained in the memory of the device external to the network element, the second queue having a plurality of work queue entries, the second queue given work queue entry including a destination address in the memory, and The output circuit is further configured to write data to the destination address in a given work queue entry of the second queue, And the buffer address control circuit is further configured to specify the next second queue entry in the following manner: When the given work queue entry of the second queue is not the last work queue entry in the second queue, specifying the next second work queue entry as the work queue entry after the given work queue entry in the second queue; and When the given work queue entry of the second queue is the last work queue entry in the second queue, the given entry from the first work queue entries is read from the first receive data queue.
10. The network element according to claim 8, and wherein the buffer address control circuit is further configured to, before reading the given work queue entry from the first receive data queue, read a selected queue given work queue entry from the selected queue maintained in the memory of the device outside the network element for each selected queue from a plurality of queues, the selected queue having a plurality of work queue entries, the plurality of work queue entries including at least a first work queue entry and a last work queue entry, the selected queue given work queue entry including a destination address in the memory, and The output circuit is further configured to write data to the destination address in a given work queue entry of the selected queue, And the buffer address control circuit is further configured to specify the next selected queue entry by: When the selected queue given work queue entry is not the last entry in the selected queue, designating the next selected work queue entry as the entry after the given work queue entry in the selected queue; and When the selected queue given entry is the last entry in the selected queue and each of the plurality of queues has been processed as a selected queue, a given work queue entry is read from the first receive data queue.
11. The network element of claim 8, and wherein the network element comprises a network interface controller (NIC).
12. The network element of claim 8, and wherein the network element comprises a switch.
13. The network element of claim 8, and wherein the packet comprises a plurality of packets, each packet containing data, and The network element is further configured to discard at least one packet of the plurality of packets before the next entry designation circuit designates the next entry as the first entry in the first receive data queue.
Citation Information
Patent Citations
Method and system for storing and retrieving data
CN103309928A
Ordering and bandwidth improvements for load and store unit and data cache
CN105765525A
Methods, queueing system, network element and network system for queueing and processing of packets
CN107431668A
Receive queue with stride-based data scattering
CN108536543A
Network interface controller with circular receive buffer
US20130103777A1
