NIC line rate hardware packet processing

Abstract

The present disclosure relates to a NIC line rate hardware packet processing. A packet processing method includes: searching a first matching action table on a network interface card (NIC) for a received packet; in response to finding a matching entry indicating an action, queuing the received packet in a first queue and storing the action data in an instruction memory; and in response to not finding a matching entry, queuing the received packet in the first queue and a second queue. The method includes: selecting a first packet from the first queue for processing, the processing including executing a corresponding action stored in the instruction memory; selecting a second packet from the second queue for processing, the processing including forwarding a portion of the second packet to a processor, the processor searching in a second matching action table; and receiving a search result from the processor, thereby allowing a third packet in the first queue corresponding to the second packet to be processed based on the search result.

Description

Technical Field

[0001] The present disclosure generally relates to packet processing in a network interface card (NIC). More specifically, the present disclosure relates to a hardware-based packet processing engine capable of offloading overflow processing to a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 A diagram illustrating an example of a high-level schematic diagram of a packet engine according to one aspect is presented.

[0003] Figure 2 A diagram illustrating an example of a packet processing system according to one aspect is presented.

[0004] FIG. 3A to FIG. 3E An example of a packet queuing and dequeuing scenario according to one aspect is illustrated.

[0005] Figure 4 A flow chart illustrating an example of packet processing operations performed by a packet engine on a network interface card (NIC) according to one aspect is presented.

[0006] Figure 5 An example of a computer system that facilitates off-NIC match action lookup according to one aspect is illustrated.

[0007] In the drawings, like reference numerals refer to like drawing elements. DETAILED DESCRIPTION

[0008] The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is not limited to the illustrated embodiments, but is intended to conform to the maximum scope consistent with the principles and features disclosed herein.

[0009] In order to meet the bandwidth requirements of high performance computing (HPC), hardware-based packet processing is required at the network interface card (NIC) of the processor. Many hardware-based packet processing engines (e.g., P4 (Packet Processor Independent of Programming Protocol) packet engine pipelines) can implement a matching action table to determine how to process packets before release. The matching action table can include multiple entries, wherein each entry includes a matching field or multiple matching fields and a corresponding set of packet processing actions. In addition to large bandwidth, HPC customers have increasing requirements for features that can support security and isolation of workloads running simultaneously on the system. In order to support a large number of tenants sharing NICs and processors, the packet engine on the NIC needs to implement a sufficiently large matching action table (e.g., a table that can be several gigabytes in size). However, such a large matching action table exceeds the memory capacity of a traditional NIC, which can typically have tens of megabytes of storage space.

[0010] The present disclosure provides a hardware-based packet engine that implements a larger off-NIC match action table in addition to a match action table on a smaller NIC. After receiving a packet for processing, the packet engine can parse the packet header and perform a match action lookup in the match action table on the smaller NIC based on the information included in the packet header. Packets with matching entries in the match action table on the NIC (i.e., the match field of the entry matches one or more header fields of the packet) can be queued in a packet class-specific queue based on the corresponding category to which the packet belongs. Packets can be classified based on various criteria (e.g., traffic category, priority category, source / destination subnet, etc.). The scope of the present disclosure is not limited by the criteria for packet classification. On the other hand, packets that do not have matching entries in the match action table on the NIC can be sent to a special queue called a fill request queue in addition to being sent to the corresponding packet class-specific queue. Packets in the fill request queue can be processed by the packet engine like ordinary packets, wherein the default action is to send the packet header to the fill request handler running on the processor. The fill request handler can perform a matching action lookup in a larger matching action table stored in the processor's DRAM and return the lookup results to the packet engine, which can then add the results to the matching action table on the NIC and process the packet accordingly. The use of packet class specific queues ensures that packets that do not have matching entries in the matching action table on the NIC do not block the flow of packets belonging to other classes. Packets of different classes can be queued in other packet class specific queues and processed according to their corresponding matching action entries. For packets within the same class, the size of the packet class specific queues can be designed to be large enough so that the response to the fill request (i.e., the lookup result) is likely to reach the packet engine before the queue with the unmatched packet is filled, thereby allowing uninterrupted packet flow within the same packet class queue.

[0011] Figure 1 A diagram illustrating an example of a high-level schematic diagram of a packet engine according to one aspect is presented. The packet engine 100 can be part of a NIC and can include lookup logic 102, action logic 104, and egress logic 106. Various logics in the packet engine 100 can be implemented using hardware or software logic. According to some aspects, the various logics in the packet engine 100 can be implemented using hardware to ensure that packet processing speeds can meet HPC bandwidth requirements.

[0012] The lookup logic 102 may be responsible for performing a match action lookup to determine the action to be performed on the incoming data packet. For example, the lookup logic 102 may receive a data packet (e.g., an Ethernet data packet) from an external or loopback interface of the NIC and parse the data packet to extract header fields to be used in the match action lookup operation. More specifically, the lookup logic 102 may use the extracted header fields as search keywords to search a match action table on the NIC (which may be implemented using a content addressable memory (CAM) or as a hash table stored in a random access memory (RAM)). For example, a match entry in the table may have a match field that matches one or more header fields of the data packet. According to some aspects, the size of the match action table on the NIC may be several megabytes. The match action table may define multiple actions to be performed on the incoming data packet, including, but not limited to: modification, steering, filtering of the data packet; triggering an update to the data packet engine state; or other management functions, such as collecting and reporting telemetry information. The lookup results (i.e., match entries) from the lookup logic 102 may be forwarded to the action logic 104, which is responsible for applying a set of actions to the incoming data packet based on the action rules included in the lookup results. Egress logic 106 may be responsible for publishing packets to other network nodes (e.g., via a network fabric). In this example, a NIC (on which packet engine 100 resides) may be interconnected with other NICs via a network fabric comprising various types of network devices (e.g., switches, routers, access points, etc.).

[0013] Note that the match action table on the NIC may not be large enough to handle all connections that may be valid at any given time. If the search result of the lookup logic 102 indicates a missing (i.e., no match action entry is found for the incoming data packet in the match action table on the NIC), the action logic 104 can perform a default action, that is, forwarding the incoming data packet (or a portion of the data packet including its header) to the NIC external lookup logic 108. According to some aspects, the NIC external lookup logic 108 can reside on a processor coupled to the NIC. The NIC external lookup logic 108 can perform a match action search in a larger match action table (e.g., a match action table stored in a dynamic random access memory (DRAM) attached to the processor). In theory, the larger match action table can have an arbitrary size to include all possible match action entries in the network, including those that cannot fit into the smaller match action table on the NIC. The NIC external lookup logic 108 can return the search result to the action logic 104 so that the appropriate action can be performed on the incoming data packet. In addition, the NIC external lookup logic 108 can also return the search result to the lookup logic 102 so that the result can be added to the match action table on the NIC. By performing a default action when the lookup logic 102 fails to find a matching action entry, the action logic 104 can handle unmatched packets (i.e., packets that do not have a matching entry in the matching action on the NIC) in a manner similar to matched packets (i.e., packets that have a matching entry), thereby simplifying the operation of the action logic 104.

[0014] In order to prevent mismatched packets from blocking services or being out of order while waiting for the lookup results from the NIC external lookup logic 108, the packet engine 100 may implement a specially designed queuing structure ( Figure 1 104) to hold packets processed by the lookup logic 102 before the action logic 104 applies an action to the packets. According to some aspects, the queuing structure may include multiple separate logical queues for queuing packets belonging to different packet classes, one logical queue for each packet class. These logical queues may be referred to as packet class-specific queues because they are specific to the specific class to which the packets belong. There are different criteria that can be used for packet classification. In one example, packets may be classified based on their business class, and these logical queues may be business class-specific queues. Note that packets from different classes flow through the packet engine independently of each other, and the packet class-specific queues are also advanced independently of each other. In addition to the packet class-specific queues, the queuing structure may also include a special queue called a fill request queue for holding unmatched packets. More specifically, unmatched packets may be queued in both the corresponding packet class-specific queues and the fill request queues.

[0015] When an unmatched packet is dequeued from the fill request queue, the action logic 104 may perform a default action for forwarding the unmatched packet to the NIC external lookup logic 108 on the dequeued packet. According to some aspects, in order to reduce bandwidth usage, the action logic 104 may extract the packet header and forward only the packet header to the NIC external lookup logic 108. According to alternative aspects, the action logic 104 may truncate the packet (e.g., retain only a predetermined initial portion of the packet and discard the rest of the packet) and forward the truncated packet (which includes the packet header) to the NIC external lookup logic 108. The NIC external lookup logic 108 may include a fill request handler executed by a management service processor on a side port of the NIC. Alternatively, the fill request handler may be executed by a processor attached to the fabric or by a main processor of the NIC. The fill request handler may also include specially constructed hardware for responding to fill requests. The fill request handler may perform a matching action lookup operation in a larger matching action table maintained in the processor's DRAM. According to one aspect, the size of the larger matching action table may be several gigabytes. Note that the processor executing the fill request handler can be the same processor responsible for managing all structures in packet engine 100. Thus, the processor can add entries to the larger match action table when a connection is established, remove entries when a connection is closed, and modify entries as needed to implement "stateful" packet processing.

[0016] Figure 2 A diagram illustrating an example of a packet processing system according to one aspect is presented. Packet processing system 200 may include NIC 202 and processor 204. According to some aspects, processor 204 may be a main processor of NIC 202. According to alternative aspects, processor 204 may be a separate management processor coupled to NIC 202 via a side port on NIC 202. Processor 204 may also be a management processor attached to a fabric.

[0017] NIC 202 may include multiple hardware logic and memory to form a hardware-based packet processing pipeline. Figure 2 In the example shown, NIC 202 may include a multiplexer 206, lookup logic 208, a hash table 210, a plurality of packet class specific queues 212, a fill request queue 214, a packet buffer 216, an instruction RAM 218, an arbiter 220, and modification and release logic 222. Processor 204 may include a management agent 224 and a memory 226.

[0018] Incoming packets arriving at NIC 202 may be sent to lookup logic 208 via multiplexer 206. Lookup logic 208 may include a packet parser ( Figure 2 ), the packet parser can parse the incoming packet to extract its header fields (e.g., L1, L2, L3, and L4 header fields). The lookup logic 208 can perform a matching action lookup based on the extracted header fields. In this example, the lookup logic 208 can search the hash table 210 (i.e., the matching action table) to obtain entries that match the extracted header fields of the incoming packet. There can be multiple hash tables, one for each header field. In addition to hash tables, other types of lookup structures (e.g., CAMs) can also be used to implement matching action tables. After the matching action lookup, the data packet can be queued in a packet class-specific queue 212 based on its corresponding packet class. More specifically, the data packet itself is stored in a packet buffer 216, and the corresponding data packet pointer is queued in a packet class-specific queue 212. Depending on the communication protocol, there can be different ways to classify the data packet. According to some aspects, the data packets can be classified based on priority, so that each packet class-specific queue queues data packets belonging to a specific priority class. According to one aspect, there may be 12 packet classes and the NIC 202 may include 12 corresponding packet class specific queues.

[0019] After performing a lookup operation on an incoming packet, the lookup logic 208 may send the lookup result along with the packet to facilitate packet queuing. The lookup result may indicate whether the match action lookup found a matching entry in the match action table. If the lookup result indicates that no matching entry was found for the packet, the packet is referred to as an unmatched packet and, in addition to being queued in the corresponding packet class-specific queue 212, the unmatched packet is queued in the fill request queue 214. The fill request queue 214 may also be referred to as a missing queue because it queues packets that do not have a match action. For matching packets that have one or more matching entries in the match action table, the lookup result may include those match entry results and may be stored in the instruction RAM 218. Each match entry may specify one or more actions to be performed on the packet.

[0020] Arbitrator 220 may be responsible for selecting the data packets to be published. Figure 2In the embodiment of the present invention, the arbiter 220 can select one queue at a time from multiple queues including the packet class-specific queue 212 and the fill request queue 214 to dequeue (i.e., remove the head of the selected queue). The arbiter 220 can make a selection based on the priority of each queue. According to some aspects, the fill request queue 214 can have a relatively high priority (which may not necessarily be the highest) to ensure that unmatched packets can be processed in a timely manner. The arbiter 220 can also use other arbitration mechanisms. Once the packet is removed from the selected queue, it can be sent to the modification and release logic 222 for processing and release. More specifically, the modification and release logic 222 can access the instruction RAM 218 to obtain the corresponding matching action search result, which specifies the action to be performed on the packet. The packet can be published to the structure after being processed (i.e., the specified action is performed on it).

[0021] For non-matching packets removed from fill request queue 214, instruction RAM 218 can store data associated with a default action. According to some aspects, in response to detecting that a matching action lookup result for a packet indicates a missing or no match is found in hash table 210, instruction RAM 218 can generate an action entry for the packet, thereby specifying that a default action is to be performed on the packet. According to some aspects, the default action can include forwarding the fill request to processor 204, such as Figure 2 As shown. The fill request may include a non-matching data packet or a portion of a non-matching data packet. To reduce bandwidth usage, the non-matching data packet may be truncated, and a predetermined number of initial bytes of the data packet (which may include a data packet header) may be sent to the processor 204. Because the data packets may have different encapsulations, they may have headers of different lengths. The number of bytes included in the truncated data packet may be designed to ensure that the header of each data packet may be included in the truncated data packet, regardless of the protocol used.

[0022] The processor 204 may include a management agent 224 and a memory 226. The management agent 224 (which may be a process executed by the processor 204) may examine the header of the unmatched packet to generate a fill response, which may indicate one or more entries to be inserted / filled into the hash table 210 to allow the packet to proceed through the packet engine. According to some aspects, the memory 226 of the processor 204 may maintain a larger match action table (which may be several orders of magnitude larger than the hash table 210). In one example, the size of the hash table 210 may be several megabytes (e.g., 2MB), and the size of the larger match action table in the memory 226 may be several gigabytes (e.g., 2GB). Upon receiving a fill request (which may include header fields of the unmatched packet), the management agent 224 may perform a match action lookup in the match action table stored in the memory 226 and generate a fill response including the lookup result. In addition to providing the fill response to the NIC 202, the processor 204 may also be responsible for updating the match action table, including the hash table 210 and the larger match action table in the memory 226. For example, when a connection is established, the processor 204 may add a new entry to the matching action table. In some cases, the processor 204 may need to remove old entries from the table to provide space for new entries.

[0023] The fill response from the processor 204 may be sent to the NIC 202. The fill response may enter the packet engine in the same manner as the incoming new packet (i.e., via the multiplexer 206). More specifically, the fill response may be provided to the instruction RAM 218 so that when the corresponding packet is removed from the packet class-specific queue 212, an appropriate action may be performed on the corresponding packet. In addition, the fill response may be provided to the hash table 210, which may add a corresponding matching action entry so that subsequent packets with similar headers may benefit from the added entry. According to some aspects, the table update logic associated with the hash table 210 may update the hash table 210 based on the received fill response. If the hash table 210 is full, the table update logic may remove older entries from the hash table 210 to make room for the fill response. According to some aspects, older entries in the hash table 210 may age and be automatically removed after a predetermined period of time.

[0024] By queuing packets including both matching packets and non-matching packets in a packet class-specific queue, the packet engine on the NIC 202 can continue to process services even when the lookup logic 208 fails to find a matching entry for each packet in the hash table 210. Non-matching packets can be queued in two different queues (i.e., the fill request queue and the queue for its corresponding packet class). While non-matching packets are waiting in these two queues, packets of other classes can continue to be processed by the packet engine without interruption. In fact, if the depth of the queue specific to the packet class is large enough, even packets within the same class can be processed without interruption. More specifically, when the management agent 224 on the processor 204 is performing a match action search, previously received packets of the same class can be moved in the queue. If the search result can be returned to the service engine before the non-matching packet reaches the head of the queue, the flow of this specific class will not be blocked. In addition, because the same non-matching packet is not only queued in the fill request queue, but also queued in the queue specific to the packet class, the non-matching packet will be able to maintain its position in the service flow. Although a fill request is issued using a non-matching packet in a fill request queue, the same non-matching packet in a packet class-specific queue is moved together with other packets in its own class so that the transmission order of packets in the same class (e.g., the same business class or the same flow) can be maintained.

[0025] FIG. 3A to FIG. 3E An example of a packet queuing and dequeuing scenario according to one aspect is illustrated. Figure 3A Multiple arbitration queues are illustrated, including a fill request queue 302 and packet class specific queues 304, 306, and 308. Packets that have been processed by the matching action lookup logic wait in these queues before they are processed by the action logic and posted to the fabric. In the accompanying drawings, some packets are numbered to distinguish one packet from another. For example, packets 1, 4, and 9 belong to the same class and are queued in the packet class specific queue 304; packets 2, 5, and 8 belong to the same class and are queued in the packet class specific queue 306; and packets 3, 6, and 7 belong to the same class and are queued in the packet class specific queue 308. In addition, packet 5 is an unmatched packet, so that packet 5 is also queued in the fill request queue 302 in addition to the packet class specific queue 306. Note that packet 5 is shown as shaded, indicating that it is an unmatched packet.

[0026] Figure 3AIt is also shown that the match action lookup results can be used to assist in packet queuing. More specifically, the match action lookup results associated with each packet can indicate whether the packet is an unmatched packet (i.e., does not have a match action entry in the match action table on the NIC). The queuing logic can send the unmatched packet (e.g., packet 5) to a packet class-specific queue and fill request queue 302 based on the packet class.

[0027] The arbiter 310 can select one queue at a time from the plurality of queues 302-308 to dequeue. According to some aspects, the dequeue operation can be priority-based, wherein packets in queues with higher priorities are removed before packets in queues with lower priorities. To reduce delays for unmatched packets while waiting for a fill response, the fill request queue 302 can be granted a relatively high priority. According to one aspect, the fill request queue 302 can be granted the highest priority of all queues. Figure 3B It shows that due to the high priority of the fill request queue 302, the unmatched packet 5 is removed from the fill request queue 302 first, and then other packets (e.g., packet 1) are removed from the corresponding queue. The packets removed from the queue are sent to the modification and release logic 312, which is responsible for performing appropriate actions on the packets before releasing them.

[0028] Figure 3C This indicates that the unmatched packet 5 is sent to the processor, while the matched packet 1 (after the modification and release logic 312 performs the appropriate action on packet 1) is released to the fabric. To reduce bandwidth usage, a truncated portion of packet 5 including the packet header may be sent to the processor to allow the processor to perform a matching action lookup in a larger matching action table stored in the processor's memory using the packet header as a search keyword, rather than sending the entire unmatched packet 5 to the processor. Figure 3CAs can be seen in FIG, the packet engine (which may include queues 302-308, arbiter 310, and modification and release logic 312) can process packet 5 in a manner similar to a matched packet, except that the packet is sent to a processor instead of a fabric. In other words, an unmatched packet 5 can flow through the packet engine even though it has not yet matched an action entry. When an unmatched packet 5 in the fill request queue 302 flows through the packet engine and leaves the packet pipeline so that its request is processed by the processor, an unmatched packet 5 in the packet class-specific queue 306 remains in the queue, waiting for the processor to return a fill response. If an unmatched packet is at the head of its queue, but the fill response has not yet been received by the packet engine, the corresponding packet class-specific queue may be blocked by the unmatched packet. However, packets from other packet class-specific queues (e.g., packets 2 and 3) can continue to flow through the service engine normally without interruption or blocking. By designing sufficiently deep queues, and by granting relatively high attributes to the fill request queue, the disclosed solution can ensure that the probability of an unmatched packet blocking its packet class-specific queue can be low. According to some aspects, the time required to obtain a fill response (ie, the time required for the processor to perform a lookup and return the lookup result) can be a few microseconds, and the queue depth can be designed accordingly. According to one aspect, each packet class specific queue can accommodate up to 512 entries.

[0029] Figure 3D indicates that the fill response has been received by the packet engine and that packet 5 in the packet class specific queue 306 can be processed by the arbitrator 310 like a regular packet (i.e., the packet can be removed from the packet class specific queue 306 like a regular packet). Figure 3D , packet 5 is no longer shaded, which means it now has a matching action entry. Figure 3D It is also shown that the order in which packets are transmitted within each packet class remains unchanged (eg, packet 5 is transmitted after packet 2).

[0030] from FIG. 3A to FIG. 3DIt can be seen that when a fill request is pending (meaning that the corresponding unmatched packet is waiting for a match action to find a fill response result), the interruption of the packet flow caused by the unmatched packet can be kept to a minimum. By queuing the packets according to their class, the solution ensures that the unmatched packets do not affect the packet flow in other classes. In addition, the unmatched packet does not block the packet flow in the same class (e.g., the same business class or priority class) unless the unmatched packet has reached the head of its queue before the fill response is returned to the business engine. By implementing an additional fill request queue with a high priority, the solution can also minimize the delay of the fill response, thereby reducing the possibility of the queue being blocked by the unmatched queue.

[0031] exist FIG. 3A to FIG. 3D In the example shown, there is only one unmatched data packet waiting for a fill response to be returned. In practice, there may be multiple unmatched data packets pending (i.e., waiting for a fill response to be returned) in the data packet engine at the same time. Figure 3E Multiple non-matching packets (i.e., shaded packets) are shown waiting to be processed in various queues. Note that one pending packet does not prevent other non-matching packets (which may be from the same or different classes) from issuing fill requests to the processor because the fill requests are issued from the fill request queue 302 rather than the packet class specific queues.

[0032] Figure 4 A flow chart illustrating an example of packet processing operations performed by a packet engine on a NIC according to one aspect is presented. During operation, a hardware-based packet engine may receive an incoming packet (operation 402) and perform a matching action lookup in a matching action table on the NIC based on header information of the packet (operation 404). The matching action table on the NIC may be implemented as a hash table stored in a memory on the NIC. Due to area limitations, the size of the matching action table on the NIC may be relatively small (e.g., a few megabytes). The packet engine may determine whether a match is found in the matching action table on the NIC (operation 406). If so, the packet engine may place the packet in a corresponding packet class-specific queue based on the packet class (e.g., business class) of the packet, and place the matching action lookup result (which may include data associated with the action to be performed on the packet) in the instruction RAM (operation 408). For example, the lookup result may specify an action to be performed on the packet before the packet is released to the structure. The packet waits in a packet class-specific queue before being selected by the arbitrator (operation 410). Once the packet is selected, the business engine may perform an appropriate action on the packet based on the lookup result and release the packet (operation 412).

[0033] If the result of the match action search on the NIC is missing (i.e., no matching entry is found in the match action table on the NIC), in addition to placing the packet in a corresponding packet class-specific queue based on the packet class determined according to the predefined packet classification criteria (operation 414), the service engine can also place the packet in a special fill request queue (operation 416). The packet can be advanced in both queues. The fill request queue typically has fewer packets in the queue and has a higher priority, so the packet is first dequeued from the fill request queue and then from the packet class-specific queue. Once it reaches the head of the fill request queue, the packet can wait for arbitration (operation 418). After the arbitrator selects the packet from the fill request queue, the service engine can perform a default action on the packet, that is, sending a fill request including at least a packet header to a process (e.g., a fill request handler) executing on the management processor (operation 420). According to some aspects, the service engine can truncate the packet and send a predetermined number of initial bytes of the packet to the management processor. Sending only the initial portion of the packet instead of the entire packet to the processor can reduce bandwidth consumption for communication between the NIC and the processor. The management processor may be the main processor of the NIC or a separate processor attached to the NIC via a side port or coupled to the NIC via a fabric. Instead of a general purpose processor, fill requests (e.g., packet headers) may also be sent to a dedicated controller circuit with a relatively large (e.g., more than a gigabyte) memory.

[0034] The processor (or dedicated controller circuit) may perform a matching action lookup in a larger NIC-external matching action table (e.g., a hash table) stored in its memory based on the received fill request, and generate a fill response and return it to the NIC. The fill response may include the matching action lookup result from the larger matching action table. Note that if the larger matching action table does not have a matching entry, the packet header may be forwarded to an exception handler, which processes the packet based on a set of predefined rules. For example, the packet may be dropped.

[0035] The packet engine receives a fill response from the processor, which may include a NIC-external match action lookup result (operation 422). According to some aspects, a receiving logic in the packet engine may receive the fill response in a manner similar to receiving an incoming packet. More specifically, the receiving logic may include a multiplexer that multiplexes the incoming packet and the fill response. The packet engine may also store the fill response in the instruction RAM and in a match action table on the NIC as a new match action entry (operation 424). Storing such an NIC-external match action lookup result in a match action table on the NIC may facilitate efficient processing of subsequently received packets with similar headers.

[0036] At the same time, the same data packet is advanced in the queue specific to the packet class. When the data packet reaches the head of the queue specific to the packet class, the service engine determines whether a corresponding fill response has been received for the data packet (operation 426). According to some aspects, the data packet engine can search the instruction memory for a matching action entry stored for the data packet. If no such entry is found, a fill response has not been received and the data packet is waiting in the queue specific to the packet class. If a fill response has been received, the data packet can wait for arbitration (operation 428). If the data packet is selected by the arbitrator, the service engine can perform an appropriate action on the data packet and publish the data packet (operation 412). More specifically, the data packet engine can perform an action based on the NIC external search results stored in the instruction memory.

[0037] By offloading some match-action lookups to the processor, the disclosed solution enables nearly arbitrarily large match-action tables to be implemented, thereby enabling support for a large number of tenants sharing a NIC. In addition, a management process running in the processor can provide table management to support dynamic offloading of functions such as managing stateful packet processing rules, which can enhance node security. Figure 5 An example of a computer system that facilitates off-NIC match action lookup according to one aspect is illustrated. Computer system 500 includes a processor 502, a memory 504, and a storage device 506. In addition, computer system 500 can be coupled to peripheral input / output (I / O) user devices 510, such as a display device 512, a keyboard 514, and a pointing device 516. Storage device 506 can store an operating system 518, a management system 520, and data 540.

[0038] Management system 520 may include instructions that, when executed by computer system 500, may cause computer system 500 or processor 502 to perform methods and / or processes described in the present disclosure. Specifically, by executing these instructions, computer system 500 may implement various functions for performing NIC off-match action lookup operations. Management system 520 may include instructions for receiving fill requests (fill request receiving instructions 522), instructions for parsing packets to extract header information (packet parsing instructions 524), instructions for looking up a NIC off-match action table based on the extracted header information (table lookup instructions 526), ​​and instructions for transmitting a fill response to a packet engine on the NIC (fill response transmission instructions 528). Data 540 may include a match action table 542.

[0039] In general, the present disclosure can provide a solution to offload a portion of the matching action lookup operation from a NIC with limited memory resources to a processor with a large memory, thereby allowing the NIC to support a large number of tenants. When the result of the matching action lookup performed on a data packet in the matching action table on the NIC is missing, the data packet can be queued in two separate queues, a packet class-specific queue for queuing all data packets belonging to the same packet class (e.g., the same service class), and a fill request or missing queue for queuing data packets that do not have a matching entry in the matching action table on the NIC. During arbitration, the data packet in the fill request queue can have a higher priority, which means that the data packet can be selected for release before the data packets in other packet class-specific queues. The data packet selected from the fill request queue will be released to the management process in the processor as a fill request. The fill request can include a data packet header, thereby facilitating the management process to perform a lookup operation in a larger NIC external matching action table stored in the RAM of the processor. The NIC external matching action lookup result can be returned to the service engine on the NIC, following the same path of the incoming service. The NIC external matching action lookup result can be used to process unmatched data packets, which are waiting in the packet class-specific queue. Additionally, the off-NIC match action lookup results can be used to "fill" the packet's entry in the match action table on the NIC to facilitate future processing of similar packets arriving at the NIC. Queuing unmatched packets in two queues allows the unmatched packets to maintain their place in the flow of the corresponding class while waiting for the off-NIC match action lookup results to be returned from the processor. Queuing all packets based on the packet's class ensures independent flows for each class. The absence of a match action lookup on the NIC in one class does not block flows in other classes. Figure 1 In the example shown in Figure 3, packets entering the NIC are outbound packets, meaning that they come from the processor and are to be issued to the fabric. Similar traffic engines can also be used to process inbound packets, meaning that they come from the fabric and are headed toward the main processor of the NIC. The NIC can include two hardware-based traffic engines, one for inbound traffic and one for outbound traffic.

[0040] In one aspect, a method for processing a data packet may be provided. The method may include: performing a lookup operation in a first matching action table located on a network interface card (NIC) based on a header of a received data packet; in response to finding a matching entry indicating an action to be performed on the received data packet, queuing the received data packet in a first queue and storing data associated with the action in an instruction memory on the NIC; and in response to not finding a matching entry in the first matching action table, queuing the received data packet in both the first queue and a second queue. The method may also include: selecting a first data packet from the first queue for processing, the processing including performing a corresponding action indicated by the data stored in the instruction memory; selecting a second data packet from the second queue for processing, the processing including forwarding at least a portion of the second data packet to a processor coupled to the NIC to allow the processor to perform a second lookup operation in the second matching action table; and receiving a lookup result of the second lookup operation from the processor indicating an action to be performed on the second data packet, thereby allowing a third data packet in the first queue corresponding to the second data packet to be processed based on the lookup result.

[0041] In variations on this aspect, the first match action table may comprise a hash table stored in a memory located on the NIC.

[0042] In variations of this aspect, the second match-action table is larger than the first match-action table, and the second match-action table may be stored in a memory associated with the processor.

[0043] In a variation of this aspect, the second queue has a higher priority than the first queue. Forwarding a portion of the second data packet may include truncating the second data packet to obtain a predetermined number of initial bytes of the second data packet (which may include a header of the second data packet), and forwarding the predetermined number of initial bytes of the second data packet to a management process executed by the processor.

[0044] In a variation of this aspect, the method may further include storing the received search result of the second search operation in the first matching action table.

[0045] In a further variation, the method may further include removing one or more old entries in the first matching action table to provide space for storing the received search results.

[0046] In variations of this aspect, the first queue is a packet class specific queue, and the method may further comprise enqueuing received packets of different packet classes into different packet class specific queues.

[0047] In a variation on this aspect, the method may further include storing a search result of the second search operation in the instruction memory.

[0048] In a further variation, the method may also include: determining whether a lookup result is stored in the instruction memory in response to selecting a third data packet in the first queue for processing; and performing an action indicated by the lookup result on the third data packet in response to determining that the lookup result is stored in the instruction memory.

[0049] In a further variation, the method may also include maintaining the third data packet in the first queue in response to determining that the lookup result is not stored in the instruction memory.

[0050] In one aspect, a packet engine for processing packets located on a network interface card (NIC) may be provided. The packet engine may include: a first matching action table, the first matching action table including a plurality of matching action entries; a lookup logic for performing a lookup operation in the first matching action table based on a header of a received packet; an instruction memory for storing a result of the lookup operation indicating an action to be performed on the received packet; a queuing logic for queuing the received packet in a first queue in response to the lookup logic finding a matching action entry corresponding to the header of the received packet and queuing the received packet in a first queue and a second queue in response to the lookup logic not finding a matching action entry; and an arbiter for selecting a packet for processing from a plurality of queues including the first queue and the second queue. The packet engine may include a packet processing logic for performing a corresponding action on a first packet selected from the first queue based on data stored in the instruction memory, and forwarding at least a portion of a second packet selected from the second queue to a processor coupled to the NIC to allow the processor to perform a second lookup operation in the second matching action table. The packet engine may further include receiving logic for receiving a search result in a second search operation from a processor indicating an action to be performed on the second packet, thereby allowing a third packet in the first queue corresponding to the second packet to be processed based on the search result.

[0051] The methods and processes described in the detailed description may be embodied as code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on a computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and codes and stored in the computer-readable storage medium.

[0052] In addition, the above methods and processes may be included in hardware devices. Hardware devices may include, but are not limited to, application specific integrated circuit (ASIC) chips, field programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software components or codes at specific times, and other programmable logic devices now known or later developed. When the hardware devices are activated, these hardware devices execute the methods and processes included therein.

[0053] The foregoing description of the embodiments has been presented for the purpose of illustration and description only. The description is not intended to be exhaustive or to limit the scope of the present disclosure to the disclosed forms. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art.

Claims

1. A method for processing a data packet, the method comprising: performing a lookup operation in a first matching action table located on a network interface card (NIC) based on a header of the received data packet; In response to finding a matching entry indicating an action to be performed on the received data packet, enqueue the received data packet in a first queue and store data associated with the action in an instruction memory on the NIC; In response to not finding a matching entry in the first matching action table, queuing the received data packet in both the first queue and the second queue; selecting a first data packet from the first queue for processing, wherein the processing includes performing a corresponding action indicated by data stored in the instruction memory; selecting a second data packet from the second queue for processing, wherein the processing includes forwarding at least a portion of the second data packet to a processor coupled to the NIC to allow the processor to perform a second lookup operation in a second match action table; and A lookup result in the second lookup operation indicating an action to be performed on the second data packet is received from the processor, thereby allowing a third data packet in the first queue corresponding to the second data packet to be processed based on the lookup result.

2. The method of claim 1, wherein: The first match action table comprises a hash table stored in a memory located on the NIC.

3. The method of claim 1, wherein: The second match-action table is larger than the first match-action table, and wherein the second match-action table is stored in a memory associated with the processor.

4. The method of claim 1, wherein: The second queue has a higher priority than the first queue, and wherein forwarding at least a portion of the second packet comprises: truncating the second data packet to obtain a predetermined number of initial bytes of the second data packet, wherein the predetermined number of initial bytes includes a header of the second data packet; and The predetermined number of initial bytes of the second data packet are forwarded to a management process executed by the processor. 5 . The method of claim 1 , further comprising storing the received search result of the second search operation in the first matching action table. 6 . The method of claim 5 , further comprising removing one or more old entries in the first matching action table to provide space for storing received search results.

7. The method of claim 1, wherein: The first queue is a packet class specific queue, and wherein the method further comprises queuing received packets of different classes into different packet class specific queues.

8. The method of claim 1, further comprising storing the lookup result of the second lookup operation in the instruction memory.

9. The method of claim 8, further comprising: In response to selecting the third data packet in the first queue for processing, determining whether the search result is stored in the instruction memory; as well as In response to determining that the lookup result is stored in the instruction memory, an action indicated by the lookup result is performed on the third data packet.

10. The method of claim 9, further comprising: In response to determining that the lookup result is not stored in the instruction memory, the third data packet is maintained in the first queue.

11. A packet engine for processing packets on a network interface card (NIC), the packet engine comprising: a first matching action table, the first matching action table comprising a plurality of matching action entries; A search logic configured to perform a search operation in the first matching action table based on a header of a received data packet; an instruction memory for storing a result of the lookup operation indicating an action to be performed on the received data packet; queuing logic to queue the received data packet in a first queue in response to the lookup logic finding a matching action entry corresponding to a header of the received data packet, and to queue the received data packet in both the first queue and a second queue in response to the lookup logic not finding the matching action entry; an arbitrator, the arbitrator being configured to select a data packet for processing from a plurality of queues including the first queue and the second queue; Data packet processing logic, the data packet processing logic is used to: performing a corresponding action on a first data packet selected from the first queue based on the data stored in the instruction memory; as well as forwarding at least a portion of a second data packet selected from the second queue to a processor coupled to the NIC to allow the processor to perform a second lookup operation in a second match action table; as well as and receiving logic for receiving from the processor a lookup result in the second lookup operation indicating an action to be performed on the second data packet, thereby allowing a third data packet in the first queue corresponding to the second data packet to be processed based on the lookup result.

12. The packet engine of claim 11, wherein: The first matching action table includes a hash table.

13. The packet engine of claim 11, wherein: The second match-action table is larger than the first match-action table, and wherein the second match-action table is stored in a memory associated with the processor.

14. The packet engine of claim 11, wherein: The second queue has a higher priority than the first queue, and wherein, in forwarding the at least a portion of the second packet, the packet processing logic is to: truncating the second data packet to obtain a predetermined number of initial bytes of the second data packet, wherein the predetermined number of initial bytes includes a header of the second data packet; and The predetermined number of initial bytes of the second data packet are forwarded to a management process executed by the processor.

15. The data packet engine of claim 11, further comprising a table update logic configured to store the received search result of the second search operation in the first matching action table.

16. The packet engine of claim 15, wherein: The table update logic is used to remove one or more old entries in the first matching action table to provide space for storing received search results.

17. The packet engine of claim 11, wherein: The first queue is a packet class specific queue, and wherein the queuing logic is used to queue received packets of different classes into different packet class specific queues.

18. The packet engine of claim 11, wherein: The receiving logic is used to store the search result of the second search operation in the instruction memory.

19. The packet engine of claim 18, wherein: The data packet processing logic is used to: In response to selecting the third data packet in the first queue for processing, determining whether the search result is stored in the instruction memory; and In response to determining that the lookup result is stored in the instruction memory, an action indicated by the lookup result is performed on the third data packet.

20. The packet engine of claim 19, wherein: The data packet processing logic is used to: In response to determining that the lookup result is not stored in the instruction memory, the third data packet is maintained in the first queue.

Images