Dpu-based intelligent network card network task offloading system

Abstract

The application discloses a DPU-based intelligent network card network task offloading system and belongs to the technical field of intelligent network card packet processing. The system comprises a network data access module, a task description generation module, an offloading decision module, a hardware execution module and a feedback module. The network data access module receives a packet and writes the packet into a receiving queue after adding an arrival timestamp. The task description generation module analyzes a packet header to generate a task descriptor containing a flow identifier, a protocol type and a load length. The offloading decision module reads the task descriptor in an arrival order, queries a flow table according to the flow identifier, extracts a stored offloading path identifier if a hit is obtained, and allocates an offloading path identifier and stores the offloading path identifier if a miss is obtained. The hardware execution module distributes the packet to a hardware engine specified by a DPU according to the offloading path identifier. The feedback module reads a completion queue depth and an occupancy count to maintain the effective state of a flow table entry. The system can reduce flow identifier calculation resource consumption and improve the accuracy of flow table entry effective state maintenance.

Description

Technical Field

[0001] This invention relates to the field of smart network interface card (NIC) packet processing technology, specifically to a smart NIC network task offloading system based on a Data Processing Unit (DPU). Background Technology

[0002] In the network packet offloading process, DPU-based smart NICs need to receive and parse arriving packets, generating task description information including flow identifier, protocol type, and payload length. Based on the flow identifier in the task description information, the flow table is queried to determine the offloading path, and the hardware execution module moves the packet payload to the designated hardware engine. After the network data access module adds the arrival timestamp to the packet and writes it to the receive queue, the task description generation module extracts the 5-tuple, protocol type, and payload length from the packet header. The offloading decision module selects a stored offloading path or allocates a new offloading path based on the flow table hit status. The feedback module samples the completion status of the hardware engine and updates the flow table entries. In existing technologies, flow identifier generation often uses hash lookup tables, cyclic redundancy check, or multiplicative hashing. The calculation process requires lookup table access, multiple rounds of shifting and XOR, multiplication, or long division circuits, consuming significant logic resources and increasing packet processing latency in high-speed data planes. Alignment processing between payload length and storage page size is generally recalculated by hardware before DMA migration. The task descriptor does not carry alignment information, resulting in additional boundary checks between descriptor ring writing and DMA startup. For hardware engine completion status feedback, existing methods mostly rely on completion interrupts or fixed-period timed sampling. The completion queue depth and the occupancy count of each engine are not fully used for maintaining the effective status of flow table entries. The update of flow table entry activity is delayed, which may result in idle flow table entries occupying flow table space for a long time or active flow table entries being prematurely reclaimed.

[0003] In terms of stream identifier generation, it is necessary to reduce the latency and logical resource overhead caused by complex hash operations, and avoid invalid stream identifiers caused by zero folding results; in terms of stream table state maintenance, it is necessary to use the dynamic changes of completion queue depth and occupancy count to adjust the valid state and priority of stream table entries, so that the resource occupancy assessment of the hardware engine is consistent with the actual completion status. Summary of the Invention

[0004] To address the issues of high resource consumption in flow identifier generation, lack of load alignment information transmission with task descriptors, and delayed maintenance of flow table entry validity status in existing network task offloading systems, a DPU-based intelligent network interface card (NIC) network task offloading system is proposed. This system generates flow identifiers by bitwise XOR folding, carries load length alignment markers in the task descriptors, and dynamically maintains the validity status of flow table entries using completion queue depth and occupancy count, thereby reducing hardware resource consumption and the probability of erroneous flow table reclamation.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a DPU-based intelligent network interface card (NIC) network task offloading system, including a network data access module, a task description generation module, an offloading decision module, a hardware execution module, and a feedback module. The network data access module receives data packets, adds an arrival timestamp to the data packets, and writes them into a receiving queue. The task description generation module parses the data packet header, generates a task descriptor containing a flow identifier, protocol type, and payload length, and writes the task descriptor into a task description queue. The offloading decision module reads task descriptors from the task description queue in arrival order, queries the flow table based on the flow identifier, and if the flow table matches, extracts the stored offloading path identifier; if the flow table does not match, it allocates an offloading path identifier based on the protocol type and payload length and stores it in the flow table. The hardware execution module distributes the data packets corresponding to the task descriptors to the hardware engines specified by the DPU according to the offloading path identifiers. The feedback module reads the completion queue depth and occupancy count of the hardware engines and maintains the valid status of each flow table entry. This system enables data packets to quickly determine the hardware engine and complete distribution after entering the receive queue by generating task descriptions and offload path identifiers and maintaining flow tables. This reduces the CPU's involvement in data movement and protocol processing, and improves the network task offloading efficiency of smart NICs on the DPU.

[0006] As one technical solution of this invention, the task description generation module extracts the 5-tuple, protocol type, and payload length from the packet header, concatenates the 5-tuple into a byte string, performs a logical XOR operation on the byte string to obtain the stream identifier, aligns the payload length with the storage page size to obtain an alignment mark, and writes the protocol type, stream identifier, payload length, and alignment mark into the task descriptor according to a preset field order, and then writes the task descriptor into the task description queue. This method makes the task descriptor fields uniform and the length controllable, facilitating hardware parsing in a fixed format and reducing branch judgments during the parsing process.

[0007] Preferably, when the unloading decision module allocates an unloading path identifier, it obtains the idle engine number pool maintained by the DPU, retrieves an engine number from the pool and marks it as occupied; it writes the flow identifier in the task descriptor into the hash table entry corresponding to the engine number, and uses the storage address of the hash table entry as the flow table index; it assembles the flow table index and the engine number into an unloading path identifier, and writes the unloading path identifier into the flow table. Therefore, once the flow table is matched, the engine number and flow table index can be obtained directly, shortening the unloading path lookup time.

[0008] Preferably, the hardware execution module parses the engine number from the unloading path identifier, reads the DPU's remapping register, and obtains the base address of the corresponding descriptor ring based on the engine number. It then writes the payload address, payload length, and stream identifier from the data packet record corresponding to the task descriptor into the next slot of the descriptor ring. The write pointer of the descriptor ring is incremented, and the DMA controller is triggered to move the payload data according to the content of the slot pointed to by the write pointer. This process directly completes the payload data movement through the descriptor ring and the DMA controller, reducing processor intervention in the data path and improving the transmission efficiency of data packets to the hardware engine.

[0009] Preferably, the feedback module retrieves completion descriptors from the completion queue in each polling cycle, locates the flow table entry based on the flow identifier in the completion descriptor, and increments the activity field of the flow table entry by the packet count value in the completion descriptor. It then performs a weighted combination of the current sampled value of the completion queue depth and the resource usage field value from the previous cycle, using the combined result as the new resource usage field value. When the resource usage field value of a flow table entry changes, the position of the flow table entry in the priority chain is adjusted to correspond to the new resource usage field value. This feedback mechanism allows the effective state and priority of flow table entries to be dynamically adjusted according to the actual completion status of the hardware engine, improving the rationality of resource scheduling.

[0010] Preferably, the task description generation module converts the source IP address, destination IP address, source port number, destination port number, and protocol number in the 5-tuple into values ​​with a fixed byte order, and concatenates them sequentially into a byte string. The byte string is then divided into multiple four-byte blocks, and each four-byte block is XORed bitwise sequentially to obtain a folded value. If the folded value is zero, it is set to a preset non-zero constant and used as the stream identifier. This method obtains a uniformly distributed stream identifier with less computational overhead, reducing the impact of hash collisions on offloading decisions.

[0011] Preferably, when the unloading decision module retrieves an engine number from the idle engine number pool, it reads the read pointer and write pointer of the idle engine number pool. If the read pointer and write pointer are equal, an engine number recycling operation is performed, scanning the status register of each hardware engine and writing the engine numbers with the idle field set in the status register into the idle engine number pool. If the read pointer and write pointer are not equal, the engine number is retrieved from the circular buffer slot pointed to by the read pointer, and the read pointer is moved one slot to the right. This method automatically triggers recycling when there are insufficient idle engine numbers, avoiding the inability to allocate unloading paths for new flows due to the exhaustion of the number pool.

[0012] Preferably, the engine number recycling operation selects an engine number to be scanned from the hardware engine number list and reads the status register corresponding to the engine number to be scanned. It checks the busy / idle flag in the status register; if the busy / idle flag indicates idle, it further reads the interrupt pending register corresponding to the engine number to be scanned. If there are no unprocessed interrupts in the interrupt pending register, the engine number to be scanned is written to the end of the idle engine number pool. The scan cursor of the engine number to be scanned in the hardware engine number list is shifted one position to the right, and the same operation is performed on the next engine number to be scanned. This recycling method can avoid engines with pending interrupts, improving recycling safety and engine reuse stability.

[0013] Preferably, when the hardware execution module writes the payload address, payload length, and flow identifier from the data packet record corresponding to the task descriptor into the next slot of the descriptor ring, it reads the write pointer and ring capacity value of the descriptor ring, and uses the sequential position corresponding to the write pointer within the ring capacity value range as the slot offset; it obtains the payload address from the data packet record, adjusts the payload address to the starting address required by the DMA controller alignment boundary; combines the starting address, payload length, and flow identifier into descriptor data, and writes the descriptor data into the slot corresponding to the slot offset. This method ensures that the starting address of the DMA transfer meets the alignment requirements, reducing the performance loss caused by unaligned access.

[0014] Preferably, when the feedback module performs a weighted combination of the current sampled value of the completed queue depth and the resource occupancy field value of the previous period, it writes the current sampled value to the tail of the sliding window array in the feedback module and removes the oldest sampled value from the head of the sliding window array. The sampled values ​​in the sliding window array are then arranged in ascending order, and the sampled value in the middle position after arrangement is taken as the new resource occupancy field value. The corresponding bit is marked in the priority bitmap of the flow table according to the new resource occupancy field value. This method smooths the resource occupancy estimate through sliding window and median calculations, avoiding frequent adjustments to the flow table priority caused by instantaneous sampling fluctuations, and improving feedback stability.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: The task description generation module converts the source IP address, destination IP address, source port number, destination port number, and protocol number in the quintuple into fixed byte order values, and concatenates them into a byte string in the order of source IP address, destination IP address, source port number, destination port number, and protocol number. The byte string is then divided into multiple four-byte blocks, and each four-byte block is XORed sequentially to obtain a folded value. When the folded value is zero, it is set to a preset non-zero constant, and this folded value serves as the flow identifier. By replacing hash lookup tables or cyclic redundancy checks with bitwise XOR folding, flow identifier calculation eliminates the need for multipliers, dividers, and large-scale lookup tables, shortening the flow identifier generation latency in high-speed data packet processing and reducing data plane logical resource consumption. Fixed byte order concatenation and four-byte block folding allow different bit fields in the quintuple to participate in mixing, reducing the concentration of flow identifiers when relying solely on the source or destination port. Setting the folded value to a non-zero constant when it is zero avoids confusion between zero-value flow identifiers and empty entries in the flow table. The task description generation module aligns the payload length with the storage page size to obtain an alignment mark, which is then written to the task descriptor. This allows the hardware execution module to directly use the alignment mark before writing to the descriptor ring slot and triggering DMA migration, reducing redundant payload address alignment calculations and out-of-bounds checks. The feedback module retrieves a completion descriptor from the completion queue in each polling cycle, locates the flow table entry based on the flow identifier in the completion descriptor, increments the activity field of the flow table entry by the packet count value in the completion descriptor, writes the current sample value of the completion queue depth to the tail of the sliding window array and removes the oldest sample value, arranges the sample values ​​in the sliding window array in ascending order, and takes the middle sample value as the new resource occupancy field value. The module then marks the corresponding bit in the priority bitmap of the flow table based on the new resource occupancy field value. When the resource occupancy field value of a flow table entry changes, the position of the flow table entry in the priority chain is adjusted to correspond to the resource occupancy field value. The completion queue depth and occupancy count directly participate in the update of the effective status of flow table entries, enabling the activity level and resource occupancy assessment of flow table entries to keep up with the actual completion status of the hardware engine in a timely manner, avoiding the retention of idle flow table entries for a long time or the premature reclamation of active flow table entries; the sliding window method of taking the median smooths out the instantaneous fluctuations in the completion queue depth, reducing the interference of occasional completion events on resource occupancy judgment; the position of the flow table entry in the priority chain is adjusted according to the value of the resource occupancy field, so that flow table entries with high resource occupancy or long-term inactivity are put into the priority maintenance scope, improving the efficiency of flow table resource reclamation and reuse. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a schematic diagram of a DPU-based intelligent network interface card network task offloading system. Figure 2 This is a flowchart of the task description generation module. Figure 3 These are curves showing the change in flow table hit rate for different protocol types as a function of the number of data packets processed. Figure 4 It is a curve showing the change in the number of hardware execution module descriptor ring slots occupied over time; Figure 5 It is a graph showing the changes in queue depth sampling and resource occupancy levels; Figure 6 It is a curve showing the change in the capacity of the idle engine number pool during the hardware engine number recycling operation. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] See Figure 1 The present invention provides a network task offloading system for intelligent network interface cards based on DPU, including a network data access module, a task description generation module, an offloading decision module, a hardware execution module and a feedback module.

[0020] The network data access module receives data packets, adds an arrival timestamp to each packet, and writes it to the receive queue. The network data access module is implemented by the network interface card (NIC) front-end receiving unit. Each data packet in the receive queue corresponds to a buffer descriptor, and the arrival timestamp is recorded in a specified field of the buffer descriptor.

[0021] The task description generation module parses the packet header, generates a task descriptor containing the stream identifier, protocol type, and payload length, and writes the task descriptor to the task description queue. The task description queue is organized in a first-in-first-out manner. Each time the task description generation module generates a task descriptor, it writes the task descriptor to the tail of the queue and updates the write pointer.

[0022] The offloading decision module reads task descriptors from the task description queue in arrival order and queries the flow table based on the flow identifier. The flow table organizes flow table entries based on the flow identifier. If the flow table matches, the stored offloading path identifier is retrieved; if the flow table does not match, an offloading path identifier is allocated based on the protocol type and payload length and stored in the flow table. The offloading path identifier is used to indicate the hardware engine to which the data packet should be distributed and the location of the flow table entry.

[0023] The hardware execution module distributes the data packet corresponding to the task descriptor to the hardware engine specified by the DPU based on the offload path identifier. The hardware execution module parses the engine number in the offload path identifier and the payload address, payload length, and stream identifier in the data packet record, writes the above information into the descriptor ring of the corresponding hardware engine, and triggers the DMA controller to move the payload data.

[0024] The feedback module reads the completion queue depth and occupancy count of the hardware engine and maintains the valid status of each flow table entry. The completion queue depth is the number of completion descriptors that have been written to but not read in the completion queue, and the occupancy count is the amount of hardware engine processing resources occupied by the flow corresponding to the flow table entry. The feedback module updates the valid status of the flow table entries based on the completion queue depth and occupancy count.

[0025] In specific implementation, please refer to Figure 2 The task description generation module retrieves data packets from the receive queue, parses the packet header, and extracts the 5-tuple, protocol type, and payload length. The 5-tuple consists of the source IP address, destination IP address, source port number, destination port number, and protocol number. The task description generation module reads the source and destination IP addresses from the network layer header and the source, destination, and protocol numbers from the transport layer header. Finally, the task description generation module subtracts the network layer header length and transport layer header length from the total packet length field to obtain the payload length. The payload length is in bytes.

[0026] In practice, the task description generation module converts the source IP address, destination IP address, source port number, destination port number, and protocol number in the 5-tuple into fixed byte order values. The fixed byte order is network byte order. The task description generation module determines the byte length of the source and destination IP addresses based on the IP version of the data packet: 4 bytes for IPv4 addresses and 16 bytes for IPv6 addresses. It then converts the source and destination IP addresses into network byte order values ​​of the corresponding byte lengths. The source port number is converted to a 2-byte network byte order value, the destination port number is converted to a 2-byte network byte order value, and the protocol number is converted to a 1-byte network byte order value.

[0027] The source IP address, destination IP address, source port number, destination port number, and protocol number are concatenated into a byte string in the following order: source IP address, destination IP address, source port number, destination port number, protocol number. For an IPv4 packet, the source IP address is 4 bytes, the destination IP address is 4 bytes, the source port number is 2 bytes, the destination port number is 2 bytes, and the protocol number is 1 byte, resulting in a 13-byte string. This string is then divided into multiple four-byte blocks. Since the string is 13 bytes long, it is divided into three complete four-byte blocks and one block with less than four bytes, containing 1 byte. Three zero bytes are padded to the block with less than four bytes to form a fourth four-byte block. The first four-byte block is XORed with the second four-byte block to obtain the first XOR result; the first XOR result is XORed with the third four-byte block to obtain the second XOR result; the second XOR result is XORed with the fourth four-byte block to obtain the folded value. The folded value is 32 bits wide. If the fold value is zero, it is set to a preset non-zero constant. The preset non-zero constant is set to 1. The reason for setting the preset non-zero constant to 1 is that a zero value in the flow identifier field is used to represent an empty entry in the flow table, while a non-zero value is used to identify a valid flow entry. 1 is the smallest non-zero positive integer, which will not conflict with other non-zero values ​​generated by calculating the fold value, and it also keeps the flow identifier bit width unchanged. The fold value is used as the flow identifier. If the fold value is non-zero, it is directly used as the flow identifier.

[0028] In the specific implementation, the task description generation module aligns the payload length with the storage page size to obtain an alignment mark. The storage page size is configured by the DPU memory management unit; in this embodiment, the storage page size is set to 4096 bytes. The reason for setting the storage page size to 4096 bytes is that the DPU memory management unit uses 4KB pages as the smallest physical address mapping granularity, and aligning the payload length to the 4KB boundary is compatible with the physical page offset in the DMA descriptor. The alignment process divides the payload length by the storage page size. If there is a remainder after dividing the payload length by the storage page size, the payload length is rounded up to an integer multiple of the storage page size, and the alignment padding status is recorded as valid; if there is no remainder after dividing the payload length by the storage page size, the payload length is maintained, and the alignment padding status is recorded as invalid. The alignment mark is a 1-bit field; the alignment mark is 1 when the alignment padding status is valid, and 0 when the alignment padding status is invalid.

[0029] In implementation, the task description generation module writes the protocol type, stream identifier, payload length, and alignment flag into the task descriptor according to a preset field order. The preset field order, from most significant bit to least significant bit, is: protocol type, stream identifier, payload length, and alignment flag. The total bit width of the task descriptor is set to 64 bits, with the protocol type field occupying 8 bits, the stream identifier field occupying 32 bits, the payload length field occupying 16 bits, the alignment flag field occupying 1 bit, and the remaining 7 bits reserved for zero values. The reason for setting the total bit width to 64 bits is that a single slot in the task description queue consists of a 64-bit storage unit, and a single write operation of the task descriptor can cover one slot without needing to concatenate across slots. The task description generation module sets the protocol type field of the task descriptor to the extracted protocol type value, the stream identifier field to the calculated stream identifier, the payload length field to the aligned payload length value, and the alignment flag field to the alignment flag value.

[0030] In practice, the task description generation module writes task descriptors to the task description queue. The task description queue is located in the shared memory area of ​​the DPU and is organized using a circular buffer. The task description generation module reads the write pointer of the task description queue and writes the task descriptor to the slot pointed to by the write pointer. The width of each slot in the task description queue is the same as the total bit width of the task descriptors. After writing is complete, the task description generation module moves the write pointer one slot to the right. If the write pointer reaches the tail of the task description queue, it wraps back to the head of the queue. Simultaneously, the task description generation module updates the written descriptor count in the task description queue, which indicates the number of task descriptors waiting to be read in the queue.

[0031] In implementation, the offloading decision module reads task descriptors from the task description queue in arrival order, retrieving the protocol type field, flow identifier field, and payload length field from the task descriptor. The offloading decision module uses the flow identifier field to query the flow table, which consists of flow table entries. Each flow table entry contains a flow identifier field, an offloading path identifier field, and a valid status field. The offloading decision module matches the flow identifier in the task descriptor with the flow identifier fields of each flow table entry. If a valid status field indicates a valid flow table entry and its flow identifier field matches the flow identifier in the task descriptor, the flow table is considered a match, and the offloading path identifier field is extracted from the matched flow table entry as the stored offloading path identifier. If no valid flow table entry matches the flow identifier in the task descriptor, the flow table is considered a miss.

[0032] When a flow table match is not found, the offloading decision module reads the protocol type and payload length fields from the task descriptor and determines the offloading path identifier that needs to be allocated based on the protocol type and payload length. The offloading decision module obtains the idle engine number pool maintained by the DPU and retrieves an engine number from it. The idle engine number pool is a circular buffer. The DPU maintains read and write pointers for the idle engine number pool. The offloading decision module retrieves the engine number from the slot pointed to by the read pointer in the circular buffer and moves the read pointer one slot forward. If the read pointer reaches the end of the circular buffer, it wraps around to the beginning of the circular buffer.

[0033] In practice, the unloading decision module marks the engine number as occupied. The module then searches the engine occupancy bitmap maintained by the DPU for the occupancy bit corresponding to the engine number and sets the occupancy bit to the occupied state. The engine occupancy bitmap records the occupancy status of each hardware engine. Each occupancy bit in the bitmap corresponds to a hardware engine; an occupied state indicates that the corresponding hardware engine has been allocated, while an idle state indicates that the corresponding hardware engine has not been allocated.

[0034] In practice, the unloading decision module writes the flow identifier from the task descriptor into the hash table entry corresponding to the engine number. The hash table entries are organized as follows: the DPU maintains a set of hash table entries, with each hash table entry address corresponding to an engine number. The unloading decision module calculates the hash table entry address based on the engine number, writes the flow identifier from the task descriptor into the flow identifier field of the corresponding hash table entry, and uses the hash table entry's storage address as the flow table index. The storage address of the hash table entry is obtained by adding the base address and the engine number offset.

[0035] In practice, the unloading decision module assembles the flow table index and engine number into an unloading path identifier. The following formula is used during the assembly process:

[0036] in, This represents the uninstallation path identifier, with a bit width of 32 bits. This represents the flow table index, which is 24 bits wide and is obtained from the lower 24 bits of the storage address of the hash table entry. The width of the engine number field is 8 bits. This indicates the engine number, with a bit width of 8 bits; This indicates a bitwise left shift operation. This indicates a bitwise OR operation. The engine number field is set to 8 bits because the DPU supports a maximum of 256 hardware engines, and an 8-bit engine number can represent hardware engine numbers from 0 to 255. The flow table index is set to 24 bits because the flow table entry storage address uses a 24-bit address range, covering the 16 MB address space in the DPU shared memory used for flow table entry storage. The offload path identifier is set to 32 bits because the descriptor ring slot in the DPU hardware execution module contains a 32-bit offload path identifier field.

[0037] In practice, the unloading decision module writes the unloading path identifier into the flow table. Specifically, it allocates a new flow table entry, writes the flow identifier from the task descriptor into the flow identifier field of the new flow table entry, writes the assembled unloading path identifier into the unloading path identifier field of the new flow table entry, and sets the valid status field of the new flow table entry to valid.

[0038] See Figure 3 The horizontal axis of the graph represents the number of data packets processed, in thousands, ranging from 0 to 500 thousand; the vertical axis represents the flow table hit rate, in percentages, ranging from 0% to 100%. The four curves in the graph represent the TCP flow table hit rate, UDP flow table hit rate, ICMP flow table hit rate, and flow table miss rate, respectively. Different line styles and markers are used to distinguish the curves in the legend.

[0039] As the number of processed packets increases, the TCP flow table hit rate curve shows a significant upward trend, initially around 20%, rapidly increasing to over 40% after processing approximately 100,000 packets, and finally approaching 100% around 400,000 packets. The UDP flow table hit rate curve exhibits an overall S-shaped upward trend, starting at around 20%, reaching 50% at approximately 200,000 packets, and then gradually stabilizing close to 95%. The ICMP flow table hit rate curve starts relatively low, at around 10%, and begins to rise significantly after 300,000 packets, eventually approaching 90%.

[0040] The flow table miss rate curve shows a clear opposite trend to the three curves mentioned above. It starts at about 80%, and as the number of packets processed increases, the miss rate gradually decreases, dropping to about 50% when there are 200,000 packets, and finally falling below 5% when there are 500,000 packets.

[0041] In practice, the hardware execution module obtains the unload path identifier corresponding to the task descriptor and parses the engine number from it. The hardware execution module reads the engine number field from the unload path identifier, which is located in the lower 8 bits of the identifier. The hardware execution module then reads the DPU's remapping registers, a set of configuration registers within the DPU. Each register unit in the remapping register corresponds to an engine number, and the remapping register unit stores the descriptor ring base address. Based on the engine number, the hardware execution module reads the remapping register unit corresponding to that engine number and obtains the base address of the corresponding descriptor ring from it. The descriptor ring base address is the starting physical address of the descriptor ring in the DPU's shared memory.

[0042] In practice, the hardware execution module writes the payload address, payload length, and stream identifier from the data packet record corresponding to the task descriptor into the next slot of the descriptor ring. The hardware execution module reads the write pointer and ring capacity value of the descriptor ring. The descriptor ring is a circular buffer containing multiple descriptor slots, the number of which is represented by the ring capacity value. The write pointer of the descriptor ring indicates the position of the next writable descriptor in the descriptor ring.

[0043] In practical implementation, the hardware execution module uses the sequential position of the write pointer within the ring capacity range as the slot offset. The slot offset calculation formula is as follows:

[0044] in, This represents the slot offset, with a value range of [value missing]. ; This represents the write pointer to the descriptor ring, and its value is an integer greater than or equal to 0. This represents the ring capacity value, which is an integer greater than 0. This indicates a modulo operation. The ring capacity is set to 1024. The reason for setting the ring capacity to 1024 is that the descriptor ring in the DPU hardware execution module can accommodate 1024 descriptor slots. Each descriptor slot has a bit width of 64 bits, and the storage space required for 1024 descriptor slots is 8192 bytes. This storage space is consistent with the size of the descriptor ring region in the DPU shared memory.

[0045] In practice, the hardware execution module obtains the payload address from the data packet record and adjusts it to the starting address required by the DMA controller alignment boundary. The data packet record is maintained by the network data access module and contains the payload address, payload length, and flow identifier. The payload address is the starting physical address of the data packet payload in the DPU memory. The DMA controller alignment boundary requires the starting address to be aligned to the DMA controller alignment boundary. The DMA controller alignment boundary is set to 64 bytes. This is based on the fact that the burst transfer length of the DPU's internal DMA controller bus is 64 bytes, and aligning the starting address to the 64-byte boundary avoids the DMA controller splitting burst transfers. The hardware execution module adjusts the payload address to the starting address required by the DMA controller alignment boundary by aligning the payload address down to the 64-byte boundary. The adjusted starting address is the result of a bitwise AND operation between the payload address and the bit-inverted DMA controller alignment boundary mask. The DMA controller alignment boundary mask is the value obtained by subtracting 1 from 64, which is 63.

[0046] In practice, the hardware execution module combines the adjusted start address, payload length, and flow identifier into descriptor data. The descriptor data uses a format consistent with the descriptor ring slot width and includes a start address field, a payload length field, and a flow identifier field. The hardware execution module writes the descriptor data to the slot corresponding to the slot offset. Specifically, it uses the descriptor ring base address as a reference, and the product of the slot offset and the width of a single descriptor slot is used as the address offset. The descriptor data is then written to the physical address obtained by adding the address offset to the descriptor ring base address.

[0047] In practice, the hardware execution module increments the write pointer of the descriptor ring. The increment method is to add 1 to the write pointer value of the descriptor ring. If the incremented write pointer value reaches the ring capacity, the write pointer value is set to 0. The hardware execution module triggers the DMA controller to move the load data according to the slot content pointed to by the write pointer. Specifically, the hardware execution module triggers the DMA controller to read the most recently written slot content in the descriptor ring, parse the start address, load length, and stream identifier in the slot content, and the DMA controller moves the load data from the start address to the target address space corresponding to the hardware engine specified by the DPU, according to the load length.

[0048] See Figure 4 In the graph, the horizontal axis represents time in milliseconds, ranging from 0 to 1000 milliseconds; the vertical axis represents the number of slots occupied by the descriptor ring, ranging from 0 to 1000. The legend shows four curves, corresponding to the change in the number of slots occupied by the descriptor ring for engines 0, 1, 2, and 3 over time.

[0049] According to the technical solution of this embodiment, the hardware execution module obtains the corresponding descriptor ring base address from the remapping register by reading the engine number in the unloading path identifier, and writes the starting address, load length, and flow identifier of the data packet payload into the corresponding slot of the descriptor ring. Then, it increments the write pointer and triggers DMA to move the load data. The number of slots occupied by each engine descriptor ring in the figure reflects the number of descriptors to be moved currently being processed by that engine, indirectly reflecting the dynamic load situation scheduled by the hardware execution module.

[0050] Analyzing the curve trends in the graph, Engine 0 (solid blue line with dots) and Engine 1 (dashed orange line with squares) exhibit significant periodic fluctuations, with peak values ​​approaching 700 to 1000 slots and trough values ​​around 200 to 300 slots, indicating periodic peaks and troughs in their load processing capacity and substantial load fluctuations. Engine 2 (dotted green line with triangles) shows more irregular fluctuations in the number of slots occupied by its descriptor ring, with peak values ​​exceeding 1000 slots and troughs dropping to approximately 100 slots, indicating significant load volatility and sudden changes. Engine 3 (dotted red line with diamonds) maintains a relatively stable number of slots occupied by its descriptor ring, remaining within the range of approximately 600 to 800 slots, with the curve showing a slow, periodic trend, indicating a relatively balanced and stable load.

[0051] In practice, the feedback module is configured with a polling period of 10 milliseconds. This 10-millisecond polling period is based on the fact that the minimum interval for the DPU hardware engine to generate completion descriptors is 1 millisecond. A 10-millisecond polling period can cover the writing of multiple completion descriptors, and the time the feedback module's polling operation occupies DPU processing resources within each polling period does not exceed 0.1 milliseconds. At the beginning of each polling period, the feedback module retrieves completion descriptors from the completion queue. The completion queue is a circular buffer in the DPU's shared memory. After completing packet processing, the hardware engine writes completion descriptors to the completion queue and updates the write pointer of the completion queue. The feedback module maintains a read pointer for the completion queue, reads completion descriptors from the slot pointed to by the read pointer, and then moves the read pointer one slot forward. If the read pointer reaches the tail of the completion queue, it wraps back to the head of the queue. The completion descriptor contains a stream identifier field and a packet count field. The stream identifier field is the stream identifier generated by the task description generation module, and the packet count field is the number of packets processed by the hardware engine corresponding to the completion descriptor in this completion operation.

[0052] In practice, the feedback module locates flow table entries based on the flow identifier in the completion descriptor. The feedback module matches the flow identifier in the completion descriptor with the flow identifier fields of each valid flow table entry, finding entries whose flow identifier field matches the flow identifier in the completion descriptor. The feedback module reads the activity field from the matched flow table entry, adds the current value of the activity field to the packet count value in the completion descriptor, and writes the sum back to the activity field of the matched flow table entry. The activity field is a 32-bit unsigned integer, with an initial value of 0.

[0053] In practice, the feedback module reads the current sampled value of the completion queue depth. The current sampled value of the completion queue depth is the difference between the write pointer of the completion queue and the read pointer maintained by the feedback module; this difference represents the number of completion descriptors that have been written to but not yet read from the completion queue. The feedback module uses the current sampled value of the completion queue depth as its current sampled value.

[0054] In practice, the feedback module performs a weighted combination of the current sampled value of the completion queue depth and the resource usage field value from the previous period. The weighted combination is implemented as follows: the feedback module maintains a sliding window array to store the sampled values ​​of the completion queue depth for the most recent seven polling periods. The feedback module writes the current sampled value to the tail of the sliding window array and removes the oldest sampled value from the head of the array. The length of the sliding window array is set to 7. The sampled values ​​stored in the sliding window array include the completion queue depth sampled values ​​from the previous period and earlier periods. The resource usage field value from the previous period is calculated from the sampled values ​​in the sliding window array from the previous period; therefore, the sampled values ​​in the sliding window array contain historical information corresponding to the resource usage field value from the previous period.

[0055] In implementation, the feedback module arranges the sampled values ​​in the sliding window array in ascending order. The sampled value at the middle position after arrangement is taken as the new resource occupancy field value. The middle position is determined as follows: when the sliding window array length is 7, the middle position is the 4th sampled value. The formula for calculating the new resource occupancy field value is:

[0056] in, This indicates the new resource usage field value, which ranges from 0 to the maximum capacity of the completion queue; This indicates that the sampled values ​​in the sliding window array are arranged in ascending order. Each sample value, It is an integer, ranging from 1 to... ; This indicates the length of the sliding window array; in this embodiment, the length of the sliding window array is 7. Indicates rounding down. When hour, The value is equal to 4, therefore the new resource usage field value is equal to the 4th sample value after sorting.

[0057] In implementation, the feedback module marks the corresponding bit in the priority bitmap of the flow table based on the new resource occupancy field value. The priority bitmap is a field in the flow table, with each bit corresponding to a resource occupancy level. The new resource occupancy field value is mapped to the priority bitmap using a resource occupancy level mapping. The resource occupancy level mapping divides the new resource occupancy field value by the resource occupancy level step size, rounding down to the nearest integer as the priority bitmap index. The resource occupancy level step size is set to 16, based on the fact that the typical range of completion queue depth is 0 to 255, divided into 16 resource occupancy levels, each covering 16 completion descriptors. The feedback module sets the bit in the priority bitmap corresponding to the priority bitmap index to 1 and sets the other bits in the priority bitmap to 0.

[0058] In practice, the feedback module compares the new resource occupancy field value with the resource occupancy field value of the previous period. If the new resource occupancy field value is different from the previous period's value, it is determined that the resource occupancy field value of the flow table entry has changed. When the resource occupancy field value of a flow table entry changes, the feedback module adjusts the position of the flow table entry in the priority chain to the position corresponding to the resource occupancy field value. The priority chain is a doubly linked list in the flow table, sorted from high to low by resource occupancy field value, and each node in the priority chain is a flow table entry. The feedback module determines the insertion position of the flow table entry in the priority chain based on the new resource occupancy field value, removes the flow table entry from the priority chain, and inserts it between flow table entries with adjacent resource occupancy field values. If the new resource occupancy field value is larger, it is inserted into a higher position in the priority chain; if the new resource occupancy field value is smaller, it is inserted into a lower position in the priority chain.

[0059] In practice, the valid status of a flow table entry is jointly represented by the activity field, resource consumption field value, priority bitmap marker, and priority chain position. The feedback module updates the activity field, resource consumption field value, and priority bitmap marker in the flow table entry and adjusts the position of the flow table entry in the priority chain. The valid status of the flow table entry is used by the feedback module to determine whether the flow table entry is still active. If the activity field does not increase within multiple consecutive polling cycles, the feedback module sets the valid status of the flow table entry to invalid.

[0060] See Figure 5In the figure, the horizontal axis represents the polling cycle number of the feedback module, ranging from 0 to 1000, and the vertical axis represents the completion queue depth and resource occupancy level, with a scale range of 0 to 100. In the legend, the blue dashed line is marked as the "raw sampled value of the completion queue depth", the solid line is marked as the "resource occupancy field value after sliding window filtering", and the black stepped line is marked as the "resource occupancy level".

[0061] The curve of the original sampled value of the completion queue depth fluctuates significantly, exhibiting periodic ups and downs, with a value range of approximately 10 to 90. This curve reflects the actual sampled data of the completion queue depth collected by the feedback module at the beginning of each polling cycle, that is, the number of completion descriptors that have been written to but not read in the completion queue, reflecting the dynamic changes in the hardware engine load.

[0062] The resource usage field value curve after sliding window filtering is smoother than the original sampled values, eliminating large short-term fluctuations. This curve is obtained by the feedback module sorting the original sampled values ​​using a sliding window array of length 7 and taking the median, reflecting the steady-state level of system resource usage and avoiding drastic changes in the resource usage field value due to abnormal sampling.

[0063] The resource occupancy level step-like change curve is calculated based on the resource occupancy field value after sliding window filtering, using a resource occupancy level mapping rule. The change in step height corresponds to a discrete interval of resource occupancy level, with a level step size of 16, indicating that the completion queue depth is divided into 16 level intervals, with a maximum mapping depth of 255. The curve remains at several stable levels, intermittently transitioning with the gradual change of the resource occupancy field value, reflecting the dynamic adjustment of the feedback module's update of the flow table priority bitmap and the resource occupancy level.

[0064] In practical implementation, when the unloading decision module needs to obtain an engine number, it first reads the read pointer and write pointer of the idle engine number pool. The idle engine number pool is implemented using a circular buffer, which contains multiple slots, each storing one engine number. The read pointer of the idle engine number pool indicates the position of the next available engine number to be read, and the write pointer indicates the position of the next available engine number to be written. Both the read and write pointers are stored in the DPU's shared memory, and both have a bit width of 16 bits.

[0065] In practice, the unloading decision module compares the read pointer with the write pointer. If the read pointer and write pointer are equal, it is determined that there are no readable engine numbers in the idle engine number pool, and an engine number recycling operation is performed. The engine number recycling operation scans the status register of each hardware engine, reclaims the engine numbers corresponding to hardware engines with an idle status, and writes the reclaimed engine numbers into the idle engine number pool.

[0066] In practice, the engine number recycling operation selects a hardware engine number to be scanned from the hardware engine number list. The hardware engine number list is stored in the DPU's shared memory. Each entry in the list corresponds to a hardware engine, and the entries are arranged in ascending order of their hardware engine numbers. The engine number recycling operation maintains a scan cursor, which is the index of the current position to be scanned in the hardware engine number list. The initial value of the scan cursor is 0, and its bit width is 10 bits. The engine number recycling operation reads the entry pointed to by the scan cursor in the hardware engine number list and retrieves the engine number to be scanned from that entry.

[0067] In practice, the engine number recycling operation reads the status register corresponding to the engine number to be scanned. The status register is located in the control register group of the DPU hardware engine, and each hardware engine has one status register. The busy / idle flag in the status register indicates whether the hardware engine is currently in an idle state. A busy / idle flag of 1 indicates that the hardware engine is busy, and a busy / idle flag of 0 indicates that the hardware engine is idle.

[0068] In practice, the engine number recycling operation checks the busy / idle flag in the status register. If the busy / idle flag indicates idle, it further reads the interrupt pending register corresponding to the engine number to be scanned. The interrupt pending register is located in the interrupt control register group of the DPU hardware engine, and each hardware engine has one interrupt pending register. The interrupt pending flag in the interrupt pending register indicates whether there are any unprocessed interrupts in the hardware engine. An interrupt pending flag of 1 indicates the existence of unprocessed interrupts, and an interrupt pending flag of 0 indicates the absence of unprocessed interrupts.

[0069] In practice, if there are no unhandled interrupts in the interrupt pending register, the engine number reclamation operation writes the engine number to be scanned to the tail of the idle engine number pool. The writing process is as follows: the engine number to be scanned is written from the circular buffer slot pointed to by the write pointer of the idle engine number pool, and then the write pointer is moved one slot to the right. If the write pointer reaches the tail of the circular buffer, it is wrapped back to the head of the circular buffer. The capacity of the circular buffer is set to 256, which is based on the fact that the DPU supports a maximum of 256 hardware engines, and the number of slots in the idle engine number pool needs to accommodate the engine numbers of all hardware engines.

[0070] In practice, if the busy / idle flag indicates busy, or if there are unprocessed interrupts in the interrupt pending register, the engine number recycling operation will not write the engine number to be scanned into the idle engine number pool. Afterward, the engine number recycling operation will shift the scan cursor of the engine number to be scanned in the hardware engine number list by one position. The calculation method for shifting the scan cursor by one position is to increment the scan cursor value by 1. If the scan cursor value reaches the total number of entries in the hardware engine number list, the scan cursor value will be set to 0.

[0071] In practice, the engine number recycling operation performs the same operation on the next engine number to be scanned. The next engine number to be scanned is the engine number in the table pointed to by the scan cursor after shifting one position in the hardware engine number list. The engine number recycling operation continues to repeat the process of reading the status register, checking the busy / idle flag, reading the interrupt pending register, determining whether to write to the idle engine number pool, and shifting the scan cursor until a round of scanning is completed.

[0072] In practice, after the unloading decision module compares the read pointer and the write pointer, if the read pointer and the write pointer are not equal, it determines that there is a readable engine number in the idle engine number pool, and retrieves the engine number from the circular buffer slot pointed to by the read pointer. The following formula is used when reading the slot:

[0073] in, This represents the slot offset of the annular buffer zone, with a value ranging from 0 to... ; This represents the read pointer to the idle engine number pool, and its value is an integer greater than or equal to 0. This represents the capacity of the circular buffer for the idle engine number pool, and its value is an integer greater than 0. In this embodiment... It equals 256; This indicates a modulo operation. The engine number is read from the address pointed to by the product of the base address of the circular buffer and the slot offset and the width of a single slot.

[0074] In practice, after retrieving the engine number, the unloading decision module moves the read pointer one slot to the right. Moving one slot is calculated by incrementing the read pointer value by 1. If the read pointer value reaches the capacity of the circular buffer, the read pointer value is set to 0. The unloading decision module then writes the moved read pointer back to the read pointer storage location in the idle engine number pool. After retrieving the engine number, the unloading decision module continues the operation of marking the engine number as occupied.

[0075] See Figure 6In the graph, the horizontal axis represents the number of scan rounds for engine number recycling operations, ranging from 0 to 1000, and the vertical axis represents the capacity of the idle engine number pool, in units of 0 to 256. The graph contains two curves: the solid curve represents "checking for interrupted engine number recycling" and the dashed curve represents "not checking for interrupted engine number recycling".

[0076] The curves show periodic fluctuations, with a cycle of approximately 150 to 200 scan rounds. Within each cycle, the idle engine number pool capacity gradually increases from a lower value to near the maximum capacity of 256, then rapidly decreases back to a lower value, reflecting the recycling and consumption process of the engine number pool. The solid curve remains below the dashed curve overall, indicating that when the interruption suspension flag is checked, the idle engine number pool capacity fluctuates less and the peak value is slightly lower than when the interruption suspension flag is not checked.

[0077] Specifically, when interruption suspension is not checked, the engine number recycling operation writes engine numbers to the idle engine number pool more frequently, causing the peak pool capacity to be closer to the upper limit of the circular buffer capacity of 256. However, when the interruption suspension flag is checked, after the recycling operation determines whether there are any unprocessed interrupts in the engine, some busy or idle engine numbers or those with unprocessed interrupts are not written to the idle engine number pool, so the peak capacity is lower and smoother.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A DPU-based intelligent network interface card (NIC) network task offloading system, characterized in that, include: The network data access module is used to receive data packets and write them to the receiving queue after adding the arrival timestamp. The task description generation module is used to parse the packet header, generate a task descriptor containing the flow identifier, protocol type, and payload length, and write the task descriptor into the task description queue. The unloading decision module is used to read the task descriptors from the task description queue in the order of arrival, query the flow table according to the flow identifier, and if the flow table is matched, extract the stored unloading path identifier; if the flow table is not matched, allocate the unloading path identifier according to the protocol type and load length and store it in the flow table. The hardware execution module is used to distribute the data packet corresponding to the task descriptor to the hardware engine specified by the DPU according to the unloading path identifier; The feedback module is used to read the completion queue depth and occupancy count of the hardware engine and maintain the valid status of each flow table entry in the flow table.

2. The DPU-based intelligent network interface card network task offloading system according to claim 1, characterized in that, The step of the task description generation module parsing the data packet header and generating a task descriptor containing a flow identifier, protocol type, and payload length includes: Extract the 5-tuple, protocol type, and payload length from the packet header; Concatenate the quintuples into a byte string, and perform a logical XOR operation on the byte string to obtain the stream identifier; Align the payload length with the storage page size to obtain the alignment mark; The protocol type, the stream identifier, the payload length, and the alignment flag are written into the task descriptor in a preset field order, and the task descriptor is written into the task descriptor queue.

3. The DPU-based intelligent network interface card network task offloading system according to claim 1, characterized in that, The step of allocating offload path identifiers based on protocol type and payload length includes: Obtain the idle engine number pool maintained by the DPU, retrieve an engine number from the idle engine number pool, and mark the engine number as occupied; Write the flow identifier in the task descriptor into the hash table entry corresponding to the engine number, and use the storage address of the hash table entry as the flow table index; The flow table index and the engine number are assembled into an unloading path identifier, and the unloading path identifier is written into the flow table.

4. The DPU-based intelligent network interface card network task offloading system according to claim 1, characterized in that, The step of the hardware execution module distributing the data packet corresponding to the task descriptor to the hardware engine specified by the DPU according to the unloading path identifier includes: The engine number is parsed from the unloading path identifier, the DPU remapping register is read, and the base address of the corresponding descriptor ring is obtained according to the engine number. Write the payload address, payload length, and flow identifier from the data packet record corresponding to the task descriptor into the next slot of the descriptor ring; Increment the write pointer of the descriptor ring and trigger the DMA controller to move the load data according to the slot content pointed to by the write pointer.

5. The DPU-based intelligent network interface card network task offloading system according to claim 1, characterized in that, The steps of the feedback module reading the completion queue depth and occupancy count of the hardware engine and maintaining the valid state of each flow table entry include: During each polling cycle, retrieve the completion descriptor from the completion queue; Locate the flow table entry based on the flow identifier in the completion descriptor, and increment the activity field of the flow table entry by the packet count value in the completion descriptor; The current sampled value of the completed queue depth is weighted and combined with the resource usage field value of the previous period, and the combined result is used as the new resource usage field value. When the resource occupancy field value of the flow table entry changes, the position of the flow table entry in the priority chain is adjusted to the position corresponding to the resource occupancy field value.

6. The DPU-based intelligent network interface card network task offloading system according to claim 2, characterized in that, The step of concatenating the quintuples into a byte string and performing a logical XOR operation on the byte string to obtain the stream identifier includes: Convert the source IP address, destination IP address, source port number, destination port number, and protocol number in the quintuple into values ​​with fixed byte order. The values ​​are concatenated into a byte string in the order of source IP address, destination IP address, source port number, destination port number, and protocol number. The byte string is divided into multiple four-byte blocks, and each four-byte block is XORed bitwise in sequence to obtain the folded value; If the fold value is zero, then the fold value is set to a preset non-zero constant, and the fold value is used as a stream identifier.

7. The DPU-based intelligent network interface card network task offloading system according to claim 3, characterized in that, The step of retrieving an engine number from the idle engine number pool includes: Read the read and write pointers from the idle engine number pool; If the read pointer and the write pointer are equal, then the engine number recycling operation is performed. The engine number recycling operation scans the status register of each hardware engine and writes the engine number with the free field set in the status register into the free engine number pool. If the read pointer and the write pointer are not equal, the engine number is retrieved from the circular buffer slot pointed to by the read pointer, and the read pointer is moved one slot to the right.

8. The DPU-based intelligent network interface card network task offloading system according to claim 4, characterized in that, The step of writing the payload address, payload length, and flow identifier from the data packet record corresponding to the task descriptor into the next slot of the descriptor ring includes: Read the write pointer and ring capacity value of the descriptor ring, and use the sequential position of the write pointer within the ring capacity value range as the slot offset; Obtain the payload address from the data packet record and adjust the payload address to the starting address required by the DMA controller alignment boundary; The starting address, load length, and flow identifier are combined into descriptor data, and the descriptor data is written into the slot corresponding to the slot offset.

9. The DPU-based intelligent network interface card network task offloading system according to claim 5, characterized in that, The step of weighting and combining the current sampled value of the completed queue depth with the resource usage field value of the previous period includes: Write the current sampled value to the end of the sliding window array in the feedback module, and remove the oldest sampled value from the head of the sliding window array; Arrange the sampled values ​​in the sliding window array in order of size, and take the sampled value in the middle position after the arrangement as the new resource occupancy field value; The corresponding bit is marked in the priority bitmap of the flow table according to the new resource occupancy field value.

10. The DPU-based intelligent network interface card network task offloading system according to claim 7, characterized in that, The step of scanning the status registers of each hardware engine and writing the engine number with the free field set in the status register into the free engine number pool includes: Select a scanned engine number from the hardware engine number list and read the status register corresponding to the scanned engine number; Check the busy / idle flag in the status register. If the busy / idle flag indicates that the machine is idle, then further read the interrupt pending register corresponding to the engine number to be scanned. If there are no unprocessed interrupts in the interrupt pending register, the engine number to be scanned is written to the end of the idle engine number pool; Move the scan cursor one position to the right of the engine number to be scanned in the hardware engine number list, and perform the same operation on the next engine number to be scanned.

Images