An adaptive dma aggregation transmission method and system for fragmented data

Abstract

The application discloses a kind of self-adapting DMA aggregation transmission methods and systems for fragmented data, it is related to computer data transmission and processing technical field, method includes: obtaining metadata;Based on metadata, the dynamic aggregation strategy containing time window condition, capacity threshold condition and logical transaction condition is evaluated, and three conditions are determined in coordination by multistage state machine to screen target data block;In the aggregation DMA descriptor chain pool, descriptor unit is allocated, and the metadata of target data block is linked according to logical order to build and generate aggregation DMA descriptor chain;Based on the chain, trigger direct memory access operation to realize batch transmission;After transmission is completed, two-level determination-based predictive interrupt inhibition and hot cache lazy recovery based on local free linked list are executed.By the synergistic aggregation of multidimensional strategy, dynamic parameter adjustment of load sensing, predictive interrupt inhibition and adaptive resource recovery, the system throughput and bus bandwidth utilization are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of computer data transmission and processing technology, specifically to Direct Memory Access (DMA) data transmission technology, and more particularly to an adaptive DMA aggregation transmission method and system for fragmented data. This application can be applied to efficient data handling and transmission in scenarios such as high-performance computing, artificial intelligence computing clusters, big data processing, network communication, and embedded multi-core processors. Background Technology

[0002] In modern computer systems such as high-performance computing, artificial intelligence training and inference, big data processing, and high-speed network communication, data transmission throughput and latency are among the key bottlenecks determining the overall system performance. To alleviate the burden on the central processing unit (CPU) during data transfer, Direct Memory Access (DMA) technology is widely used. It allows hardware subsystems to independently access system memory for data reading and writing, thereby freeing the CPU from heavy data transfer tasks and allowing it to focus on the execution of computation and control logic.

[0003] Modern DMA controllers generally support Scatter-Gather (SG) DMA transfer mode, which uses a set of descriptors to record the physical addresses and data lengths of multiple non-contiguous memory regions. The DMA controller sequentially moves each data segment according to the descriptor chain. SG-DMA allows a single DMA operation to access multiple physically non-contiguous memory spaces, and an interrupt is triggered to notify the CPU only after all data segments have been transferred. Compared to traditional block DMA, this represents a significant improvement in transfer efficiency.

[0004] Building upon this foundation, researchers have proposed several further optimization schemes for DMA transfer efficiency. For example, one scheme improves flexibility and response speed for urgent messages by setting priority logic for short messages to change the descriptor sending queue from sequential to out-of-order execution; another scheme reduces hardware / software interaction overhead by sending transmission parameters to the hardware accelerator in advance; and yet another scheme reduces interrupt wait time and improves parallel processing capabilities by separating read / write channels through a double-buffering mechanism or designing a multi-core collaborative DMA architecture. Furthermore, research has proposed a modular and configurable intelligent DMA engine architecture that supports the on-demand combination of functional units for different application scenarios.

[0005] However, the existing DMA data transfer technology still has the following obvious drawbacks when dealing with high-concurrency, fragmented, small-granular data streams: First, DMA startup overhead is high, failing to fully leverage the advantages of hardware batch transfer. Existing solutions generally adopt a passive, request-sequence-based processing mode. When data producers generate discrete data blocks, the system directly maps them to DMA descriptors and submits them sequentially, lacking an intermediate layer for intelligent analysis and reorganization of data and its metadata. Each small data block may trigger an independent DMA transaction, leading to frequent start-stop of the DMA controller. Although SG-DMA can process multiple discontinuous regions in a single operation, the construction of its descriptor chain is still triggered on a request-by-request basis, lacking proactive identification and batch integration of logical or spatiotemporal correlations between data blocks.

[0006] Secondly, the high interrupt frequency easily triggers interrupt storms. Due to the lack of policy-based proactive aggregation, hardware interrupts are triggered after each or batch of small-scale transmissions. In high-throughput scenarios, hundreds of thousands or even millions of interrupts can occur per second, causing the CPU to frequently enter the interrupt context for descriptor reclamation and buffer processing. This results in significant context switching overhead and severely strains business computing resources. Although some solutions support configuring interrupt aggregation parameters (such as aggregation count and latency), they are essentially passive merging based on fixed thresholds or fixed timeouts. They cannot perceive the arrival patterns and logical contexts of subsequent tasks, leading to a dilemma of insufficient aggregation and the introduction of additional latency in scenarios with sudden or fluctuating traffic.

[0007] Third, it lacks data awareness and cannot identify the logical relationships between data blocks. In existing solutions, the construction of DMA descriptors relies solely on the physical address and length information of data blocks, without considering the transaction, data stream, or business context to which the data blocks belong. Even if multiple data blocks belong to the same logical transaction (such as different segments of the same video frame or multiple segments of the same RDMA message), the system cannot proactively identify and aggregate them into a single batch transmission, thus missing the opportunity for optimization through logical reorganization.

[0008] Fourth, the tight coupling between buffer reclamation and data transfer completion exacerbates competition for multi-core resources. Current solutions immediately release the buffer and return it to the global memory pool after DMA transfer, a process involving the acquisition and release of a global mutex lock. In multi-core concurrent scenarios, this global lock becomes a severe performance bottleneck. Furthermore, the immediate reclamation of the buffer causes premature invalidation of "hot" data in the CPU cache, preventing the use of data locality principles to improve subsequent access performance.

[0009] In summary, existing DMA data transfer technologies, when faced with high-concurrency, fragmented data streams, suffer from problems such as underutilization of DMA bandwidth potential, excessive CPU interrupt load, and limited multi-core scalability due to the lack of an intelligent intermediate coordination layer between data production and hardware execution. Therefore, constructing a highly efficient DMA transfer mechanism capable of proactively sensing data attributes, dynamically aggregating, and intelligently scheduling data is of significant practical importance for improving overall system throughput, reducing CPU load, and optimizing bus bandwidth utilization. Summary of the Invention

[0010] The purpose of this invention is to provide an adaptive DMA aggregation transmission method and system for fragmented data, addressing the aforementioned problems, and to optimize the transmission efficiency of scattered data blocks in computer systems.

[0011] The technical solution of the present invention is as follows: An adaptive DMA aggregation transfer method for fragmented data includes the following steps: Obtain metadata, which is associated with the data block to be transmitted stored in the distributed data buffer pool. The metadata includes at least the storage address and attribute information of the data block to be transmitted. The preset dynamic aggregation strategy is evaluated based on storage address and attribute information. The dynamic aggregation strategy defines the time window conditions, capacity threshold conditions or logical transaction conditions that trigger the aggregation operation. Based on the evaluation results, target data blocks are selected from the metadata; Descriptor units are allocated in the aggregated DMA descriptor chain pool, and the metadata corresponding to the target data block is written and linked in logical order to build and generate the aggregated DMA descriptor chain. Based on the aggregated DMA descriptor chain, transfer instructions are generated and direct memory access operations are triggered to enable the batch transfer of target data blocks scattered in the distributed data buffer pool to the target address.

[0012] Furthermore, the evaluation of the preset dynamic aggregation strategy based on storage address and attribute information includes: The multi-level state machine coordinates the determination of time window conditions, capacity threshold conditions and logical transaction conditions. The multi-level state machine includes at least idle state, cumulative state, emergency state and transaction ready state. In the idle state, only the time window condition takes effect, and the capacity threshold condition is bypassed; in the cumulative state, the time window condition and the capacity threshold condition take effect in parallel, and the logical transaction condition has priority to interrupt; in the emergency state, the capacity threshold condition is bypassed, and aggregation is triggered as soon as the queue is not empty. The formula for determining the time window condition is: ,when The time condition is met, among which The write timestamp of the earliest data block in the queue. The current system time at the time of this assessment. This is the preset maximum waiting time; The capacity threshold condition is determined by: accumulating the data lengths of all data blocks to be transmitted in the queue. ,when The condition is met when the storage addresses of each data block do not violate the physical page boundary limits or alignment requirements of the DMA controller. This is a preset capacity threshold; The determination of logical transaction conditions includes: the condition is met when there is a set of data blocks in the queue with the same transaction identifier and containing a transaction end tag, or when there is a data block carrying a high priority tag.

[0013] Furthermore, after obtaining the metadata, the process also includes: Real-time monitoring of the percentage of idle resources in the distributed data buffer pool ; when Below the preset low water level At that time, the dynamic aggregation strategy is adjusted according to the segmented step algorithm: when At that time, according to the first attenuation ratio Adjustment: , ;in The low waterline, The adjusted capacity threshold, This is the adjusted time window value; when At that time, according to the second attenuation ratio Adjustment: , ; in , , This is the baseline value for the capacity threshold. This serves as the baseline value for the time window. when At that time, restore , .

[0014] Furthermore, the step of filtering target data blocks from metadata based on the evaluation results includes: Extract the transaction identifier from the metadata; Matching is performed using a two-level hash index structure. The first-level hash index uses the transaction identifier as the key, and the second-level hash index uses the sequence number as the key. Multiple data blocks with the same transaction identifier but non-contiguous physical addresses in the distributed data buffer pool are identified as target data blocks. An ordered linked list is constructed based on the sequence number of the target data block, and the blocks are arranged in ascending order of sequence number as the logical order of aggregation and transmission. When some data blocks of the same transaction have not arrived, the consecutive arrived data blocks starting from sequence number 0 will be constructed into an aggregate descriptor chain for transmission, and the unarrived data blocks will be kept in the two-level hash index to wait for the next round of aggregation window processing.

[0015] Furthermore, after generating the transfer instruction based on the aggregated DMA descriptor chain and triggering the direct memory access operation, the process further includes: Upon detecting a transmission completion event, a two-level interrupt suppression determination is performed: Level 1 Decision: Retrieve the head node of the queue to be executed. If the head node exists and belongs to the same logical context, suppress the current interrupt notification and continuously send the transmission instructions to the head node. Second-level decision: If the queue to be executed is empty, calculate the average arrival time interval of the most recent N transmission tasks. ,like If a predictive wait timer is started and the interrupt notification is delayed, the interrupt notification is merged if a new task arrives before the timer expires; otherwise, an interrupt notification is triggered. As an interrupt delay threshold, This represents the time difference between two consecutive transmission tasks arriving at the system. Additionally, it sets the maximum number of consecutive interruption suppression thresholds MAX_SUPPRESS_COUNT and the maximum total delay threshold MAX_SUPPRESS_DURATION. When either threshold is reached, the merged interrupt notification is forcibly triggered and the suppression count is reset.

[0016] Furthermore, the step of generating transfer instructions and triggering direct memory access operations based on the aggregated DMA descriptor chain includes: Calculate the credit consumption of the aggregated DMA descriptor chain. ,in The total data size of the aggregate DMA descriptor chain. Credit value per unit; Calculate your current available credit score ,in This represents the maximum credit limit for the current business flow. The sum of credits consumed by submitted but incomplete transfers; like The aggregated DMA descriptor chain is inserted into the priority heap, which is sorted by the chain's priority. Each chain is submitted to the DMA controller for execution according to the pop order of the priority heap.

[0017] Furthermore, the step of submitting each chain to the DMA controller for execution according to the priority heap popping order also includes: like Then the aggregated DMA descriptor chain will be split into multiple sub-chains, and the data volume of each sub-chain will not exceed MAX_TX_PER_BURST, where MAX_TX_PER_BURST is a preset single transaction threshold. Multiple subchains are configured to the transport interface in batches, and other chains in the priority heap are allowed to jump the queue during the transport intervals between adjacent subchains.

[0018] Furthermore, after generating the transfer instruction based on the aggregated DMA descriptor chain and triggering the direct memory access operation, the process further includes: Based on the source address field of each descriptor unit in the aggregated DMA descriptor chain, locate all the scattered buffer blocks occupied by this transfer in batches; Update the status flag of the scatter buffer block to be locally available, and attach the memory pointer of the scatter buffer block to the local free list exclusively used by the current CPU core. When the current CPU core requests a buffer, it will prioritize allocating a buffer block from the local free list. When the number of buffer blocks in the local free list reaches the preset capacity limit MAX_LOCAL_CACHE, the excess buffer blocks are returned in batches to the global distributed data buffer pool.

[0019] Furthermore, the step of constructing an ordered linked list based on the sequence number of the target data block also includes: The second-level hash index records the arrangement status of each sequence number; When subsequent data blocks of the same transaction arrive, the second-level hash index is queried to check whether the sequence number of the data block has been marked as orchestrated. If it has been marked, it is determined to be a duplicate data block and deduplication is triggered; If it is not marked, it is inserted into the corresponding position of the ordered linked list according to the sequence number of the data block, and marked as arranged in the second-level hash index.

[0020] This application also includes an adaptive DMA aggregation transfer system for fragmented data, implementing an adaptive DMA aggregation transfer method for fragmented data, including: The data acquisition module is used to acquire metadata, which is associated with the data block to be transmitted stored in the distributed data buffer pool. The metadata includes at least the storage address and attribute information of the data block to be transmitted. The strategy evaluation module is used to evaluate the preset dynamic aggregation strategy based on storage address and attribute information. The dynamic aggregation strategy defines the time window conditions, capacity threshold conditions or logical transaction conditions that trigger the aggregation operation. The association identification module is used to filter target data blocks from metadata based on the evaluation results; The chain building module is used to allocate descriptor units in the aggregated DMA descriptor chain pool, write and link the metadata corresponding to the target data block in logical order, so as to build and generate an aggregated DMA descriptor chain. The transmission scheduling module is used to generate transmission instructions based on the aggregated DMA descriptor chain and trigger direct memory access operations so that target data blocks scattered in the distributed data buffer pool are transmitted in batches to the target address.

[0021] Compared with existing technologies, the advantages of this invention are: 1. Significantly reduces DMA startup overhead and improves batch transfer efficiency. This application introduces a dynamic aggregation strategy to logically aggregate originally scattered small-granular DMA requests into a large-granularity descriptor chain, enabling the DMA controller to complete data transfer tasks that originally required multiple startups with a single startup. This significantly reduces the number of DMA controller start-stop cycles and effectively releases the hardware's batch transfer potential.

[0022] 2. Effectively suppress terminal storms and reduce CPU load. This application uses a predictive interrupt suppression mechanism to proactively merge interrupt notifications based on queue status and historical arrival patterns. This merges multiple small transmission interrupts into a single notification, significantly reducing the number of interrupts per unit time, reducing CPU interrupt context switching overhead, and freeing up more computing resources for business processing.

[0023] 3. Achieve data awareness and logical transaction reorganization to improve the semantic integrity of transmission. This application actively identifies discrete data blocks belonging to the same logical transaction or business flow through a metadata mechanism and a two-level hash index structure, and rearranges the transmission order according to their logical sequence numbers, ensuring that the receiving end directly obtains continuous and ordered data, avoiding the overhead of additional reordering required after data arrival in traditional solutions.

[0024] 4. Reduce multi-core resource contention and improve system scalability. This application uses lazy reclamation and hot caching mechanisms to retain the completed buffer locally in a hot cache state for subsequent reuse by the same CPU core. This eliminates high-frequency global lock contention, improves cache hit rate, and has excellent scalability in multi-core high-concurrency scenarios.

[0025] 5. End-to-end collaborative optimization for adaptive load adjustment. This application forms a complete closed-loop feedback chain from metadata acquisition, aggregation decision-making, descriptor chain construction, transmission scheduling to resource reclamation. It can automatically adjust aggregation strategy parameters according to real-time load status, achieving a dynamic balance between low latency and high throughput to adapt to diverse business scenario requirements. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the method steps of this application.

[0027] Figure 2 This is a detailed flowchart of the method described in this application.

[0028] Figure 3 This is a structural block diagram of the system of this application.

[0029] Figure 4 A structural block diagram of an electronic device provided in this application. Detailed Implementation

[0030] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0031] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0032] Please see Figure 1-4 An adaptive DMA aggregation transfer method for fragmented data, such as Figure 1 As shown, it includes the following steps: Step S101: Obtain metadata.

[0033] In this embodiment, the system first pre-builds and maintains a distributed data buffer pool in memory to temporarily store discrete data blocks generated by data producers (such as applications, network protocol stacks, or device drivers), whose physical addresses may not be contiguous. Unlike the traditional direct DMA operation, the producer does not directly initiate a DMA request after writing data, but instead generates metadata describing the characteristics of the data block.

[0034] Specifically, this metadata is a lightweight data structure that includes, but is not limited to, the following fields: The metadata structure includes a physical address field to record the starting address of the data block in physical memory. This address is used to fill the source address field of the DMA descriptor later. The metadata structure also includes a data length field to record the size of the data block in bytes. This length value is filled into the descriptor's length field when the descriptor is constructed and is also used to accumulate the total amount of data in the current queue during capacity threshold evaluation. The metadata structure also includes a timestamp field to record the time the data block was written to the buffer by the producer. This timestamp is used to calculate the data block's residence time in the queue during time window evaluation, i.e., the difference between the current system time and the write timestamp. The metadata structure also includes a transaction identifier field to identify the logical transaction to which the data block belongs. All data blocks within the same logical transaction have the same transaction identifier value. This transaction identifier is used to identify the set of data blocks belonging to the same transaction during logical transaction evaluation and is also used as the primary hash key in the first-level hash index. The metadata structure also includes a sequence number field to record the logical order position of the data block within its transaction, such as fragment offset or message sequence number. This sequence number is used as a secondary hash key in the second-level hash index and as a sorting criterion in the ordered linked list, ensuring that the receiving end can restore the data in the correct order. The metadata structure also includes a priority field to identify the quality of service level of the data block. This priority is used in credit value scheduling to determine the sorting key value of the priority heap; the aggregation descriptor chain corresponding to higher-priority data blocks will be submitted to the DMA controller for execution first. The metadata structure also includes a data type label field to identify the service type to which the data block belongs, such as video stream data, control signaling data, or batch file data. This data type label can be used as a substitute field for transaction identifiers for hash index matching, suitable for aggregation scenarios of data blocks that do not belong to a specific transaction but have the same processing requirements.

[0035] This metadata is asynchronously submitted to a processing queue in an intermediate layer, thereby achieving dual decoupling of data production and transmission control in both time and control dimensions—the producer can continue to produce new data without waiting for DMA transmission to complete, and the transmission control logic does not need to be aware of the producer's specific implementation details.

[0036] Step S102: Evaluate the preset dynamic aggregation strategy based on storage address and attribute information.

[0037] In this embodiment, the specific process of evaluating the preset dynamic aggregation strategy is as follows: First, the system parses the metadata of the data blocks to be transmitted in the current buffer queue in real time, extracting their corresponding physical storage addresses and attribute information including data length, write timestamp, and transaction identifier. Then, the extracted information is compared and combined with the various judgment conditions in the preset dynamic aggregation strategy.

[0038] (a) Evaluation and determination of time window conditions: In evaluating the time window condition, the system reads the write timestamp of the earliest data block to be transmitted that entered the current buffer queue, and calculates the difference between the current system time and the write timestamp. This difference is the dwell time of the data block in the queue. If the dwell time exceeds the preset maximum waiting time, the time window condition is deemed met, and the aggregation operation is triggered.

[0039] In this embodiment, the maximum waiting time is set to 500 microseconds. This value is based on the following considerations: In high-speed network scenarios, the end-to-end transmission delay of data packets is typically in the range of tens to hundreds of microseconds. Setting the waiting time to 500 microseconds allows sufficient data blocks to be accumulated to form an effective batch, without introducing additional latency exceeding one typical network round-trip time, thus achieving a balance between transmission latency and DMA startup overhead. It should be noted that this value can be flexibly adjusted according to the actual application scenario. For example, it can be shortened to 100 microseconds in latency-sensitive scenarios and extended to 1 millisecond in throughput-priority scenarios.

[0040] (ii) Evaluation and determination of capacity threshold conditions: In evaluating the capacity threshold, the system reads the data length field of all data blocks to be transferred in the current buffer queue and sums them to obtain the current cumulative data volume. Simultaneously, the system also considers the extracted physical storage addresses to evaluate whether these data blocks comply with the physical page boundary limits or alignment requirements of the DMA controller during DMA batch transfer. Specifically, the system checks whether the starting address of each data block matches the address alignment requirements of the DMA controller, such as whether it meets the requirements of 64-byte alignment or 4KB page alignment, and also checks for page table continuity limitations during cross-page access. If the cumulative data volume reaches the preset batch processing capacity threshold, and the storage addresses of all data blocks to be transferred do not violate the physical boundary limits of the DMA hardware, the capacity threshold condition is deemed met, and the aggregation operation is triggered.

[0041] In this embodiment, the batch processing capacity threshold is set to 64KB. This value is based on the following considerations: 64KB is an integer multiple of the optimal burst transfer length for most modern DMA controllers, while remaining compatible with the huge page size of the memory management unit. This allows for the transfer of sufficient data in a single DMA transaction to fully utilize the bus bandwidth, without causing other tasks to starve due to excessively large single transfers occupying the DMA channel for extended periods. This threshold can also be adaptively adjusted on different hardware platforms.

[0042] (III) Evaluation and judgment of logical transaction conditions: In evaluating logical transaction conditions, the system analyzes the transaction identifier field in the attribute information of the data blocks to be transmitted. If it detects that data blocks with the same transaction identifier have accumulated in the current queue, and that these include data blocks with a specific control tag (which indicates that all data blocks for this transaction have arrived), or if it detects that a data block with a specific high-priority business tag has entered the queue, then the logical transaction condition is deemed met, and the aggregation operation is triggered directly without waiting for the capacity threshold or time window timeout. This logical transaction condition has a higher priority than the time window condition and the capacity threshold condition; that is, as soon as this condition is triggered, the system immediately interrupts the current accumulated waiting state and enters the aggregation process.

[0043] (iv) State machine mechanism for multi-condition collaborative evaluation: In this embodiment, the evaluation of the above three conditions is not a simple parallel or logical relationship of "triggering upon satisfaction of any one condition," but rather a coordinated scheduling process using a multi-level state machine. This state machine includes the following four working states: The first state is the idle state. In this state, the current buffer queue is empty or contains only a few scattered data blocks, and the amount of data in the queue is far from reaching the benefit threshold for batch transfer. In this state, the system uses only the time window condition as the sole trigger—that is, when the dwell time of the earliest data block entering the queue exceeds the preset maximum waiting time, the state machine transitions out of the idle state and triggers aggregation. The capacity threshold condition in the idle state is actively bypassed and ignored by the system because even if DMA transfer is triggered when the data volume is too small, the ratio of effective data load to transfer overhead is too low, leading to a waste of bus bandwidth. When the amount of data in the queue reaches 30% of the capacity threshold, the state machine transitions from the idle state to the accumulation state.

[0044] The second state is the accumulation state. In this state, a certain number of data blocks have accumulated in the queue, and the process of accumulating data for batch transmission is underway. In the accumulation state, the time window condition and the capacity threshold condition take effect in parallel—the aggregation operation is triggered when either condition is met. Simultaneously, logical transaction conditions have the highest priority for interruption in the accumulation state: when the system detects that all data blocks belonging to the same transaction have arrived, regardless of whether the current time window has expired or the capacity threshold has been reached, the state machine immediately jumps to the transaction-ready state and triggers aggregation. This ensures that the transaction can be transmitted atomically in one go, avoiding additional reassembly waiting time at the receiving end due to transaction splitting.

[0045] The third state is the emergency state. In this state, either the time window has expired (i.e., the earliest data block in the queue has been waiting longer than the maximum waiting time), or the buffer idle rate is lower than the preset low watermark (i.e., system resources are strained). In the emergency state, the capacity threshold condition is temporarily bypassed by the system—as long as there are data blocks to be transmitted in the queue, an aggregation operation is triggered to move the data out as quickly as possible and release the buffer space, preventing the system from blocking or losing packets due to resource exhaustion. After the transmission is completed, if the queue is empty, the state machine rolls back to the idle state; if there is still data in the queue, the state machine remains in the emergency state until the queue is emptied or the idle rate recovers.

[0046] The fourth state is the transaction-ready state. In this state, the logical transaction conditions have been triggered—that is, all data shards with the same transaction identifier have arrived, or data blocks carrying high-priority tags have been enqueued. In the transaction-ready state, the system does not wait for the time window to expire or for the capacity threshold to be reached; it immediately triggers an aggregation operation, packaging all data blocks belonging to the transaction into an aggregation descriptor chain and committing it for transmission. After the transaction transmission is complete, the state machine rolls back to the accumulated state or the idle state based on the queue state.

[0047] The transition relationships between the four states are as follows: From the idle state, transition to the emergency state when the time window times out, and to the accumulation state when the queue data volume reaches 30% of the capacity threshold. From the accumulation state, transition to the emergency state when the time window times out or the capacity threshold is met, and to the transaction ready state when the logical transaction condition is triggered. From the emergency state, transition to the idle state if the queue is empty after the transmission is completed. From the transaction ready state, transition to the accumulation state if the queue data volume exceeds 30% of the capacity threshold after the transaction transmission is completed; otherwise, transition to the idle state.

[0048] In the state machine described above, the logical transaction condition has the highest interruption priority—regardless of whether the current state machine is in the accumulating or urgent state, once it is detected that all data blocks for the same transaction have arrived, the state machine immediately and unconditionally jumps to the transaction-ready state. The time window condition is next in priority, effective in both the idle and accumulating states. The capacity threshold condition only takes effect in the accumulating state and is bypassed in both the idle and urgent states. Through this hierarchical priority design, the three aggregation conditions are no longer isolated "OR" relationships, but form an ordered decision-making system with state dependencies and priority levels.

[0049] Step S103: Based on the evaluation results, filter the target data blocks from the metadata.

[0050] Once any condition in the above dynamic aggregation strategy is met, or the state machine determines that aggregation is triggered, the assembly engine starts the filtering action. Based on the type of the currently triggered condition and the state of the state machine, the engine extracts metadata for a set of target data blocks from the processing queue. The filtering strategy differs under different triggering conditions.

[0051] Specifically, when a logical transaction condition is triggered or the state machine is in a transaction ready state, the engine will filter out all metadata with the same transaction identifier. These data blocks may be physically discontinuous in the queue, but they logically belong to the same business transaction and need to be aggregated together for transmission to ensure the atomicity of the transaction and data integrity.

[0052] When a capacity threshold condition is triggered, or when the state machine is in an accumulation state and the capacity condition is met, the engine extracts metadata in a first-in, first-out (FIFO) order until the total length of the extracted data blocks reaches or exceeds the batch processing capacity threshold. This sequential extraction method is suitable for ordinary data streams that do not have strong logical relationships, and can achieve batch transmission while ensuring basic fairness.

[0053] When a time window condition is triggered or the state machine is in an emergency, the engine extracts all metadata to be transmitted from the current queue, regardless of its total amount. This full extraction method aims to clear the queue as quickly as possible, freeing up buffer resources and preventing system blockage or data loss in extreme cases of resource scarcity or timeouts.

[0054] Through the aforementioned differentiated screening strategies, the system can adopt the most appropriate aggregation granularity under different load conditions and triggering reasons, thereby achieving the optimal balance between transmission efficiency and resource utilization.

[0055] Step S104: Allocate descriptor units in the aggregated DMA descriptor chain pool, write and link the metadata corresponding to the target data block in logical order to build and generate the aggregated DMA descriptor chain.

[0056] In this embodiment, the system pre-maintains an independent resource pool, called the aggregated DMA descriptor chain pool, which contains a large number of pre-allocated, hardware-recognizable DMA descriptor units. Each descriptor unit contains at least the following fields at the hardware level: a source address field, used to store the physical memory starting address of the data block to be transferred; a length field, used to store the size of the data block; a next descriptor pointer pair field, used to point to the next descriptor unit in the chain, thereby realizing the link between descriptors; and a control flag field, used to store various control information required by the DMA controller when executing the descriptor, such as interrupt enable flag, transfer completion flag, etc.

[0057] Once the target data blocks are identified, the system does not physically copy the data content. Instead, it requests a corresponding number of free descriptor units from the descriptor chain pool, equal to the number of target data blocks. Then, the system sequentially fills the source address and length fields of the descriptor units with the physical address and length information of each target data block selected in step S103, and links them together by setting the next descriptor pointer field, forming a complete aggregated DMA descriptor chain. The head pointer of this descriptor chain is recorded in the execution queue, awaiting subsequent scheduling and commit. Because the descriptor chain pool is pre-allocated, the allocation and release operations of descriptor units can be completed in constant time, avoiding the latency uncertainty caused by dynamic memory allocation on the critical path of data transmission.

[0058] Step S105: Generate a transfer instruction based on the aggregated DMA descriptor chain and trigger a direct memory access operation.

[0059] Once the aggregation chain is built, the system generates a transfer instruction pointing to the head of the chain and writes this instruction to the DMA controller's command register to trigger execution. Upon receiving this instruction, the DMA controller, guided by the descriptors in the chain, automatically and continuously reads data blocks from the distributed data buffer pool using the hardware's scatter-aggregate capability and batch-transfers them to the target address. The target address can be the network interface card's transmit queue address, the disk controller's data buffer address, or the address of another memory region, depending on the specific application scenario.

[0060] During the transfer, the DMA controller processes each descriptor sequentially according to the descriptor chain: it reads the amount of data indicated by the length field from the location indicated by the source address field, writes it to the location indicated by the internal destination address register, and then automatically reads the next descriptor and repeats the above process until the end of the chain. The entire transfer process requires no further CPU intervention. Once all descriptors in the chain have been processed, the DMA controller determines whether to trigger a transfer completion interrupt based on the control flags of the last descriptor.

[0061] Through the steps described above, this embodiment transforms the previously numerous, fragmented, and high-frequency CPU-interventional operations—such as configuring a DMA descriptor individually for each small data packet—into a small number of batch chain construction operations. This not only significantly reduces the CPU instruction cycle consumption per unit of data but also substantially improves the bus utilization and effective throughput of the DMA controller.

[0062] Please see Figure 2 A detailed step-by-step method for adaptive DMA aggregation transfer of fragmented data includes: Step S201: Obtain metadata.

[0063] This step is basically the same as step S101, and will not be repeated here.

[0064] Step S202: Monitor the proportion of idle resources in the distributed data buffer pool in real time, and dynamically adjust the dynamic aggregation strategy when the proportion of idle resources is lower than the preset low water level.

[0065] In this embodiment, after obtaining the metadata, the descriptor assembly engine or background daemon logic will continuously detect the current status of the distributed data buffer pool and calculate the percentage of the current number of free buffers to the total capacity of the buffer pool, which will be used as a measure of the proportion of free resources.

[0066] When the proportion of idle resources falls below a preset low watermark, the system initiates a segmented, step-by-step dynamic parameter adjustment logic. In this embodiment, the low watermark is set to 20%, meaning that when the proportion of idle buffers falls below 20%, the system is considered to be in a resource-scarce state, requiring proactive adjustment of the aggregation strategy to accelerate buffer release.

[0067] Specifically, the segmented step adjustment logic includes the following three intervals and corresponding adjustment actions: The first interval is the medium load interval. When the detected idle resource ratio is greater than or equal to 10% and less than 20%, the system determines that it is currently in a medium load state. Although the buffer resources are tight, they have not yet reached a dangerous level. At this time, the system adjusts the dynamic aggregation strategy downward according to the first attenuation ratio. This first attenuation ratio is 50%, that is, the system lowers the capacity threshold condition from the baseline value to 50% of the baseline value, and at the same time lowers the time window condition from the baseline value to 50% of the baseline value.

[0068] The second interval is the heavy load interval. When the proportion of idle resources further decreases to less than 10%, the system determines that it is currently under heavy load, and the buffer resources are severely insufficient, posing a risk of imminent exhaustion. At this point, the system switches to a second attenuation ratio to further reduce the dynamic aggregation strategy. This second attenuation ratio is 20%, meaning the system lowers the capacity threshold condition from the baseline value to 20% of the baseline value, and simultaneously lowers the time window condition from the baseline value to 20% of the baseline value.

[0069] Through the aforementioned segmented, step-by-step adjustments, as the buffer idle rate decreases from 20% to 10%, the aggregation threshold linearly reduces from the baseline value to 50% of the baseline value; when the idle rate further decreases to below 10%, the aggregation threshold is further reduced to 20% of the baseline value. This segmented design allows the system to gradually increase the aggregation frequency as resources transition from mild to severe strain, rather than jumping to the most aggressive state all at once, thus avoiding system jitter caused by sudden parameter changes.

[0070] The third interval is the recovery interval. When the proportion of idle resources is subsequently monitored to rise and equal to or above the low water level, the system determines that the resource pressure has been alleviated and restores the capacity threshold conditions and time window conditions to the originally set baseline values. This recovery mechanism ensures that the system can automatically return to high-throughput aggregation mode after the load decreases, without manual intervention.

[0071] In this embodiment, the baseline value for the capacity threshold is set to 64KB, and the baseline value for the time window is set to 500 microseconds. Therefore, under medium load, the capacity threshold is adjusted to 32KB, and the time window is adjusted to 250 microseconds; under heavy load, the capacity threshold is adjusted to 12.8KB, and the time window is adjusted to 100 microseconds. These specific values ​​can be flexibly configured according to the actual hardware platform and application scenario.

[0072] Through this dynamic adjustment logic based on segmented steps within the idle rate range, when buffer resources are tight, the system can sacrifice part of the depth of a single DMA transfer in exchange for faster data turnover and release speed in the buffer, thereby adaptively avoiding the risk of buffer overflow and system blocking due to waiting for aggregation.

[0073] Step S203: Evaluate the preset dynamic aggregation strategy based on storage address and attribute information.

[0074] In this step, the system uses the strategy parameters dynamically adjusted in step S202—namely, the adjusted capacity threshold and time window value—to evaluate the currently acquired metadata according to the multi-condition collaborative evaluation mechanism described in step S102. Specifically, the system replaces the time window determination benchmark from the benchmark value of 500 microseconds with the adjusted value, and replaces the capacity threshold determination benchmark from the benchmark value of 64KB with the adjusted value, while the determination logic of the logical transaction conditions remains unchanged. If any of the adjusted conditions is met, the subsequent filtering and assembly process is triggered.

[0075] Step S204: Extract the transaction ID or data type tag from the metadata, match it using a multi-level hash index structure, find multiple data blocks with the same transaction ID or belonging to the same data type tag and whose physical addresses are not contiguous in the distributed data buffer pool, and arrange the logical order of the target data blocks during aggregation and transmission.

[0076] In this embodiment, the multi-level hash index structure is implemented using a two-level hash table, and its specific working process is as follows: The first-level hash table is used for fast transaction location, using the transaction identifier as the primary hash key. The system extracts the transaction identifier field value from the currently pending metadata, inputs it as the key into the first-level hash function, calculates the corresponding hash bucket index, and thus quickly locates the entry pointer of the second-level hash table corresponding to that transaction identifier. Each bucket in the first-level hash table stores, in addition to a pointer to the second-level hash table, the total number of data blocks accumulated for the transaction, the total data volume, and a status flag indicating whether the transaction's end tag has been reached.

[0077] The second-level hash table is used to quickly locate data fragments within the same transaction, using the sequence number as the secondary hash key. After locating the second-level hash table of a specific transaction through the first-level hash table, the system extracts the sequence number field from the metadata and inputs it as the key into the second-level hash function for precise matching. Through the fast lookup using two levels of hashing, the system can efficiently classify multiple scattered data blocks generated in high-concurrency scenarios, which have non-contiguous physical storage addresses in the distributed data buffer pool but share the same transaction identifier, into the same transaction set.

[0078] After finding all target data blocks with the same transaction identifier in the multi-level hash table, the system enters the logical order orchestration phase. The orchestration process proceeds as follows: First, the system extracts the sequence number field from the metadata of each matched target data block in turn. This sequence number represents the logical position of the data block in its respective transaction.

[0079] Secondly, the system creates an ordered linked list in memory bound to the transaction identifier. Each node in this linked list contains at least three fields: a physical address field, which stores the physical starting address of the data block in the distributed data buffer pool; a data length field, which stores the size of the data block; a sequence number field, which stores the logical sequence number of the data block; and a pointer field pointing to the next node.

[0080] Then, the system traverses each matched target data block and inserts the corresponding linked list node into the corresponding position in the ordered linked list according to its sequence number. The specific insertion logic is as follows: if the sequence number of the node to be inserted is less than the sequence number of the current head node, then that node becomes the new head node; if the sequence number of the node to be inserted is greater than the sequence number of the current tail node, then that node is appended to the tail of the linked list; if the sequence number of the node to be inserted is in the middle of the linked list, then traversing from the head node, finding the first position with a sequence number greater than that of the node to be inserted, and inserting the node before that position. Through this insertion logic, the nodes in the linked list are always arranged in ascending order of sequence number.

[0081] Finally, when the aggregation operation is triggered, the scheduling engine reads the physical address field and data length field from each node in the ordered linked list from head to tail, using this as the filling order when constructing the aggregate DMA descriptor chain. Since the linked list is already sorted by sequence number, the order in the descriptor chain naturally corresponds to the correct logical order of the data blocks in the transaction.

[0082] Furthermore, this embodiment also implements a linkage update mechanism between the index and the linked list, as well as a dynamic truncation strategy. The linkage update mechanism works as follows: When a target data block is successfully inserted into the ordered linked list, the system updates the head and tail pointers of the linked list in the corresponding bucket of the first-level hash table, and simultaneously marks the sequence number of the data block as "arranged" in the second-level hash table. When subsequent data blocks of the same transaction arrive, the system first queries the second-level hash table to check if its sequence number has been marked—if it has been marked, it means that the data block corresponding to that sequence number has arrived and been arranged, and the current data block is a duplicate copy. The system triggers deduplication processing, discarding duplicate data blocks and recording the corresponding log information; if it has not been marked, it means that the data block is a newly arrived fragment, and the system locates the insertion position in the ordered linked list based on its sequence number and updates the linked list.

[0083] The dynamic truncation strategy works as follows: When an aggregation operation is triggered (e.g., time window expires or capacity threshold is met), if some shards for a transaction are still missing, the system does not wait indefinitely for all shards to arrive. Instead, it scans the ordered linked list, checking if consecutive shards starting from sequence number 0 are complete. Specifically, the system traverses the list from the head node, checking if the sequence number of the first node is 0, the sequence number of the second node is 1, and so on, until it encounters a position where the first sequence number is not consecutive or reaches the end of the list. The system constructs the consecutive data blocks into an aggregation descriptor chain for transmission, while retaining the information of the missing shards in the second-level hash index, waiting for the next aggregation window to continue processing.

[0084] By organically combining the above two-level hash matching, ordered linked list arrangement, linkage update and dynamic truncation strategy, the system realizes efficient search, classification and sequential reorganization of small data blocks with discrete physical addresses. This ensures that after these data blocks are transferred to the target address in batches by DMA, the receiving end can directly restore the complete and continuous data content in the correct business logic order without additional rearrangement operations.

[0085] Step S205: Allocate descriptor units in the aggregated DMA descriptor chain pool, write and link the metadata corresponding to the target data block in logical order to build and generate the aggregated DMA descriptor chain.

[0086] This step is basically the same as step S104. Using the pre-allocated aggregated DMA descriptor chain pool, the information of multiple discrete data blocks that have been filtered and sorted in step S204 is sequentially filled into the hardware descriptor according to the order of the ordered linked list, forming a logically complete aggregated transfer task.

[0087] Step S206: Calculate the credit value consumption corresponding to the aggregated DMA descriptor chain and determine the task submission order based on the priority heap structure; if the total data volume of the aggregated DMA descriptor chain exceeds the preset single transaction threshold, the aggregated DMA descriptor chain is logically split into multiple sub-chains, and the multiple sub-chains are configured to the transmission interface in batches to trigger execution.

[0088] In this embodiment, the system maintains a credit value counter for each service flow or each DMA channel. This counter quantifies the currently available bandwidth resources for that flow or channel. The credit value is calculated as follows: the system pre-allocates a credit limit for each flow or channel, which is determined based on the bus bandwidth, the service quality level of the flow, and the overall QoS policy of the system. Credit values ​​consumed by submitted but not yet completed transmission tasks are deducted from this credit limit, and the remaining difference is the currently available credit value.

[0089] When a newly constructed aggregated DMA descriptor chain is ready to be committed, the system first calculates the credits required to execute the chain. The amount of credits consumed is equal to the total data size of the chain divided by the amount of data represented by a unit of credit. For example, if a unit of credit represents 1KB of data, then the credits consumed by the chain are equal to the quotient of the total data size of the chain divided by 1KB. If the consumption is less than or equal to the currently available credits, the chain is marked as committable and enters the priority heap to await scheduling; if the consumption is greater than the currently available credits, the chain is suspended, waiting for the committed transfer tasks to complete and release credits before being committed again.

[0090] All aggregated chains in a committable state are inserted into a priority heap structure. The sorting key of this priority heap is the chain's priority—chains with higher priority values ​​are closer to the top of the heap and will be popped and submitted to the DMA controller for execution first. The chain's priority is derived from the priority field of the data blocks it contains. If a chain contains multiple data blocks with different priorities, the highest priority among them is taken as the chain's priority. Through this priority heap-based scheduling mechanism, high-priority service flows (such as real-time control signaling) can obtain the right to use the DMA channel before large blocks of low-priority data are transmitted, thereby ensuring quality of service.

[0091] Furthermore, to prevent a single, excessively large aggregate chain from monopolizing the DMA channel for an extended period, thus blocking other tasks, the system checks whether the total data volume of the chain exceeds a preset single-transaction threshold before submission. In this embodiment, this single-transaction threshold is set to 1MB. If the total data volume of the chain exceeds this threshold, the original chain is automatically split into multiple sub-chains according to the threshold's granularity, with each sub-chain's data volume not exceeding the threshold. These sub-chains are submitted to the DMA controller sequentially for execution, but other chains in the priority heap have the opportunity to be submitted in the queue during the intervals between sub-chains. Through this splitting and queuing mechanism, the system ensures efficient transmission of large blocks of data while preventing the DMA channel from being monopolized for extended periods, thus ensuring fair scheduling among multiple data streams.

[0092] Step S207: When a transfer completion event of a direct memory access operation is detected, the queue of aggregated DMA descriptor chains to be executed is queried to predict whether there is a next aggregated chain that is about to be ready following the current chain and belonging to the same logical context. If so, an interrupt suppression operation is performed.

[0093] In this embodiment, when a current aggregated DMA descriptor chain transfer completion event is detected, the system does not immediately send an interrupt notification to the CPU. Instead, it performs predictive evaluation and suppression control according to the following two-level decision steps: The first level of determination involves immediate retrieval of the queue head status. The system first searches for the head node of the queue to be executed. If the head node exists, meaning there is at least one aggregate descriptor chain waiting to be executed in the queue, the system further checks whether the head chain and the currently completed chain belong to the same logical context. The criteria for determining whether they belong to the same logical context can be any of the following conditions: they belong to the same receiving thread, they belong to the same business flow, they have the same transaction identifier, or their target addresses fall within the same contiguous address range. If the head chain and the current chain belong to the same logical context, the system directly determines to trigger interrupt suppression, masks the interrupt notification for the completion of this transmission, and immediately sends the transmission instruction of the head chain to the DMA controller, allowing the DMA to continuously execute the next transmission task without CPU intervention.

[0094] If the first-level judgment is not triggered, i.e., the execution queue is empty, then the second-level judgment, i.e., arrival interval trend prediction, is initiated. The system maintains a historical arrival time record array, which records the arrival times of the most recent transmission tasks. In this embodiment, the array has a length of 8, recording the arrival times of the most recent 8 transmission tasks. The system calculates the time interval between each pair of these 8 arrival times, and then calculates the average arrival time interval. The system compares this average arrival time interval with a preset interrupt delay threshold. The value of this interrupt delay threshold is approximately equal to the time required for the CPU to process one interrupt context switch; in this embodiment, it is set to 20 microseconds.

[0095] If the average arrival time interval is less than or equal to the interrupt delay threshold, the system determines that subsequent tasks have a high probability of arriving within a short period. Therefore, it starts a predictive wait timer with a timer length equal to the interrupt delay threshold and actively delays the triggering of the current interrupt. During the timer's operation, if a new transmission task arrives, the system cancels the timer and merges the interrupt for the current transmission completion with the interrupt for the next transmission completion into a single interrupt notification, which is triggered together upon the next transmission completion. If no new task arrives after the timer expires, the system determines that it is currently in a low-traffic idle period and there are no continuous high-density transmission tasks. It immediately triggers the interrupt notification for the current transmission completion and stops waiting.

[0096] If the average arrival time interval is greater than the interrupt delay threshold, the system will immediately trigger the current interrupt notification without waiting.

[0097] The above mechanism introduces the following handling for interrupt notification delays: When the system is in a high-throughput burst period, multiple consecutive transmission completion events are merged into one interrupt trigger, significantly reducing the number of CPU interrupt responses and effectively suppressing interrupt storms; When the system is in a low-traffic idle period, the system does not introduce additional waiting delays, and triggers an interrupt promptly after each transmission is completed, ensuring the real-time performance of data processing.

[0098] To prevent data reception timeouts caused by indefinite interruption delays under extremely high load scenarios, the system also sets up dual protection limits. The first layer of protection is the maximum threshold for the number of consecutive interrupt suppressions, which is set to 3 times in this embodiment. This means the system can suppress a maximum of 3 consecutive interrupts. After reaching this number, regardless of the prediction result, a merged interrupt notification will be forcibly triggered. The second layer of protection is the maximum total delay threshold, which is set to 100 microseconds in this embodiment. This threshold accumulates from the first interrupt suppression; if the total delay reaches 100 microseconds, an interrupt notification will be forcibly triggered and the suppression count will be reset. Through these dual protections, the system ensures both the real-time performance and reliability of data transmission while pursuing interrupt merging efficiency.

[0099] Step S208: After the direct memory access operation is completed, according to the records of the aggregated DMA descriptor chain, the scattered buffer blocks occupied by this transfer are located in batches, a lazy reclamation operation is performed, and they are temporarily kept in the local cache.

[0100] In this embodiment, after the DMA data transfer is completed, the system does not immediately return the distributed buffer blocks occupied by this transfer to the global distributed data buffer pool, but instead executes a lazy reclamation mechanism. The specific working process of this mechanism is as follows: The system first locates all buffer blocks involved in this transfer. The location method is as follows: directly read the source address field of each descriptor unit in the aggregated DMA descriptor chain constructed in step S104, and extract the physical addresses of all buffer blocks used in this transfer from these source address fields. Since the descriptor chain has recorded the addresses of all data blocks during construction, the system does not need to re-traverse the buffer pool for searching, but directly extracts them in batches from the descriptor chain, thereby reducing the time complexity of the location operation to a single traversal proportional to the number of descriptors.

[0101] After location is complete, the system marks these buffer blocks as "hot cached". "Hot cached" means that the system updates the status flag to "locally available" in the control header structure of the scattered buffer block and attaches the memory pointer of that buffer block to the local free list exclusively controlled by the current CPU core. This local free list is a data structure accessible only to the current CPU core; other CPU cores cannot allocate or release buffers from this list, therefore, operations on this list do not require any global mutex locks.

[0102] Buffer blocks in a hot cache state have the following two functions: The first benefit is the elimination of global lock contention. When the current CPU core generates a new data block and needs to allocate a buffer, the system prioritizes popping a buffer block in a hot cache state from the local free list for allocation. Since the local free list is exclusively owned by the current CPU core, the allocation operation does not require requesting a mutex lock from the globally distributed data buffer pool, thus completely eliminating the overhead of lock acquisition and release in multi-core concurrent allocation scenarios.

[0103] The second benefit is improving access performance by leveraging CPU cache locality. A buffer block in a hot cache state retains its physical memory address in the current CPU core's L1 and L2 data caches; that is, the cache line still contains the data recently written to that buffer. When this buffer is reallocated to a new data block, the new data write operation will directly hit the CPU cache, avoiding the tens to hundreds of nanoseconds of access latency introduced by loading data from main memory due to cache misses. This effect is particularly significant for scenarios involving frequent read and write operations of small data sizes.

[0104] The system also sets a capacity limit for the local free list of each CPU core. In this embodiment, the capacity limit is set to 32 buffer blocks. The system monitors the number of buffer blocks currently mounted in the local free list of each CPU core in real time. When this number reaches the capacity limit, the system triggers a batch return operation: clearing the "locally available" flag of the buffer blocks in the local free list that exceed the capacity limit, and returning the pointers of these buffer blocks in batches to the global distributed data buffer pool. By setting a capacity limit, the system utilizes the advantage of cache locality while avoiding the problem of a single CPU core excessively occupying buffer resources, leading to insufficient resources for other cores.

[0105] The lazy garbage collection mechanism in this embodiment differs fundamentally from existing immediate garbage collection schemes in terms of operation frequency: immediate garbage collection requires a global lock operation after each transfer, while this embodiment only performs a batch global return when the local free list accumulates 32 buffer blocks, reducing the lock operation frequency to approximately one-thirty-second of the traditional scheme. Simultaneously, because local allocation hits the hot cache, memory allocation latency is significantly reduced, and cache hit rate is significantly improved.

[0106] Please see Figure 3 An adaptive DMA aggregation transfer system for fragmented data includes: The data acquisition module 301 is used to acquire metadata, which is associated with the data block to be transmitted stored in the distributed data buffer pool. The metadata includes at least the storage address and attribute information of the data block to be transmitted. The strategy evaluation module 302 is used to evaluate the preset dynamic aggregation strategy based on storage address and attribute information. The dynamic aggregation strategy defines the time window conditions, capacity threshold conditions or logical transaction conditions that trigger the aggregation operation. The association identification module 303 is used to filter target data blocks from metadata based on the evaluation results; Chain building module 304 is used to allocate descriptor units in the aggregated DMA descriptor chain pool, write and link the metadata corresponding to the target data block in logical order, so as to build and generate an aggregated DMA descriptor chain. The transmission scheduling module 305 is used to generate transmission instructions based on the aggregated DMA descriptor chain and trigger direct memory access operations so that target data blocks scattered in the distributed data buffer pool are transmitted in batches to the target address.

[0107] The strategy evaluation module 302 also includes a resource monitoring submodule and a strategy dynamic adjustment submodule. The resource monitoring submodule monitors the idle resource ratio of the distributed data buffer pool in real time. The strategy dynamic adjustment submodule dynamically adjusts the dynamic aggregation strategy according to a segmented step algorithm when the idle resource ratio falls below a preset low watermark. The segmented step algorithm includes: when the idle resource ratio is in a first interval, reducing the capacity threshold condition or shortening the time window condition by a first decay ratio; when the idle resource ratio is in a second interval, reducing the capacity threshold condition or shortening the time window condition by a second decay ratio, wherein the idle resource ratio in the second interval is lower than that in the first interval, and the second decay ratio is less than the first decay ratio.

[0108] The association identification module 303 includes a feature extraction submodule, a hash matching submodule, and a logical orchestration submodule. The feature extraction submodule extracts transaction identifiers or data type tags from the metadata. The hash matching submodule uses a two-level hash index structure to find multiple data blocks with the same transaction identifier or belonging to the same data type tag, and whose physical addresses are not contiguous in the distributed data buffer pool. The first-level hash index uses the transaction identifier as the key, and the second-level hash index uses the sequence number as the key. The logical orchestration submodule arranges the target data blocks in the logical order of aggregation and transmission using an ordered linked list, with the arrangement based on the sequence number of each data block in ascending order.

[0109] The transmission scheduling module 305 includes a quality of service (QoS) calculation submodule and a chain splitting submodule. The QoS calculation submodule calculates the credit value consumption corresponding to the aggregated DMA descriptor chain and determines the task submission order based on a priority heap structure, where the credit value consumption equals the total data volume of the aggregated chain divided by the data volume represented by a unit credit value. The chain splitting submodule logically splits the aggregated DMA descriptor chain into multiple subchains when the total data volume exceeds a preset single transaction threshold, and configures these subchains to the transmission interface in batches to trigger execution.

[0110] The system also includes an interrupt control module and a resource reclamation module. The interrupt control module, upon detecting a direct memory access operation's transfer completion event, queries the queue of pending aggregated DMA descriptor chains. If it predicts that a next aggregated DMA descriptor chain, immediately following the current chain and belonging to the same logical context, is about to become ready, it performs an interrupt suppression operation to proactively delay triggering the interrupt notification for the current transfer completion. It then merges and triggers an interrupt notification once the transfer operation corresponding to the next aggregated DMA descriptor chain is completed. The resource reclamation module, after a direct memory access operation completes, locates the scattered buffer blocks occupied by the current transfer in batches according to the records in the aggregated DMA descriptor chain. It performs a lazy reclamation operation, temporarily storing the buffer blocks in the local cache of the current CPU core. The lazy reclamation operation includes marking the scattered buffer blocks as hot cached and adding them to the local free list. The buffer blocks are only returned to the global distributed data buffer pool in batches when the local cache reaches its capacity limit.

[0111] Based on the same inventive concept, this application also provides an electronic device. For example... Figure 4 As shown, this application provides an electronic device, including: one or more processors, a memory, and one or more communication interfaces.

[0112] One or more computer programs are stored in the memory, and when the one or more computer programs are executed by the one or more processors, the one or more processors implement the adaptive DMA aggregation transfer method for fragmented data as described above.

[0113] The communication interface is connected between the processor and memory, or to an external network or external device, and is configured to enable information interaction and data transmission between the processor and memory, and between electronic devices and external devices.

[0114] The processor is a device with data processing capabilities, which may include, but is not limited to, a central processing unit, a network processor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The memory is a device with data storage capabilities, including but not limited to random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EPR), and flash memory.

[0115] The communication interface connects the processor and the memory, enabling information exchange between the processor and the memory. Its bus type includes, but is not limited to, peripheral interconnect standard bus, system bus, etc.

[0116] The processor, memory, and communication interface are interconnected via a bus, which in turn connects to other components of the computing device.

[0117] The one or more processors can be implemented as a hardware offload engine, dedicated to performing dynamic aggregation strategy evaluation and descriptor chain construction steps, thereby offloading the computational load from the main CPU. In such embodiments, computationally intensive operations such as aggregation decisions and descriptor chain construction are performed independently by the offload engine, while the main CPU is only responsible for submitting metadata and triggering the final transmission instructions, further reducing the load on the main CPU.

[0118] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of an adaptive DMA aggregation transfer method for fragmented data.

[0119] Computer-readable storage media can include various media that can store program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of an adaptive DMA aggregation transfer method for fragmented data.

[0121] This embodiment explains the linkage relationship and systematic optimization effect between the steps in the above embodiments from the perspective of overall synergy.

[0122] The steps described above in this application are not simply an isolated combination of technical means, but rather form an organic whole through the following synergistic relationship: The first layer of collaboration involves decoupling the data plane from the control plane. Step S101 decouples data block production from DMA transfer control through a metadata mechanism—data producers only need to submit metadata to the intermediate queue to continue production, without waiting for DMA transfer to complete. This allows the aggregation decision-making process in steps S102 to S103 to proceed independently without blocking data production, achieving asynchronous matching between producers and consumers in the time dimension. The collaborative effect of this decoupling is that regardless of fluctuations in the data production rate, the aggregation decision layer can independently perform batch processing with the optimal strategy, without causing DMA channels to be idle or overloaded due to instantaneous changes in the production rate.

[0123] The second layer of collaboration involves adapting aggregation decisions to hardware execution constraints. In step S102, during the evaluation of the capacity threshold condition, it simultaneously checks whether the physical storage address of the data block to be transferred violates the DMA controller's page boundary limits or alignment requirements. This design ensures that each descriptor in the aggregation descriptor chain constructed in step S104 can be correctly executed by the DMA hardware when submitted, without being rejected or split by the hardware due to address alignment issues. In traditional schemes, software-level aggregation decisions and hardware-level execution constraints are separate—software aggregates according to logic, while the hardware may re-disassemble the aggregated chain due to address issues during execution, causing the software aggregation effort to be negated. This collaborative design of the present invention completes the adaptation of software decisions to hardware constraints from the aggregation stage, avoiding the aforementioned performance-degrading cycle of "aggregation-splitting-re-aggregation."

[0124] The third layer of collaboration is an integrated pipeline for searching, sorting, and chain construction. The two-level hash index in step S204 not only completes the search and grouping of the target data block, but the sequence number information recorded in its second-level hash table also serves the ordered linked list sorting in step S204. The sorted linked list then directly serves as the basis for the filling order of the descriptor chain in step S104. These three elements form a pipeline-like linkage of "searching equals grouping, sorting equals chain construction"—the data block is traversed only once throughout the entire process, instead of the multiple traversals of searching once, sorting once, and descriptor construction once in traditional schemes. This integrated design optimizes the time overhead from three traversals to a single traversal.

[0125] The fourth layer of coordination is a closed-loop feedback between transmission scheduling and interrupt suppression. The credit value scheduling and priority heap structure in step S206 determine the submission order of each aggregation chain to the DMA controller, which directly affects the fullness of the queue to be executed. The first-level determination (queue head search) of the interrupt suppression mechanism in step S207 is based on the state of the queue to be executed—interrupts are suppressed when there are subsequent tasks in the queue, and a second-level prediction is attempted when the queue is empty. This forms a closed loop of forward scheduling and backward feedback: the scheduling policy affects the queue state, the queue state in turn affects the interrupt suppression decision, and the effect of interrupt suppression affects the CPU's efficiency in processing subsequent tasks, thus affecting the enqueue rate of new tasks. This closed loop enables the system to achieve an adaptive dynamic balance between transmission efficiency and interrupt response.

[0126] The fifth layer of coordination involves pipelined parallelism between data transfer completion and resource reclamation. The DMA transfer operation in step S105 and the lazy buffer reclamation in step S208 are decoupled in time—after the DMA transfer is completed, the buffer is not immediately returned to the global pool, but is retained in the local free list of the current CPU core as a hot cache. This allows the next transfer in step S105 to start immediately, without waiting for the reclamation operation in step S208 to complete. Simultaneously, the reclamation operation in step S208 only needs to update the local list and does not require acquiring a global lock, thus not blocking transfer operations on other cores. The pipelined parallelism of transfer and reclamation further improves the overall system throughput.

[0127] The sixth level of collaboration is a state machine-driven multi-dimensional strategy fusion. The multi-level state machine in step S102 transforms the time window condition, capacity threshold condition, and logical transaction condition from three isolated parallel conditions into an organic decision-making system with priority hierarchy and state dependencies: in the idle state, the capacity condition is bypassed to avoid wasting small-size transmissions; in the accumulation state, the three conditions work together and allow logical transactions to be interrupted while waiting; in the emergency state, the capacity condition is bypassed again to pursue maximum release speed. This state-based adaptive decision-making enables the system to consistently select the optimal aggregation strategy across the entire load range, from idle to light load, from light load to heavy load, and from heavy load to overload.

[0128] In summary, the seven steps of this invention, through synergistic effects, aggregate scattered fragmented data blocks at the logical level, reorganize them at the sorting level, link them at the descriptor level, optimize them at the scheduling level, and reuse them at the resource level, forming a full-link collaborative optimization system from data generation to hardware transmission completion. Its overall technical effect is far beyond what can be achieved by simply adding up the technical effects of each step.

[0129] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. An adaptive DMA aggregation transfer method for fragmented data, characterized in that, Includes the following steps: Obtain metadata, which is associated with the data block to be transmitted stored in the distributed data buffer pool. The metadata includes at least the storage address and attribute information of the data block to be transmitted. The preset dynamic aggregation strategy is evaluated based on storage address and attribute information. The dynamic aggregation strategy defines the time window conditions, capacity threshold conditions or logical transaction conditions that trigger the aggregation operation. Based on the evaluation results, target data blocks are selected from the metadata; Descriptor units are allocated in the aggregated DMA descriptor chain pool, and the metadata corresponding to the target data block is written and linked in logical order to build and generate the aggregated DMA descriptor chain. Based on the aggregated DMA descriptor chain, transfer instructions are generated and direct memory access operations are triggered to enable the batch transfer of target data blocks scattered in the distributed data buffer pool to the target address.

2. The adaptive DMA aggregation transfer method for fragmented data according to claim 1, characterized in that, The evaluation of the preset dynamic aggregation strategy based on storage address and attribute information includes: The multi-level state machine coordinates the determination of time window conditions, capacity threshold conditions and logical transaction conditions. The multi-level state machine includes at least idle state, cumulative state, emergency state and transaction ready state. In the idle state, only the time window condition takes effect, and the capacity threshold condition is bypassed; in the cumulative state, the time window condition and the capacity threshold condition take effect in parallel, and the logical transaction condition has priority to interrupt; in the emergency state, the capacity threshold condition is bypassed, and aggregation is triggered as soon as the queue is not empty. The formula for determining the time window condition is: ,when The time condition is met, among which The write timestamp of the earliest data block in the queue. The current system time at the time of this assessment. This is the preset maximum waiting time; The capacity threshold condition is determined by: accumulating the data lengths of all data blocks to be transmitted in the queue. ,when The condition is met when the storage addresses of each data block do not violate the physical page boundary limits or alignment requirements of the DMA controller. This is a preset capacity threshold; The determination of logical transaction conditions includes: the condition is met when there is a set of data blocks in the queue with the same transaction identifier and containing a transaction end tag, or when there is a data block carrying a high priority tag.

3. The adaptive DMA aggregation transfer method for fragmented data according to claim 2, characterized in that, After obtaining the metadata, the process also includes: Real-time monitoring of the percentage of idle resources in the distributed data buffer pool ; when Below the preset low water level At that time, the dynamic aggregation strategy is adjusted according to the segmented step algorithm: when At that time, according to the first attenuation ratio Adjustment: , ;in This is the low water level. The adjusted capacity threshold, This is the adjusted time window value; when At that time, according to the second attenuation ratio Adjustment: , ; in , , This is the baseline value for the capacity threshold. This serves as the baseline value for the time window. when At that time, restore , .

4. The adaptive DMA aggregation transfer method for fragmented data according to claim 1, characterized in that, The process of filtering target data blocks from metadata based on evaluation results includes: Extract the transaction identifier from the metadata; Matching is performed using a two-level hash index structure. The first-level hash index uses the transaction identifier as the key, and the second-level hash index uses the sequence number as the key. Multiple data blocks with the same transaction identifier but non-contiguous physical addresses in the distributed data buffer pool are identified as target data blocks. An ordered linked list is constructed based on the sequence number of the target data block, and the blocks are arranged in ascending order of sequence number as the logical order of aggregation and transmission. When some data blocks of the same transaction have not arrived, the consecutive arrived data blocks starting from sequence number 0 will be constructed into an aggregate descriptor chain for transmission, and the unarrived data blocks will be kept in the two-level hash index to wait for the next round of aggregation window processing.

5. The adaptive DMA aggregation transfer method for fragmented data according to claim 1, characterized in that, After generating transfer instructions based on the aggregated DMA descriptor chain and triggering direct memory access operations, the process further includes: Upon detecting a transmission completion event, a two-level interrupt suppression determination is performed: Level 1 Decision: Retrieve the head node of the queue to be executed. If the head node exists and belongs to the same logical context, suppress the current interrupt notification and continuously send the transmission instructions to the head node. Second-level decision: If the queue to be executed is empty, calculate the average arrival time interval of the most recent N transmission tasks. ,like If a predictive wait timer is started and the interrupt notification is delayed, the interrupt notification is merged if a new task arrives before the timer expires; otherwise, an interrupt notification is triggered. As an interrupt delay threshold, This represents the time difference between two consecutive transmission tasks arriving at the system. Additionally, it sets the maximum number of consecutive interruption suppression thresholds MAX_SUPPRESS_COUNT and the maximum total delay threshold MAX_SUPPRESS_DURATION. When either threshold is reached, the merged interrupt notification is forcibly triggered and the suppression count is reset.

6. The adaptive DMA aggregation transfer method for fragmented data according to claim 1, characterized in that, The process of generating transfer instructions and triggering direct memory access operations based on the aggregated DMA descriptor chain includes: Calculate the credit consumption of the aggregated DMA descriptor chain. ,in The total data size of the aggregate DMA descriptor chain. Credit value per unit; Calculate your current available credit score ,in This represents the maximum credit limit for the current business flow. The sum of credits consumed by submitted but incomplete transfers; like The aggregated DMA descriptor chain is inserted into the priority heap, which is sorted by the priority of the chain. Each chain is submitted to the DMA controller for execution according to the pop order of the priority heap.

7. The adaptive DMA aggregation transfer method for fragmented data according to claim 6, characterized in that, The step of submitting each chain to the DMA controller for execution according to the priority heap popping order also includes: like Then the aggregated DMA descriptor chain will be split into multiple sub-chains, and the data volume of each sub-chain will not exceed MAX_TX_PER_BURST, where MAX_TX_PER_BURST is a preset single transaction threshold. Multiple subchains are configured to the transport interface in batches, and other chains in the priority heap are allowed to jump the queue during the transport intervals between adjacent subchains.

8. The adaptive DMA aggregation transfer method for fragmented data according to claim 1, characterized in that, After generating transfer instructions based on the aggregated DMA descriptor chain and triggering direct memory access operations, the process further includes: Based on the source address field of each descriptor unit in the aggregated DMA descriptor chain, locate all the scattered buffer blocks occupied by this transfer in batches; Update the status flag of the scatter buffer block to be locally available, and attach the memory pointer of the scatter buffer block to the local free list exclusively used by the current CPU core. When the current CPU core requests a buffer, it will prioritize allocating a buffer block from the local free list. When the number of buffer blocks in the local free list reaches the preset capacity limit MAX_LOCAL_CACHE, the excess buffer blocks are returned in batches to the global distributed data buffer pool.

9. The adaptive DMA aggregation transfer method for fragmented data according to claim 4, characterized in that, The step of constructing an ordered linked list based on the sequence number of the target data block further includes: The second-level hash index records the arrangement status of each sequence number; When subsequent data blocks of the same transaction arrive, the second-level hash index is queried to check whether the sequence number of the data block has been marked as orchestrated. If it has been marked, it is determined to be a duplicate data block and deduplication is triggered; If it is not marked, it is inserted into the corresponding position of the ordered linked list according to the sequence number of the data block, and marked as arranged in the second-level hash index.

10. An adaptive DMA aggregation transfer system for fragmented data, characterized in that, Implementing the adaptive DMA aggregation transfer method for fragmented data as described in any one of claims 1-9, comprising: The data acquisition module is used to acquire metadata, which is associated with the data block to be transmitted stored in the distributed data buffer pool. The metadata includes at least the storage address and attribute information of the data block to be transmitted. The strategy evaluation module is used to evaluate the preset dynamic aggregation strategy based on storage address and attribute information. The dynamic aggregation strategy defines the time window conditions, capacity threshold conditions or logical transaction conditions that trigger the aggregation operation. The association identification module is used to filter target data blocks from metadata based on the evaluation results; The chain building module is used to allocate descriptor units in the aggregated DMA descriptor chain pool, write and link the metadata corresponding to the target data block in logical order, so as to build and generate an aggregated DMA descriptor chain. The transmission scheduling module is used to generate transmission instructions based on the aggregated DMA descriptor chain and trigger direct memory access operations so that target data blocks scattered in the distributed data buffer pool are transmitted in batches to the target address.

Images