Timer adjusting method and system based on DPDK dynamic load awareness

Abstract

The invention relates to a timer adjusting method and system based on DPDK dynamic load awareness, and the method comprises the steps: calculating a first load rate of a first buffer area and a second load rate of a second buffer area when a timer triggers a pre-constructed collection thread according to a triggering period, and adjusting the first load rate of the first buffer area and the second load rate of the second buffer area based on the first load rate and the second load rate; adjusting a trigger period through a disk scanning period adjusting rule, and updating a timer based on the trigger period; and scanning the target collection directory, and writing the scanned file into the first buffer area. The step of executing the pre-constructed processing thread comprises the steps of obtaining data from the first buffer area and processing the data, and storing a processing result in the second buffer area. And based on the updated timer, circularly executing the acquisition thread and the processing thread, and balancing the data acquisition rate and the data processing rate. The problem of how to improve the resource utilization efficiency while keeping the balance between file collection and file processing in the prior art is solved.

Description

A timer adjustment method and system based on DPDK dynamic load awareness Technical Field

[0001] This application relates to the field of data acquisition and processing technology, and in particular to a timer adjustment method and system based on DPDK dynamic load sensing. Background Technology

[0002] In computer systems, file acquisition and processing is a common data processing method. The file acquisition end is typically responsible for scanning the target directory, discovering new files, and adding them to the task queue; the file processing end then retrieves files from the queue and performs operations such as parsing, analysis, or dumping. To ensure that files enter the processing flow in a timely manner, existing technologies generally adopt a fixed-period timed disk scanning mechanism. A timer triggers the file acquisition thread to scan the directory, writes the scan results to the task queue, and then the file processing thread retrieves files from the queue for processing.

[0003] The existing patent publication number CN111208985B describes a data processing method, system, and storage medium based on a producer-consumer model. This method optimizes the data processing flow through a sentinel consumer polling mechanism and multi-consumer collaborative work. The sentinel consumer polls for a globally available processing quantity. If there is, the sentinel consumer wakes up other consumers, who then process the data in their respective data queues until the globally available processing quantity falls below a threshold. At this point, other consumers enter a dormant state, and the sentinel consumer returns to the polling step. After the producer generates data, it distributes it evenly to the data queues corresponding to each consumer. This patent monitors the data generated by the producer through the sentinel consumer polling mechanism and improves data processing performance when there is a large amount of producer data by increasing the number of consumers, thus improving parallel data processing capabilities to some extent. However, some shortcomings remain: First, while the polling mechanism can monitor the data processing volume in real time, it may incur performance overhead, especially when a large number of consumers are involved, where polling itself may become a performance bottleneck. Second, this patent is geared towards multi-consumer scenarios and does not offer an effective solution for scenarios where the producer produces a large amount of data in a single-consumer setting. Finally, the sentinel consumer mechanism is highly sensitive to changes in the number of consumers. If there are large fluctuations in the number of consumers, this mechanism may not be able to adapt to real-time changes in load demand. When faced with highly dynamic and unpredictable workloads, it may not be able to quickly adjust its processing strategy.

[0004] Therefore, no effective solution has yet been proposed for adjusting the dynamic load timer in file processing to maintain a balance between file acquisition and file processing while improving resource utilization efficiency. Summary of the Invention

[0005] This application provides a timer adjustment method and system based on DPDK dynamic load awareness, which at least solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0006] In a first aspect, embodiments of this application provide a timer adjustment method based on DPDK dynamic load awareness. The method includes: when a timer triggers a pre-built acquisition thread according to a trigger cycle, calculating a first load rate of a first buffer and a second load rate of a second buffer; adjusting the trigger cycle based on the first load rate and the second load rate using a disk scan cycle adjustment rule; and updating the timer based on the trigger cycle; scanning a target acquisition directory and writing the scanned files into the first buffer; executing a pre-built processing thread, including acquiring and processing data from the first buffer and storing the processing result in the second buffer; and cyclically executing the acquisition thread and the processing thread based on the updated timer to balance the data acquisition rate and the data processing rate.

[0007] In one embodiment, before the timer triggers the pre-built acquisition thread according to the trigger cycle, the method further includes: creating an acquisition thread and a processing thread for the file; creating a pending lock-free circular queue based on the DPDK framework as a first buffer, wherein the first buffer is a buffer of files to be processed; and creating a complete lock-free circular queue based on the DPDK framework as a second buffer, wherein the second buffer is a buffer of files that have been processed.

[0008] In one embodiment, calculating the first load rate of the first buffer and the second load rate of the second buffer includes: obtaining the queue occupancy rate of the first buffer based on the ratio of the number of files to be processed in the first buffer to the total capacity of the queue; obtaining the queue backlog rate of the second buffer based on the ratio of the number of processed files in the second buffer to the total capacity of the queue; using the queue occupancy rate as the first load rate of the first buffer and the queue backlog rate as the second load rate of the second buffer.

[0009] In one embodiment, adjusting the trigger cycle based on the first load rate and the second load rate using a disk sweep cycle adjustment rule includes: setting an idle threshold and a first overload threshold for the first buffer, setting a second overload threshold for the second buffer; and adjusting the trigger cycle to shorten, maintain, or extend based on a comparison between the first load rate and the idle threshold and the first overload threshold, and a comparison between the second load rate and the second overload threshold.

[0010] In one embodiment, adjusting the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, through a disk sweep cycle adjustment rule, includes: determining the adjustment state of the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold; and based on the adjustment state, shortening, maintaining, or extending the trigger cycle through a corresponding adjustment rule.

[0011] Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the adjustment state of the triggering cycle is determined; based on the adjustment state, the triggering cycle is shortened, maintained, or extended according to the corresponding adjustment rules.

[0012] In one embodiment, determining the state of the trigger cycle adjustment based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, includes: if the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the trigger cycle is determined to be a first state; if the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be a second state; if the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the trigger cycle is determined to be a first state; if the first load rate is greater than the first overload threshold, then the adjustment state of the trigger cycle is determined to be a third state; if the first load rate is equal to a preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be a first state.

[0013] In one embodiment, the step of shortening, maintaining, or extending the triggering period based on the adjustment state and using corresponding adjustment rules includes: if the adjustment state of the triggering period is the first state, then maintaining the previous triggering period; if the adjustment state of the triggering period is the second state, then multiplying the previous triggering period by a first adjustment coefficient to obtain the shortened next triggering period; if the adjustment state of the triggering period is the third state, then multiplying the previous triggering period by a second adjustment coefficient to obtain the extended next triggering period.

[0014] In one embodiment, the step of obtaining and processing data from the first buffer and storing the processing result in the second buffer includes: obtaining data to be processed from the first buffer according to a preset interface; performing parsing and dumping operations on the data to be processed to obtain the processing result; and storing the result in the second buffer.

[0015] In one embodiment, the step of cyclically executing the acquisition thread and the processing thread to balance the data acquisition rate and the data processing rate includes: the timer being configured to periodically trigger the acquisition thread according to the updated trigger period; the acquisition thread, in response to receiving trigger information, performing calculations of the first load rate and the second load rate, and updating the trigger period and the timer based on the first load rate and the second load rate through the disk scanning cycle adjustment rule; the processing thread being configured to execute concurrently with the acquisition thread, performing operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer; and balancing the data acquisition rate and the data processing rate by cyclically executing the acquisition thread and the processing thread and adjusting the trigger period based on the first load rate and the second load rate.

[0016] Secondly, embodiments of this application provide a timer adjustment system based on DPDK dynamic load awareness. The system is used to execute the method described above. The system includes an execution acquisition thread module, an execution processing thread module, and a loop module. Specifically: the execution acquisition thread module is used to calculate a first load rate of a first buffer and a second load rate of a second buffer when the timer triggers a pre-built acquisition thread according to a trigger cycle; adjust the trigger cycle based on the first load rate and the second load rate using a disk scan cycle adjustment rule; and update the timer based on the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer; the execution processing thread module is used to execute a pre-built processing thread, including acquiring and processing data from the first buffer and storing the processing result in the second buffer; the loop module is used to cyclically execute the acquisition thread and the processing thread based on the updated timer, balancing the data acquisition rate and the data processing rate.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a timer adjustment based on DPDK dynamic load awareness as described in the first aspect above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a timer adjustment method based on DPDK dynamic load awareness as described in the first aspect above.

[0019] The timer adjustment method and system based on DPDK dynamic load awareness provided in this application embodiment have at least the following technical effects.

[0020] When the timer triggers the pre-built acquisition thread according to the trigger cycle, the first load rate of the first buffer and the second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. The pre-built processing thread is executed, including acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and the processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, so that the file acquisition cycle can adapt to the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing. This improves the system's stability, real-time performance of file acquisition and processing, and system resource utilization efficiency, solving the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0021] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and constitute a part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: FIG1 is a flowchart of a timer adjustment method based on DPDK dynamic load awareness; FIG2 is a structural schematic diagram of resource initialization according to an exemplary embodiment; FIG3 is a structural schematic diagram of adjusting the trigger period according to an exemplary embodiment; FIG4 is a flowchart of adjustment step S102 according to an exemplary embodiment; FIG5 is a system structure block diagram of a timer adjustment system based on DPDK dynamic load awareness according to an exemplary embodiment; FIG6 is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0024] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0025] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0026] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0027] In computer systems, file acquisition and processing is a common data processing method. The file acquisition end is typically responsible for scanning the target directory, discovering new files, and adding them to the task queue; the file processing end then retrieves files from the queue and performs operations such as parsing, analysis, or dumping. To ensure that files enter the processing flow in a timely manner, existing technologies generally adopt a fixed-period timed disk scanning mechanism. A timer triggers the file acquisition thread to scan the directory, writes the scan results to the task queue, and then the file processing thread retrieves files from the queue for processing.

[0028] However, the fixed-period disk scanning mechanism has many shortcomings in practical applications: 1. Static mismatch between acquisition and processing capabilities: When the file acquisition thread generates files at a rate that far exceeds the processing speed of the file processing thread during peak periods, the task queue is prone to continuous accumulation, which may lead to queue overflow or delayed processing of some files in severe cases; conversely, when the long-term processing capability is strong and few new files are generated in the directory, the queue will be idle for a long time, resulting in waste of resources.

[0029] 2. Unpredictable Dynamic Fluctuations in File Growth Rate: In scenarios such as log collection, batch file generation, and monitoring data storage, the file generation rate often fluctuates significantly over time, exhibiting unpredictability. Fixed-period disk scanning mechanisms cannot keep up with these changes in real time. When file growth surges, it can easily lead to a sudden backlog of disk directory files and queues, while when growth slows down, system resources become idle.

[0030] 3. Balancing efficiency and resource consumption is difficult when setting a fixed scan cycle. The scan cycle typically needs to be manually set, but its length is difficult to control, often resulting in a trade-off between acquisition and processing efficiency and resource usage. Setting a short scan cycle to reduce the acquisition delay of new files can lead to frequent invalid scans when few new files are added, consuming CPU and I / O resources. Conversely, setting a long cycle can prevent new files from being detected in a timely manner when many new files are added, affecting real-time processing.

[0031] Some optimization methods attempt to alleviate the above problems by increasing the number of file acquisition and processing threads or expanding the queue capacity, but these methods cannot fundamentally solve the contradiction between the acquisition rate and the processing rate.

[0032] Therefore, there is an urgent need for a timer control method that can automatically adjust the disk scanning cycle according to the file acquisition and processing load, so as to maintain a dynamic balance between file acquisition and file processing, thereby improving the system's stability and file acquisition and processing efficiency.

[0033] The existing patent publication number CN118018494A discloses a data processing method, system, and storage medium based on a producer-consumer model. This patent optimizes the data processing flow through a sentinel consumer polling mechanism and multi-consumer collaborative work. The sentinel consumer polls for a globally available processing quantity. If a globally available quantity exists, the sentinel consumer wakes up other consumers, who then process the data in their respective data queues until the globally available processing quantity falls below a threshold. At this point, other consumers enter a sleep state, and the sentinel consumer returns to the polling step. After the producer generates data, it is distributed evenly to the data queues corresponding to each consumer.

[0034] First, while polling can monitor data processing volume in real time, it may incur performance overhead, especially when a large number of consumers are involved, where polling itself can become a performance bottleneck. Second, this patent addresses multi-consumer scenarios and does not offer an effective solution for single-consumer scenarios where producers generate large amounts of data. Finally, the sentinel consumer mechanism is highly sensitive to changes in the number of consumers. If there are significant fluctuations in the number of consumers, this mechanism may not be suitable for real-time changes in load demands. It may also be unable to quickly adjust processing strategies when facing highly dynamic and unpredictable workloads.

[0035] In summary, the dynamic load timer adjustment method for file processing aims to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0036] In this document, it should be understood that the terms used may be technical means used to implement part of the present invention or other summary technical terms. For example, the term may include: DPDK: (Data Plane Development Kit), a set of user-space open-source software libraries and driver frameworks for fast packet processing.

[0037] Scan cycle adjustment rules: A set of predefined decision logic that dynamically calculates the next file directory scan interval based on real-time load status. The core inputs are the occupancy rate of the first buffer (pending queue) and the backlog rate of the second buffer (complete queue). By comparing these with preset thresholds, the system load status (such as idle, normal, overload) is determined, and then a new trigger cycle (shortened, maintained, or extended) is non-linearly decided.

[0038] Data acquisition rate: The number of new files that the file acquisition thread discovers and stores in the first buffer (pending queue) per unit of time.

[0039] Data processing rate: The number of files that a file processing thread retrieves from the first buffer (pending queue) and completes processing (such as parsing and dumping) per unit of time.

[0040] Pending lock-free circular queue: A first-in-first-out (FIFO) circular buffer implemented based on DPDK, serving as the first buffer and a buffer for files to be processed. The acquisition thread writes newly discovered files into this queue, and the processing thread retrieves files from this queue for processing.

[0041] Complete lock-free circular queue: A circular buffer implemented based on DPDK, serving as a secondary buffer and a buffer for processed files. Processing threads place successfully processed files into this queue.

[0042] In a first aspect, embodiments of this application provide a timer adjustment method based on DPDK dynamic load awareness. Figure 1 is a flowchart of a timer adjustment method based on DPDK dynamic load awareness. As shown in Figure 1, the method includes: step S101, when the timer triggers a pre-built acquisition thread according to the trigger cycle, calculating the first load rate of the first buffer and the second load rate of the second buffer, adjusting the trigger cycle according to the first load rate and the second load rate through the disk scan cycle adjustment rule, and updating the timer according to the trigger cycle; scanning the target acquisition directory and writing the scanned files into the first buffer.

[0043] Step S102, executing the pre-built processing thread includes retrieving data from the first buffer and processing it, and storing the processing result in the second buffer.

[0044] Step S103: Based on the updated timer, the acquisition thread and processing thread are executed in a loop to balance the data acquisition rate and the data processing rate.

[0045] In summary, this application provides a timer adjustment method based on DPDK dynamic load awareness. When the timer triggers a pre-built acquisition thread according to the trigger cycle, a first load rate of the first buffer and a second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time performance of file acquisition and processing, and system resource utilization efficiency. This solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0046] In one embodiment, before the timer triggers the pre-built acquisition thread according to the trigger cycle in step S101, FIG2 is a schematic diagram of the resource initialization structure according to an exemplary embodiment. As shown in FIG2, the method further includes: creating an acquisition thread and a processing thread for the file; creating a pending lock-free circular queue as a first buffer according to the DPDK framework, wherein the first buffer is a buffer of files to be processed; and creating a complete lock-free circular queue as a second buffer according to the DPDK framework, wherein the second buffer is a buffer of files that have been processed.

[0047] Optionally, a lock-free circular queue is created using the `rte_ring_create()` interface, and a DPDK timer is initialized. The DPDK timer uses SINGLE mode to control the periodic triggering of the file acquisition thread. Creating independent acquisition and processing threads decouples tasks and enables parallel execution, thereby improving the overall system throughput. Based on the DPDK framework, lock-free circular queues for pending and completed tasks are created as dual buffers, isolating pending and processed files respectively. This not only achieves secure and efficient data exchange between acquisition and processing but also avoids thread contention and context switching overhead through lock-free design. The occupancy rate of the pending queue and the backlog rate of the completed queue provide real-time and accurate feedback signals for subsequent dynamic load sensing, enabling the system to dynamically adjust the acquisition frequency based on the actual queue status. This fundamentally improves resource utilization, system stability, and real-time processing performance, and allows for dynamic adjustment of the timer period. Compared to a fixed-period disk scanning mechanism, this approach can adapt to the actual system load in real time, avoiding queue backlog during peak periods and frequent idling during low-load periods.

[0048] In one embodiment, step S101 calculates the first load rate of the first buffer and the second load rate of the second buffer, specifically including the following steps: Step 1: Obtain the queue occupancy rate of the first buffer based on the ratio of the number of files to be processed in the first buffer to the total capacity of the queue; Step 2: Obtain the queue backlog rate of the second buffer based on the ratio of the number of processed files in the second buffer to the total capacity of the queue; Step 3: Use the queue occupancy rate as the first load rate of the first buffer and the queue backlog rate as the second load rate of the second buffer.

[0049] Optionally, the queue occupancy rate is obtained by calculating the ratio of the number of files to be processed to the total queue capacity, and the queue backlog rate is obtained by calculating the ratio of the number of processed files to the total queue capacity. These two are used as the first load rate and the second load rate, respectively. This provides objective and real-time quantifiable metrics.

[0050] In one embodiment, FIG3 is a schematic diagram of the structure for adjusting the trigger cycle according to an exemplary embodiment. As shown in FIG3, step S101 adjusts the trigger cycle based on the first load rate and the second load rate through the disk sweep cycle adjustment rule, specifically including the following steps: Step 1, setting the idle threshold and the first overload threshold of the first buffer, and setting the second overload threshold of the second buffer.

[0051] Optionally, in this embodiment, the idle threshold P_low can be set to 0.3, the first overload threshold P_high can be set to 0.7, and the second overload threshold C_thr can be set to 0.5. That is: when the pending queue load rate is <0.3, the pending queue is considered idle; when the pending queue load rate is in the range of [0.3, 0.7], the pending queue load is considered normal; when the pending queue load rate is >0.7, the pending queue is considered overloaded; and when the complete queue load rate is >0.5, the complete queue is considered overloaded.

[0052] Step 2: Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, adjust the trigger cycle to shorten, maintain, or extend. Specifically, this includes: determining the adjustment state of the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold. This includes: if the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0053] Optionally, if the pending queue load rate is less than the threshold of 0.3 and the complete queue load rate is less than the threshold of 0.5, it indicates that the new file collection rate is insufficient and the file processing capacity is limited. The state of the trigger cycle is determined to be the first state, that is, the queue load is normal.

[0054] If the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be the second state.

[0055] Optionally, if the pending queue load rate is less than the threshold of 0.3 and the complete queue load rate is greater than or equal to the threshold of 0.5, it indicates that the new file acquisition rate is insufficient, but the file processing capability is strong. The adjustment state of the trigger cycle is determined to be the second state, that is, the queue load is idle.

[0056] If the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0057] Optionally, if the pending queue load rate is in the range of [0.3, 0.7], it indicates that the file acquisition and processing capabilities are in a balanced state, the system is running smoothly overall, and the state for determining the trigger cycle is the first state, that is, the queue load is normal.

[0058] If the first load rate is greater than the first overload threshold, the adjustment state of the trigger cycle is determined to be the third state.

[0059] Optionally, if the pending queue load rate is greater than or equal to the threshold of 0.7, it indicates that the file acquisition rate is too high and the file processing capacity is insufficient, and there is a risk of queue backlog. The state of the trigger cycle is determined to be the third state, that is, the queue overload state.

[0060] If the first load rate is equal to the preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0061] Optionally, if the pending queue load rate is approximately 0 and the complete queue load rate is approximately 0, it means that there are no files to process at this time, and the state of the trigger cycle is determined to be the first state, that is, the queue load is normal.

[0062] In one embodiment, based on the adjustment state, the trigger period is shortened, maintained, or extended according to the corresponding adjustment rules. This includes: if the adjustment state of the trigger period is the first state, then the previous trigger period is maintained; optionally, if the adjustment state of the trigger period is the queue load normal state, it indicates that the new file collection rate is insufficient and the file processing capacity is limited. In this case, the previous timer trigger period is maintained or the default timer trigger period when the program starts is set to avoid invalid frequent scanning.

[0063] If the trigger cycle adjustment state is the second state, the previous trigger cycle is multiplied by the first adjustment coefficient to obtain the shortened next trigger cycle. Optionally, if the trigger cycle adjustment state is the queue load idle state, it indicates that the new file acquisition rate is insufficient, but the file processing capability is strong. In this case, the timer trigger cycle is shortened, that is, the previous timer cycle is multiplied by 0.7 as the next timer trigger cycle to speed up the directory scan and ensure that the processing end can obtain the files to be processed in a timely manner.

[0064] If the adjustment state of the trigger cycle is the third state, then the previous trigger cycle is multiplied by the second adjustment coefficient to obtain the extended next trigger cycle.

[0065] Optionally, if the adjustment state of the trigger cycle is the queue overload state, it indicates that the file acquisition rate is too high and the file processing capacity is insufficient, and there is a risk of queue accumulation. In this case, the timer trigger cycle is extended, that is, the previous timer cycle is multiplied by 1.3 as the next timer trigger cycle, so as to reduce the acquisition frequency and avoid pending queue overflow.

[0066] The beneficial effects of adjusting the trigger cycle are as follows: Firstly, it improves system stability and real-time performance. When the rate of new file additions far exceeds the file processing capacity, the collection cycle is automatically extended to alleviate disk directory file and queue backlogs. Secondly, when file processing capacity is high, the collection cycle is shortened to accelerate file discovery. This maintains a dynamic balance between collection and processing under different load scenarios, significantly improving system stability and task processing real-time performance. Thirdly, it reduces resource consumption and improves resource utilization. Fixed-cycle methods often require a trade-off between response latency and resource consumption. This application, through dynamic cycle adjustment, avoids frequent invalid scans in low-load scenarios, saving CPU and I / O resources; and ensures that new files are collected promptly in high-load scenarios, improving resource utilization efficiency.

[0067] In one embodiment, after updating the timer based on the trigger period, the method further includes: clearing the processed files in the second buffer to prevent the complete queue from accumulating and affecting the work of the file processing thread; scanning the target collection directory and writing newly discovered files into a pending lock-free circular queue (first buffer) without locking operations; using the rte_ring_enqueue() interface to write newly discovered files into the first buffer; and reconfiguring and starting the DPDK timer according to the calculated timer trigger period, using the SINGLE mode to prepare for the next file collection.

[0068] By clearing the second buffer when executing the collection thread, invalid accumulation of processed files is effectively prevented, ensuring the efficiency of the processing thread and the availability of buffer resources. By writing new files to the first buffer without locks, the lock overhead and context switching caused by thread contention are avoided, significantly improving the concurrency performance and throughput of data exchange. Finally, by reconfiguring the DPDK timer in SINGLE mode according to the dynamically calculated trigger period, the collection action is ensured to be triggered periodically and accurately in strict accordance with the latest load-aware strategy. This achieves the coordination of resource cleanup, efficient data entry, and precise scheduling at the system level, ensuring the stable and efficient operation of the adaptive loop.

[0069] It is important to note that in the DPDK (Data Plane Development Kit) timer library, SINGLE mode is a timer operating mode. Its core principle is that after the timer triggers a preset callback function once, it will automatically enter a stopped state and will not automatically trigger again.

[0070] In one embodiment, FIG4 is a flowchart illustrating step S102 according to an exemplary embodiment. As shown in FIG4, step S102, executing the pre-built processing thread, includes obtaining data from a first buffer and processing it, and storing the processing result in a second buffer. Specifically, it includes the following steps: Step S1021, obtaining the data to be processed from the first buffer according to a preset interface.

[0071] Optionally, files to be processed can be retrieved directly from the pending lock-free circular queue without locking operations. In this embodiment, the rte_ring_dequeue_burst() interface is used to retrieve files to be processed directly from the pending lock-free circular queue.

[0072] Step S1022: Perform parsing and dumping operations on the data to be processed to obtain the processing results.

[0073] Optionally, file processing operations such as parsing and dumping can be performed on the retrieved files to be processed.

[0074] Step S1023: Store the result in the second buffer.

[0075] Optionally, the processed files can be added to the complete lock-free circular queue.

[0076] By using a specific interface of a lock-free circular queue, data to be processed is efficiently retrieved from the first buffer, avoiding the contention overhead and thread blocking caused by traditional locking mechanisms, and significantly improving the throughput and response speed of the data retrieval stage. By parsing and dumping the data, the file content is effectively converted and persisted, forming a complete processing chain. Finally, the processing results are stored in the second buffer, thus collaboratively realizing a high-concurrency, low-latency data processing pipeline.

[0077] In one embodiment, step S103, cyclically executing the acquisition thread and the processing thread to balance the data acquisition rate and the data processing rate, specifically includes the following steps: Step S1031, the timer is configured to periodically trigger the acquisition thread according to the updated trigger cycle; Step S1032, in response to receiving the trigger information, the acquisition thread performs calculations of the first load rate and the second load rate, and updates the trigger cycle and the timer based on the first load rate and the second load rate through the disk scan cycle adjustment rule; Step S1033, the processing thread is configured to execute concurrently with the acquisition thread, performing operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer; Step S1034, by cyclically executing the acquisition thread and the processing thread, and adjusting the trigger cycle based on the first load rate and the second load rate, the data acquisition rate and the data processing rate are balanced.

[0078] Optionally, the file acquisition thread and the file processing thread work together continuously under timer scheduling. The timer is adjusted through a dynamic load awareness mechanism to achieve a real-time balance between acquisition rate and processing rate, ensuring that the system has good stability and processing efficiency in both high-load and low-load scenarios.

[0079] Specifically, the timer sends a trigger signal to the acquisition thread when the set time point is reached, based on the currently effective trigger cycle. Upon responding to this signal, the acquisition thread immediately samples and calculates the current occupancy rate of the first buffer, using it as the first load rate, and samples and calculates the current backlog rate of the second buffer, using it as the second load rate. Subsequently, the acquisition thread compares the first load rate with preset idle and overload thresholds, and simultaneously compares the second load rate with the preset overload threshold. Based on these multiple comparisons, it dynamically determines the next trigger cycle to be used (which may be shortened, maintained, or extended) according to the built-in scan cycle adjustment rules. After the decision is made, the acquisition thread first clears the second buffer to reclaim resources, then scans the target directory and writes newly discovered files into the first buffer using lock-free operations. Finally, it reconfigures and starts the timer using the new cycle value obtained from the decision (usually set to SINGLE mode), thus completing this acquisition scheduling. Simultaneously, the processing thread, as an independent concurrent execution entity, continuously extracts files to be processed from the first buffer through the lock-free interface, performs parsing, dumping, and other business operations, and places the processed result files into the second buffer. The operation is achieved by continuously repeating this closed-loop process of timed triggering, load sensing, decision-making cycle, data collection, and parallel processing.

[0080] Step S103 transforms the system from a passive, static, timed-driven model to an active, dynamic one by constructing a complete closed loop of perception, decision-making, and execution. This enables adaptive control of the acquisition frequency: when the system senses strong processing capacity but insufficient files to process (i.e., low first load rate and high second load rate), it automatically shortens the cycle to accelerate file discovery, fully utilizing processing resources and improving system responsiveness and resource utilization in low-load input scenarios; when it senses that an excessively fast acquisition rate may lead to queue overflow (i.e., excessively high first load rate), it automatically extends the cycle to alleviate acquisition pressure, effectively preventing queue accumulation and data loss in high-load scenarios and enhancing system stability; through continuous, decoupled concurrent execution of acquisition and processing threads, combined with lock-free queues for efficient data exchange, it maximizes the overall system throughput while ensuring real-time performance. Ultimately, this mechanism enables the system to maintain a dynamic balance between acquisition and processing capabilities even in unpredictable environments with drastic fluctuations in file generation rates.

[0081] In summary, this application provides a timer adjustment method based on DPDK dynamic load awareness. When the timer triggers a pre-built acquisition thread according to the trigger cycle, a first load rate of the first buffer and a second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time performance of file acquisition and processing, and system resource utilization efficiency. This solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0082] This application also includes the following beneficial effects: High-efficiency concurrency based on lock-free circular queues: The acquisition thread and processing thread interact using a DPDK lock-free circular queue, avoiding the contention and context switching overhead of traditional locked queues. In a multi-threaded concurrent environment, higher throughput and lower latency can be achieved.

[0083] Good scalability and versatility: The method in this application does not depend on specific business logic and can be applied to various scenarios such as log collection, batch file processing, and monitoring data disk storage, and has good scalability and engineering feasibility.

[0084] Secondly, embodiments of this application provide a timer adjustment system based on DPDK dynamic load awareness. Figure 5 is a system structure block diagram of a timer adjustment system based on DPDK dynamic load awareness according to an exemplary embodiment. As shown in Figure 5, the system includes an execution acquisition thread module, an execution processing thread module, and a loop module; wherein: the execution acquisition thread module is used to calculate a first load rate of a first buffer and a second load rate of a second buffer when the timer triggers a pre-built acquisition thread according to the trigger cycle; adjust the trigger cycle based on the first load rate and the second load rate according to the scan cycle adjustment rule; and update the timer based on the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer; the execution processing thread module is used to execute a pre-built processing thread, including acquiring data from the first buffer and processing it, and storing the processing result in the second buffer; the loop module is used to cyclically execute the acquisition thread and the processing thread based on the updated timer, balancing the data acquisition rate and the data processing rate.

[0085] In summary, this application provides a timer adjustment system based on DPDK dynamic load awareness. By executing a data acquisition thread module, a processing thread module, and a loop module, it addresses the problem in related technologies of maintaining a balance between file acquisition and file processing while improving resource utilization efficiency. Specifically, when the timer triggers the pre-built acquisition thread according to the trigger cycle, it calculates the first load rate of the first buffer and the second load rate of the second buffer. Based on the first and second load rates, it adjusts the trigger cycle according to the disk scan cycle adjustment rules and updates the timer based on the trigger cycle. It then scans the target acquisition directory and writes the scanned files into the first buffer. Executing the pre-built processing thread includes acquiring and processing data from the first buffer and storing the processing results in the second buffer. Based on the updated timer, it loops through the acquisition thread and processing thread to balance the data acquisition rate and the data processing rate. By introducing queue load rate as a feedback signal, the trigger period of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time file acquisition and processing, and system resource utilization efficiency.

[0086] It should be noted that the timer adjustment system based on DPDK dynamic load awareness provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0087] Thirdly, embodiments of this application provide an electronic device, and FIG6 is a block diagram of an electronic device according to an exemplary embodiment. As shown in FIG6, the electronic device may include a processor 61 and a memory 62 storing computer program instructions.

[0088] Specifically, the processor 61 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0089] The memory 62 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include removable or non-removable (or fixed) media. Where appropriate, the memory 62 may be internal or external to a data processing device. In a particular embodiment, the memory 62 is non-volatile memory. In a particular embodiment, the memory 62 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0090] The memory 62 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 61.

[0091] The processor 61 reads and executes computer program instructions stored in the memory 62 to implement any of the timer adjustment methods based on DPDK dynamic load awareness in the above embodiments.

[0092] In one embodiment, a timer adjustment device based on DPDK dynamic load awareness may further include a communication interface 63 and a bus 60. As shown in FIG6, the processor 61, memory 62, and communication interface 63 are connected through the bus 60 and communicate with each other.

[0093] The communication interface 63 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 63 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0094] Bus 60 includes hardware, software, or both, that couples together components of a DPDK-based dynamic load-aware timer regulation device. Bus 60 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, and Local Bus. For example, and not as a limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 60 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0095] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a timer adjustment method based on DPDK dynamic load awareness provided in the first aspect.

[0096] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0097] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the DPDK-based dynamic load-aware timer adjustment method provided in the first aspect.

[0098] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A timer adjustment method based on DPDK dynamic load awareness, characterized in that, The method includes: when a timer triggers a pre-built acquisition thread according to a trigger cycle, calculating a first load rate of a first buffer and a second load rate of a second buffer; adjusting the trigger cycle based on the first load rate and the second load rate using a scan cycle adjustment rule; and updating the timer based on the trigger cycle; scanning a target acquisition directory and writing the scanned files into the first buffer; executing a pre-built processing thread, including acquiring and processing data from the first buffer and storing the processing result in the second buffer; and cyclically executing the acquisition thread and the processing thread based on the updated timer to balance the data acquisition rate and the data processing rate.

2. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The step of adjusting the triggering cycle based on the first load rate and the second load rate using a scan cycle adjustment rule includes: setting an idle threshold and a first overload threshold for the first buffer, and setting a second overload threshold for the second buffer; adjusting the triggering cycle based on a comparison result between the first load rate and the idle threshold and the first overload threshold, and a comparison result between the second load rate and the second overload threshold, using a scan cycle adjustment rule.

3. The timer adjustment method based on DPDK dynamic load awareness according to claim 2, characterized in that, The step of adjusting the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, through the disk sweep cycle adjustment rules, includes: determining the adjustment state of the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold; and based on the adjustment state, shortening, maintaining, or extending the trigger cycle through the corresponding adjustment rules.

4. The timer adjustment method based on DPDK dynamic load awareness according to claim 3, characterized in that, The step of determining the adjustment state of the trigger cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, includes: if the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the trigger cycle is determined to be a first state; if the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be a second state; if the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the trigger cycle is determined to be a first state; if the first load rate is greater than the first overload threshold, then the adjustment state of the trigger cycle is determined to be a third state; if the first load rate is equal to a preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be a first state.

5. A timer adjustment method based on DPDK dynamic load awareness according to claim 4, characterized in that, The step of shortening, maintaining, or extending the trigger cycle based on the adjustment state and using corresponding adjustment rules includes: if the adjustment state of the trigger cycle is the first state, then maintaining the previous trigger cycle; if the adjustment state of the trigger cycle is the second state, then multiplying the previous trigger cycle by a first adjustment coefficient to obtain the shortened next trigger cycle; if the adjustment state of the trigger cycle is the third state, then multiplying the previous trigger cycle by a second adjustment coefficient to obtain the extended next trigger cycle.

6. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, Before the timer triggers the pre-built acquisition thread according to the trigger cycle, the method further includes: creating an acquisition thread and a processing thread for the file; creating a pending lock-free circular queue based on the DPDK framework as a first buffer, wherein the first buffer is a buffer of files to be processed; and creating a complete lock-free circular queue based on the DPDK framework as a second buffer, wherein the second buffer is a buffer of files that have been processed.

7. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The calculation of the first load rate of the first buffer and the second load rate of the second buffer includes: obtaining the queue occupancy rate of the first buffer based on the ratio of the number of files to be processed in the first buffer to the total capacity of the queue; obtaining the queue backlog rate of the second buffer based on the ratio of the number of processed files in the second buffer to the total capacity of the queue; using the queue occupancy rate as the first load rate of the first buffer and the queue backlog rate as the second load rate of the second buffer.

8. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The step of obtaining and processing data from the first buffer and storing the processing result in the second buffer includes: obtaining data to be processed from the first buffer according to a preset interface; performing parsing and dumping operations on the data to be processed to obtain the processing result; and storing the result in the second buffer.

9. A timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The method of cyclically executing the acquisition thread and the processing thread to balance the data acquisition rate and the data processing rate includes: the timer being configured to periodically trigger the acquisition thread according to the updated trigger cycle; the acquisition thread responding to receiving trigger information, performing calculations of the first load rate and the second load rate, and updating the trigger cycle and the timer based on the first load rate and the second load rate according to the disk scan cycle adjustment rule; the processing thread being configured to execute concurrently with the acquisition thread, performing operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer; and balancing the data acquisition rate and the data processing rate by cyclically executing the acquisition thread and the processing thread and adjusting the trigger cycle based on the first load rate and the second load rate.

10. A timer regulation system based on DPDK dynamic load sensing, characterized in that, The system is used to execute the method of any one of claims 1 to 9. The system includes an execution acquisition thread module, an execution processing thread module, and a loop module; wherein: the execution acquisition thread module is used to calculate a first load rate of a first buffer and a second load rate of a second buffer when a timer triggers a pre-built acquisition thread according to a trigger cycle; adjust the trigger cycle based on the first load rate and the second load rate according to a disk scan cycle adjustment rule; and update the timer based on the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer; the execution processing thread module is used to execute a pre-built processing thread, including acquiring data from the first buffer and processing it, and storing the processing result in the second buffer; the loop module is used to cyclically execute the acquisition thread and the processing thread based on the updated timer, balancing the data acquisition rate and the data processing rate.

Images