A timer adjustment method and system based on DPDK dynamic load awareness

Abstract

This application relates to a timer adjustment method and system based on DPDK dynamic load awareness. The method includes: when a timer triggers a pre-built acquisition thread according to a trigger cycle, calculating a first load rate of a first buffer and a second load rate of a second buffer; adjusting the trigger cycle based on the first and second load rates using a disk scan cycle adjustment rule; and updating the timer based on the trigger cycle; scanning the target acquisition directory and writing the scanned files into the first buffer; executing the pre-built processing thread, including acquiring and processing data from the first buffer and storing the processing results in the second buffer; and cyclically executing the acquisition thread and processing thread based on the updated timer to balance the data acquisition rate and the data processing rate. This addresses the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

Description

Technical Field

[0001] This application relates to the field of data acquisition and processing technology, and in particular to a timer adjustment method and system based on DPDK dynamic load sensing. Background Technology

[0002] In computer systems, file acquisition and processing is a common data processing method. The file acquisition end is typically responsible for scanning the target directory, discovering new files, and adding them to the task queue; the file processing end then retrieves files from the queue and performs operations such as parsing, analysis, or dumping. To ensure that files enter the processing flow in a timely manner, existing technologies generally adopt a fixed-period timed disk scanning mechanism. A timer triggers the file acquisition thread to scan the directory, writes the scan results to the task queue, and then the file processing thread retrieves files from the queue for processing.

[0003] The existing patent publication number CN111208985B discloses a data processing method, system, and storage medium based on a producer-consumer model. This method optimizes the data processing flow through a sentinel consumer polling mechanism and multi-consumer collaborative work. The sentinel consumer polls for a globally available processing quantity. If such a quantity exists, the sentinel consumer wakes up other consumers, who then process the data in their respective queues until the globally available processing quantity falls below a threshold. At this point, other consumers enter a sleep state, and the sentinel consumer returns to the polling step. After producers generate data, it is distributed evenly to the data queues corresponding to each consumer. This patent monitors the data generated by producers through the sentinel consumer polling mechanism and improves data processing performance when there is excessive producer data by increasing the number of consumers, thus enhancing parallel data processing capabilities to some extent. However, some shortcomings remain:

[0004] First, while polling can monitor data processing volume in real time, it may incur performance overhead, especially when a large number of consumers are involved, where polling itself can become a performance bottleneck. Second, this patent addresses multi-consumer scenarios and does not offer an effective solution for single-consumer scenarios where producers generate large amounts of data. Finally, the sentinel consumer mechanism is highly sensitive to changes in the number of consumers. If there are significant fluctuations in the number of consumers, this mechanism may not be suitable for real-time changes in load demands. It may also be unable to quickly adjust processing strategies when facing highly dynamic and unpredictable workloads.

[0005] Therefore, no effective solution has yet been proposed for adjusting the dynamic load timer in file processing to maintain a balance between file acquisition and processing while improving resource utilization efficiency. Summary of the Invention

[0006] This application provides a timer adjustment method and system based on DPDK dynamic load awareness, which at least solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0007] In a first aspect, embodiments of this application provide a timer adjustment method based on DPDK dynamic load awareness, the method comprising:

[0008] When the timer triggers the pre-built acquisition thread according to the trigger cycle, the first load rate of the first buffer and the second load rate of the second buffer are calculated. Based on the first load rate and the second load rate, the trigger cycle is adjusted according to the disk scan cycle adjustment rule, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer.

[0009] Executing the pre-built processing thread includes retrieving data from the first buffer, processing it, and storing the processing result in the second buffer;

[0010] Based on the updated timer, the acquisition thread and the processing thread are executed cyclically to balance the data acquisition rate and the data processing rate.

[0011] In one embodiment, before the timer triggers a pre-built acquisition thread according to the trigger period, the method further includes:

[0012] Create a file acquisition thread and a file processing thread;

[0013] A pending lock-free circular queue is created based on the DPDK framework as the first buffer, wherein the first buffer is a buffer of files to be processed.

[0014] A lock-free circular queue is created based on the DPDK framework as a second buffer, where the second buffer is a buffer of processed files.

[0015] In one embodiment, calculating the first load rate of the first buffer and the second load rate of the second buffer includes:

[0016] The queue occupancy rate of the first buffer is obtained by the ratio of the number of files to be processed in the first buffer to the total capacity of the queue.

[0017] The queue backlog rate of the second buffer is obtained by comparing the number of processed files in the second buffer with the total capacity of the queue.

[0018] The queue occupancy rate is used as the first load rate of the first buffer, and the queue backlog rate is used as the second load rate of the second buffer.

[0019] In one embodiment, adjusting the triggering cycle based on the first load rate and the second load rate using a disk sweep cycle adjustment rule includes:

[0020] Set the free threshold and the first overload threshold for the first buffer, and set the second overload threshold for the second buffer;

[0021] Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the triggering cycle is adjusted to be shortened, maintained, or extended.

[0022] In one embodiment, adjusting the triggering cycle based on the comparison result of the first load rate with the idle threshold and the first overload threshold, and the comparison result of the second load rate with the second overload threshold, through a disk sweep cycle adjustment rule, includes:

[0023] Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the adjustment state of the triggering cycle is determined;

[0024] Based on the adjustment state, the triggering cycle is shortened, maintained, or extended according to the corresponding adjustment rules.

[0025] Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the adjustment state of the triggering cycle is determined;

[0026] Based on the adjustment state, the triggering cycle is shortened, maintained, or extended according to the corresponding adjustment rules.

[0027] In one embodiment, determining the state of the trigger cycle adjustment based on the comparison result of the first load rate with the idle threshold and the first overload threshold, and the comparison result of the second load rate with the second overload threshold, includes:

[0028] If the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the triggering cycle is determined to be the first state.

[0029] If the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be the second state.

[0030] If the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the triggering cycle is determined to be the first state.

[0031] If the first load rate is greater than the first overload threshold, then the adjustment state of the triggering cycle is determined to be the third state;

[0032] If the first load rate is equal to a preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0033] In one embodiment, the step of shortening, maintaining, or extending the triggering period based on the adjustment state using corresponding adjustment rules includes:

[0034] If the adjustment state of the triggering cycle is the first state, then the previous triggering cycle is maintained;

[0035] If the adjustment state of the triggering cycle is the second state, then the previous triggering cycle is multiplied by the first adjustment coefficient to obtain the shortened next triggering cycle;

[0036] If the adjustment state of the triggering cycle is the third state, then the previous triggering cycle is multiplied by the second adjustment coefficient to obtain the extended next triggering cycle.

[0037] In one embodiment, the step of retrieving and processing data from the first buffer and storing the processing result in the second buffer includes:

[0038] According to the preset interface, the data to be processed is obtained from the first buffer;

[0039] The data to be processed is parsed and dumped to obtain the processing result;

[0040] The result is stored in the second buffer.

[0041] In one embodiment, the cyclic execution of the acquisition thread and the processing thread to balance the data acquisition rate and the data processing rate includes:

[0042] The timer is configured to periodically trigger the acquisition thread according to the updated trigger period;

[0043] In response to receiving the trigger information, the acquisition thread performs the calculation of the first load rate and the second load rate, and updates the trigger period and the timer based on the first load rate and the second load rate through the disk scanning cycle adjustment rule;

[0044] The processing thread is configured to execute concurrently with the acquisition thread, performing operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer;

[0045] By cyclically executing the acquisition thread and the processing thread, and adjusting the triggering cycle based on the first load rate and the second load rate, the data acquisition rate and the data processing rate are balanced.

[0046] Secondly, embodiments of this application provide a timer adjustment system based on DPDK dynamic load awareness. The system is used to execute the method described above, and includes an execution acquisition thread module, an execution processing thread module, and a loop module; wherein:

[0047] The execution acquisition thread module is used to calculate the first load rate of the first buffer and the second load rate of the second buffer when the timer triggers the pre-built acquisition thread according to the trigger cycle; adjust the trigger cycle according to the first load rate and the second load rate through the disk scan cycle adjustment rules; and update the timer according to the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer.

[0048] The execution processing thread module, used to execute a pre-built processing thread, includes obtaining data from the first buffer and processing it, and storing the processing result in the second buffer;

[0049] The loop module is used to repeatedly execute the acquisition thread and the processing thread based on the updated timer, thereby balancing the data acquisition rate and the data processing rate.

[0050] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a timer adjustment based on DPDK dynamic load awareness as described in the first aspect above.

[0051] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a timer adjustment method based on DPDK dynamic load awareness as described in the first aspect above.

[0052] The timer adjustment method and system based on DPDK dynamic load awareness provided in this application embodiment have at least the following technical effects.

[0053] When the timer triggers the pre-built acquisition thread according to the trigger cycle, the first load rate of the first buffer and the second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. The pre-built processing thread is executed, including acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and the processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, so that the file acquisition cycle can adapt to the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing. This improves the system's stability, real-time performance of file acquisition and processing, and system resource utilization efficiency, solving the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0054] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 This is a flowchart of a timer adjustment method based on DPDK dynamic load awareness;

[0057] Figure 2 This is a schematic diagram illustrating the structure of resource initialization according to an exemplary embodiment;

[0058] Figure 3 This is a schematic diagram of a structure for adjusting the trigger cycle according to an exemplary embodiment;

[0059] Figure 4 This is a schematic flowchart of step S102 according to an exemplary embodiment;

[0060] Figure 5 This is a system structure block diagram of a timer regulation system based on DPDK dynamic load awareness, according to an exemplary embodiment.

[0061] Figure 6 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0063] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0064] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0065] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0066] In computer systems, file acquisition and processing is a common data processing method. The file acquisition end is typically responsible for scanning the target directory, discovering new files, and adding them to the task queue; the file processing end then retrieves files from the queue and performs operations such as parsing, analysis, or dumping. To ensure that files enter the processing flow in a timely manner, existing technologies generally adopt a fixed-period timed disk scanning mechanism. A timer triggers the file acquisition thread to scan the directory, writes the scan results to the task queue, and then the file processing thread retrieves files from the queue for processing.

[0067] However, fixed-period sweeping mechanisms have many shortcomings in practical applications:

[0068] 1. Static mismatch between data acquisition and processing capabilities.

[0069] When the file acquisition thread generates files at a rate that consistently exceeds the processing speed of the file processing thread during peak periods, the task queue is prone to continuous accumulation, which may lead to queue overflow or delayed processing of some files in severe cases. Conversely, when the long-term processing capacity is strong but the number of new files generated in the directory is small, the queue may remain idle for a long time, resulting in wasted resources.

[0070] 2. The rate of file growth fluctuates dynamically and is unpredictable.

[0071] In scenarios such as log collection, batch file generation, and monitoring data storage, the file generation rate often fluctuates significantly over time and is unpredictable. Fixed-period disk scanning mechanisms cannot keep up with this change in real time. When file growth suddenly increases, it can easily lead to a sudden backlog of disk directory files and queues, while when growth slows down, system resources become idle.

[0072] 3. The sweep cycle setting makes it difficult to balance efficiency and resources.

[0073] Fixed scan cycles typically require manual setting, but controlling the size of the scan cycle is difficult, often resulting in a trade-off between acquisition and processing efficiency and resource consumption. Setting a short scan cycle to reduce the acquisition delay of new files can lead to frequent invalid scans when few new files are added, consuming CPU and I / O resources. Conversely, setting a long cycle can prevent new files from being detected in a timely manner when many new files are added, affecting real-time processing.

[0074] Some optimization methods attempt to alleviate the above problems by increasing the number of file acquisition and processing threads or expanding the queue capacity, but these methods cannot fundamentally solve the contradiction between the acquisition rate and the processing rate.

[0075] Therefore, there is an urgent need for a timer control method that can automatically adjust the disk scanning cycle according to the file acquisition and processing load, so as to maintain a dynamic balance between file acquisition and file processing, thereby improving the system's stability and file acquisition and processing efficiency.

[0076] The existing patent publication number CN118018494A discloses a data processing method, system, and storage medium based on a producer-consumer model. This patent optimizes the data processing flow through a sentinel consumer polling mechanism and multi-consumer collaborative work. The sentinel consumer polls for a globally available processing quantity. If a globally available quantity exists, the sentinel consumer wakes up other consumers, who then process the data in their respective data queues until the globally available processing quantity falls below a threshold. At this point, other consumers enter a sleep state, and the sentinel consumer returns to the polling step. After the producer generates data, it is distributed evenly to the data queues corresponding to each consumer.

[0077] First, while polling can monitor data processing volume in real time, it may incur performance overhead, especially when a large number of consumers are involved, where polling itself can become a performance bottleneck. Second, this patent addresses multi-consumer scenarios and does not offer an effective solution for single-consumer scenarios where producers generate large amounts of data. Finally, the sentinel consumer mechanism is highly sensitive to changes in the number of consumers. If there are significant fluctuations in the number of consumers, this mechanism may not be suitable for real-time changes in load demands. It may also be unable to quickly adjust processing strategies when facing highly dynamic and unpredictable workloads.

[0078] In summary, the dynamic load timer adjustment method for file processing aims to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0079] In this document, it should be understood that the terms used may be technical means used to implement part of the present invention or other summary technical terms. For example, the terms may include:

[0080] DPDK (Data Plane Development Kit) is a user-space open-source software library and driver framework for fast packet processing.

[0081] Scan cycle adjustment rules: A set of predefined decision logic that dynamically calculates the next file directory scan interval based on real-time load status. The core inputs are the occupancy rate of the first buffer (pending queue) and the backlog rate of the second buffer (complete queue). By comparing these with preset thresholds, the system load status (such as idle, normal, overload) is determined, and then a new trigger cycle (shortened, maintained, or extended) is non-linearly decided.

[0082] Data acquisition rate: The number of new files that the file acquisition thread discovers and stores in the first buffer (pending queue) per unit of time.

[0083] Data processing rate: The number of files that a file processing thread retrieves from the first buffer (pending queue) and completes processing (such as parsing and dumping) per unit of time.

[0084] Pending lock-free circular queue: A first-in-first-out (FIFO) circular buffer implemented based on DPDK, serving as the first buffer and a buffer for files to be processed. The acquisition thread writes newly discovered files into this queue, and the processing thread retrieves files from this queue for processing.

[0085] Complete lock-free circular queue: A circular buffer implemented based on DPDK, serving as a secondary buffer for processed files. Processing threads place successfully processed files into this queue.

[0086] In a first aspect, embodiments of this application provide a timer adjustment method based on DPDK dynamic load awareness. Figure 1 This is a flowchart of a timer tuning method based on DPDK dynamic load awareness, such as... Figure 1 As shown, the method includes:

[0087] Step S101: When the timer triggers the pre-built acquisition thread according to the trigger cycle, calculate the first load rate of the first buffer and the second load rate of the second buffer. Based on the first load rate and the second load rate, adjust the trigger cycle according to the disk scan cycle adjustment rule, and update the timer based on the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer.

[0088] Step S102, executing the pre-built processing thread includes retrieving data from the first buffer and processing it, and storing the processing result in the second buffer.

[0089] Step S103: Based on the updated timer, the acquisition thread and processing thread are executed in a loop to balance the data acquisition rate and the data processing rate.

[0090] In summary, this application provides a timer adjustment method based on DPDK dynamic load awareness. When the timer triggers a pre-built acquisition thread according to the trigger cycle, a first load rate of the first buffer and a second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time performance of file acquisition and processing, and system resource utilization efficiency. This solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0091] In one embodiment, before the timer triggers the pre-built acquisition thread according to the trigger cycle in step S101, Figure 2 This is a schematic diagram illustrating the structure of resource initialization according to an exemplary embodiment, such as... Figure 2 As shown, the method also includes:

[0092] Create a file acquisition thread and a file processing thread;

[0093] A pending lock-free circular queue is created based on the DPDK framework as the first buffer, where the first buffer is a buffer of files to be processed.

[0094] A lock-free circular queue is created based on the DPDK framework as a second buffer, where the second buffer is a buffer of processed files.

[0095] Optionally, a lock-free circular queue is created using the `rte_ring_create()` interface, and a DPDK timer is initialized. The DPDK timer uses SINGLE mode to control the periodic triggering of the file acquisition thread. Creating independent acquisition and processing threads decouples tasks and enables parallel execution, thereby improving the overall system throughput. Based on the DPDK framework, lock-free circular queues for pending and completed tasks are created as dual buffers, isolating pending and processed files respectively. This not only achieves secure and efficient data exchange between acquisition and processing but also avoids thread contention and context switching overhead through lock-free design. The occupancy rate of the pending queue and the backlog rate of the completed queue provide real-time and accurate feedback signals for subsequent dynamic load sensing, enabling the system to dynamically adjust the acquisition frequency based on the actual queue status. This fundamentally improves resource utilization, system stability, and real-time processing performance, and allows for dynamic adjustment of the timer period. Compared to a fixed-period disk scanning mechanism, this approach can adapt to the actual system load in real time, avoiding queue backlog during peak periods and frequent idling during low-load periods.

[0096] In one embodiment, step S101 calculates the first load rate of the first buffer and the second load rate of the second buffer, specifically including the following steps:

[0097] Step 1: Obtain the queue occupancy rate of the first buffer based on the ratio of the number of files to be processed in the first buffer to the total capacity of the queue;

[0098] Step 2: Obtain the queue backlog rate of the second buffer based on the ratio of the number of processed files in the second buffer to the total queue capacity;

[0099] Step 3: Use the queue occupancy rate as the first load rate of the first buffer and the queue backlog rate as the second load rate of the second buffer.

[0100] Optionally, the queue occupancy rate is obtained by calculating the ratio of the number of files to be processed to the total queue capacity, and the queue backlog rate is obtained by calculating the ratio of the number of processed files to the total queue capacity. These two are used as the first load rate and the second load rate, respectively. This provides objective and real-time quantifiable metrics.

[0101] In one embodiment, Figure 3 This is a schematic diagram illustrating the structure for adjusting the trigger cycle according to an exemplary embodiment, such as... Figure 3 As shown, step S101 adjusts the trigger cycle based on the first load rate and the second load rate using the disk scan cycle adjustment rule, specifically including the following steps:

[0102] Step 1: Set the free threshold and the first overload threshold of the first buffer, and set the second overload threshold of the second buffer.

[0103] Optionally, in this embodiment, the idle threshold P_low can be set to 0.3, the first overload threshold P_high can be set to 0.7, and the second overload threshold C_thr can be set to 0.5. That is:

[0104] When the load rate of the pending queue is less than 0.3, the pending queue is considered to be idle.

[0105] When the pending queue load rate is in the range of [0.3, 0.7], the pending queue load is considered normal.

[0106] When the pending queue load rate is greater than 0.7, the pending queue is considered overloaded.

[0107] When the load rate of the complete queue is greater than 0.5, the complete queue is considered overloaded.

[0108] Step 2: Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, adjust the triggering cycle by shortening, maintaining, or extending it. Specifically, this includes:

[0109] Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the adjustment state of the trigger cycle is determined. This includes:

[0110] If the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0111] Optionally, if the pending queue load rate is less than the threshold of 0.3 and the complete queue load rate is less than the threshold of 0.5, it indicates that the new file collection rate is insufficient and the file processing capacity is limited. The state of the trigger cycle is determined to be the first state, that is, the queue load is normal.

[0112] If the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be the second state.

[0113] Optionally, if the pending queue load rate is less than the threshold of 0.3 and the complete queue load rate is greater than or equal to the threshold of 0.5, it indicates that the new file acquisition rate is insufficient, but the file processing capability is strong. The adjustment state of the trigger cycle is determined to be the second state, that is, the queue load is idle.

[0114] If the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0115] Optionally, if the pending queue load rate is in the range of [0.3, 0.7], it indicates that the file acquisition and processing capabilities are in a balanced state, the system is running smoothly overall, and the state for determining the trigger cycle is the first state, that is, the queue load is normal.

[0116] If the first load rate is greater than the first overload threshold, the adjustment state of the trigger cycle is determined to be the third state.

[0117] Optionally, if the pending queue load rate is greater than or equal to the threshold of 0.7, it indicates that the file acquisition rate is too high and the file processing capacity is insufficient, and there is a risk of queue backlog. The state of the trigger cycle is determined to be the third state, that is, the queue overload state.

[0118] If the first load rate is equal to the preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be the first state.

[0119] Optionally, if the pending queue load rate is approximately 0 and the complete queue load rate is approximately 0, it means that there are no files to process at this time, and the state of the trigger cycle is determined to be the first state, that is, the queue load is normal.

[0120] In one embodiment, based on the adjustment state, the triggering period is shortened, maintained, or extended according to corresponding adjustment rules. This includes:

[0121] If the adjustment state of the trigger cycle is in the first state, then the previous trigger cycle will be maintained.

[0122] Optionally, if the adjustment status of the trigger cycle is normal queue load, it indicates that the new file collection rate is insufficient and the file processing capacity is limited. In this case, maintain the previous timer trigger cycle or set the default timer trigger cycle when the program starts to avoid invalid frequent scanning.

[0123] If the adjustment state of the trigger cycle is the second state, then the previous trigger cycle is multiplied by the first adjustment coefficient to obtain the shortened next trigger cycle;

[0124] Optionally, if the trigger cycle adjustment status is queue load idle state, it indicates that the new file acquisition rate is insufficient, but the file processing capability is strong. In this case, the timer trigger cycle is shortened, that is, the previous timer cycle is multiplied by 0.7 as the next timer trigger cycle, in order to speed up the directory scan and ensure that the processing end can obtain the files to be processed in a timely manner.

[0125] If the adjustment state of the trigger cycle is the third state, then the previous trigger cycle is multiplied by the second adjustment coefficient to obtain the extended next trigger cycle.

[0126] Optionally, if the adjustment state of the trigger cycle is the queue overload state, it indicates that the file acquisition rate is too high and the file processing capacity is insufficient, and there is a risk of queue accumulation. In this case, the timer trigger cycle is extended, that is, the previous timer cycle is multiplied by 1.3 as the next timer trigger cycle, so as to reduce the acquisition frequency and avoid pending queue overflow.

[0127] The beneficial effects of adjusting the trigger cycle are as follows: Firstly, it improves system stability and real-time performance. When the rate of new file additions far exceeds the file processing capacity, the collection cycle is automatically extended to alleviate disk directory file and queue backlogs. Secondly, when file processing capacity is high, the collection cycle is shortened to accelerate file discovery. This maintains a dynamic balance between collection and processing under different load scenarios, significantly improving system stability and task processing real-time performance. Thirdly, it reduces resource consumption and improves resource utilization. Fixed-cycle methods often require a trade-off between response latency and resource consumption. This application, through dynamic cycle adjustment, avoids frequent invalid scans in low-load scenarios, saving CPU and I / O resources; and ensures that new files are collected promptly in high-load scenarios, improving resource utilization efficiency.

[0128] In one embodiment, after updating the timer based on the trigger period, the method further includes: clearing the processed files in the second buffer to prevent the complete queue from accumulating and affecting the work of the file processing thread; scanning the target collection directory and writing newly discovered files into a pending lock-free circular queue (first buffer) without locking operations; using the rte_ring_enqueue() interface to write newly discovered files into the first buffer; and reconfiguring and starting the DPDK timer according to the calculated timer trigger period, using the SINGLE mode to prepare for the next file collection.

[0129] By clearing the second buffer when executing the collection thread, invalid accumulation of processed files is effectively prevented, ensuring the efficiency of the processing thread and the availability of buffer resources. By writing new files to the first buffer without locks, the lock overhead and context switching caused by thread contention are avoided, significantly improving the concurrency performance and throughput of data exchange. Finally, by reconfiguring the DPDK timer in SINGLE mode according to the dynamically calculated trigger period, the collection action is ensured to be triggered periodically and accurately in strict accordance with the latest load-aware strategy. This achieves the coordination of resource cleanup, efficient data entry, and precise scheduling at the system level, ensuring the stable and efficient operation of the adaptive loop.

[0130] It is important to note that in the DPDK (Data Plane Development Kit) timer library, SINGLE mode is a timer operating mode. Its core principle is that after the timer triggers a preset callback function once, it will automatically enter a stopped state and will not automatically trigger again.

[0131] In one embodiment, Figure 4 This is a schematic flowchart of step S102 according to an exemplary embodiment, as shown below. Figure 4 As shown, step S102, executing the pre-built processing thread, includes retrieving data from the first buffer, processing it, and storing the processing result in the second buffer. Specifically, it includes the following steps:

[0132] Step S1021: Obtain the data to be processed from the first buffer according to the preset interface.

[0133] Optionally, files to be processed can be retrieved directly from the pending lock-free circular queue without locking operations. In this embodiment, the rte_ring_dequeue_burst() interface is used to retrieve files to be processed directly from the pending lock-free circular queue.

[0134] Step S1022: Perform parsing and dumping operations on the data to be processed to obtain the processing results.

[0135] Optionally, file processing operations such as parsing and dumping can be performed on the retrieved files to be processed.

[0136] Step S1023: Store the result in the second buffer.

[0137] Optionally, the processed files can be added to the complete lock-free circular queue.

[0138] By using a specific interface of a lock-free circular queue, data to be processed is efficiently retrieved from the first buffer, avoiding the contention overhead and thread blocking caused by traditional locking mechanisms, and significantly improving the throughput and response speed of the data retrieval stage. By parsing and dumping the data, the file content is effectively converted and persisted, forming a complete processing chain. Finally, the processing results are stored in the second buffer, thus collaboratively realizing a high-concurrency, low-latency data processing pipeline.

[0139] In one embodiment, step S103, which involves cyclically executing the acquisition thread and the processing thread to balance the data acquisition rate and the data processing rate, specifically includes the following steps:

[0140] Step S1031: The timer is configured to periodically trigger the acquisition thread according to the updated trigger period;

[0141] Step S1032: The acquisition thread responds to the received trigger information, performs the calculation of the first load rate and the second load rate, and updates the trigger period and timer based on the first load rate and the second load rate through the disk scan cycle adjustment rule;

[0142] Step S1033: The processing thread is configured to execute concurrently with the acquisition thread to perform operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer.

[0143] Step S1034: By cyclically executing the acquisition thread and the processing thread, and adjusting the trigger cycle based on the first load rate and the second load rate, the data acquisition rate and the data processing rate are balanced.

[0144] Optionally, the file acquisition thread and the file processing thread work together continuously under timer scheduling. The timer is adjusted through a dynamic load awareness mechanism to achieve a real-time balance between acquisition rate and processing rate, ensuring that the system has good stability and processing efficiency in both high-load and low-load scenarios.

[0145] Specifically, the timer sends a trigger signal to the acquisition thread when the set time point is reached, based on the currently effective trigger cycle. Upon responding to this signal, the acquisition thread immediately samples and calculates the current occupancy rate of the first buffer, using it as the first load rate, and samples and calculates the current backlog rate of the second buffer, using it as the second load rate. Subsequently, the acquisition thread compares the first load rate with preset idle and overload thresholds, and simultaneously compares the second load rate with the preset overload threshold. Based on these multiple comparisons, it dynamically determines the next trigger cycle to be used (which may be shortened, maintained, or extended) according to the built-in scan cycle adjustment rules. After the decision is made, the acquisition thread first clears the second buffer to reclaim resources, then scans the target directory and writes newly discovered files into the first buffer using lock-free operations. Finally, it reconfigures and starts the timer using the new cycle value obtained from the decision (usually set to SINGLE mode), thus completing this acquisition scheduling. Simultaneously, the processing thread, as an independent concurrent execution entity, continuously extracts files to be processed from the first buffer through the lock-free interface, performs parsing, dumping, and other business operations, and places the processed result files into the second buffer. The operation is achieved by continuously repeating this closed-loop process of timed triggering, load sensing, decision-making cycle, data collection, and parallel processing.

[0146] Step S103 transforms the system from a passive, static, timed-driven model to an active, dynamic one by constructing a complete closed loop of perception, decision-making, and execution. This enables adaptive control of the acquisition frequency: when the system senses strong processing capacity but insufficient files to process (i.e., low first load rate and high second load rate), it automatically shortens the cycle to accelerate file discovery, fully utilizing processing resources and improving system responsiveness and resource utilization in low-load input scenarios; when it senses that an excessively fast acquisition rate may lead to queue overflow (i.e., excessively high first load rate), it automatically extends the cycle to alleviate acquisition pressure, effectively preventing queue accumulation and data loss in high-load scenarios and enhancing system stability; through continuous, decoupled concurrent execution of acquisition and processing threads, combined with lock-free queues for efficient data exchange, it maximizes the overall system throughput while ensuring real-time performance. Ultimately, this mechanism enables the system to maintain a dynamic balance between acquisition and processing capabilities even in unpredictable environments with drastic fluctuations in file generation rates.

[0147] In summary, this application provides a timer adjustment method based on DPDK dynamic load awareness. When the timer triggers a pre-built acquisition thread according to the trigger cycle, a first load rate of the first buffer and a second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing the queue load rate as a feedback signal, the trigger cycle of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in the file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time performance of file acquisition and processing, and system resource utilization efficiency. This solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency.

[0148] This application also includes the following beneficial effects:

[0149] High-efficiency concurrency based on lock-free circular queues: The acquisition thread and processing thread interact using a DPDK lock-free circular queue, avoiding the contention and context switching overhead of traditional locked queues. In a multi-threaded concurrent environment, this achieves higher throughput and lower latency.

[0150] Good scalability and versatility: The method in this application does not depend on specific business logic and can be applied to various scenarios such as log collection, batch file processing, and monitoring data disk storage, and has good scalability and engineering feasibility.

[0151] Secondly, embodiments of this application provide a timer adjustment system based on DPDK dynamic load awareness. Figure 5 This is a system architecture block diagram illustrating a DPDK-based dynamic load-aware timer regulation system according to an exemplary embodiment. For example... Figure 5 As shown, the system includes an execution acquisition thread module 510, an execution processing thread module 520, and a loop module 530; wherein:

[0152] The acquisition thread module 510 is used to calculate the first load rate of the first buffer and the second load rate of the second buffer when the timer triggers the pre-built acquisition thread according to the trigger cycle. Based on the first load rate and the second load rate, the trigger cycle is adjusted according to the disk scan cycle adjustment rule, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned and the scanned files are written to the first buffer.

[0153] The execution processing thread module 520 is used to execute a pre-built processing thread, including retrieving data from a first buffer and processing it, and storing the processing result in a second buffer.

[0154] The loop module 530 is used to execute the acquisition thread and the processing thread in a loop based on the updated timer, balancing the data acquisition rate and the data processing rate.

[0155] In summary, this application provides a timer adjustment system based on DPDK dynamic load awareness. By executing a data acquisition thread module 510, a processing thread module 520, and a loop module 530, it solves the problem in related technologies of how to maintain a balance between file acquisition and file processing while improving resource utilization efficiency. Specifically, when the timer triggers the pre-built acquisition thread according to the trigger cycle, a first load rate of the first buffer and a second load rate of the second buffer are calculated. Based on the first and second load rates, the trigger cycle is adjusted according to the disk scan cycle adjustment rules, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes acquiring data from the first buffer, processing it, and storing the processing results in the second buffer. Based on the updated timer, the acquisition thread and processing thread are executed cyclically to balance the data acquisition rate and the data processing rate. By introducing queue load rate as a feedback signal, the trigger period of the timer is dynamically adjusted, enabling the file acquisition cycle to adapt to changes in file acquisition and processing load. This effectively avoids queue accumulation or overflow in high-load scenarios and avoids excessive resource consumption in low-load scenarios, achieving a dynamic balance between file acquisition and processing, and improving system stability, real-time file acquisition and processing, and system resource utilization efficiency.

[0156] It should be noted that the timer adjustment system based on DPDK dynamic load awareness provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] Thirdly, embodiments of this application provide an electronic device, Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 6 As shown, the electronic device may include a processor 61 and a memory 62 storing computer program instructions.

[0158] Specifically, the processor 61 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0159] The memory 62 may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include removable or non-removable (or fixed) media. Where appropriate, the memory 62 may be internal or external to a data processing device. In a particular embodiment, the memory 62 is non-volatile memory. In a particular embodiment, the memory 62 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0160] The memory 62 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 61.

[0161] The processor 61 reads and executes computer program instructions stored in the memory 62 to implement any of the timer adjustment methods based on DPDK dynamic load awareness in the above embodiments.

[0162] In one embodiment, a timer adjustment device based on DPDK dynamic load awareness may further include a communication interface 63 and a bus 60. Wherein, as Figure 6 As shown, the processor 61, memory 62, and communication interface 63 are connected through bus 60 and complete communication with each other.

[0163] The communication interface 63 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 63 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0164] Bus 60 includes hardware, software, or both, that couples together components of a DPDK-based dynamic load-aware timer regulation device. Bus 60 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, and Local Bus. For example, and not as a limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 60 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0165] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a timer adjustment method based on DPDK dynamic load awareness provided in the first aspect.

[0166] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0167] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to perform steps implementing the DPDK-based dynamic load-aware timer adjustment method provided in the first aspect.

[0168] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A timer adjustment method based on DPDK dynamic load awareness, characterized in that, The method includes: Create a file acquisition thread and a file processing thread; create a pending lock-free circular queue based on the DPDK framework as the first buffer, wherein the first buffer is a buffer of files to be processed; create a complete lock-free circular queue based on the DPDK framework as the second buffer, wherein the second buffer is a buffer of files that have been processed. When the timer triggers the pre-built acquisition thread according to the trigger cycle, the first load rate of the first buffer and the second load rate of the second buffer are calculated. The idle threshold and the first overload threshold of the first buffer are set, and the second overload threshold of the second buffer is set. Based on the comparison result of the first load rate with the idle threshold and the first overload threshold, and the comparison result of the second load rate with the second overload threshold, the trigger cycle is adjusted according to the disk scan cycle adjustment rule, and the timer is updated based on the trigger cycle. The target acquisition directory is scanned, and the scanned files are written to the first buffer. Executing the pre-built processing thread includes retrieving data from the first buffer, processing it, and storing the processing result in the second buffer; Based on the updated timer, the acquisition thread and the processing thread are executed cyclically to balance the data acquisition rate and the data processing rate.

2. The method of claim 1, wherein the method is based on DPDK dynamic load awareness. The step of adjusting the triggering cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, through the disk sweep cycle adjustment rules, includes: Based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, the adjustment state of the triggering cycle is determined; Based on the adjustment state, the triggering cycle is shortened, maintained, or extended according to the corresponding adjustment rules.

3. The method of claim 2, wherein the method further comprises: The step of determining the adjustment state of the triggering cycle based on the comparison results of the first load rate with the idle threshold and the first overload threshold, and the comparison results of the second load rate with the second overload threshold, includes: If the first load rate is less than the idle threshold and the second load rate is less than the second overload threshold, then the adjustment state of the triggering cycle is determined to be the first state. If the first load rate is less than the idle threshold and the second load rate is greater than or equal to the second overload threshold, then the adjustment state of the trigger cycle is determined to be the second state. If the first load rate is greater than or equal to the idle threshold and less than or equal to the first overload threshold, then the adjustment state of the triggering cycle is determined to be the first state. If the first load rate is greater than the first overload threshold, then the adjustment state of the triggering cycle is determined to be the third state; If the first load rate is equal to a preset threshold and the second load rate is equal to the preset threshold, then the adjustment state of the trigger cycle is determined to be the first state.

4. The method of claim 3, wherein the method further comprises: The step of shortening, maintaining, or extending the triggering period based on the adjustment state and using corresponding adjustment rules includes: If the adjustment state of the triggering cycle is the first state, then the previous triggering cycle is maintained; If the adjustment state of the triggering cycle is the second state, then the previous triggering cycle is multiplied by the first adjustment coefficient to obtain the shortened next triggering cycle; If the adjustment state of the triggering cycle is the third state, then the previous triggering cycle is multiplied by the second adjustment coefficient to obtain the extended next triggering cycle.

5. The method of claim 1, wherein the method is based on DPDK dynamic load awareness. The calculation of the first load rate of the first buffer and the second load rate of the second buffer includes: The queue occupancy rate of the first buffer is obtained based on the ratio of the number of files to be processed in the first buffer to the total capacity of the queue. The queue backlog rate of the second buffer is obtained by comparing the number of processed files in the second buffer with the total capacity of the queue. The queue occupancy rate is used as the first load rate of the first buffer, and the queue backlog rate is used as the second load rate of the second buffer.

6. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The step of retrieving and processing data from the first buffer and storing the processing result in the second buffer includes: According to the preset interface, the data to be processed is obtained from the first buffer; The data to be processed is parsed and dumped to obtain the processing result; The result is stored in the second buffer.

7. The timer adjustment method based on DPDK dynamic load awareness according to claim 1, characterized in that, The cyclic execution of the acquisition thread and the processing thread, balancing the data acquisition rate and the data processing rate, includes: The timer is configured to periodically trigger the acquisition thread according to the updated trigger period; In response to receiving the trigger information, the acquisition thread performs the calculation of the first load rate and the second load rate, and updates the trigger period and the timer based on the first load rate and the second load rate through the disk scanning cycle adjustment rule; The processing thread is configured to execute concurrently with the acquisition thread, performing operations such as acquiring data from the first buffer, processing data, and writing processing results to the second buffer; By cyclically executing the acquisition thread and the processing thread, and adjusting the triggering cycle based on the first load rate and the second load rate, the data acquisition rate and the data processing rate are balanced.

8. A DPDK-based dynamic load-aware timer adjustment system, comprising: The system is used to execute the method according to any one of claims 1 to 7, and the system includes an execution acquisition thread module, an execution processing thread module, and a loop module; wherein: The execution acquisition thread module is used to calculate the first load rate of the first buffer and the second load rate of the second buffer when the timer triggers the pre-built acquisition thread according to the trigger cycle; adjust the trigger cycle according to the first load rate and the second load rate through the disk scan cycle adjustment rules; and update the timer according to the trigger cycle; scan the target acquisition directory and write the scanned files into the first buffer. The execution processing thread module, used to execute a pre-built processing thread, includes obtaining data from the first buffer and processing it, and storing the processing result in the second buffer; The loop module is used to repeatedly execute the acquisition thread and the processing thread based on the updated timer, thereby balancing the data acquisition rate and the data processing rate.

Images