A method, device and medium for balancing network flow packets

Abstract

The application provides a network flow data packet balancing discarding method, device and medium, relates to the technical field of network flow data packet balancing discarding, and comprises the following steps: determining a stress index used for representing the overall balancing state of a system; judging whether the system is in a balancing state according to the numerical range of the stress index; if the system is not in the balancing state, adjusting the transfer rate through a preset feedback control algorithm; the control amount of the feedback control algorithm is determined according to the errors of the stress index, the occupancy rate of a network card cache area relative to a target value and the occupancy rate of a discarding cache area relative to a target value; transferring the network flow data packets temporarily stored in the network card cache area to the discarding cache area according to the adjusted transfer rate; allocating corresponding hard disk identifiers to the network flow data packets in the discarding cache area, and writing the network flow data packets into corresponding hard disks in batches; and the application can enhance the running stability and reliability in a high-load or performance heterogeneous environment.

Description

Technical Field

[0001] This invention relates to the field of balanced disk persistence technology for network flow data packets, and in particular to a method, device and medium for balanced disk persistence of network flow data packets. Background Technology

[0002] In network traffic monitoring, security auditing, and big data acquisition systems, the real-time and reliable writing of massive data packets captured by high-speed network interfaces to storage media is a key technical aspect ensuring data integrity and the timeliness of subsequent analysis. To buffer the speed difference between network traffic bursts and relatively slow disk I / O, existing systems typically employ a multi-level caching architecture, such as setting up network interface card (NIC) caches and disk write-to-disk caches. However, existing technologies face significant challenges in handling such multi-level cached data streams: the system lacks the ability to collaboratively perceive and dynamically control the entire data pipeline (including data reception, inter-cache transfer, and final disk write-to-disk). Specifically, each processing stage (such as NIC reception rate, inter-cache transfer rate, and disk write-to-disk rate) often operates independently or is simply controlled based on local thresholds, failing to adjust in real-time and collaboratively according to the overall system load, easily leading to data flow imbalances. For example, improper transfer rate settings may cause the upstream NIC cache to overflow while the downstream disk write-to-disk cache remains idle, or insufficient disk write-to-disk rate may cause a large backlog of data in the disk write-to-disk cache, resulting in increased overall processing latency, data packet loss, or a sharp decline in storage performance. Therefore, existing technologies urgently need a strategy that can intelligently sense the overall equilibrium state of the system and dynamically adjust the distribution of data flow among various links in order to achieve high throughput, low latency and stable network streaming data disking. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of this application, a method for balanced disk persistence of network flow data packets is provided, comprising:

[0005] Real-time monitoring and acquisition of the system's first real-time occupancy rate of the network card cache, the second real-time occupancy rate of the disk cache, the current network card data receiving rate, the current data transfer rate from the network card cache to the disk cache, and the current aggregate disk write rate of the hard disk array;

[0006] Based on the mismatch between the network card data receiving rate, transfer rate and aggregate disk write rate, the first deviation between the real-time occupancy rate of the network card buffer and the preset target occupancy rate, and the second deviation between the real-time occupancy rate of the disk write buffer and the preset target occupancy rate, a pressure index is determined to characterize the overall equilibrium state of the system.

[0007] Based on the range of values ​​for the pressure index, determine whether the system is in equilibrium.

[0008] If the system is not in a balanced state, the transfer rate is adjusted by a preset feedback control algorithm; the control quantity of the feedback control algorithm is determined based on the pressure index, the error of the network card buffer occupancy rate relative to the target value, and the error of the disk buffer occupancy rate relative to the target value.

[0009] According to the adjusted transfer rate, the network flow data packets temporarily stored in the network card buffer are transferred to the disk buffer.

[0010] Assign corresponding hard disk identifiers to network stream data packets in the disk buffer, and write the network stream data packets to the corresponding hard disks in batches.

[0011] According to another aspect of this application, a non-transitory computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above method.

[0012] According to another aspect of this application, an electronic device is also provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0013] The present invention has at least the following beneficial effects:

[0014] The network flow data packet balanced disk write method of the present invention calculates a pressure index characterizing the overall system balance by real-time monitoring of the occupancy status of the network card buffer and disk write buffer, and combining the rates of the three key links of data reception, transfer and disk write. This achieves a global and quantitative perception of the health status of the data pipeline. Based on this pressure index, the data transfer rate of the buffer area is dynamically and accurately adjusted through a preset feedback control algorithm, enabling the system to intelligently cope with network traffic bursts and hard disk I / O fluctuations. This effectively avoids buffer overflow or idleness caused by imbalance in local links, and significantly reduces the risk of data packet loss. At the same time, this balancing mechanism ensures the smooth flow of data in the entire process of reception, buffering and disk write, maximizing the hardware I / O potential, thereby significantly improving the overall system data throughput, reducing processing latency, and enhancing the operational stability and reliability under high load or heterogeneous performance environments. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1A flowchart of a method for balanced disk placement of network flow data packets provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that, based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Furthermore, this device and / or practice the method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0019] Example 1:

[0020] The following will refer to Figure 1 The flowchart shown illustrates a method for balanced disk persistence of network flow data packets, introducing one such method.

[0021] The method for balanced disk delivery of network flow packets may include the following steps:

[0022] R100 monitors and acquires the system's first real-time occupancy rate of the network card cache, the second real-time occupancy rate of the disk cache, the current network card data receiving rate, the current data transfer rate from the network card cache to the disk cache, and the current aggregate disk write rate of the hard disk array.

[0023] In this embodiment, the first real-time occupancy rate of the network interface card (NIC) buffer is achieved as follows: the NIC buffer is implemented as a fixed-size circular buffer, allocated in memory. Two pointers are maintained: a producer pointer (write position) and a consumer pointer (read / transfer position).

[0024] Real-time occupancy rate = (Producer pointers - Consumer pointers) mod total buffer capacity / total buffer capacity. To avoid modulo operations, the number of used slots and the total number of slots can be recorded directly. Occupancy rate is usually expressed as a percentage.

[0025] For example, if the total capacity of the network card buffer is 1GB (i.e., 1,073,741,824 bytes), and 256MB (268,435,456 bytes) is currently used, then the first real-time occupancy rate is 25%.

[0026] The second real-time occupancy rate of the disk write-to-buy buffer is implemented as follows: The disk write-to-buy buffer is typically organized as a collection of multiple linked lists or queues, each queue corresponding to a hard drive, but all sharing a fixed memory pool. A global variable is maintained to record the total memory occupied by all data packets awaiting disk write-to-buy. Second real-time occupancy rate = Total currently allocated memory / Total size of the disk write-to-buy buffer memory pool.

[0027] The current network card data reception rate is achieved as follows: A counter is set in the network card driver or DPDK PMD (polling mode driver) layer to accumulate the number of bytes received per second. A sliding time window (e.g., 1 second) is used for smoothing. Rate = Total number of bytes received in the past second / 1 second.

[0028] The current data transfer rate is achieved as follows: In the thread transferring data from the network card buffer to the disk buffer, a counter is set to accumulate the number of bytes transferred per second. A sliding time window is also used for calculation. Rate = Total number of bytes transferred in the past second / 1 second.

[0029] The current aggregated write rate of the hard drive array is achieved as follows: A performance counter is created for each hard drive. In the callback after each asynchronous write operation, the number of bytes successfully written is incremented. Aggregated write rate = (Total number of bytes successfully written to all hard drives in the past second) / 1 second.

[0030] In Linux, the amount of data written within an interval can be obtained by reading the number of write sectors for each hard drive in / proc / diskstats, multiplying it by the sector size (e.g., 512 bytes), and then calculating the difference.

[0031] This step provides the system with accurate, real-time "health check data." By simultaneously monitoring the flow rate at the data pipeline's inlet (receiving), intermediate stages (transfer), and outlet (disk landing), as well as the "water level" of two key buffer zones, the system can perceive its own operational status as if it possesses a nervous system. This comprehensive monitoring forms the basis for any subsequent intelligent decision-making, making the system no longer a black box, but completely observable and measurable.

[0032] R200 determines the pressure index to characterize the overall system equilibrium state based on the mismatch between the network card data receiving rate, transfer rate and aggregation disk write rate, the first deviation between the real-time occupancy rate of the network card buffer and the preset target occupancy rate, and the second deviation between the real-time occupancy rate of the disk write buffer and the preset target occupancy rate.

[0033] Furthermore, the mismatch between the network card's data reception rate, transfer rate, and aggregated disk write rate is determined through the following steps:

[0034] R210 calculates the square of the difference between the network card's data reception rate and transfer rate, and the square of the difference between the transfer rate and the aggregation disk transfer rate.

[0035] R211 is obtained by summing the two squared values ​​and taking the square root to get a total rate deviation value.

[0036] R212, normalize the total rate deviation by dividing the network card data receiving rate to obtain the degree of mismatch.

[0037] In this embodiment, R in R is the data reception rate of the network card. trans R is the transfer rate. out Given the polymerization drop rate; calculate the sum of squares of the differences between any two rates: .

[0038] Mismatch To normalize, it can be divided by a reference rate, for example... (if If the value is 0, a local minimum is used to avoid division by zero, resulting in the normalized value of M. .

[0039] Furthermore, the first deviation is obtained through the following steps:

[0040] R220, calculate the first absolute difference between the first real-time occupancy rate and the preset target occupancy rate.

[0041] The system first calculates the absolute difference between the first real-time network interface card (NIC) buffer occupancy rate (U_nic) and the preset target occupancy rate (U_target). It reads the latest U_nic value (e.g., 0.75, or 75%) from the monitoring module. It also reads the preset U_target value (e.g., 0.5, or 50%) from the system configuration. The first absolute difference is then calculated: Δ1 = |U_nic - U_target|. This first absolute difference operation ensures the difference is always non-negative, as subsequent calculations only consider the magnitude of the deviation, not its direction (too high or too low a deviation is also undesirable).

[0042] R221, divide the first absolute difference by the preset target occupancy rate to obtain a relative deviation value.

[0043] Dividing the first absolute difference Δ1 by the preset target occupancy rate U_target yields a normalized relative deviation value δ = Δ1 / U_target. The relative deviation value δ indicates how many times the current deviation is equivalent to the target value. δ = 0.5 indicates that the deviation is 50% of the target value.

[0044] R222, the relative deviation value is input into a preset mathematical function for processing, the mathematical function is configured to assign a higher penalty weight to larger relative deviation values.

[0045] In this embodiment, the preset mathematical function can be f(x) = x². This function is a non-linear penalty function, which has a fast calculation speed.

[0046] R223, the output value of the mathematical function is used as the first deviation.

[0047] The first deviation is a dimensionless scalar that will be used to calculate the stress index later. The larger this value, the more serious the deviation of the network card's cache occupancy status from the ideal target, and due to the effect of the nonlinear function, the "badness" of the serious deviation is amplified.

[0048] In this embodiment, when the network card buffer occupancy rate deviates significantly from the target (e.g., about to overflow or severely idle), the nonlinear function outputs a significantly amplified deviation value. This causes the stress index to rise sharply, triggering the control algorithm to take more decisive and faster corrective measures (such as drastically adjusting the transfer rate), effectively preventing buffer overflow or system idleness.

[0049] For normal fluctuations within a small range around the target value, the output of the nonlinear function increases slowly. This means that the system will not "overreact" to small, harmless fluctuations, avoiding frequent oscillations in control commands and improving the stability and smoothness of the system.

[0050] Furthermore, the second deviation is determined through the following steps:

[0051] R230, calculate the second absolute difference between the second real-time occupancy rate and the preset target occupancy rate.

[0052] The system calculates the second absolute difference between the second real-time occupancy rate (U_buf) of the disk buffer and the preset target occupancy rate (U_target_buf), which is Δ_buf = |U_buf - U_target_buf|.

[0053] R231, compare the second absolute difference with a preset allowable deviation range. If the second absolute difference does not exceed the allowable deviation range, then determine that the deviation between the real-time occupancy rate of the disk buffer and the preset target occupancy rate is zero.

[0054] The system presets a tolerance range (ε), which is a small positive number representing the safe range within which the disk buffer occupancy rate is allowed to fluctuate around the target value. For example, ε = 0.1 (i.e., 10%). This means that when the occupancy rate is within ±10% of the target value (i.e., 40%~60%), it is considered a normal fluctuation.

[0055] R232, if the second absolute difference exceeds the allowable deviation range, the second deviation is calculated based on the size of the excess portion through a preset nonlinear mapping relationship; wherein, the growth rate of the second deviation increases with the increase of the excess portion.

[0056] Calculate the excess: δ_exceed = Δ_buf - ε. The nonlinear mapping function can be a square function or an exponential function, causing the deviation to increase rapidly with the excess.

[0057] Square function: D_buf = (δ_exceed) 2 The calculation is simple, and the penalty is moderate.

[0058] Exponential function: D_buf=exp(α4×δ_exceed)-1, where α4 is the sensitivity coefficient, which has a stronger penalty.

[0059] The disk cache directly faces disk I / O, and its volatility is typically greater than that of the network card cache. The introduction of a tolerance range allows the system to distinguish between normal fluctuations and abnormal deviations, avoiding unnecessary adjustments to minor fluctuations, reducing frequent oscillations in the control system, and improving overall stability.

[0060] When the disk buffer occupancy rate deviates significantly (e.g., close to full or empty), the nonlinear mapping will cause the deviation to increase sharply, thus dominating the system stress index and triggering the control algorithm to take strong corrective measures (e.g., significantly adjusting the transfer rate, initiating emergency disk write, etc.) to effectively prevent data backlog or interruption.

[0061] Furthermore, the pressure index is determined through the following steps:

[0062] R240 calculates the quantized values ​​of the rate mismatch, network interface card (NIC) buffer deviation, and disk write-to-disk buffer deviation, respectively. The rate mismatch is calculated based on the difference between any two of the NIC data receiving rate, transfer rate, and aggregated disk write-to-disk rate. The NIC buffer deviation is calculated based on a first deviation, and the disk write-to-disk buffer deviation is calculated based on a second deviation.

[0063] Calculation of rate mismatch (R_mismatch):

[0064] R_mismatch = sqrt((R_in - R_trans)² + (R_trans - R_out)²); To make this term consistent with the other two terms (ranging from 0 to 1), it can be divided by a reference standard to obtain the normalized R_mismatch'. This is used to quantify the consistency of data flow rates in the "receive-transfer-disk" pipeline.

[0065] Calculation of NIC cache deviation (D_nic): The first deviation D_nic is directly calculated using steps R220-R223. This value already includes a non-linear penalty and typically ranges from [0, 1+). It is used to quantify the degree to which the first-level cache (NIC cache) occupancy deviates from the ideal target.

[0066] The calculation of the disk cache deviation (D_buf): The second deviation D_buf is directly obtained from steps R230-R232. This value is 0 within the tolerance range, and is obtained through non-linear mapping when it exceeds the tolerance range. It is used to quantify the degree to which the occupancy rate of the second-level cache (disk cache) deviates from the ideal target.

[0067] R241 assigns a first weighting coefficient to the rate mismatch item, a second weighting coefficient to the network card buffer deviation item, and a third weighting coefficient to the disk buffer deviation item; the sum of the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient is a fixed value; when the system starts up or there is a sudden change in traffic, the first weighting coefficient is greater than the second weighting coefficient and the third weighting coefficient; when the system is running stably, both the second weighting coefficient and the third weighting coefficient are greater than the first weighting coefficient.

[0068] Let the first weighting coefficient be W_m (corresponding to the rate mismatch), the second weighting coefficient be W_n (corresponding to the network card buffer deviation), and the third weighting coefficient be W_b (corresponding to the disk buffer deviation). W_m+W_n+W_b=1 (a fixed value, such as 100%) to ensure the stability of the stress index scale.

[0069] "Startup" refers to the first N seconds (e.g., 30 seconds) after system initialization; "traffic surge" can be determined by detecting whether the derivative (rate of change) of the network card's receive rate R_in exceeds a threshold, for example, |dR_in / dt| > 200MB / s². W_m is set to a relatively large value (e.g., 0.6), while W_n and W_b are set to relatively small values ​​(e.g., 0.2 each). The logic is: at this stage, the system's primary task is to quickly establish or restore the balance of the data flow, avoiding large blockages or interruptions in the pipeline; therefore, rate matching is of greater concern.

[0070] The system has been running beyond the startup phase and no sudden traffic spikes have been detected for a continuous period (e.g., 60 seconds), while the pressure index remains low (e.g., <0.2). W_m is set to a relatively small value (e.g., 0.2), while W_n and W_b are set to relatively large values ​​(e.g., 0.4 each). The logic is: at this stage, the flow rate is basically balanced, and the system's primary goal shifts to fine-grained cache management to prevent any cache overflow or idleness, thereby ensuring service stability and low latency. Therefore, more attention is paid to the cache status.

[0071] To avoid sudden changes in the pressure index caused by abrupt changes in weights, which could lead to oscillations in control commands, the weight switching should be done smoothly (e.g., adjusting by 0.1 every 10 seconds to gradually reach the target value).

[0072] R242, multiply the rate mismatch, network card cache deviation, and disk cache deviation by their respective weighting coefficients, and add the three products together to obtain the stress index.

[0073] This pressure index determination method, through a three-step strategy of "itemized quantization, dynamic weighting, and linear synthesis," achieves accurate and adaptive assessment of the state of complex systems, bringing significant benefits: First, it transforms the abstract system health into a concrete and calculable scalar, providing a clear and unified decision-making basis for upper-level control algorithms. Second, by designing separate quantification methods for the rate matching term and the buffer deviation term (the former focusing on the overall flow rate difference, and the latter introducing nonlinear penalties and tolerance intervals), it accurately characterizes the imbalance features of different links. Most importantly, the innovative dynamic weighting mechanism enables the system to possess "state awareness" capabilities, prioritizing pipeline smoothness during startup or abrupt changes, and focusing on buffer safety during stable periods. This adaptive switching of strategies allows the system to respond nimbly to sudden flow events while maintaining fine-grained operation in steady state, greatly enhancing the system's robustness and environmental adaptability. The entire calculation process involves only basic arithmetic operations with extremely low overhead, ensuring that the core judgment logic can be executed in real time at high frequencies (e.g., 10Hz), laying a solid foundation for achieving closed-loop intelligent flow control.

[0074] R300 determines whether the system is in equilibrium based on the numerical range of the pressure index.

[0075] The system presets one or more pressure index thresholds to define different state ranges.

[0076] Threshold setting: Typically, two thresholds are set, for example... , .

[0077] Status judgment:

[0078] like The system is determined to be in equilibrium. At this point, the system is running smoothly and requires no adjustment or only minor tweaking.

[0079] like The system is determined to be in a state of slight imbalance. The system needs to activate its adjustment mechanism.

[0080] like The system is determined to be in a state of severe imbalance. Aggressive measures are required and may trigger an alarm.

[0081] To avoid frequent state switching around the threshold, a hysteresis mechanism can be added. For example, the threshold for transitioning from equilibrium to slight imbalance can be set to 0.12, and the threshold for recovering from slight imbalance to equilibrium can be set to 0.08.

[0082] By using threshold judgments, the system achieves a clear understanding and classification of its own state. This avoids the control system overreacting to minor, normal fluctuations (preventing "neuroticism"), while ensuring decisive intervention when real problems arise. The tiered state also provides a basis for subsequently adopting adjustment strategies of varying degrees of intensity.

[0083] R400: If the system is not in a balanced state, the transfer rate is adjusted through a preset feedback control algorithm; the control quantity of the feedback control algorithm is determined based on the pressure index, the error of the network card buffer occupancy rate relative to the target value, and the error of the disk buffer occupancy rate relative to the target value.

[0084] When the system determines that it is not in equilibrium (the pressure index σ exceeds the threshold), the feedback control algorithm is activated. Its goal is to calculate the adjustment (ΔR_trans) of the transfer rate (R_trans) to bring the system back to equilibrium. The algorithm's input signals incorporate three key feedbacks:

[0085] Stress index (σ): A scalar measure of the overall system imbalance. The objective is to minimize σ.

[0086] Network interface card (NIC) buffer error (e_nic): defined as e_nic = U_target_nic - U_nic. When U_nic > U_target (over-occupancy), e_nic is negative, indicating that R_trans needs to be increased to free up the buffer.

[0087] Disk buffer error (e_buf): defined as e_buf = U_target_buf - U_buf. When U_buf > U_target (over-utilization), e_buf is negative, indicating that R_trans needs to be reduced to alleviate downstream pressure.

[0088] Furthermore, the feedback control algorithm includes:

[0089] R410 calculates the adjustment amount of the transfer rate based on the error and pressure index changes of the network card buffer occupancy rate and disk buffer occupancy rate relative to the target value between the current time and the previous time, through a weighted combination of proportional, integral, and derivative components; wherein, the proportional component is proportional to the current error, the integral component is proportional to the historical cumulative error, and the derivative component is proportional to the error change rate.

[0090] Synthesis of error signals (inputs of proportional and integral terms):

[0091] The three input signals (σ, e_nic, e_buf) are combined into a single comprehensive error signal (e_combined), which serves as the direct input to the proportional (P) and integral (I) components of the PID controller. e_combined = k_σ × σ + k_n × e_nic + k_b × e_buf; k_σ, k_n, and k_b are the combined weights. Typically, k_σ has a higher weight for faster response to system imbalances. The weights (k_n, k_b) of e_nic and e_buf have opposite signs because their control directions are opposite (accelerating when the network card buffer is full, and decelerating when the disk buffer is full). For example, k_σ can be set to 0.5, k_n to 0.3, and k_b to -0.2.

[0092] Synthesis of rate of change signals (input of the differential term):

[0093] The differential (D) stage requires the rate of change of error. The rate of change of the combined error (de_combined) and the rate of change of the stress index (dσ) are calculated as differential inputs, the latter of which can predict the deterioration trend of the system more quickly.

[0094] de_combined = (e_combined(t) - e_combined(t-1)) / Δt; dσ = (σ(t) - σ(t-1)) / Δt; The differential inputs can be synthesized as: d_input = k_d1 × de_combined + k_d2 × dσ. k_d2 is usually positive and sensitive because a rapid increase in the pressure index is a danger signal.

[0095] PID three-term calculation and weighted combination:

[0096] Proportional term (P): P_out = Kp × e_combined(t). Directly responds to the current "comprehensive discomfort".

[0097] Integral term (I): I_out = Ki × ∑(e_combined × Δt). The sum of historical accumulated errors, used to eliminate steady-state errors (i.e., the system being in a state of slight imbalance for a long period). To prevent integral saturation (windup), upper and lower limits for the output need to be set.

[0098] Differential term (D): D_out = Kd × d_input. Suppresses system oscillations, providing "predictive" damping.

[0099] Output synthesis: The final transfer rate adjustment ΔR_trans = P_out + I_out + D_out.

[0100] Example parameter tuning (empirical values): Kp=50 (MB / s per unit error), Ki=5, Kd=20. k_d1=1.0, k_d2=2.0. These parameters can be optimized in a real system.

[0101] Apply adjustment amount and limit:

[0102] R_trans_new = R_trans_old + ΔR_trans; A hard limit must be imposed on R_trans_new:

[0103] Lower bound (Min): A value that guarantees basic data flow, such as 10MB / s.

[0104] Maximum (Max): The theoretical maximum value, which is the minimum value among the network card buffer memory bandwidth, disk buffer receiving capacity, and current network card receiving rate R_in. It is usually slightly less than R_in.

[0105] Ultimately, the restricted R_trans_new was used to control the token bucket generation rate of the transfer thread.

[0106] The implementation of this feedback control algorithm achieves adaptive, forward-looking, and stable closed-loop control of the data transfer rate by ingeniously integrating multi-dimensional system states (global pressure index, two-level buffer errors and their changing trends) into an improved PID control framework. Its core advantages are: First, multi-signal fusion avoids the limitations of a single indicator. For example, when both network card buffer and disk buffer are high, the algorithm can identify a systemic overload rather than a local problem through the pressure index and derivative term, thus making a more reasonable compromise adjustment. Second, the clear division of labor among the three PID terms ensures control quality—the proportional term provides fast response, the integral term eliminates long-term deviations, and the derivative term suppresses overshoot and oscillations. This allows the system to smoothly converge to the equilibrium point, avoiding the "stop-and-go" jitter caused by manual control or simple threshold control. Third, dynamic synthetic weights (such as k_σ, k_n, k_b) allow the system to adjust the focus of the control strategy at different stages (such as startup, steady state, and sudden changes), enhancing environmental adaptability. Ultimately, the algorithm acts like a tireless intelligent speed regulator, ensuring that the data stream remains efficient and stable throughout the complex pipeline of receiving, buffering, and disk storage, maximizing system throughput while minimizing latency and packet loss risks.

[0107] R500, according to the adjusted transfer rate, transfers network flow data packets temporarily stored in the network card buffer to the disk buffer.

[0108] In the transfer thread, implement a token bucket algorithm to control the rate. The token bucket capacity is set to RB (e.g., 100MB), and the token generation rate is exactly [value missing]. (Unit: bytes / second). Before transferring a data packet from the network interface card (NIC) buffer to the disk buffer, a token equal to the size of the data packet needs to be obtained from the bucket. If there are insufficient tokens, the transfer thread sleeps until enough tokens accumulate in the bucket.

[0109] After the transfer thread is awakened, it reads one or more packets from the consumer pointer position of the network interface card (NIC) buffer (until the token runs out or the end of the buffer is encountered). The packets and their metadata are copied to a pre-allocated memory location in the disk cache. The NIC's consumer pointer and the disk cache usage statistics are updated.

[0110] Because the transfer is performed strictly in the order it is read from the network card buffer, and the network card buffer itself is FIFO, the timestamp order of the data packets is preserved.

[0111] The R600 assigns a corresponding hard disk identifier to network stream data packets in the disk buffer and writes the network stream data packets to the corresponding hard disk in batches.

[0112] In this embodiment, the data writing method in Embodiments 3 to 5 can be used to assign a corresponding hard disk identifier to the network stream data packets in the disk cache area, and write the network stream data packets to the corresponding hard disk in batches, which will not be elaborated here.

[0113] Example 2:

[0114] Based on the above embodiment one, when gigabit network cards and 10-gigabit network cards are used interchangeably, in order to enable the 10-gigabit network card to perform at its best, the following method can be used, that is, the method in embodiment one can also include the following steps:

[0115] S100 retrieves the number of idle CPU cores on the server, the type of network interface card (NIC) on the server, and the maximum number of queues supported by each NIC.

[0116] In this embodiment, the method is applied to the scenario of network flow data packets being written to disk. The data written to disk provides a data foundation for subsequent auditing. The total number of physical CPU cores can be obtained by parsing / proc / cpuinfo or using the lscpu command. After subtracting the cores that are fixedly occupied by critical system services or user processes (such as cores isolated by the taskset or isocupus kernel parameters), a list of truly available free cores can be obtained. The lspci command is used in conjunction with the ethtool command to identify these cores. For example, ethtool <NIC name>|grep "Speed" can obtain speed information (1000Mb / s is gigabit, 10000Mb / s is 10 gigabit). The "Combined" value in "Pre-set maximums" (i.e., the maximum number of queues supported by the hardware) can be viewed by ethtool -l <NIC name>.

[0117] Example: The server has 16 physical cores (CPU0~15), of which CPU0~3 are reserved by the system, and CPU4~15 are 12 idle cores. Two network cards are detected: eth0 (Gigabit, maximum queue size 4) and eth1 (10 Gigabit, maximum queue size 8).

[0118] S200, if the server's network card type includes gigabit network cards and 10-gigabit network cards, and the number of idle CPU cores is less than or equal to a first preset number, then the queue number of each gigabit network card and each 10-gigabit network card is set to the minimum value between the maximum number of queues supported by each network card and the number of idle CPU cores.

[0119] Furthermore, the first preset quantity is 8.

[0120] In this embodiment, the first preset number is an empirical threshold, for example, set to 8. This means that when the number of idle cores is less than or equal to 8, core resources are considered scarce, and it is impossible to allocate more cores to the 10 Gigabit network card, so a balancing strategy is adopted.

[0121] Calculate the minimum value for each network interface card (NIC) (maximum number of queues, number of idle cores). Example: Assume the number of idle cores is 4, the maximum queue count for a gigabit NIC is 4, and the maximum queue count for a 10-gigabit NIC is 8. If we directly take the minimum value, then the gigabit queue count = 4, the 10-gigabit queue count = 4, and the total queue count 8 > 4. In this case, we can adjust the gigabit queue count to 4 and the 10-gigabit queue count to 4.

[0122] Furthermore, the first preset quantity is determined through the following steps:

[0123] S210, retrieve the number of gigabit network cards and 10-gigabit network cards in the server.

[0124] List all network interfaces using system commands or programming interfaces. For each identified physical interface (excluding virtual network cards such as virbr0, docker0, etc.), use the ethtool tool to query its "supported speed" and "current speed". Calculate the number of gigabit (1G) and 10 Gigabit (10G) network cards respectively.

[0125] This step forms the factual basis of the entire resource allocation scheme. By automatically detecting and accurately determining the type and quantity of physical network interface cards (NICs), errors or omissions that might occur with manual configuration are avoided, ensuring the objectivity and accuracy of subsequent core resource calculations. This allows the scheme to adapt to different server hardware configurations, whether it's a pure gigabit, pure 10 gigabit, or hybrid environment, correctly initiating the corresponding optimization logic.

[0126] S220: Based on the performance requirements of each type of network card, allocate different baseline core requirements to each gigabit network card and each 10-gigabit network card; wherein, the baseline core requirements are obtained by querying a preset performance parameter table, which is determined comprehensively based on the server's CPU model, network card model and service performance targets.

[0127] A performance parameter table is a crucial configuration or database where each entry defines a baseline number of CPU cores required for a single network interface card (NIC) to handle traffic under specific conditions. Key dimensions include:

[0128] CPU model: CPUs of different generations and architectures have different single-core processing capabilities (such as Intel Xeon Gold 6348 and AMD EPYC 7763).

[0129] Network card model: Different network card models have different hardware offloading capabilities, queue efficiency, and driver optimization levels (such as Intel i350-T2 and Intel X710-DA2).

[0130] Business performance goals: These are application-level requirements, such as: Line-speed forwarding: Requires CPU processing power to keep pace with network card port bandwidth. Deep packet inspection: Requires additional core resources for complex pattern matching. Low-latency transactions: May require higher single-core clock speeds and less queue sharing.

[0131] Based on the actual CPU model and network card model of the current server, find the most matching entry in the performance parameter table and read its defined "baseline cores per card".

[0132] This step uses a pre-defined parameter table to scientifically map hardware performance (CPU, network card) to software objectives (business requirements). This allows the pre-allocation of core resources to closely match the objective needs of actual workloads, reserving sufficient processing power for high-performance network cards (such as 10 Gigabit cards) under computationally intensive tasks, while avoiding waste caused by allocating too many resources to low-load network cards, thus achieving precise resource planning from the source.

[0133] S230, add the sum of the baseline core requirements of all gigabit network cards in the server to the sum of the baseline core requirements of all 10-gigabit network cards to obtain the first preset number.

[0134] This step is a simple arithmetic aggregation, which summarizes the individual requirements of each network card obtained in S220 into the total minimum guaranteed requirements of the entire server network subsystem for CPU cores.

[0135] This step generates the critical decision threshold for the entire solution—the first preset number. This value is no longer a guess based on experience, but a quantitative derivation based on hardware capabilities and business objectives. Its core function is to provide the system with a clear boundary for judging whether resources are sufficient. When the number of idle cores in the system is less than or equal to this value, it indicates that CPU resources are scarce or just enough, and a conservative balanced allocation strategy must be adopted (such as the minimum value (maximum queue number, number of idle cores) in the original solution) to ensure the basic functions of all network cards. When the number of idle cores is greater than this value, it indicates that the CPU is a "sufficient resource," and a more aggressive optimization strategy (such as proportional tilt allocation) can be enabled, allowing high-performance network cards such as 10 Gigabit Ethernet to utilize the surplus cores to further improve performance. This achieves adaptive switching of resource allocation strategies, enabling the system to make optimal decisions in both resource-constrained and resource-abundant states.

[0136] The S300 binds each queue of each gigabit network interface card (NIC) and each 10 gigabit network interface card (NIC) to an idle CPU core.

[0137] Furthermore, the S300 includes:

[0138] S310 identifies the NUMA node to which each gigabit network interface card (NIC) and each 10 gigabit NIC belong.

[0139] In a NUMA architecture, CPUs, memory, and PCIe devices (such as network cards) belong to specific physical nodes. Accessing resources across nodes introduces significant latency; the PCI bus address of a network card (such as 0b:00.0) can be found using lspci, and then the NUMA node to which each gigabit network card and each 10-gigabit network card belongs can be identified by following the / sys / class / net / <network card name> / device symbolic link or directly querying the / sys / bus / pci / devices / <PCI address> / numa_node file.

[0140] For example: A dual-socket server (2 CPUs, i.e., 2 NUMA nodes: Node0 and Node1). Learned from lspci that eth0 (gigabit) is at PCI address 03:00.0, and querying its numa_node file shows 0; eth1 (10-gigabit) is at PCI address 84:00.0, and querying its numa_node file shows 1. This indicates that eth0 is closer to the CPU and memory of Node0, and eth1 is closer to the CPU and memory of Node1.

[0141] This step provides crucial hardware topology knowledge for the entire optimization process. By precisely identifying the NUMA node where the network card is located, the system can know the "local path" for data to be written from the network card to memory via DMA. This is the basis for implementing the subsequent "proximity binding" strategy, avoiding cross-node memory access caused by blind binding, and creating conditions for reducing data stream processing latency from the source.

[0142] S320, preferentially bind the queues of each gigabit network card and each 10-gigabit network card to the idle CPU cores corresponding to the same NUMA node as the network card.

[0143] After determining the home node of the network card, the system needs to filter out the list of idle CPU cores within that node and preferentially bind the interrupts of the network card queues to these cores. The numactl --hardware or lscpu command can be used to view the system NUMA topology. From the global list of idle cores obtained in step S100, filter out the cores belonging to the NUMA node where the target network card is located. For each queue of the network card, allocate a core from the pool of idle cores of its所属 NUMA node and use the interrupt affinity setting command to bind it. The DPDK technology can also be used to implement the binding operation of idle CPU cores.

[0144] This step ensures that once network interface card (NIC) DMA data is in local memory, the interrupt service routine handling that process and subsequent application threads (if also scheduled on the same node) can access this data at the fastest possible speed by fixing the interrupt handling process of the NIC queue to the CPU core of its local NUMA node. This minimizes the high latency (typically up to 2-3 times) caused by cross-node memory access, significantly reduces packet processing latency, and avoids cross-node bus congestion, thus providing a crucial guarantee for line-speed processing of high-speed NICs such as 10 Gigabit Ethernet.

[0145] S330: If there are insufficient idle CPU cores on the same NUMA node, a cross-node binding queue is used.

[0146] When the number of queues that a network interface card (NIC) needs to be configured exceeds the number of available idle CPU cores within its NUMA node, cores from other nodes (i.e., "remote nodes") must be used. You can choose other NUMA nodes with lower current system load or more idle cores. Alternatively, you can simply select from the global list of idle cores in sequence.

[0147] This step is crucial for ensuring system robustness and functional integrity. It handles boundary cases in hardware resource allocation, ensuring that even when the computing resources of a NUMA node are relatively scarce, all network interface card queues can be bound and enabled through cross-node scheduling, avoiding the situation where queues are discarded or processing power is reduced due to insufficient cores. Although cross-node binding introduces additional access latency, the performance loss is usually far outweighed by the throughput reduction caused by queues not being enabled. This is a necessary strategy to strike a balance between "optimal performance" and "maximum availability," ensuring the universality of the configuration scheme across various hardware configurations.

[0148] S400, if the number of idle CPU cores is greater than the first preset number, then all the idle CPU cores are divided into a first idle CPU core group and a second idle CPU core group according to a preset ratio; the number of idle CPU cores in the first idle CPU core group is less than the number of idle CPU cores in the second idle CPU core group.

[0149] Furthermore, the preset ratio is 1 / 3.

[0150] The number of cores in the two groups is calculated based on the ratio. For example, if there are 12 idle cores, they are divided in a 1:3 ratio: 3 cores in the gigabit group and 9 cores in the 10-gigabit group.

[0151] Furthermore, the preset ratio is determined through the following steps:

[0152] S410: Obtain the average throughput and average packet loss rate of the gigabit network card group and the average throughput and average packet loss rate of the 10-gigabit network card group within a preset historical time period.

[0153] The system periodically (e.g., every second) reads the kernel statistics file for each network interface card (NIC). Throughput is obtained by summing the `rx_bytes` (received bytes) and `tx_bytes` (sent bytes) values ​​under ` / sys / class / net / <interface> / statistics / `; packet loss is obtained by summing the `rx_dropped` and `tx_dropped` values. The total number of packets can be calculated using `rx_packets` and `tx_packets`. The metrics for all gigabit NICs are summed separately to obtain the total throughput, total packet loss, and total number of packets for the gigabit NIC group; the same applies to the 10-gigabit NIC group. Within a preset historical time period (e.g., the most recent 300 seconds), the group-level metrics (throughput, packet loss rate) calculated for each sampling period are arithmetically averaged to obtain the average throughput and average packet loss rate. The packet loss rate is calculated as: Packet Loss Rate = Total Packet Losses / Total Number of Packets.

[0154] The S420 calculates the proportion of input / output wait time by reading the system's global CPU time status file.

[0155] Periodically (e.g., every 10 seconds), read the first line of the ` / proc / stat` file (the cumulative system-level CPU time). Pay attention to the `iowait` field, which represents the CPU's wait time for I / O to complete. Take the difference between two consecutive `iowait` samples and divide it by the difference in total CPU time within that time interval (total CPU time = user + nice + system + idle + iowait + irq + softirq + steal). Formula: iowait ratio = Δiowait / Δtotal CPU time.

[0156] S430: If the proportion of input / output waiting time exceeds the preset proportion threshold for several consecutive cycles, the type of the current business being run by the server is determined to be input / output intensive; otherwise, it is determined to be compute intensive.

[0157] Set a percentage threshold (e.g., 30%) and a number of consecutive cycles (e.g., 5 cycles). If the percentage of continuously calculated iowait exceeds the threshold, the current service is determined to be I / O intensive; otherwise, it is compute intensive.

[0158] By analyzing CPU time distribution, this step identifies the dominant business load characteristics at the system level. This provides crucial contextual information for subsequent ratio adjustments, ensuring that the core allocation strategy aligns with overall business characteristics and preventing resource allocation from becoming disconnected from actual needs.

[0159] S440, based on the average throughput and average network packet loss rate of the gigabit network card group and the 10-gigabit network card group, as well as the type of service currently running on the server, the preset base ratio is adjusted to obtain the preset ratio.

[0160] Furthermore, step S440 includes the following steps:

[0161] S441, determine the first adjustment coefficient based on the average throughput of the gigabit network card group and the 10-gigabit network card group; the larger the ratio of the average throughput of the gigabit network card group to the average throughput of the 10-gigabit network card group, the larger the first adjustment coefficient.

[0162] The first adjustment factor (f1) reflects the relative load of throughput. It is calculated as the ratio of the average throughput of gigabit and 10-gigabit networks, R_t = gigabit throughput / 10-gigabit throughput. F1 is set to 1 + α × R_t, where α is a gain factor (e.g., 0.5). A larger R_t indicates a heavier relative load on gigabit networks, resulting in a larger f1 and thus increasing the proportion of gigabit networks.

[0163] S442, determine the second adjustment coefficient based on the average packet loss rate of the gigabit network card group and the 10-gigabit network card group; the larger the ratio of the average packet loss rate of the gigabit network card group to the average packet loss rate of the 10-gigabit network card group, the larger the second adjustment coefficient.

[0164] The second adjustment factor (f2) reflects the relative pressure of packet loss rate. The packet loss rate ratio R_l is calculated as: Gigabit packet loss rate / 10 Gigabit packet loss rate. F2 is set to 1 + β × R_l, where β is a gain factor (e.g., 0.2). A larger R_l indicates more severe packet loss on the gigabit side, and a larger f2 is needed to allocate more cores to mitigate packet loss.

[0165] S443, determine the third adjustment factor based on the type of service currently running on the server; the third adjustment factor determined when the type of service currently running on the server is input / output intensive is less than the third adjustment factor determined when the type of service currently running on the server is compute intensive.

[0166] The third adjustment factor (f3) reflects the service type preference. For I / O-intensive systems, f3 = 0.9; for compute-intensive systems, f3 = 1.1. For I / O-intensive systems, the system may need to support high-speed network I / O, so the gigabit ratio should be appropriately reduced (f3 < 1) to prioritize 10 gigabit resources; the opposite is true for compute-intensive systems.

[0167] S444, the product of the first adjustment coefficient, the second adjustment coefficient, the third adjustment coefficient and the preset base ratio is determined as the preset ratio.

[0168] Set an initial allocation ratio as the preset base ratio P_base, for example, gigabit cores: 10-gigabit cores = 1:4 (i.e. gigabit accounts for 20%), denoted as scalar P_base = 0.25 (number of gigabit cores / number of 10-gigabit cores).

[0169] The pre-set ratio P is calculated as P_base × f1 × f2 × f3. This P value represents the adjusted ratio of gigabit to 10-gigabit core counts. In actual partitioning, if the total number of idle cores is N, then the gigabit core count = N × P / (1 + P), and the 10-gigabit core count = N × 1 / (1 + P). If the calculation result is a decimal, the gigabit core count can be rounded up.

[0170] This step utilizes a multi-factor product model to organically integrate network load metrics (throughput, packet loss rate) with system service characteristics (I / O intensive / computation intensive), enabling dynamic and fine-grained adjustments to preset ratios. This makes CPU core allocation no longer static or empirical, but adaptive to real-time workloads, intelligently balancing resource allocation between network cards of different speeds. This improves the processing capacity of high-load network cards while optimizing overall system resource utilization and network performance.

[0171] The S500 sets the number of queues for each gigabit network card to the minimum of the maximum number of queues supported by each gigabit network card and the number of idle CPU cores, and sets the number of queues for each 10-gigabit network card to the minimum of the maximum number of queues supported by each 10-gigabit network card and the number of idle CPU cores.

[0172] Gigabit network card queue count = min(maximum number of Gigabit network cards queues, number of Gigabit network card cores); 10 Gigabit network card queue count = min(maximum number of 10 Gigabit network cards queues, number of 10 Gigabit network card cores).

[0173] S600 binds each queue of each gigabit network card to an idle CPU core in the first idle CPU core group, and binds each queue of each 10 gigabit network card to an idle CPU core in the second idle CPU core group, so that each queue processes the corresponding data stream through an idle CPU.

[0174] The idle CPU cores have been divided into two logical groups according to a preset ratio (e.g., 1:4). For example, assuming there are a total of 16 idle cores (CPU0-15), the first group (Gigabit group) is allocated 4 cores, and the second group (10 Gigabit group) is allocated 12 cores.

[0175] Queue configuration: The number of queues has been set for each gigabit network card and 10-gigabit network card to ensure that the total number of queues for gigabit network cards does not exceed the number of cores in the first group, and the total number of queues for 10-gigabit network cards does not exceed the number of cores in the second group.

[0176] Interrupt identification: The hardware interrupt number (IRQ) corresponding to each queue of each network card has been identified.

[0177] The binding process is achieved by setting interrupt affinity (IRQ Affinity), which fixes a specific interrupt to a specific CPU core for processing.

[0178] Allocation strategy:

[0179] Gigabit network card queues: allocated sequentially from the first group of cores. For example, if the first group of cores is CPU0-3, then the first queue of the gigabit network card will be bound to CPU0, the second to CPU1, and so on.

[0180] 10 Gigabit Ethernet NIC queues: allocated sequentially from the second group of cores. If the second group of cores consists of CPU4-15, then the first queue of the 10 Gigabit Ethernet NIC is bound to CPU4, the second to CPU5, and so on.

[0181] Multi-NIC scenario: If there are multiple gigabit NICs, continue to allocate them in a loop from the first core group; the same applies to 10-gigabit NICs.

[0182] This step achieves several key benefits. First, it allocates more dedicated cores to high-throughput 10 Gigabit network interface cards (NICs), fully matching their high computational demands for packet processing and avoiding performance bottlenecks caused by core contention, thus ensuring the line-speed processing capability of 10 Gigabit links. Second, it isolates NIC traffic of different speeds to different core groups, eliminating interference from low-speed traffic to high-speed traffic and improving the determinism and low latency of high-speed network processing. Third, the one-to-one binding relationship ensures that interrupt handling for each queue is always executed on a fixed core, greatly improving CPU cache hit rate and reducing context switching overhead, thereby significantly improving single-stream processing performance. Finally, this explicit isolation architecture simplifies performance monitoring and fault diagnosis, allowing administrators to clearly observe the load of each core group and quickly locate performance anomalies. Overall, this step is a key technical guarantee for achieving resource optimization and predictable performance in mixed-rate network environments.

[0183] Example 3:

[0184] Based on the above embodiment two, after the network flow data received by the network card is divided into network path data packets, it is written to disk using the following method, that is, after step S600, the following steps may also be included:

[0185] Q100 determines the amount of data to be written to disk in a single operation for each hard drive based on the write speed of each hard drive and the size of the preset disk cache.

[0186] Furthermore, Q100 includes the following steps:

[0187] Q110 defines the write percentage of each hard drive as the ratio of the write speed of each hard drive to the sum of the write speeds of all hard drives.

[0188] Q120 determines the amount of data written to disk per drive by multiplying the write percentage of each hard drive by the size of the preset disk cache area.

[0189] The core of this step is to allocate a reasonable write quota to each hard drive based on their performance differences, serving as the maximum amount of data each hard drive can receive during this round of disk write operations. In implementation, the nominal or measured sustained write rates (in units such as MB / s) of all hard drives to be used first need to be obtained. Assuming there are N hard drives, the write rate of the i-th hard drive is v_i, and the preset total size of the disk write buffer is S (in units such as MB). The amount of data (quota) for each hard drive in a single disk write operation, Q_i, can be calculated as a proportion of its write rate to the total: Q_i = (v_i / Σv) × S.

[0190] In practice, hard drive speed can be tested using system tools (such as hdparm or fio) or by reading manufacturer specifications. Quota calculations are typically performed during system initialization or configuration changes, and the results are stored in memory for later use.

[0191] For example: The system has 3 hard drives: HDD1 (speed 100MB / s), SSD1 (speed 500 MB / s), and SSD2 (speed 300MB / s), with a disk cache size of 900MB. The total speed is 900MB / s. Therefore, the quotas are: HDD1: (100 / 900) × 900 = 100MB; SSD1: (500 / 900) × 900 = 500MB; SSD2: (300 / 900) × 900 = 300MB.

[0192] This step allocates cache space proportional to write speed, allowing high-performance hard drives to obtain a larger data write quota in a single disk write cycle. This fully leverages their I / O potential and prevents low-speed hard drives from becoming the bottleneck for overall throughput. This differentiated quota mechanism lays the foundation for subsequent intelligent data allocation, ensuring that the overall system disk write bandwidth matches hardware capabilities and improving resource utilization efficiency.

[0193] Furthermore, the method also includes:

[0194] Q130 provides real-time data on the current I / O operation queue depth and average write latency for each hard drive.

[0195] Current I / O queue depth: This refers to the number of read and write requests waiting to be processed by the hard drive at the sampling time. In Linux systems, this can be obtained directly by reading the file ` / sys / block / <disk device name (e.g., sda)> / inflight`. Its content typically contains two comma-separated numbers, representing the number of pending read and write requests, respectively. These numbers can be summed to obtain the total queue depth. For more detailed monitoring, the `iostat -x1` command can be used to focus on the `aqu-sz` field (average queue length) or relevant kernel statistics can be directly parsed.

[0196] Average write latency: This refers to the average time (from request issuance to completion confirmation) that the hard drive takes to process each write request over a recent period. In Linux, it can be calculated using the `await` field (average wait time for all I / O operations) output by `iostat -x`, or more precisely, by combining data from ` / proc / diskstats`. The formula is: Average write latency = (Total write operation time) / (Number of write operation completions). The total write operation time can be obtained from the "Cumulative Write Time" field (in milliseconds) in ` / proc / diskstats`.

[0197] Q131, Calculate the real-time load impact factor for each hard disk based on the current I / O operation queue depth and average write latency.

[0198] Normalization: Set a "full load threshold" (Q_max, L_max) for both queue depth (Q) and average write latency (L). For example, based on experience, set Q_max = 32 (queue depth) and L_max = 50ms (latency).

[0199] Calculate the load factor: r_Q = min(Q / Q_max, 1.0); r_L = min(L / L_max, 1.0). When the actual value exceeds the threshold, the load factor is set to 1.

[0200] The comprehensive calculation factor is: F_load = 1.0 - max(r_Q, r_L) × γ. Here, γ is a decay coefficient (e.g., 0.3) used to control the degree of influence of load on the quota. Alternatively, a smoother formula can be used: F_load = exp(-β1 × (α1 × r_Q + (1-α1) × r_L)), where α1 is the weight (e.g., 0.6), and β1 is the sensitivity coefficient.

[0201] This step uses a quantitative mathematical model to transform the abstract concept of "hard drive workload" into a directly usable adjustment coefficient. Its core value lies in achieving intelligent assessment and expression of load pressure. This model not only considers a single metric but also integrates two dimensions: queue depth (reflecting backlog) and latency (reflecting response speed), providing a more comprehensive characterization of the hard drive's true load. By setting thresholds and coefficients, system administrators can flexibly define at what level of load "throttling" should begin, making resource adjustment strategies both automated and strategically optimized.

[0202] Q132, multiply the single disk write data size determined based on the write rate ratio by the real-time load impact factor to obtain the final single disk write data size for each hard drive; wherein, the real-time load impact factor is negatively correlated with the I / O operation queue depth and the average write latency. When the queue depth increases or the write latency increases, the single disk write data size allocated to the corresponding hard drive will be reduced accordingly.

[0203] The base quota Qi_i_base, statically calculated based on the write rate ratio in step Q100, is multiplied by the real-time load impact factor F_load_i calculated in step Q131 to obtain the final effective quota Qi_i_final for this hard drive in this disk write cycle. That is: Qi_i_final = Qi_i_base × F_load_i. Then, Qi_i_final is used to update the initial value of "current single disk write data volume of the hard drive" maintained by the system, which is used for data packet allocation in subsequent steps Q400 and Q500.

[0204] This step is the final execution stage of the dynamic adjustment mechanism, directly feeding back the monitoring and analysis results from the preceding steps into resource allocation decisions. Its core benefit is the realization of "quota elastic scaling based on real-time load." When a hard drive experiences a performance bottleneck, its quota is automatically reduced, guiding the system to allocate more new data streams to currently less busy hard drives, thereby achieving better load balancing and pressure distribution across multiple hard drives. This not only prevents a single hot hard drive from slowing down the overall disk write speed due to overload, but also improves the system's resilience and stability in the face of sudden loads, enabling the entire storage subsystem to handle continuous data streams with smoother and more predictable performance.

[0205] Q200, in response to the total number of network flow data packets cached in the preset disk cache reaching the maximum capacity of the preset disk cache, obtains the network session ID, size and preset unfinished network session ID table corresponding to each network traffic data packet; the unfinished network session ID table includes several unfinished network session IDs, and each unfinished network session ID corresponds to a hard disk ID.

[0206] The system maintains a fixed-size disk write buffer (usually located in memory) to accumulate packets received from the network. When the total amount of data in the buffer reaches its maximum capacity (e.g., 80% or 100%), a disk write schedule is triggered. At this time, the system iterates through the metadata of each packet in the buffer, extracting key information: network session ID (usually a hash value of a 5-tuple or 3-tuple) and packet size (length). Simultaneously, the system reads a globally maintained "unfinished network session ID table," which stores the mapping between currently active session IDs and their assigned disk IDs in key-value pairs. This table can be implemented using a hash table to ensure efficient lookup.

[0207] This step, by setting up a cache full trigger mechanism, enables batch data processing, aggregating scattered small I / O requests into large batches of I / O operations. This significantly reduces disk addressing overhead and write amplification, thereby increasing disk throughput. Simultaneously, it centrally acquires session information for all data packets and provides a unified decision context for subsequent steps, ensuring consistency in allocation strategies.

[0208] Q300: For any network traffic data packet RQ, if the network session ID corresponding to RQ exists in the unfinished network session ID table, then assign the hard disk ID corresponding to the unfinished network session ID that is the same as the network session ID corresponding to RQ in the unfinished network session ID table to RQ; otherwise, determine RQ as a new network session network traffic data packet.

[0209] For each data packet in the buffer, the "Unfinished Network Session ID Table" is queried using its network session ID as the key. If the query finds a match, the disk ID mapped in the table is assigned to the data packet, and it is removed from the decision list. If the query fails, the data packet is marked as a "new session data packet" and placed in an allocation queue, awaiting processing by step Q400. This process ensures that the data flow of existing sessions remains unchanged.

[0210] This step adheres to the "session persistence" principle, ensuring the continuity of all data packets within the same network session in physical storage. This greatly facilitates subsequent session-based data retrieval, integrity verification, or in-depth analysis, avoiding performance degradation caused by data fragmentation. Simultaneously, directly reusing existing allocation results reduces the number of calls to new session allocation logic, lowering computational overhead.

[0211] Q400: For any new network session network traffic data packet YQ, compare YQ with the remaining disk data size corresponding to any hard drive WE. If the remaining disk data size corresponding to WE is greater than the size corresponding to YQ, then assign the hard drive ID corresponding to WE to YQ; otherwise, compare YQ with the next uncompared hard drive.

[0212] When allocating hard drives for each new session data packet, a greedy algorithm based on remaining quota is used. The system maintains a list recording the current "single remaining disk write amount" for each hard drive (initially the quota calculated by Q100). The hard drive list is traversed (usually sorted in descending order of write speed). For the data packet YQ to be allocated, the remaining quota of each hard drive WE is checked sequentially to see if it is greater than the size of YQ. If so, the hard drive ID of WE is assigned to YQ, and the remaining quota of that hard drive is updated: remaining quota = remaining quota - size of YQ. If not, the next hard drive is checked.

[0213] This step enables fine-grained, capacity-based load balancing. By dynamically tracking the remaining write capacity of each hard drive and prioritizing drives capable of receiving current data packets, it ensures that the write load of each hard drive matches its performance quota, preventing the imbalance where some hard drives are filled up prematurely while others remain idle, thereby optimizing the parallel write efficiency of multi-hard drive systems.

[0214] Furthermore, comparing YQ with the remaining data volume for a single disk write operation corresponding to any hard drive WE includes:

[0215] Q410 prioritizes comparison with the hard drive that received the most new session data packets in the previous write cycle. If the remaining write data volume of the hard drive that received the most new session data packets in the previous write cycle is insufficient, then it is compared with the hard drive that has the largest remaining write data volume in the current write cycle. If it is still insufficient, then it is compared with other hard drives in descending order of write speed.

[0216] In this embodiment, a three-level priority intelligent comparison order is introduced to optimize the allocation decision of new session data packets.

[0217] First layer: Prioritization based on historical activity level

[0218] After each disk write operation is completed, the system records the number of new session data packets received by each hard drive in that round (only the first allocated session is counted). These statistics are maintained in a counter and used to guide allocation at the start of the next disk write cycle.

[0219] When it is necessary to allocate a hard disk for a new session data packet YQ, first query the hard disk that received the most new session data packets in the previous disk write cycle (denoted as H_most).

[0220] Check if the current "remaining amount of data to be written to disk in a single run" of H_most (i.e., its real-time adjusted quota minus the amount of data already allocated in this round) is greater than the size of YQ. If yes, allocate YQ to H_most and update its remaining quota; otherwise, proceed to the next level of comparison.

[0221] This strategy leverages the principle of locality of reference in session establishment. In most network environments, the generation of new sessions tends to be clustered in time and space (e.g., a server establishes connections with a large number of clients in a short period). Prioritizing the allocation of new sessions to the most active hard drive from the previous round increases the density of new sessions on the same hard drive, which is beneficial for improving write continuity (reducing head movement or FTL conversion overhead) and may provide better locality for subsequent session-based data analysis. At the same time, this simplifies allocation decisions and avoids scanning all hard drives every time.

[0222] Second layer: Greedy comparison based on the current remaining capacity

[0223] If the remaining quota of H_most is insufficient to accommodate YQ, the system then searches for the hard drive with the largest remaining data write volume in the current single push (denoted as H_max). "Remaining data write volume in a single push" refers to the initial quota after real-time load factor adjustment, minus the total size of all data packets (including persistent and new sessions) already allocated to that hard drive in this round. The system checks if the remaining quota of H_max is greater than the size of YQ. If yes, it allocates the quota; otherwise, it proceeds to the third-level comparison.

[0224] When a historically active hard drive becomes unable to receive new data due to excessive quota consumption, the strategy at this layer shifts to the hard drive with the highest current capacity. This is a real-time load-aware greedy strategy designed to maximize the utilization of the remaining write capacity of each hard drive, avoiding the allocation of new sessions to hard drives that are about to be full, thereby achieving a more balanced load distribution globally. This helps prevent individual hard drives from becoming bottlenecks prematurely, improving the overall throughput and resource utilization of multi-hard drive systems.

[0225] Third layer: Degradation comparison based on basic performance

[0226] If the remaining quotas of H_most and H_max are insufficient to accommodate YQ, the system will check the remaining quotas of the other hard drives in descending order of their write speeds (excluding H_most and H_max, which have already been checked). Once the first hard drive with a remaining quota larger than YQ is found, YQ will be allocated to it immediately.

[0227] If no matching hard drive is found after traversing all hard drives, a fallback strategy is triggered (such as step Q500 in the original scheme, where YQ is allocated to the hard drive with the highest write speed, even if its quota is insufficient).

[0228] This layer's strategy is a performance-oriented degradation solution when the first two layers of optimization fail (usually indicating high overall system load or very large data packets). It compares data packets in descending order of write speed, prioritizing high-performance hard drives. Even if their remaining quota is insufficient, their powerful I / O capabilities allow them to handle excess data at a relatively low cost. This ensures the completeness of system decision-making and the continuity of processing under high pressure, preventing the allocation process from becoming deadlocked. At the same time, this order implicitly follows the intuitive logic of "assigning difficult tasks to the most capable," helping to minimize the impact of abnormal data packets on overall system performance.

[0229] This step integrates three key factors: temporal locality, real-time load, and hardware performance. This ensures that the allocation of new session data packets is both forward-looking (utilizing historical patterns) and flexible in responding to real-time changes, while also guaranteeing system robustness in extreme cases. This strategy significantly reduces the randomness and arbitrariness in the allocation process, improves decision-making quality and execution efficiency, and thus optimizes the overall write load balancing, storage locality, and service stability under high load in multi-disk systems.

[0230] Q500: If the amount of data remaining to be written to each hard drive in a single operation is less than the amount corresponding to YQ, then the disk ID corresponding to the hard drive with the highest write speed will be assigned to YQ.

[0231] When a data packet exceeds the remaining quota of all hard drives, the allocation logic in step Q400 will fail. In this case, the system initiates a fallback strategy: selecting the hard drive with the highest current write rate (which can be pre-sorted or queried in real-time) and assigning its ID to the data packet. After performing this allocation, it is usually necessary to accept that the actual write volume of that hard drive temporarily exceeds its quota. The system can log such events for performance monitoring reference.

[0232] This step serves as a system resilience mechanism, ensuring that the allocation process can continue even in extreme situations (such as oversized packets or severely depleted quotas), preventing system congestion or data loss. Selecting the hard drive with the highest write speed as a fallback option aims to leverage its high performance to minimize the additional latency caused by processing such abnormal data packets, thus maintaining the overall system throughput and responsiveness.

[0233] The Q600 writes each network traffic data packet to disk based on the hard drive ID corresponding to each network traffic data packet.

[0234] After all data packets are assigned disk IDs, the system groups the data packets according to their disk IDs. Then, a separate write thread or I / O task is started for each disk to write the data packets belonging to that disk to its corresponding physical storage device in batches and in order. During writing, multiple data packets from the same disk can be concatenated into larger data blocks in memory before writing to reduce the number of I / O operations. After writing is complete, the disk write buffer is cleared, and the remaining quota for each disk is reset to the initial quota (Q_i), preparing for the next round of accumulation and disk write.

[0235] This step, by writing in parallel groups according to the target hard drive, fully utilizes the I / O concurrency of multi-hard drive systems, making the total disk write time approximately equal to the write time of the slowest hard drive, rather than the sum of the write times of each hard drive, thus greatly improving overall throughput. Batch sequential writing also reduces head movement (for HDDs) or write amplification (for SSDs), further optimizing storage performance. The closed-loop design of the entire process enables the system to continuously and stably process high-speed network data streams.

[0236] Furthermore, the method also includes:

[0237] When acquiring network traffic data packets, the Q700 analyzes the sequence characteristics of the network traffic data packets. If a data packet sequence that matches a preset abnormal pattern is detected, an independent logical container is created for the identified abnormal data packet sequence, and all subsequent data packets of the abnormal data packet sequence are temporarily marked as monitored flows and no longer participate in the routine comparison based on the amount of data left on disk in a single transaction.

[0238] The system integrates a real-time analysis engine along the data packet reception path to perform deep inspection of network flow data packets. This engine maintains an updatable library of anomaly patterns, which can be defined as follows:

[0239] Rule-based matching: Matches packet payload or header features using predefined rule sets (such as Suricata or Snort rule syntax).

[0240] Statistical anomaly detection: Establish a traffic baseline model (such as an autoregressive model based on historical data), calculate the deviation of traffic characteristics (such as packet rate, byte rate, connection frequency) from the baseline in real time, and trigger an anomaly if it exceeds a preset standard deviation multiple.

[0241] Behavioral analysis: detects protocol state machine violations (such as establishing a TCP connection with a non-SYN packet), geographical anomalies (source IP suddenly coming from an unfamiliar country), or port scanning patterns (sending probe packets to multiple ports in a short period of time).

[0242] When a packet sequence matching an anomaly pattern is detected, the system immediately creates a logical container object in memory. This object contains the following metadata: anomaly session ID (usually a 5-tuple hash), anomaly type, first detection timestamp, and a pointer to the associated packet list. Simultaneously, the system sets a "monitored flow" flag for this session ID, redirecting subsequent packets to the anomaly handling pipeline, bypassing the regular Q300-Q500 allocation logic.

[0243] This step enables proactive identification and isolation of threatening traffic and performance anomalies. Through real-time deep packet inspection and behavioral analysis, the system can capture and classify abnormal traffic before it impacts overall storage performance, laying the foundation for subsequent differentiated processing. Creating independent logical containers achieves fault domain isolation, preventing abnormal traffic from crowding out processing resources for normal traffic and ensuring the Quality of Service (QoS) for core business data.

[0244] Q710: Allocate the entire logical container to the target hard drive with the lightest current load and redundant I / O capability, and specially mark the session ID of the abnormal data packet sequence in the unfinished network session ID table so that all subsequent data packets of the abnormal data packet sequence directly point to the target hard drive.

[0245] The system maintains a hard drive health rating table, and updates the following metrics for each hard drive periodically (e.g., every second):

[0246] Load score: a weighted normalized value based on I / O queue depth (weight 0.4), write latency (weight 0.4), and CPU utilization (weight 0.2).

[0247] Redundancy: Calculate the ratio of the hard drive's current remaining quota to its base quota, as well as the difference between its nominal write rate and the current actual throughput.

[0248] When a target hard drive needs to be allocated to an abnormal logical container, the system selects the hard drive with the lowest load score and redundancy capacity above a threshold (e.g., remaining quota percentage > 30%). After allocation, a special field is added to the "Unfinished Network Session ID Table" for the abnormal session ID. Subsequently, all packets belonging to this session ID will be directly located to the logical container via pointers and directed to the target hard drive's dedicated write queue.

[0249] This step, through fine-grained resource scheduling, directs abnormal traffic to the storage devices with the highest processing capacity in the system, achieving load isolation and resource optimization. This prevents abnormal traffic from impacting the hard drives of normal business operations while making full use of idle I / O capabilities. A special marking mechanism ensures the shortest possible path for abnormal data, reducing decision-making overhead, and providing a clear tracking anchor for centralized monitoring and dynamic adjustments.

[0250] Q720: Start a monitoring thread to continuously track the traffic rate of the abnormal data packet sequence and the real-time performance metrics of the target hard drive.

[0251] The system creates a lightweight monitoring coroutine (instead of an operating system thread) for each active logical container to reduce context switching overhead. This coroutine performs the following operation every 100 milliseconds:

[0252] Traffic metering: Read the number of new data packets and bytes added during the period from the metadata of the logical container, and calculate the instantaneous rate (pps, Mbps) and cumulative total.

[0253] Disk performance sampling: Obtain the latest performance counters by reading / sys / block / sdb / stat from the target disk or by using iostat -d sdb1 2, and calculate the current queue depth, average write latency, and throughput.

[0254] Threshold check: Compare the real-time rate with a preset threshold (which can be dynamically adjusted, such as 3 times the baseline value); compare the hard disk latency with a degradation threshold (such as 50ms).

[0255] Logs and alerts: Write monitoring data to a time-series database (such as Prometheus) and send alert events when the rate continues to exceed the threshold.

[0256] Continuous, fine-grained monitoring provides the system with situational awareness. By tracking the evolution of abnormal traffic and the response of the target hard drive in real time, the system can anticipate potential risks (such as impending hard drive overload), providing data support for proactive, elastic scaling decisions. This closed loop of monitoring and decision-making is key to achieving system adaptability.

[0257] Q730, if the traffic rate exceeds the threshold or the target hard disk performance degrades, the logical container is split into multiple segments, and according to the instantaneous load of each hard disk, the data packets of different segments are redirected to multiple hard disks for parallel disk writing, and the session ID mapping relationship is updated to reflect the storage status of this segment.

[0258] When the monitoring thread detects a trigger condition (such as traffic rate exceeding a threshold or hard disk latency exceeding a threshold), the following dynamic adjustment process is executed:

[0259] Splitting Decision: Based on the total data volume of the current abnormal traffic and the number of available hard drives in the system, determine the number of splits N (e.g., N = min(current active hard drives, ceil(current rate / safe throughput per hard drive))). Divide the queue of data packets to be written to disk within the logical container into N sub-segments according to time windows or data volume. Each sub-segment maintains independent metadata and a write state machine.

[0260] Disk reselection: The system queries the instantaneous load of each disk in real time and selects the disk with the lightest current load for each segment (the original target disk should be avoided unless its load has recovered). This is achieved through a fast greedy selection algorithm.

[0261] Mapping Update: In the "Unfinished Network Session ID Table", create N sub-mapping entries for the original session ID.

[0262] Parallel disk write: Data packets for each segment are distributed to the corresponding disk's write queue and written in parallel by their respective write threads. The system ensures that the order of data packets within each segment remains unchanged.

[0263] For example, DNS attack traffic continues to grow to 12 Gbps, and the latency on the target hard drive sdb reaches 55ms. The system decides to split it into three segments (N=3). Based on the real-time load, sda, sdc, and sdb (although sdb has high latency, it can still process some data due to its remaining capacity) are selected to process the three segments respectively. After updating the mapping table, subsequent packets are distributed to the three hard drives according to hash.

[0264] This step demonstrates the system's horizontal scalability and resilience. When a single hard drive cannot handle abnormal traffic, the load is dynamically distributed across multiple physical devices, achieving parallelization of storage I / O and significantly improving the system's ability to handle traffic surges. The segmented mapping mechanism ensures data logical integrity while avoiding single-point bottlenecks, thus improving the overall throughput and resilience of the system.

[0265] Q740, when the abnormal pattern disappears or the sequence ends, the system automatically cleans up the special marker and restores the subsequent regular data packets of the session to the normal allocation process.

[0266] The abnormal traffic monitoring goroutine continuously detects and recovers from the following conditions:

[0267] Flow rate decline: The flow rate is lower than 50% of the abnormal threshold for M consecutive cycles (e.g., 10 cycles, i.e., 1 second).

[0268] Session termination signal: A TCP FIN / RST packet is detected, or the UDP stream exceeds the idle timeout period (e.g., 300 seconds of inactivity).

[0269] Attack signatures disappear: The rules engine confirms that the data packets no longer match any abnormal patterns.

[0270] When the recovery conditions are met, the system stops the monitoring coroutine for the abnormal session, clears the special flags (such as MONITORD, SEGMENTED) of the session ID in the "Unfinished Network Session ID Table", but retains its current disk mapping (if the session has not ended). After the remaining data packets in the logical container are written to disk according to the normal process, the container memory is released.

[0271] If subsequent data packets arrive in the session, they will re-enter the Q300 step for routine session persistence determination and allocation.

[0272] This step enables automatic resource reclamation and state normalization. By intelligently detecting abnormal termination events, the system can promptly release monitored resources and resume normal optimized processing of traffic. This avoids efficiency degradation caused by long-term occupation of system resources due to historical abnormal states, ensuring that the system always processes current traffic with the optimal strategy, demonstrating the system's self-healing and adaptive characteristics.

[0273] Furthermore, the unfinished network session ID table also includes a timestamp corresponding to each unfinished network session ID; the method further includes:

[0274] Q800 periodically scans the unfinished network session ID table and removes unfinished network sessions I whose current time differs from the timestamp by more than a preset timeout duration from the unfinished network session ID table.

[0275] In each entry of the unfinished network session ID table, in addition to storing the mapping between the session ID and the disk ID, a last active timestamp must also be stored. Whenever a packet belonging to that session arrives (whether it is a new allocation or a session persistence), the system updates this timestamp to the current system time (with millisecond precision).

[0276] The system starts a low-priority background daemon thread that scans the entire mapping table at fixed intervals (e.g., every 30 seconds). This scanner uses incremental or piecewise traversal to avoid prolonged lock holding affecting normal packet processing. For each entry in the table, the difference (ΔT) between the current time and the last active timestamp is calculated. The preset timeout duration is dynamically set according to the protocol type: TCP sessions: Because TCP is a stateful connection, the timeout duration is longer, typically set to twice the maximum message lifespan (2MSL), for example, 120 seconds. UDP / other connectionless streams: Because there is no connection, the timeout duration is shorter, for example, 60 seconds.

[0277] If ΔT > preset timeout duration, remove the entry from the table. Before removal, logs (session ID, duration, total data volume) can be recorded for auditing purposes.

[0278] This step effectively addresses the issue of session state remnants caused by network anomalies (such as sudden connection interruptions or client crashes) by introducing a time-based session garbage collection mechanism. This ensures that the unfinished session mapping table only contains truly active sessions, significantly reducing table storage space usage and improving query efficiency (reduced hash table collisions). Simultaneously, periodic cleanup releases the memory required for metadata maintained by the system for each session, preventing memory leaks and enhancing the long-term stability of the system.

[0279] Q810: If the received network traffic data packet carries a transport layer connection termination flag, then after the received network traffic data packet is written to disk, the corresponding unfinished network session ID in the unfinished network session ID table will be removed.

[0280] During the packet parsing phase, a deep inspection of the transport layer protocol header is performed: TCP: checking whether the FIN (End) or RST (Reset) flag is set to 1. Other protocols: such as SCTP, QUIC, etc., identify their corresponding connection closing signals.

[0281] When a data packet carrying a termination flag is detected, the system does not immediately delete the mapping entry. Instead, it: a. allows the data packet to be assigned a hard disk ID through normal session persistence logic (Q300). b. places the data packet in the disk write queue of the corresponding hard disk. c. performs the deletion operation only after receiving confirmation from the hard disk controller that the data packet has been physically written (or at least committed to the disk cache).

[0282] When deleting an entry from the mapping table, concurrency safety must be ensured. Typically, a lookup and delete operation is performed while holding a write lock on the session table.

[0283] This step enables protocol-aware and precise resource reclamation. Compared to the latency of timeout cleanup, it releases resources immediately after a session's legitimate termination, keeping the mapping table in a minimal state and further improving query performance. More importantly, it ensures the reliable storage of critical control information (such as FIN / RST packets), as the state is only cleaned up after confirming that these flag packets have been written to disk. This provides assurance for subsequent network forensics and connection integrity analysis. This mechanism is particularly suitable for scenarios with frequent short connections, enabling efficient management of session lifecycles.

[0284] The Q800 and Q810 steps constitute a multi-layered session state cleanup system: Q800 provides a "grace period" fallback cleanup to handle abnormally terminated sessions; Q810 provides "instant" precise cleanup to handle normally terminated sessions. These two mechanisms complement each other, ensuring that the unfinished session mapping table is always maintained at a near-optimal size, preventing premature deletion from compromising session persistence and avoiding performance degradation from accumulating junk entries. This dynamic maintenance mechanism is key to the system's ability to efficiently handle massive network traffic over the long term. It balances memory overhead, query efficiency, and data locality, reflecting refined and intelligent resource management.

[0285] Furthermore, the method also includes:

[0286] Q900 will skip a hard drive during subsequent allocation processes if it detects multiple consecutive disk write failures or a write rate that consistently falls below the minimum threshold.

[0287] This step effectively integrates fault diagnosis results into the entire resource allocation process, achieving automatic fault masking and load redistribution. By dynamically eliminating faulty hard drives and recalculating resources, the system ensures that data allocation decisions are always based on healthy and available storage subsystems, avoiding the writing of valuable data to unreliable media. Simultaneously, the migration mechanism for existing sessions guarantees the integrity of data logic and achieves seamless service failover. This significantly enhances the system's self-healing capabilities and business continuity in the face of hardware failures, reduces the need for operational intervention, and demonstrates the system's high robustness.

[0288] In this embodiment, an intelligent multi-disk write allocation mechanism is employed to collaboratively optimize data write efficiency, storage locality, and heterogeneous hardware utilization. Specifically, by maintaining an unfinished network session ID table, this invention directs all data packets from the same session to the same hard drive for storage, effectively ensuring the logical locality and physical continuity of data. This significantly improves the read performance for subsequent session-based retrieval and in-depth analysis, resolving the low read efficiency problem caused by scattered data storage in existing technologies. Simultaneously, a dynamic "budget" for the amount of data written to disk per session is allocated based on the write rate ratio of each hard drive. Furthermore, load balancing is performed on new session data packets based on this budget, enabling high-performance hard drives to handle a greater load commensurate with their capabilities. This fully leverages the performance potential of heterogeneous storage hardware, improving the overall system write throughput. In addition, the solution incorporates a fallback allocation strategy for very large packets, ensuring that the data processing flow is not blocked and enhancing the system's robustness. In summary, this method achieves comprehensive optimization of three key objectives—write throughput, read friendliness, and hardware adaptability—in complex production environments.

[0289] Example 4:

[0290] Based on the above embodiment two, in order to avoid disk buffer overflow, a preset network card buffer can also be set, and then the following steps can be included after step S600:

[0291] H100 stores network flow data packets received by the network card queue into a preset network card buffer for temporary storage.

[0292] In this embodiment, during system initialization, a fixed-size circular buffer is pre-allocated in memory as the network interface card (NIC) buffer. When the NIC driver (or DPDK PMD) receives data packets from the hardware queue via interrupt or polling, it directly segments the network flow data into several network flow data packets and stores them in the NIC buffer. To prevent the producer (NIC) from overwriting data that has not been processed by the consumer (transfer thread) too quickly, the circular buffer needs to implement a watermark alarm mechanism. When the number of free slots falls below the low watermark (e.g., 20% of the total capacity), flow control or an alarm can be triggered.

[0293] This step decouples and buffers network traffic from subsequent processing logic. High-speed, bursty network data streams are first stored losslessly in a safe area of ​​memory, avoiding packet loss due to momentary delays or blockages in subsequent processing stages (such as disk I / O). The circular buffer design ensures efficient memory reuse and sequential access, providing the first line of defense against traffic peaks.

[0294] H200: If the disk cache has free capacity, then the network flow data packets in the network card cache are transferred to the disk cache in timestamp order; otherwise, the transfer of network flow data packets in the preset network card cache is paused, and the network session ID corresponding to each network flow data packet in the disk cache is read.

[0295] Furthermore, the step of transferring network flow data packets in the network card buffer to the disk cache in timestamp order includes the following steps:

[0296] H210, establish and maintain a dynamic priority session list; the dynamic priority session list is statistically analyzed based on the network session ID of network flow data packets, and records the arrival rate and total data volume of network flow data packets for each network session within the most recent preset time window.

[0297] The system initializes and maintains a core data structure in memory—a dynamic priority session list. This list is a container that supports efficient insertion, deletion, updating, and sorting, such as using a skip list or a balanced binary search tree (like a red-black tree), with the session's "priority score" as the sorting key.

[0298] Create and maintain a sliding time window statistician for each observed network session ID (based on the hash value of a 5-tuple or triple). This statistician is typically a fixed-length circular buffer (e.g., with a length corresponding to 60 time slices), and each slot records the number of packets and the total number of bytes arriving for that session within a fixed time interval (e.g., 1 second).

[0299] Whenever a new time slice ends (or a new data packet is received for the session), the system updates the session's statistics and recalculates two core metrics:

[0300] Arrival rate: The average number of data packets arriving per second over the most recent N time slices (e.g., the most recent 10 seconds).

[0301] Total data volume: The total number of bytes that have arrived in the last N time slices.

[0302] Based on a pre-defined weighting formula, a real-time priority score is calculated for each active session. An example formula is: Score = α² × Normalized arrival rate + β² × Normalized total data volume, where α² and β² are adjustable weights (e.g., α² = 0.7, β² = 0.3), designed to give higher rankings to high-frequency, consistent sessions. After the score is calculated, the session's position in the ordered list is updated immediately.

[0303] This step enables refined, real-time quantitative perception of the activity levels of different sessions within network traffic. Through sliding time window statistics, the system can accurately identify the currently "hottest" or largest data volume sessions, overcoming the limitations of relying on static configurations or simple rules. This lays a solid data foundation for subsequent implementation of intelligent priority scheduling, allowing the system to distinguish between "elephant flows" and "mouse flows," and focus on processing the data flows that consume the most system resources (especially cache space).

[0304] H220, when transferring network flow data packets from the network card buffer to the disk buffer, prioritizes transferring data packets belonging to the network sessions ranked higher in the dynamic priority session list, so as to ensure that data from highly active network sessions can enter the subsequent processing flow first.

[0305] When the transfer condition is met (there is free capacity in the disk buffer), the transfer thread does not directly retrieve packets from the physical header of the network interface card (NIC) buffer. Instead, it first queries the dynamic priority session list to obtain the top-ranked (e.g., Top 5) high-priority session IDs. Then, the transfer thread scans the queue of pending packets in the NIC buffer. It maintains a pointer to record the current scan position, but prioritizes retrieving all packets belonging to the aforementioned high-priority sessions, regardless of their physical order in the queue. These packets are copied to the disk buffer. After a round of transfers of high-priority session packets, if there is still free capacity in the disk buffer, the transfer thread returns to the original scan pointer position and continues to transfer the remaining low-priority packets in FIFO order until the buffer is full or the NIC buffer is emptied.

[0306] This step effectively translates the prioritization strategy into performance advantages. By allowing packets from highly active sessions to "jump the queue" and enter the processing core (disk buffer) first, the end-to-end processing latency of these critical data streams is significantly reduced. This is crucial for ensuring the quality of service for services with high real-time requirements (such as interactive video and online transactions). Simultaneously, by prioritizing the transfer of sessions with large data volumes, the space occupied by these sessions in the network interface card (NIC) buffer can be released more quickly, improving the overall turnover efficiency of the buffer and indirectly enhancing the system's ability to handle sudden surges in traffic.

[0307] H230, the update cycle of the dynamic priority session list is synchronized with the allocation trigger cycle of the disk cache. Each time it is updated, the priority ranking of the network session is recalculated and adjusted by combining the historical data volume of the network session and the remaining unwritten data volume of the network session in the current disk cache.

[0308] To ensure the timeliness and accuracy of priority ranking, the list update mechanism needs to be closely synchronized with the system's core driving cycle—the allocation and disk write cycle of the disk cache.

[0309] The system binds the update operation of the dynamic priority session list to the event "disk buffer is full and disk allocation is triggered" and before the start of a new round of data transfer. This ensures that priority calculation uses the latest statistics from the previous complete processing cycle.

[0310] When updating the priority score calculation, a key feedback factor is introduced: the amount of remaining unwritten data for the session in the current write buffer. The new priority score formula evolves to: New Score = (α² × Arrival Rate + β² × Total Data Amount) × (1 + γ² × Normalized Remaining Data Amount); where γ² is a gain coefficient. This means that if a session has a large backlog of unwritten data in the write buffer (possibly due to a slow target hard drive), the system will further increase its priority, prompting upstream systems to replenish the data more quickly, thereby attempting to maintain the session's processing pipeline and prevent cache space from being blocked due to backlog.

[0311] Each update also removes sessions that have been inactive for several recent periods or have been confirmed terminated, in order to keep the list compact and reduce maintenance overhead.

[0312] This step achieves closed-loop feedback of the priority strategy and system state awareness. By feeding back the backlog status of the downstream (disk buffer) to the upstream (transfer scheduling) decision-making, the system acquires a preliminary "congestion control" awareness. This mechanism can dynamically allocate processing resources to sessions "encountering bottlenecks," helping to balance the progress between different sessions and preventing individual sessions from being severely delayed due to differences in I / O performance, thereby improving the fairness and overall throughput efficiency of the system in multi-session concurrent scenarios. The synchronous update mechanism ensures the timeliness of the decision-making basis and evenly distributes the computational overhead across the processing cycle, avoiding impact on the real-time data path.

[0313] H300: For any network flow data packet RQ, if the network session ID corresponding to RQ exists in the preset unfinished network session ID table, then the hard disk ID mapped to the network session ID corresponding to RQ in the unfinished network session ID table is assigned to RQ; otherwise, RQ is determined as a new session data packet; the unfinished network session ID table is used to record the mapping relationship between unfinished network session IDs and the assigned hard disk IDs.

[0314] When the transfer is paused and data in the disk buffer needs to be processed, the system first iterates through each data packet. For any network flow packet RQ, the IP header and transport layer header of the RQ are parsed to extract the 5-tuple (source IP, source port, destination IP, destination port, protocol). A fixed-length session ID (e.g., a 128-bit hash value) is calculated from this 5-tuple using an efficient hash function (such as Jenkins hash). Subsequently, a lookup is performed in a global hash table that supports high concurrency access (i.e., the "unfinished network session ID table") using this session ID as the key. The values ​​in this hash table store the disk IDs previously allocated to the session. If the lookup matches, this disk ID is immediately recorded as the allocation target for the packet's RQ. If the lookup does not match, the RQ is marked as a "new session packet" and placed in a pending decision list. The hash table requires a background cleanup thread to remove ended session entries based on timeouts or the detection of TCP FIN / RST packets.

[0315] This step is the core implementation of the Session Affinity principle. By forcing all data packets of the same session to point to the same physical hard drive, the physical continuity of session data on the storage medium is ensured. This brings significant performance advantages to subsequent session-based data retrieval, complete stream reconstruction, or in-depth analysis, avoiding expensive random read operations caused by scattered data storage, and is the cornerstone for improving the efficiency of later data analysis in the system.

[0316] H400 assigns a disk ID to RQ based on the current amount of data remaining to be written to disk in a single operation for each disk.

[0317] Furthermore, step H400 includes the following steps:

[0318] The H410 monitors the current I / O operation queue depth, average response time, and historical throughput fluctuations of each hard drive in real time.

[0319] Current I / O operation queue depth monitoring: In Linux systems, this is achieved by directly reading / sys / block / <sdx>The ` / inflight` file or the `aqu-sz` (average queue length) field from the `iostat -x1` command can be used to obtain the queue depth. The ` / sys / block / sda / inflight` file typically contains two comma-separated numbers, such as 12,8, representing the number of incomplete read and write requests, respectively. Adding these numbers together gives the total queue depth. A lightweight monitoring thread can periodically (e.g., every 200 milliseconds) read this file from all disks and update the results to a disk status structure in shared memory.

[0320] Average response time monitoring: Obtained by parsing the ` / proc / diskstats` file or the `await` field of `iostat -x`. ` / proc / diskstats` provides cumulative counts since system startup. For a given hard drive, field 10 is the total number of milliseconds spent on read operations, field 11 is the total number of milliseconds spent on write operations; field 4 is the number of completed read operations, and field 8 is the number of completed write operations. At two adjacent sampling times t1 and t2, the average response time within the interval is calculated as: [(Write time_t2 - Write time_t1) + (Read time_t2 - Read time_t1)] / [(Number of completed writes_t2 - Number of completed writes_t1) + (Number of completed reads_t2 - Number of completed reads_t1)]; this value reflects the average time (including queuing and servicing time) required for the hard drive to process an I / O request.

[0321] Historical throughput fluctuation monitoring: At each sampling time, in addition to the above indicators, the instantaneous throughput of the hard disk is also recorded (which can be accessed via / sys / block / ). <sdx>(Calculated from the changes in the number of sectors read and written in / stat).

[0322] Volatility calculation: The system maintains a fixed-length historical throughput sequence for each hard drive (e.g., the most recent 100 sample values). Periodically (e.g., every 10 seconds), the ratio of the standard deviation (or variance) to the mean of this sequence (i.e., the coefficient of variation) is calculated as the "volatility coefficient". The larger the volatility coefficient, the more unstable the historical throughput performance of the hard drive.

[0323] For example, for the hard drive sdb, the monitoring thread reads at time t1: total number of read / write completions = 10000, total time = 50000ms. At time t2 (200ms later): total number of completions = 10050, total time = 51000ms. Therefore, the average response time within the interval = (51000 - 50000) / (10050 - 10000) = 1000 / 50 = 20ms. Meanwhile, the queue depth reading is 15. The throughput fluctuation coefficient over the past 100 sampling points is calculated to be 0.3 (relatively stable).

[0324] This step establishes a multi-dimensional, real-time awareness of hard drive performance. Compared to considering only static write rate or remaining quota, by collecting queue depth and response time, the system can keenly capture the instantaneous pressure on the hard drive (whether it is congested); by analyzing historical throughput fluctuations, the system can assess the long-term reliability of the hard drive (whether its performance is stable). This provides subsequent intelligent decision-making with insights far deeper than "whether it is idle," and is a key prerequisite for avoiding the allocation of data to hard drives that are "about to be overloaded" or have "unpredictable performance."

[0325] H420 calculates a real-time load weight for each hard drive based on monitoring data; the real-time load weight is positively correlated with I / O queue depth and average response time, and negatively correlated with historical throughput stability.

[0326] First, map the three metrics to a uniform scale (e.g., between 0 and 1). Queue depth: Divide by an empirical "depth threshold" (e.g., 32); values ​​exceeding the threshold are counted as 1. Average response time: Divide by an acceptable "latency threshold" (e.g., 50ms); values ​​exceeding the threshold are counted as 1. Historical throughput volatility coefficient: This is already a relative value and can be used directly or scaled appropriately.

[0327] Real-time load weight = W_q × normalized queue depth + W_l × normalized average response time + W_v × historical volatility coefficient; where W_q, W_l, and W_v are configurable weight coefficients, and W_q + W_l + W_v = 1. By design, W_q and W_l are positive coefficients (usually large), making the weight positively correlated with queue depth and response time; W_v is also a positive coefficient (usually small), making the weight negatively correlated with stability (because a larger volatility coefficient indicates greater instability).

[0328] The weighting coefficients themselves are not entirely fixed. For example, when the overall system load is light, the influence of W_v can be reduced, and more attention can be paid to capacity; when the system is close to full load, W_q and W_l should be increased to avoid selecting hard drives that are already congested.

[0329] For example: Assume W_q=0.5, W_l=0.4, W_v=0.1, depth threshold=32, and delay threshold=50ms.

[0330] SDB metrics: Queue depth = 15 (normalized 15 / 32 ≈ 0.47), Response time = 20ms (normalized 20 / 50 = 0.4), Volatility coefficient = 0.3.

[0331] The calculated weight is 0.5 × 0.47 + 0.4 × 0.4 + 0.1 × 0.3 = 0.235 + 0.16 + 0.03 = 0.425.

[0332] This step achieves intelligent fusion from multi-dimensional indicators to a single decision variable. It's not merely addition; rather, it uses configurable weights to reflect the system's different strategic emphases under varying operating conditions. Positively correlated terms (queues, latency) ensure the system remains highly sensitive to immediate congestion, proactively avoiding hotspots; negatively correlated terms (stability), under equal conditions, prioritize reliable, long-lasting hard drives with predictable performance, rather than performance spikes that may suddenly slow down even when momentarily idle. This comprehensive evaluation significantly improves the quality and long-term benefits of allocation decisions.

[0333] H430, when assigning a hard disk ID to an RQ, prioritizes the hard disk with the lowest real-time load weight and a single remaining data write volume greater than the RQ size.

[0334] When a disk needs to be allocated for a new session data packet (RQ), the system first iterates through all disks, filtering out those with a "remaining amount of data to be written to disk in a single session greater than the RQ size," forming a preliminary qualified candidate set. This ensures basic capacity constraints. Within the preliminary qualified candidate set, the system sorts them in ascending order according to real-time load weight (lowest weight first). Then, the disk with the lowest weight is selected as the target. If multiple disks have the same lowest weight, their remaining capacity (selecting the one with the most remaining) or write rate can be further considered as auxiliary factors in the decision.

[0335] If the initial qualified candidate set is empty (i.e., the remaining capacity of all hard drives is insufficient to accommodate the RQ), then a degradation strategy is initiated. In this case, the capacity constraint can be ignored, and the hard drive with the lowest real-time load weight can be directly selected from all hard drives, and the RQ can be assigned to it.

[0336] This step is the final execution stage of intelligent load balancing. It perfectly combines the two objectives of "capacity feasibility" and "load optimization." By first filtering capacity, the basic feasibility of the allocation operation is ensured; then, by ranking by weight, the most beneficial choice for the overall health of the system is made among the feasible options. This two-stage decision-making logic of "filtering first, then optimizing" is both efficient and effective. It guides new data flows naturally to those hard drives that have both available space and the least workload, thereby proactively balancing I / O pressure globally, avoiding the Matthew effect of "the busy getting busier and the idle getting idler," and significantly improving the overall utilization efficiency and long-term operational stability of multi-hard drive systems.

[0337] H500: For network flow data packets in the disk cache that have been allocated to a hard drive, group them by hard drive ID and perform disk write operation; wherein, for network flow data packet groups belonging to the same network session ID, the amount of data written in a single disk write operation does not exceed the single disk write data size of the corresponding hard drive; the remaining network flow data packets belonging to the same network session that are not written in this operation have their allocated hard drive IDs remain unchanged and continue to be stored in the disk cache.

[0338] Furthermore, the amount of data written to the hard drive in a single operation is determined through the following steps:

[0339] H510 defines the write percentage of each hard drive as the ratio of the write speed of each hard drive to the sum of the write speeds of all hard drives.

[0340] Calculate the total write rate of all hard drives involved in the write operation: Total rate = Σ (write rate of each hard drive). For each hard drive i, calculate its write percentage: Write percentage_i = Write rate of hard drive i / Total rate. This value is a decimal between 0 and 1, and the sum of the write percentages of all hard drives is 1.

[0341] H520 determines the amount of data written to disk per drive in a single operation by multiplying the write percentage of each hard drive by the size of the preset disk cache area.

[0342] Multiply the write percentage by the preset total disk cache size: Base quota_i = Write percentage_i × Disk cache size. The result is the theoretical maximum amount of data that the hard drive can receive in this disk write cycle. The system maintains a "current remaining quota" variable for each hard drive, with the initial value being this base quota.

[0343] This step allocates a large quota to high-performance hard drives (such as SSDs) proportional to their capacity, while allocating a smaller quota to low-speed hard drives (such as HDDs). This ensures, from a planning perspective, that the overall system throughput potential is fully utilized, preventing high-performance hard drives from becoming underutilized and idle resources, and providing a reasonable and efficient initial state for subsequent load balancing. This is the strategic cornerstone for achieving optimal overall system performance.

[0344] Furthermore, the method also includes:

[0345] H530 can obtain the current I / O operation queue depth and average write latency of each hard drive in real time.

[0346] In Linux systems, the file / sys / block / is read periodically (e.g., every 100 milliseconds). <sdx>Use ` / inflight` to get the number of I / O requests. This file typically contains two comma-separated numbers (e.g., 5, 12), representing the number of incomplete read and write requests, respectively. Adding them together gives the total number of pending I / O requests, i.e., the queue depth.

[0347] The calculation is performed by parsing the corresponding fields in ` / proc / diskstats`. Pay attention to field 10 (cumulative milliseconds spent on write I / O) and field 8 (cumulative number of write I / O completions). Within the sampling interval Δt, calculate: Average write latency = (cumulative write time for this operation - cumulative write time for the previous operation) / (number of write completions for this operation - number of write completions for the previous operation). This value directly reflects the average speed at which the hard drive responds to write requests, typically in milliseconds (ms).

[0348] H540 calculates the real-time load impact factor for each hard disk based on the current I / O operation queue depth and average write latency.

[0349] Set empirical "full load thresholds" Q_max and L_max for queue depth and average latency, respectively (e.g., Q_max=32, L_max=50ms). Calculate the normalized load values: r_Q=min(Q / Q_max,1.0), r_L=min(L / L_max,1.0). When the actual value exceeds the threshold, calculate as full load (1.0).

[0350] The real-time load impact factor F_load = 1 / (1+k×(α3×r_Q+(1-α3)×r_L)) is calculated using a pre-defined function that smoothly reduces the impact of load as the load increases. Here, k is a sensitivity coefficient (e.g., k=1.0), controlling the strength of the load's influence on the factor; α3 is a weighting coefficient (e.g., 0.6), used to adjust the relative importance of queue depth and latency. This function ensures that F_load monotonically decreases from 1 as r_Q or r_L increases, but never reaches zero.

[0351] In this step, increased load only leads to a decrease in the factor (but not a drastic drop to zero), providing a stable and controllable adjustment "handle" for the next step of proportionally reducing quotas. By combining queue depth and latency, this factor more comprehensively reflects the overall stress on the hard drive and is more reliable than a single indicator.

[0352] H550 multiplies the single disk write data volume determined based on the write rate ratio with the real-time load impact factor to obtain the final single disk write data volume for each hard drive; wherein, the real-time load impact factor is negatively correlated with I / O operation queue depth and average write latency.

[0353] For the i-th hard drive, its base quota Q_base_i is multiplied by its real-time load impact factor F_load_i to obtain the corresponding single-time disk write size Q_final_i = Q_base_i × F_load_i. The calculated Q_final_i is then set as the initial value of the "current single-time remaining disk write size" for the i-th hard drive. When the system subsequently allocates hard drives for new data packets, it will strictly adhere to this updated remaining quota. This means that for a hard drive with increased load, its "budget" available for receiving new data will be reduced in real time.

[0354] This step is a crucial execution stage in closed-loop negative feedback control. It achieves a complete "sensing-decision-execution" cycle: detecting increased hard drive load, calculating the reduction factor, and immediately reducing its data reception quota. This mechanism brings revolutionary benefits: it enables the system to have proactive, preventative load balancing capabilities. The system no longer passively waits for the hard drive queue to fill up and latency to spike before reacting; instead, it can gently guide new data to other, less busy hard drives as soon as pressure begins to appear, thus preventing congestion. This significantly improves the efficiency, stability, and overall throughput of multi-hard drive systems, and is the core intelligence ensuring long-term smooth system operation.

[0355] After completing the disk write operation, H600 continues to transfer network flow data packets from the network card buffer to the disk write buffer, and prepares for the next allocation and disk write cycle.

[0356] Once all asynchronous write operations for the current batch on all hard drives have been initiated, the current round of core disk write processing is complete. The system then updates the management status of the disk write buffer: marking successfully written data packet entries as reclaimable and clearing their contents, while increasing the free capacity count. The process then jumps back to step H200. Since space has been released in the disk write buffer, the transfer conditions are met, and the system restarts the data transfer from the network interface card buffer to the disk write buffer, filling the newly released space and preparing for the next disk write cycle. The entire process thus forms a closed loop and continues to run.

[0357] This process forms a continuous, self-driven processing pipeline. Disk write operations free up space, triggering the inflow of new data, which in turn prepares data for the next disk write. This cyclical mechanism ensures continuous and saturated utilization of the system's processing capacity without the need for frequent external scheduling. The entire design forms a negative feedback closed-loop system that automatically adapts to changes in input flow, maintaining long-term stable operation and providing operational assurance for 24 / 7 uninterrupted data capture and storage.

[0358] Furthermore, the method also includes:

[0359] H700 determines that the network session corresponding to a network session ID is a persistent overload session when the total number of remaining network stream data packets waiting to be written to disk in the disk cache corresponding to a network session ID exceeds the single disk write data size of the corresponding hard drive for a preset number of consecutive times.

[0360] The system maintains a dedicated continuous excess counter for each active, unfinished network session ID (stored in the unfinished network session mapping table). The check is triggered after each allocation operation for packets in the disk write buffer (i.e., step H400) and before the actual disk write operation. During the check, the system counts the total amount of data packets for that network session ID that have been allocated a hard disk ID in the disk write buffer but have not yet been actually written to the hard disk; this is called the "total amount of remaining packets waiting to be written to disk" (denoted as W_remaining).

[0361] Retrieve the disk ID currently mapped to this network session, and query the current effective single disk write volume for that disk (i.e., the dynamically adjusted quota, denoted as Q_disk). Compare W_remaining with Q_disk. If W_remaining > Q_disk, increment the continuous overrun counter for this session by 1; otherwise, reset the counter to zero.

[0362] When the continuous overage counter of a session reaches a preset threshold (e.g., 3 times), the system determines that the session is a "continuous overage session" and triggers the segmentation process.

[0363] By introducing a continuous threshold determination, this approach achieves accurate and stable identification of abnormally large flows. It avoids false triggering of segmentation due to single instantaneous fluctuations (such as network bursts), taking action only when sessions consistently exhibit "supply exceeding demand" characteristics. This mechanism ensures that segmentation strategies are applied only to "elephant flows" that truly require horizontal scaling processing capabilities, preventing unnecessary complex management of a large number of ordinary sessions, thus achieving an optimal balance between functionality and overhead.

[0364] H710, create a new mapping record for the network session ID in the unfinished network session ID table.

[0365] Once a session is marked as a persistent overload session, the system needs to modify its storage mapping policy from "single-disk affinity" to "multi-disk parallel".

[0366] The original "Unfinished Network Session ID Table" is a key-value pair structure: {session_id:primary_disk_id}. It now needs to be expanded to support a main record with multiple segment records. For example, upgrading session entries to:

[0367] { "session_id": "0xVIDEO123",

[0368] "segments": [

[0369] {"segment_id": 0, "disk_id": "sdb", "status": "active"},

[0370] {"segment_id": 1, "disk_id": "sdc", "status": "active"}

[0371] ],

[0372] "flags": ["PERSISTENT_OVERLOAD", "SEGMENTED"]}.

[0373] Upon detecting an excessive session, the system immediately creates a new segment record (e.g., segment_id:1) in the session's entry. The disk_id field of the new segment record is initially empty and will be determined in step H720. The system also sets a segmentation flag and records metadata such as the trigger time.

[0374] This step is crucial for enabling dynamic switching of storage policies. By instantly modifying the core mapping data structure, the system logically opens new, parallel data channels for the same session. This provides a transparent routing basis for subsequently directing data streams to different physical hard drives, ensuring the correct forwarding of data packets within the system without altering the upper-layer session identification logic. This design maintains architectural clarity and scalability.

[0375] H720, subsequently distributing some network flow packets arriving with that network session ID to another available hard drive with the lightest load.

[0376] The system first obtains real-time load metrics (such as I / O queue depth and write latency, as in step H410) for all currently available hard drives (excluding those marked as faulty). Using the same method as in steps H420-H430, it calculates the load weight of each hard drive, but the goal is to select the hard drive with the "lightest current load" for the new segment. This typically means selecting the hard drive with the lowest load weight. An important constraint is to prioritize selecting another available hard drive different from the original hard drive (sdb) to achieve true load distribution.

[0377] From the moment of determination, new data packets arriving in the disk buffer belonging to this overloaded session will no longer be assigned to the original hard drive (sdb), but will be assigned to a newly selected hard drive (e.g., sdc) according to the new mapping relationship. The system updates the mapping table entry for this session, setting the disk_id field of segment1 to the selected new hard drive ID (e.g., sdc).

[0378] For session data packets that already exist in the disk cache and have been allocated to the original hard drive at the time of the decision, the original allocation remains unchanged, and the original hard drive will still write them. The splitting operation only affects subsequent new data.

[0379] This step enables dynamic, load-aware traffic segmentation. Instead of simple polling or random allocation, it intelligently redirects excess load to the most idle storage node based on real-time system conditions. This mechanism can quickly respond to changes in the load of each hard drive within the system, achieving automatic global load balancing. It effectively eliminates performance bottlenecks that may arise from a single hard drive serving a single large flow, thereby significantly improving the overall system throughput and resource utilization.

[0380] H730 generates and maintains a metadata index, which is used to record the logical association and physical location information of the same network session being split and stored on different hard drives.

[0381] While partitioned storage improves write performance, it disrupts the physical continuity of data. Therefore, a robust metadata indexing system must be established and maintained to ensure the correct reassembly of the original session data stream during reads. Create a separate, persistent session partitioned index table. This table, unlike the session mapping table, is designed to be stored long-term (e.g., written to a database or index file) for subsequent queries.

[0382] For example, session 0xVIDEO123 is split into sdb and sdc. The index table records are as follows:

[0383] { "session_id": "0xVIDEO123",

[0384] "segments": [

[0385] {"disk_id": "sdb", "file": " / data / sdb / stream_20240520_1.bin", "range": "0-104857600"},

[0386] {"disk_id": "sdc", "file": " / data / sdc / stream_20240520_1.bin", "range": "104857601-209715200"} ]}.

[0387] This step is crucial for ensuring the logical integrity and subsequent analyzability of the data. It addresses the core challenge of partitioned storage—physical data dispersion. By building and maintaining a precise global index containing logical-physical mappings, the system hides the complexity of the storage from the outside world, providing upper-layer applications with a unified and coherent view of session data. This allows all session-based analysis tools (such as network forensics, traffic replay, and deep testing) to directly and efficiently retrieve and reassemble complete data from any session without needing to concern themselves with the underlying storage details. This results in a significant improvement in write performance without compromising data availability or analytical value.

[0388] Furthermore, the method also includes:

[0389] H800 will skip a hard drive during subsequent allocation processes if it detects multiple consecutive disk write failures or a write rate that consistently falls below the minimum threshold.

[0390] The system sets up explicit success / failure status checks for each write operation to the hard drive. Specifically, after initiating a system call (such as write, pwrite) or asynchronous I / O callback, the return value or error code is checked. Critical errors include: I / O error, timeout, and device unresponsiveness. Each occurrence of one of these errors constitutes a "disk write failure".

[0391] The system periodically (e.g., every second) calculates the actual write throughput for each hard drive. The instantaneous write rate (MB / s) is obtained by reading / proc / diskstats or using iostat, multiplying the number of write sectors completed within each sampling period by the sector size (typically 512 bytes), and then dividing by the sampling interval.

[0392] Historical trend analysis: The system maintains a sequence of write rates over a rolling time window (e.g., the past 60 seconds) to determine "persistent" low performance.

[0393] This step isolates the hard drive from the work cluster before it experiences serious performance problems or complete failure, ensuring the overall stability of the system and the reliability of the data.

[0394] In this embodiment, firstly, addressing the issue of excessive data backlog caused by continuous high-bandwidth sessions, this invention allows only a portion of the data packets from the same session to be written in a single disk write operation, with the remaining data remaining in a cache pending write state. This "batch disk write" mechanism breaks through the cache blocking bottleneck caused by waiting for the complete session to be written in traditional solutions, ensuring the continuity and low latency of data stream processing and fundamentally avoiding the risks of cache overflow and data loss. Secondly, by strictly maintaining the "unfinished network session ID table" and ensuring that all data packets from the same session (including those written in batches) are directed to the same hard drive, this solution achieves efficient pipelined processing while perfectly maintaining the physical storage locality of data, laying a solid foundation for subsequent high-performance session-level retrieval and analysis. Furthermore, the two-level buffer structure of the network card cache and the disk write cache works in conjunction with the aforementioned intelligent allocation and batch disk write process, achieving decoupling and smooth connection of the receiving, buffering, allocation, and writing stages, enabling the system to handle traffic peaks with ease and stably coordinate high-speed network I / O with relatively low-speed disk storage. In summary, this method successfully optimizes system throughput, data locality, and processing real-time performance in complex high-throughput scenarios, achieving a unified improvement in reliability, efficiency, and analyzability.

[0395] Example 5:

[0396] In the above embodiments, the network flow data received by the network card can be segmented into several network flow data packets through the following steps; the network flow data packets in the above embodiments can also be obtained through the following steps:

[0397] Y100, in response to acquiring target network stream data, determines the communication type between the terminal devices corresponding to the target network stream data; the target network stream data is any network stream data received by the network card queue.

[0398] This step is the entry point of the method, responsible for capturing raw data packets from the network interface card (NIC) queue and performing initial classification. The system receives raw data link layer frames from a specified network interface (e.g., eth0) at the NIC driver layer or through a high-performance network framework such as DPDK (Data Plane Development Kit). For each received data packet (i.e., the target network flow data), the system first parses its Ethernet header to determine the type of the upper-layer protocol it carries (e.g., an Ethernet type field of 0x0800 represents IPv4). Next, it continues to parse the IP header, extracting its "Protocol" field. Based on the value of this field, the system can determine the communication type: if the protocol field value is 6, it indicates that the upper layer is TCP. TCP is a typical connection-oriented protocol, requiring a three-way handshake to establish a connection, and possesses reliable transmission, data reordering, and other state management mechanisms. Therefore, the system determines that the communication type corresponding to the target network flow data is a connection-oriented communication type. If the protocol field value is 17 (UDP), 1 (ICMP), or other connectionless protocols, the system determines that the communication type is a connectionless communication type.

[0399] This step achieves basic traffic splitting through protocol parsing, laying the foundation for subsequent implementation of differentiated and refined processing strategies. Distinguishing between connection-oriented and connectionless traffic is a primary and crucial step in respecting the inherent characteristics of network protocols. This enables the system to adopt stable strategies for the stateful nature of TCP streams, while simultaneously enabling more flexible intelligent processing channels for the variable UDP / ICMP streams, thus avoiding the blind application of strategies from the outset.

[0400] Y200: If the communication type between the terminal devices corresponding to the target network stream data is a connection-oriented communication type, then the target network stream data is segmented using a preset fixed duration.

[0401] For target network flow data determined to be connection-oriented (i.e., TCP), the system employs a simple and stable strategy: segmentation based on a preset fixed duration. Internally, the system maintains a session state machine for each TCP session (consisting of a 5-tuple: source IP, source port, destination IP, destination port, and protocol unique identifier), which includes a timer and a data block buffer. When the first packet belonging to a TCP session arrives, the system starts timing with the packet's timestamp and begins buffering all subsequent packets of that session (associated with the same 5-tuple) into the same memory block. When the timer reaches the preset fixed duration (e.g., T_fixed = 5 seconds), the system immediately packages all packets belonging to that TCP session in the current buffer into a single "network flow packet" block (e.g., stored as a pcap file or sent to the next processing stage), and resets the timer and buffer to prepare for the data in the next time window. This process repeats cyclically throughout the TCP session until TCP connection termination (e.g., a FIN packet) or timeout is detected.

[0402] For streams like TCP, which have inherent time and state dependencies, fixed-duration segmentation produces data blocks of relatively uniform size with clear time boundaries. This is highly beneficial for subsequent storage, indexing, and time-window-based analysis (such as "TCP throughput in the past 5 seconds"). The fixed-duration strategy is logically simple, has extremely low overhead, and effectively maintains the logical continuity of the TCP data stream and the analytical context, avoiding the uncertainties and additional overhead that complex algorithms may introduce, achieving a good balance between processing efficiency and predictability.

[0403] Y300: If the communication type between the terminal devices corresponding to the target network stream data is a connectionless communication type, then obtain the target task identifier corresponding to the target network stream data; the target task identifier is obtained based on the source MAC address, destination MAC address and communication protocol corresponding to the target network stream data.

[0404] Furthermore, the target task identifier is obtained through the following steps:

[0405] Y310 concatenates the source MAC address, destination MAC address, and communication protocol corresponding to the target network flow data in a preset order to obtain the fusion information corresponding to the target network flow data.

[0406] Y320 determines the target task identifier by using the hash value of the fusion information corresponding to the target network flow data.

[0407] The system extracts three key fields from the data packet: Source MAC address: the physical address of the sending network device, located in the Ethernet header; Destination MAC address: the physical address of the receiving network device, located in the Ethernet header; Communication protocol: the protocol field value in the IP header (e.g., 17 represents UDP).

[0408] The three fields mentioned above are concatenated into a string or byte stream in a predetermined order. For example, it can be formatted as {source MAC}:{destination MAC}:{protocol number} (e.g., 00:1A:2B:3C:4D:5E:00:1A:2B:3C:4D:60:17). Then, a collision-resistant hash function (such as SHA-256 or MurmurHash3) is applied to this string to generate a fixed-length (e.g., 128-bit) hash value. This hash value is the target task identifier. It can uniquely identify "a specific connectionless protocol task running between two specific physical devices" with a very high probability.

[0409] This step moves beyond the traditional five-tuple (IP + port) perspective, instead defining tasks from the dimension of physical devices and protocol types. This is particularly applicable to scenarios such as the Internet of Things (IoT) and industrial control, where device MAC addresses are relatively stable, and specific protocols on specific devices (such as a camera sending a video stream via UDP) often correspond to a fixed business task. The resulting task identifier is more directly linked to the underlying physical entity and business logic than traditional stream identifiers, providing a precise anchor for intelligent optimization based on task characteristics.

[0410] Y400: If the target task identifier exists in the preset segmentation duration mapping table, the target network flow data is segmented using the segmentation duration corresponding to the task identifier that is the same as the target task identifier in the segmentation duration mapping table. The preset segmentation duration mapping table includes several task identifiers, and each task identifier corresponds to a segmentation duration. The segmentation duration is determined based on the configuration information of the terminal device associated with the corresponding task identifier and the task feature information being processed.

[0411] The system maintains a pre-defined segmentation duration mapping table (e.g., a hash table or key-value database), where the key is the task identifier and the value is the segmentation duration. When the target task identifier (from Y300) of a connectionless data packet is obtained, the system searches this mapping table.

[0412] If a matching task identifier is found, its corresponding segmentation duration is immediately retrieved. The system maintains a context (buffer and timer) similar to Y200 for this task identifier, but uses this potentially dynamically changing duration obtained from the mapping table as the timer threshold to segment the data stream. The segmentation logic is similar to Y200, but the duration is customized.

[0413] If not found, the system can use a default conservative splitting time (such as a small value, 100 milliseconds) for processing, and may trigger an asynchronous process to try to calculate an optimized splitting time for the new task identifier and add it to the mapping table.

[0414] Each "Task Identifier-Segmentation Duration" record in the mapping table is determined offline or online through the Y410-Y440 steps. The network controller or management module periodically or as needed executes the Y410-Y440 process to calculate the latest optimized segmentation duration for each task identifier of interest and sends the updated segmentation mapping table in the data plane.

[0415] Furthermore, the segmentation duration corresponding to each task identifier in the segmentation duration mapping table is determined through the following steps:

[0416] Y410: Obtain the hardware configuration information of the terminal device associated with the specified task identifier and the task characteristic information of the running application; wherein, the task characteristic information includes the application type, typical data transmission cycle, historical data packet size distribution and task criticality level; the specified task identifier is any task identifier in the segmentation duration mapping table.

[0417] Hardware configuration information acquisition: The network management platform uses standard protocols (such as SNMP, NETCONF / YANG) or vendor-specific APIs to query network access devices (such as switches, APs) or direct management terminals associated with the source MAC address in the task identifier to obtain their hardware profile. Key information includes:

[0418] Network interface capabilities: maximum negotiated rate of the connected port (e.g., 1Gbps), duplex mode, and supported physical / data link layer standards (e.g., 802.11ax).

[0419] Device processing capabilities (e.g., for smart terminals): CPU model, number of cores, and memory size reported by the terminal management agent. For example, the hardware configuration of an IoT camera used for video surveillance might be recorded as "CPU: Dual-core ARMA53, 1.2GHz; RAM: 512MB; Network: 100Mbps wired".

[0420] Task feature information acquisition: Analysis is performed at the gateway using a lightweight application performance management (APM) probe or deep packet inspection (DPI) engine deployed on the endpoint to collect:

[0421] Application type: Based on traffic behavior pattern recognition or port mapping, it is classified into "real-time video streaming", "VoIP", "intermittent sensor reporting", "batch file transfer", etc. For example, traffic that continuously sends fixed-size packets on port 5060 can be classified as "VoIP".

[0422] Typical data transmission cycle: The mode or average of the time interval between two consecutive actively transmitted data packets in a statistical application under steady-state conditions. For example, an environmental sensor application might have a cycle of "5 minutes," while a heartbeat packet in an online game might have a cycle of "10 seconds."

[0423] Historical packet size distribution: Collect a histogram of all packet sizes for this task within the past statistical window (e.g., 1 hour) and calculate the main concentration interval (e.g., 80% of the packet sizes are between 100-200 bytes).

[0424] Task criticality level: Pre-configured by the network policy server or passed based on business system tags. For example, in an industrial network, control command flows are labeled "Critical", status monitoring flows are labeled "Important", and ordinary log flows are labeled "Normal".

[0425] This step establishes a refined cognitive foundation at the task level, mapping abstract network traffic to specific physical device capabilities and upper-layer business intents. By integrating the static capability limits of devices (hardware configuration) with the dynamic behavioral patterns of applications (task characteristics), it provides comprehensive and differentiated input dimensions for subsequent intelligent decision-making. This transcends the limitations of traditional network management that only focuses on link status, achieving a leap from "pipe management" to "joint awareness of business and pipelines," ensuring that the formulation of segmentation strategies fully considers all key constraints on the end-to-end path, and laying a solid data foundation for personalized optimization.

[0426] Y420, based on the hardware configuration information and task feature information, constructs a multi-dimensional feature vector corresponding to the specified task identifier, and inputs the multi-dimensional feature vector into the pre-trained duration prediction model; the duration prediction model is trained on a historical dataset using machine learning methods, and the historical dataset contains network stream data samples of different task types with known optimal duration segments and their corresponding multi-dimensional feature vectors.

[0427] Constructing multidimensional feature vectors:

[0428] Encode hardware configuration information: network speed is converted into a logarithmic scale value (e.g., log10(speed / Mbps)); CPU capability can be normalized into a relative performance score (between 0 and 1) based on benchmark tests.

[0429] The task feature information is encoded as follows: "Application type" uses one-hot encoding; "typical sending cycle" and "median historical packet size" are normalized by max-min; "task criticality level" is mapped to an ordered numerical sequence (e.g., ordinary = 1, important = 2, critical = 3).

[0430] All encoded values ​​are concatenated in a predetermined order into a fixed-length numerical vector, i.e., a multidimensional feature vector. For example, a vector may be represented as: [0.8 (network capacity), 0.3 (CPU capacity), 0,1,0 (video stream type), 0.15 (normalized period), 0.05 (normalized packet size), 3 (criticality level)].

[0431] Train a regression model (such as Gradient Boosting Machine (GBM), Random Forest, or Lightweight Neural Network) using a historical dataset. Each sample in the dataset contains two parts: 1) a "known optimal segmentation duration" label (this label can be derived from historical experiments, aiming to optimize the overall metrics of throughput, latency, and packet loss rate); 2) a "multidimensional feature vector" constructed at the corresponding time point of the historical task using the above method. The goal of the model is to learn a mapping function from the feature vector to the optimal duration.

[0432] The real-time multidimensional feature vector of the currently specified task identifier is input into the loaded pre-trained model. The model performs forward propagation calculation and outputs a continuous value representing the base segmentation duration proposal (T_base), typically in milliseconds.

[0433] This step utilizes a machine learning model to achieve intelligent mapping from multi-dimensional features to optimized segmentation time. It effectively mines hidden empirical patterns from massive amounts of historical operational data, avoiding the one-sidedness and rigidity of relying on manual empirical formulas. The model can automatically handle complex interactions and non-linearities between features, providing high-quality initial segmentation suggestions even for unprecedented combinations of new tasks. This is equivalent to equipping the network system with an "experienced expert system," enabling it to quickly allocate appropriate initial time windows for various connectionless services in a data-driven manner, significantly improving the scientific rigor and efficiency of strategy formulation.

[0434] Y430, based on the output of the duration prediction model, obtains the basic segmentation duration suggestion value corresponding to the specified task identifier.

[0435] The output value is limited to a reasonable physical range [T_min, T_max] preset by the system. For example, T_min is 10 milliseconds (to avoid excessive overhead due to overly fine segmentation), and T_max is 10 seconds (to avoid losing flexibility due to overly coarse segmentation). If the output value exceeds this range, the nearest boundary value is used.

[0436] Y440 dynamically adjusts the basic segmentation duration recommendation value based on real-time network status information to obtain the segmentation duration corresponding to the specified task identifier; the real-time network status information includes the current network link latency, the real-time CPU load of the terminal device corresponding to the specified task identifier, and the current queue depth of the switch on the data flow path.

[0437] Furthermore, the basic segmentation duration recommendation value is dynamically adjusted based on real-time network status information, including:

[0438] Y441 compares the obtained network link latency, terminal device CPU load, and switch queue depth on the data flow path with the corresponding preset thresholds to determine the impact coefficients of network link latency, terminal device CPU load, and switch queue depth on the data flow path on the segmentation time.

[0439] Determine the network link delay impact factor (K_delay):

[0440] Preset threshold: Set a baseline network latency threshold D_base (e.g., the historical average RTT of the task under good network conditions).

[0441] Calculation logic: K_delay = D_base / max(D_current, D_min). D_current is the average round-trip time currently measured. D_min is a protective value to prevent the denominator from being too small (e.g., 1ms). If the current delay D_current is higher than the baseline D_base (network degradation), K_delay is less than 1, and the system will tend to shorten the splitting time to reduce the queuing time of a single data block in the congestion queue. The lower the delay, the closer K_delay is to or greater than 1.

[0442] Determine the impact factor of terminal device CPU load (K_cpu):

[0443] Preset threshold: Set a threshold C_base that represents the load safety (e.g., 70% of CPU utilization).

[0444] Calculation logic: A piecewise function is used. For example: if C_current ≤ C_base, then K_cpu = 1.0 (normal load, no impact). If C_current > C_base, then K_cpu = 1.0 + (C_current - C_base) / (100 - C_base). Here, C_current is the current CPU utilization percentage.

[0445] When the CPU load exceeds the safety threshold (K_cpu > 1), the system will tend to extend the segmentation time to reduce the frequency of data packet arrival and processing, thereby reducing the pressure on terminal devices.

[0446] Determine the influence coefficient of the switch queue depth (K_queue):

[0447] Preset threshold: Set a queue depth safety threshold Q_base (e.g., 40% of the total length of the switch port queue).

[0448] Calculation logic: K_queue = Q_base / max(Q_current, 1). Q_current is the current queue depth.

[0449] When the queue depth Q_current exceeds the safety threshold Q_base (the buffer is about to become congested), K_queue < 1. The system will strongly tend to shorten the splitting time, speed up data flow, and prevent packet loss.

[0450] This step uses concise and clear mathematical relationships to uniformly map real-time status indicators (latency, load, queue depth) with different dimensions and meanings to adjustment coefficients for the control variable "segmentation duration". The physical meaning of this mapping relationship is clear: when the network is congested (high latency, deep queues), the coefficient < 1, prompting the system to adopt finer-grained segmentation to improve agility; when computing resources are strained (high CPU load), the coefficient > 1, prompting the system to adopt coarser-grained segmentation to reduce frequency. This provides standardized and quantifiable input for subsequent intelligent weighted decision-making, serving as a crucial bridge connecting "state awareness" and "policy generation".

[0451] Y442, based on the criticality level of the associated business of the specified task identifier, assigns weights to each influence coefficient and performs weighted fusion to obtain a comprehensive correction factor.

[0452] This step involves weighting and summing the three impact coefficients based on the criticality level of the business, generating a unified comprehensive correction factor K_total.

[0453] Preset weight configuration: Three different weight vectors (W_delay, W_cpu, W_queue) are preset for different task criticality levels (e.g., "critical", "important", "normal"). The sum of the weights is 1. For example: Critical business (e.g., control commands): (0.5, 0.2, 0.3). Network latency (W_delay=0.5) is given the highest weight because it is extremely sensitive to latency and reliability and should be prioritized in responding to network congestion. Important business (e.g., video surveillance): (0.3, 0.4, 0.3). Relatively balanced, possibly with slight attention to CPU load (W_cpu=0.4) to ensure smooth video encoding / decoding. Normal business (e.g., file download): (0.2, 0.3, 0.5). Queue depth (W_queue=0.5) is given a higher weight, mainly adjusted from the perspective of global network buffer resource utilization efficiency.

[0454] Calculate the comprehensive correction factor: Based on the task criticality level associated with the currently specified task identifier, select the corresponding weighted reorganization (W_d, W_c, W_q). Calculate the weighted sum: K_total = (K_delay × W_d) + (K_cpu × W_c) + (K_queue × W_q). K_total is the comprehensive correction factor that integrates multi-dimensional states and business strategies.

[0455] This step achieves a "business intent-driven" strategy integration. Instead of mechanically treating all network metrics equally, it allows for dynamic adjustments to decision-making priorities based on the core business needs (criticality level). For example, for life-or-death control commands, the system will respond more aggressively to changes in network latency, sacrificing some terminal load to ensure timeliness; for background downloads, it will focus more on avoiding network queue congestion. This design directly translates high-level network service level agreements (SLAs) or business policies into low-level, real-time parameter adjustment behaviors, enabling network resource scheduling to truly possess business-aware capabilities and differentiated service assurance capabilities.

[0456] Y443, multiply the basic segmentation duration suggestion value by the comprehensive correction factor to obtain the final corrected segmentation duration; wherein, the comprehensive correction factor causes the output segmentation duration to be automatically shortened when the network condition deteriorates.

[0457] Multiply the suggested base segmentation duration T_base by the comprehensive correction factor K_total to obtain the initial correction value: T_temp = T_base × K_total. Clamp T_temp within a globally valid range allowed by the system [T_min, T_max]. For example, T_min is 10 milliseconds (to avoid protocol header overhead from excessive segmentation), and T_max is 5 seconds (to avoid sluggish response). Finally, T_final = clamp(T_temp, T_min, T_max).

[0458] The calculated T_final is used as the latest segmentation duration for the specified task identifier, and the corresponding entry in the segmentation duration mapping table is updated. Afterward, the data plane module (performing step Y400) will directly use this updated duration value to segment the data stream for the task until the next dynamic correction cycle arrives.

[0459] This step completes the final transformation from intelligent decision-making to execution parameters. Through multiplicative correction and safety clamping, it ensures that the final segmentation duration incorporates both the baseline prediction based on long-term characteristics (T_base) and sensitively reflects real-time network conditions and business strategies (K_total), while avoiding system instability caused by extreme values. In particular, due to the mechanism that K_delay and K_queue are less than 1 when the network deteriorates, it ensures that K_total also tends to be less than 1 when network conditions worsen, thus achieving the core design goal of "automatically shortening the segmentation duration" and effectively dealing with congestion. The entire mechanism forms a complete "monitoring-analysis-decision-execution" closed loop, enabling the segmentation behavior of network data flow to be continuously and adaptively optimized, maintaining high performance and high reliability in dynamic environments.

[0460] Y500: If the target task identifier does not exist in the preset segmentation duration mapping table, the target network flow data is segmented using a preset fixed duration to obtain several network flow data packets corresponding to the target network flow data.

[0461] During the Y400 step, the system failed to find an entry in the preset segmentation duration mapping table that matched the target task identifier of the current data packet (generated by the Y300 step). This usually means that the task is appearing for the first time, or its strategy has not yet been calculated and injected. The system immediately adopts a preset fixed duration (T_default) as the segmentation basis. This T_default is a conservative value that has undergone global trade-offs, for example, set to 100 milliseconds.

[0462] Using a globally conservative preset fixed duration (such as 100ms) can achieve a broadly acceptable balance between protocol overhead, processing latency, and memory usage during the initial processing, avoiding data backlog or loss due to the lack of available strategies.

[0463] Furthermore, the method also includes the following steps:

[0464] If Y600 receives a certain number of pending connectionless network stream data with different target task identifiers but the same source MAC address and communication protocol type within a preset time period, and the destination MAC address of the pending connectionless network stream data belongs to the same broadcast domain or server cluster, then the pending connectionless network stream data will be merged into the same logical aggregation task.

[0465] The system monitors all unknown connectionless data streams within an observation window. If multiple data streams originating from the same source MAC device, using the same protocol (such as UDP), are found to exceed a threshold, and the destination MAC addresses of these streams belong to the same server cluster or broadcast domain, these streams are merged into a single logical aggregation task.

[0466] Y610 generates an aggregation task identifier for the logical aggregation task, associates the aggregation task identifier with an optimized aggregation segmentation duration, and stores it in the segmentation duration mapping table; the aggregation segmentation duration is used to guide the segmentation of subsequent similar aggregation data streams.

[0467] The source MAC address, protocol, and target cluster identifier are concatenated and hashed to generate a unique aggregation task identifier. Based on the overall characteristics of the aggregation task (such as the most critical member requirements and overall traffic patterns) and the real-time network status from the source device to the target cluster, an optimized aggregation segmentation duration is obtained through model calculation and dynamic correction (the same model as Y420 can be used, but the training data must include aggregation task samples). The identifier and duration are stored as a new record in the segmentation duration mapping table. Thereafter, any member flow belonging to this aggregation task can match this strategy and be segmented using a unified aggregation duration.

[0468] This step establishes a coordinated and unified optimization strategy for the aggregation task. By adopting a "one-time calculation, unified application" approach, not only is the efficiency of strategy management greatly improved, but it also ensures that a large number of data streams sent to the same target can maintain a synchronized rhythm, thus smoothing network traffic from a macro perspective, avoiding sudden congestion, and optimizing the overall network resource utilization efficiency and transmission performance.

[0469] Furthermore, the method also includes the following steps:

[0470] Y700: If the target task identifier exists in the segmentation duration mapping table and the target network flow data has been segmented using the corresponding segmentation duration, then the packet arrival rate of the target network flow data will be continuously monitored.

[0471] For each task identifier that uses a duration segmentation table, a lightweight traffic monitor is deployed on the data plane. This monitor counts the number of packets arriving for that task identifier within a fixed time window (e.g., every 100 milliseconds), calculates the real-time packet arrival rate (packets / second), and calculates the rate change rate (growth rate) between adjacent windows. The monitoring results are fed back to the policy control module.

[0472] Y710 If the detected packet arrival rate growth rate is greater than the preset growth rate threshold, a burst mode segmentation duration is created for the target task identifier, and the segmentation duration corresponding to the target task identifier in the segmentation duration mapping table is overwritten.

[0473] When the packet arrival rate growth rate of a certain task continuously exceeds a preset threshold (e.g., 200% per second), it is determined to be a burst mode. Immediately, based on the current real-time network status (latency, queue depth) and the severity of the burst, a temporary, shorter burst mode segmentation duration is calculated. For example, the original duration can be multiplied by an emergency reduction factor (e.g., 0.3). This burst segmentation duration is immediately written into the segmentation duration mapping table, overwriting the original duration. Simultaneously, a preset timeout timer (e.g., set to 10 seconds) is started.

[0474] This step provides a second-level emergency response capability for sudden traffic surges. By quickly switching to a more aggressive segmentation strategy, the system can transform sudden large traffic surges into finer-grained data blocks, thereby reducing the load on a single transmission, effectively alleviating network buffer pressure, and avoiding instantaneous packet loss and latency spikes caused by sudden traffic surges.

[0475] Y720: Start the preset timeout timer. If the packet arrival rate drops back to the normal range before the timer expires, the original segmentation duration in the mapping table will be restored. If the packet arrival rate does not drop back to the normal range after the timer expires, this sudden behavior will be recorded as a new feature of the target task identifier, and the segmentation duration of the target task identifier in the segmentation duration mapping table will be permanently updated.

[0476] During the timeout period, the packet arrival rate is continuously monitored. If the rate drops below the burst threshold and stabilizes for a period of time (e.g., three consecutive monitoring windows are normal), the value in the segmentation duration mapping table is immediately restored to the original segmentation duration. If the packet arrival rate remains above the burst threshold after the timer expires, this burst is determined to be the new normal for this task. The system records this burst behavior characteristic (such as peak rate and duration) as a new feature for identifying this task. Based on this new feature, steps Y410-Y440 are re-executed (which can be simplified to incremental calculation) to generate a new permanent segmentation duration adapted to the burst pattern and update it in the mapping table.

[0477] This step enables intelligent judgment and policy adaptation to changes in traffic patterns. Through a "temporary coverage-timeout evaluation" mechanism, the system can handle both short-term traffic spikes and identify persistent changes in business patterns. When a temporary surge ends, the policy automatically reverts to maintain normal efficiency; when a long-term pattern change is detected, the new experience is incorporated into a permanent policy. This design gives the system the ability to continuously learn and evolve, enabling it to track and adapt to the dynamic evolution of business traffic over the long term, ensuring the timeliness and optimality of the policies.

[0478] Furthermore, the method also includes the following steps:

[0479] The Y800 stores the segmented network flow data packets into a preset network card buffer, and uses a preset disk write buffer to write the network flow data packets to disk.

[0480] In this embodiment, Y800 can be implemented using the methods described in Embodiments 2 to 5, which will not be elaborated here.

[0481] In this embodiment, the basic strategy of traffic splitting is first implemented by intelligently identifying the communication type of network flows (connection-oriented and connectionless). For stateful sessions such as TCP, stable fixed-duration segmentation is adopted to ensure the temporal continuity and predictability of data flow. More importantly, for connectionless and variable traffic such as UDP and ICMP, the solution creatively introduces the concept of "target task identifier" based on source MAC, destination MAC, and communication protocol, and constructs a dynamically learnable and updateable segmentation duration mapping table. This allows the system to allocate appropriate and personalized segmentation durations for different connectionless business logics (such as video streams, sensor data, and control signaling) based on the hardware configuration of the terminal device and the task characteristics of the running application (such as cycle, data volume, and criticality). On the one hand, this avoids the additional latency introduced by setting excessively long segmentation durations for high real-time tasks (such as audio and video streams); on the other hand, it also prevents increased processing overhead and storage fragmentation caused by excessively fragmented segmentation of low-frequency tasks (such as device status reporting). Ultimately, this method achieves intelligent and refined network packet segmentation strategies as a whole, significantly improving the processing efficiency and resource utilization of subsequent packet caching, transmission, disk storage, and analysis.

[0482] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0483] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0484] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0485] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0486] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0487] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0488] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0489] The electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0490] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and a bus connecting different system components (including memory and processor).

[0491] The memory stores program code that can be executed by the processor, causing the processor to perform the steps in the various embodiments described in this specification.

[0492] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0493] The memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0494] A bus can represent one or more of several bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0495] Electronic devices can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable users to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, electronic devices can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapters. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0496] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0497] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0498] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.< / sdx> < / sdx> < / sdx>

Claims

1. A method for balanced disk persistence of network flow data packets, characterized in that, include: Real-time monitoring and acquisition of the system's first real-time occupancy rate of the network card cache, the second real-time occupancy rate of the disk cache, the current network card data receiving rate, the current data transfer rate from the network card cache to the disk cache, and the current aggregate disk write rate of the hard disk array; Based on the mismatch between the network card data receiving rate, transfer rate and aggregate disk write rate, the first deviation between the real-time occupancy rate of the network card buffer and the preset target occupancy rate, and the second deviation between the real-time occupancy rate of the disk write buffer and the preset target occupancy rate, a pressure index is determined to characterize the overall equilibrium state of the system. Based on the range of values ​​for the pressure index, determine whether the system is in equilibrium. If the system is not in an equilibrium state, the transfer rate is adjusted through a preset feedback control algorithm; The control quantity of the feedback control algorithm is determined comprehensively based on the pressure index, the error of the network card buffer occupancy rate relative to the target value, and the error of the disk buffer occupancy rate relative to the target value. According to the adjusted transfer rate, the network flow data packets temporarily stored in the network card buffer are transferred to the disk buffer. Assign corresponding hard disk identifiers to network stream data packets in the disk buffer, and write the network stream data packets to the corresponding hard disks in batches.

2. The method according to claim 1, characterized in that, The feedback control algorithm includes: Based on the error and pressure index changes of the network card buffer occupancy rate and disk buffer occupancy rate relative to the target value at the current and previous times, the adjustment amount of the transfer rate is calculated by a weighted combination of proportional, integral, and derivative components. Among them, the proportional component is proportional to the current error, the integral component is proportional to the historical cumulative error, and the derivative component is proportional to the error change rate.

3. The method according to claim 1, characterized in that, The mismatch between the network card's data reception rate, transfer rate, and aggregated disk write rate is determined through the following steps: Calculate the square of the difference between the network card's data receiving rate and the transfer rate, and the square of the difference between the transfer rate and the aggregation disk write rate. The sum of the two calculated square values ​​is taken as the square root to obtain a total rate deviation value. The mismatch degree is obtained by normalizing the total rate deviation by dividing it by the network card data receiving rate.

4. The method according to claim 1, characterized in that, The first deviation is obtained through the following steps: Calculate the first absolute difference between the first real-time occupancy rate and the preset target occupancy rate; Divide the first absolute difference by the preset target occupancy rate to obtain a relative deviation value; The relative deviation value is input into a preset mathematical function for processing, and the mathematical function is configured to assign a higher penalty weight to larger relative deviation values. The output value of the mathematical function is used as the first deviation.

5. The method according to claim 1, characterized in that, The second deviation is determined through the following steps: Calculate the second absolute difference between the second real-time occupancy rate and the preset target occupancy rate; The second absolute difference is compared with a preset allowable deviation range. If the second absolute difference does not exceed the allowable deviation range, the deviation between the real-time occupancy rate of the disk buffer and the preset target occupancy rate is determined to be zero. If the second absolute difference exceeds the allowable deviation range, the second deviation degree is calculated based on the size of the excess portion using a preset nonlinear mapping relationship; wherein, the growth rate of the second deviation degree increases as the excess portion increases.

6. The method according to claim 1, characterized in that, The pressure index is determined through the following steps: The quantized values ​​of the rate mismatch, network interface card (NIC) buffer deviation, and disk write-to-disk buffer deviation are calculated separately. The rate mismatch is calculated based on the difference between any two of the NIC data receiving rate, transfer rate, and aggregated disk write-to-disk rate. The NIC buffer deviation is calculated based on a first deviation, and the disk write-to-disk buffer deviation is calculated based on a second deviation. A first weighting coefficient is assigned to the rate mismatch item, a second weighting coefficient is assigned to the network card buffer deviation item, and a third weighting coefficient is assigned to the disk buffer deviation item; the sum of the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient is a fixed value; when the system starts up or there is a sudden change in traffic, the first weighting coefficient is greater than the second weighting coefficient and the third weighting coefficient; when the system is running stably, both the second weighting coefficient and the third weighting coefficient are greater than the first weighting coefficient. The rate mismatch, network card cache deviation, and disk cache deviation are multiplied by their respective weighting coefficients, and the three products are summed to obtain the stress index.

7. The method according to claim 6, characterized in that, The sum of the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient is 1.

8. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the method as described in any one of claims 1-7.

9. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 8.

Images