A method for batch data parsing, an electronic device, a storage medium, and a program product.

Abstract

This invention discloses a data batch parsing method, electronic device, storage medium, and program product. The method includes: determining a load pressure index based on processor utilization, processing time, upper limit of processing time, load weight, and timeliness weight; determining a dynamic batch processing quantity based on the load pressure index, upper limit of batch processing quantity, and lower limit of batch processing quantity; acquiring transmitted data and caching the transmitted data in a data cache area; and when the data cache size in the data cache area reaches the dynamic batch processing quantity, batch acquiring data from the data cache area for data parsing. By determining the load pressure index in real time and dynamically determining the batch processing quantity for data parsing based on the real-time load pressure index, data parsing can adapt to changes in system load, fully utilizing load capacity while avoiding load pressure.

Description

Technical Field

[0001] This invention relates to the field of communication network congestion control technology, and in particular to a data batch parsing method, electronic device, storage medium, and program product. Background Technology

[0002] In network communication, high-concurrency traffic scenarios place stringent demands on system stability, message processing efficiency, and resource utilization.

[0003] In existing technologies, a fixed batch processing threshold is typically used to trigger batch data parsing. However, while a fixed batch processing threshold simplifies the processing logic, it cannot dynamically adjust data parsing based on system load. This approach may result in excessive processing pressure per cycle or reduced real-time performance of data parsing due to excessively long waiting times.

[0004] Therefore, there is an urgent need to provide a method for batch data parsing that dynamically follows load pressure, adapting to data transmission, parsing, and flow control in high-concurrency scenarios. Summary of the Invention

[0005] This invention provides a data batch parsing method, electronic device, storage medium, and program product to enable data parsing to adapt to changes in system load, thereby fully utilizing load capacity while avoiding load pressure.

[0006] According to one aspect of the present invention, a method for batch data parsing is provided, the method comprising: The load pressure index is determined based on processor utilization, processing time, processing time limit, load weight, and timeliness weight. The dynamic batch size is determined based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size. Acquire the transmission data and cache the transmission data in the data cache area; When the amount of data cached in the data cache reaches the dynamic batch processing quantity, data in the data cache is retrieved in batches for data parsing.

[0007] According to another aspect of the present invention, a data batch parsing apparatus is provided, the apparatus comprising: The load stress index determination module is used to determine the load stress index based on processor utilization, processing time, processing time limit, load weight, and timeliness weight. The dynamic batch processing quantity determination module is used to determine the dynamic batch processing quantity based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity. The data caching module is used to acquire the transmitted data and cache the transmitted data in the data cache area; The data parsing module is used to batch retrieve data from the data cache area and perform data parsing when the data cache size in the data cache area reaches the dynamic batch processing quantity.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the data batch parsing method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data batch parsing method according to any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the data batch parsing method described in any embodiment of the present invention.

[0011] The technical solution of this invention determines the load pressure index based on processor utilization, processing time, upper limit of processing time, load weight, and timeliness weight; determines the dynamic batch processing quantity based on the load pressure index, upper limit of batch processing quantity, and lower limit of batch processing quantity; acquires the transmitted data and caches the transmitted data in the data cache area; when the data cache amount in the data cache area reaches the dynamic batch processing quantity, batches of data in the data cache area are acquired for data parsing. This solves the flow control problem in high-concurrency scenarios in network communication. By determining the load pressure index in real time and dynamically determining the batch processing quantity for data parsing based on the real-time load pressure index, data parsing can adapt to changes in system load, making full use of load capacity while avoiding load pressure.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a data batch parsing method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a batch data parsing method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the application process of a batch data parsing method provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of a data batch parsing device provided in Embodiment 3 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the data batch parsing method of this invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] Example 1 Figure 1This is a flowchart of a data batch parsing method according to Embodiment 1 of the present invention. This embodiment is applicable to high-concurrency scenarios where data transmission, parsing, and flow control are performed based on the Transmission Control Protocol (TCP). This method can be executed by a data batch parsing device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers, servers, or gateway devices. Figure 1 As shown, the method includes: Step 110: Determine the load pressure index based on processor utilization, processing time, processing time limit, load weight, and timeliness weight.

[0018] For example, the load pressure index can be expressed by the formula Confirmed. Among them, This refers to processor utilization, which can be CPU utilization collected in real time. To address processing time consumption, processing time can be collected within a sliding window, and the average processing time can be determined based on the collected results. The maximum processing time is set, for example, 100 milliseconds (ms). For load weight, As a time-sensitive factor, and The sum is 1. Load weight and timeliness weight can be determined based on the business scenario. For business scenarios that prioritize processor usage, such as compute-intensive tasks, the load weight can be set higher than the timeliness weight. For example, compute-intensive tasks could be data encryption and compression, with a load weight of 0.7 and a timeliness weight of 0.3. For business scenarios that prioritize processing time, such as I / O-intensive tasks, the load weight can be set lower than the timeliness weight. For example, I / O-intensive tasks could be database writes or forwarding, with a load weight of 0.3 and a timeliness weight of 0.7.

[0019] By determining the real-time load pressure index under different business scenarios based on processor utilization and processing time, the load capacity can be quantified, which facilitates subsequent batch data parsing adjustments and enables batch data parsing to adapt to dynamic load conditions.

[0020] Step 120: Determine the dynamic batch size based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size.

[0021] For example, the number of dynamic batches can be expressed by the formula Confirmed. Among them, This is the lower limit for the batch processing quantity, i.e., the minimum number of batch aggregations allowed by the system. Setting it to a value in the range of 10 to 50 can help avoid frequent context switching. This is the maximum batch size allowed by the system, such as a value between 1000 and 5000. The maximum number of batches can be determined based on the memory capacity used for data caching in the data cache area. For example, the maximum number of batches can be determined based on the memory capacity, data size, and preset caching factor. For instance, this can be achieved using a formula... Determine the maximum number of items that can be processed in a batch. In the formula, This refers to the memory capacity used for data caching in the data cache area. This refers to the data size, such as the average size of data transferred within batch processing or a sliding window. In practical applications, the preset buffer coefficient can be set to a value in the range of 0.7 to 0.9. For example, in the formula for determining the upper limit of batch processing quantity, the preset buffer coefficient can be set to 0.8.

[0022] Based on the method for determining the dynamic batch size, there is a negative correlation between the dynamic batch size and the load pressure index. When the load pressure index is low, the load is idle, and a larger dynamic batch size tends to aggregate in large batches to save CPU. When the load pressure index is high, the load is busy, and a smaller dynamic batch size reduces the risk of single-time blocking. Specifically, when both processor utilization and processing time are low, the load pressure index is low, and a larger dynamic batch size saves CPU usage through large-batch aggregation. When processor utilization is low and processing time is high, the load pressure index is moderate, and a moderate dynamic batch size is appropriate. When both processor utilization and processing time are high, the load pressure index tends to 1, and a small dynamic batch size is used to alleviate load pressure through fast refreshing.

[0023] In this embodiment of the invention, the memory capacity used for data caching in the data cache area can be dynamically determined by the queue level of the data cache area. Optionally, the dynamic batch processing quantity is determined based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity, including: determining the queue level based on the current queue length, the upper limit of the queue capacity, the queue enqueue rate, the queue dequeue rate, the static capacity weight, and the dynamic trend weight of the data cache area; adjusting the memory capacity used for data caching in the data cache area based on the queue level; determining the upper limit of the batch processing quantity based on the memory capacity, the data size, and the preset caching coefficient; and determining the dynamic batch processing quantity based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity.

[0024] For example, according to the formula Determine the water level in the queue. In the formula, The current queue length can be determined by real-time queue monitoring; This represents the maximum queue capacity. The queue enqueue rate, such as the average enqueue rate over the past second; This represents the queue dequeue rate, such as the average dequeue rate over the past second. For static capacity weights, For dynamic trend weights. The expression for the activation function Sigmoid is: , where e is a natural constant. Static capacity weights and dynamic trend weights can be preset values, such as 0.6 and 0.4 respectively. Determining the queue level by considering the amount of space occupied by the data and the flow rate allows for accurate feedback of the queue level based on flow trends while taking into account data storage volume, enabling dynamic adjustments to data parsing. The advantage of considering the flow rate as a sigmoid function when determining the queue level is that when the queue inbound rate exceeds the queue outbound rate, even if the current queue length is low, the second term in the queue level formula will increase rapidly, triggering an early warning and predicting back pressure. In other words, by combining instantaneous capacity and flow acceleration in a multi-dimensional queue level prediction model, back pressure can be triggered before the queue is full, eliminating control lag.

[0025] When determining the queue level, the static capacity weight and dynamic trend weight can be determined based on the business scenario. Optionally, the queue level can be determined based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer. This includes: determining the static capacity weight and dynamic trend weight based on the business's sensitivity to sudden traffic surges and the business's expected throughput; and determining the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer.

[0026] For example, burst traffic sensitivity is positively correlated with dynamic trend weight; that is, the higher the burst traffic sensitivity, the greater the dynamic trend weight. Throughput expectation is positively correlated with static capacity weight; that is, the greater the throughput expectation, the greater the static capacity weight. The sum of the static capacity weight and the dynamic trend weight is 1. For instance, for services with a balance between burst traffic sensitivity and throughput expectation, the static capacity weight can be set to 0.6 and the dynamic trend weight to 0.4. For services with high burst traffic sensitivity, such as those where no overflow is allowed, the static capacity weight can be set to 0.4 and the dynamic trend weight to 0.6 to allow for earlier trend warnings. For services with high throughput expectation, such as those with large system memory, the static capacity weight can be set to 0.8 and the dynamic trend weight to 0.2 to ensure that batch data parsing takes into account continuous throughput.

[0027] After determining the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight, the memory capacity used for data caching in the data buffer can be adjusted according to the queue level. For example, a larger memory capacity can be used when the queue level is normal; as the queue level increases, i.e., data processing congestion increases, a smaller memory capacity can be used to reduce congestion.

[0028] Furthermore, the upper limit of the batch processing quantity can be determined based on memory capacity, data size, and preset caching coefficient; the dynamic batch processing quantity can be determined based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity. Thus, by dynamically adjusting the batch processing quantity, congestion problems in data processing can be resolved, and the performance of batch data parsing can be improved.

[0029] Step 130: Obtain the transmission data and cache it in the data cache area.

[0030] The transmitted data can be data traffic received based on the TCP protocol. The transmitted data can be directly cached in the data buffer, decoupling data storage and parsing, allowing asynchronous execution of data storage and parsing, and improving the concurrency capabilities of the data processing system.

[0031] One or more cache queues can be set in the data cache area. When setting a single cache queue in the data cache area, data can be cached according to data priority. When setting multiple cache queues in the data cache area, the cache queues can be set as multiple priority queues, each caching data of different priorities.

[0032] Optionally, acquiring and caching the transmitted data in the data cache includes: acquiring the static priority of the data in the transmitted data, and determining the dynamic priority of the data based on the static priority, the data arrival time, the current time, and the data aging step size; and caching the transmitted data in the data cache according to the dynamic priority of the data.

[0033] The static priority of transmitted data can be priority information contained within the data itself. For example, each piece of transmitted data can be encapsulated as a tuple when enqueued: (effective_priority, arrival_timestamp, data), where the initial value of effective_priority is equal to the configured static priority, and arrival_timestamp is the data arrival time. Dynamic data priority can be expressed using the formula... Confirmed. In the formula, For static data priority, For the current time, For data arrival time, The data aging step can be set to 100ms. For example, for a low-priority transmission data, such as a priority of 3, after waiting in the data buffer for 200ms, the data's dynamic priority can be updated to 3-2=1, meaning it becomes high-priority transmission data and competes for message processing. In this embodiment of the invention, a lower priority value indicates a higher priority level. Caches transmission data in the data buffer according to its dynamic priority for processing, ensuring that any data, even in the worst-case scenario, will eventually be elevated to the highest priority and processed, completely resolving the "starvation" phenomenon and achieving dynamic fairness in data processing. The priority mechanism avoids the problem of high-priority data not being processed in a timely manner due to a large volume of low-priority data. Furthermore, the dynamic data aging method in this embodiment of the invention also avoids the problem of low-priority data constantly yielding to high-priority data processing due to its low priority level.

[0034] Step 140: When the amount of data cached in the data cache reaches the dynamic batch processing quantity, batch retrieve data from the data cache for data parsing.

[0035] For example, a zero-copy circular buffer and a resilient asynchronous task pool can be used to ensure that I / O reception is not interrupted under high load.

[0036] The technical solution of this embodiment determines the load pressure index based on processor utilization, processing time, upper limit of processing time, load weight, and timeliness weight; determines the dynamic batch processing quantity based on the load pressure index, upper limit of batch processing quantity, and lower limit of batch processing quantity; acquires the transmitted data and caches the transmitted data in the data cache area; when the data cached amount in the data cache area reaches the dynamic batch processing quantity, the data in the data cache area is acquired in batches for data parsing. This solves the flow control problem in high-concurrency scenarios in network communication. By determining the load pressure index in real time and dynamically determining the batch processing quantity for data parsing based on the real-time load pressure index, data parsing can adapt to changes in system load, making full use of load capacity while avoiding load pressure.

[0037] Specifically, by determining the dynamic load pressure index and introducing traffic acceleration to determine the queue level, this invention no longer relies on rigid batch processing thresholds. Instead, it dynamically fits the system load curve and the dynamic batch processing quantity through a continuous function. Whether it's a sudden surge in short messages or complex parsing with sustained high CPU consumption, the system can automatically find the optimal throughput balance point, avoiding the oscillations and avalanches found in traditional solutions. The dynamic aging priority mechanism fills the gaps in standard priority queues, ensuring millisecond-level responses for high-priority services such as alarms and control commands, while ensuring that low-priority services such as heartbeats and logs are not lost or experience connection timeouts under high loads, maintaining the topological stability of the distributed system. The multi-dimensional water level model has predictive capabilities, intervening before the buffer is truly full. Combined with fine-tuning of the TCP receive window, it achieves smooth traffic shaping, avoiding severe packet loss and retransmissions at the network layer.

[0038] Example 2 Figure 2 This is a flowchart of a batch data parsing method according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes: Step 210: Determine the load pressure index based on processor utilization, processing time, processing time limit, load weight, and timeliness weight.

[0039] Step 220: Determine the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer.

[0040] Optionally, the queue level is determined based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer. This includes: determining the static capacity weight and dynamic trend weight based on the business's sensitivity to sudden traffic surges and the business's expected throughput; and determining the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer.

[0041] Step 230: Adjust the memory capacity used for data caching in the data buffer area according to the queue level.

[0042] Optionally, the memory capacity used for data caching in the data buffer is adjusted according to the queue level, including: when the queue level is less than a first preset threshold, setting the memory capacity used for data caching in the data buffer to the upper limit of the buffer capacity; when the queue level is greater than or equal to the first preset threshold and less than a second preset threshold, setting the memory capacity used for data caching in the data buffer to the upper limit of the buffer capacity at a first preset ratio; when the queue level is greater than or equal to the second preset threshold and less than a third preset threshold, setting the memory capacity used for data caching in the data buffer to the upper limit of the buffer capacity at a second preset ratio, and performing a preset time pause on data access in the data buffer; when the queue level is greater than or equal to the third preset threshold, performing a pause on data access in the data buffer.

[0043] The third preset water level threshold is greater than the second preset water level threshold, which in turn is greater than the first preset water level threshold. For example, the first preset water level threshold is a value less than or equal to 0.5, such as 0.4. When the queue water level is less than the first preset water level threshold, the memory capacity used for data caching in the data buffer is set to the upper limit of the buffer capacity, allowing for full-speed data reading and avoiding data processing delays.

[0044] The second preset water level threshold can be a value between 0.6 and 0.8, such as 0.7. When the queue water level is greater than or equal to the first preset water level threshold but less than the second preset water level threshold (e.g., greater than or equal to 0.4 but less than 0.7), data processing experiences slight congestion. In this case, the memory capacity used for data caching in the data buffer can be set to the upper limit of the buffer capacity of the first preset ratio. The first preset ratio can be a value in the range of [0.4, 0.8], such as 0.5. By performing slight rate limiting during periods of slight congestion, upstream data reading can be restricted, avoiding pressure on the load.

[0045] The third preset water level threshold can be a value between 0.8 and 0.95, such as 0.9. When the queue water level is greater than or equal to the second preset water level threshold but less than the third preset water level threshold (e.g., greater than or equal to 0.7 but less than 0.9), data processing is severely congested. In this case, the memory capacity used for data caching in the data buffer can be set to the upper limit of the buffer capacity of the second preset ratio, and data access to the data buffer can be paused for a preset time. The second preset ratio can be the same as or different from the first preset ratio. For example, the second preset ratio can be a value in the range of [0.3, 0.6], such as 0.4. By performing strong flow limiting during severe congestion, upstream data reading can be restricted to avoid putting pressure on the load. At the same time, data access to the data buffer can be paused for a preset time, which reduces the TCP data receiving window by back-pressure on upstream reading, avoiding huge pressure on downstream processing and causing system crash. The preset time can be a value between 1ms and 20ms, such as 10ms.

[0046] When the queue level is greater than or equal to the third preset threshold (e.g., greater than or equal to 0.9), data processing is circuit-broken, and data access to the data buffer is suspended to protect the system. At this time, low-priority new connection requests can also be discarded.

[0047] By limiting the upstream data reception based on the queue level, system congestion can be alleviated and load pressure reduced.

[0048] Step 240: Determine the upper limit of the batch processing quantity based on memory capacity, data size, and preset cache coefficient.

[0049] Step 250: Determine the dynamic batch size based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size.

[0050] Step 260: Obtain the transmission data and cache it in the data cache area.

[0051] Optionally, acquiring and caching the transmitted data in the data cache includes: acquiring the static priority of the data in the transmitted data, and determining the dynamic priority of the data based on the static priority, the data arrival time, the current time, and the data aging step size; and caching the transmitted data in the data cache according to the dynamic priority of the data.

[0052] Step 270: When the amount of data cached in the data cache reaches the dynamic batch processing quantity, batch retrieve data from the data cache for data parsing.

[0053] Optionally, when the amount of data cached in the data cache reaches the dynamic batch processing limit, data in the data cache is retrieved in batches for data parsing, including: when the amount of data cached in the data cache reaches the dynamic batch processing limit, determining the data parsing strategy based on the queue level; and retrieving data in the data cache in batches for data parsing according to the data parsing strategy.

[0054] Data parsing strategies include: normal processing, thread scaling, service degradation, and data discarding. Different downstream data parsing strategies can be adapted to different queue levels to achieve congestion control during batch data parsing and maximize throughput within the limits of hardware resources. For example, when the queue level increases, the data parsing strategy can be tightened, i.e., the load pressure can be reduced by adjusting the data parsing strategy.

[0055] Optionally, the data parsing strategy is determined based on the queue level, including: when the queue level is less than a first preset threshold, the data parsing strategy is to perform data parsing through the basic parsing thread; when the queue level is greater than or equal to the first preset threshold and less than a second preset threshold, the data parsing strategy is to expand the basic parsing thread and perform data parsing through the expanded parsing thread; when the queue level is greater than or equal to the second preset threshold and less than a third preset threshold, the data parsing strategy is to expand the basic parsing thread and perform downgraded data parsing through the expanded parsing thread; when the queue level is greater than or equal to the third preset threshold, the data parsing strategy is to expand the basic parsing thread and perform downgraded data parsing for data with a preset priority through the expanded parsing thread.

[0056] For example, when the queue level is lower than a first preset threshold, a normal processing strategy is adopted, which involves parsing the data using the basic parsing thread. For instance, idle parsing threads are released, and the basic parsing thread performs normal data parsing. Normal parsing can involve parsing the complete validation protocol fields. For example, four parsing threads can be used to parse the complete validation protocol fields to save resources.

[0057] When the queue level is greater than or equal to the first preset threshold and less than the second preset threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing threads, and then use the expanded parsing threads to perform data parsing. Expansion can involve requesting more parsing threads, such as expanding from 4 to 8 parsing threads, to improve processing efficiency and alleviate queue congestion through concurrent parsing.

[0058] When the queue level is greater than or equal to the second preset threshold but less than the third preset threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing thread and then perform downgraded data parsing using the expanded parsing thread. Downgraded data parsing can be a fast parsing mode for the data, such as extracting only key tags and values, skipping data verification (such as CRC check) or log formatting processing, in order to improve processing efficiency and alleviate queue congestion.

[0059] When the queue level is greater than or equal to the third preset threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing thread, and then use the expanded parsing thread to perform downgraded data parsing on data with preset priority. The preset priority can be high priority (P1), medium priority (P2), or high priority, that is, to discard low priority (P3) data to alleviate queue congestion.

[0060] The technical solution of this invention utilizes a collaborative mechanism of "upstream rate limiting" and "downstream expansion or degradation." In the initial stages of congestion, it prioritizes absorbing the backlog by increasing downstream computing power (expanding threads) rather than blindly reducing upstream speed, thereby maximizing throughput within the limits of hardware resources. By constructing a bidirectional elastic scaling backpressure mechanism between upstream and downstream, it dynamically triggers the expansion of downstream parsing resources or the degradation of parsing strategies while limiting the upstream read rate, thus resolving the data batch parsing congestion problem.

[0061] Table 1 Table 1 is a processing table for a bidirectional elastic expansion backpressure mechanism according to Embodiment 2 of the present invention. As shown in Table 1, when the queue water level is different, upstream and downstream strategies can be adopted accordingly. The congestion problem in batch data parsing can be solved by the cooperation of upstream and downstream strategies.

[0062] For example, an application example of the bidirectional elastic scaling backpressure mechanism provided in this embodiment of the invention can be as follows: In the initial state: the queue level is normal, the CPU utilization rate is 20%, the average processing time is 5ms, and the load pressure index is determined to be 0.2, which is low; the dynamic batch processing quantity is determined to be 1000, the timeout time is 50ms, and the system performs data batch parsing processing once every 1000 data entries or after waiting for 50ms. Stage 1: Sudden traffic surge: For example, the client simultaneously sends a large amount of high-priority (P1) alarm data, the queue inbound rate is determined to be 20000 TPS, and the queue outbound rate is 5000 TPS. The queue is currently occupied at 30%, but due to the extremely high acceleration, the queue level soars to 0.8, reaching severe congestion. Stage 2: Cooperative control: The upstream triggers a feedback mechanism, the TCP receive coroutine is forcibly inserted to pause access processing at the microsecond level, and the memory capacity is reduced, the TCP receive window is shrunk, and the client's sending speed is physically slowed down to perform traffic shaping. Downstream parsing layer expands thread capacity (e.g., from 4 to 16) and switches to degraded data parsing, skipping non-critical field validation. At this point, CPU utilization increases, the load pressure index rises to 0.9, the dynamic batch processing quantity is reduced to 100 to reduce single-time blocking, and the timeout is reduced to 5ms for faster response. Phase Three: Anti-starvation aging: Due to the continuous flow of high-priority alarm data, a large amount of low-priority log data accumulates in the queue and cannot be dequeued. Through data aging processing, a low-priority log data is found to have waited for more than 300ms during scanning, determining its dynamic priority as medium, which may be updated to high priority later depending on real-time conditions. In the next batch parsing, the "aged" log data is extracted and parsed together with the newly arrived alarm data to avoid connection timeouts. Phase Four: Recovery: After alarm data processing is completed, the queue inbound rate decreases, the queue level recovers to 0.3, redundant threads are automatically recycled, and complete data parsing is adopted to remove upstream data reception rate limits, restoring to the initial high-efficiency state.

[0063] The technical solution of this invention solves the flow control problem in high-concurrency scenarios of network communication by determining the load pressure index based on processor utilization, processing time, upper limit of processing time, load weight, and timeliness weight; determining the queue level based on the current queue length, upper limit of queue capacity, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer; adjusting the memory capacity used for data caching in the data buffer based on the queue level; determining the upper limit of batch processing based on memory capacity, data size, and preset caching coefficient; determining the dynamic batch processing number based on the load pressure index, upper limit of batch processing number, and lower limit of batch processing number; acquiring and caching the transmitted data in the data buffer; and when the data cached amount in the data buffer reaches the dynamic batch processing number, batch acquiring data in the data buffer for data parsing. By determining the load pressure index in real time and dynamically determining the batch processing number for data parsing based on the real-time load pressure index, data parsing can adapt to changes in system load, making full use of load capacity while avoiding load pressure.

[0064] Specifically, by combining a multi-dimensional water level prediction model of "instantaneous capacity" and "traffic acceleration," backpressure is triggered in advance before the queue is full, eliminating control lag and avoiding severe packet loss and retransmission at the network layer. A bidirectional elastic scaling backpressure mechanism is constructed to dynamically trigger the expansion of downstream parsing resources or the degradation of parsing strategies while limiting the upstream read rate. By establishing a load pressure coefficient model, a continuous function mapping between batch threshold and timeout time is achieved, covering all load conditions. Whether it is a sudden short message surge or a complex parsing with sustained high CPU consumption, the system can automatically find the optimal throughput balance point, avoiding the oscillation and avalanche problems found in traditional solutions. A dynamic aging algorithm is introduced, which automatically increases the weight of low-priority messages as the waiting time increases, ensuring that all messages are eventually processed. This guarantees millisecond-level response for high-priority services such as alarms and control commands, while ensuring that low-priority services such as heartbeats and logs are not lost or experience connection timeouts under high load, maintaining the topological stability of the distributed system. By using a zero-copy circular buffer and an elastic asynchronous task pool, we ensure that I / O reception is uninterrupted under high load, thus achieving complete decoupling between parsing and I / O.

[0065] Figure 3 This is a schematic diagram illustrating the application process of a batch data parsing method provided in Embodiment 2 of the present invention. Figure 3As shown, the TCP receive coroutine can receive alarm information and log data from multiple clients. The load pressure index can be determined in real time based on the resource status during batch data parsing. The dynamic batch processing quantity can be determined based on the load pressure index. When the dynamic batch processing quantity is reached, data can be retrieved from the queue in the data buffer for data parsing. The transmitted data received by the TCP receive coroutine can be recorded using priority marking and timestamps, updating the dynamic priority of the data. The queue level can be updated based on the transmitted data received by the TCP receive coroutine and the batch data parsing status. The queue level enables upstream data reception feedback control and downstream data batch parsing control. When retrieving data for data parsing, processing can be performed according to the dynamic data priority based on the queue level.

[0066] Example 3 Figure 4 This is a schematic diagram of a data batch parsing device provided according to Embodiment 3 of the present invention. Figure 4 As shown, the device includes: a load pressure index determination module 410, a dynamic batch processing quantity determination module 420, a data caching module 430, and a data parsing module 440. Wherein: The load pressure index determination module 410 is used to determine the load pressure index based on processor utilization, processing time, processing time limit, load weight, and timeliness weight. The dynamic batch processing quantity determination module 420 is used to determine the dynamic batch processing quantity based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity. Data caching module 430 is used to acquire transmitted data and cache the transmitted data in the data cache area; The data parsing module 440 is used to batch retrieve data from the data cache area and perform data parsing when the data cache volume in the data cache area reaches the dynamic batch processing quantity.

[0067] Optional, the dynamic batch size determination module 420 includes: The queue level determination unit is used to determine the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer. The memory capacity adjustment unit is used to adjust the memory capacity used for data caching in the data buffer area according to the queue level. The batch processing quantity limit determination unit is used to determine the upper limit of the batch processing quantity based on memory capacity, data size, and preset cache coefficient; The dynamic batch processing quantity determination unit is used to determine the dynamic batch processing quantity based on the load pressure index, the upper limit of the batch processing quantity, and the lower limit of the batch processing quantity.

[0068] Optional, the memory capacity adjustment unit includes: The first memory capacity adjustment subunit is used to set the memory capacity used for data caching in the data cache area to the upper limit of the cache area capacity when the queue water level is less than the first preset water level threshold. The second memory capacity adjustment subunit is used to set the memory capacity used for data caching in the data cache area to the upper limit of the cache area capacity of the first preset ratio when the queue water level is greater than or equal to the first preset water level threshold and less than the second preset water level threshold. The third memory capacity adjustment subunit is used to set the memory capacity used for data caching in the data buffer to the upper limit of the buffer capacity of the second preset ratio when the queue water level is greater than or equal to the second preset water level threshold and less than the third preset water level threshold, and to perform a preset time pause data access processing on the data buffer. The fourth memory capacity adjustment subunit is used to pause data access to the data buffer when the queue level is greater than or equal to the third preset threshold level.

[0069] Optionally, the data parsing module 440 includes: The data parsing strategy determination unit is used to determine the data parsing strategy based on the queue level when the data cache size in the data cache area reaches the dynamic batch processing quantity. The data parsing unit is used to retrieve data from the data cache in batches and parse it according to the data parsing strategy.

[0070] Optionally, the data parsing strategy determination unit includes: The first data parsing strategy determination sub-unit is used to determine the data parsing strategy as data parsing through the basic parsing thread when the queue level is less than the first preset water level threshold. The second data parsing strategy determination subunit is used to determine the data parsing strategy as expanding the basic parsing thread when the queue level is greater than or equal to the first preset water level threshold and less than the second preset water level threshold, and then performing data parsing through the expanded parsing thread. The third data parsing strategy determination subunit is used to determine the data parsing strategy as expanding the basic parsing thread when the queue level is greater than or equal to the second preset water level threshold and less than the third preset water level threshold, and then performing downgraded data parsing through the expanded parsing thread. The fourth data parsing strategy determination subunit is used to determine the data parsing strategy as expanding the basic parsing thread when the queue level is greater than or equal to the third preset water level threshold, and then using the expanded parsing thread to perform downgraded data parsing on data with preset priority.

[0071] Optional, the data caching module 430 includes: The data dynamic priority determination unit is used to obtain the static priority of the data in the transmitted data, and determine the dynamic priority of the data based on the static priority of the data, the data arrival time, the current time, and the data aging step size; The data caching unit is used to cache transmitted data into the data cache area according to the dynamic data priority.

[0072] Optionally, the queue water level determination unit includes: The weight determination subunit is used to determine the static capacity weight and dynamic trend weight based on the service's sensitivity to burst traffic and the service's expected throughput. The queue level determination subunit is used to determine the queue level based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer.

[0073] The data batch parsing device provided in the embodiments of the present invention can execute the data batch parsing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0074] Example 4 Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0075] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) or random access memory (RAM), communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. Input / output (I / O) interfaces are also connected to the bus 14.

[0076] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0077] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data batch parsing methods.

[0078] In some embodiments, the data bulk parsing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data bulk parsing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data bulk parsing method by any other suitable means (e.g., by means of firmware).

[0079] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0080] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0083] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0084] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0085] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0086] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for batch data parsing, characterized in that, include: The load pressure index is determined based on processor utilization, processing time, processing time limit, load weight, and timeliness weight. The dynamic batch size is determined based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size. Acquire the transmission data and cache the transmission data in the data cache area; When the amount of data cached in the data cache reaches the dynamic batch processing quantity, data in the data cache is retrieved in batches for data parsing.

2. The method according to claim 1, characterized in that, The dynamic batch size is determined based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size, including: The queue level is determined based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer. Adjust the memory capacity used for data caching in the data buffer area according to the queue level; The maximum number of batch processes is determined based on the memory capacity, data size, and preset cache coefficient. The dynamic batch size is determined based on the load pressure index, the upper limit of the batch size, and the lower limit of the batch size.

3. The method according to claim 2, characterized in that, Adjusting the memory capacity used for data caching in the data buffer area according to the queue level includes: When the queue level is less than the first preset water level threshold, the memory capacity used for data caching in the data cache area is set as the upper limit of the cache area capacity. When the queue level is greater than or equal to the first preset water level threshold and less than the second preset water level threshold, the memory capacity used for data caching in the data cache area is set to the upper limit of the cache area capacity of the first preset ratio. When the queue level is greater than or equal to the second preset water level threshold and less than the third preset water level threshold, the memory capacity used for data caching in the data cache is set to the upper limit of the cache capacity of the second preset ratio, and the data cache is subjected to a data access pause for a preset time. When the queue level is greater than or equal to the third preset water level threshold, data access to the data buffer is paused.

4. The method according to claim 2, characterized in that, When the amount of data cached in the data cache reaches the dynamic batch processing quantity, data in the data cache is retrieved in batches for data parsing, including: When the amount of data cached in the data cache area reaches the dynamic batch processing quantity, the data parsing strategy is determined according to the queue level. According to the data parsing strategy, data in the data cache is retrieved in batches for data parsing.

5. The method according to claim 4, characterized in that, The data parsing strategy is determined based on the queue level, including: When the queue level is lower than the first preset water level threshold, the data parsing strategy is determined to be data parsing through the basic parsing thread; When the queue level is greater than or equal to the first preset water level threshold and less than the second preset water level threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing thread and perform data parsing through the expanded parsing thread. When the queue level is greater than or equal to the second preset water level threshold and less than the third preset water level threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing thread and perform downgraded data parsing through the expanded parsing thread. When the queue level is greater than or equal to the third preset water level threshold, the data parsing strategy is determined to be to expand the capacity of the basic parsing thread, and to perform downgraded data parsing on the data with the preset priority through the expanded parsing thread.

6. The method according to claim 1, characterized in that, Acquiring transmission data and caching the transmission data in a data buffer includes: Obtain the static priority of the data in the transmitted data, and determine the dynamic priority of the data based on the static priority, data arrival time, current time, and data aging step size; The transmitted data is cached in the data cache area according to the data dynamic priority.

7. The method according to claim 2, characterized in that, The queue level is determined based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer, including: Based on the business's sensitivity to sudden traffic surges and the expected throughput, determine the static capacity weight and the dynamic trend weight. The queue level is determined based on the current queue length, queue capacity limit, queue enqueue rate, queue dequeue rate, static capacity weight, and dynamic trend weight of the data buffer.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data batch parsing method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data batch parsing method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data batch parsing method according to any one of claims 1-7.

Images