Data adaptive processing method and system, electronic device and storage medium

Abstract

Embodiments of the present application disclose a data adaptive processing method and system, electronic equipment and storage medium, relating to the technical field of data processing. The data adaptive processing method comprises: storing to-be-processed data into a cache space; when a continuous duration reaches a timing trigger period, packaging the to-be-processed data into a data batch; distributing the data batch to a queue; calling a data processing resource to process the data batch; and adjusting the timing trigger period and the calling amount of the data processing resource according to the queue pressure of the queue. The timing trigger period and the calling amount of the data processing resource are adjusted in coordination, the fluctuation of the to-be-processed data is adaptively adjusted, the queue pressure is reduced, and the data processing effect is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data adaptive processing method, system, electronic device, and storage medium. Background Technology

[0002] Data is information typically generated by front-end devices and processed by back-end devices. It is ubiquitous in real life. For example, in a monitoring scenario, sensors at the front end transmit the detected information to the back-end server. After receiving the data, the server analyzes it to determine the real-time situation in the scene where the sensor is located. Another example is in an aircraft swarm scenario, where a large number of aircraft feed back the collected or generated data to the central processing unit. After analyzing the data from all the aircraft, the central processing unit determines the real-time situation and then further controls each aircraft.

[0003] For backend devices, the amount of data transmitted by frontend devices is unstable. The amount of data may be extremely low at certain times and extremely high at others. This instability poses a significant challenge to the data processing capabilities of backend devices. Taking an aircraft swarm as an example, multiple aircraft transmit data to the central processor in real time during the same period. Initially, because no targets are detected or the aircraft need to perform few actions, the amount of data transmitted is small. However, in the later stages, due to the appearance of targets or increased actions by the aircraft, the amount of data transmitted increases continuously, resulting in significant fluctuations in the amount of data received by the central processor.

[0004] To enable backend devices to adapt to scenarios with large fluctuations in data volume, the current approach is to set a time window. The backend devices receive data within this window, and then process the received data in batches after the window ends. While this method can address the issue of data volume fluctuations to some extent, when the data volume is small, the need to wait for the time window to close before processing leads to unused data processing resources on the backend devices. Conversely, when the data volume is large, the large volume of data within a single batch can overburden the backend devices' data processing resources. In short, the current data processing method has poor data processing efficiency. Summary of the Invention

[0005] The purpose of this invention is to propose a data adaptive processing method, system, electronic device, and storage medium, which aims to solve the problem of poor data processing effect of existing data processing methods.

[0006] A data adaptive processing method, applied to a data adaptive processing system, the method comprising: The acquired data to be processed is stored in the cache space of the adaptive data processing system; According to the preset timing trigger mechanism, when the duration for which the data to be processed is stored in the cache space reaches the timing trigger cycle, the data to be processed in the cache space is packaged into a data batch. The data is distributed in batches to the queue; The data processing resources of the adaptive data processing system are invoked to process the data batches in the queue; The timing triggering period is adjusted according to the queue pressure, and the amount of data processing resources called is adjusted according to the queue pressure, so as to control the queue pressure within a preset range. The queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches.

[0007] Optionally, adjusting the timing triggering period according to the queue pressure of the queue includes: Using the time series of queue pressure and a preset smoothing coefficient, the timing triggering period at time t is calculated.

[0008] Optionally, the queue pressure includes an occupancy ratio, wherein the occupancy ratio represents the proportion of the total number of data batches in the queue to the total capacity of the queue; The step of calculating the timing trigger period at time t using the time series of queue pressure and a preset smoothing coefficient includes: Based on the preset maximum timing window, the preset minimum timing window, and the occupancy ratio, determine the reference value for the timing trigger cycle at time t; Calculate the product of the smoothing coefficient and the timing trigger period at time t-1 to obtain a first value, wherein the smoothing coefficient is between 0 and 1; The smoothing margin of the smoothing coefficient is calculated by multiplying it by the reference value of the timing trigger cycle to obtain a second value, wherein the smoothing margin = 1 - smoothing coefficient; Summing the first value and the second value yields the timing trigger period at time t.

[0009] Optionally, the data adaptive processing system includes multiple triggering mechanisms, each of which has a corresponding triggering condition. The triggering mechanism includes the timing triggering mechanism, and the triggering condition of the timing triggering mechanism is whether the duration reaches the timing triggering cycle. After storing the acquired data to be processed into the cache space of the data adaptive processing system, the method further includes: When the data to be processed in the cache space reaches the triggering condition of any of the triggering mechanisms, the data to be processed in the cache space is packaged to form a data batch.

[0010] Optionally, the triggering mechanism includes a capacity triggering mechanism, wherein the triggering condition of the capacity triggering mechanism is whether the amount of data cached in the cache space reaches a preset cache amount threshold. After storing the acquired data to be processed into the cache space of the data adaptive processing system, the method further includes: According to the capacity triggering mechanism, when the amount of data to be processed stored in the cache space reaches the cache amount threshold, the data to be processed in the cache space is packaged into a data batch.

[0011] Optionally, adjusting the amount of data processing resource calls based on the queue pressure includes: Based on the difference between the queue pressure and the preset target pressure, the call volume is calculated using a feedback control algorithm, wherein the larger the call volume, the more data processing threads are in the data adaptive processing system.

[0012] Optionally, after distributing the data batch to the queue, the method further includes: At least one batch of data from the queue is pre-extracted into the local memory of the data processing resource based on the queue pressure, wherein the greater the queue pressure, the more batches of data are pre-extracted.

[0013] This application also provides a data adaptive processing system, including a producer module and a consumer module, wherein the producer module has a cache space and the consumer module has local memory; The producer module is used to store the acquired data to be processed into the cache space; The producer module is also used to package the data to be processed in the cache space into a data batch when the duration of the data to be processed stored in the cache space reaches the timer triggering period, according to a preset timer triggering mechanism. The producer module is also used to distribute the data batches to the queue; The consumer module is used to invoke data processing resources to process the data batches in the queue; The producer module is further configured to adjust the timing triggering period according to the queue pressure of the queue, and the consumer module is further configured to adjust the amount of data processing resources called according to the queue pressure, so as to control the queue pressure within a preset range, wherein the queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches.

[0014] This application also provides an electronic device, which includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the steps of the method described above.

[0015] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the method described above.

[0016] The embodiments of the present invention have the following beneficial effects: After receiving the data to be processed, it is first packaged into a data batch according to the timed trigger cycle. Then, data processing resources are invoked to retrieve the data batch from the queue for processing. During this process, the timed trigger cycle is adjusted according to the queue pressure, thereby reducing the queue pressure and achieving adaptive adjustment of the timed trigger cycle. When the timed trigger cycle is adjusted, the waiting time for the data to be processed in the cache space to be packaged into a data batch will change. This helps to reduce the waiting time for data batch packaging when there is little data to be processed, ensuring that data processing resources are used promptly and fully. Conversely, when there is a lot of data to be processed, the waiting time for data batch packaging will be extended, allowing data processing resources sufficient time to process the data batches in the queue and ensuring processing effectiveness.

[0017] At the same time, as the timing trigger cycle is adjusted, the amount of data processing resources called will also be adjusted accordingly. The coordinated adjustment of the two allows the data processing resources to reduce the amount of calls when there is less data to be processed, avoiding unnecessary resource occupation and waste. When there is more data to be processed, the amount of calls will be increased to improve the data processing throughput and reduce data congestion.

[0018] Therefore, the data adaptive processing method provided in this application can adapt to the fluctuation of the data to be processed by coordinating the timing trigger cycle and the amount of data processing resource calls, thereby reducing queue pressure and improving data processing effect. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] in: Figure 1 This is a flowchart of a data adaptive processing method in one embodiment; Figure 2 A flowchart illustrating a data adaptive processing method with multiple triggering mechanisms in one embodiment; Figure 3 This is a block diagram of a data adaptive processing system in one embodiment. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] First, the relevant terms in the embodiments of this application will be explained as follows: Time window: refers to the length of time. In the data processing scenario of aircraft swarms, the time window specifically refers to the waiting time for batch packaging of data transmitted by aircraft. The larger the time window, the longer the waiting time; conversely, the smaller the time window, the shorter the waiting time. For example, if the time window is 10 seconds, the data will be batch packaged after waiting for 10 seconds.

[0023] Producer: A module used to acquire and / or store data.

[0024] Consumer: The module used to process data.

[0025] In the data processing scenario of aircraft swarms, the current data processing of aircraft has the following shortcomings: 1. Static configuration problem: Traditional batch aggregation methods usually use a fixed time window, which cannot be dynamically adjusted according to the load of the aircraft cluster, resulting in performance degradation during periods of intensive task execution; 2. Low resource utilization: The number of consumers is fixed and cannot be dynamically adjusted according to the number of aircraft clusters. This can easily cause processing bottlenecks when large-scale aircraft are connected, and waste resources during idle periods. 3. I / O wait overhead: Consumers experience blocking and waiting when retrieving data from the queue, resulting in poor data processing performance; 4. Queue overflow risk: When a cluster of aircraft reports data simultaneously, the queue may accumulate rapidly, causing overflow and resulting in the loss of critical flight control data.

[0026] To overcome the above-mentioned deficiencies, this application provides a data adaptive processing method applied to a data adaptive processing system. The data adaptive processing system can be any system with data receiving and processing functions, specifically a computer, controller, or server, or other intelligent device. For ease of understanding, the data adaptive processing system is taken as the central processing unit of an aircraft cluster as an example. Figure 1 As shown, the adaptive data processing method includes the following:

[0027] S100. Store the acquired data to be processed into the cache space of the data adaptive processing system.

[0028] During the operation of an aircraft swarm, multiple aircraft will feed back their collected data (such as ambient temperature or captured images) and their own operational data (such as flight speed and altitude) to the central processing unit in real time or periodically. The central processing unit is connected to each aircraft to receive and process the data fed back by each aircraft. The purpose of processing may be to control the flight trajectory of the aircraft based on the image data collected by the aircraft itself, or to determine the relative positional relationship between the aircraft based on the aircraft's own operational data, thereby adjusting the position of the aircraft to avoid collisions or to make the aircraft swarm form a specific attitude (such as forming a digital shape).

[0029] The data fed back from the aircraft to the central processing unit is the data to be processed in this embodiment. The communication between the aircraft and the central processing unit can be implemented according to existing communication protocols, and this embodiment does not make specific limitations on this.

[0030] Once the central processing unit (CPU) receives the data to be processed from the aircraft, it stores the data in its cache. In one embodiment, the data stored in the cache can be sequential, meaning it is ordered according to the order in which the CPU receives the data.

[0031] S102. According to the preset timing trigger mechanism, when the duration for which the data to be processed is stored in the cache space reaches the timing trigger period, the data to be processed in the cache space is packaged into a data batch.

[0032] The central processing unit (CPU) has a preset timing trigger mechanism. Specifically, this mechanism controls the time window for packaging data to be processed in the cache into a data batch. In other words, the CPU continuously receives data to be processed from the aircraft and stores it in the cache. Simultaneously, the CPU runs the timing trigger mechanism so that when the duration of receiving data reaches the timing trigger period, the data stored in the cache during that duration is packaged into a data batch. The so-called time window is the timing trigger period in this embodiment.

[0033] To facilitate understanding, in one application scenario, the central processing unit (CPU) has an adaptive aggregator. The adaptive aggregator maintains a configurable timer. Taking a 3-second timing trigger period as an example, when the CPU starts up and begins receiving data to be processed, the timer starts. When the timer reaches 3 seconds, the data to be processed stored in the cache during these 3 seconds is packaged into a data batch. At the same time, the timer is reset and starts timing again. When the timer reaches 3 seconds again, the data to be processed stored in the cache during these 3 seconds is packaged into the next data batch, and so on.

[0034] S104. Distribute the data batches to the queue.

[0035] The central processing unit has pre-set queues, such as blocking queues and message queues. Taking message queues as an example, you can use Kafka, RocketMQ, RabbitMQ, etc. The specific choice can be made according to the actual application scenario and data processing requirements. This embodiment does not make any specific limitations.

[0036] After the central processing unit packages the data to be processed into a data batch, it distributes the data batch into a queue. The data batches are arranged in a certain order in the queue, such as according to the order in which the data batches were distributed into the queue, or according to the size of the data batches.

[0037] S106. Call the data processing resources of the adaptive data processing system to process the data batches in the queue.

[0038] The central processing unit (CPU) calls upon data processing resources to process the data to be processed in the data batch. In other words, the CPU will take the data batch from the queue and process it using the corresponding data processing resources.

[0039] Among them, data processing resources, such as CPU resources, are specifically represented by the number of threads used to process the data to be processed. If there are 2 threads, the data to be processed in 2 batches can be processed simultaneously. If there are 10 threads, the data to be processed in 10 batches can be processed simultaneously.

[0040] It should be noted that processing data batches takes time, and the duration depends on the CPU's data processing capabilities as well as the size of the data to be processed in each batch. In other words, if a data batch contains a large amount of data to be processed, the thread will be occupied for a longer period of time. If each data batch contains a large amount of data to be processed, all threads will be occupied, and no data batches will be retrieved from the queue in a short period of time, but new data batches will still be distributed to the queue, causing the number of data batches in the queue to increase continuously, increasing the queue pressure, and the queue will tend to saturate or even become unable to receive new data batches, resulting in the loss of data batches.

[0041] S108. Adjust the timing trigger cycle according to the queue pressure, and adjust the amount of data processing resources called according to the queue pressure, so as to control the queue pressure within a preset range.

[0042] To adaptively adjust queue pressure, the CPU monitors queue pressure in real time or periodically and adjusts the timing trigger period and the amount of data processing resources allocated accordingly. In other words, when the CPU detects high queue pressure, it increases the timing trigger period, making it take longer to form a new data batch, thus reducing the frequency of new data batches entering the queue. Simultaneously, it increases the allocation of data processing resources, allowing the CPU to process more data batches simultaneously, thereby increasing the frequency of data batches being retrieved from the queue. This results in a situation where the frequency of data batches entering the queue is low, and the frequency of data batches being retrieved is high, thus reducing queue pressure.

[0043] Similarly, when the CPU detects that the queue pressure is low, it will reduce the timing trigger cycle, which increases the frequency of new data batches entering the queue and reduces the amount of data processing resources called, thus reducing the frequency of data batches being retrieved from the queue. This results in a phenomenon where the frequency of data batches entering the queue is high and the frequency of data batches being retrieved is low, increasing the queue pressure and ensuring that the data to be processed can be processed in a timely manner, while shortening the idle time of the CPU's data processing resources.

[0044] Queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches. Specifically, the number of data batches can be determined by the number of data batches in the queue. For example, if the queue can hold a maximum of 100 data batches and there are currently 80 data batches, the queue pressure can be determined to be 0.8, indicating relatively high queue pressure. In addition, the number of data batches can be determined by the length of the data batches in the queue. For example, if the space used to store data batches in the queue is 100, and the total length of the currently stored data batches has reached 80, the queue pressure can be determined to be 0.8, indicating relatively high queue pressure.

[0045] Specifically, the rate of change of data batches can be determined by the rate of change of the number of data batches in the queue or the rate of change of the length of data batches in the queue. The rate of change of the number is the amount of change in the number of data batches in the queue per unit time, and the rate of change of the length is the amount of change in the length of data batches in the queue per unit time.

[0046] It should be noted that the preset range can be set manually. For example, if the preset range is 0-100, then the queue pressure range is also 0-100. A preset range of 20-80 means that when the queue pressure is less than 20 or greater than 80, the timing trigger period and call volume will be adjusted. It should also be noted that queue pressure can be calculated in real-time or periodically, and the adjustment amounts for the timing trigger period and call volume can be calculated based on the queue pressure. Adjustments are made only when the adjustment amount for the timing trigger period meets the preset adjustment range or the adjustment amount for the call volume meets the preset requirements. In other words, keeping queue pressure within the preset range can be triggered by whether the queue pressure is within the preset range, or by the adjustment amount for the timing trigger period and / or the adjustment amount for the call volume, with the ultimate goal of keeping queue pressure within the preset range.

[0047] Based on the above, after obtaining the data to be processed, it is first packaged into a data batch according to the timed trigger cycle. Then, data processing resources are invoked to retrieve the data batch from the queue for processing. During this process, the timed trigger cycle is adjusted according to the queue pressure, thereby reducing the queue pressure and achieving adaptive adjustment of the timed trigger cycle. When the timed trigger cycle is adjusted, the waiting time for the data to be processed in the cache space to be packaged into a data batch will change. This helps to reduce the waiting time for data batch packaging when there is little data to be processed, ensuring that data processing resources are used promptly and fully. Conversely, when there is a lot of data to be processed, the waiting time for data batch packaging is extended, allowing data processing resources sufficient time to process the data batches in the queue and ensuring processing effectiveness.

[0048] At the same time, as the timing trigger cycle is adjusted, the amount of data processing resources called will also be adjusted accordingly. The coordinated adjustment of the two allows the data processing resources to reduce the amount of calls when there is less data to be processed, avoiding unnecessary resource occupation and waste. When there is more data to be processed, the amount of calls will be increased to improve the data processing throughput and reduce data congestion.

[0049] Therefore, the data adaptive processing method provided in this application can adapt to the fluctuation of the data to be processed by coordinating the timing trigger cycle and the amount of data processing resource calls, thereby reducing queue pressure and improving data processing effect.

[0050] In another embodiment of this application, adjusting the timing triggering period according to the queue pressure includes: By utilizing the time series of queue pressure and combining it with a preset smoothing coefficient, the timing triggering cycle at time t is calculated.

[0051] In one embodiment, queue pressure is calculated in real time or periodically to obtain multiple queue pressures. The queue pressures at different times constitute a time series. The timing triggering period at time t is calculated using the time series of queue pressures and a preset smoothing coefficient.

[0052] Here, time t can be the current time or any time when the timing trigger cycle is to be calculated, and the time series of queue pressure includes time t, time t-1 and time t+1; the smoothing coefficient ranges from 0 to 1.

[0053] As described above, the use of a time series in calculating the timing trigger period facilitates the combination of the calculated timing trigger period at time t with the timing trigger periods before or after time t in the time series, thus improving the accuracy of the calculated timing trigger period at time t. Furthermore, the inclusion of a smoothing coefficient enhances the smoothness of the timing trigger period at time t.

[0054] In another embodiment of this application, queue pressure includes occupancy ratio, wherein the occupancy ratio characterizes the proportion of the total number of data batches in the queue to the total capacity of the queue; Using the time series of queue pressure and a preset smoothing coefficient, the timing triggering period at time t is calculated, including: Based on the preset maximum timing window, the preset minimum timing window, and the occupancy ratio, determine the reference value for the timing trigger cycle at time t; Calculate the product of the smoothing coefficient and the timing trigger period at time t-1 to obtain the first value, where the smoothing coefficient is between 0 and 1; The second value is obtained by multiplying the smoothing margin of the smoothing coefficient by the reference value of the timing trigger cycle, where the smoothing margin = 1 - smoothing coefficient; Summing the first and second values ​​yields the timing trigger period at time t.

[0055] In one embodiment, the occupancy ratio can be calculated based on the queue length of the data batch in the queue. For example, β = L1 / L0, where β is the occupancy ratio, L1 is the queue length of the data batch in the queue at time t, and L0 is the maximum queue length of the queue at time t.

[0056] In one embodiment, the timing trigger period at time t and the reference value of the timing trigger period are calculated using the following formula: ; ; Where θt is the reference value of the timing trigger cycle at time t, min_window is the minimum timing window, β is the occupancy ratio, max_window is the maximum timing window, Vt is the timing trigger cycle at time t, a is the smoothing coefficient, V(t-1) is the timing trigger cycle at time t-1, and (1-a) is the smoothing margin.

[0057] The maximum and minimum timing windows are preset boundary values ​​for the timing trigger period. In other words, the timing trigger period is between the maximum and minimum timing windows.

[0058] In one embodiment, after calculating the timing trigger period, it is necessary to determine whether the adjustment conditions are met. Specifically, the adjustment conditions include: When the absolute value of the difference between the timing trigger period calculated at time t and the timing trigger period at time t-1 is greater than the minimum timing window, the timing trigger period is adjusted.

[0059] The above content helps to improve the accuracy of the timing trigger cycle.

[0060] In another embodiment of this application, the data adaptive processing system includes multiple triggering mechanisms, each of which has a corresponding triggering condition. The triggering mechanism includes a timed triggering mechanism, and the triggering condition of the timed triggering mechanism is whether the duration reaches the timed triggering cycle. After storing the acquired data to be processed into the cache space of the adaptive data processing system, the method further includes: When the data to be processed in the cache space reaches the trigger condition of any triggering mechanism, the data to be processed in the cache space is packaged into a data batch.

[0061] In other words, the central processing unit (CPU) not only has a timing trigger mechanism, but also other trigger mechanisms, such as capacity or flags. Taking flags as an example, if a flag trigger mechanism is preset, when data to be processed with a preset flag is stored in the cache space, the data to be processed in the cache space is packaged into a data batch.

[0062] Multiple triggering mechanisms can be active simultaneously. Packaging will occur as long as the triggering condition of any one of the triggering mechanisms is met.

[0063] Through the above, and through multiple triggering mechanisms, it is beneficial to adaptively optimize the size of data batches and the frequency of data being sent to the queue based on the operating status of the data adaptive system, thereby balancing throughput and latency performance.

[0064] In another embodiment of this application, the triggering mechanism includes a capacity triggering mechanism, the triggering condition of which is whether the amount of data cached in the cache space reaches a preset cache amount threshold. After storing the acquired data to be processed into the cache space of the adaptive data processing system, the method further includes: According to the capacity triggering mechanism, when the amount of data to be processed stored in the cache space reaches the cache size threshold, the data to be processed in the cache space is packaged into a data batch.

[0065] In one embodiment, such as Figure 2 As shown, this embodiment includes two triggering mechanisms: a capacity triggering mechanism and a timer triggering mechanism. After the data to be processed is input, it is written into the cache space. It is then determined whether the condition for sending the data to be processed to the queue is triggered. The timer triggering mechanism triggers a time condition, i.e., whether the duration reaches the timer trigger cycle. The capacity triggering mechanism triggers a capacity condition, i.e., whether the data cache size reaches the cache size threshold. If either condition is met, the data to be processed in the current buffer space is packaged into a data batch, which is then sent to the downstream distribution queue. The buffer space is then cleared or reclaimed, and the system waits for subsequent data to be processed to be written.

[0066] By introducing both timing-triggered and capacity-triggered mechanisms, we can ensure the real-time processing of the data to be processed and improve the throughput of the data adaptive system.

[0067] In another embodiment of this application, adjusting the amount of data processing resources called based on queue pressure includes: Based on the difference between queue pressure and preset target pressure, a feedback control algorithm is used to calculate the call volume. The larger the call volume, the more data processing threads are in the data adaptive processing system.

[0068] In one embodiment, the call quantity is calculated using a feedback control algorithm using the following formula: ; : Number of calls; Proportional gain; Deviation, i.e., the degree of difference; Immediately respond to the current error and quickly eliminate large deviations; Amplifier: Eliminates historical accumulated errors and overcomes steady-state errors caused by friction. Suppressor: Predicts future error trends and suppresses overshoot.

[0069] In this embodiment, to clearly illustrate the deviation amount The determination method first involves quantifying queue pressure and target pressure. As mentioned above, queue pressure can be expressed as a percentage of occupancy. Representation, i.e. The preset target pressure can also be set as a target occupancy ratio. The specific value of this characterization can be determined based on the actual application scenario. For example, setting it to 0.5 indicates that the queue is expected to maintain a 50% occupancy level. Based on this, the deviation... The calculation formula is: in, This represents the queue occupancy rate monitored at time t. The sign of this deviation has a clear control meaning: when When the target occupancy rate is greater than the actual occupancy rate, it indicates that the current queue pressure is lower than the expected level, and the system has surplus processing capacity. At this time, the PID controller outputs the call volume. This will be increased to increase the number of data processing threads; when When the target occupancy rate is less than the actual occupancy rate, it indicates that the current queue backlog exceeds the expected level, and the system is in an overload state. At this time, the PID controller outputs the call volume. This will be reduced to decrease the number of data processing threads; when This indicates that the queue pressure is exactly at the target level, and the number of calls remains unchanged.

[0070] To facilitate understanding, the following explanation is provided in conjunction with specific numerical values: Assuming a preset target occupancy rate =0.5. At time t, the queue occupancy rate was monitored. Then the deviation This negative deviation indicates that the queue pressure is too high and the system is in an overload state. Based on this deviation, the PID controller will combine the effects of the proportional, integral, and derivative terms to output a control command to reduce the number of calls, thereby reducing the number of data processing threads. This, along with the adjustment of the timing trigger cycle on the production side, will work together to bring the queue pressure back to the target level.

[0071] The above-mentioned method of defining the deviation gives the input of the PID controller a clear sign and physical meaning, ensuring the correctness of the feedback control loop in regulating the queue pressure.

[0072] In one embodiment, the following adjustment rules are provided: 1. Call volume constraint: The absolute value of each call volume shall not exceed the preset upper limit; 2. Thread pool boundaries: The core size of the thread pool is not less than the minimum value, and the maximum size does not exceed a preset multiple of the number of CPU cores; 3. Adjustment frequency: The feedback control algorithm is calculated every 50ms; 4. Shrinkage protection: Shrinkage operation is only allowed when the thread's idle time exceeds a preset threshold.

[0073] Based on the above, feedback control algorithms can help improve the accuracy of call volume.

[0074] In another embodiment of this application, after distributing data batches to the queue, the method further includes: Based on queue pressure, at least one batch of data in the queue is pre-fetched into the local memory of the data processing resource. The greater the queue pressure, the more batches of data are pre-fetched.

[0075] In one embodiment, the number of data batches prefetched into the local memory of the data processing resources varies depending on the queue pressure. Specifically, the queue pressure is proportional to the number of prefetched data batches.

[0076] To make it easier to understand, let's take queue pressure as an example. When the occupancy rate is higher than 70%, the number of data batches to be pre-extracted is twice the number of current threads. When the occupancy rate is between 30% and 70%, the number of data batches to be pre-extracted is once the number of current threads. When the occupancy rate is lower than 30%, the number of data batches to be pre-extracted is once.

[0077] Based on the above, by pre-fetching, we can ensure that threads have enough data batches to process when the queue pressure is high, reducing queue blocking; ensure processing continuity when the queue pressure is normal; and avoid wasting local memory by pre-fetching too much data when the queue pressure is low.

[0078] This application embodiment also provides a data adaptive processing system, including a producer module and a consumer module, wherein the producer module has a cache space and the consumer module has local memory; The producer module is used to store the acquired data to be processed into a cache space; The producer module is also used to package the data to be processed in the cache space into a data batch when the duration of the data stored in the cache space reaches the timer trigger period, according to a preset timer trigger mechanism. The producer module is also used to distribute data in batches to queues; The consumer module is used to call data processing resources to process batches of data in the queue; The producer module is also used to adjust the timing triggering cycle according to the queue pressure, and the consumer module is also used to adjust the amount of data processing resources called according to the queue pressure, so as to control the queue pressure within a preset range. The queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches.

[0079] Specifically, in one embodiment, such as Figure 3 As shown, the producer module is connected to a data receiving module. The data receiving module receives the data to be processed and transmits it to the producer module. The producer module adjusts its timing trigger cycle based on queue pressure. The producer module batches data and sends it to the queue, which then transmits queue pressure to both the producer and consumer modules. The consumer module controls the number of consumers (i.e., the number of threads) and distributes batches of data from the queue to each consumer. When a pre-fetching mechanism is in place, the consumer module pre-fetches batches of data from the queue and stores them in local memory. Specifically, local memory can be a shared memory buffer for all consumers or a dedicated memory buffer for each consumer.

[0080] In one embodiment, the producer module is responsible for continuously collecting messages (i.e., collecting data to be processed) from the data source and temporarily storing them in an internal buffer. To adapt to changes in system load, this embodiment introduces a time series analysis and forecasting algorithm to adaptively adjust the data aggregation time window. Its design principles are as follows: When the system load is high, a larger time window is used to process multiple messages in batches to improve throughput. When the system load is low, a smaller time window is used to achieve near real-time processing while ensuring system responsiveness and stability. The dynamic adjustment process of the time window can be predicted based on historical load trends or queue status, achieving smooth and gradual adjustment and reducing the impact of sudden fluctuations on system performance.

[0081] The system features an adaptive aggregator, which is the core module for implementing dynamic time window adjustment. Its main functions are as follows: Timed trigger mechanism: The aggregator maintains a configurable high-precision timer, whose timing trigger period is dynamically adjusted by the system monitoring and control center according to the load. The period range can be set from 10 ms to 1 s. Whenever the timer is triggered, the aggregator packages all the cached data in the buffer space into a data batch and sends the batch to the downstream data distribution queue.

[0082] Capacity triggering mechanism: The aggregator monitors the amount of data in the buffer space in real time; when the amount of cached data reaches a preset threshold, it can immediately perform data packaging and distribution operations without waiting for the timer to be triggered. This mechanism ensures low-latency processing even under high load or burst traffic conditions, preventing queue backlog.

[0083] The innovation of the dual-trigger strategy: By simultaneously introducing time-triggered and capacity-triggered mechanisms, this method ensures both the real-time performance of data processing and improves system throughput. Compared to traditional fixed window or single trigger methods, this strategy can adaptively optimize the data batch size and distribution frequency according to the system's operating status, thereby balancing throughput and latency performance.

[0084] Adaptive aggregation time window adjustment mechanism: This embodiment provides an adaptive aggregation time window adjustment mechanism based on queue occupancy ratio, which is used to dynamically determine the time window for batch processing of the producer module to adapt to changes in system load and achieve a balance between high throughput and low latency.

[0085] In one embodiment, the producer module is further configured to use the time series of the queue pressure, combined with a preset smoothing coefficient, to calculate the timing triggering period at time t.

[0086] In one embodiment, queue pressure includes occupancy ratio, wherein the occupancy ratio represents the proportion of the total number of data batches in the queue to the total capacity of the queue; the producer module is further configured to determine a reference value for the timing trigger cycle at time t based on a preset maximum timing window, a preset minimum timing window, and the occupancy ratio; Calculate the product of the smoothing coefficient and the timing trigger period at time t-1 to obtain a first value, wherein the smoothing coefficient is between 0 and 1; The smoothing margin of the smoothing coefficient is calculated by multiplying it by the reference value of the timing trigger cycle to obtain a second value, wherein the smoothing margin = 1 - smoothing coefficient; Summing the first value and the second value yields the timing trigger period at time t.

[0087] In one embodiment, the data adaptive processing system includes multiple triggering mechanisms, each of which corresponds to a triggering condition. The triggering mechanism includes a timing triggering mechanism, and the triggering condition of the timing triggering mechanism is whether the duration reaches the timing triggering cycle. The producer module is also used to package the data to be processed in the cache space into a data batch when the data to be processed in the cache space reaches the triggering condition of any of the triggering mechanisms.

[0088] In one embodiment, the triggering mechanism includes a capacity triggering mechanism, wherein the triggering condition of the capacity triggering mechanism is whether the amount of data cached in the cache space reaches a preset cache amount threshold. The producer module is also used to package the data to be processed in the cache space into a data batch when the amount of data cached in the cache space reaches the cache amount threshold, according to the capacity triggering mechanism.

[0089] In one embodiment, the consumer module is further configured to calculate the call volume using a feedback control algorithm based on the degree of difference between the queue pressure and the preset target pressure, wherein the larger the call volume, the more data processing threads are in the data adaptive processing system.

[0090] In one embodiment, the consumer module is further configured to pre-extract at least one of the data batches in the queue into the local memory of the data processing resource based on the queue pressure, wherein the greater the queue pressure, the more data batches are pre-extracted.

[0091] In summary, this embodiment has at least one of the following beneficial effects: 1. By combining time-triggered and capacity-triggered aggregation mechanisms, batch processing throughput is significantly improved while ensuring real-time performance.

[0092] 2. The batch size of the aggregation is dynamically changed according to the actual load, avoiding performance mismatch issues caused by fixed batch configurations in different traffic scenarios.

[0093] 3. The time window adjustment algorithm based on time series analysis smooths the aggregation rhythm and effectively suppresses system jitter caused by frequent window enlargement or shrinking.

[0094] 4. The aggregation time window is strictly limited to the minimum and maximum threshold range to prevent uncontrollable delays or resource consumption under extreme load conditions.

[0095] 5. Consumer quantity adjustment mechanism based on control feedback control has higher stability and anti-oscillation capability compared with fixed threshold strategy.

[0096] 6. The synergistic effect of proportional, integral, and differential terms enables the system to respond quickly to sudden traffic surges, eliminate long-term backlogs, and suppress excessive expansion.

[0097] 7. Multiple constraint strategies effectively prevent system instability caused by disordered scaling up and down of consumer threads in high-concurrency scenarios.

[0098] 8. Load-aware prefetching mechanism significantly reduces the blocking time of consumer threads and improves processing continuity by obtaining data in advance.

[0099] 9. The prefetch quantity changes dynamically with the queue occupancy ratio, improving throughput under high load and avoiding unnecessary memory consumption under low load.

[0100] 10. The prefetching mechanism and dynamic consumer adjustment form a synergistic positive feedback, improving overall processing capacity without blindly increasing threads.

[0101] This application also provides an electronic device, which includes a memory and a processor; Memory, used to store computer programs; A processor is used to execute computer programs to implement the steps of the methods described above.

[0102] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the above-described method.

[0103] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A data adaptive processing method, applied to a data adaptive processing system, the method comprising: The acquired data to be processed is stored in the cache space of the adaptive data processing system; According to the preset timing trigger mechanism, when the duration for which the data to be processed is stored in the cache space reaches the timing trigger cycle, the data to be processed in the cache space is packaged into a data batch. The data is distributed in batches to the queue; The data processing resources of the adaptive data processing system are invoked to process the data batches in the queue; The timing triggering period is adjusted according to the queue pressure, and the amount of data processing resources called is adjusted according to the queue pressure, so as to control the queue pressure within a preset range. The queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches.

2. The data adaptive processing method according to claim 1, characterized in that, The step of adjusting the timing trigger period according to the queue pressure includes: Using the time series of queue pressure and a preset smoothing coefficient, the timing triggering period at time t is calculated.

3. The data adaptive processing method according to claim 2, characterized in that, The queue pressure includes the occupancy ratio, wherein the occupancy ratio represents the proportion of the total number of data batches in the queue to the total capacity of the queue; The step of calculating the timing trigger period at time t using the time series of queue pressure and a preset smoothing coefficient includes: Based on the preset maximum timing window, the preset minimum timing window, and the occupancy ratio, determine the reference value for the timing trigger cycle at time t; Calculate the product of the smoothing coefficient and the timing trigger period at time t-1 to obtain a first value, wherein the smoothing coefficient is between 0 and 1; The smoothing margin of the smoothing coefficient is calculated by multiplying it by the reference value of the timing trigger cycle to obtain a second value, wherein the smoothing margin = 1 - smoothing coefficient; Summing the first value and the second value yields the timing trigger period at time t.

4. The data adaptive processing method according to claim 1, characterized in that, The data adaptive processing system includes multiple triggering mechanisms, each of which has a corresponding triggering condition. The triggering mechanism includes the timing triggering mechanism, and the triggering condition of the timing triggering mechanism is whether the duration reaches the timing triggering cycle. After storing the acquired data to be processed into the cache space of the data adaptive processing system, the method further includes: When the data to be processed in the cache space reaches the triggering condition of any of the triggering mechanisms, the data to be processed in the cache space is packaged to form a data batch.

5. The data adaptive processing method according to claim 4, characterized in that, The triggering mechanism includes a capacity triggering mechanism, the triggering condition of which is whether the amount of data cached in the cache space reaches a preset cache amount threshold. After storing the acquired data to be processed into the cache space of the data adaptive processing system, the method further includes: According to the capacity triggering mechanism, when the amount of data to be processed stored in the cache space reaches the cache amount threshold, the data to be processed in the cache space is packaged into a data batch.

6. The data adaptive processing method according to claim 1, characterized in that, The step of adjusting the amount of data processing resources called based on the queue pressure includes: Based on the difference between the queue pressure and the preset target pressure, the call volume is calculated using a feedback control algorithm, wherein the larger the call volume, the more data processing threads are in the data adaptive processing system.

7. The data adaptive processing method according to any one of claims 1-6, characterized in that, After distributing the data batch to the queue, the method further includes: At least one batch of data from the queue is pre-extracted into the local memory of the data processing resource based on the queue pressure, wherein the greater the queue pressure, the more batches of data are pre-extracted.

8. A data adaptive processing system, characterized in that, It includes a producer module and a consumer module, wherein the producer module has a cache space and the consumer module has local memory; The producer module is used to store the acquired data to be processed into the cache space; The producer module is also used to package the data to be processed in the cache space into a data batch when the duration of the data to be processed stored in the cache space reaches the timer triggering period, according to a preset timer triggering mechanism. The producer module is also used to distribute the data batches to the queue; The consumer module is used to invoke data processing resources to process the data batches in the queue; The producer module is further configured to adjust the timing triggering period according to the queue pressure of the queue, and the consumer module is further configured to adjust the amount of data processing resources called according to the queue pressure, so as to control the queue pressure within a preset range, wherein the queue pressure is used to characterize the number of data batches in the queue and / or the rate of change of data batches.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the method according to any one of claims 1-7.

Images