Optimization processing method of streaming data dynamic window

Abstract

The invention relates to an optimization processing method for a streaming data dynamic window, and the method comprises the steps: taking streaming data as a dynamic system, and calculating the data disturbance energy at each moment on a time axis; constructing a window trigger point according to the potential energy curve; when the trigger potential locally reaches a peak value or the trend changes, adaptively dividing the window; the starting boundary and the ending boundary of each window are defined as elastic intervals; before the window slides or is closed, window boundary self-adjustment and complementation are carried out according to compensatory analysis of subsequent data streams; taking a large window as a container, and dividing sub-segments according to data density change; partial states are shared among the sub-windows, and fine-grained analysis including anomaly detection and instantaneous peak value recognition is executed; introducing a life cycle termination relationship driven by an entropy threshold: when the window activeness is lower than the entropy threshold, triggering state compression or early release; the accuracy and dynamics of window division are realized, the response capability of a system to data fluctuation and burst is enhanced, and the elasticity and fault-tolerant capability of window boundaries are improved.

Description

Technical Field

[0001] The present invention relates to a method for optimizing a dynamic window, and more particularly to a method for optimizing a dynamic window of streaming data. Background Art

[0002] In current technical solutions such as the real-time stream data processing method and device based on stream window implementation in Chinese patent CN111210156A, although attempts have been made to realize the connection and real-time calculation of waybill flow data and distribution facility flow data through the stream window mechanism, and to perform indicator aggregation analysis by calling different types of window processing processes to support business scheduling and resource optimization, the existing technology still has significant technical limitations and practical application drawbacks in terms of the dynamics, intelligence, data adaptability, and resource optimization of window management. Its shortcomings are mainly reflected in the following aspects: First, the stream window division is usually dependent on the time definition of the static configuration of the task, that is, the start and end boundaries of the window need to be clearly set in the development or scheduling stage. Although different window types (such as sliding windows, rolling windows, session windows, etc.) can be called to meet the aggregation needs of different indicators, their windows The partitioning logic is still driven by preset rules and lacks the ability to adaptively perceive the behavior of the data stream itself. As a result, when the data arrival frequency suddenly changes, the event semantics turn, or the structural distribution changes, it is still aggregated according to a fixed granularity, which can easily cause the data within the window to be fragmented, incomplete, or mixed with irrelevant information, which is not conducive to capturing and identifying fine content such as sudden behaviors and abnormal patterns; secondly, although the stream window processing process used in the existing technology supports the concurrent execution of multiple windows, it does not provide a dynamic monitoring and lifecycle adjustment mechanism for the window's own operating status. Once the window is started, it will continue to run until the preset time or the end of the event capacity, ignoring the situations that may occur in actual operation, such as a sharp drop in data value, a long period of static status, or an overflow of duplicate content, which can easily lead to a waste of system memory and computing resources, and is not conducive to resource scheduling and priority management in a high-load environment; Furthermore, the window boundaries in the existing technical solutions are basically static configurations, and there is a lack of effective processing strategies for abnormal flow situations such as disordered data and delayed data. In scenarios where there is a deviation between the actual arrival time of the data and the logical time of the event, it is easy for some data to be incorrectly divided into the wrong window, affecting the accuracy of the aggregation results. In addition, although the window aggregation method of the existing technology can achieve personalized processing based on indicators, it still uses full-state detailed storage for state management. Especially when multiple windows run concurrently, it is easy to cause system state expansion and increased state storage pressure. At the same time, the existing technology does not provide a resource scheduling optimization strategy during the window processing process, that is, whether the window continues to run, whether it can be released in advance, whether it should be merged with other windows, etc., lacks global perception and system-level linkage. Finally, the existing technology focuses on the calculation of single-volume indicators in specific business scenarios, and does not provide a universal data behavior-driven window management model, which limits its adaptability in different types of stream data scenarios (such as financial flow, user behavior flow, IoT perception flow, etc.). Summary of the Invention

[0003] The purpose of the present invention is to provide a method for optimizing the processing of dynamic windows of streaming data, thereby solving some of the drawbacks and deficiencies pointed out in the background art.

[0004] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: a method for optimizing the dynamic window of streaming data, comprising: treating the streaming data as a dynamic system and calculating the data disturbance energy at each moment on the time axis, including the burst rate, first-order derivative change, and distribution skewness; constructing a window trigger point based on the potential energy curve; and adaptively dividing the window when the trigger potential reaches a local peak or the trend changes. The start and end boundaries of each window are defined as elastic intervals. Before the window slides or closes, the window boundaries are self-adjusted and padded based on compensatory analysis of subsequent data streams. If out-of-order data is found to have missed the original window, it is re-inserted or merged using delayed self-healing logic. The large window is regarded as a container and divided into sub-segments according to changes in data density. Sub-windows share some states but perform fine-grained analysis including anomaly detection and instantaneous peak identification. The sub-window structure is temporary and does not affect the external window boundary, but improves the insight within the window. An entropy threshold-driven lifecycle termination relationship is introduced: when the window activity falls below the entropy threshold, state compression or early release is triggered.

[0005] Furthermore, the method of adaptively dividing the window includes: Continuously map the received real-time streaming data into a data evolution sequence; extract multi-dimensional flow feature information based on the data evolution sequence, including event arrival frequency, field change rate, data density, and time interval fluctuation; Utilize flow characteristic information to construct a potential energy curve reflecting data state changes; monitor local peaks, mutation points, or trend inflection points of the potential energy curve on the time axis; and use the local change points as window trigger points to adaptively divide the streaming data window. The divided data window is input into the subsequent calculation or processing module.

[0006] Furthermore, the potential energy curve is constructed in real time based on the structure and behavior of the data itself, and the generation parameters are dynamically adjusted to adapt to changes in the data stream distribution. The window trigger point is determined based on the combined judgment of multiple local change indicators, including: the rate of change of the first-order derivative of the potential energy, the fluctuation amplitude in a short period of time, and the difference between consecutive events. Specifically, we define a data behavior-driven potential energy function to reflect the evolution of streaming data in real time. We then combine the derivative changes of this function with local trends to determine the window trigger point, thereby building an intelligent window with perceptual capabilities. Define the window-triggered potential function as follows: in: Indicates the current time point; Indicates the behavior deviation rate of the data field within the neighboring window, measuring the field change trend at the current time point compared with the previous period; Indicates the fluctuation of data density per unit time, reflecting the change in the arrival frequency of inflow events in a short period; Indicates the continuous difference of events, that is, the structural difference between adjacent data, which is used to measure the semantic mutation of data; is the weight coefficient, which is used to reflect the comprehensive influence of different change indicators on the potential energy function; the overall function A curve is formed over time to capture key trigger points; System real-time calculation function , and process it as follows: Calculate trend changes of a function (e.g., consecutive rising or falling segments); Detect local extreme points or inflection points; Determine the sign change of the derivative (i.e., from rising to falling, from stable to sudden); When the following composite conditions are met, the current time point is determined For the window trigger point: It is a local peak or an obvious transition; And in a short time The rate of change of the first derivative of increases or decreases significantly; and at least one component term (e.g. ) exceeds the set threshold; Once the trigger point is confirmed, the system immediately starts a new window division at that point, thereby realizing adaptive streaming window boundary generation based on the data evolution structure.

[0007] Furthermore, after the window trigger point is divided, a window boundary elastic adjustment mechanism is executed for subsequent out-of-order or delayed data to support dynamic correction or expansion of the window boundary.

[0008] Furthermore, the entropy threshold-driven lifecycle termination relationship construction method includes: Monitor the real-time data stream in the divided dynamic window; continuously calculate the window activity based on the variability, density, and distribution diversity indicators of the data in the window; when the window activity falls below the preset information entropy threshold, trigger the window life cycle termination operation; The lifecycle termination operation includes at least one of the following: a) compressing the intermediate state information stored in the window to release system resources; b) Close the window in advance and submit the processing results.

[0009] Furthermore, the activity is calculated based on the following: the frequency of field value changes within the window, the degree of change in event intervals, the range of field distribution changes, and the deduplication rate of events within the window; the information entropy threshold is used to reflect the current information density or novelty of the window. When this indicator is continuously lower than the threshold for more than a continuous period, it is judged that the window has entered the decay stage; For each divided dynamic window, periodically evaluate the data activity within it, including: How often the field value changes; Fluctuations in the arrival intervals between adjacent events; the degree of spread of field distribution; Changes in data deduplication rates reflect content diversity; The above dimensions jointly determine the window activity intensity curve and are further mapped into an overall metric value for comparison with the information entropy threshold set by the system. The activity potential entropy function is defined as follows to determine the window life cycle status: in: :Indicates window Activity potential entropy value in the current evaluation period; 、 : The evaluation start and end time points of the current window; : It represents the frequency function of field changes, reflecting the degree of change of each field in unit time; : event interval fluctuation function, which quantifies the instability of the time interval between consecutive events; : Field distribution concentration function, which measures whether the distribution of the current field value tends to be single; 、 : Adjustable parameters that control the weights of different indicators; : The density function of the deduplication rate of the data within the window, reflecting the change in data diversity, serves as the weight coefficient; symbol : Indicates the continuous accumulation of dynamic features within the window time range; The system calculates in each time period : If the value is continuously lower than the set information entropy threshold , and the holding time exceeds the tolerance duration period , the window is determined to have entered the information decay phase; this triggers the window lifecycle termination process, including: Compress state information to save memory; Close the window and release resources; Or merge into other active windows.

[0010] Furthermore, the state compression operation expresses the state by introducing an approximate statistical structure, including count estimation, frequency compression, summary coding or probability-based data structure.

[0011] Furthermore, the early release operation is completed before the window reaches its scheduled time or event capacity condition, and is linked with the system resource management module to reclaim the memory and computing resources occupied by the window.

[0012] The beneficial effects of the present invention are as follows: by constructing a potential energy function to identify local peaks, trend mutations, or behavioral inflection points of data states, automatic window division based on endogenous changes in data is achieved, avoiding data fragmentation or misaggregation caused by traditional fixed time or event number division, and improving window semantic integrity and analysis accuracy; the introduced potential energy derivative trend monitoring and change amplitude determination mechanism can timely divide windows when data streams experience drastic fluctuations, thereby quickly capturing abnormal patterns or emergencies, and improving the real-time performance and sensitivity of the system in scenarios such as financial risk control and anomaly detection; The window boundary adopts a soft adjustment model, combined with a tolerance strategy for data delay and out-of-order arrival, to ensure that delayed data can be accurately classified into the corresponding window within the legal range, effectively preventing analysis errors caused by timing misalignment; the window information value is dynamically evaluated through the activity potential entropy function, and the life cycle is terminated when the activity is continuously lower than the entropy threshold. It supports operations such as state compression, early release or window merging to release memory, computing power and state storage resources, and avoid invalid windows occupying system resources for a long time; a variety of approximate statistical structures are used to compress and express window state information, greatly reducing state storage and transmission overhead while ensuring the availability of results, which is especially suitable for ultra-large-scale distributed stream processing scenarios; after releasing the window in advance, it is linked with the resource scheduling module to promptly reallocate the released computing resources to high-load and high-value windows, thereby improving the computing power utilization and load responsiveness of the overall system and avoiding performance bottlenecks and delay peaks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the process of adaptively dividing streaming data into dynamic windows according to the present invention.

[0014] Figure 2 This is a functional relationship diagram of the elastic window boundary and the self-healing mechanism of the present invention.

[0015] Figure 3 This is a timing diagram of the large window and sub-window decomposition and lifecycle management of the present invention.

[0016] Figure 4 This is a simplified diagram of the adaptive window and elastic boundary processing of vehicle flow data according to an embodiment of the present invention.

[0017] Figure 5 This is a flowchart of window lifecycle termination and resource recycling driven by entropy threshold according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a detailed description of the specific embodiments of the present invention with reference to the accompanying drawings.

[0019] Combined with attachment Figure 1, the optimization processing method of the dynamic window of streaming data of the present invention regards the data stream as a continuously evolving dynamic system, and by analyzing its disturbance behavior characteristics in the time series, constructs a potential energy-driven window division mechanism to achieve adaptive triggering and intelligent management of windows. The system inputs the received streaming data into the data behavior modeling module, which extracts the characteristic changes of each data in the stream at the arrival time point in real time, including but not limited to: data burst rate, that is, the rapid increase or decrease in the number of events per unit time; first-order derivative change, which refers to the rate of change of field values over time; and distribution skewness, which is used to measure the offset trend and concentration degree of field values in the statistical distribution. Based on the combined characteristics of the above-mentioned multiple change dimensions, the system constructs a continuous disturbance potential energy curve, which is used to describe the fluctuation intensity and behavior change trend of the data system in the time dimension, where the potential energy value corresponding to each time point reflects the degree of instability or structural change of the system at that moment. When the potential energy curve shows a local peak, a sharp jump, or a sudden change in the sign of the derivative at a certain point in time, the system determines that the current data stream state has undergone a significant turning point or mutation, that is, the trigger condition has been met. At this time, the system marks this time point as a window trigger point and immediately divides a new dynamic window based on it; this window will be used to carry the data of the current disturbance segment for subsequent computing tasks, including aggregation, detection, or anomaly identification. This adaptive window division method does not rely on traditional fixed time intervals or event number thresholds, but is completely triggered based on the data's own evolutionary trend. It has high flexibility and environmental adaptability, and can effectively capture key behavioral segments in complex streaming scenarios. It is particularly suitable for data stream processing tasks with nonlinear fluctuations, burst characteristics, or semantic segment switching, such as online log analysis, financial market transaction flow monitoring, IoT perception data clustering, and other typical application scenarios.

[0020] Combined with attachment Figure 2, introduces a window boundary adjustment mechanism with boundary elasticity and delay self-healing capabilities to address issues such as overly rigid boundary demarcation, high data omission rates, and rigid window granularity in traditional windows facing complex stream environments such as out-of-order data, delays, and data catch-up. During the window demarcation phase, the system constructs a set of elastic boundary intervals. This means that both the start and end times of the window are set within a time range that allows for fluctuations or adjustments. This is called a boundary elasticity buffer. This interval allows subsequent data to re-adjust the original window boundaries under specific conditions. Before a window slides or is about to close, the system performs a compensatory analysis of the incoming data in the subsequent short period of time. It determines whether the newly arrived data has a significant correlation with the window that is about to close in terms of time, field characteristics, data density or event semantics. If certain similarity or continuity conditions are met, the system performs a fine-tuning operation on the window boundary, extending the start or end time of the current window backward or forward to form a window range that is more in line with the actual data structure, thereby enhancing the window's adaptability to data flow changes. At the same time, to solve the data omission problem caused by out-of-order data, a delay self-healing logic based on the timestamp tolerance mechanism is designed. This logic marks data that arrives out of order or is pushed late. When the timestamp of such data falls within the elastic boundary interval of a closed window, the system reinserts the data into the original window for supplementary recording, or merges it with the adjacent new window if the original window has completed processing, ensuring that data is not lost, duplicated, or terminated prematurely, thereby improving the overall fault tolerance and semantic integrity of the window. Through this method, each window has the ability to dynamically self-adjust, can timely correct its own structure according to the changing characteristics of the actual data stream, and has the recovery ability to meet the data integrity protection needs in disordered scenarios. It is particularly suitable for real-time computing environments such as log stream reconstruction, user behavior sequence reorganization, and sensor network flow correction that have high requirements for timing accuracy and data integrity.

[0021] Combined with attachment Figure 3, introduces a window enhancement mechanism with decomposable internal structure, monitorable behavior, and controllable life cycle to solve the performance bottlenecks of traditional window structures in terms of uneven data distribution, intensive event fluctuations, and resource utilization efficiency. After the window division is completed, the system treats the large window as a logical container perspective and dynamically divides it into several sub-segment structures based on the changes in data density within the window. That is, during the window life cycle, the window is further subdivided into several sub-windows according to the local density of data arrival, the time interval change trend, or the field behavior mutation point. Each sub-window exists as a temporary analysis unit, which is used to carry data within a specific time range in the window and perform refined processing operations, including but not limited to anomaly detection, mutation peak identification, short-cycle pattern analysis, etc. In this way, while maintaining the overall boundary of the original large window unchanged, its internal analysis granularity and structural sensitivity are improved. Sub-windows can share some intermediate calculation states, such as basic statistics, historical context, or pre-processing cache, to reduce repeated calculations and state overhead, but each performs independent local feature mining tasks to avoid pattern dilution or anomaly omission caused by excessive window granularity. During the sub-window construction process, the system continuously tracks the active behavior changes of each sub-window and performs refined processing operations based on data behavior. The system constructs its information activity index based on features and further introduces a lifecycle termination mechanism driven by information entropy threshold. That is, the system calculates the activity function value of the sub-window in each cycle. When it is found that the value is continuously lower than the set information entropy threshold and the maintenance time exceeds the preset duration period, the sub-window is determined to have entered the inefficient decay stage. At this time, the system will trigger corresponding optimization operations, including compressing its intermediate state data structure into an approximate expression to release memory resources, or directly closing the sub-window to release computing power and state space in advance, thereby avoiding redundant calculations and resource waste. This mechanism ensures the lightweight, adaptive and controllable termination of the sub-window structure, while not affecting the boundary stability and output consistency of the large window to which it belongs. It improves the insight ability within the window and the resource efficiency of system processing as a whole. It is particularly suitable for online data analysis tasks with multi-peak distribution streams, non-stationary event sequences, or extremely uneven scene density, such as real-time user behavior sequence analysis, IoT perception peak monitoring, financial anomaly event location, and other application scenarios with high requirements on local accuracy and processing efficiency.

[0022] Example 1: Combined with attachment Figure 4In this embodiment, a real-time vehicle perception system is deployed at the entrance of the highway in the intelligent traffic monitoring system. Each vehicle passing through the entrance will report a data event in real time. The event field contains information such as vehicle ID, passing timestamp, vehicle speed, license plate type, and region. These data form a typical streaming data source. The system hopes to use data window partitioning technology to identify behavioral patterns such as sudden traffic flow, slow-moving areas, or abnormal speed sections. However, the traditional method based on fixed time intervals (such as a window every 10 seconds) or a fixed number of events (such as a window every 100 events) will lead to unreasonable window partitioning when the traffic flow is sparse or dense, which easily leads to the phenomenon of congested sections being divided or traffic sections being diluted. Therefore, the system applies a streaming data dynamic window adaptive partitioning method to optimize the processing.The system maps the vehicle events received in real time into a time-ordered data evolution sequence, which grows in real time as vehicles pass by. After entering the processing flow, the system extracts multi-dimensional flow features from the sequence, counts the arrival frequency of vehicle events in each second time window, forms the current traffic density curve, records the range of change and mean offset of the speed field in each event to form a field change rate indicator, and analyzes whether the time interval fluctuations between vehicles are stable, such as whether there is a sudden dense convoy or sparse tail flow. The above three types of information are used together as the current data status Based on the flow characteristics of the vehicle, the system constructs a potential energy curve for the vehicle passing state. During a certain period of time, such as between 7:56 and 7:57 in the morning, the system monitors that the traffic volume suddenly increases from 2 vehicles per second to 15 vehicles per second. At the same time, the speed fluctuation increases from an average of ±4km / h to ±13km / h, and the interval time decreases from an average of 0.6 seconds to 0.1 seconds. This state is manifested on the potential energy curve as a significant peak after a continuous rise. Then the potential energy trend suddenly turns downward around 7:57:06. The system recognizes this as a local peak of the data state and guides the trigger. When a change occurs, this time point is marked as a window trigger point, and a complete data window is divided from the previous trigger point to the current inflection point. This window covers the entire period of traffic flow growth. The system then continues to observe potential energy changes and divides the window again at the next mutation point. The resulting window is not restricted by fixed time or the number of events, but is adaptively divided based on the flow rhythm and state changes of the data itself. The divided window is input into the calculation module for aggregation analysis and anomaly identification. For example, the system marks a high-density traffic window as a potential congestion indicator. The vehicle speed dispersion and frequency peak detection modules further confirm its anomaly level. This information is sent to the front-end prompt system or the linked traffic control module. Throughout this process, the window division accurately captures the structural fluctuation characteristics of traffic flow, ensuring that the data is not fragmented at inflection points and that sudden states are not missed due to fixed granularity. The system further observes traffic flow monitoring within the next minute, noting that vehicle speed and flow have returned to a stable state, and the potential energy curve has returned to a low level with no obvious trend fluctuations. Therefore, the system determines that this section does not have significant state disturbances and only slowly slides the window for basic statistical processing.

[0023] The system further introduces a window-triggered potential function for adaptive window partitioning: This function is used to implement intelligent judgment and division of window boundaries. First, the system sets the time granularity to 1 sampling point per second and calculates three core sub-functions based on the vehicle data received per second. Indicates the field behavior deviation rate, which is calculated by the difference between the current time point t and the standard deviation of the speed field change in the past 3 seconds sliding window. At 7:57:00 am, the system observed that the standard deviation of the speed field jumped from the previous 2.8km / h to 7.9km / h, and the behavior deviation rate at that moment was calculated. , then calculate Indicates density volatility, that is, the rate of change of vehicle event frequency per unit time. In the previous 3 seconds, the average was 3 vehicles per second, and the current second instantly reached 18 vehicles. The density change ratio ,at last, Indicates the continuous difference of events, that is, the distance change in the field structure of each pair of adjacent vehicle events. Here, the system uses the joint coding difference of license plate type and area field to measure it. There are about 19 inconsistent field structures in 30 adjacent data. The continuous structure difference is inferred. , combined with the set weight parameters , , , these parameter values all fall within the closed interval [0,1]. The system can be optimized through the parameter adjustment mechanism or pre-training model to meet This normalized constraint is brought into the calculation to obtain the value of the potential energy function at this moment: The system records the potential energy value and compares it with the value of the last second. and the previous second By comparison, it is found that the potential energy value shows an accelerating upward trend. At 7:57:01, The value dropped to 4.2, forming an obvious local peak and the first-order derivative change rate was negative, rapidly decreasing from positive to negative. At the same time, the density fluctuation index in this area Once exceeded the set fluctuation threshold , meeting the composite conditions for determining the window trigger point, namely: the potential energy function at the current time point is a local extreme value and the trend of change is reversed, and at least one component function exceeds the threshold set by the system. Therefore, the system identifies 7:57:00 as the window trigger point, and immediately uses this time point as the starting or ending boundary for the new window division, and combines the previous trigger point to establish a complete data window at 7:56:45. This window effectively covers the sudden increase in traffic flow. In the subsequent calculation module, the system performs abnormal vehicle speed identification and predictive aggregation processing on the events in the window, and determines that it is an instantaneous traffic flow impact section induced by upstream road construction. At the same time, because the window division point falls precisely at the inflection point of the potential energy trend, the problem of traffic flow fluctuation behavior being split across windows is avoided, which improves the integrity and responsiveness of the analysis results. The mechanism can still maintain sensitivity during the sparse traffic period in the afternoon. Between 14:20 and 14:25, due to the decrease in traffic volume, the field changes tend to be stable, and the system continues to calculate The value fluctuates between 1.2 and 1.5, and there are no drastic changes in various indicators. The sign of the derivative remains close to zero for a long time. The system determines that the current state is stable and there is no need to divide a new window. The existing sliding window structure remains unchanged, thus saving computing resources.

[0024] After the system completes windowing using the adaptive trigger points of the potential energy function, it applies a flexible window boundary adjustment mechanism to further enhance the robustness and integrity of the system's handling of complex real-world traffic data. This mechanism addresses issues such as data out-of-order or delayed arrival in streaming data due to network latency, sensor jitter, or cache congestion. This mechanism ensures that even after windowing is complete, the system can still rationally adjust and dynamically expand window boundaries to ensure data integrity and semantic accuracy within the analysis window. After the window was completed at 7:57:00 AM and fed into the downstream computing engine, the system received a vehicle event data entry at 7:57:06 AM, with a timestamp of 7:56:59.500, the field is vehicle speed: 88km / h, the license plate type is operational, the system immediately identifies the event as delayed data, because its timestamp is before the window end 7:57:00, theoretically it should belong to the previous window but missed insertion due to network delay. If the traditional stream processing method is used, the event will be discarded or mistakenly included in the next window because the window has been closed, which will cause the congestion mode judgment to shift and the speed distribution statistics to deviate. The boundary elastic adjustment mechanism described in the present invention provides a solution strategy for this. The system maintains a time range called elastic back-boundary buffer for each divided window. In this case, the buffer time period is set to 5 seconds, that is, within the window Legal late events are still allowed to be included in the window processing within 5 seconds after the end time of the port. Therefore, the event meets the delay tolerance interval condition when it arrives. The system queries its timestamp to fall within the tolerance range of 7:57:00±5 seconds, confirming that it belongs to the legal delayed data of the previous window, triggering the boundary extension logic, temporarily inserting the event into the tail buffer area of the window, and synchronously updating the internal state value of the window, including recalculating the maximum value, standard deviation, field distribution and other indicators of the vehicle speed, and deriving the local field offset of the window tail based on the new information of the event, and updating its decision variables that affect downstream judgment, such as re-evaluating whether the window still meets the requirements. The system meets the abnormal standard of high discrete vehicle speed to ensure the accurate continuation of data semantics. In another period, at 15:02:15 in the afternoon, due to the restart of the upstream sensing device, a large number of historical buffer events were reissued, including dozens of data with timestamps between 15:01:55 and 15:02:05. At this time, the time interval has been covered by multiple windows. The system compares and inserts the timestamp of each data in the elastic boundary interval of the corresponding window. If a data overlaps with multiple windows, the target window for insertion is determined based on the minimum field similarity error principle to ensure the correct attribution of data and avoid duplication. Statistics are also available. Furthermore, after the window boundary expansion operation is completed, the system sets a short-term review state for the expanded window. This means that the window remains temporarily writable for 5 seconds after expansion to accommodate more potential supplemental data. Only after the review period does the window officially lock its output and hand it over to downstream processing, preventing frequent writebacks from inflating system overhead. This window boundary elasticity mechanism allows the system to maintain high throughput while also achieving a high degree of fault tolerance for streaming data uncertainty, significantly improving event processing accuracy and semantic continuity. It is particularly suitable for business scenarios with high concurrency, low latency, and a high risk of data out-of-order, such as IoT, video streaming logs, and transaction matching systems.

[0025] Example 2: Combined with attachment Figure 5On the basis of Example 1, in order to further improve the resource utilization efficiency and window scheduling intelligence under high-concurrency streaming data processing, the system deploys the proposed window life cycle termination mechanism driven by entropy threshold, which can dynamically evaluate each divided window during operation and make a decision on whether to release or compress it in advance, effectively avoiding low-value windows from occupying system resources while maintaining the integrity and real-time nature of the processing results. During the low traffic period from 16:05 to 16:25 in the afternoon, the frequency of vehicles passing through the entrance dropped significantly. According to the mechanism of the present invention, the system first periodically monitors each active window and counts the data change index in the window every 30 seconds. The data includes field value variability (such as vehicle speed and license plate type), event density (number of vehicles passing per unit time), and distribution diversity (such as coverage of the license plate's assigned area). In window ID#W102, recording began at 16:10:00, and within 30 seconds, only 11 vehicles were recorded. This represents a 95% decrease in event density compared to the system average (5 vehicles per second). Furthermore, eight of the 11 vehicles were from the same area, with a standard deviation of 1.2 km / h in speed. Field variability was extremely low, and there was no switching in license plate type. Based on this information, the system calculated the activity value for this window and converted it to information entropy, resulting in a current entropy value of 0.41. The system's default entropy threshold is 0. 75. Combining historical model experience and resource load, the system determines that the window has entered a state of behavioral decay and has no value to be retained. Therefore, the life cycle termination logic is triggered immediately. First, the operation described in a) is executed to compress the intermediate state of the window, such as the average vehicle speed, vehicle frequency graph, and deduplicated vehicle list. The entire state structure is compressed into a lightweight statistical summary, retaining only the core indicators for global trend analysis and releasing the original state memory of about 32MB. Then, the operation b) is continued to close the window in advance and the compressed processing results are packaged and output to the result storage area. At the same time, the window is marked as terminated in advance for subsequent window schedulers to participate in. At this time, the original planned life cycle of the window was 60 seconds, but the termination operation was completed at the 36th second of operation. The system released about 2.7% of the window resource quota and dispatched the processing capacity to the next active window ID#W103. In another case, window ID#W108 had a sudden data flow retransmission after running at 16:17:00. The original activity was lower than the entropy threshold, but the entropy value rebounded to 0.83 during the observation period. The system determined that the window had not entered the termination state, maintained operation and entered the entropy state lock protection period to avoid accidental closure due to short-term fluctuations. This design reflects that the mechanism of the present invention has dynamic and progressive state assessment and self-repair capabilities.

[0026] In the period when vehicles are sparse and data changes slowly, in order to ensure the efficiency of resource allocation and the accuracy of event processing, the activity potential entropy function is introduced In this scenario, the system performs an activity evaluation on all running windows every 60 seconds to determine whether they are in an information decay state. The system selects window ID #W125 as the sample window, which runs from 16:45:00 to 16:46:00, and performs feature extraction on the raw data collected from this window. Indicates the frequency of field changes. In this window, the system mainly monitors changes in the vehicle speed and vehicle type fields. In the evaluation interval, the standard deviation of vehicle speed remained below 1.5 km / h in the first half, the vehicle type did not change, and the average value of the change frequency function was only 0.08. It represents the event interval fluctuation function. The actual sampling shows that the event interval is very stable, with an average of 1.2 seconds. The fluctuation function value is extremely small, with an average of about 0.05. The system calculates the dispersion of the license plate area field and finds that 93% of the vehicles come from the same area, resulting in a serious skewed field distribution. The average value of the concentration function reaches 0.91. The deduplication rate density function represents the data. The system records a very high repetition rate in the vehicle IDs during this window, with only 18% of independent vehicles. The deduplication rate density function value is 0.18. The system then substitutes the above variables into the activity potential entropy function: Take the evaluation period as 60 seconds and set the coefficient , these two weight coefficients are derived from the offline training results of the model, and the control range is set to In order to ensure the flexible control of fluctuation and concentration factors within the interval, the potential entropy function value of the window is calculated. ≈0.016, which is significantly lower than the information entropy threshold set by the system , and the low value state has been maintained for two consecutive cycles (i.e. 120 consecutive seconds), exceeding the set tolerance duration period Seconds, the system therefore determines that the window has entered the information decay stage according to the strategy of the present invention, and immediately triggers the life cycle termination process. First, a compression operation is performed to abstractly aggregate the complete vehicle trajectory and speed sequence originally recorded in the window, retaining only lightweight indicators such as window aggregation indicators such as average speed, peak traffic time point, vehicle area coverage, etc. After compressing the state information, the window occupies about 42MB of memory and is released. The window is then removed from the execution queue and marked as mergeable. The system further identifies that the currently running adjacent window ID#W126 is in an active state and has similar field features, so the aggregation information of W125 is merged into the state container of W126 to form a state continuation, avoiding full This process not only reduces the resource occupation of redundant windows, but also maintains the continuity of data context. During the overall operation of this mechanism, the system can recycle about 1.8 inefficient windows per minute on average and reallocate computing resources to high-entropy windows, such as ID#W130 and ID#W131, to cope with sudden traffic flow sections. Through this method, the system realizes data-behavior-based perceptual scheduling of the window life cycle, with the ability of autonomous judgment, dynamic adjustment and fine release, which significantly improves resource allocation efficiency and computing density control. It is especially suitable for traffic scenarios with drastic traffic fluctuations and high window redundancy risks, such as highway entrances, urban main roads and toll stations.

[0027] Start the state compression operation mechanism to efficiently compress the intermediate state information accumulated in the window, release system memory and retain useful summaries to support subsequent statistical and query needs. Based on the introduction of approximate statistical structures to replace the original detailed storage, the system chooses to use a combination of four compression strategies for processing, namely count estimation (CountEstimation), frequency compression (FrequencySketching), digest encoding (DigestSummarization) and probability-based data structures (such as BloomFilter or HyperLogLog) to replace state expression in the window. During the operation of port ID#W125, the system received a total of 113 vehicle events. The original state contained full field details such as the unique ID of each vehicle, speed time series, license plate area, timestamp, and original entry order. The overall state structure occupied about 42MB of memory. In order to perform compression operations, the frequency compression operation was first performed on the license plate area field, and the Count-MinSketch structure was used to record the area coding frequency. The system hashed the area field to a Sketch matrix of 4 columns and 512 rows, allowing an error range of up to 5% to estimate the frequency of each area, and successfully reduced the original area details to an approximate frequency structure of about 32KB. Subsequently, for the unique identification item in the vehicle ID sequence, the system introduced Hyp The cardinality is estimated using the erLogLog structure, confirming that at a 95% confidence level the number of unique vehicles in the window is 26±1, rather than the 113 details originally stored. The historical fluctuation information in the speed sequence is further represented by the TDigest summary structure, retaining key quantiles such as P50, P90, and P99 to support subsequent statistical analysis, while discarding atypical points to save computing resources. In addition, to support subsequent fuzzy search scenarios, the system converts the event timestamp sequence into two sliding time bucket structures to represent the peak period, and combines the compressed index for anomaly tracking. Finally, the compressed state structure of the window is condensed to 1.7MB, which is more than 96% smaller than the original 42MB state. The system registers the compressed structure to the state pool The state compression operation not only improves the system storage and memory usage efficiency, but also significantly reduces downstream processing latency and communication load, allowing the system to maintain stable operation in the presence of multiple low-activity windows, without triggering frequent GC or scheduling blockage due to state expansion. At the same time, the compressed structure has serializability and supports fusion with other window results to achieve high-level aggregation and trend analysis across windows. This mechanism has also been applied to other windows such as ID#W130. In windows with drastic speed changes, a combination of summary coding and fluctuation map compression is used to compress complex dynamic states, providing an effective low-cost data source for traffic event modeling and pattern learning systems.

[0028] With the continuous intervention of the dynamic window entropy perception mechanism, the system's processing of low-activity windows no longer relies on fixed life cycles or event arrival conditions. Instead, after meeting a specific activity threshold, the system uses an early release mechanism to cooperate with the resource management module to complete window-level resource recovery and scheduling rebalancing. In the processing cycle of window ID#W132, the window was originally planned to have a life cycle of 60 seconds, and the triggering condition was to reach at least 120 events or a running time of 60 seconds. However, when the system actually ran to the 28th second, it had judged the activity of the window through two consecutive rounds of activity potential entropy evaluation. Stable below the system set entropy threshold , the frequency of field changes in the window within the evaluation interval The vehicle type field in this section is only a private car, the speed standard deviation fluctuation value is only ±1.1km / h, the event interval mean is 1.2 seconds, the volatility is less than 0.04, the vehicle ID repetition rate exceeds 83%, and the field distribution concentration is high. Reach 0.95, and the data deduplication rate At most evaluation points, it is only 0.13. After the system calculates the , the continuous evaluation period is lower than the threshold, and the duration has exceeded the set seconds, so the system determines it as an invalid window and decides to execute the release operation in advance. At this time, the system uses the early release mechanism of the present invention to realize linkage with the resource management module in the stream processing platform, searches for the resource handle of the window in the state directory, identifies the memory objects, state snapshots, cache file handles and thread binding queues occupied by it, and then initiates a release instruction, calls the memory release API and thread unbinding operation, clears the 45MB window state storage space occupied by it, and releases 1 local state cache queue and the task threads bound to 2 CPU cores. After receiving the release completion signal, the resource management module immediately triggers the scheduling reallocation logic, reallocating the resources originally occupied by the window to the window ID#W135 which is currently in an active high-entropy state. This window is in the traffic burst area during the evening rush hour, with an event arrival frequency of up to 18 vehicles per second. The field changes drastically and requires rapid aggregation calculation. The original scheduling strategy is planned to be delayed scheduling. Due to the early release of W132, the scheduler immediately allocates the released two-core computing resources to the W135 thread execution pool to ensure the timeliness and stability of high-entropy window analysis. In addition, the early release operation marks W132 as a short-lifecycle window and writes it to the system monitoring log, which is used as the training input sample for the subsequent model-adaptive window lifecycle estimation module. Through this mechanism, the system not only prevents invalid windows from continuing to occupy resources until the end of the natural closing cycle, but also realizes dynamic resource reuse and load optimization, making the entire stream processing system have elastic processing capabilities during peak periods. Within one hour of system operation, a total of 43 low-activity windows similar to W132 were released in advance, accounting for approximately 21.8% of the total number of windows in the full cycle, reclaiming a total of approximately 1.73GB of memory resources and releasing 26 cores of thread resources. The overall average response delay of the system was reduced by 17.6%. This example verifies that the early release mechanism proposed in this invention can significantly improve resource utilization efficiency, reduce resource idle cycles, and optimize the computing performance of active windows in actual scenarios. It constitutes an indispensable part of the dynamic window lifecycle optimization system and is particularly suitable for task systems with extremely high requirements for computing power allocation and window scheduling accuracy, such as high-speed data streams and high-resource-sensitive scenarios such as intelligent transportation, IoT perception networks, and online risk control engines.

[0029] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. The optimization processing method of the dynamic window of streaming data is characterized by The following steps are involved: Treating streaming data as a dynamic system, the data disturbance energy at each moment, including burst rate, first-order derivative change, and distribution skewness, is calculated on the time axis. Window trigger points are constructed based on the potential energy curve: when the trigger potential reaches a local peak or the trend changes, the window is adaptively divided. The start and end boundaries of each window are defined as elastic intervals. Before the window slides or closes, the window boundaries are self-adjusted and padded based on compensatory analysis of subsequent data streams. If out-of-order data is found to have missed the original window, it is re-inserted or merged using delayed self-healing logic. The large window is regarded as a container and divided into sub-segments according to changes in data density. Sub-windows share some states but perform fine-grained analysis including anomaly detection and instantaneous peak identification. The sub-window structure is temporary and does not affect the external window boundary, but improves the insight within the window. An entropy threshold-driven lifecycle termination relationship is introduced: when the window activity falls below the entropy threshold, state compression or early release is triggered.

2. The method for optimizing the dynamic window of streaming data according to claim 1, characterized in that The method of adaptively dividing the window includes: Continuously map the received real-time streaming data into a data evolution sequence; extract multi-dimensional flow feature information based on the data evolution sequence, including event arrival frequency, field change rate, data density, and time interval fluctuation; Utilize flow characteristic information to construct a potential energy curve reflecting data state changes; monitor local peaks, mutation points, or trend inflection points of the potential energy curve on the time axis; and use the local change points as window trigger points to adaptively divide the streaming data window. The divided data window is input into the subsequent calculation or processing module.

3. The method for optimizing the dynamic window of streaming data according to claim 2, characterized in that The potential energy curve is constructed in real time based on the structure and behavioral state of the data itself, and the generation parameters are dynamically adjusted to adapt to changes in the data stream distribution; the judgment of the window trigger point is based on the joint judgment of multiple local change indicators, including: the rate of change of the first-order derivative of potential energy, the fluctuation amplitude in a short period of time, and the difference between consecutive events.

4. The method for optimizing the dynamic window of streaming data according to claim 3, characterized in that After the window trigger point is divided, the window boundary elastic adjustment mechanism is executed for the subsequent out-of-order or delayed data to support dynamic correction or expansion of the window boundary.

5. The method for optimizing the dynamic window of streaming data according to claim 1, characterized in that The entropy threshold-driven lifecycle termination relationship construction method includes: Monitor the real-time data stream in the divided dynamic window; continuously calculate the window activity based on the variability, density, and distribution diversity indicators of the data in the window; when the window activity falls below the preset information entropy threshold, trigger the window life cycle termination operation; The lifecycle termination operation includes at least one of the following: a) compressing the intermediate state information stored in the window to release system resources; b) Close the window in advance and submit the processing results.

6. The method for optimizing the dynamic window of streaming data according to claim 5, characterized in that The calculation basis of the activity includes: the frequency of field value changes in the window, the degree of change in event intervals, the range of change in field distribution and the deduplication rate of events in the window; the information entropy threshold is used to reflect the current information density or novelty of the window. When the indicator is continuously lower than the threshold for more than a continuous period, it is judged that the window has entered the attenuation stage.

7. The method for optimizing the dynamic window of streaming data according to claim 6, characterized in that The state compression operation expresses the state by introducing an approximate statistical structure, including count estimation, frequency compression, summary coding or probability-based data structure.

8. The method for optimizing the dynamic window of streaming data according to claim 7, characterized in that The early release operation is completed before the window reaches its scheduled time or event capacity condition, and is linked with the system resource management module to reclaim the memory and computing resources occupied by the window.

Images