A time series data processing method and system based on dynamic hierarchical clustering and LSTM

Abstract

A time series data processing method and system based on dynamic hierarchical clustering and LSTM, the method comprising: acquiring multi-modal time series data, generating a dynamic tree structure through incremental hierarchical clustering; determining a target node based on the dynamic tree structure and generating a hierarchical position encoding vector of the target node; inputting the hierarchical position encoding vector into an LSTM network to modulate its gating state and outputting an LSTM prediction value; performing double threshold judgment based on the target node and the LSTM prediction value, and generating a structure adjustment instruction to update the dynamic tree structure when the condition is met. Through the cooperation of hierarchical clustering and LSTM, real-time perception and adaptive prediction of time series data pattern evolution are realized.

Description

Technical Field

[0001] This invention relates to the field of time series data analysis and prediction, and in particular to a time series data processing method and system based on dynamic hierarchical clustering and LSTM. Background Technology

[0002] In the field of time series data analysis and prediction, existing technologies have significant systemic limitations when facing complex dynamic environments, mainly in the following aspects:

[0003] Static clustering methods are ill-suited for time-series scenarios: While traditional methods such as hierarchical clustering and K-means can effectively partition high-dimensional data structures and generate rich hierarchical topological information, they face fundamental challenges in time-series data analysis. On the one hand, these methods cannot capture the dynamic features of the data over time. For example, in industrial equipment monitoring, although they can identify "normal / abnormal" operating conditions, they cannot track the gradual evolution of abnormal patterns. On the other hand, the hierarchical semantic information generated by clustering (such as node depth and subtree topology) is only used for initial data partitioning and is not effectively integrated into the subsequent time-series modeling process, leading to a disconnect in the analysis of data evolution patterns.

[0004] Limitations and Adaptability Issues of Deep Learning Models: While Long Short-Term Memory (LSTM) networks excel at handling long-term dependencies, they exhibit significant shortcomings in practical applications. First, their ability to extract correlation features from multimodal heterogeneous data sources (such as device temperature, pressure, and log text) is limited. Second, when data distribution changes drastically, the model struggles to adapt quickly to the new distribution, exhibiting knowledge degradation; experimental results show that its generalization performance may significantly decline.

[0005] The traditional hybrid architecture lacks a collaborative mechanism: the existing "clustering + LSTM" cascade architecture has obvious design flaws. The clustering results are fixed in the early stages of training and cannot be dynamically adjusted according to LSTM prediction errors or changes in data distribution, causing the model to fail rapidly in data drift scenarios. At the same time, the feature fusion layer only uses the discrete category labels or low-dimensional embedding vectors of the clusters, failing to fully utilize the deep correlation information contained in the hierarchical structure, such as inter-cluster distance relationships and node hierarchical features, so that important multimodal coupling features are not effectively mined.

[0006] Performance bottlenecks in real-world applications: In the financial risk control field, traditional hybrid models cannot adaptively adjust cluster boundaries, resulting in a 3-5 working day lag in fraud pattern evolution identification; in industrial predictive maintenance scenarios, the false alarm rate for bearing failures is as high as 35% due to the mixed processing of operating condition data; in edge computing environments, existing solutions suffer from high computational complexity and a lack of collaborative optimization mechanisms, resulting in latency exceeding 500ms when processing large-scale data on resource-constrained devices, failing to meet real-time requirements.

[0007] The core bottleneck in technological development lies in the fact that current optimizations primarily focus on improving individual modules, such as developing more efficient clustering algorithms or LSTM variants, without overcoming the key technological hurdle of deep collaboration between hierarchical clustering and LSTM. Specifically, this manifests in: a lack of online update mechanisms for the hierarchical clustering tree structure, with update strategies disconnected from LSTM prediction performance; a lack of effective interaction channels between hierarchical topological information and the internal state of LSTM; and the continued high computational complexity of existing methods, making it difficult to meet the large-scale time-series processing needs of edge devices. Summary of the Invention

[0008] In view of the above-mentioned shortcomings in current time series data processing, this invention provides a time series data processing method based on dynamic hierarchical clustering and LSTM. Through the synergy of hierarchical clustering and LSTM, real-time perception and adaptive prediction of the evolution of time series data patterns are achieved.

[0009] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0010] A time-series data processing method based on dynamic hierarchical clustering and LSTM, the method comprising:

[0011] Acquire multimodal time-series data and generate a dynamic tree structure through incremental hierarchical clustering;

[0012] The target node is determined based on the dynamic tree structure, and the hierarchical position encoding vector of the target node is generated.

[0013] The hierarchical location encoding vector is input into the LSTM network to modulate its gating state, and the LSTM prediction value is output.

[0014] Based on the target node and the LSTM prediction value, a dual threshold judgment is performed, and a structure adjustment instruction is generated when the condition is met to update the dynamic tree structure.

[0015] According to one aspect of the present invention, the time series data processing method includes: preprocessing the acquired multimodal time series data, specifically including: missing value processing, outlier processing, standardization and encoding, and timestamp alignment.

[0016] According to one aspect of the invention, the incremental hierarchical clustering employs a sliding window mechanism.

[0017] According to one aspect of the invention, the sliding window mechanism uses a fixed-size window to process the data stream, the default size of which is 1000 data points.

[0018] According to one aspect of the present invention, the time-series data processing method is executed in an architecture that coordinates edge devices and cloud servers, wherein,

[0019] The edge device compresses the key information of the generated dynamic tree structure into an 18-byte topology summary data packet and uploads it to the cloud server.

[0020] The cloud server compresses the structural adjustment instructions into a 12-byte structural adjustment instruction data packet and sends it to the edge device.

[0021] According to one aspect of the present invention, the 18-byte topology summary data packet includes: cluster center coordinates, maximum depth, and node density; the 12-byte structure adjustment instruction data packet includes: opcode, target node, and child node list.

[0022] According to one aspect of the present invention, the generation of the hierarchical position encoding vector of the target node specifically involves:

[0023] The formula for calculating the hierarchical position encoding vector is as follows:

[0024]

[0025] in, , is the logarithmic transformation value of the node depth. The transformed depth. Original depth; , which is the normalized sibling node order. According to the order of brothers, It is the eldest sibling in the order of rank; , where is the entropy value of the disorder of the subtree rooted at the target node. The entropy value. This refers to the substructure consisting of all descendant nodes rooted at the current node.

[0026] According to one aspect of the present invention, the step of inputting the hierarchical position encoding vector into the LSTM network to modulate its gating state is specifically implemented as follows:

[0027] In the forget gate computation of LSTM, the hierarchical position encoding vector is... As a modulation term, the forget gate calculation formula is as follows:

[0028]

[0029] in, Traditional forget gate weight matrix, For bias terms, The state was hidden in the previous moment. For the current timing input, Hierarchical position encoding vector, This is the sigmoid function.

[0030] According to one aspect of the present invention, the step of performing a dual threshold judgment based on the target node and the LSTM prediction value, and generating a structure adjustment instruction when the condition is met, specifically comprises:

[0031] Monitor the target node and the LSTM prediction value and calculate their metrics, wherein the metrics include LSTM prediction error and node density change rate;

[0032] A dual threshold judgment is performed based on the LSTM prediction error and the node density change rate.

[0033] Generate corresponding structural adjustment instructions based on the node density change rate.

[0034] A time-series data processing system based on dynamic hierarchical clustering and LSTM, the system comprising:

[0035] Data processing module: acquires multimodal time series data and generates a dynamic tree structure through incremental hierarchical clustering;

[0036] HPE encoding module: Determines the target node based on the dynamic tree structure and generates the hierarchical position encoding vector of the target node;

[0037] Gated fusion module: Inputs the hierarchical position encoding vector into the LSTM network to modulate its gating state and outputs the LSTM prediction value;

[0038] Collaborative control module: Based on the target node and the LSTM prediction value, a dual threshold judgment is performed, and a structure adjustment instruction is generated when the conditions are met to update the dynamic tree structure.

[0039] The advantages of this invention are as follows: First, by using a dynamic tree structure and a dual-threshold triggering mechanism, a closed-loop linkage between the clustering structure and the prediction model is achieved. This enables real-time perception and rapid response to sudden changes in data distribution, effectively solving the recognition lag problem caused by the solidification of clustering in traditional hybrid models. Second, by fusing hierarchical positional encoding vectors with LSTM, the topological semantics of the clustering tree are embedded into the memory unit of the temporal model, achieving a gene-level fusion of structure and temporal sequence. This allows the model to fundamentally understand the intrinsic physical coupling relationship between multimodal features. Finally, based on an architecture and incremental update strategy that combines lightweight edge devices and cloud servers, network transmission volume is effectively compressed and computational complexity is significantly reduced. This allows the entire highly dynamic model to run smoothly on embedded devices with only 180MB of memory, successfully overcoming the resource bottleneck of deploying intelligent models at the edge. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic flowchart of a time-series data processing method based on dynamic hierarchical clustering and LSTM as described in this invention.

[0042] Figure 2 This is a schematic diagram of the structure of a time-series data processing system based on dynamic hierarchical clustering and LSTM as described in this invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] like Figure 1 As shown, a time-series data processing method based on dynamic hierarchical clustering and LSTM is presented. The method specifically includes the following steps:

[0046] Step S1: Obtain multimodal time series data and generate a dynamic tree structure through incremental hierarchical clustering;

[0047] Step S2: Determine the target node based on the dynamic tree structure and generate the hierarchical position encoding vector of the target node;

[0048] Step S3: Input the hierarchical position encoding vector into the LSTM network to modulate its gating state and output the LSTM prediction value;

[0049] Step S4: Based on the target node and the LSTM prediction value, perform a dual threshold judgment, and generate a structure adjustment instruction when the conditions are met to update the dynamic tree structure.

[0050] Further, in step S1, multimodal time series data is acquired; specifically:

[0051] The multimodal time series data is collected by edge devices deployed at the front end of the service. The multimodal time series data includes, but is not limited to:

[0052] Industrial scenarios: Sensor readings for equipment vibration spectrum, temperature, pressure, noise, etc.

[0053] Financial scenarios: real-time transaction logs, price sequences, order book changes, etc.;

[0054] Environmental monitoring scenarios: monitoring data such as temperature, humidity, PM2.5 concentration, and wind speed.

[0055] It should be noted that this embodiment uses an industrial scenario as an example.

[0056] Furthermore, the time-series data processing method includes: preprocessing the acquired multimodal time-series data, specifically including:

[0057] Furthermore, the time-series data processing method includes: preprocessing the acquired multimodal time-series data, including: missing value handling, outlier handling, standardization and encoding, and timestamp alignment. Specifically:

[0058] (1) Missing value handling: For single missing points, linear interpolation or spline interpolation is used for accurate filling; for short-term continuous missing points, such as fewer than 3 consecutive missing points, the mean of the short time segment is used for filling; for long-term continuous missing points, such as more than 10 consecutive missing points, the entire missing data segment is directly removed to avoid introducing too much noise.

[0059] (2) Outlier handling: First, based on statistical principles, using... The criterion, three standard deviations, is used to identify outliers in the data. Subsequently, differentiated processing strategies are adopted based on the business characteristics of different application scenarios: in industrial scenarios, outliers are replaced with the median of the data within the window to maintain trend stability; in financial scenarios, identified outliers are directly removed, and the aforementioned missing value handling method is used for filling.

[0060] (3) Standardization and Encoding: Min-Max normalization is performed on all numerical features to compress their numerical range to the [0, 1] interval. For non-numerical categorical attributes, such as device ID and operating mode, one-hot encoding is used to convert them into numerical vectors.

[0061] (4) Timestamp alignment: The nearest neighbor interpolation method is adopted, and the data of other modalities are aligned to a unified timestamp based on a main time axis to ensure that the data of each modality are synchronized in the time dimension.

[0062] Furthermore, multimodal time-series data is acquired, and a dynamic tree structure is generated using an incremental hierarchical clustering algorithm; specifically:

[0063] The incremental hierarchical clustering employs a sliding window mechanism. This sliding window mechanism processes the data stream using a fixed-size window, with a default window size of 1000 data points. That is, the clustering tree is updated only for the latest 1000 data points.

[0064] Dynamic tree structure construction: Within each sliding window, the Euclidean distance between data points within the window is calculated as a similarity measure. Subsequently, a bottom-up aggregation strategy, such as the AGNES algorithm, is used to iteratively merge the most similar clusters, gradually constructing a hierarchical dynamic tree structure.

[0065] The clustering process is incremental. As new data enters the window and old data is removed, the tree structure is updated locally rather than rebuilt globally, which significantly reduces computational complexity.

[0066] In the final output dynamic tree structure, each node contains a set of core information, including the cluster center coordinates. ), maximum node depth ), node density ( ) and the list of child nodes ( )wait.

[0067] Furthermore, the time-series data processing method is executed in an architecture where edge devices and cloud servers collaborate. At this time, the edge device compresses the key information of the generated dynamic tree structure into an 18-byte topology summary data packet and uploads it to the cloud server. The 18-byte topology summary data packet includes: center coordinates (…). ), maximum node depth ) and node density ( ).

[0068] Further, in step S2, the target node is determined based on the dynamic tree structure, and the hierarchical position encoding vector of the target node is generated; specifically:

[0069] (1) Determine the target node: The target node refers to the leaf node to which the data point currently being processed belongs, or its parent node on the path. The system determines the target node corresponding to each input data point by traversing the dynamic tree structure.

[0070] (2) Calculate the hierarchical position encoding vector:

[0071] For a target node in a dynamic tree structure, its topological features are extracted from the following three dimensions:

[0072] Depth feature extraction: Obtain the depth of the target node in the tree structure, i.e., the path length from the root node to the target node. Then, perform a logarithmic transformation on this depth value:

[0073]

[0074] in, The transformed depth. Original depth;

[0075] Sibling order feature extraction: Determine the ordered sibling order of the target node in the list of its direct child nodes under its parent node. Then, normalize the sibling order.

[0076]

[0077] in, For normalized ordinal position, According to the order of brothers, It is the eldest sibling in the order of rank;

[0078] Subtree disorder feature extraction: Calculate the entropy of the entire subtree rooted at the target node. :

[0079]

[0080] in, The entropy value. This refers to the substructure consisting of all descendant nodes rooted at the current node.

[0081] Vector integration: The three extracted topological features are integrated in a fixed order to form a three-dimensional Hierarchical Position Encoding (HPE) vector. Its form is as follows:

[0082]

[0083] in, It is a differentiable continuous numerical vector.

[0084] In summary, the discrete tree topology is transformed into a differentiable numerical feature vector.

[0085] Further, in step S3, the hierarchical position encoding vector is input into the LSTM network to modulate its gating state, and the LSTM prediction value is output; specifically:

[0086] In the forget gate calculation of LSTM, the HPE vector is introduced as a modulation term. Its calculation formula is as follows:

[0087]

[0088] in, Traditional forget gate weight matrix, For bias terms, The state was hidden in the previous moment. For the current timing input, This is the sigmoid function.

[0089] In summary, this makes the LSTM cell state update process dependent not only on the temporal context ( , It is also affected by the cluster structure of the current data point ( ), Direct regulation of ).

[0090] The fused LSTM network completes cell state updates and output gate calculations according to the standard procedure, and finally the output layer generates the predicted values. .

[0091] Step S4 involves performing a dual threshold judgment based on the target node and the LSTM prediction value, and generating a structure adjustment instruction to update the dynamic tree structure when the conditions are met; specifically, this includes the following steps:

[0092] Step S401: Monitor the target node and the LSTM prediction value and calculate their indices, wherein the indices include the LSTM prediction error and the node density change rate; specifically:

[0093] LSTM prediction error: Calculate the relative error percentage and maintain a fixed-size sliding window (e.g., the last 5 predictions), obtaining the mean error through the arithmetic mean. .

[0094] Node density change rate:

[0095]

[0096] in, The target node density within the current sliding window. This is the density of the previous window.

[0097] Step S402, perform a dual threshold judgment based on the LSTM prediction error and the node density change rate; specifically:

[0098] Set error threshold Density change rate threshold .

[0099] When both conditions are met and When this occurs, a subtree reconstruction for the target node is triggered.

[0100] Step S403: Generate corresponding structural adjustment instructions based on the node density change rate. Specifically:

[0101] when When the value is >0: This indicates data pattern differentiation, and a "split" command is output. The specific strategy for subtree reconstruction is as follows: calculate the KL divergence of the vibration spectrum of all data points under the target node, and split the subsets with divergence greater than 0.3 to form new subtrees;

[0102] when When <0: This indicates data pattern fusion, and the "Merge" command is output. The specific strategy for subtree reconstruction is to find leaf node pairs with a temperature reading similarity greater than 0.85 under the target node and merge them.

[0103] Furthermore, the cloud server compresses the structural adjustment instructions into a 12-byte structural adjustment instruction data packet and sends it to the edge device. The 12-byte structural adjustment instruction data packet includes: (op), the target node (node), and a list of child nodes (child).

[0104] At this point, HPE will immediately recalculate the node encoding vectors based on the new structure, thus affecting the next round of LSTM prediction. This process repeats itself, forming a closed-loop optimization system of structure-aware prediction, performance monitoring triggering, and dynamic structural adjustment, enabling the model to continuously adapt to the evolution of data distribution.

[0105] Example 2

[0106] like Figure 2 As shown, a time-series data processing system based on dynamic hierarchical clustering and LSTM is disclosed. The system includes:

[0107] Data processing module M1: Acquires multimodal time series data and generates a dynamic tree structure through incremental hierarchical clustering;

[0108] HPE encoding module M2: Determines the target node based on the dynamic tree structure and generates the hierarchical position encoding vector of the target node;

[0109] Gated fusion module M3: Inputs the hierarchical position encoding vector into the LSTM network to modulate its gating state and outputs the LSTM prediction value;

[0110] Collaborative control module M4: Based on the target node and the LSTM prediction value, it performs a dual threshold judgment and generates a structure adjustment instruction when the conditions are met to update the dynamic tree structure.

[0111] The advantages of this invention are as follows: First, by using a dynamic tree structure and a dual-threshold triggering mechanism, a closed-loop linkage between the clustering structure and the prediction model is achieved. This enables real-time perception and rapid response to sudden changes in data distribution, effectively solving the recognition lag problem caused by the solidification of clustering in traditional hybrid models. Second, by fusing hierarchical positional encoding vectors with LSTM, the topological semantics of the clustering tree are embedded into the memory unit of the temporal model, achieving a gene-level fusion of structure and temporal sequence. This allows the model to fundamentally understand the intrinsic physical coupling relationship between multimodal features. Finally, based on an architecture and incremental update strategy that combines lightweight edge devices and cloud servers, network transmission volume is effectively compressed and computational complexity is significantly reduced. This allows the entire highly dynamic model to run smoothly on embedded devices with only 180MB of memory, successfully overcoming the resource bottleneck of deploying intelligent models at the edge.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A time-series data processing method based on dynamic hierarchical clustering and LSTM, characterized in that, The time-series data processing method is executed in an architecture that coordinates edge devices and cloud servers, and the method includes: The edge device acquires multimodal time-series data and generates a dynamic tree structure through incremental hierarchical clustering. The key information of the generated dynamic tree structure is compressed into an 18-byte topology summary data packet and uploaded to the cloud server. The target node is determined based on the dynamic tree structure, and a hierarchical position encoding vector of the target node is generated, wherein the formula for calculating the hierarchical position encoding vector is: in, , is the logarithmic transformation value of the node depth. The transformed depth. Original depth; , which is the normalized sibling node order. According to the order of brothers, It is the eldest sibling in the order of rank; , where is the entropy value of the disorder of the subtree rooted at the target node. The substructure consisting of all descendant nodes rooted at the current node; The hierarchical location encoding vector is input into the LSTM network to modulate its gating state, and the LSTM prediction value is output. Based on the target node and the LSTM prediction value, a dual threshold judgment is performed. When the condition is met, a structure adjustment instruction is generated to update the dynamic tree structure. The cloud server compresses the structure adjustment instruction into a 12-byte structure adjustment instruction data packet and sends it to the edge device.

2. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 1, characterized in that, The time series data processing method includes: preprocessing the acquired multimodal time series data, specifically including: missing value processing, outlier processing, standardization and encoding, and timestamp alignment.

3. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 1, characterized in that, The incremental hierarchical clustering adopts a sliding window mechanism.

4. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 3, characterized in that, The sliding window mechanism uses a fixed-size window to process the data stream, with a default window size of 1000 data points.

5. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 1, characterized in that, The 18-byte topology summary data packet includes: cluster center coordinates, maximum depth, and node density; the 12-byte structure adjustment instruction data packet includes: opcode, target node, and child node list.

6. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 1, characterized in that, The specific implementation of inputting the hierarchical position encoding vector into the LSTM network to modulate its gating state is as follows: In the forget gate computation of LSTM, the hierarchical position encoding vector is... As a modulation term, the forget gate calculation formula is as follows: in, Traditional forget gate weight matrix, For bias terms, The state was hidden in the previous moment. For the current timing input, Hierarchical position encoding vector, This is the sigmoid function.

7. The time-series data processing method based on dynamic hierarchical clustering and LSTM according to claim 1, characterized in that, The step of performing a dual threshold judgment based on the target node and the LSTM prediction value, and generating a structure adjustment instruction when the condition is met, specifically involves: Monitor the target node and the LSTM prediction value and calculate their metrics, wherein the metrics include LSTM prediction error and node density change rate; A dual threshold judgment is performed based on the LSTM prediction error and the node density change rate. Generate corresponding structural adjustment instructions based on the node density change rate.

8. A time-series data processing system based on dynamic hierarchical clustering and LSTM, characterized in that, The system is implemented based on the time-series data processing method based on dynamic hierarchical clustering and LSTM as described in any one of claims 1 to 7, and the system includes: Data processing module: acquires multimodal time series data and generates a dynamic tree structure through incremental hierarchical clustering; HPE encoding module: Determines the target node based on the dynamic tree structure and generates the hierarchical position encoding vector of the target node; Gated fusion module: Inputs the hierarchical position encoding vector into the LSTM network to modulate its gating state and outputs the LSTM prediction value; Collaborative control module: Based on the target node and the LSTM prediction value, a dual threshold judgment is performed, and a structure adjustment instruction is generated when the conditions are met to update the dynamic tree structure.

Images