A machine learning based data flow analysis method
By integrating Long Short-Term Memory Networks and Transformer Models, a self-learning time series analysis mechanism is constructed, which solves the adaptability and accuracy problems of existing data flow analysis methods in complex network environments. It realizes accurate modeling and dynamic analysis of multi-source data flow, and improves prediction accuracy and anomaly detection capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUYANG GUJIU WEILAN TECHNOLOGY CO LTD
- Filing Date
- 2025-11-17
- Publication Date
- 2026-05-12
AI Technical Summary
Existing data flow analysis methods struggle to effectively capture data interaction characteristics across multiple nodes, protocols, and time windows in complex and dynamic network environments. They also lack self-learning capabilities, resulting in poor model adaptability, decreased prediction accuracy, and delayed anomaly detection.
By integrating Long Short-Term Memory Networks and Transformer Models, a self-learning temporal analysis mechanism is constructed. Through feature extraction, fusion, and self-learning modules, dynamic parameter adjustment and continuous optimization are achieved. Combined with dynamic attention weight calculation and synchronous calculation control, a multi-layer feature fusion and self-learning closed-loop structure is established.
It enables accurate modeling and dynamic analysis of complex data flow behavior in multi-source systems, improves the model's adaptability, prediction accuracy and anomaly identification capabilities, has real-time response capabilities, and is adaptable to data flow monitoring and intelligent decision-making in complex network environments.
Smart Images
Figure CN121479276B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data analysis technology, and in particular to a data flow analysis method based on machine learning. Background Technology
[0002] Existing data flow analysis methods are mostly based on static statistical features or single deep learning models, such as traditional support vector machines, random forests, or methods that only use long short-term memory networks. Although these methods can identify data flow patterns to a certain extent, they are difficult to effectively capture the data interaction features under multiple nodes, multiple protocols, and multiple time windows, resulting in poor model adaptability to complex dynamic network environments. At the same time, although some global modeling methods based on Transformers have strong feature representation capabilities, they lack coupling mechanisms with time-dependent features and are difficult to reflect the continuous change patterns of data flow.
[0003] Furthermore, existing methods typically keep parameters static after model training, lacking the ability to learn independently of prediction biases or anomalous samples. They cannot dynamically correct and continuously optimize model parameters during operation, which can easily lead to decreased prediction accuracy and delayed anomaly detection. Existing technologies generally suffer from problems such as single feature extraction levels, fixed feature fusion methods, and insufficient model adaptability, which limit their performance in real-time analysis and prediction of data flow in complex network environments. Summary of the Invention
[0004] One objective of this invention is to propose a data flow analysis method based on machine learning. This invention integrates long short-term memory networks and the Transformer model to construct a self-learning time series analysis mechanism, thereby realizing intelligent modeling and anomaly prediction of data flow. It has the advantages of strong adaptability, high prediction accuracy, and excellent real-time performance.
[0005] A data flow analysis method based on machine learning according to an embodiment of the present invention includes the following steps:
[0006] Data flow information is collected from multi-source systems to generate data flow feature sequences;
[0007] A data flow time series analysis model is constructed, which includes: an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer.
[0008] The data flow feature sequence is input into the model, and the feature extraction layer performs temporal and global correlation feature extraction on the data flow feature sequence to generate a data flow temporal embedding vector.
[0009] The data flow temporal embedding vector is weighted and fused using a feature fusion layer. The importance of features is assigned through a dynamic attention weight calculation mechanism to generate a data flow fusion representation vector.
[0010] The output layer performs nonlinear mapping calculations on the data flow fusion representation vector to obtain data flow path information, node interaction intensity information, prediction deviation information and anomaly probability information, and marks abnormal samples when the anomaly probability information exceeds the threshold.
[0011] When the prediction deviation information or anomaly probability information exceeds the preset threshold, the corresponding abnormal samples, data flow path information, node interaction strength information and prediction deviation information are input into the model self-learning module, and the model parameters are dynamically adjusted through the self-learning feedback mechanism.
[0012] Based on the updated model, real-time data flow information is continuously analyzed, and the analysis results include data flow status information, data flow prediction results information, and data flow anomaly risk level information.
[0013] Optionally, the generation of the data flow feature sequence specifically includes:
[0014] Collect data flow information;
[0015] The collected data flow information is denoised.
[0016] Perform missing data completion on the denoised data flow information;
[0017] The completed data flow information is then processed to ensure a uniform format.
[0018] Normalize the data flow information after it has been formatted and standardized.
[0019] The data flow information, after being processed by denoising, missing data completion, format unification, and normalization, is organized and encoded in chronological order to generate a data flow feature sequence.
[0020] Optionally, the construction of the data flow time series analysis model specifically includes:
[0021] The basic framework of the feature extraction layer is established, and the Long Short-Term Memory network module and the Transformer network module are defined as two independent feature sub-modules.
[0022] The feature extraction layer includes a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computing control node;
[0023] A feature alignment layer is set between the output of the Long Short-Term Memory Network module and the input of the Dynamic Gated Interaction Unit. The feature alignment layer is used to perform dimension matching, channel mapping and feature standardization on the time-dependent feature values of the Long Short-Term Memory Network module output to generate aligned feature values.
[0024] A dynamic gating interaction unit is constructed between the feature alignment layer and the Transformer network module;
[0025] A synchronous computation control node is set in the feature extraction layer. The synchronous computation control node is used to coordinate the computation step size and parameter synchronization relationship between the long short-term memory network module and the Transformer network module.
[0026] The feature extraction layer is generated based on a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computation control node.
[0027] A feature fusion layer is set after the feature extraction layer. The feature fusion layer includes a feature grouping unit, a dynamic attention weight calculation unit, and a normalization mapping unit.
[0028] A model self-learning module is set after the feature fusion layer. The model self-learning module includes a feedback data receiving unit, a difference calculation unit, and a parameter update unit.
[0029] A data flow time series analysis model is constructed based on an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer.
[0030] Optionally, the construction of the dynamic gating interaction unit specifically includes:
[0031] In the basic framework of the feature extraction layer, a bidirectional feature transfer and parameter co-evolution interaction channel is established. The interaction channel connects the hidden node layer of the long short-term memory network module and the attention calculation layer of the Transformer network module, and achieves this through the interdependence of feature weights and gating states.
[0032] In the interaction channel, a weight mapping layer, a gating adjustment layer, and a feedback interface layer are set sequentially.
[0033] The weight mapping layer calculates the correlation coefficient between the attention gradient direction and the gating gradient direction based on the global feature weights from the Transformer network module and the current input gate and forget gate states of the Long Short-Term Memory network module, and generates a parameter coordination signal.
[0034] The gating adjustment layer synchronously and iteratively adjusts the input gate state value and forget gate state value of the long short-term memory network module and the attention matrix parameters of the Transformer network module according to the parameter coordination signal, so that the weight updates of the two modules achieve directional consistency and amplitude mutual constraint within the same training step, and generate output results.
[0035] The feedback interface layer maps the output of the gating adjustment layer to a new attention weighting matrix and gating state response. In each iteration, it simultaneously updates the global attention distribution of the Transformer network and the temporal memory parameters of the Long Short-Term Memory network, realizing a dynamic co-evolution process driven by bidirectional dependency, and generating dynamic gating interaction units with parameter adaptation and feature collaboration.
[0036] Optionally, the generation of the data flow time-series embedding vector specifically includes:
[0037] Input the data flow characteristic sequence into the input layer of the data flow time series analysis model;
[0038] The Long Short-Term Memory (LSTM) network module of the feature extraction layer extracts the time dependency of the input data flow feature sequence, generating a data flow time-series intermediate vector containing the temporal change features of nodes;
[0039] The feature alignment layer performs dimension matching and feature standardization on the intermediate vectors of the data flow time series to generate aligned time series features.
[0040] The dynamic gating interaction unit performs bidirectional interactive processing on the aligned temporal features and outputs data flow interaction feature values adjusted by gating feedback.
[0041] The synchronous computation control node performs synchronous control on the computation step size, parameter update and gradient direction of the data flow interaction feature value, and outputs the synchronously optimized data flow feature result;
[0042] The Transformer network module performs multi-dimensional global feature association calculations on the data flow characteristic results and outputs multi-dimensional global representation results.
[0043] The output layer performs a unified mapping on the global representation results and outputs a temporal embedding vector of the data flow.
[0044] Optionally, the generation of the data flow fusion representation vector specifically includes:
[0045] The feature fusion layer groups the data flow temporal embedding vector according to node identification information, communication protocol information and time window information to generate multi-source feature grouping vectors;
[0046] The dynamic attention weight calculation unit of the feature fusion layer performs weighted fusion of multi-source feature group vectors, adaptively adjusts the group weight allocation according to the temporal dynamic changes of the input features, and outputs the weighted fusion feature result;
[0047] The attention weight calculation unit performs weight calculation on each feature group in the multi-source feature grouping vector. Based on the node interaction frequency corresponding to the node identification information, the protocol layer feature weight corresponding to the communication protocol information, and the temporal correlation factor corresponding to the time window information, it performs dependency calculation on the input feature value inside each feature group to generate a group attention weight value that is adaptively updated with feature input and temporal changes.
[0048] The weighted fusion feature results are input into the normalized mapping unit of the feature fusion layer. The fusion results are then subjected to feature standardization and multi-source scale alignment to generate a data flow fusion representation vector.
[0049] Optionally, the generation of the abnormal samples specifically includes:
[0050] The data flow fusion representation vector is input into the output layer of the data flow time series analysis model;
[0051] In the output layer, feature mapping is calculated on the data flow fusion representation vector, and a prediction output is generated based on the correlation results of the time dimension and the node dimension.
[0052] The data flow path information is calculated based on the predicted output, and the node interaction strength information is also calculated.
[0053] The predicted output is compared with the actual observed data to calculate the prediction deviation information and anomaly probability information;
[0054] When the anomaly probability exceeds a preset threshold, the data flow feature segment at the corresponding time step is marked as an anomaly sample.
[0055] Optionally, the dynamic adjustment specifically includes:
[0056] When the detected anomaly probability information exceeds the preset anomaly threshold or the prediction deviation information exceeds the preset deviation threshold, the model self-learning module is triggered to extract the corresponding anomaly samples, data flow path information, node interaction intensity information and prediction deviation information to generate an anomaly feedback dataset.
[0057] The abnormal feedback dataset is input into the model self-learning module. The model self-learning module calculates the difference information between the model output and the target expected value based on the feedback dataset and generates the model parameter adjustment factor.
[0058] The weight parameters, learning rate, and gating parameters of the data flow time series analysis model are dynamically updated based on the model parameter adjustment factor to generate new model parameter state values.
[0059] The dynamic update specifically includes: the model self-learning module receives the model parameter adjustment factor and the existing model parameter state values; iteratively updates the weight parameters of the data flow time series analysis model according to the model parameter adjustment factor to generate new weight parameter values; adjusts the model's learning rate parameter according to the model parameter adjustment factor to obtain updated learning rate parameter values; corrects the gating parameters corresponding to the input gate state, forget gate state, and output gate state in the long short-term memory network module according to the model parameter adjustment factor to obtain updated gating parameter values; during the parameter update process, a temporal consistency constraint is introduced to dynamically align the time step sequence of the abnormal feedback dataset with the temporal weight distribution within the model; and a parameter linkage update mechanism is used to achieve dynamic coupling between multi-dimensional parameters, forming an adaptive fusion structure of feedback learning and temporal modeling, generating new model parameter state values, which include updated weight parameter values, learning rate parameter values, and gating parameter values, thus completing the model's self-learning and adaptive update.
[0060] Optionally, the generation of the analysis results specifically includes:
[0061] The real-time data flow information is input into the updated data flow time series analysis model for continuous analysis.
[0062] Based on the analysis results, output data flow status information, data flow prediction results information, and data flow anomaly risk level information to generate analysis results.
[0063] The beneficial effects of this invention are:
[0064] This invention constructs a machine learning-based data flow temporal analysis model, achieving accurate modeling and dynamic analysis of complex data flow behavior in multi-source systems. The model's structural design integrates the advantages of Long Short-Term Memory (LSTM) networks and Transformer networks, establishing a bidirectional feature transfer and parameter collaborative update channel between them through a dynamic gating interaction unit. This allows the fusion of temporal memory features and global attention features, effectively solving the problems of local feature distortion and insufficient global dependency in traditional models when processing non-stationary multi-source data. Simultaneously, a synchronous computation control node ensures consistent control of step size, parameters, and gradients between the LSTM and Transformer modules, guaranteeing temporal coordination of feature extraction and parameter updates across multiple modules, making the feature extraction process more stable and efficient.
[0065] This invention designs a dynamic attention weight calculation unit in the feature fusion stage, which can dynamically allocate feature importance based on multi-dimensional information such as node identifiers, communication protocols, and time windows, realizing adaptive weighted fusion of multi-source features. This significantly improves the model's ability to identify and express complex data flow patterns. In addition, a model self-learning module is introduced in the model training and running stages. When a prediction deviation or anomaly probability exceeds a threshold, it can automatically generate model parameter adjustment factors based on abnormal samples, node interaction strength, and path information, and update the weight parameters, learning rate, and gating parameters in a linked manner. This design constructs a self-learning closed-loop structure with temporal consistency constraints and parameter linkage control mechanisms, enabling the model to have continuous evolution capabilities and dynamic adaptive performance, and maintain prediction accuracy and convergence stability under changes in data distribution or environmental disturbances.
[0066] In summary, this invention achieves deep correlation modeling and intelligent prediction of data flow through multi-layer feature fusion, dynamic gating interaction, and self-learning closed-loop optimization. It has technical advantages such as strong adaptability, high accuracy of anomaly identification, excellent model stability, and outstanding real-time response capability, effectively improving the level of data flow monitoring and intelligent decision-making in complex network environments. Attached Figure Description
[0067] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0068] Figure 1 This is a flowchart of a data flow analysis method based on machine learning proposed in this invention;
[0069] Figure 2 This is a schematic diagram of the interaction structure between the Long Short-Term Memory network module and the Transformer network module in the feature extraction layer of the data flow analysis method based on machine learning proposed in this invention. Detailed Implementation
[0070] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0071] refer to Figure 1 and Figure 2 A data flow method based on machine learning includes the following steps:
[0072] Data flow information is collected from multi-source systems to generate data flow feature sequences;
[0073] A data flow time series analysis model is constructed, which includes: an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer.
[0074] The data flow feature sequence is input into the model, and the feature extraction layer performs temporal and global correlation feature extraction on the data flow feature sequence to generate a data flow temporal embedding vector.
[0075] The data flow temporal embedding vector is weighted and fused using a feature fusion layer. The importance of features is assigned through a dynamic attention weight calculation mechanism to generate a data flow fusion representation vector.
[0076] The output layer performs nonlinear mapping calculations on the data flow fusion representation vector to obtain data flow path information, node interaction intensity information, prediction deviation information and anomaly probability information, and marks abnormal samples when the anomaly probability information exceeds the threshold.
[0077] When the prediction deviation information or anomaly probability information exceeds the preset threshold, the corresponding abnormal samples, data flow path information, node interaction strength information and prediction deviation information are input into the model self-learning module, and the model parameters are dynamically adjusted through the self-learning feedback mechanism.
[0078] Based on the updated model, real-time data flow information is continuously analyzed, and the analysis results include data flow status information, data flow prediction results information, and data flow anomaly risk level information.
[0079] In this embodiment, the generation of the data flow feature sequence specifically includes:
[0080] Collect data flow information, which includes node identification information, timestamp information, data traffic information, communication protocol information, payload characteristic information, and path marking information;
[0081] The collected data flow information is denoised to eliminate abnormal data caused by network jitter, sampling error or equipment noise, so as to maintain the continuity and stability of the data flow information.
[0082] The data flow information after noise reduction is filled with missing information, such as node identification information, timestamp information, data traffic information, communication protocol information, payload characteristic information or path marking information, according to the similar characteristics of the same type of data, so as to maintain the integrity of the data flow information.
[0083] The completed data flow information is processed to unify the format, and data flow information from different sources, with different sampling frequencies and different encoding formats is converted according to a unified data structure and time order;
[0084] Normalize the data flow information after it has been formatted and mapped all feature dimensions to the same numerical range.
[0085] The data flow information, after being processed by denoising, missing data completion, format unification, and normalization, is organized and encoded in chronological order to generate a data flow feature sequence.
[0086] In this embodiment, the construction of the data flow time series analysis model specifically includes:
[0087] The basic framework of the feature extraction layer is established, and the Long Short-Term Memory network module and the Transformer network module are defined as two independent feature sub-modules.
[0088] The feature extraction layer includes a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computing control node;
[0089] The Long Short-Term Memory (LSTM) network module includes an input node layer, a hidden node layer, and an output node layer. The hidden node layer contains an input gate, a forget gate, and an output gate, which are used to realize the state memory and update control of the input features. The Transformer network module includes an encoder stack layer, an attention calculation layer, and a feature mapping layer. The encoder stack layer is used to realize multi-layer feature representation, the attention calculation layer is used to calculate global feature association, and the feature mapping layer is used to generate the attention weight matrix.
[0090] A feature alignment layer is set between the output of the Long Short-Term Memory Network module and the input of the dynamic gating interaction unit. The feature alignment layer is used to perform dimension matching, channel mapping and feature standardization on the time-dependent feature values output by the Long Short-Term Memory Network module to generate aligned feature values, providing structured input for subsequent feature interaction.
[0091] A dynamic gating interaction unit is constructed between the feature alignment layer and the Transformer network module. The dynamic gating interaction unit includes a weight mapping layer, a gating adjustment layer and a feedback interface layer. It is used to realize bidirectional feature interaction and parameter collaborative update between the long short-term memory network module and the Transformer network module, so as to realize the adaptive fusion of temporal memory features and global attention features during the feature extraction process, thereby improving the model's ability to express the temporal correlation of multi-source data flow features.
[0092] A synchronous computation control node is set in the feature extraction layer. The synchronous computation control node is used to coordinate the computation step size and parameter synchronization relationship between the long short-term memory network module and the Transformer network module.
[0093] The construction of the synchronous computation control node specifically includes: establishing a structural framework for the synchronous computation control node in the logical control layer of the feature extraction layer. The synchronous computation control node includes a step size adjustment unit, a parameter synchronization unit, and a gradient coordination unit. The step size adjustment unit is connected to the training control interface of the Long Short-Term Memory network module and the Transformer network module, and defines a unified computation step size register structure and a synchronous trigger clock structure within the node. The parameter synchronization unit is connected to the dynamic gating interaction unit and the weight storage interface of the two network modules, and establishes a parameter mapping table and a shared register channel within the node to form parameter correspondences between multiple modules. The gradient coordination unit is connected to the backpropagation interface of the Long Short-Term Memory network module and the Transformer network module, sets up a gradient buffer and a direction index table within the node, and establishes a data path with the parameter synchronization unit to form a unified gradient propagation path in the node structure. The step size adjustment unit, the parameter synchronization unit, and the gradient coordination unit are interconnected through an internal signal bus to form a comprehensive control structure with synchronous computation logic, parameter consistency logic, and gradient coordination logic, generating an integrated logic unit for the synchronous computation control node.
[0094] The feature extraction layer is generated based on a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computation control node.
[0095] A feature fusion layer is set after the feature extraction layer. The feature fusion layer includes a feature grouping unit, a dynamic attention weight calculation unit, and a normalization mapping unit.
[0096] The feature grouping unit is used to divide features according to node identification information, communication protocol information and time window information based on the input temporal feature results; the dynamic attention weight calculation unit is used to define the weight calculation path and attention mapping relationship of each feature group during the model building stage; the normalization mapping unit is used to define the mapping between the output dimension of the feature fusion result and the input dimension of the model self-learning module in order to achieve structural alignment of the internal hierarchy of the model.
[0097] A model self-learning module is set after the feature fusion layer. The model self-learning module includes a feedback data receiving unit, a difference calculation unit, and a parameter update unit.
[0098] The feedback data receiving unit is used to receive abnormal sample information, prediction bias information, and node interaction strength information from the model output layer; the difference calculation unit is used to calculate the difference value between the model prediction result and the expected target, and generate parameter adjustment signals; the parameter update unit is used to establish the mapping relationship between the parameter adjustment signals and the model weight parameters, learning rate, and gating parameters during the model construction stage, so as to realize the updatable structure definition of model parameters.
[0099] A data flow time series analysis model is constructed based on an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer.
[0100] In this embodiment, the construction of the dynamic gating interaction unit specifically includes:
[0101] In the basic framework of the feature extraction layer, a bidirectional feature transfer and parameter co-evolution interaction channel is established. The interaction channel connects the hidden node layer of the long short-term memory network module and the attention calculation layer of the Transformer network module. Through the interdependence of feature weights and gating states, a parameter coupling path that can be updated synchronously is formed.
[0102] In the interactive channel, a weight mapping layer, a gating adjustment layer, and a feedback interface layer are set sequentially. The weight mapping layer receives global feature weights from the Transformer network module and, in conjunction with the current input gate and forget gate states of the Long Short-Term Memory network module, calculates the correlation coefficient between the attention gradient direction and the gating gradient direction to generate a parameter coordination signal.
[0103] The gating adjustment layer receives parameter coordination signals and synchronously iteratively adjusts the input gate state values and forget gate state values of the long short-term memory network module and the attention matrix parameters of the Transformer network module. This enables the weight updates of the two modules to achieve directional consistency and amplitude mutual constraint within the same training step, forming an adaptive collaborative optimization mechanism with temporal memory strength and global dependency as joint objectives.
[0104] The feedback interface layer receives the output of the gating adjustment layer and maps it to a new attention weighting matrix and gating state response. In each iteration, it simultaneously updates the global attention distribution of the Transformer network and the temporal memory parameters of the Long Short-Term Memory network, realizing a dynamic co-evolution process driven by bidirectional dependency, and generating dynamic gating interaction units with parameter adaptation and feature collaboration.
[0105] In this embodiment, the generation of the data flow time-series embedding vector specifically includes:
[0106] Input the data flow characteristic sequence into the input layer of the data flow time series analysis model;
[0107] The Long Short-Term Memory (LSTM) network module of the feature extraction layer extracts the time dependency of the input data flow feature sequence, generating a data flow time-series intermediate vector containing the temporal change features of nodes;
[0108] The feature alignment layer performs dimension matching and feature standardization on the intermediate vectors of the data flow time series to generate aligned time series features.
[0109] The dynamic gating interaction unit performs bidirectional interactive processing on the aligned temporal features and outputs data flow interaction feature values adjusted by gating feedback.
[0110] The synchronous computation control node performs synchronous control on the computation step size, parameter update and gradient direction of the data flow interaction feature value, and outputs the synchronously optimized data flow feature result;
[0111] The Transformer network module performs multi-dimensional global feature association calculations on the data flow characteristic results and outputs multi-dimensional global representation results.
[0112] The output layer performs a unified mapping on the global representation results and outputs a temporal embedding vector of the data flow.
[0113] In this embodiment, the generation of the data flow fusion representation vector specifically includes:
[0114] The feature fusion layer groups the data flow temporal embedding vector according to node identification information, communication protocol information and time window information to generate multi-source feature grouping vectors;
[0115] The dynamic attention weight calculation unit of the feature fusion layer performs weighted fusion of multi-source feature group vectors, adaptively adjusts the group weight allocation according to the temporal dynamic changes of the input features, and outputs the weighted fusion feature result;
[0116] The attention weight calculation unit performs weight calculation on each feature group in the multi-source feature grouping vector. Based on the node interaction frequency corresponding to the node identification information, the protocol layer feature weight corresponding to the communication protocol information, and the temporal correlation factor corresponding to the time window information, it performs dependency calculation on the input feature value inside each feature group to generate a group attention weight value that is adaptively updated with feature input and temporal changes.
[0117] The weighted fusion feature results are input into the normalized mapping unit of the feature fusion layer. The fusion results are then subjected to feature standardization and multi-source scale alignment to generate a data flow fusion representation vector.
[0118] In this embodiment, the generation of the abnormal samples specifically includes:
[0119] The data flow fusion representation vector is input to the output layer of the data flow time series analysis model, and the output layer is used to perform nonlinear mapping calculation on the input data flow fusion representation vector.
[0120] In the output layer, feature mapping is calculated on the data flow fusion representation vector, and a prediction output is generated based on the correlation results of the time dimension and the node dimension.
[0121] The data flow path information is calculated based on the predicted output to characterize the data flow relationship between each node. At the same time, the node interaction strength information is calculated to describe the communication association strength between nodes.
[0122] The calculation of the data flow path information specifically includes:
[0123] The predicted outputs are grouped according to node identification information to generate a node prediction feature set. Feature association analysis is performed on the predicted output values between any two nodes in the node prediction feature set to obtain the node association degree value. Based on the node association degree value and the adjacency relationship in the time series, a prediction association matrix between nodes is constructed. The prediction association matrix is normalized to generate normalized association weights between nodes. The dominant transmission direction between nodes is determined based on the normalized association weights to generate an initial prediction path set. The initial prediction path set is filtered for path continuity and weight threshold to obtain a data flow path sequence that conforms to the time series characteristics, and the data flow path information is output.
[0124] The calculation of the node interaction strength information specifically includes:
[0125] The predicted output is matched with node identification information and communication protocol information to generate a node interaction feature set. Based on the predicted output change magnitude and protocol layer feature weights for each node in the node interaction feature set, the node interaction response value is calculated. The node interaction response values are then aggregated over a time window to obtain the average node interaction feature over time. This average node interaction feature is then weighted and fused with the node communication stability index to generate a node interaction strength value. Finally, the node interaction strength value is standardized and sorted to output the node interaction strength information.
[0126] The predicted output is compared with the actual observed data to calculate the prediction deviation information and the anomaly probability information. The prediction deviation information is used to characterize the degree of deviation between the model output and the real data, and the anomaly probability information is used to describe the probability of anomalies occurring in the data flow in the current time series.
[0127] When the anomaly probability exceeds a preset threshold, the data flow feature segment at the corresponding time step is marked as an anomaly sample.
[0128] In this embodiment, the dynamic adjustment specifically includes:
[0129] When the detected anomaly probability information exceeds the preset anomaly threshold or the prediction deviation information exceeds the preset deviation threshold, the model self-learning module is triggered to extract the corresponding anomaly samples, data flow path information, node interaction intensity information and prediction deviation information to generate an anomaly feedback dataset.
[0130] The abnormal feedback dataset is input into the model self-learning module. The model self-learning module calculates the difference information between the model output and the target expected value based on the feedback dataset and generates the model parameter adjustment factor.
[0131] The generation of the model parameter adjustment factor specifically includes: the model self-learning module receiving the abnormal feedback dataset, calculating the prediction residual information based on the prediction deviation information corresponding to the abnormal samples; calculating the node importance coefficient value based on the node interaction strength information between the prediction residual information and the abnormal samples; calculating the path correlation weight result based on the data flow path information and the node importance coefficient value; performing correlation mapping on the prediction residual information, node importance coefficient value, and path correlation weight result to generate the parameter offset response value; calculating the self-learning feedback gain value based on the joint comparison between the parameter offset response value and the prediction deviation information corresponding to the abnormal samples; and generating the model parameter adjustment factor based on the self-learning feedback gain value.
[0132] The weight parameters, learning rate, and gating parameters of the data flow time series analysis model are dynamically updated based on the model parameter adjustment factor to generate new model parameter state values.
[0133] The dynamic update specifically includes: the model self-learning module receives the model parameter adjustment factor and the existing model parameter state values; iteratively updates the weight parameters of the data flow time series analysis model according to the model parameter adjustment factor to generate new weight parameter values; adjusts the model's learning rate parameter according to the model parameter adjustment factor to obtain updated learning rate parameter values; corrects the gating parameters corresponding to the input gate state, forget gate state, and output gate state in the long short-term memory network module according to the model parameter adjustment factor to obtain updated gating parameter values; during the parameter update process, a temporal consistency constraint is introduced to dynamically align the time step sequence of the abnormal feedback dataset with the temporal weight distribution within the model; and a parameter linkage update mechanism is used to achieve dynamic coupling between multi-dimensional parameters, forming an adaptive fusion structure of feedback learning and temporal modeling, generating new model parameter state values, which include updated weight parameter values, learning rate parameter values, and gating parameter values, thus completing the model's self-learning and adaptive update.
[0134] In this embodiment, the generation of the analysis results specifically includes:
[0135] The real-time data flow information is input into the updated data flow time series analysis model for continuous analysis.
[0136] Based on the analysis results, output data flow status information, data flow prediction results information, and data flow anomaly risk level information to generate analysis results.
[0137] Example 1:
[0138] To verify the feasibility of this invention in practice, it was applied to a multi-source data flow monitoring and intelligent analysis system of a large industrial internet platform. This system has a complex network environment with diverse node types, including heterogeneous data flows from the production control layer, equipment communication layer, and cloud scheduling layer. Traditional monitoring schemes mainly rely on a single time series prediction algorithm to analyze equipment operation data. However, when faced with a large number of nodes, inconsistent protocol standards, and significant differences in communication frequencies, the model often struggles to maintain global feature consistency, resulting in fluctuating prediction results and low anomaly detection rates. Especially under high-concurrency data flow conditions, a single model cannot simultaneously consider both time-dependent features and global correlation features, leading to a delayed response to potential abnormal states and difficulty in timely detection of communication anomalies, data congestion, or potential link risks. The machine learning-based data flow analysis method proposed in this invention aims to achieve real-time modeling and dynamic prediction of complex data flows by constructing a multi-level feature extraction and self-learning feedback mechanism, effectively improving the visualization and anomaly detection capabilities of data flows in multi-source network environments.
[0139] In the specific implementation process, the system first acquires data flow information from different communication layers through a multi-source acquisition module. The data sources include real-time operation data uploaded by field control equipment, communication status logs of edge nodes, and task scheduling records generated by the cloud scheduling platform. Before entering the analysis model, all data undergoes denoising, missing data completion, format unification, and normalization preprocessing to ensure that data from different sources, formats, and sampling frequencies can be mapped to a unified feature space, forming a feature sequence with comparability and temporal continuity. This processing provides clean and structured input for subsequent model training and online inference.
[0140] After data preprocessing, the proposed data flow time series analysis model is constructed. The model consists of an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer. The feature extraction layer is the key part for achieving innovative breakthroughs. This layer introduces a bidirectional interactive structure between a Long Short-Term Memory (LSTM) network and a Transformer network, and uses a dynamic gating interaction unit to establish a feature transfer and parameter co-evolution channel between the two. The LSM network is responsible for capturing the time-dependent features of the data flow, while the Transformer network is responsible for establishing global feature associations across nodes and time periods. The two networks achieve synchronous parameter adjustment through a gating feedback mechanism, enabling the model to simultaneously optimize local temporal features and global dependency features during training. Compared with traditional serial hybrid networks, this dynamic interaction mechanism can automatically adjust memory strength and attention distribution according to changes in input features, thereby achieving adaptive modeling of non-stationary multi-source data.
[0141] Following the feature extraction layer, the feature fusion layer is used to perform multi-dimensional fusion of feature embedding results from different nodes, different communication protocol layers, and different time windows. In this stage, the present invention introduces a dynamic attention weight calculation unit. By comprehensively considering the correlation between node identification, communication frequency, and time window, the importance of each feature group is dynamically allocated, thereby realizing the weighted fusion of multi-source information. This dynamic attention mechanism enables the model to automatically adjust the fusion strategy when the distribution of input data changes, and has an adaptive response capability to changes in the interaction frequency between nodes. It significantly enhances the model's accuracy in characterizing global temporal correlations. In actual operation, this mechanism enables the model to automatically identify key node features that have a significant impact on the overall data flow, avoiding the insensitivity problem of fixed-weight models to data distribution drift.
[0142] The result after feature fusion is input to the output layer for calculation. The output layer comprehensively utilizes nonlinear mapping and feature regression calculation to obtain data flow path information, node interaction strength information, prediction deviation information, and anomaly probability information. Through this structure, the system can perform trend prediction and anomaly analysis on each data flow path and automatically identify nodes that may experience communication delays, traffic mutations, or data packet loss. When the anomaly probability information generated by the output layer exceeds a preset threshold, the system will automatically mark the feature segment of the corresponding time step as an anomaly sample and trigger the model self-learning module.
[0143] The introduction of the model self-learning module is a crucial step in achieving intelligent evolution in this invention. When the model detects prediction bias or anomaly probability exceeding limits, the self-learning module automatically collects abnormal samples, data flow path information, and node interaction strength information to form an anomaly feedback dataset. It then calculates a model parameter adjustment factor, which is generated through the correlation mapping of residual information, node importance, and path association weights. This factor accurately reflects the parameter offset trend of the model in a specific time period. Subsequently, the model self-learning module dynamically corrects the model's weight parameters, learning rate, and gating parameters based on this adjustment factor, achieving adaptive model updates. This mechanism effectively avoids the problem of traditional models requiring manual retraining after changes in the data environment, enabling the model to automatically optimize performance during continuous operation and maintain long-term stable prediction accuracy.
[0144] When this invention is applied to the actual environment of the industrial internet platform, the system completes multiple rounds of self-learning iterations within a continuous operating cycle. The model can automatically update parameters when the data distribution changes. The application results show that the stability of the model in handling high concurrency and heterogeneous data streams is significantly improved, the response speed to data flow anomalies is significantly accelerated, and the overall prediction error is effectively suppressed. Compared with the original analysis method based on a single time series model, the solution of this invention can more accurately characterize the interaction relationship between multiple nodes, more timely identify potential abnormal flow risks, and form a self-calibrating learning loop within the model.
[0145] In summary, this invention achieves temporal modeling and adaptive learning of multi-source data streams in complex network environments. Through dynamic gating interaction structures, dynamic attention fusion mechanisms, and self-learning feedback optimization, the model possesses stronger generalization and self-evolution capabilities. This method maintains high-efficiency prediction and anomaly detection performance under different load states, communication modes, and data sampling periods without manual intervention, providing a reliable, intelligent, and sustainable optimization solution for data flow monitoring, intelligent early warning, and resource scheduling in complex systems.
[0146] Table 1. Performance Comparison of the Invention Method and Traditional Models in Multi-Source Data Stream Analysis Tasks
[0147] Performance indicators LSTM LSTM+Transformer Method of the present invention Average prediction error 0.218 0.162 0.089 Mean Absolute Error 0.187 0.134 0.071 Anomaly detection accuracy 83.5% 89.2% 95.6% Anomaly detection recall rate 78.9% 86.3% 94.1% F1 Overall Score 0.81 0.88 0.95 Model adaptive iteration time (single round) 12.5s 15.7s 9.8s Number of parameter convergence steps (number of iterations) 460 380 240 Long-term stability of the model (error drift rate after 24 hours of continuous operation) 12.8% 7.4% 2.3%
[0148] As shown in Table 1, the data flow analysis method based on machine learning proposed in this invention outperforms traditional solutions in all core performance indicators, especially in prediction accuracy and anomaly detection. Traditional LSTM time series prediction models can only capture local time dependencies, with an average prediction error of 0.218. However, this invention introduces a dynamic gating interaction mechanism and a global attention feature fusion structure, enabling the model to capture both time series changes and global dependency features simultaneously, thereby significantly reducing the prediction error to 0.089, a reduction of more than 59%. This result demonstrates that the model of this invention has stronger cross-node feature representation capabilities and complex data flow trend learning capabilities.
[0149] In terms of anomaly detection performance, the method of this invention achieves an accuracy of 95.6% and a recall of 94.1%, both significantly higher than traditional models. The key reason for this performance improvement is that, through the dynamic attention weight calculation unit, the model can adjust the feature weight distribution under different nodes and different time windows in real time, so that the model can still maintain a high sensitivity to anomaly recognition when the data flow characteristics change. At the same time, the model introduces dynamic calculation logic of anomaly probability information in the output layer, so that the detection results can better reflect the fluctuation trend of the system's operating state and reduce false alarms and false negatives.
[0150] In terms of model training efficiency and stability, the self-learning closed-loop structure of this invention significantly improves parameter update efficiency. By generating model parameter adjustment factors through differential analysis of the feedback dataset, the model can achieve multi-parameter linkage updates in a single iteration, reducing the number of parameter convergence steps from 460 rounds in the traditional LSTM model to 240 rounds, and improving the convergence speed by about 47%. In addition, in the stability test of continuous operation for 24 hours, the error drift rate of the model of this invention is only 2.3%, which is more than 80% lower than that of the traditional model, indicating that the model has good self-correction and self-stability in the long-term online operation environment.
[0151] Further analysis reveals that the fundamental reason for the performance improvement lies in the three innovative mechanisms of this invention: the dynamic gating interaction unit enables bidirectional feature flow and collaborative parameter updates between the Long Short-Term Memory network and the Transformer module during the feature extraction stage, solving the problem of "independent feature evolution and asynchronous parameters" in previous models; the dynamic attention weight calculation mechanism introduces temporal correlation factors and node importance factors during the feature fusion stage, enabling the model to adaptively adjust the weight distribution of different feature groups, significantly improving its ability to model the uncertainty of multi-source data; and the model self-learning module constructs a feedback-optimization-update closed loop, enabling the model to automatically optimize parameters based on abnormal feedback during operation, possessing continuous learning and self-evolution capabilities.
[0152] In summary, the method of this invention has achieved significant improvements in prediction accuracy, anomaly detection rate, parameter convergence speed, and operational stability. Through verification in a complex industrial internet environment, this invention proves that it can not only effectively solve the technical bottleneck of traditional models that cannot simultaneously take into account local temporal features and global correlation features, but also maintain analytical performance in dynamic environments. It provides an efficient, stable, and adaptive intelligent solution for multi-source data flow monitoring and anomaly prediction.
[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A data flow analysis method based on machine learning, characterized in that, Includes the following steps: Data flow information is collected from multi-source systems to generate data flow feature sequences; A data flow time series analysis model is constructed, which includes: an input layer, a feature extraction layer, a feature fusion layer, a model self-learning module, and an output layer. The data flow feature sequence is input into the model, and the feature extraction layer performs temporal and global correlation feature extraction on the data flow feature sequence to generate a data flow temporal embedding vector. The data flow temporal embedding vector is weighted and fused using a feature fusion layer. The importance of features is assigned through a dynamic attention weight calculation mechanism to generate a data flow fusion representation vector. The output layer performs nonlinear mapping calculations on the data flow fusion representation vector to obtain data flow path information, node interaction intensity information, prediction bias information and anomaly probability information, and marks abnormal samples when the anomaly probability information exceeds the threshold. When the prediction deviation information or anomaly probability information exceeds the preset threshold, the corresponding abnormal samples, data flow path information, node interaction strength information and prediction deviation information are input into the model self-learning module, and the model parameters are dynamically adjusted through the self-learning feedback mechanism. Based on the updated model, real-time data flow information is continuously analyzed, and the analysis results include data flow status information, data flow prediction results information, and data flow anomaly risk level information. The construction of the data flow time series analysis model specifically includes: The basic framework of the feature extraction layer is established, and the Long Short-Term Memory network module and the Transformer network module are defined as two independent feature sub-modules. The feature extraction layer includes a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computing control node; A feature alignment layer is set between the output of the Long Short-Term Memory Network module and the input of the Dynamic Gated Interaction Unit. The feature alignment layer is used to perform dimension matching, channel mapping and feature standardization on the time-dependent feature values output by the Long Short-Term Memory Network module to generate aligned feature values. A dynamic gating interaction unit is constructed between the feature alignment layer and the Transformer network module; A synchronous computation control node is set in the feature extraction layer. The synchronous computation control node is used to coordinate the computation step size and parameter synchronization relationship between the long short-term memory network module and the Transformer network module. The feature extraction layer is generated based on a long short-term memory network module, a feature alignment layer, a dynamic gating interaction unit, a Transformer network module, and a synchronous computation control node. A feature fusion layer is set after the feature extraction layer. The feature fusion layer includes a feature grouping unit, a dynamic attention weight calculation unit, and a normalization mapping unit. A model self-learning module is set after the feature fusion layer. The model self-learning module includes a feedback data receiving unit, a difference calculation unit, and a parameter update unit. A data flow time series analysis model is constructed based on the input layer, feature extraction layer, feature fusion layer, model self-learning module, and output layer; The construction of the dynamic gating interaction unit specifically includes: In the basic framework of the feature extraction layer, a bidirectional feature transfer and parameter co-evolution interaction channel is established. The interaction channel connects the hidden node layer of the long short-term memory network module and the attention calculation layer of the Transformer network module, and achieves this through the interdependence of feature weights and gating states. In the interaction channel, a weight mapping layer, a gating adjustment layer, and a feedback interface layer are set sequentially. The weight mapping layer calculates the correlation coefficient between the attention gradient direction and the gating gradient direction based on the global feature weights from the Transformer network module and the current input gate and forget gate states of the Long Short-Term Memory network module, and generates a parameter coordination signal. The gating adjustment layer synchronously and iteratively adjusts the input gate state value and forget gate state value of the long short-term memory network module and the attention matrix parameters of the Transformer network module according to the parameter coordination signal, so that the weight updates of the two modules achieve directional consistency and amplitude mutual constraint within the same training step, and generate output results. The feedback interface layer maps the output of the gating adjustment layer to a new attention weighting matrix and gating state response. In each iteration, it simultaneously updates the global attention distribution of the Transformer network and the temporal memory parameters of the Long Short-Term Memory network, realizing a dynamic co-evolution process driven by bidirectional dependency, and generating dynamic gating interaction units with parameter adaptation and feature collaboration.
2. The data flow analysis method based on machine learning according to claim 1, characterized in that, The generation of the data flow feature sequence specifically includes: Collect data flow information; The collected data flow information is denoised. Perform missing data completion on the denoised data flow information; The completed data flow information is then processed to ensure a uniform format. Normalize the data flow information after it has been formatted and standardized. The data flow information, after being processed by denoising, missing data completion, format unification, and normalization, is organized and encoded in chronological order to generate a data flow feature sequence.
3. The data flow analysis method based on machine learning according to claim 1, characterized in that, The generation of the data flow time-series embedding vector specifically includes: Input the data flow characteristic sequence into the input layer of the data flow time series analysis model; The Long Short-Term Memory (LSTM) network module of the feature extraction layer extracts the time dependency of the input data flow feature sequence, generating a data flow time-series intermediate vector containing the temporal change features of nodes; The feature alignment layer performs dimension matching and feature standardization on the intermediate vectors of the data flow time series to generate aligned time series features. The dynamic gating interaction unit performs bidirectional interactive processing on the aligned temporal features and outputs data flow interaction feature values adjusted by gating feedback. The synchronous computation control node performs synchronous control on the computation step size, parameter update and gradient direction of the data flow interaction feature value, and outputs the synchronously optimized data flow feature result; The Transformer network module performs multi-dimensional global feature association calculations on the data flow characteristic results and outputs multi-dimensional global representation results. The output layer performs a unified mapping on the global representation results and outputs a temporal embedding vector of the data flow.
4. The data flow analysis method based on machine learning according to claim 1, characterized in that, The generation of the data flow fusion representation vector specifically includes: The feature fusion layer groups the data flow temporal embedding vector according to node identification information, communication protocol information and time window information to generate multi-source feature grouping vectors; The dynamic attention weight calculation unit of the feature fusion layer performs weighted fusion of multi-source feature group vectors, adaptively adjusts the group weight allocation according to the temporal dynamic changes of the input features, and outputs the weighted fusion feature result; The attention weight calculation unit performs weight calculation on each feature group in the multi-source feature grouping vector. Based on the node interaction frequency corresponding to the node identification information, the protocol layer feature weight corresponding to the communication protocol information, and the temporal correlation factor corresponding to the time window information, it performs dependency calculation on the input feature value inside each feature group to generate a group attention weight value that is adaptively updated with feature input and temporal changes. The weighted fusion feature results are input into the normalized mapping unit of the feature fusion layer. The fusion results are then subjected to feature standardization and multi-source scale alignment to generate a data flow fusion representation vector.
5. The data flow analysis method based on machine learning according to claim 1, characterized in that, The generation of the abnormal samples specifically includes: The data flow fusion representation vector is input into the output layer of the data flow time series analysis model; In the output layer, feature mapping is calculated on the data flow fusion representation vector, and a prediction output is generated based on the correlation results of the time dimension and the node dimension. The data flow path information is calculated based on the predicted output, and the node interaction strength information is also calculated. The predicted output is compared with the actual observed data to calculate the prediction deviation information and anomaly probability information; When the anomaly probability exceeds a preset threshold, the data flow feature segment at the corresponding time step is marked as an anomaly sample.
6. The data flow analysis method based on machine learning according to claim 1, characterized in that, The dynamic adjustment specifically includes: When the detected anomaly probability information exceeds the preset anomaly threshold or the prediction deviation information exceeds the preset deviation threshold, the model self-learning module is triggered to extract the corresponding anomaly samples, data flow path information, node interaction intensity information and prediction deviation information to generate an anomaly feedback dataset. The abnormal feedback dataset is input into the model self-learning module. The model self-learning module calculates the difference information between the model output and the target expected value based on the feedback dataset and generates the model parameter adjustment factor. The weight parameters, learning rate, and gating parameters of the data flow time series analysis model are dynamically updated based on the model parameter adjustment factor to generate new model parameter state values. The dynamic update specifically includes: the model self-learning module receives the model parameter adjustment factor and the existing model parameter state values; iteratively updates the weight parameters of the data flow time series analysis model according to the model parameter adjustment factor to generate new weight parameter values; adjusts the model's learning rate parameter according to the model parameter adjustment factor to obtain updated learning rate parameter values; corrects the gating parameters corresponding to the input gate state, forget gate state, and output gate state in the long short-term memory network module according to the model parameter adjustment factor to obtain updated gating parameter values; during the parameter update process, a temporal consistency constraint is introduced to dynamically align the time step sequence of the abnormal feedback dataset with the temporal weight distribution within the model; and a parameter linkage update mechanism is used to achieve dynamic coupling between multi-dimensional parameters, forming an adaptive fusion structure of feedback learning and temporal modeling, generating new model parameter state values, which include updated weight parameter values, learning rate parameter values, and gating parameter values, thus completing the model's self-learning and adaptive update.
7. The data flow analysis method based on machine learning according to claim 1, characterized in that, The generation of the analysis results specifically includes: The real-time data flow information is input into the updated data flow time series analysis model for continuous analysis. Based on the analysis results, output data flow status information, data flow prediction results information, and data flow anomaly risk level information to generate analysis results.