Bayesian and LSTM and GNN integrated algorithm research

Abstract

The application belongs to the technical field of machine learning, and is especially based on the integrated algorithm research of Bayes and LSTM and GNN, including a graph data construction module, a graph neural network module and a Bayes long short-term memory network module; the graph data construction module is used for receiving original data and constructing graph structure data, the graph structure data including nodes, edges and node features; the graph neural network module is connected with the graph data construction module and is used for processing the graph structure data to generate node embedding vectors containing topological relationship information; in the application, the graph data construction module, the GNN module and the Bayes LSTM module are designed in a serial integrated manner, and a complete and coherent data flow from constructing a graph structure from original data, to extracting spatial features by using GNN and finally to time series modeling by Bayes LSTM is defined.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically involving the research of an integrated algorithm based on Bayesian, LSTM, and GNN. Background Technology

[0002] An integrated algorithm is a comprehensive solution that integrates multiple different technologies, models, or methods into a unified framework. It aims to fully leverage the advantages of each component to solve complex problems that are difficult to address with a single technology.

[0003] A search revealed that invention patent CN119250120A discloses a prediction method, device, and medium based on LSTM and Bayesian uncertainty. The method includes: acquiring raw data and generating an initial feature set; dividing the initial feature set into a training set and a prediction set; inputting the training set and prediction set into an optimization function to obtain the optimal parameters of an XGBoost model through Bayesian optimization; training the XGBoost model using the optimal parameters, the training set, and the prediction set, and extracting intermediate features; fusing the training set and intermediate features to obtain a final feature set; training an initial prediction model using the final feature set to obtain a prediction model; and using the prediction model to make a prediction to obtain a prediction result. By using Bayesian uncertainty, the prediction result takes into account the uncertainty inherent in the prediction, and then, through finding the optimal parameters and extensive computational inference, a more accurate prediction result that includes most of the risk is obtained.

[0004] The aforementioned disclosed technologies lack the ability to process relationships and topological structures within the data. Their methodologies revolve solely around traditional tabular data concepts such as "raw data," "feature set," and "training set," as described in the original text: "obtaining raw data and generating an initial feature set from the raw data" and "merging the training set and intermediate features to obtain the final feature set." This design implies that the model treats all data points as isolated individuals or entities with only temporal correlations, failing to identify, construct, and utilize any complex spatial, logical, or physical connections between data points (e.g., road connections in traffic sensor networks, follow relationships in social networks, and power line connections in power grids). Therefore, the core architecture of this patent has inherent limitations, preventing its application to any real-world scenario with graph-structured characteristics, such as traffic flow prediction, social network information propagation analysis, or collaborative decision-making by IoT devices, significantly limiting its application scope and upper limit of predictive performance. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a research on an integrated algorithm based on Bayesian, LSTM, and GNN, aiming to solve the technical problem that existing models treat all data points as isolated individuals or only temporally related entities, failing to identify, construct, and utilize any complex spatial, logical, or physical connections between data points.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a research system based on an integrated algorithm of Bayesian, LSTM, and GNN, comprising a graph data construction module, a graph neural network module, and a Bayesian long short-term memory network module; the graph data construction module is used to receive raw data and construct graph structure data, the graph structure data including nodes, edges, and node features; the graph neural network module is connected to the graph data construction module and is used to process the graph structure data to generate node embedding vectors containing topological relationship information; the Bayesian long short-term memory network module is connected to the graph neural network module and is used to receive the node embedding vector sequence, perform time-series modeling, and output prediction results with uncertainty estimation.

[0007] Furthermore, the graph data construction module constructs an adjacency matrix based on the physical connections, logical dependencies, or spatial relationships between data points.

[0008] Furthermore, the graph neural network module is a graph convolutional network, a graph attention network, or a graph sampling aggregation network.

[0009] Furthermore, the Bayesian Long Short-Term Memory network module achieves uncertainty quantification by introducing a probability distribution onto the weight parameters of the LSTM.

[0010] Furthermore, the probability distribution is approximated using variational inference or Monte Carlo Dropout methods.

[0011] An integrated algorithm based on Bayesian, LSTM, and GNN, applied to the system as described in any one of claims 1-5, includes: Step 1: receiving the original dataset and constructing graph structure data; Step 2: processing the graph structure data using a graph neural network, learning node representations, and obtaining a node embedding vector sequence; Step 3: inputting the node embedding vector sequence into a Bayesian long short-term memory network for training and inference, and obtaining a prediction result containing uncertainty information.

[0012] Furthermore, the step of constructing the graph structure data includes: defining an entity in the system as a node, defining the relationship between entities as an edge, and defining the observation data of the entity as node features.

[0013] Furthermore, the graph neural network processing step employs a message passing mechanism to update the target node's representation by aggregating information from neighboring nodes.

[0014] Furthermore, "obtaining a prediction result containing uncertainty information" refers to the probability distribution or confidence interval of the output prediction value.

[0015] Furthermore, the graph neural network module and the Bayesian long short-term memory network module are jointly trained in an end-to-end manner.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. In this invention, the graph data construction module, GNN module, and Bayesian LSTM module are designed in a serial, integrated manner. This architecture is not a simple stacking of models, but rather defines a complete and coherent data flow from constructing a graph structure from the original data, to extracting spatial features using GNN, and finally handing it over to Bayesian LSTM for time series modeling. This directly solves the fundamental deficiency of the comparative patent, which can only process isolated time series data and cannot utilize system topological relationships; this architecture endows the model with the ability to simultaneously capture spatiotemporal dual-dimensional information in the data. It can understand dynamic changes in time like LSTM, and understand spatial or logical correlations and dependencies like GNN; the "node embedding vectors" generated by the GNN module are a higher-level feature representation containing rich neighborhood information. Using these vectors as input to LSTM is equivalent to providing "features with contextual background" for the time series prediction model, greatly improving the quality and dimensionality of the input information; in scenarios such as traffic prediction, social network propagation, and power grid load prediction, there are strong correlations between data points. By utilizing this relational information, this architecture can make predictions that are far more accurate than those of the comparative patents. This solution is no longer limited to traditional time series forecasting and can be directly applied to time series analysis problems of all graph structure data, opening up entirely new application areas, such as supply chain risk transmission prediction, IoT device cluster collaborative fault early warning, and dynamic prediction of biological protein interactions. The analytical basis of the model has been upgraded from "points" to "networks", making the prediction results more reflective of the real operating state of complex systems.

[0017] 2. In this invention, Bayesian principles are deeply integrated into the LSTM network kernel, rather than merely serving as an external optimizer. By introducing a probability distribution into the LSTM weights, the model itself becomes a probabilistic generative model, with its final output being a prediction distribution containing confidence intervals. Unlike traditional LSTMs that output a definitive predicted value, this Bayesian LSTM outputs a probability distribution, clearly informing users "what range the predicted value might fall within" and "how confident this prediction is." This method effectively distinguishes between noise in the data itself and cognitive uncertainty arising from a lack of training data. This is crucial for risk assessment; users not only know "what the predicted result will be," but also "how high the risk of this result is." For example, in financial forecasting, the model can provide a large confidence interval along with the stock price trend, which serves as a high-risk warning, guiding users to make cautious decisions. In high-risk fields such as autonomous driving and medical diagnosis, the model can "refuse to make a prediction" or "request human intervention" when its uncertainty is too high, greatly enhancing the safety and reliability of the entire system. By analyzing the sources of uncertainty, it is possible to identify which regions or types of data are insufficient, thereby guiding more targeted data collection and saving costs.

[0018] 3. In this invention, the GNN module and the Bayesian LSTM module must be jointly trained end-to-end. This means that the parameters of all modules are synchronously updated and optimized through backpropagation algorithm based on a unified global loss function, avoiding the error accumulation and suboptimal solution problems caused by the "staged training" in the comparative patent. The feature extractor and predictor can provide feedback to each other and evolve co-evolv, with the GNN learning to extract the features most beneficial to the final time series prediction task. The entire system becomes a highly adaptive organism, capable of dynamically adjusting the operation of each internal module according to different tasks and objectives to achieve optimal overall performance; joint training can unlock the model's maximum performance potential, typically achieving higher prediction accuracy than staged training. Users do not need to train and debug multiple models separately; they only need to build a unified network and train it, reducing engineering complexity. The integrated model can better learn the general patterns behind the data, thus exhibiting stronger generalization ability when faced with unseen data. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Please see Figure 1 This embodiment provides the following technical solution: a research system based on an integrated algorithm of Bayesian, LSTM, and GNN, including a graph data construction module, a graph neural network module, and a Bayesian long short-term memory network module; the graph data construction module is used to receive raw data and construct graph structure data, which includes nodes, edges, and node features; the graph neural network module is connected to the graph data construction module and is used to process the graph structure data and generate node embedding vectors containing topological relationship information; the Bayesian long short-term memory network module is connected to the graph neural network module and is used to receive the node embedding vector sequence, perform time series modeling, and output prediction results with uncertainty estimation.

[0022] Specifically, the system includes: a sequentially connected graph data construction module, a graph neural network (GNN) module, and a Bayesian long short-term memory (LSTM) network module; the graph data construction module is configured to receive the original time-series dataset as input, which contains time-series data of multiple entities; according to predefined inter-entity relationship rules, each entity is mapped to a graph node, each set of relationships between entities is mapped to a connecting edge, and the observation data vector of an entity at a certain time step is mapped to the feature vector of the corresponding node at that time step, thereby constructing a dynamic graph structure data, which includes a node set, an edge set, and node feature vectors. The system includes an adjacency matrix describing the connections between nodes; a graph neural network (GNN) module configured to receive dynamic graph structure data from the graph data construction module, and aggregate the feature information of the neighboring nodes of each target node through at least one layer of graph convolution operation or message passing mechanism to generate a sequence of node embedding vectors containing network topology information; and a Bayesian long short-term memory (LSTM) module configured to receive the sequence of node embedding vectors from the GNN module, perform time series modeling through its internal LSTM units with probability distribution weights, and finally output the prediction result for future time steps and an estimate of the uncertainty of the prediction result. An integrated core architecture was established for the system, addressing the fundamental limitation of existing technologies in simultaneously processing spatiotemporally correlated data. By sequentially connecting the graph data construction module, graph neural network module, and Bayesian long short-term memory network module, a paradigm shift was achieved in the data processing workflow, moving from traditional isolated time-series analysis to spatiotemporally fused analysis. This architecture provides the top-level design foundation for all subsequent claims, ensuring the uniformity and integrity of the technical solutions.

[0023] The graph data construction module builds an adjacency matrix based on the physical connections, logical dependencies, or spatial relationships between data points.

[0024] Specifically, the predefined inter-entity relationship rules include: relationships based on physical connection links, logical dependencies, geospatial proximity, or relationships defined by data correlation analysis; the adjacency matrix is ​​a binary matrix or a weighted matrix, where the weights are used to represent the strength or type of inter-entity relationship; The system's applicability and flexibility have been enhanced. By clearly defining the rules for relationships between entities—based on physical connections, logical dependencies, spatial proximity, or data relevance—and by allowing the adjacency matrix to be binary or weighted, the system can accurately adapt to various application scenarios. Furthermore, by expressing the strength of relationships through weights, the model's accuracy in depicting complex relationships in the real world has been improved.

[0025] The graph neural network module can be a graph convolutional network, a graph attention network, or a graph sampling aggregation network.

[0026] Specifically, the implementation architecture of the Graph Neural Network (GNN) module is selected from one of the following: Graph Convolutional Network (GCN), Graph Attention Network (GAT), Graph Sampling Aggregation Network (GraphSAGE), or a variant thereof; the message passing mechanism follows the "aggregation-update" paradigm, that is, first aggregating the features of neighboring nodes, then combining them with the features of the target node itself, and updating them through a learnable neural network; It provides advanced and diverse graph structure information extraction capabilities. By specifying that the graph neural network module can adopt architectures such as GCN, GAT, or GraphSAGE and their "aggregation-update" paradigm, it ensures that the module can efficiently and robustly learn the feature representations of nodes in their local neighborhoods, providing high-quality input features rich in spatial correlation information for subsequent time series prediction modules.

[0027] The Bayesian Long Short-Term Memory (LSTM) network module quantifies uncertainty by introducing a probability distribution onto the weight parameters of the LSTM. This probability distribution is approximated using variational inference or Monte Carlo Dropout methods.

[0028] Specifically, the Bayesian Long Short-Term Memory (LSTM) module achieves Bayesianization by introducing a prior probability distribution into the weight parameters of the LSTM units. The weights of the LSTM units are not treated as fixed values, but rather as random variables following a normal distribution with mean μ and variance σ², thus quantifying uncertainty. The posterior distribution of the weight parameters is approximated using variational inference, i.e., a trainable variational distribution q(ω) is used to approximate the true posterior distribution p(ω|D), and the variational parameters are optimized by maximizing the evidence lower bound ELBO. Alternatively, the Monte Carlo Dropout method is used as an approximate Bayesian inference, simulating the parameter distribution by performing multiple forward propagation samplings on the LSTM with Dropout during the testing phase. This achieves a leap from deterministic to probabilistic prediction, providing crucial decision support information. Claim 4, by specifying the introduction of a prior probability distribution into the LSTM weights, lays the foundation for uncertainty quantification. Claim 5 further provides an feasible and efficient technical path by explicitly employing variational inference or Monte Carlo Dropout methods for approximate calculation. Together, these two approaches enable the system to output not only the predicted value but also the confidence level of that predicted value, providing a quantitative basis for users to assess risk.

[0029] An integrated algorithm based on Bayesian, LSTM, and GNN, applied to any of the systems described in claims 1-5, includes: Step 1: receiving the original dataset and constructing graph structure data; Step 2: processing the graph structure data using a graph neural network, learning node representations, and obtaining a node embedding vector sequence; Step 3: inputting the node embedding vector sequence into a Bayesian long short-term memory network for training and inference, and obtaining a prediction result containing uncertainty information.

[0030] Specifically, the method includes the following steps: S1: Graph structure data construction step: Receive the original time series dataset, abstract the entities in the data into graph nodes according to predefined mapping rules, abstract the relationships between entities into edges, construct the adjacency matrix of the graph, and use the entity observation data as node features to form graph structure data; S2: Spatial feature learning step: Input the graph structure data constructed in S1 into a pre-defined graph neural network (GNN) model. The GNN model iteratively updates the node features through message passing functions and update functions, and finally outputs a low-dimensional node embedding vector sequence that can represent the node and its neighborhood structure information; S3: Time series prediction and uncertainty quantification step: Input the node embedding vector sequence obtained in S2 into a Bayesian Long Short-Term Memory (LSTM) network according to time steps. The Bayesian LSTM learns the long-term dependency patterns in the time series through its probabilistic forward computation process, and outputs the predicted values ​​for future time steps and their confidence intervals or complete posterior distributions. A novel and complete integrated algorithm method is defined. Through three core steps—planning graph structure data construction, spatial feature learning, and temporal prediction and uncertainty quantification—the scope of protection of the claims is extended from the device to the method, providing a clear and operable process specification for the specific implementation of this technology and enhancing the breadth of patent protection.

[0031] The steps for constructing graph-structured data include: defining an entity in the system as a node, defining the relationships between entities as edges, and defining the observed data of entities as node features.

[0032] Specifically, in step S1, "entities" are sensors in traffic prediction scenarios, users in social network analysis, and buses or generators in power systems; "relationships" are actual connecting lines in physical networks and interaction frequency, similarity, or influence strength in virtual networks. The method's universality and specific application scenarios were clarified, enhancing the patent's implementability and protective strength. By specifically listing the counterparts of "entity" and "relationship" in different scenarios such as transportation, social interaction, and electricity, the invention's broad application potential was fully revealed, laying a solid foundation for potential future application patent layouts.

[0033] The graph neural network processing steps employ a message passing mechanism, updating the target node's representation by aggregating information from neighboring nodes.

[0034] Specifically, the "message passing mechanism" in step S2 is as follows: for each node in the graph, feature information is collected from its directly adjacent neighbor nodes. The collection method is summation, averaging, taking the maximum value, or weighted summation through an attention mechanism. The aggregated neighbor information is combined with the target node's own information and then fed into a fully connected neural network to generate a new embedding representation for the node. This ensures the effectiveness and sophistication of spatial feature learning. By specifically defining the aggregation method in the message passing mechanism and its combination with neural network updates, it guarantees that the graph neural network module can extract the feature representations that are most helpful in improving the final prediction performance from the original graph data.

[0035] "Obtaining prediction results that include information about uncertainty" refers to outputting the probability distribution or confidence interval of the predicted value.

[0036] Specifically, in step S3, the “output prediction posterior distribution” is manifested as follows: for the same input, the Bayesian LSTM module performs T forward propagation samplings to obtain T different prediction output results. Based on these T results, the mean and variance of the final prediction value are calculated. The mean is used as the final prediction value, and the variance is used to measure the uncertainty of the prediction. It provides a specific and operable technical solution for uncertainty estimation. By explicitly specifying the use of T forward propagation sampling and calculating the mean and variance to output the predicted posterior distribution, "uncertainty quantification" is no longer an abstract concept, but a practical function with clear calculation steps and deliverable results.

[0037] The graph neural network module and the Bayesian long short-term memory network module are jointly trained in an end-to-end manner.

[0038] Specifically, the GNN model in step S2 and the Bayesian LSTM model in step S3 are integrated into a unified deep learning framework, using stochastic gradient descent and its variants to jointly optimize all parameters end-to-end through backpropagation, including the parameters of the GNN, the parameters of the LSTM, and the variational parameters in variational inference. This ensures optimal system performance and simplified training. By employing an end-to-end joint training approach to optimize all parameters, it avoids error accumulation and suboptimal solutions caused by staged training. This allows the feature extraction and prediction modules to optimize collaboratively, ultimately improving the model's overall accuracy and generalization ability. Simultaneously, it simplifies the model development and deployment process.

[0039] Example 2: Please see Figure 1 In this embodiment, the staff uses the method disclosed in the present invention in a smart city traffic flow prediction system; This invention applies an integrated algorithm based on Bayesian, LSTM, and GNN. The system uses microwave traffic detectors at 256 road intersections in an urban area as data acquisition nodes, continuously collecting lane-level traffic flow, average speed, and occupancy data at 5-minute intervals. Each detector is abstracted as a graph node through a graph data construction module, and a directed weighted adjacency matrix is ​​constructed based on the actual road connection topology (weights are set according to road level and number of lanes), forming a dynamic graph structure containing 256 nodes and 812 edges. This data is input to a GNN module containing a three-layer graph attention network, which aggregates the features of adjacent nodes through a multi-head attention mechanism, generating a 128-dimensional node embedding vector sequence. This sequence is then input into a Bayesian LSTM module (using Monte Carlo Dropout for variational inference, with a Dropout rate of 0.3) for end-to-end joint training on an NVIDIA DGX A100 server, with the loss function combining mean squared error and KL divergence. After training, the system outputs real-time probability distribution predictions of traffic flow at each node for the next 15 minutes. It not only provides traffic flow point estimates (with an accuracy of 87.2%) but also generates 90% confidence intervals (average interval width of ±85 vehicles / hour). When the predicted confidence interval width exceeds the threshold of 150 vehicles / hour, the system automatically marks the prediction as a high-uncertainty output, prompting the traffic management center to combine it with manual analysis for decision-making. This effectively avoids misleading predictions during periods of abnormal sensor data, improving the reliability of the traffic guidance system to 99.6%.

[0040] The working principle of this invention is as follows: The system first receives the original time-series dataset through the graph data construction module, and maps each entity to a graph node, the association relationship to an edge, and the observation data to node features according to the physical connections, logical dependencies, or spatial relationships between entities, thereby constructing a graph structure data containing an adjacency matrix. This graph data is input to the graph neural network module (using architectures such as GCN, GAT, or GraphSAGE), which aggregates neighbor node information through a message passing mechanism to update the target node representation and generates a sequence of node embedding vectors containing topological relationships. This sequence is then input to the Bayesian Long Short-Term Memory network module, which learns the temporal dependencies of node embeddings by introducing a probability distribution into the LSTM weight parameters (approximately calculated using variational inference or Monte Carlo Dropout methods), and outputs the probability distribution or confidence interval of the predicted value through multiple forward propagation sampling, thereby realizing uncertainty quantification. In the entire system, the graph neural network module and the Bayesian LSTM module adopt an end-to-end joint training method, and optimize all parameters synchronously through the backpropagation algorithm to ensure the coordinated optimization of spatial feature extraction and temporal prediction, ultimately achieving high-precision prediction and risk assessment of complex spatiotemporal data.

[0041] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A research system based on an integrated algorithm combining Bayesian, LSTM, and GNN, characterized by: It includes a graph data construction module, a graph neural network module, and a Bayesian long short-term memory network module; The graph data construction module is used to receive raw data and construct graph structure data, which includes nodes, edges and node features. The graph neural network module is connected to the graph data construction module and is used to process the graph structure data to generate node embedding vectors containing topological relationship information. The Bayesian Long Short-Term Memory network module is connected to the graph neural network module and is used to receive the node embedding vector sequence, perform time series modeling, and output prediction results with uncertainty estimation.

2. The research system based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 1, characterized in that: The graph data construction module constructs an adjacency matrix based on the physical connections, logical dependencies, or spatial relationships between data points.

3. The research system based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 1, characterized in that: The graph neural network module is a graph convolutional network, a graph attention network, or a graph sampling aggregation network.

4. The research system based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 1, characterized in that: The Bayesian Long Short-Term Memory (LSTM) network module achieves uncertainty quantification by introducing a probability distribution onto the weight parameters of the LSTM.

5. The research system based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 1, characterized in that: The probability distribution is approximated using variational inference or Monte Carlo Dropout methods.

6. An integrated algorithm method based on Bayesian, LSTM, and GNN, applied to the system described in any one of claims 1-5, characterized in that, include: Step 1: Receive the raw dataset and construct the graph structure data; Step 2: Process the graph structure data using a graph neural network to learn node representations and obtain a sequence of node embedding vectors; Step 3: Input the node embedding vector sequence into the Bayesian Long Short-Term Memory network for training and inference to obtain prediction results containing uncertainty information.

7. The research method based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 6, characterized in that: The steps for constructing the graph structure data include: defining an entity in the system as a node, defining the relationships between entities as edges, and defining the observed data of entities as node features.

8. The research method based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 6, characterized in that: The graph neural network processing step employs a message passing mechanism, which updates the representation of the target node by aggregating information from neighboring nodes.

9. The research method based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 6, characterized in that: The phrase "obtaining a prediction result containing uncertainty information" refers to the probability distribution or confidence interval of the output prediction value.

10. The research method based on the integrated algorithm of Bayesian, LSTM, and GNN as described in claim 6, characterized in that: The graph neural network module and the Bayesian long short-term memory network module are jointly trained in an end-to-end manner.

Images