Check patentability & draft patents in minutes with Patsnap Eureka AI!

Data flow dynamic prediction method and system based on Flink and LSTM

A technology of dynamic prediction and data flow, applied in neural learning methods, biological neural network models, design optimization/simulation, etc., can solve the problem of disconnection of data situation, lack of data flow traction, prediction program does not know when there is a new model, etc. problem, achieve the effect of improving efficiency and saving manpower

Active Publication Date: 2022-03-11
TECH & ENG CENT FOR SPACE UTILIZATION CHINESE ACAD OF SCI
View PDF3 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The modeling and modeling of the existing time series prediction are two separate tasks, and there are two problems: 1. The whole process lacks data flow traction, when to start modeling for the first time, and when to update the model, manual work is required 2. Modeling and model use are not carried out under the same framework, and there is a lack of interaction between modeling and model use. The forecasting program does not know when a new model is available, which leads to a disconnect between the forecast and the current time series data situation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data flow dynamic prediction method and system based on Flink and LSTM
  • Data flow dynamic prediction method and system based on Flink and LSTM
  • Data flow dynamic prediction method and system based on Flink and LSTM

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0059] Embodiment 1, with the real-time generation and influx of the monitoring data flow of the monitored system, the data flow needs to be cached so as to invoke the latest known data during modeling. Establish a data channel from Flink source (data source) to sink (data sink): use Kafka as the data source, read the Kafka source operator through Flink, and continuously ingest data streams. Transform the original data stream into a marked data stream, where the mark is the key to realize dynamic prediction. Then the data stream is incrementally written to an external system through the Flinksink operator, which can be a file system or a database (such as a MySQL database).

[0060] For a specific process of judging whether to trigger a modeling action instruction according to the real-time state of the input data stream, reference may be made to Embodiment 2.

[0061] The real-time status here can be understood as whether the conditions for predictive analysis are met.

Embodiment 2

[0062] Embodiment 2 judges the state of the data in the input data stream in real time, and triggers the first modeling action command when the conditions for predictive analysis are met. The conditions for predictive analysis to meet the needs can be understood as: set according to user needs, for example, the temperature is 38°C at the beginning, the data flow has been rising slowly, and when it rises to 39°C, the modeling instruction is triggered. In fact, it is a threshold judgment, and there is no hard requirement.

[0063] When the judgment result is yes, the model is constructed based on the data set, and the specific process of obtaining the prediction model can refer to Embodiment 3.

Embodiment 3

[0064] In embodiment 3, after the first modeling instruction is issued, a cycle timer is used to create a periodic update mechanism for the LSTM model, and an action instruction for periodically updating the model is triggered after the timer expires.

[0065] After the two instructions are issued, the LSTM modeling program is automatically called, and the latest data set will be read when the modeling program is running. Among them, the latest data set is: from the time when the data stream is written into the external system to the moment when the modeling instruction is issued, the external system will continuously accumulate data, and the latest data set is the data read from the external system for the most recent period. After the modeling is completed, immediately change the Flag value. Here, the Flag value needs to be used by the forecasting program to judge whether a new model is generated, so as to carry out predictive analysis and notify the forecasting program to st...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the field of industrial big data, in particular to a data flow dynamic prediction method and system based on Flink and LSTM. The method comprises the following steps: acquiring time sequence data generated by a monitored system in real time, forming an input data stream according to the time sequence data, converting the input data stream to obtain a marked data stream, and processing the marked data stream to form a data set which is continuously accumulated along with time; judging whether to trigger a modeling action instruction or not according to the real-time state of the input data stream; when the judgment result is yes, performing model construction based on the data set to obtain a prediction model; and predicting the data elements in the marked data stream through the prediction model to obtain a prediction result. According to the method, key modeling and prediction tasks do not need to be executed manually and are automatically completed under the traction of the data flow, so that the efficiency is improved while a large amount of manpower is saved.

Description

technical field [0001] The invention relates to the field of industrial big data, in particular to a data flow dynamic prediction method and system based on Flink and LSTM. Background technique [0002] In aerospace, aviation, nuclear power, energy, chemical industry, shipbuilding, rail transit and other industrial fields, with the improvement of the automation level of industrial equipment, more and more sensors are integrated into industrial systems to monitor and predict the operating status of equipment. Supported by rich data sources. The performance parameter data collected by sensors is generally time-series data with time stamps, which contains the law of health degradation of the monitored system over time. In order to predict the future health situation, it is necessary to establish a predictive model based on the data that has occurred, and then use the model to carry out predictive analysis, which is of great significance for guiding the use and maintenance plan...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F30/27G06N3/04G06N3/08
CPCG06F30/27G06N3/08G06N3/044
Inventor 施建明刘亦飞
Owner TECH & ENG CENT FOR SPACE UTILIZATION CHINESE ACAD OF SCI
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com