Lake and warehouse integrated Internet of Things time series data storage architecture system and method

Abstract

The invention discloses a lake and warehouse integrated Internet of Things time series data storage architecture system and method in the crossing field of Internet of Things time series data storage and engineering machinery intelligent technology. The system comprises a data source layer, a data access layer, a data access and preprocessing layer, a lake and warehouse integrated storage layer, a data fusion and analysis layer and an application layer. The data source layer collects multi-source heterogeneous time sequence data, after the multi-source heterogeneous time sequence data is buffered and temporarily stored by the data access layer, an adaptation module of the data access and preprocessing layer analyzes and converts the data, and a preprocessing module performs quality cleaning to form standardized data; the lake-warehouse integrated storage layer carries out classified storage according to data types; the data fusion and analysis layer extracts fusion features and constructs an equipment health state evaluation model; and the application layer realizes the functions of equipment monitoring, fault prediction, construction optimization and the like according to an analysis result, effectively solves the problems of storage, fusion and query optimization of multi-source heterogeneous time series data, improves the storage efficiency and the query speed, and assists in intelligent operation, maintenance and management of equipment.

Description

Technical Field

[0001] The present invention relates to a lake-warehouse integrated Internet of Things time-series data storage architecture system and method, belonging to the intersection of Internet of Things time-series data storage and engineering machinery intelligent technology. Background Art

[0002] At present, although traditional relational databases can effectively manage structured data, in the IoT scenario, when faced with massive, highly concurrent time series data, their fixed table structures are difficult to adapt to multi-source heterogeneous data formats, and storage and writing speeds are limited. Although time series databases are optimized for time series data, most focus on regular sampling scenarios. For multi-source heterogeneous and irregularly sampled time series data in complex equipment manufacturing, they are deficient in fusion, multimodal processing, and correlation analysis. While the integration of data lakes and data warehouses to build a lake-warehouse integrated architecture is becoming a trend, there is no mature solution in the field of IoT time series data storage that deeply combines the concept of lake-warehouse integration to achieve full-process optimization from efficient storage to fast query and analysis. The shortcomings of existing technologies include storage architecture limitations, which make it difficult to adapt to multi-source heterogeneous time series data generated by open-pit mining equipment, etc., resulting in data dispersion, inability to uniformly manage and analyze, and difficulty in achieving elastic resource expansion, affecting the normal operation of business. At the same time, there are problems with data writing and query efficiency. In the scenario of high-concurrency data generation in construction machinery, the existing storage system cannot guarantee low-latency writing, which easily causes data backlog and is not conducive to real-time monitoring and analysis. In addition, for complex query requests for multi-source heterogeneous time series data, the query speed is slow and cannot meet the requirements for data immediacy in engineering construction scenarios.

[0003] In addition, existing technologies have difficulty achieving efficient integration of multi-source heterogeneous data and are unable to fully tap the value of data. For example, it is impossible to organically combine the physical vibration data of the equipment with real-time operating condition data and environmental image data to build a comprehensive equipment health assessment model or an accurate operation process analysis model. This limits the improvement of the intelligent level of construction machinery and is not conducive to the accurate prediction and early intervention of potential equipment failures. Summary of the Invention

[0004] The purpose of the present invention is to provide an integrated lake-warehouse IoT time-series data storage architecture system and method, which can solve the problems of multi-source heterogeneous data storage bottlenecks, write-query efficiency imbalance, cross-modal fusion barriers and insufficient support for intelligent applications.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] In one aspect, the present invention provides a lake-warehouse integrated IoT time series data storage architecture system, comprising: The data source layer is used to obtain heterogeneous time series data from multiple sources; The data access layer includes a data acquisition module, which is used to receive the multi-source heterogeneous time series data and temporarily store the multi-source heterogeneous time series data in a buffer of the data acquisition module; Data access and preprocessing layer, including: A data adaptation module, configured to parse and convert the multi-source heterogeneous data to obtain adapted data; A data preprocessing module is used to perform quality inspection and cleaning on the adapted data to obtain preprocessed standardized data; The lake-warehouse integrated storage layer is used to store the standardized data in different locations according to type; The data fusion and analysis layer is used to extract features from the standardized data stored in different locations, fuse the extracted features, construct a comprehensive feature vector, and build an equipment health status assessment model based on the comprehensive feature vector; The application layer is used to perform equipment operation status monitoring, fault prediction and construction optimization based on the analysis results output by the equipment health status assessment model.

[0007] In combination with the first aspect, further, the source heterogeneous time series data includes real-time working condition data, physical vibration data and environmental image data.

[0008] In combination with the first aspect, further parsing and converting the multi-source heterogeneous data to obtain adapted data includes: Structuring the real-time working condition data and converting it into a relational data format to obtain structured working condition data; Performing time series encoding and compression storage on the physical vibration data to obtain physical vibration time series data; Adopting an adapted image storage format for the environmental image data and adding metadata description to obtain environmental image data with metadata description added; Based on the structured working condition data, physical vibration time series data and environmental image data with added metadata description, adapted data that meets the system storage and analysis requirements is obtained.

[0009] In combination with the first aspect, further, the adapted data is quality checked and cleaned to obtain pre-processed standardized data, including: filtering out noise in the adapted data to obtain filtered data; Filling missing values ​​in the filtered data to obtain filled data; The outliers in the filled data are corrected to obtain preprocessed standardized data.

[0010] In combination with the first aspect, further storing the standardized data in different locations according to type includes: Storing the structured working condition data in a structured data storage area of ​​a data warehouse; The physical vibration data is stored in layers according to access frequency; wherein the compressed physical vibration time series data with high frequency access is stored in the time series optimization storage area of ​​the data warehouse; the original physical vibration data or the physical vibration time series data with low frequency access is stored in the original data storage area of ​​the data lake; The environmental image data with metadata description is stored in the raw data storage area of ​​the data lake in an unstructured format.

[0011] In combination with the first aspect, the lake-warehouse integrated storage layer further includes a metadata management module for recording and managing the metadata information of the standardized data stored in different locations in real time; wherein the metadata information includes data source, type, timestamp, storage location and device ID, and a composite index is constructed based on the timestamp and device ID to optimize data query efficiency.

[0012] In combination with the first aspect, further, feature extraction is performed on the standardized data stored in different locations, and the extracted features are fused, including: A feature extraction module, configured to simultaneously extract the time series features of the physical vibration time series data and the equipment state structured features of the structured working condition data using a long short-term memory network; Using a convolutional neural network to extract visual features of the environmental image data with added metadata description; A data fusion module, configured to perform cross-modal fusion of the temporal features, device state structured features, and visual features to generate a comprehensive feature vector; The real-time analysis module is used to perform real-time data analysis based on the comprehensive feature vector and trigger an early warning mechanism according to a set threshold.

[0013] In combination with the first aspect, further, the application layer includes: Equipment monitoring unit, used to display equipment operating status and monitoring interface in real time; Fault prediction unit, used to predict potential equipment failures in advance and plan repair and maintenance work; The construction optimization unit is used to optimize the construction plan and resource scheduling according to the analysis results.

[0014] The second aspect is a lake-warehouse integrated IoT time series data storage architecture method, including: Acquire multi-source heterogeneous time series data; Receiving the multi-source heterogeneous time series data, and temporarily storing the multi-source heterogeneous time series data in a buffer of a data acquisition module; Parse and convert temporarily stored multi-source heterogeneous data to obtain adapted data; Performing quality inspection and cleaning on the adapted data to obtain pre-processed standardized data; Storing the standardized data in different locations according to type; Extracting features from the standardized data stored in different locations, fusing the extracted features to construct a comprehensive feature vector, and training a device health status assessment model based on the comprehensive feature vector; Based on the analysis results output by the equipment health status assessment model, equipment operation status monitoring, fault prediction and construction optimization are performed.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Acquire multi-source heterogeneous time series data generated by open-pit mining equipment and engineering machinery during operation, including real-time working condition structured data, physical vibration time series data, and environmental image unstructured data; receive and buffer the multi-source heterogeneous time series data through the data access layer, temporarily store it in the data acquisition module, and perform type analysis and conversion processing on the temporarily stored multi-source heterogeneous data by the data adaptation module. Perform quality inspection and cleaning on the adapted data through the preprocessing module, remove noise and process missing values ​​and outliers, generate preprocessed standardized data, and store the standardized data by type in different locations of the lake warehouse integrated storage layer After acquiring multi-source heterogeneous time series data from the lake warehouse integrated storage layer, a long short-term memory network is used to extract the time series features of physical vibration data and operating condition data, and a convolutional neural network is used to extract the visual features of environmental image data. The time series features and visual features are cross-modally fused through the data fusion module to construct a comprehensive feature vector. Based on this feature vector, an equipment health status assessment model is constructed and trained. Based on the analysis results output by the model, a real-time analysis module is used to monitor vibration exceeding the limit and abnormal fluctuations in operating parameters and trigger visual warnings. At the same time, it supports the equipment monitoring system to display the operating status in real time, the fault prediction system to plan maintenance in advance, and the construction optimization system to adjust resource scheduling to improve engineering construction safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure shows a system block diagram of an IoT time series data storage architecture provided by an embodiment of the present invention; Figure 2 The figure shows a data storage management flow chart provided by an embodiment of the present invention; Figure 3 Shown is a data fusion analysis flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0018] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects. Example 1

[0019] See also Figure 1 , a lake-warehouse integrated IoT time series data storage architecture system, including a data source layer, a data access layer, a data access and preprocessing layer, a lake-warehouse integrated storage layer, a data fusion and analysis layer, and an application layer.

[0020] Among them, the data source layer is used to obtain multi-source heterogeneous time series data; The data acquisition module in the data access layer is used to receive multi-source heterogeneous time series data and temporarily store the multi-source heterogeneous time series data in the buffer area of ​​the module through the data acquisition module; The data access and preprocessing layer includes a data adaptation module and a data preprocessing module. The data adaptation module is used to parse and convert multi-source heterogeneous data to obtain processed adapted data; the data preprocessing module is used to perform quality inspection and cleaning on the adapted data to obtain preprocessed standardized data. The lake-warehouse integrated storage layer is used to store the standardized data in different locations according to type to achieve efficient storage and management; The data fusion and analysis layer is used to extract features from standardized data stored in different locations, fuse the extracted features, and construct a comprehensive feature vector. This comprehensive feature vector is then used to train the equipment health status assessment model, improving data fusion and analysis capabilities and enhancing comprehensive perception and accurate assessment of equipment operating status. The application layer is used to perform equipment operation status monitoring, fault prediction and construction optimization based on the analysis results output by the equipment health status assessment model, providing support for the intelligent application of construction machinery.

[0021] In summary, the system solves the limitations of traditional storage architecture in processing IoT time series data through the integration of lake warehouse integration and distributed architecture, improves storage efficiency and scalability, reduces storage costs, and improves system reliability and scalability.

[0022] Furthermore, the system employs a low-latency write strategy and intelligent query optimization algorithms, significantly improving the write efficiency and query speed of multi-source heterogeneous time-series data. This meets the stringent real-time data requirements of engineering construction scenarios and provides higher-level applications with faster and smoother data access. Real-time data analysis and early warning mechanisms can promptly detect equipment anomalies and potential failures, providing construction personnel with timely and accurate warning information. This helps ensure proactive troubleshooting and prevention measures, reducing equipment downtime and maintenance costs, and ensuring the safety and efficiency of engineering construction. Example 2

[0023] Data collection phase: In daily production processes, open-pit mining equipment, ultra-large cranes and other construction machinery can generate multi-source heterogeneous time series data in real time during operation. Among them, multi-source heterogeneous time series data includes real-time operating condition data (such as equipment operating status parameters: temperature, pressure, speed, etc.), physical vibration data (time series data collected by sensors, which may contain noise and irregular sampling characteristics) and environmental image data (such as video or image data around the equipment, etc., that is, unstructured data).

[0024] Next, on the construction machinery, various sensors, monitoring devices and data acquisition modules deployed on the equipment are used to collect the multi-source heterogeneous data generated above in real time to ensure the integrity and accuracy of the data collection.

[0025] It should be noted that the collected multi-source heterogeneous data covers key parameters and environmental information of equipment operation.

[0026] Data access and preprocessing stage: The multi-source heterogeneous data collected in real time from the device is transmitted to the data access layer of the system architecture. The transmitted multi-source heterogeneous data is first temporarily stored in the buffer of the data acquisition module to prepare for subsequent data processing; In the data access and preprocessing layer, the data adaptation module parses and converts the temporarily stored multi-source heterogeneous data to obtain the processed adapted data; The process of obtaining the adaptation module includes: Structural processing is performed on the real-time working condition data, and it is converted into a relational data format (such as a table) to obtain structured working condition data; Perform time series encoding and compressed storage on physical vibration data to obtain adaptively encoded physical vibration time series data, thereby reducing storage space and optimizing query efficiency; Adopting an appropriate image storage format (such as JPEG, PNG, etc.) for the environmental image data and adding metadata descriptions, such as shooting time, to obtain environmental image data with metadata descriptions added; Based on structured working condition data, adaptively encoded physical vibration time series data, and environmental image data with added metadata description, adapted data that meets the system storage and analysis requirements is obtained.

[0027] Furthermore, the data preprocessing module performs quality inspection and cleaning on the adapted data to obtain preprocessed standardized data. The processing process is as follows: Filter out the noise in the adapted data to obtain filtered data, thereby improving the purity of the data; Fill in the missing values ​​in the filtered data to obtain filled data; Correct or delete the outliers in the filled data to obtain the preprocessed standardized data.

[0028] Through the above preprocessing process, the data quality is improved, the accuracy and reliability of the data are ensured, and a good foundation is laid for subsequent data storage and analysis.

[0029] Data storage stage: See also Figure 2 , the pre-processed standardized data is stored in different locations of the lake warehouse integrated storage layer according to type, and the storage in different locations specifically includes: For structured data that requires frequent complex queries and analyses, such as structured process data, store it in the structured data storage area of ​​the data warehouse; Physical vibration data is stored in layers according to access frequency. The compressed physical vibration time series data with high frequency access is stored in the time series optimization storage area of ​​the data warehouse. The original physical vibration data or the physical vibration time series data with low frequency access is stored in the original data storage area of ​​the data lake. The environmental image data with metadata description is stored in the raw data storage area of ​​the data lake in an unstructured format.

[0030] Furthermore, the lake-warehouse integrated storage layer also includes a metadata management module for real-time recording and management of metadata information of standardized data stored in different locations; wherein, metadata information includes key information such as data source, type, timestamp, storage location and device ID, and constructs a composite index based on the timestamp and device ID, that is, a multi-level indexing mechanism is adopted to construct an index based on the time characteristics of time series data, device identification and other dimensions, such as constructing a composite index for device ID and time range to optimize data query efficiency and facilitate subsequent rapid positioning and retrieval of data.

[0031] Data fusion and analysis stage: See also Figure 3The data fusion module obtains the required different types of multi-source heterogeneous data from the lake-warehouse integrated storage layer. At the same time, the real-time analysis module performs real-time calculation and analysis on the fused data. The fusion and analysis process is as follows: The feature extraction module uses a long short-term memory (LSTM) network to simultaneously extract the time series features of physical vibration time series data and the structured features of equipment status from structured working condition data, thereby capturing the long-term dependencies in the time series data and the characteristics of equipment operating state changes. Use convolutional neural networks (CNNs) to extract visual features of environmental image data with metadata descriptions; The data fusion module is used to cross-modally fuse time series features, equipment status structured features, and visual features to generate a comprehensive feature vector, thereby achieving deep fusion of multi-source data at the feature level. Based on the comprehensive feature vector, it is also possible to further construct an equipment health status assessment model or an operation process analysis model.

[0032] The real-time analysis module is used to perform real-time data analysis based on the comprehensive feature vector. When an anomaly is detected, such as vibration outside the normal range or abnormal fluctuation of operating parameters, an early warning mechanism is triggered, and the early warning information is notified to relevant personnel through a visual interface or message push, so that measures can be taken quickly to ensure the safety of engineering construction and the normal operation of equipment.

[0033] Application service stage: The fused multi-source heterogeneous data provides service support for the application layer, such as equipment monitoring systems, fault prediction systems, and construction optimization systems, as follows: Equipment monitoring unit, used to display the equipment operating status in real time and provide operators with an intuitive monitoring interface; Fault prediction unit, used to predict potential equipment failures in advance and help maintenance personnel plan repair and maintenance work; The construction optimization unit is used to optimize construction plans and resource scheduling based on data analysis results, thereby improving construction efficiency and quality.

[0034] Feedback optimization phase: Establish a feedback optimization mechanism to evaluate the results of data fusion and analysis, as well as the effectiveness of application services. Based on the evaluation results, optimize and adjust the data fusion module and real-time analysis module, continuously improve system performance and analysis accuracy, and better meet the needs of intelligent applications of construction machinery. This forms a closed loop of continuous improvement and promotes the continuous evolution and improvement of the system. Example 3

[0035] A lake-warehouse integrated IoT time series data storage architecture method, including: Acquire multi-source heterogeneous time series data; Receiving the multi-source heterogeneous time series data, and temporarily storing the multi-source heterogeneous time series data in a buffer of a data acquisition module; Parse and convert temporarily stored multi-source heterogeneous data to obtain adapted data; Performing quality inspection and cleaning on the adapted data to obtain pre-processed standardized data; Storing the standardized data in different locations according to type; Extracting features from the standardized data stored in different locations, fusing the extracted features to construct a comprehensive feature vector, and constructing a device health status assessment model based on the comprehensive feature vector; Based on the analysis results output by the equipment health status assessment model, equipment operation status monitoring, fault prediction and construction optimization are performed.

[0036] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A lake-warehouse integrated IoT time series data storage architecture system, characterized by: include: The data source layer is used to obtain heterogeneous time series data from multiple sources; The data access layer includes a data acquisition module, which is used to receive the multi-source heterogeneous time series data and temporarily store the multi-source heterogeneous time series data in a buffer of the data acquisition module; Data access and preprocessing layer, including: A data adaptation module, configured to parse and convert the multi-source heterogeneous data to obtain adapted data; A data preprocessing module is used to perform quality inspection and cleaning on the adapted data to obtain preprocessed standardized data; The lake-warehouse integrated storage layer is used to store the standardized data in different locations according to type; The data fusion and analysis layer is used to extract features from the standardized data stored in different locations, fuse the extracted features, construct a comprehensive feature vector, and build an equipment health status assessment model based on the comprehensive feature vector; The application layer is used to perform equipment operation status monitoring, fault prediction and construction optimization based on the analysis results output by the equipment health status assessment model.

2. The lake-warehouse integrated IoT time series data storage architecture system according to claim 1 is characterized in that: The source heterogeneous time series data includes real-time working condition data, physical vibration data and environmental image data.

3. The lake-warehouse integrated IoT time series data storage architecture system according to claim 2 is characterized in that: Parsing and converting the multi-source heterogeneous data to obtain adapted data includes: Structuring the real-time working condition data and converting it into a relational data format to obtain structured working condition data; Performing time series encoding and compression storage on the physical vibration data to obtain physical vibration time series data; Adopting an adapted image storage format for the environmental image data and adding metadata description to obtain environmental image data with metadata description added; Based on the structured working condition data, physical vibration time series data and environmental image data with added metadata description, adapted data that meets the system storage and analysis requirements is obtained.

4. The lake-warehouse integrated IoT time series data storage architecture system according to claim 3 is characterized in that: The adapted data is quality checked and cleaned to obtain pre-processed standardized data, including: filtering out noise in the adapted data to obtain filtered data; Filling missing values ​​in the filtered data to obtain filled data; The outliers in the filled data are corrected to obtain preprocessed standardized data.

5. The lake-warehouse integrated IoT time series data storage architecture system according to claim 3 is characterized in that: The standardized data is stored in different locations according to type, including: Storing the structured working condition data in a structured data storage area of ​​a data warehouse; The physical vibration data is stored in layers according to access frequency; wherein the compressed physical vibration time series data with high frequency access is stored in the time series optimization storage area of ​​the data warehouse; the original physical vibration data or the physical vibration time series data with low frequency access is stored in the original data storage area of ​​the data lake; The environmental image data with metadata description is stored in the raw data storage area of ​​the data lake in an unstructured format.

6. The lake-warehouse integrated IoT time series data storage architecture system according to claim 5 is characterized in that: The lake-warehouse integrated storage layer also includes a metadata management module for recording and managing the metadata information of the standardized data stored in different locations in real time; wherein the metadata information includes data source, type, timestamp, storage location and device ID, and a composite index is constructed based on the timestamp and device ID to optimize data query efficiency.

7. The lake-warehouse integrated IoT time series data storage architecture system according to claim 5 is characterized in that: Extract features from standardized data stored in different locations and fuse the extracted features, including: A feature extraction module, configured to simultaneously extract the time series features of the physical vibration time series data and the equipment state structured features of the structured working condition data using a long short-term memory network; Using a convolutional neural network to extract visual features of the environmental image data with added metadata description; A data fusion module, configured to perform cross-modal fusion of the temporal features, device state structured features, and visual features to generate a comprehensive feature vector; The real-time analysis module is used to perform real-time data analysis based on the comprehensive feature vector and trigger an early warning mechanism according to a set threshold.

8. The lake-warehouse integrated IoT time series data storage architecture system according to claim 1 is characterized in that: The application layer includes: Equipment monitoring unit, used to display equipment operating status and monitoring interface in real time; Fault prediction unit, used to predict potential equipment failures in advance and plan repair and maintenance work; The construction optimization unit is used to optimize the construction plan and resource scheduling according to the analysis results.

9. A lake-warehouse integrated IoT time series data storage architecture method, characterized in that: include: Acquire multi-source heterogeneous time series data; Receiving the multi-source heterogeneous time series data, and temporarily storing the multi-source heterogeneous time series data in a buffer of a data acquisition module; Parse and convert temporarily stored multi-source heterogeneous data to obtain adapted data; Performing quality inspection and cleaning on the adapted data to obtain pre-processed standardized data; Storing the standardized data in different locations according to type; Extracting features from the standardized data stored in different locations, fusing the extracted features to construct a comprehensive feature vector, and training a device health status assessment model based on the comprehensive feature vector; Based on the analysis results output by the equipment health status assessment model, equipment operation status monitoring, fault prediction and construction optimization are performed.

Images