Data lake-based data processing method, device, equipment and medium

Abstract

The application provides a data lake-based data processing method, device, equipment and medium. The method comprises the following steps: determining a preset data lake table; obtaining time series data and model data of a target device based on a flink framework and the data lake table; comparing the time series data with historical time series data stored in a target state backend of the flink framework to obtain a first comparison result; comparing the model data with historical model data stored in the target state backend to obtain a second comparison result; updating the historical time series data and the historical model data in the target state backend based on the first comparison result and the second comparison result to obtain an update result; and storing the time series data and the model data in a target database according to the update result. Through the method, unified real-time access to a data lake can be realized based on a data lake storage technology, stream batch integration at a storage level is realized, and stream batch integration at a computing level is realized based on a flink computing engine.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method, apparatus, device, and medium based on a data lake. Background Technology

[0002] With the rapid development of the Internet of Things and the trend of digital transformation of traditional enterprises, the volume of data is growing at an unprecedented rate. Data timeliness is particularly important in enterprise operations, and enterprises have higher requirements for processing massive amounts of data.

[0003] Existing storage methods for massive IoT data scenarios mostly store data separately for different scenarios. For example, model data is stored in relational databases, while historical data analysis and query scenarios synchronize IoT data to the database for offline analysis and processing.

[0004] However, existing data processing methods are not effective in addressing data storage and processing issues. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and medium based on a data lake to address the problem that existing data processing methods have poor processing effects on data storage and data processing.

[0006] Firstly, this application provides a data processing method based on a data lake, the method comprising:

[0007] Define the preset data lake table, which is a data table in the data lake used to store data of the target device;

[0008] Based on the Flink framework and a pre-defined data lake, time-series data and model data of the target device are obtained.

[0009] The first comparison result is obtained by comparing the time series data with the historical time series data stored in the target state backend of the Flink framework.

[0010] By comparing the model data with the historical model data stored in the target state backend, a second comparison result is obtained;

[0011] Based on the first comparison result and the second comparison result, the historical time series data and historical model data in the target state backend are updated to obtain the update result;

[0012] Based on the update results, the time series data and model data are stored in the target database.

[0013] In this embodiment of the application, determining a preset data lake includes:

[0014] Determine the initial time-series data lake and the initial model data lake;

[0015] Based on the Flink framework and target message protocol, the time series data of the target device is stored in the initial time series data lake table to obtain the time series data lake table. The target message protocol is used to synchronize the time series data of the target device from the external database to the initial time series data lake table.

[0016] Based on the Flink framework and the target database, the model data of the target device is stored in the initial data lake table to obtain the model data lake table;

[0017] The data lake table is determined based on the time-series data lake table and the model data lake table.

[0018] In this embodiment of the application, based on the Flink framework and data lake, time-series data and model data of the target device are obtained, including:

[0019] Based on the Flink framework and MQTT protocol, time-series data of the target device is obtained in real time from the time-series data lake table;

[0020] Based on the Flink framework and a preset streaming method, model data of the target device is obtained in real time from the model data lake.

[0021] In this embodiment of the application, before obtaining the first comparison result by comparing the time-series data and the historical time-series data stored in the target state backend of the Flink framework, the method further includes:

[0022] Determine the initial state cache in the Flink framework. The initial state cache is the target state backend of the Flink framework.

[0023] Based on preset detection conditions, the data of zombie devices stored in the initial state cache is cleaned up to obtain the state cache;

[0024] Time series data and model data are stored in the state cache.

[0025] In this embodiment of the application, based on the first comparison result and the second comparison result, the historical time-series data and historical model data in the target state backend are updated to obtain the update result, including:

[0026] If the first comparison result indicates that the time series data has been updated, then the historical time series data in the target state backend is updated according to the time series data to obtain the updated time series data.

[0027] If the second comparison result indicates that the model data has been updated, then the historical model data in the target state backend is updated according to the model data to obtain the updated model data.

[0028] The update results are obtained based on the updated time series data and the updated model data.

[0029] In this embodiment of the application, it also includes:

[0030] Read the data processing rules from the data lake rule table. The data lake rule table is a pre-set rule table in the data lake. The data lake rule table is used to define the data calculation rules, and the data processing rules are the data processing rules of the target device.

[0031] When acquiring time-series data of the target device, the time-series data is processed based on data processing rules to obtain processed data. The processed data is used to characterize the incremental data of the target device in the time series.

[0032] Read the processed data in real time using the Flink framework in streaming mode;

[0033] If the processed data does not meet the first preset data threshold requirement, the processed data is determined to be abnormal data.

[0034] The abnormal data is sent to the target device based on the MQTT protocol, so that the target device can repair the abnormal state of the device according to the abnormal data.

[0035] In this embodiment of the application, after performing calculations on time-series data based on data processing rules to obtain processed data, the method further includes:

[0036] Based on a pre-set prediction model, the processed data is predicted to obtain the predicted value of the time series data. The prediction model is constructed based on historical prediction values ​​and actual values. The prediction model is used to determine the predicted value after a preset time period based on the change pattern of the processed data.

[0037] When the predicted value of the time series data does not meet the second preset data threshold requirement, the predicted value of the time series data is determined to be abnormal data.

[0038] The abnormal data is sent to the target device so that the target device can make adjustments based on the abnormal data.

[0039] In this embodiment of the application, it also includes:

[0040] Based on the model data in the data lake, determine the product and attribute tags of the target device;

[0041] Write attribute tags as product tags into the tag table to establish an index relationship between products, attribute tags, model data, and time series data;

[0042] When a user's search command for a target device is received, a results table is displayed to the user based on the search command and the index relationship of the target device.

[0043] Secondly, this application provides a data processing apparatus based on a data lake, the apparatus comprising:

[0044] The data lake table determination module is used to determine the preset data lake table, which is a data table in the data lake used to store data of the target device;

[0045] The data acquisition module is used to acquire time-series data and model data from the target device based on the Flink framework and a pre-defined data lake.

[0046] The time series data comparison module is used to compare the time series data with the historical time series data stored in the target state backend of the Flink framework to obtain the first comparison result.

[0047] The model data comparison module compares the model data with the historical model data stored in the target state backend to obtain a second comparison result.

[0048] The data update module is used to update the historical time-series data and historical model data in the target state backend based on the first comparison result and the second comparison result, and obtain the update result.

[0049] The data storage module is used to store time-series data and model data into the target database based on the update results.

[0050] Thirdly, this application provides an apparatus, including: a processor, and a memory communicatively connected to the processor;

[0051] The memory stores the instructions that the computer executes;

[0052] The processor executes computer execution instructions stored in memory to implement the method of this application.

[0053] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of this application.

[0054] The data processing method, apparatus, device, and medium based on a data lake provided in this application involve: determining a preset data lake table, which is a data table in the data lake used to store data of the target device; acquiring time-series data and model data of the target device based on the Flink framework and the preset data lake table; comparing the time-series data with historical time-series data stored in the target state backend of the Flink framework to obtain a first comparison result; comparing the model data with historical model data stored in the target state backend to obtain a second comparison result; updating the historical time-series data and historical model data in the target state backend based on the first and second comparison results to obtain an update result; and storing the time-series data and model data in the target database according to the update result.

[0055] In this way, data stored in the data lake table can be obtained through the Flink framework, thereby obtaining the time-series data and model data of the target device. The obtained time-series data is then compared with the historical time-series data stored in the target state backend of the Flink framework, and the model data is compared with the historical model data stored in the target state backend. Based on the comparison results, the historical time-series data and historical model data are updated, and based on the update results, the time-series data and model data are stored in the target database. Based on this, the update frequency of the downstream target database can be reduced, which greatly improves the system performance, provides the application side with the latest status and indicator information of the device, and enables unified real-time access to the data lake based on data lake storage technology. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 This is a flowchart illustrating a data processing method based on a data lake, as provided in an embodiment of this application.

[0058] Figure 2 This is a flowchart illustrating another data processing method based on a data lake, provided as an embodiment of this application.

[0059] Figure 3 This is a schematic diagram of the structure of a data processing device based on a data lake, provided in an embodiment of this application.

[0060] Figure 4 This is a structural block diagram of an apparatus for performing a data lake-based data processing method according to an embodiment of this application.

[0061] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] In existing technologies, for scenarios with high real-time requirements, such as real-time alarms and real-time analysis queries, data is stored in low-latency storage, such as time-series databases or message queues, for real-time computation. However, for offline computation, data is stored in the corresponding database for offline computation. This separate storage system solution brings additional system maintenance costs, and because the data is stored in two copies, there may be data inconsistencies between real-time and offline data, which brings additional usage costs.

[0064] To address the aforementioned issues, this application provides a data processing method based on a data lake. This method utilizes the streaming computing framework Flink to read data from the data lake table in real time, synchronizes upstream object model system data to the data lake in real time, and synchronizes change information to the data lake metadata storage in real time. This solves the problem of real-time updates of device object model data and device-reported IoT data to the data lake.

[0065] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0066] This embodiment does not impose any particular restrictions on the implementation method of the execution entity, as long as the execution entity can determine the preset data lake table, which is a data table in the data lake used to store the data of the target device; based on the Flink framework and the data lake table, the time-series data and model data of the target device are obtained; the time-series data is compared with the historical time-series data stored in the target state backend of the Flink framework to obtain a first comparison result; the model data is compared with the historical model data stored in the target state backend to obtain a second comparison result; based on the first comparison result and the second comparison result, the historical time-series data and historical model data in the target state backend are updated to obtain an update result; according to the update result, the time-series data and model data are stored in the target database.

[0067] In this context, a data lake can be understood as a system or storage that stores data in its natural or raw format, typically in the form of object blocks or files. A data lake is usually a single storage for all data in an enterprise, which includes copies of the raw data generated by the original system as well as transformed data generated for various tasks.

[0068] The Flink (Apache Flink) framework is a framework and distributed processing engine for stateful computation on unbounded and bounded data streams; Flink can run in all common cluster environments and can perform computations at memory speeds and arbitrary scales.

[0069] Figure 1 This is a flowchart illustrating a data processing method based on a data lake, provided as an embodiment of this application. Figure 1 As shown, this data lake-based data processing method may include the following steps:

[0070] S110. Determine the preset data lake table. The data lake table is a data table in the data lake used to store data of the target device.

[0071] The preset data lake table is a pre-set data table that stores data about the target device.

[0072] Based on this, a data lake is defined so that data can be retrieved from the data lake and processed subsequently.

[0073] In some implementations, determining a preset data lake includes:

[0074] Determine the initial time-series data lake and the initial model data lake;

[0075] Based on the Flink framework and target message protocol, the time series data of the target device is stored in the initial time series data lake table to obtain the time series data lake table. The target message protocol is used to synchronize the time series data of the target device from the external database to the initial time series data lake table.

[0076] Based on the Flink framework and the target database, the model data of the target device is stored in the initial data lake table to obtain the model data lake table;

[0077] Based on time-series data lake tables and model data lake tables, determine the data lake tables.

[0078] Among them, the initial time-series data lake table is the model data lake device time-series data table without stored data. This table supports efficient sequential writing of appended data. The initial model data lake table is the model data lake table without stored data. This table model supports real-time updates based on the primary key and can also update its own table structure in real-time based on changes in the upstream table structure.

[0079] The target message protocol can be the Message Queuing Telemetry Transport (MQTT) protocol, or any other protocol used to synchronize the timing data of the target device from an external database to the initial timing data lake. This embodiment does not impose any restrictions on this.

[0080] The target database can be a relational database or other types of databases.

[0081] The model data is object model data, which refers to digitizing entities in physical space and building a data model of those entities in the cloud; the product functions are defined from three dimensions: attributes, services, and events.

[0082] Based on this, time-series data is collected and fed into the data lake using the Flink framework and target message protocol; model data is collected and fed into the data lake using the Flink framework and target database; and device model data and IoT data reported by devices are updated to the data lake in real time.

[0083] S120, based on the Flink framework and data lake, obtains time-series data and model data from the target device.

[0084] Based on this, data obtained from the data lake table through the Flink framework can directly replace the Kafka message queue component, avoiding the resource usage and cluster maintenance of the Kafka message queue and reducing the system's storage costs.

[0085] In some implementations, time-series data and model data of the target device are obtained based on the Flink framework and data lake, including:

[0086] Based on the Flink framework and MQTT protocol, time-series data of the target device is obtained in real time from the time-series data lake table;

[0087] Based on the Flink framework and a preset streaming method, model data of the target device is obtained in real time from the model data lake.

[0088] MQTT is a message protocol based on the publish / subscribe paradigm. It operates on the TCP / IP protocol suite and is a publish / subscribe message protocol designed for remote devices with low hardware performance and poor network conditions. It can be used by the Flink framework to obtain time-series data from a time-series data lake in real time.

[0089] The default streaming read method allows you to read and process a portion of the data at a time.

[0090] Based on this, the Flink framework acquires device time-series data from the time-series data lake in real time via the MQTT protocol, and acquires device object model data from the model data lake in real time via streaming reads, thus achieving the acquisition of time-series and model data of the target device. Based on the Flink framework, low-latency query responses can be achieved under massive datasets, thereby directly providing data services. This solves the problems of high latency in conventional data lakes and the need to synchronize processed data to a low-latency database to provide services, reducing related usage and maintenance costs.

[0091] S130. Compare the time series data with the historical time series data stored in the target state backend of the Flink framework to obtain the first comparison result.

[0092] The target state backend is the state storage for data in the data lake table stored by the Flink framework.

[0093] The first comparison result is used to determine the data update status of the time series data compared to the historical time series data stored in the target state backend. For example, the time series data has been updated compared to the historical time series data, or the time series data has not been updated compared to the historical time series data, so that subsequent processing can be carried out based on the updated time series data.

[0094] Based on this, by comparing the time-series data obtained by the Flink framework with historical time-series data, it is possible to determine whether the time-series data of the target device has been updated based on the first comparison result.

[0095] In some implementations, before obtaining the first alignment result by comparing the time-series data with the historical time-series data stored in the target state backend of the Flink framework, the method further includes:

[0096] Determine the initial state cache in the Flink framework. The initial state cache is the target state backend of the Flink framework.

[0097] Based on preset detection conditions, the data of zombie devices stored in the initial state cache is cleaned up to obtain the state cache;

[0098] Time series data and model data are stored in the state cache.

[0099] The state cache is RocksDB state backend, which is a type of state backend in Flink. Using RocksDB as a state backend can improve the performance and reliability of Flink applications. Furthermore, RocksDB state backend supports incremental checkpointing, which allows for fast checkpointing operations without affecting application performance.

[0100] The preset detection condition is to clean up zombie device data in a timely manner by setting the time to live (TTL) of the state, so as to avoid the problem of query efficiency decreasing as the Flink state grows larger over time due to the large number of devices.

[0101] Based on this, by cleaning up the data of zombie devices stored in the initial state cache, a state cache is obtained, which enables the storage of time-series data and model data of the target device into the state cache.

[0102] S140. Compare the model data with the historical model data stored in the target state backend to obtain the second comparison result.

[0103] The second comparison result is used to determine the data update status of the model data compared to the historical model data stored in the target state backend.

[0104] Based on this, by comparing the model data obtained by the Flink framework with historical model data, it is possible to determine whether the model data of the target device has been updated based on the second comparison result.

[0105] S150. Based on the first comparison result and the second comparison result, update the historical time series data and historical model data in the target state backend to obtain the update result.

[0106] Based on this, by using the first comparison result and the second comparison result, it is determined that the time series data and model data of the target device have been updated, thereby updating the historical time series data and historical model data in the target state backend and realizing data synchronization.

[0107] In some implementations, based on the first comparison result and the second comparison result, the historical time-series data and historical model data in the target state backend are updated to obtain the update result, including:

[0108] If the first comparison result indicates that the time series data has been updated, then the historical time series data in the target state backend is updated according to the time series data to obtain the updated time series data.

[0109] If the second comparison result indicates that the model data has been updated, then the historical model data in the target state backend is updated according to the model data to obtain the updated model data.

[0110] The update results are obtained based on the updated time series data and the updated model data.

[0111] Among them, the updated time series data refers to the time series data of the target device that has been updated compared to the historical time series data, and the updated model data refers to the model data of the target device that has been updated compared to the historical model data.

[0112] Based on this, by comparing the results, the updated time series data and the updated model data are determined, thereby determining the update result.

[0113] S160. Based on the update results, store the time series data and model data in the target database.

[0114] The target database is the Remote Dictionary Server (Redis), an open-source NoSQL database written in C, also known as a non-relational database.

[0115] Based on this, by determining the update result, the updated portion of the target device's time-series data compared to the historical time-series data is stored in the Redis database, and the updated portion of the model data compared to the historical model data is also stored in the Redis database. Thus, by using state caching, the update frequency of downstream Redis can be reduced, greatly improving system performance and providing the application side with the latest device status and indicator information.

[0116] In some implementations, the method further includes:

[0117] Read the data processing rules from the data lake rule table. The data lake rule table is a pre-set rule table in the data lake. The data lake rule table is used to define the data calculation rules, and the data processing rules are the data processing rules of the target device.

[0118] When acquiring time-series data of the target device, the time-series data is processed based on data processing rules to obtain processed data. The processed data is used to characterize the incremental data of the target device in the time series.

[0119] Read the processed data in real time using the Flink framework in streaming mode;

[0120] If the processed data does not meet the first preset data threshold requirement, the processed data is determined to be abnormal data.

[0121] The abnormal data is sent to the target device based on the MQTT protocol, so that the target device can repair the abnormal state of the device according to the abnormal data.

[0122] The first preset data threshold is a preset data threshold used to determine abnormal data. It can be a filter operator used to filter out abnormal data based on the first preset data threshold.

[0123] Based on this, the time-series data of the target device is processed using the data processing rules in the data lake rule table. Abnormal data is identified according to a first preset data threshold, and instructions are sent to the device side via the MQTT protocol to complete the repair of abnormal device states. For example, if the temperature is too high, a cooling instruction is sent to the device control system to cool the device in a timely manner.

[0124] In this implementation, after processing the time-series data based on data processing rules to obtain the processed data, the method further includes:

[0125] Based on a pre-set prediction model, the processed data is predicted to obtain the predicted value of the time series data. The prediction model is constructed based on historical prediction values ​​and actual values. The prediction model is used to determine the predicted value after a preset time period based on the change pattern of the processed data.

[0126] When the predicted value of the time series data does not meet the second preset data threshold requirement, the predicted value of the time series data is determined to be abnormal data.

[0127] The abnormal data is sent to the target device so that the target device can make adjustments based on the abnormal data.

[0128] The prediction model can be an efficient algorithm library that comes with the Flink framework, using a linear regression prediction algorithm and performing calculations and predictions in a streaming mode.

[0129] The second preset data threshold requirement is a preset data threshold used to determine abnormal data in the predicted values ​​of time series data. It can be compared with a defined threshold, and predicted values ​​that exceed the threshold range are filtered out to identify abnormal data.

[0130] Based on this, the time-series data of the target device is predicted using a preset prediction model to obtain the predicted value, which is then compared with the threshold defined by the corresponding indicator. Predicted values ​​that exceed the threshold range are filtered out, and then abnormal values ​​are fed back to the application platform as abnormal warning information through message events, providing early warning capabilities to the outside world.

[0131] In other embodiments, the method further includes:

[0132] Based on the model data in the data lake, determine the product and attribute tags of the target device;

[0133] Write attribute tags as product tags into the tag table to establish an index relationship between products, attribute tags, model data, and time series data;

[0134] When a user's search command for a target device is received, a results table is displayed to the user based on the search command and the index relationship of the target device.

[0135] Based on this, Flink reads the product model table in real time, writing all attributes of the product model as tags to the tag table in real time. While viewing the tags, users can also view the time-series data of the devices with the corresponding tags. Based on the partition prefix index of the data lake, the platform can perform real-time retrieval based on products and tags.

[0136] In this embodiment, by integrating Flink, it is possible to synchronize upstream object model system data to the data lake in real time and synchronize change information to the data lake metadata storage in real time. This solves the problem of real-time updates of device object model data and device-reported IoT data to the data lake. By basing the data lake query service on a distributed computing engine, it can meet the low-latency query response under massive data, thereby directly providing data services to the outside world. This solves the problems of high latency in conventional data lakes and the need to synchronize processed data to a low-latency database to provide data services, thereby reducing related usage and maintenance costs.

[0137] Figure 2 This is a flowchart illustrating another data processing method based on a data lake, provided as an embodiment of this application. Figure 2 As shown, this data lake-based data processing method may include the following steps:

[0138] S210 and Flink acquire device timing data in real time via the MQTT protocol and acquire device object model data in real time via streaming.

[0139] Based on this, Flink obtains the device time-series data and device model data from the data lake table.

[0140] S220: Cache the latest data of the latest device into the state store.

[0141] Based on this, the newly acquired device timing data and device model data from Flink are cached in the state store.

[0142] S230. Configure RocksDB as the Flink state storage backend.

[0143] Therefore, given the large number of devices, the TTL of the state is set to clean up zombie device data in a timely manner, avoiding the problem of Flink state becoming larger and larger over time, which leads to a decrease in query efficiency.

[0144] S240. After obtaining new data, it is first compared with the status data, and the updated device data is written to Redis in real time.

[0145] Based on this, state caching can reduce the update frequency of downstream Redis, greatly improving system performance and providing the application side with the latest device status and indicator information.

[0146] In some implementations, real-time data collection into the lake includes:

[0147] Create a new primary key and update the model data lake table;

[0148] For databases that can enable CDC mode, Flink-CDC technology is used to synchronize the object model data from the relational database to the object model table in the data lake in real time.

[0149] For databases that do not support enabling CDC mode, data can be written to the data lake in near real-time by directly polling the client interface;

[0150] Create and append time series data tables for devices in the model data lake;

[0151] The Flink computing engine and MQTT protocol are used to write device time-series data into the data lake table in real time.

[0152] Among them, the Flink-CDC technology involves Flink first synchronizing all the data in the table to the data lake table, and then continuing to collect incremental data in a CDC manner based on the checkpoint position obtained in the last full data acquisition.

[0153] Based on this, the model data lake table and the data lake device time series data table are used to collect and import device time series data and model data into the lake.

[0154] In other implementations, the construction of a real-time rule-based alerting model includes:

[0155] Enable change log mode for the rule-based data lake table;

[0156] Create and append the calculation results table of the model data lake rules;

[0157] Flink reads the data lake rule table in streaming mode;

[0158] Flink acquires time-series data in real time via the MQTT protocol and associates it with rule flow tables;

[0159] Flink parses rule table data in real time, dynamically performs aggregation calculations based on aggregation fields and functions through keyed process functions and state storage, and writes the calculation results to the downstream data lake calculation result table;

[0160] Flink reads the data lake computation results table in real time in streaming mode, filters out abnormal data using the filter operator, calls the alarm system interface to send abnormal data information, and sends instructions to the device side via the MQTT protocol to complete the repair of abnormal device states.

[0161] In the change log mode, an additional change log file is generated based on the characteristics of Flink synchronizing upstream data. This change log file can support Flink's high-performance real-time reading capability in streaming mode, thereby obtaining changes in upstream computing rules in real time.

[0162] The model data lake rule calculation result table supports efficient sequential writing of appended data.

[0163] The data lake rule table defines the computational logic, involving the attributes that need to be aggregated in the object model, the aggregation function, and information such as thresholds.

[0164] Based on this, Flink is used to parse the rule table data in real time and perform calculations based on the device's time-series data to realize the device's rule-based alarms. For example, if the temperature is too high, a cooling command is sent to the device control system to complete the timely cooling of the device.

[0165] In this implementation, the construction of the alarm prediction model based on historical rule calculation results includes:

[0166] Create a table of equipment object model prediction results and enable change log streaming mode;

[0167] Flink streams data from the result table of rule-based computation.

[0168] Using a linear regression prediction algorithm, the predicted values ​​of each monitoring indicator are calculated in streaming mode after a preset time, and the predicted data is written to the prediction result table in the lake in real time.

[0169] Flink reads the prediction results table in real time and compares it with the threshold defined by the corresponding metric. Predictions that exceed the threshold range are filtered out, and then outliers are fed back to the application platform as anomaly warning information via message events.

[0170] The module automatically compares the predicted alarm values ​​with the actual values, optimizes the algorithm model parameters, adjusts the weights of various influencing factors, and continuously improves the accuracy of early warnings.

[0171] Among them, the result table data based on rule calculation is the data lake calculation result table that stores the calculation results.

[0172] Linear regression prediction algorithms can be derived from Alink, a high-efficiency algorithm library that comes with the Flink framework.

[0173] The preset time is a predicted time set by the user, usually one hour.

[0174] Based on this, an alarm prediction system is built. Through early warning capabilities, if an equipment anomaly occurs one hour later, the system can promptly adjust the equipment response through manual intervention to prevent the anomaly from occurring.

[0175] In other implementations, the device tag module construction includes:

[0176] Flink streams the product model table in real time, writing all attributes from the product model into the tag table as tags for the product.

[0177] While viewing the tags, you can also view the timing data of the devices with the corresponding tags;

[0178] The tag computation engine leverages Flink's capabilities and the data lake's streaming capabilities to acquire tag computation logic in real time, perform corresponding calculations, and output the results to downstream data lake tables for querying.

[0179] The platform can perform real-time searches based on products and tags in the tag table.

[0180] Viewing the time-series data of devices tagged with the corresponding tags can be done through data lake partitioning and prefix indexing, which can quickly obtain the changes in device indicators within any range of historical data.

[0181] The query can be provided by directly creating a corresponding data lake engine table for simple aggregation tags. When device table data is written in real time, the downstream aggregation table automatically aggregates the data according to the corresponding aggregation logic and directly generates the corresponding result table, which simplifies the data processing chain and accelerates the tag production speed.

[0182] Based on this, a device labeling module is built within the lake, allowing users to determine the attribute labels and time-series data of the corresponding product model of the device according to the label table.

[0183] In this embodiment, based on the data lake model table, Flink can read data in real time in streaming mode, which can directly replace the Kafka message queue component, avoiding the resource usage and cluster maintenance of the Kafka message queue and reducing the system's storage cost.

[0184] Figure 3 A schematic diagram of the structure of a data processing device 300 based on a data lake provided in this application embodiment is shown below. Figure 3 As shown, the data processing device 300 based on the data lake includes: a data lake table determination module 310, a data acquisition module 320, a time series data comparison module 330, a model data comparison module 340, a data update module 350, and a data storage module 360.

[0185] The data lake table determination module 310 is used to determine a preset data lake table, which is a data table in the data lake used to store data of the target device.

[0186] The data acquisition module 320 is used to acquire time-series data and model data of the target device based on the Flink framework and a preset data lake.

[0187] The time series data comparison module 330 is used to compare the time series data with the historical time series data stored in the target state backend of the Flink framework to obtain the first comparison result.

[0188] The model data comparison module 340 compares the model data with the historical model data stored in the target state backend to obtain a second comparison result.

[0189] The data update module 350 is used to update the historical time series data and historical model data in the target state backend based on the first comparison result and the second comparison result to obtain the update result;

[0190] The data storage module 360 ​​is used to store time series data and model data into the target database based on the update results.

[0191] In this embodiment of the application, the data lake table determination module 310 can also be specifically used for:

[0192] Determine the initial time-series data lake and the initial model data lake;

[0193] Based on the Flink framework and target message protocol, the time series data of the target device is stored in the initial time series data lake table to obtain the time series data lake table. The target message protocol is used to synchronize the time series data of the target device from the external database to the initial time series data lake table.

[0194] Based on the Flink framework and the target database, the model data of the target device is stored in the initial data lake table to obtain the model data lake table;

[0195] The data lake table is determined based on the time-series data lake table and the model data lake table.

[0196] In this embodiment of the application, the data acquisition module 320 can also be specifically used for:

[0197] Based on the Flink framework and MQTT protocol, time-series data of the target device is obtained in real time from the time-series data lake table;

[0198] Based on the Flink framework and a preset streaming method, model data of the target device is obtained in real time from the model data lake.

[0199] In this embodiment of the application, the data processing device 300 based on the data lake can also be specifically used for:

[0200] Determine the initial state cache in the Flink framework. The initial state cache is the target state backend of the Flink framework.

[0201] Based on preset detection conditions, the data of zombie devices stored in the initial state cache is cleaned up to obtain the state cache;

[0202] Time series data and model data are stored in the state cache.

[0203] In this embodiment of the application, the data update module 350 can also be specifically used for:

[0204] If the first comparison result indicates that the time series data has been updated, then the historical time series data in the target state backend is updated according to the time series data to obtain the updated time series data.

[0205] If the second comparison result indicates that the model data has been updated, then the historical model data in the target state backend is updated according to the model data to obtain the updated model data.

[0206] The update results are obtained based on the updated time series data and the updated model data.

[0207] In this embodiment of the application, the data processing device 300 based on the data lake can also be specifically used for:

[0208] When acquiring time-series data of the target device, the time-series data is processed based on data processing rules to obtain processed data. The processed data is used to characterize the incremental data of the target device in the time series.

[0209] Read the processed data in real time using the Flink framework in streaming mode;

[0210] If the processed data does not meet the first preset data threshold requirement, the processed data is determined to be abnormal data.

[0211] The abnormal data is sent to the target device based on the MQTT protocol, so that the target device can repair the abnormal state of the device according to the abnormal data.

[0212] In this embodiment of the application, the data processing device 300 based on the data lake can also be specifically used for:

[0213] Based on a pre-set prediction model, the processed data is predicted to obtain the predicted value of the time series data. The prediction model is constructed based on historical prediction values ​​and actual values. The prediction model is used to determine the predicted value after a preset time period based on the change pattern of the processed data.

[0214] When the predicted value of the time series data does not meet the second preset data threshold requirement, the predicted value of the time series data is determined to be abnormal data.

[0215] The abnormal data is sent to the target device so that the target device can make adjustments based on the abnormal data.

[0216] In this embodiment of the application, the data processing device 300 based on the data lake can also be specifically used for:

[0217] Based on the model data in the data lake, determine the product and attribute tags of the target device;

[0218] Write attribute tags as product tags into the tag table to establish an index relationship between products, attribute tags, model data, and time series data;

[0219] When a user's search command for a target device is received, a results table is displayed to the user based on the search command and the index relationship of the target device.

[0220] Figure 4 This is a schematic diagram of the device provided in an embodiment of this application. Figure 4 As shown, the device 400 includes:

[0221] The device 400 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403, and other components. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0222] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the data processing method based on the data lake as described above.

[0223] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0224] In the above Figure 4 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0225] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0226] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0227] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the data lake-based data processing methods described above.

[0228] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0229] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0230] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer-executable instructions, which can be loaded by a processor to execute the steps in any of the data processing methods based on a data lake provided in embodiments of this application.

[0231] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0232] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0233] Since the instructions stored in the storage medium can execute the steps of any of the data lake-based data processing methods provided in the embodiments of this application, the beneficial effects that any of the data lake-based data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0234] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0235] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data lake-based data processing method, characterized in that, The method comprises: determining an initial time series data lake table and an initial model data lake table; storing time series data of a target device into the initial time series data lake table based on a flink framework and a target message protocol to obtain a time series data lake table, the target message protocol being used to synchronize the time series data of the target device from an external database to the initial time series data lake table; storing model data of the target device into the initial model data lake table based on the flink framework and a target database to obtain a model data lake table; determining a data lake table based on the time series data lake table and the model data lake table, the data lake table being a data table in a data lake for storing data of the target device; real-time acquiring the time series data of the target device from the time series data lake table based on a flink framework and an mqtt protocol; real-time acquiring the model data of the target device from the model data lake table based on the flink framework and a preset stream reading mode; comparing the time series data with historical time series data stored in a target state backend of the flink framework to obtain a first comparison result; comparing the model data with historical model data stored in the target state backend to obtain a second comparison result; updating the historical time series data and the historical model data in the target state backend based on the first comparison result and the second comparison result to obtain an update result; storing the time series data and the model data into a target database according to the update result.

2. The method of claim 1, wherein, Before the comparing the time series data with historical time series data stored in a target state backend of a flink framework to obtain a first comparison result, the method further comprises: determining an initial state cache in the flink framework, the initial state cache being a target state backend of the flink framework; cleaning data of a zombie device stored in the initial state cache based on a preset detection condition to obtain a state cache; storing the time series data and the model data into the state cache.

3. The method of claim 1, wherein, The updating the historical time series data and the historical model data in the target state backend based on the first comparison result and the second comparison result to obtain an update result comprises: if the first comparison result indicates that the time series data has data update, updating the historical time series data in the target state backend according to the time series data to obtain updated time series data; if the second comparison result indicates that the model data has data update, updating the historical model data in the target state backend according to the model data to obtain updated model data; obtaining an update result according to the updated time series data and the updated model data.

4. The method of claim 1, wherein, The method further comprises: read a data processing rule in a data lake rule table, the data lake rule table being a rule table preset in the data lake table, the data lake rule table being used to define a calculation rule of data, the data processing rule being a data processing rule of the target device; when the time series data of the target device is acquired, the time series data is calculated and processed based on the data processing rule to obtain processed data, the processed data being used to represent incremental data of the target device in time series; the processed data is read in a streaming mode in real time through a flink framework; when the processed data does not meet a first preset data threshold requirement, it is determined that the processed data is abnormal data; the abnormal data is sent to a target device end based on an mqtt protocol, so that the target device end repairs a device abnormal state according to the abnormal data.

5. The method of claim 4, wherein, after the time series data is calculated and processed based on the data processing rule to obtain the processed data, the method further comprises: a prediction model is used to determine a predicted value after a preset time period according to a change rule of the processed data, the prediction model being constructed according to historical predicted values and actual values; when the predicted value of the time series data does not meet a second preset data threshold requirement, it is determined that the predicted value of the time series data is abnormal data; the abnormal data is sent to the target device end, so that the target device end adjusts the target device according to the abnormal data.

6. The method of claim 1, wherein, the method further comprises: a product and an attribute label of the target device are determined based on model data in the data lake table; the attribute label is written into a label table as a label of the product to establish an index relationship among the product, the attribute label, the model data and the time series data; when a retrieval instruction of the target device is received from a user, a result table is displayed to the user according to the retrieval instruction and the index relationship of the target device.

7. A data lake based data processing apparatus, characterized by, the device comprises: a data lake table determination module, configured to determine an initial time series data lake table and an initial model data lake table; store time series data of a target device into the initial time series data lake table based on a flink framework and a target message protocol to obtain a time series data lake table, the target message protocol being used to synchronize the time series data of the target device from an external database to the initial time series data lake table; store model data of the target device into the initial model data lake table based on the flink framework and a target database to obtain a model data lake table; determine a data lake table based on the time series data lake table and the model data lake table, the data lake table being a data table in a data lake used to store data of a target device; The data acquisition module is configured to acquire the time series data of the target device from the time series data lake table in real time based on a Flink framework and an MQTT protocol, and acquire the model data of the target device from the model data lake table in real time based on the Flink framework and a preset stream reading mode. The time series data comparison module is configured to compare the time series data with historical time series data stored in a target state backend of the Flink framework to obtain a first comparison result. The model data comparison module is configured to compare the model data with historical model data stored in the target state backend to obtain a second comparison result. The data update module is configured to update the historical time series data and the historical model data in the target state backend based on the first comparison result and the second comparison result to obtain an update result. The data storage module is configured to store the time series data and the model data into a target database according to the update result.

8. An apparatus, comprising: The computer readable storage medium stores computer execution instructions, and the computer execution instructions can be invoked by a processor to execute the method according to any one of claims 1 to 6. The computer readable storage medium stores computer execution instructions, and the computer execution instructions can be invoked by a processor to execute the method according to any one of claims 1 to 6. ​ ​ 9. A computer-readable storage medium, characterized in that, ​

Images