Data processing system, method, device, storage medium and program product

Abstract

An embodiment of the present application provides a data processing system, method, device, storage medium, and program product. The data processing system includes a streaming data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system. Among them, the streaming data processing platform is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively. The streaming data processing platform is configured to receive the original telemetry data reported by the target device, and store the real-time data and query data extracted from the original telemetry data. The real-time data is stored in the cache database, the query data is stored in the relational database, and the original telemetry data is stored in the distributed file storage system in the form of files. Through the above system, the speed of data processing and response generated by the target device can be effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing system, method, device, storage medium, and program product. Background Art

[0002] During the operation of some devices, data related to the operation process of the device will be generated, and these data play a crucial role in maintaining the safe, stable, and efficient operation of the device.

[0003] In the related art, a database management system can be used to store and process the data generated during the operation of the device. However, if the amount of data generated during the operation of the device is huge, the processing and response speed of the database management system for this data is poor. Summary of the Invention

[0004] Embodiments of this application provide a data processing system, method, device, storage medium, and program product, which can effectively improve the processing and response speed of data.

[0005] In a first aspect, embodiments of this application provide a data processing system, which includes a streaming data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system; wherein, the streaming data processing platform is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively, and the big data processing platform is also communicatively connected to the distributed file storage system;

[0006] The streaming data processing platform is configured to receive the original telemetry data reported by the target device, and store the real-time data and query data extracted from the original telemetry data into the cache database, store the query data into the relational database, and store the original telemetry data in the form of a file into the distributed file storage system; wherein, the real-time data is the data that dynamically changes during the operation of the target device;

[0007] The big data processing platform is configured to, in response to the data processing instruction of the streaming data processing platform, perform statistical analysis processing on the original telemetry data stored in the distributed file storage system to obtain the statistical data of the target device, and store the statistical data into the relational database.

[0008] In some embodiments, the streaming data processing platform is communicatively connected to the target device through an Internet of Things platform, and specifically, the streaming data processing platform is configured to:

[0009] Receive the JSON message sent by the Internet of Things platform;

[0010] Parse the original telemetry data from the JSON message according to the preset table structure;

[0011] Extract initial real-time data and initial query data from the original telemetry data; the initial query data includes at least one of the following: fault alarm data, system data, version change data;

[0012] After deduplicating the initial real-time data, obtain the real-time data, and store the obtained real-time data in the cache database;

[0013] Filter and split the initial query data to obtain the query data, and store it in the relational database in the form of key-value pairs;

[0014] Store the original telemetry data in the distributed file storage system in a columnar file storage format.

[0015] In some embodiments, the stream data processing platform is specifically configured to:

[0016] Extract the initial real-time data within a time window from the original telemetry data;

[0017] Partition the initial real-time data within the time window according to the device type;

[0018] Use the latest data in each partition as the real-time data;

[0019] According to the device identifier corresponding to the real-time data, store the real-time data in the partition corresponding to the device identifier in the cache database in a way that overwrites the old data of the device identifier.

[0020] In some embodiments, the original telemetry data includes: alarm class data, the query data includes alarm data, and N bits of the alarm class data correspond to one type of alarm event;

[0021] The stream data processing platform is specifically configured to:

[0022] Split the alarm class data according to bits to obtain multiple alarm data; one alarm data corresponds to one alarm event;

[0023] Write the alarm data into the corresponding alarm record in the relational database, where the alarm record includes: a first byte and a second byte; the first byte is used to indicate whether there is an alarm event, and the second byte is used to indicate whether the alarm event has ended.

[0024] In some embodiments, the stream data processing platform is specifically configured to:

[0025] Obtain the alarm value corresponding to the alarm data, and query the alarm records matching the alarm data from the relational database;

[0026] When the alarm value indicates that no alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, then update the alarm value in the alarm record;

[0027] When the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has ended, or if the alarm record is not queried, then update the alarm value in an added manner;

[0028] When the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, then update the alarm value in the alarm record.

[0029] In some embodiments, the target device is a device in an energy storage power station, and the big data processing platform is further configured to:

[0030] Obtain the operation data of the energy storage device of the energy storage power station from the distributed file storage system;

[0031] Use the machine learning model to predict the operation status of the energy storage device and generate a prediction result;

[0032] Write the prediction result into the relational database;

[0033] And / or, when the prediction result indicates that the energy storage device is abnormal, output an alarm message indicating that the energy storage device is abnormal.

[0034] In some embodiments, the data processing system further includes: a web application end; the web application end is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively;

[0035] The web application end is configured to:

[0036] Obtain real-time data from the cache database;

[0037] Obtain query data from the relational database and / or the big data processing platform;

[0038] Obtain the original telemetry data from the distributed file storage system.

[0039] In some embodiments, the distributed file storage system is further configured to receive the original telemetry data of the target device batch-uploaded by the web application end.

[0040] In some embodiments, the stream data processing platform is further configured to:

[0041] Send a device provisioning program to the Internet of Things platform; the device provisioning program is used to define an Internet of Things device in the Internet of Things platform for receiving the original telemetry data.

[0042] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a data processing system. The data processing system includes a stream data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system. Among them, the stream data processing platform is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively, and the big data processing platform is also communicatively connected to the distributed file storage system;

[0043] The stream data processing platform is configured to receive the original telemetry data reported by the target device, extract real-time data and query data from the original telemetry data, store the real-time data in the cache database, store the query data in the relational database, and store the original telemetry data in the distributed file storage system. Among them, the real-time data is the data that dynamically changes during the operation of the target device;

[0044] The big data processing platform is configured to, in response to the data processing instruction of the stream data processing platform, perform statistical analysis processing on the original telemetry data stored in the distributed file storage system to obtain the statistical data of the target device, and store the statistical data in the relational database.

[0045] In a third aspect, an embodiment of the present application provides a data processing device, including: a memory and a processor;

[0046] The memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to implement the method according to any one of the first aspect.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method according to any one of the first aspect.

[0048] In a fifth aspect, the present application provides a computer program product, including a computer program, and the computer program implements the method according to any one of the first aspect when executed by a processor.

[0049] The data processing system, method, device, storage medium, and program product provided by the embodiments of the present application. The data processing system includes a streaming data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system. Among them, the streaming data processing platform is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively, and the big data processing platform is also communicatively connected to the distributed file storage system. The streaming data processing platform is configured to receive the original telemetry data reported by the target device, and store the real-time data and query data extracted from the original telemetry data into the cache database, store the query data into the relational database, and store the original telemetry data into the distributed file storage system. Among them, the real-time data is the data that dynamically changes during the operation of the target device. The big data processing platform is configured to perform statistical analysis processing on the original telemetry data stored in the distributed file storage system to obtain the statistical data of the target device, and store the statistical data into the relational database. Through the above system, the speed of data processing and response generated by the target device can be effectively improved, meeting the usage requirements of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Structural schematic diagram of a data processing system provided by an embodiment of the present application Figure 1 ;

[0051] Figure 2 Process schematic diagram of storing real-time data into a relational database provided by an embodiment of the present application;

[0052] Figure 3 Process schematic diagram of writing alarm data into a relational database provided by an embodiment of the present application;

[0053] Figure 4 Structural schematic diagram of a data processing system provided by an embodiment of the present application Figure 2 ;

[0054] Figure 5 Process schematic diagram of a data processing method provided by an embodiment of the present application;

[0055] Figure 6 Structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0057] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects, and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0058] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0059] An energy storage power station is a solution proposed to address issues such as the volatility of renewable energy, peak load demand, and stability faced by the power system. It achieves the balance and stability of power supply and demand in the power system while reducing carbon emissions, providing stronger and more sustainable support for the future power system, and is of great significance to the power industry and the sustainable energy development strategy.

[0060] During the operation of an energy storage power station, a large amount of data is generated, and this data plays a crucial role in maintaining the safe, stable, and efficient operation of the power station. Therefore, how to collect, store, analyze, and process the data generated during the operation of an energy storage power station is of great significance.

[0061] In related technologies, a database management system (DBMS) can be used to store and process the data generated during the operation of energy storage power station equipment. For example, the data of the energy storage power station is collected through sensors, and the collected data is stored in a traditional relational database (such as MySQL, SQL Server), and SQL statements are used to query and analyze the data for users or managers.

[0062] However, traditional relational databases have performance bottlenecks when the amount of data in a single table reaches tens of millions. For example, query performance, memory pressure, etc. The daily data volume of an energy storage power station can easily reach tens of millions. Therefore, traditional relational databases obviously cannot provide sufficient performance and response speed, and cannot handle filtering queries and statistical analysis of massive data, which may lead to delays in the monitoring and management of the energy storage power station. When a fault alarm occurs in the energy storage power station, abnormal situations cannot be processed in time, resulting in unnecessary losses.

[0063] In view of this, the embodiments of the present application provide a data processing system, method, device, storage medium, and program product. By analyzing and processing the received data and storing different data in different types of databases, sufficient performance and response speed can be provided, and a big data processing platform is deployed in the data processing system to handle filtering queries and statistical analysis of massive data.

[0064] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments can be implemented independently or in combination with each other. Concepts or processes that are the same or similar may not be repeated in some embodiments.

[0065] Figure 1 A data processing system provided by an embodiment of the present application, as Figure 1 shown, the data processing system includes: a stream data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system.

[0066] Among them, the stream data processing platform is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively, and the big data processing platform is also communicatively connected to the distributed file storage system.

[0067] In some embodiments, the stream data processing platform is communicatively connected to the target device.

[0068] In some embodiments, the stream data processing platform is communicatively connected to the target device through an Internet of Things platform.

[0069] In some embodiments, the target device may be a device in an energy storage power station, or other devices that can generate a large amount of data. The embodiments of the present application do not limit this. That is, the data processing system provided by the embodiments of the present application can not only store and process the data of the energy storage power station, but also store and process the data generated by other devices or systems (for example, power grid data, operator data, etc.). The embodiments of the present application do not limit the application scenarios of the data processing system.

[0070] For ease of understanding, in the following, an example will be given where the target device is a device of an energy storage power station, and the original telemetry data is transmitted through an Internet of Things platform.

[0071] In some embodiments, the stream data processing platform is used to receive the original telemetry data reported by the energy storage power station.

[0072] Data acquisition devices are provided in the energy storage power station. These data acquisition devices can collect data (original telemetry data) during the operation of each device in the energy storage power station through sensors, and report the original telemetry data to the stream data processing platform in the data processing system through the Internet of Things platform (IoT Hub).

[0073] In a possible implementation, the data acquisition device can periodically collect the original telemetry data at a preset time interval, and send the collected original telemetry data to the Internet of Things platform. When the Internet of Things platform receives the original telemetry data, it can forward the original telemetry data to the stream data processing platform.

[0074] In another possible implementation, the Internet of Things platform can periodically send a data acquisition instruction to the data acquisition device at a preset time interval, receive the original telemetry data collected based on the data acquisition instruction sent by the data acquisition device, and forward the original telemetry data to the stream data processing platform.

[0075] In some embodiments, when the Internet of Things platform receives the original telemetry data, it can encapsulate the original telemetry data in a target format (for example, JSON format), and send the encapsulated original telemetry data to the stream data processing platform in the form of a message (for example, a JSON message).

[0076] When the stream data processing platform receives the JSON message sent by the Internet of Things platform, it can parse the JSON message according to a preset table structure and extract the original telemetry data of the energy storage power station.

[0077] In some embodiments, after the stream data processing platform obtains the original telemetry data of the energy storage power station, it can process the original telemetry data. For example, the stream data processing platform can extract real-time data and query data from the original telemetry data according to a preset data type, store the real-time data in the cache database, store the query data in the relational database, and store the original telemetry data in the distributed file storage system in the form of a file.

[0078] Among them, the real-time data are the data that dynamically change during the operation of the equipment of the energy storage power station. For example, equipment status, power generation, battery pack status, etc. The query data are some static information during the operation of the equipment of the energy storage power station. For example, user information, system data, fault alarm information, version change information, etc.

[0079] In some embodiments, the streaming data processing platform can extract initial real-time data and initial query data from the original telemetry data.

[0080] The streaming data processing platform can perform deduplication processing on the initial real-time data to obtain the real-time data, and store the real-time data in the cache database. Among them, the streaming data processing platform can use a preset automated deduplication tool to perform deduplication on the initial real-time data to obtain the real-time data. By storing the real-time data in the cache database, users can monitor the storage power station in real time through the cache database, effectively improving the timeliness of monitoring the energy storage power station.

[0081] The streaming data processing platform can filter and split the initial query data to obtain the query data, and store it in the relational database in the form of key-value pairs.

[0082] The streaming data processing platform can perform data filtering on the initial query data, eliminate redundant information in the initial query data, improve the simplicity of the data, and split the initial query data into different query data according to different type identifiers, and store it in the relational data in the form of key-value pairs.

[0083] Among them, a key-value pair is a simple correspondence relationship. The key is used as the index of the element, and the value represents the data stored and read. When the streaming data processing platform obtains the query data, for any query data, the streaming data processing platform can use the device identifier corresponding to the query data as the key of the query data, and use the query data as the corresponding value. Subsequently, users can quickly obtain the corresponding value (query data) through the key value, effectively improving the performance and response speed of the data processing system.

[0084] The streaming data processing platform can store the original telemetry data in a columnar file (e.g., Parquet file format) storage format in the distributed file storage system. Optionally, the streaming data processing platform can also compress the original telemetry data and then store it in the distributed file storage system in a columnar file storage format to reduce the occupancy of the storage space of the distributed file storage system. By storing the original telemetry data in the distributed file storage system, the original telemetry data can be traced and the system performance will not be affected due to the huge amount of data.

[0085] Optionally, when writing the original telemetry data into the distributed file storage system, the streaming data processing platform can also perform data cleaning on the original telemetry data, thereby improving the quality of the original telemetry data and reducing the probability of errors occurring when using the original telemetry data subsequently.

[0086] In some embodiments, in the original telemetry data, there are also some valuable statistical analysis data (referred to as statistical data), for example, the number of charge-discharge cycles and the statistical analysis of charge-discharge amounts of the station. These statistical analysis data need to be obtained through some complex statistical analysis calculations on the original telemetry data. To enable users to quickly obtain these data, the data processing system can process the original telemetry data through a big data processing platform to obtain the statistical data.

[0087] Exemplarily, the big data processing platform is used to perform statistical analysis processing on the original telemetry data stored in the distributed file storage system to obtain the statistical data of the energy storage power station, and store the statistical data in the relational database.

[0088] In some embodiments, a big data computing engine (e.g., Spark big data computing engine) is built into the big data processing platform. When the big data processing platform receives a data processing instruction sent by the streaming data processing platform, it can read the original telemetry data corresponding to the data processing instruction from the distributed file storage system, and execute the data processing instruction through the big data computing engine to obtain the statistical data.

[0089] In some embodiments, a data processing program can also be preset in the big data processing platform. By executing the preset computing program through the big data computing engine, the corresponding original telemetry data can be read from the distributed file storage system, and the original telemetry data can be statistically analyzed to obtain the statistical data.

[0090] After the big data processing platform obtains the statistical data, it can store the statistical data in the relational data in the form of key-value pairs, which is convenient for users to quickly query subsequently.

[0091] In the data processing system provided by the embodiment of the present application, the original telemetry data of the energy storage power station is processed through a streaming data processing platform, the real-time data is stored in a cache database, the query data is stored in a relational database, and the original telemetry data is stored in a distributed file storage system. Moreover, the original telemetry data is statistically analyzed through a big data processing platform, and the statistical data is stored in the relational database, which can effectively improve the performance and response speed of the data processing system, meet the requirements of filtering queries and statistical analysis for massive data, and improve the ability and efficiency of monitoring the energy storage power station.

[0092] Next Figure 1 Based on the embodiment shown below, the process of the streaming data processing platform storing real-time data into a relational database will be described in detail.

[0093] Figure 2 FIG. is a schematic diagram of the process of a streaming data processing platform provided by an embodiment of the present application storing real-time data into a relational database. As Figure 2 shown, it includes:

[0094] S201. Extract the initial real-time data within a time window from the original telemetry data.

[0095] In some embodiments, in order to prevent the streaming data processing platform from writing real-time data into the cache database too frequently, which may cause excessive pressure on its server and affect the performance of the cache database, during the processing, the streaming data processing platform can set a time window (for example, set the time window by creating a window function Tumbling Window()), and extract the initial real-time data within the time window from the original telemetry data. For example, if the original telemetry data is data within 10 consecutive minutes and the time window set by the streaming data processing platform is the most recent 1 minute, then the streaming data processing platform extracts the initial real-time data of the most recent 1 minute from the original telemetry data.

[0096] S202. Partition the initial real-time data within the time window according to the device type.

[0097] In some embodiments, the real-time data of different devices may be different, and the addresses stored in the cache database may also be different. Therefore, the streaming data processing platform can partition the obtained initial real-time data according to the device type, and each partition corresponds to an initial real-time data of a certain type.

[0098] S203. Use the latest data in each partition as the real-time data.

[0099] In some embodiments, since the cache database caches the real-time data of each device, the stream data processing platform can use the latest data in each partition as the real-time data, thereby obtaining the real-time data at the minimum data transmission cost, effectively reducing the number of write requests, and reducing the probability of server downtime or processing delay of the cache database.

[0100] S204. According to the device identifier corresponding to the real-time data, store the real-time data in the partition corresponding to the device identifier in the cache database in a way that overwrites the old data of the device identifier.

[0101] In some embodiments, when the stream data processing platform writes the real-time data of each device into the cache database, it can use the device identifier of each device as the unique ID for writing into the cache database. When writing the real-time data of each device, if there is already data corresponding to the unique ID in the cache database (that is, there is old data), the stream data processing platform can store the real-time data in the partition corresponding to the device identifier in the cache database in a way that overwrites the old data of the device identifier. If there is no data corresponding to the unique ID in the cache database (that is, there is no old data), the stream data processing platform can create a new data partition in the cache database and store the real-time data in the new data partition in an added manner. By using the created unique ID as the cache Key, it is possible to ensure that each data item with the same ID can be overwritten, ensuring the real-time nature and uniqueness of the data cached by the current device.

[0102] In some embodiments, the original telemetry data reported by the energy storage power station includes alarm data. When the stream data processing platform extracts query data including alarm data from the telemetry data, since N bits (for example, 1 bit) in the alarm data correspond to one type of alarm event, when the stream data processing platform writes the alarm data into the relational database, it is necessary to split the alarm data. The following is combined with Figure 3 to illustrate this process.

[0103] Figure 3 FIG. is a schematic diagram of a process for writing alarm data into a relational database provided by an embodiment of the present application. As Figure 3 shown, it includes:

[0104] S301. Split the alarm data according to bits to obtain multiple alarm data; one alarm data corresponds to one alarm event.

[0105] Exemplarily, the alarm data includes 8 bits (bit), and each byte corresponds to one type of alarm of the device. For example, the alarm data is 00110101.

[0106] The streaming data processing platform splits the alarm data by bits to obtain multiple alarm data. For example, the alarm data 00110101 is split into 8 alarm data. For the first bit with a value of 0, it can indicate that the alarm event of this type has not been sent, or the alarm event of this type has occurred. Subsequently, taking 0 indicating that the alarm event of this type has not been sent as an example for illustration.

[0107] It should be understood that the alarm type corresponding to each bit is predefined in the streaming data processing platform.

[0108] Optionally, the streaming data processing platform also reorganizes the 8 alarm data. For example, adding a device identifier to obtain 8 alarm data.

[0109] S302. Update the alarm data to the corresponding alarm record in the relational database. The alarm record includes: a first byte and a second byte; the first byte is used to indicate whether there is an alarm event, and the second byte is used to indicate whether the alarm event has ended.

[0110] In some embodiments, the data included in the alarm record includes: the station identifier of the energy storage power station, the device identifier, the device type, the fault type, the fault start flag, the fault start time, the fault end flag, the fault end time, etc.

[0111] When the streaming data processing platform obtains multiple alarm data, it can update the alarm data to the corresponding alarm record according to the device identifier and the alarm type. The alarm record includes a first byte (bitValue1, the fault start flag) and a second byte (bitValue2, the fault end flag).

[0112] Among them, the value of bitValue1 is 1 or null. A value of 1 for bitValue1 indicates that the current device has a fault of this type. The value of bitValue2 is 0 or null. A value of 0 for bitValue2 indicates that the fault of this type of the current device has disappeared.

[0113] In some embodiments, the streaming data processing platform can write the alarm data into the alarm record according to the following steps:

[0114] S3021. Obtain the alarm value corresponding to the alarm data, and query the alarm record matching the alarm data from the relational database.

[0115] Among them, the alarm value corresponding to the alarm data is the value of the corresponding bit (bitValue). The alarm record matching the alarm data is the value of the latest first byte and second byte of this type of the device.

[0116] S3022. When the alarm value indicates that no alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, update the alarm value in the alarm record.

[0117] For any alarm data, if bitValue is 0, it means that the current device does not have a fault of the current type. If the alarm record is queried and the corresponding alarm record is: the value of bitValue1 is 1 and the value of bitValue2 is empty (the alarm record indicates that the alarm event corresponding to the alarm data has not ended), write the alarm value in the alarm record.

[0118] Exemplarily: The alarm record is as follows:

[0119] bitValue1(1), bitValue2(); (queried alarm record) bitValue1(), bitValue2(0); (updated alarm value).

[0120] Or,

[0121] bitValue1(1, null);

[0122] bitValue2(null, 0);

[0123] The first value in bitValue1 and bitValue2 is the value not queried, and the latter value is the alarm value not written.

[0124] That is, the streaming data processing platform updates the alarm data to the corresponding fault alarm record in an updated manner, indicating that the current fault has disappeared, and the current fault alarm record is a complete record data.

[0125] S3023. When the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has ended, or if the alarm record is not queried, update the alarm value in an added manner.

[0126] For any alarm data, if bitValue is 1, it means that the current device has a fault of the current type. If the alarm record is not queried, or if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has ended (the corresponding alarm record is: the value of bitValue1 is empty, the value of bitValue2 is 0, or the value of bitValue1 is 1 and the value of bitValue2 is 0), indicating that a new alarm has occurred for the device, write the alarm value in the alarm record in an added manner.

[0127] Exemplarily, the alarm records are as follows:

[0128] bitValue1(), bitValue2(0); (queried alarm record)

[0129] bitValue1(1), bitValue2(); (updating alarm value).

[0130] Or;

[0131] bitValue1(null);

[0132] bitValue2(0);

[0133] bitValue1(1);

[0134] bitValue2(null);

[0135] Among them, the first two bitValue1(null) and bitValue2(0) are the queried values, and the last two bitValue1(1) and bitValue2(null) are the newly added values.

[0136] S3024. When the alarm value indicates that an alarm event occurs, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, then update the alarm value in the alarm record.

[0137] For any alarm data, if bitValue is 1, it means that the current device has a fault of the current type. If the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended (the corresponding alarm record is: the value of bitValue1 is 1 and the value of bitValue2 is empty), it means that the device is continuously having an alarm, then update the alarm value in the alarm record.

[0138] Exemplarily: The alarm records are as follows:

[0139] bitValue1(1), bitValue2(); (queried alarm record) bitValue1(1), bitValue2(); (updating alarm value).

[0140] Or,

[0141] bitValue1(1, 1);

[0142] bitValue2(null, null).

[0143] Optionally, when updating the alarm start identifier and / or the alarm end identifier in the alarm record, the time corresponding to the alarm start identifier and / or the alarm end identifier can also be updated simultaneously.

[0144] Optionally, for any alarm data, if the bitValue is 1, it indicates that the current device has a fault of the current type. If the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended (the corresponding alarm record is: the value of bitValue1 is 1 and the value of bitValue2 is empty), it means that the device is continuously generating an alarm newly. The streaming data processing platform can also not update the alarm record to facilitate the user to easily query the first occurrence time of the alarm.

[0145] In the embodiments of the present application, the streaming data processing platform processes fault alarm data by creating a stored procedure, so that the same fault or alarm data has both occurrence records and disappearance records, thereby simplifying the operation of the fault alarm closed-loop query, saving the database disk space and improving the query speed of the fault alarm data.

[0146] On the basis of the above embodiments, the data processing system provided by the embodiments of the present application further includes a web application end.

[0147] Figure 4 The structural schematic diagram of the data processing system provided by the embodiments of the present application is as Figure 4 shown. The data processing system further includes: a web application end; the web application end is respectively communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform.

[0148] In some embodiments, the web application end obtains real-time data from the cache database. For example, the web application end can execute a predefined query command to obtain real-time data from the cache database and display the real-time data to monitor the operation status of the energy storage power station in real time.

[0149] Optionally, the cache database can also process the real-time data. For example, the real-time data is processed into a preset format to facilitate the web application end to display it, or operations such as filtering the real-time data are performed.

[0150] In some embodiments, the web application terminal obtains query data from the relational database and / or the big data processing platform. For example, the web application terminal may execute a predefined query statement to obtain the query data from the relational database and display the query data to the user. Alternatively, when the big data processing platform obtains the query data and stores the query data in the relational database, the big data processing platform sends the query data to the web application terminal.

[0151] In some embodiments, the web application terminal obtains the original telemetry data from the distributed file storage system. The distributed file storage system is preset with a data interface, and the web application terminal can directly obtain the original telemetry data from the distributed file storage system through the data interface. Alternatively, the web application terminal can obtain the original telemetry data from the distributed file storage system through the big data processing platform.

[0152] In some embodiments, some energy storage power stations may also be offline devices and are not connected to the Internet of Things platform. For the original telemetry data of this part of the energy storage power stations (referred to as offline data), the web application terminal can batch upload the original telemetry data of the devices of this part of the energy storage power stations and store it in the distributed file system.

[0153] In some embodiments, to further improve the monitoring of the operation process of the energy storage power station and ensure the normal operation of the energy storage power station, the big data processing platform can predict the state of the energy storage device (for example, the battery) in the energy storage power station.

[0154] Exemplarily, the big data processing platform obtains the operation data of the energy storage device of the energy storage power station from the distributed file storage system; uses the machine learning model to predict the operation condition of the energy storage device to generate a prediction result; writes the prediction result into the relational database; and / or, when the prediction result indicates that the energy storage device is abnormal, outputs an alarm message indicating the abnormality of the energy storage device.

[0155] The big data processing platform is built with a pre-trained machine learning model. When the big data processing platform obtains the operation data of the energy storage device of the energy storage power station from the distributed file system, it can input the operation data into the machine learning model and obtain the prediction result of the machine learning model for predicting the operation condition of the energy storage device. The big data processing platform can write the prediction result into the relational database and / or send it to the web application terminal.

[0156] The big data processing platform can also judge the prediction result to identify whether the prediction result indicates that there is an abnormality in the energy storage device. If the big data platform identifies that there is an abnormality in the energy storage device according to the prediction result, it can generate an abnormality warning message and output the warning message of the abnormality of the energy storage device to the web application end.

[0157] In some embodiments, when the devices in the energy storage power station send the original telemetry data to the Internet of Things platform, it is necessary to define Internet of Things devices in the Internet of Things platform for receiving the original telemetry data of the devices corresponding to the energy storage power station. Since the energy storage power station is large in scale and has numerous devices, it is impossible to define the Internet of Things devices one by one. Therefore, the stream data processing platform can also send a device pre-provisioning program (for example, Device Provisioning Service, DPS program) to the Internet of Things platform; the device pre-provisioning program is used to define Internet of Things devices in the Internet of Things platform for receiving the original telemetry data of the devices corresponding to the energy storage power station.

[0158] In some embodiments, when the stream data processing platform stores the original telemetry data in the distributed file storage system, it creates a file format of type parquet and sets the maximum number of data rows that can be written to the file and the longest write time, so as to ensure that the file size is within a controllable range.

[0159] In some embodiments, the stream data processing platform includes multiple data processing nodes. To achieve high concurrency and high throughput of the stream data processing platform, when the stream data processing platform receives the original telemetry data, it can split the original telemetry data into multiple sub-data (for example, split by device type) and distribute them to multiple data processing nodes for processing.

[0160] In some embodiments, when the stream data processing platform distributes multiple sub-data to multiple data processing nodes for processing, it can adopt a load balancing strategy for distribution. For example, if a certain data processing node currently has few computing tasks, more sub-data can be distributed; if a certain data processing node currently has many computing tasks, more sub-data can be distributed or no sub-data can be distributed.

[0161] In summary, the data processing system provided by the embodiments of the present application can store the real-time data of each device in the cache database, effectively improving the system performance and response speed. For a large-scale, globally distributed energy storage power station, the generated data is necessarily massive, and only the real-time data part has high value among these massive data. Therefore, it is not necessary to store the historical data in the relational database to affect the performance. This method directly stores the original telemetry data in the distributed file storage system in the form of files, so that the original data can be traced without affecting the system performance. The complex analysis and calculation of the historical data of the energy storage power station are carried out by loading the files in the distributed file storage system through the big data processing platform, and the calculation results are saved in the relational database for the server to query and display. At the same time, this method can easily handle the problems of data field changes and complex data nesting brought by different projects of the energy storage power station. Whether it is the stream data processing platform or the distributed cache, the data storage format can be easily expanded, greatly reducing the maintenance cost in the later stage of the project.

[0162] Based on the above embodiments, the embodiments of the present application further provide a data processing method.

[0163] Figure 5 It is a schematic flowchart of a data processing method provided by the embodiments of the present application. The method is applied to a data processing system, and the system includes a stream data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system. Among them, the stream data processing platform is respectively communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform, and the big data processing platform is also communicatively connected to the distributed file storage system. As Figure 5 shown, it includes:

[0164] S501. The stream data processing platform receives the original telemetry data reported by the target device, and stores the real-time data and query data extracted from the original telemetry data. The real-time data is stored in the cache database, the query data is stored in the relational database, and the original telemetry data is stored in the distributed file storage system. Among them, the real-time data is the data that dynamically changes during the operation of the devices in the energy storage power station.

[0165] S502. The big data processing platform performs statistical analysis processing on the original telemetry data stored in the distributed file storage system in response to the data processing instruction of the stream data processing platform, obtains the statistical data of the target device, and stores the statistical data in the relational database.

[0166] In some embodiments, the streaming data processing platform is communicatively connected to the devices of multiple energy storage power stations through the Internet of Things platform. The streaming data processing platform receives the JSON messages sent by the Internet of Things platform, parses the original telemetry data of the energy storage power stations from the JSON messages according to a preset table structure, extracts initial real-time data and initial query data from the original telemetry data. The initial query data includes at least one of the following: fault alarm data, system data, version change data. After deduplicating the initial real-time data, real-time data is obtained, and the obtained real-time data is stored in the cache database. The initial query data is filtered and split to obtain the query data, which is stored in the relational database in the form of key-value pairs. The original telemetry data is stored in the distributed file storage system in a columnar file storage format.

[0167] In some embodiments, the streaming data processing platform extracts the initial real-time data within a time window from the original telemetry data, partitions the initial real-time data within the time window according to device types, takes the latest data in each partition as the real-time data, and stores the real-time data in the partition corresponding to the device identifier in the cache database in a manner that overwrites the old data corresponding to the device identifier.

[0168] In some embodiments, the original telemetry data includes alarm data, the query data includes alarm data, and N bits of the alarm data correspond to one type of alarm event.

[0169] The streaming data processing platform splits the alarm data according to bits to obtain multiple alarm data. One alarm data corresponds to one alarm event. The alarm data is written into the alarm record corresponding to the relational database. The alarm record includes a first byte and a second byte. The first byte is used to indicate whether there is an alarm event, and the second byte is used to indicate whether the alarm event ends.

[0170] In some embodiments, the streaming data processing platform obtains the alarm value corresponding to the alarm data, and queries the alarm record matching the alarm data from the relational database; when the alarm value indicates that no alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, then update the alarm value in the alarm record; when the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has ended, or if the alarm record is not queried, then update the alarm value in an added manner; when the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, then update the alarm value in the alarm record.

[0171] In some embodiments, the target device is a device in an energy storage power station, and the big data processing platform obtains the operation data of the energy storage device of the energy storage power station from the distributed file storage system; uses the machine learning model to predict the operation status of the energy storage device to generate a prediction result; writes the prediction result into the relational database; and / or, when the prediction result indicates that the energy storage device is abnormal, outputs an alarm message indicating that the energy storage device is abnormal.

[0172] In some embodiments, the data processing system further includes: a web application; the web application is communicatively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform respectively.

[0173] The web application obtains real-time data from the cache database; obtains query data from the relational database and / or the big data processing platform; obtains the original telemetry data from the distributed file storage system.

[0174] In some embodiments, the distributed file storage system receives the original telemetry data of the devices of the energy storage power station uploaded in batches by the web application.

[0175] In some embodiments, the streaming data processing platform sends a device provisioning program to the Internet of Things platform; the device provisioning program is used to define an Internet of Things device in the Internet of Things platform for receiving the original telemetry data of the devices of the corresponding energy storage power station.

[0176] In some embodiments, the stream data processing platform includes multiple computing nodes. After receiving the original telemetry data, the stream data processing platform will first shard the original telemetry data, and use load balancing to distribute the sharded original telemetry data to multiple computing nodes for parallel processing, thereby achieving high concurrency and high throughput of stream data processing.

[0177] In some embodiments, the stream data processing platform uses an internal optimizer to perform logical optimization on query operations during the processing of raw telemetry data, thereby reducing computing overhead and resource consumption. Secondly, the stream data processing platform manages state information in memory and stores intermediate calculation results in memory, thereby avoiding frequent disk I / O operations, so that the processing speed of stream data reaches near real-time processing effect. In addition, the stream data processing platform will regularly create checkpoints to save processing status and intermediate results, and can resume processing after node failure to ensure data consistency. Finally, the stream data processing platform can dynamically adjust resource allocation according to the load to ensure efficient operation of the system.

[0178] For the specific implementation of each step in the data processing method for the energy storage power station equipment provided in the embodiment of the present application, reference can be made to the specific implementation of each component in the data processing system of the energy storage power station equipment in the above embodiment. The principles and technical effects are similar and will not be repeated here.

[0179] The embodiment of the present application further provides a data processing device, which can implement the technical solution executed by any device (eg, stream data processing platform) in the data processing system described in any of the above embodiments.

[0180] Figure 6 A schematic diagram of the structure of a data processing device 60 provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the electronic device may include: a transceiver 601 , a processor 602 , and a memory 603 .

[0181] The processor 602 executes the computer-executable instructions stored in the memory, so that the processor 602 executes the solution in the above embodiment. The processor 602 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0182] The memory 603 is connected to the processor 602 via a system bus and completes communication between them. The memory 603 is used to store computer program instructions.

[0183] The transceiver 601 can perform receiving data / instructions and sending data / instructions.

[0184] Optionally, the electronic device 60 may further include a communication interface 604, through which it can communicate and interact with external or internal devices via the communication interface 603. The external device may be, for example, a client (such as a mobile phone, a tablet). In a specific implementation, if the communication interface 604, the memory 603, and the processor 602 are implemented independently, the communication interface 604, the memory 603, and the processor 602 can be interconnected through a bus and communicate with each other.

[0185] The system bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to implement communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include a Random Access Memory (RAM), and may also include a non-volatile memory.

[0186] Optionally, in a specific implementation, if the communication interface 604, the memory 603, and the processor 602 are integrated on a single chip, the communication interface 604, the memory 603, and the processor 602 can communicate through an internal interface.

[0187] An embodiment of this application also provides a chip for running instructions. The chip is used to execute the technical solutions performed by any device in the above data processing system in the above embodiments.

[0188] An embodiment of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the technical solutions performed by any device in the above data processing system. The implementation principle and technical effects are similar and will not be elaborated here.

[0189] In one possible implementation, the computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CD-ROM), or other optical disc storage, magnetic disk storage, or any other medium targeted to carry or store the required program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is properly termed a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, disk and optical disc include optical disc, laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. Combinations of the above should also be included within the scope of computer-readable media.

[0190] In an embodiment of the present application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the technical solutions executed by any of the devices in the above data processing system. The implementation principle and technical effects are similar and will not be elaborated here.

[0191] In the specific implementation of the above terminal device or server, it should be understood that the processor may be a central processing unit (CPU for short), and may also be other general-purpose processors, digital signal processors (DSP for short), application-specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0192] Those skilled in the art can understand that all or part of the steps of any of the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, all or part of the steps of the above method embodiments are executed.

[0193] If the technical solution of the present application is implemented in the form of software and sold or used as a product, it can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the technical solution of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a computer program or several instructions. The computer software product enables a computer device (which can be a personal computer, a server, a network device or a similar electronic device) to execute all or part of the steps of the method described in the embodiments of the present application.

[0194] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0195] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0196] It should be understood that the above device embodiments are illustrative only, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0197] In addition, unless otherwise specified, in each embodiment of the present application, each functional unit / module can be integrated into one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0198] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0199] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned memory includes: USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical discs, etc., which can store program codes.

[0200] In the above embodiments, the descriptions of the respective embodiments each have their own emphasis. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0201] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing system, characterized in that: The data processing system includes a stream data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system; wherein the stream data processing platform is respectively connected in communication with the cache database, the relational database, the distributed file storage system, and the big data processing platform, and the big data processing platform is also connected in communication with the distributed file storage system; The stream data processing platform is used to receive the original telemetry data reported by the target device, and extract real-time data and query data from the original telemetry data, store the real-time data in the cache database, store the query data in the relational database, and store the original telemetry data in the distributed file storage system; wherein the real-time data is the data that changes dynamically during the operation of the target device; The big data processing platform is used to respond to the data processing instructions of the stream data processing platform, perform statistical analysis on the original telemetry data stored in the distributed file storage system, obtain statistical data of the target device, and store the statistical data in the relational database; The original telemetry data includes alarm data, and the stream data processing platform is further used to split the alarm data according to bits to obtain multiple alarm data; one alarm data corresponds to one alarm event; the alarm data is updated to the alarm record corresponding to the relational database, and the alarm record includes: a first byte and a second byte; the first byte is used to indicate whether there is an alarm event, and the second byte is used to indicate whether the alarm event is over; one bit of the alarm data corresponds to one type of the alarm event; the value of the first byte is 1 or empty, and the value of the second byte is 0 or empty; The stream data processing platform is specifically used for: Acquire an alarm value corresponding to the alarm data, and query an alarm record matching the alarm data from the relational database; In the case where the alarm value indicates that no alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, updating the alarm value in the alarm record; In the case where the alarm value indicates that an alarm event has occurred, if the alarm record is found and the alarm record indicates that the alarm event corresponding to the alarm data has ended, or if the alarm record is not found, then the alarm value is updated in a newly added manner; In the case where the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, the alarm value is updated in the alarm record.

2. The system according to claim 1, characterized in that The stream data processing platform is connected to the target device through the Internet of Things platform, and the stream data processing platform is specifically used for: Receive the JSON message sent by the IoT platform; Parsing the original telemetry data from the JSON message according to a preset table structure; extracting initial real-time data and initial query data from the raw telemetry data; The initial query data includes at least one of the following: fault alarm data, system data, and version change data; Performing deduplication processing on the initial real-time data to obtain real-time data, and storing the real-time data in the cache database; Filtering and splitting the initial query data to obtain the query data, and storing the query data in the relational database in a key-value pair manner; The raw telemetry data is stored in the distributed file storage system in a columnar file storage format.

3. The system according to claim 2, characterized in that The stream data processing platform is specifically used for: extracting initial real-time data within a time window from the raw telemetry data; The initial real-time data within the time window is partitioned according to the device type; Using the latest data in each partition as the real-time data; According to the device identification corresponding to the real-time data, the real-time data is stored in the partition corresponding to the device identification in the cache database in a manner of overwriting old data of the device identification.

4. The system according to any one of claims 1 to 3, characterized in that: The target device is a device in an energy storage power station, and the big data processing platform is also used for: Acquire operation data of the energy storage device of the energy storage power station from the distributed file storage system; Using a machine learning model to predict the operating status of the energy storage device and generate a prediction result; Writing the prediction result into the relational database; And / or, when the prediction result indicates that the energy storage device is abnormal, outputting alarm information of the abnormality of the energy storage device.

5. The system according to any one of claims 1 to 3, characterized in that: The data processing system further includes: a web application end; the web application end is respectively connected to the cache database, the relational database, the distributed file storage system, and the big data processing platform for communication; The web application end is used for: Acquire real-time data from the cache database; Acquire query data from the relational database and / or the big data processing platform; The raw telemetry data is obtained from the distributed file storage system.

6. The system according to claim 5, characterized in that The distributed file storage system is also used to receive the original telemetry data of the target device uploaded in batches by the web application end.

7. The system according to claim 2 or 3, characterized in that: The stream data processing platform is also used for: Sending a device provisioning program to the Internet of Things platform; the device provisioning program is used to define an Internet of Things device for receiving the raw telemetry data in the Internet of Things platform.

8. The system according to any one of claims 1 to 3, characterized in that: The stream data processing platform further includes a plurality of data processing nodes, and the stream data processing platform is further used for: The original telemetry data is distributed to the multiple data processing nodes in a load balancing manner, so that the multiple data processing nodes extract the real-time data and query data.

9. A data processing method, characterized in that: The method is applied to a data processing system, which includes a stream data processing platform, a big data processing platform, a cache database, a relational database, and a distributed file storage system; wherein the stream data processing platform is respectively connected in communication with the cache database, the relational database, the distributed file storage system, and the big data processing platform, and the big data processing platform is also connected in communication with the distributed file storage system; The stream data processing platform receives the raw telemetry data reported by the target device, extracts real-time data and query data from the raw telemetry data, stores the real-time data in the cache database, stores the query data in the relational database, and stores the raw telemetry data in the distributed file storage system; wherein the real-time data is data that changes dynamically during the operation of the target device; The big data processing platform performs statistical analysis on the raw telemetry data stored in the distributed file storage system in response to the data processing instruction of the stream data processing platform, obtains statistical data of the target device, and stores the statistical data in the relational database; The original telemetry data includes alarm data, and the stream data processing platform further splits the alarm data according to bits to obtain multiple alarm data; one alarm data corresponds to one alarm event; the alarm data is updated to the alarm record corresponding to the relational database, and the alarm record includes: a first byte and a second byte; the first byte is used to indicate whether there is an alarm event, and the second byte is used to indicate whether the alarm event is over; one bit of the alarm data corresponds to one type of the alarm event; the value of the first byte is 1 or empty, and the value of the second byte is 0 or empty; The stream data processing platform obtains an alarm value corresponding to the alarm data, and queries the relational database for an alarm record matching the alarm data; In a case where the alarm value indicates that no alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, the alarm value is updated in the alarm record; in a case where the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has ended, or, if the alarm record is not queried, the alarm value is updated in a new manner; in a case where the alarm value indicates that an alarm event has occurred, if the alarm record is queried and the alarm record indicates that the alarm event corresponding to the alarm data has not ended, the alarm value is updated in the alarm record.

10. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method as claimed in claim 9.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method described in claim 9.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method as claimed in claim 9 when the computer program is executed by a controller.

Images