Energy storage Internet of Things platform and its data visualization method

Abstract

The present application belongs to the field of energy storage Internet of Things, and relates to an energy storage Internet of Things platform and its data visualization method. The energy storage Internet of Things platform includes: a data reporting module, which is used to obtain the original data packets uploaded by each energy storage device, and transmit the original data packets to the data storage module; the data storage module is used to parse and process the original data packets, and write them into multiple basic data tables according to categories; a data integration module is used to form multiple conventional views based on multiple basic data tables, and then form at least one materialized view based on multiple conventional views; a data visualization module is used to obtain the corresponding fields of at least one conventional view or at least one materialized view based on user settings to generate charts for display. The present application breaks through the limitations of traditional physical tables, adopts a view-centric architecture design, significantly reduces storage costs, and improves the real-time query capability of data and the ability to respond to flexible changes in historical data in a timely manner.

Description

Technical Field

[0001] The present application belongs to the field of energy storage Internet of Things, and specifically relates to an energy storage Internet of Things platform and a data visualization method thereof. Background Art

[0002] In the energy storage industry, with the rapid development of energy management and distributed energy systems, building an IoT data platform has become the key to improving the intelligent management level of energy storage systems. However, building such a data platform faces many challenges.

[0003] First, the number of energy storage devices is huge and covers a wide range. Usually, these devices are mass-produced by powerful companies, and each model may involve tens of thousands of devices, all of which need to upload data regularly. In order to handle such a large amount of data, the data platform must have a high degree of stability and strong processing capabilities to tap the potential value of the data and provide support for the expansion of new businesses.

[0004] Secondly, energy storage equipment consists of multiple key components, such as charge controllers, inverters, battery management systems, battery cells, temperature control systems, and communication modules. Each component is deployed with multiple sensors to monitor operating status and collect relevant parameters. This makes each data collection involve a large number of indicators, increasing the complexity of data upload and the difficulty of success rate.

[0005] As the amount of data continues to increase, it becomes increasingly difficult to query data in a timely and effective manner. In order to improve data usage efficiency, data warehouses are usually used to logically integrate scattered source data in layers. The integration process takes a lot of time to put data into physical tables. The physical tables at different data warehouse levels are interdependent. When a dimension change or data change occurs, all the dependent physical tables need to update the data. The process is very cumbersome and time-consuming, and it is impossible to provide real-time query capabilities for data and flexible changes to historical data. Summary of the invention

[0006] The embodiments of the present application provide an energy storage Internet of Things platform and a data visualization method thereof, which can provide real-time query capabilities for data and flexible changes to historical data.

[0007] In a first aspect, an embodiment of the present application provides an energy storage Internet of Things platform, including a data reporting module, a data storage module, a data integration module and a data visualization module, wherein:

[0008] The data reporting module is used to obtain the original data packets uploaded by each energy storage device and transmit the original data packets to the data storage module;

[0009] The data storage module is used to parse and process the original data packet to obtain original sub-packet data and the category of the original sub-packet data, and write the original sub-packet data into multiple basic data tables of the data warehouse according to its category;

[0010] The data integration module is used to perform a first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse, and then perform a second data integration based on the multiple conventional views to form at least one materialized view in the data warehouse;

[0011] The data visualization module is used to obtain corresponding fields of at least one conventional view or at least one materialized view from the data warehouse based on user settings and generate charts for display.

[0012] Optionally, the basic data table includes an aggregation table, and the data storage module is further used to merge multiple original sub-package data belonging to the same category based on a preset aggregation key and write them into a basic data of the aggregation table corresponding to the category.

[0013] Optionally, the original data packet includes a data type ID, the data type ID includes an inverter ID, a battery pack ID and a user ID, and the basic data table includes a user basic information table and a device log aggregation table.

[0014] The data storage module is also used to write the original sub-package data whose data type ID is the inverter ID or the battery pack ID into the device log aggregation table, and write the original sub-package data whose data type ID is the user ID into the user basic information table.

[0015] Optionally, the original data packet also includes a device ID and a data batch ID, and the device log aggregation table includes a date field, a device ID field, a data batch field, and a plurality of indicator information fields.

[0016] The data warehousing module is also used to use "date + device ID + data batch ID" as the aggregation key to merge the original sub-package data whose data type ID is inverter ID or battery pack ID and whose device ID and data batch ID are the same and write them into the same basic data of the device log aggregation table.

[0017] Optionally, the data integration module is used to extract a group of related field data from one of the multiple basic data tables to form a first normal view, and / or to extract a group of related field data from at least two of the multiple basic data tables respectively, and to associate and combine the extracted at least two groups of related field data to form a second normal view.

[0018] Optionally, the data integration module is further used to perform data conversion on the data content of some fields in the extracted set of related field data before forming the first conventional view; and / or to perform data conversion on the data content of some fields of at least one set of related field data in the at least two sets of related field data before forming the second conventional view.

[0019] In a second aspect, an embodiment of the present application provides a data visualization method based on an energy storage Internet of Things platform, the method comprising:

[0020] Obtain the original data packets uploaded by each energy storage device;

[0021] Parsing and processing the original data packet to obtain original sub-packet data, and writing the original sub-packet data into multiple basic data tables of the data warehouse according to their categories;

[0022] Performing a first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse, and then performing a second data integration based on the multiple conventional views to form at least one materialized view in the data warehouse;

[0023] Based on user settings, corresponding fields of at least one conventional view or at least one materialized view are obtained from the data warehouse to generate a chart for display.

[0024] Optionally, the basic data table includes an aggregate table, and writing the original sub-packaged data into multiple basic data tables of the data warehouse according to its category includes:

[0025] Based on a preset aggregation key, multiple pieces of original sub-packaged data belonging to the same category are merged and written into a basic data piece of the aggregation table corresponding to the category.

[0026] Optionally, the original data packet includes a device ID, a data type ID and a data batch ID, the data type ID includes an inverter ID, a battery pack ID and a user ID, the basic data table includes a device log aggregation table, the device log aggregation table includes a date field, a device ID field, a data batch field and multiple indicator information fields, and the writing of the original sub-packaged data into multiple basic data tables of the data warehouse according to its category includes:

[0027] Using "date + device ID + data batch ID" as the aggregation key, the original sub-packaged data whose data type ID is inverter ID or battery pack ID and whose device ID and data batch ID are the same are merged and written into the same basic data of the device log aggregation table.

[0028] Optionally, performing the first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse includes:

[0029] A group of related field data is extracted from one of the multiple basic data tables to form a first conventional view, and / or a group of related field data is respectively extracted from at least two of the multiple basic data tables, and the extracted at least two groups of related field data are associated and combined to form a second conventional view.

[0030] In an embodiment of the present application, a storage Internet of Things platform is provided, including a data reporting module, a data storage module, a data integration module and a data visualization module, wherein the data reporting module is used to obtain the original data packet uploaded by each energy storage device and transmit the original data packet to the data storage module; the data storage module is used to parse and process the original data packet to obtain the original sub-package data, and then write the original sub-package data into multiple basic data tables of the data warehouse according to its category; the data integration module is used to perform a first data integration based on multiple basic data tables to form multiple conventional views in the data warehouse, and then perform a second data integration based on multiple conventional views to form at least one materialized view in the data warehouse; the data visualization module is used to obtain at least one conventional view or at least one materialized view from the data warehouse based on user settings. The corresponding fields generate a chart for display. This application breaks through the limitations of traditional physical tables, adopts a view-centered architecture design, and significantly reduces storage costs through the organic combination of conventional views and materialized views, and improves the real-time query capability of data and the timely response capability to flexible changes in historical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The module structure diagram of the energy storage Internet of Things platform is shown as an example;

[0032] Figure 2 The technical architecture diagram of the energy storage Internet of Things platform is shown as an example;

[0033] Figure 3 The schematic diagram of data reporting of the main components of the energy storage device is shown as an example;

[0034] Figure 4 The encapsulation protocol structure diagram of the original data packet is shown as an example;

[0035] Figure 5 An exemplary diagram of automatic aggregation of original sub-packet data is shown;

[0036] Figure 6 The schematic diagram of a data warehouse implemented in a view manner is shown as an example;

[0037] Figure 7 The flowchart of the data visualization method based on the energy storage Internet of Things platform is exemplified. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The professional terms used in this application are explained as follows:

[0040] 1. Energy storage equipment: a device or system used to store energy and release it when needed, usually composed of a battery pack, inverter and control system.

[0041] 2. Inverter: Including unidirectional inverter and bidirectional inverter, it is a device that can convert direct current into alternating current. It is widely used in photovoltaic systems, energy storage systems, etc. to supply power to the power grid or other equipment.

[0042] 3. Battery pack: It is a collection of multiple battery cells, usually including a battery management system (BMS) for storing and releasing electrical energy.

[0043] 4. Protocol: refers to the rules and standards for communication between computers or electronic devices, which stipulates the format, transmission method and receiving and processing flow of data to ensure that each device can correctly understand and exchange information with each other.

[0044] 5. Protocol bit: It is a bit specified in the communication protocol. Each bit represents a byte position agreed in the communication protocol and is used to transmit data or control information. It defines the specific content and function of the data, such as identifying the start bit, end bit or check bit.

[0045] 6. Batch Packetization: It is a data transmission method that divides a large data packet into multiple small packets for transmission to reduce the burden of a single transmission and ensure the reliability and integrity of the data in the network.

[0046] 7. Conventional view: It is a common view in the data warehouse, that is, a virtual table, which consists of query statements and does not store actual data. Instead, it dynamically executes queries to generate results each time it is accessed. Its main function is to encapsulate complex query logic.

[0047] 8. Materialized view: It is the actual table in the data warehouse that stores the query result data. It can be accessed directly after it is generated, without the need to re-execute the complex query process every time. It is suitable for scenarios where query efficiency needs to be improved.

[0048] 9. Starrocks: is a high-performance distributed SQL database, mainly used for real-time analysis of large-scale data, providing high throughput and low-latency query performance, suitable for big data scenarios.

[0049] 10. RabbitMQ: It is an open source message queue middleware used for asynchronous message transmission in distributed systems. It supports multiple protocols and message routing strategies and is widely used in microservice architecture and event-driven applications.

[0050] 11. Flink: It is an open source stream processing framework that supports batch and stream processing tasks, can process massive amounts of data, and provides high-throughput, low-latency data stream analysis capabilities, making it suitable for real-time data processing scenarios.

[0051] 12. Superset: is an open source data visualization and data exploration platform that provides an intuitive user interface that enables users to easily create various types of visualization charts. It supports multiple data sources, including SQL, CSV, JSON, etc., and provides powerful query and filtering functions.

[0052] 13. Dashboards: Its Chinese meaning is Kanban, dashboard, and instrument panel. Its function is to display and monitor key data indicators, usually presenting data in the form of charts, tables, numbers, etc.

[0053] 14. Charts: represents a chart set, which can include tables, proposition charts, trend charts, pie charts and other analysis icons. A Dashboard can contain multiple different types of Charts.

[0054] See also Figure 1 and Figure 2 , Figure 1 The structure of the energy storage Internet of Things platform 100 is shown as an example. Figure 2 The following is an exemplary diagram of the technical architecture of the energy storage Internet of Things platform 100. Figure 1 and Figure 2 As shown, the energy storage Internet of Things platform 100 includes a data reporting module 101, a data storage module 102, a data integration module 103 and a data visualization module 104.

[0055] The data reporting module 101 is used to obtain the original data packets uploaded by each energy storage device and transmit the original data packets to the data visualization module 102. Specifically, Figure 3As shown in the figure, an energy storage device may include multiple inverters and multiple battery packs. The energy storage device sensors regularly monitor the various indicator data of these inverters and battery packs, and report these indicator data uniformly through the IOT (Internet of Things) box. When reporting, it is packaged according to the preset protocol. Figure 4 As shown, the protocol includes a device ID field, a data type ID field, a data batch field, a data packet number field, and a reported data content field. For example, the 1st to 4th bits store the device ID, the 5th to 8th bits store the data type ID, the 9th bit stores the data batch, the 10th bit stores the data packet number, and the 11th to Nth bits store the reported data content. Among them, the reported data content includes any one of the inverter monitoring indicators, the battery pack monitoring indicators, and the user information, and the data type ID includes any one of the inverter ID, the battery pack ID, or the user ID. When the data type ID is the inverter ID, the reported data content is the inverter monitoring indicator, when the data type ID is the battery pack ID, the reported data content is the battery pack monitoring indicator, and when the data type ID is the user ID, the reported data content is the user information.

[0056] Specifically, the inverter monitoring indicators include inverter model, inverter power type, inverter software version, inverter alarm, inverter fault, inverter operating temperature, inverter charging energy, inverter charging power, number of inverters, inverter power, inverter voltage, inverter current, number of PV accesses, PV charging power, PV charging voltage, PV charging current, PV working status, grid power, grid voltage, grid current, 5V DC load power, 12V load power, 24V load power, DC turn-on time, AC load power, AC load voltage, AC load current, etc.

[0057] The battery pack monitoring indicators include the number of battery packs, total battery pack voltage, total battery pack current, total battery pack SOC, battery pack SOH, battery pack operating status, average battery pack temperature, total battery pack charging energy, total battery pack discharging energy, battery pack low voltage alarm, number of battery pack cells, cell voltage, etc.

[0058] User information includes user name, country, gender, email address, address, usage status, etc.

[0059] In order to save space and transmission bandwidth, the various indicator data of the energy storage device are encoded and stored in hexadecimal format. After encoding, they are encrypted by an algorithm, encapsulated into original data packets, and uploaded to the RabbitMQ message queue under the security gateway, waiting to be consumed by downstream.

[0060] In the process of uploading data, uploading all indicators at a time will consume a lot of network bandwidth and there is a risk of upload failure. Therefore, the data of the same time batch is divided into K packages for upload, and one package is uploaded every m seconds. When all K packages of data are uploaded, the receiving end merges the data of the same time batch into a complete data.

[0061] The data storage module 102 is used to receive the original data packets uploaded by the data upload module 101 in batches, parse and process the original data packets to obtain the original sub-package data and the categories of the original sub-package data, and write the original sub-package data into multiple basic data tables of the data warehouse according to their categories. The multiple basic data tables are pre-created by the data storage module 102 in the database (such as the Starrocks database), including aggregate tables and ordinary tables. Among them, a basic data in the aggregate table is used to store the data content after the multiple original sub-package data are merged, and a basic data in the ordinary table is used to store the data content of an original sub-package data.

[0062] In one embodiment, the data warehousing module 102 pre-creates a user basic information table and a device log aggregation table in the data warehouse. Among them, the user basic information table is a common table or an aggregation table, and the device log aggregation table is an aggregation table. The user basic information table is used to store user information, and the device log aggregation table is used to store device information, inverter monitoring indicators, and battery pack monitoring indicators. Each basic data in the device log aggregation table contains all monitoring indicators reported in a batch, and each basic data in the user basic information table contains the latest information of all users. For example, assuming that the number of all indicators reported in a batch is 200, these 200 indicators are divided into 10 original data packets and uploaded, and each original data packet contains 20 indicators. When creating the device log aggregation table, it is also necessary to create 200 fields of indicator information to receive each sub-package data in turn, and automatically merge each sub-package data into a basic data. It can be understood that the inverter monitoring indicators and the battery pack monitoring indicators can also be stored in different aggregation tables respectively.

[0063] Specifically, the data storage module 102 uses Flink to subscribe to the original data packets reported in the RabbitMQ topic. Each time an original data packet is received, the data is first decrypted, and then the content in the original data packet is parsed according to the agreed protocol. For example, the original data parsed for the first packet of data in the i-th batch is:

[0064] 1-4 bits are the device ID, hexadecimal value: 0x7A6B5C3B

[0065] Bits 5-8 are the inverter ID / battery pack ID, hexadecimal value: 0x1A2B3C4D

[0066] 9 bits are data batches, hexadecimal value: 0x1B14

[0067] 10 bits are the data packet number, hexadecimal value: 0x01

[0068] Bit 11 is the inverter 1 model, hexadecimal value: 0x3E

[0069] Bit 12 is the inverter 1 power, hexadecimal value: 0x3E

[0070] Bit 13 is the voltage of inverter 1, hexadecimal value: 0x4D

[0071] Bit 14 is the inverter 1 current, hexadecimal value: 0x5A

[0072] The original data obtained by parsing the second package of data in the i-th batch is:

[0073] 1-4 bits are the device ID, hexadecimal value: 0x7A6B5C3B

[0074] Bits 5-8 are the inverter ID / battery pack ID, hexadecimal value: 0x1A2B3C4D

[0075] 9 bits are data batches, hexadecimal value: 0x1B14

[0076] The 10th bit is the data packet number, hexadecimal value: 0x02

[0077] Bit 11 is the inverter 1 alarm, hexadecimal value: 0x7A

[0078] Bit 12 indicates inverter 1 fault, hexadecimal value: 0x89

[0079] Bit 13 is the operating temperature of inverter 1, hexadecimal value: 0x6B

[0080] Bit 14 is the charging energy of inverter 1, hexadecimal value: 0x3C

[0081] After the data entry module 102 parses the original data based on the original data packet, it is also used to process the parsed original data to obtain original sub-packet data and the classification of the original sub-packet data. First, the data entry module 102 performs diversion processing according to the parsed data type ID, and uses the original data with the data type ID of the inverter ID or the battery pack ID as the first branch flow data, and uses the original data with the data type ID of the user ID as the second branch flow data. Secondly, the first branch flow data is subjected to a first processing to obtain the first original sub-packet data, and then the first original sub-packet data is written into the device log aggregation table in the data warehouse; the second branch flow data is subjected to a second processing to obtain the second original sub-packet data, and then the second original sub-packet data is written into the user basic information table in the data warehouse.

[0082] In one embodiment, the first processing and the second processing include data conversion processing, such as converting the parsed hexadecimal raw data into decimal raw data. In the above case, the data of the first package of data in the i-th batch after being converted into decimal is: {2050471867}{439041101}{6932}{01}{62}{62}{77}{90}, and the data of the second package of data in the i-th batch after being converted into decimal is: {2050471867}{439041101}{6932}{02}{122}{137}{107}{60}.

[0083] In other embodiments, the first processing also includes data cleaning, for example, cleaning data with empty indicators, converting units of numerical values, filling default values ​​in designated fields, etc.

[0084] After obtaining the original sub-package data, the specific method of writing the original sub-package data into multiple basic data tables of the data warehouse according to its category is: use the Stream load micro-batch method provided by Starrocks to write the original sub-package data into the corresponding basic data table. When large quantities of data are stored in real time, the use of the Stream load micro-batch method can not only avoid putting pressure on the performance of the data warehouse, but also improve writing efficiency and stability compared to the conventional insert and batch insert methods. In addition, Stream load writes data using HTTP requests to submit file stream data to the Starrocks database. It has nothing to do with the programming language and can be used as long as the Http protocol is implemented.

[0085] In the embodiment of the present application, by creating an aggregation table in the Starrocks database, multiple original sub-package data can be automatically merged according to the aggregation key without coding to obtain a complete basic data to form the basic data of the data warehouse, which greatly saves development costs and time costs. Specifically, the data warehousing module 102 is also used to merge multiple original sub-package data belonging to the same category into a basic data of the aggregation table corresponding to the category based on the preset aggregation key. Figure 5As shown, it is a schematic diagram of automatic aggregation of original sub-packaged data. Taking the device log aggregation table as an example, the device log aggregation table includes a date field, a device ID field, a data batch field and multiple indicator information fields. The data warehousing module 102 uses "date + device ID + data batch ID" as the aggregation key, and merges and writes each original sub-packaged data whose data type ID is the inverter ID or the battery pack ID and the device ID and the data batch ID are the same into the same basic data of the device log aggregation table. For example, assuming that the number of all indicators reported in a batch is 200, these 200 indicators are divided into 10 original data packets and uploaded, and each original data packet contains 20 indicators, then a device log aggregation table containing 200 indicator information fields is created in advance to receive data. When the first package of 20 indicators arrives, the corresponding 20 indicators in the table are filled, and the other 180 indicator values ​​in the table are empty. When the second package of 20 indicators arrives, 20 of the remaining 180 indicators will be filled. After filling, there are still 160 indicators left empty, and so on until 10 packages of data are filled. At this time, a complete basic data is formed.

[0086] In some embodiments, the device log aggregation table also includes a timestamp field. Different batches of reported data correspond to different timestamps, so that the data integration module 103 and the data visualization module 104 can extract, summarize and analyze the indicator information from multiple time dimensions (minutes, hours, days, months, etc.) according to the timestamp information to obtain the corresponding view.

[0087] In some embodiments, the user basic information table includes a user ID field and multiple user information fields. The data storage module 102 parses and processes the original data packet whose data type ID is user ID, obtains the original sub-packet data containing user information, and then updates the original sub-packet data to the existing user basic information table. Specifically, when the user basic information table contains the basic data of the user ID in the original sub-packet data, the user information field of the basic data is updated; when the user basic information table does not contain the basic data of the user ID in the original sub-packet data, the original sub-packet data is written into the user basic information table as a new data. In this way, the user information stored in the user basic information table can always be the latest state. When the amount of user information is small, it can be uploaded through an original data packet, and a user basic information table of a common table type is created, and the original sub-packet data containing user information is directly updated to the user basic information table. When the amount of user information is large, it is uploaded through multiple original data packets, and a user basic information table of an aggregate table type is created, and multiple original sub-packet data are merged and updated to the user basic information table. When merging, "user ID" is used as the aggregation key for merging.

[0088] The data integration module 103 is used to perform a first data integration based on multiple basic data tables to form multiple conventional views in the data warehouse, and then perform a second data integration based on the multiple conventional views to form at least one materialized view in the data warehouse.

[0089] The data warehouse is a collection of data established to serve specific business topics and to achieve efficient data integration for reporting systems. In this application, the data warehouse is mainly implemented using views, which can greatly reduce storage costs and support flexible updates of current historical data. The data warehouse of this application is divided into three logical layers to store data, namely the original data layer, the data warehouse detail layer, and the data mart layer, as follows:

[0090] 1) Original data layer

[0091] The data in the original data layer is composed of physical tables, which are mainly aggregated from the real-time data reported by energy storage devices. Figure 6 As shown, the original data layer is used to store basic data tables, which include user basic information tables and device log aggregation tables.

[0092] 2) Data warehouse detail layer

[0093] The data at the data warehouse detail layer is composed of conventional views, and the conventional views are obtained by performing a first data integration by the data integration module 103 based on multiple basic data tables. Specifically, the data integration module 103 extracts a group of related field data from one basic data table of the multiple basic data tables to form a first conventional view, and / or extracts a group of related field data from at least two basic data tables of the multiple basic data tables, respectively, and associates and combines the extracted at least two groups of related field data to form a second conventional view. Among them, the number of first conventional views and second conventional views is not limited, and they can be integrated as needed. Each group of related field data includes at least one field and the data content corresponding to the at least one field. Figure 6 As shown, the data integration module 103 performs the first data integration based on the user basic information table and the device log aggregation table to form a user device information view, an energy detail view, an alarm detail view, and an operation and maintenance indicator view. Among them, the user device information view is a combination of a group of related field data in the user basic information table and another group of related field data extracted from the device log aggregation table, which includes the user basic information and the status information of the device to which it belongs.

[0094] In some embodiments, the data integration module 103 is further configured to perform data conversion on the data content of some fields in the extracted set of related field data before forming the first conventional view; and / or to perform data conversion on the data content of some fields in at least one set of related field data in the extracted at least two sets of related field data before forming the second conventional view. Figure 6As shown in the figure, the energy detail view is converted from the energy indicators extracted from the device log aggregation table. The conversion process includes converting the data content of fields such as software version, charging status, and running status into corresponding Chinese, and disassembling the energy of multiple inverters in one record into multiple energy records for subsequent aggregation. The alarm detail view is converted from the alarm indicators extracted from the device log aggregation table. The conversion process includes converting the data content of fields such as alarm type, alarm status, and software version into corresponding Chinese, and disassembling the alarms of multiple inverters in one record into multiple alarms for subsequent aggregation.

[0095] 3) Data mart layer

[0096] The data mart layer is usually directly facing the reporting tool, and the reporting tool directly queries the result table of the data warehouse for display. The result table includes the conventional view of the data warehouse detail layer and the materialized view of the data mart layer. The data mart layer is mainly composed of materialized views, which are obtained by the data integration module 103 through the second data integration based on multiple conventional views. Among them, the second data integration includes associating and aggregating part of the field data of at least two conventional views in the multiple conventional views. Figure 6 As shown in the figure, the data mart layer includes the equipment management materialized view, energy summary materialized view, alarm summary materialized view and operation and maintenance summary materialized view.

[0097] a. Materialized view of equipment management, including basic attributes of users and equipment, operation logs, comparison of key equipment indicators, equipment working status, and equipment composition. The materialized view data is queried from the user equipment information view and updated in real time.

[0098] b. Energy summary materialized view, including cumulative power generation, cumulative power consumption, cumulative purchased power, cumulative savings, PV input power, AC input and output power, DC input and output power, and various voltage and current data. The materialized view data is queried from the energy detail view associated with the user equipment information view, which is summarized from multiple dimensions and can be automatically updated on a scheduled basis by minute, hour, day, month, or year.

[0099] c. Alarm summary materialized view, including alarm type ratio, alarm component ratio, alarm location and number, alarm period trend, alarm level ratio, etc. The materialized view data is queried from the user device information view associated with the alarm detail view, which is summarized from multiple dimensions and can be automatically updated on a scheduled basis by minute, hour, day, month, or year.

[0100] d. Operation and maintenance summary materialized view, including user operation logs (power on and off, networking, charging and discharging, and other function start and stop) and sensor logs (name, location, status, heartbeat time). The materialized view data is queried from the operation and maintenance indicator view, and supports queries by device dimension as well as hour, day, month, and year dimensions.

[0101] The data visualization module 104 is used to obtain corresponding fields of at least one conventional view or at least one materialized view from the data warehouse based on user settings to generate charts for display. The information set by the user includes the selected view, the selected icon display style, the selected field, the set aggregation parameters, etc. In one embodiment, the data visualization module 104 is implemented using the Superset visualization tool.

[0102] The main display method of Superset is Dashboards, which contains many Charts. Each Chart can customize the chart. Select the data table in Chats, and then bind the field data in the data table to the chart for display. The following are four examples of the most common application scenarios for the energy storage IoT platform:

[0103] (1) Equipment management

[0104] Create a dashboard named "Device Management", create a Chart named "Total Number of Devices", import the user device information view, select the graphic as "Title Box", drag the "Device ID" field into the aggregation field of the title box, set the aggregation parameter to "Remove Duplicates", and the total number of devices obtained by aggregation will be displayed in the title box. Create another Chart named "Online Device Proportion", import the user device information view, select the graphic as "Pie Chart", drag the "Device Status" field into the pie chart's Proportion option, and get a pie chart of the online device proportion.

[0105] (2) Energy statistics

[0106] Create a dashboard named "Energy Statistics", create two chats named "Total Power Generation" and "Total Discharge", select "Title Box" for the graphic, drag the "Total Energy" field into the title box graphic, and set the aggregation type to "Sum". Create two chats named "Power Generation Source Proportion" and "Discharge Source Proportion", select "Title Box" for the graphic, drag the "Total Energy" field into the title box graphic, set the aggregation type to "Sum", and then put the "Power Generation Type" field into the dimension attributes of the pie chart.

[0107] (3) Alarm analysis

[0108] Create a dashboard named "Alarm Analysis", create a chart named "Alarm Type Proportion", select "Pie Chart" as the graphic, drag the "Alarm ID" field into the pie chart aggregation property, set the aggregation type to "Count", set the dimension to the "Alarm Type" field, and save the chart. Create a chart named "Alarm Location and Frequency Distribution", select "Bar Chart" as the graphic, drag the "Alarm ID" field into the bar graph, set the aggregation type to "Count", and then put the "Alarm Location" field into the bar graph dimension property, and save the graphic.

[0109] (4) Operation and maintenance analysis

[0110] Create a dashboard named "Operation and Maintenance Analysis", create a chart named "Operation Log", select "Table" for the graph, drag the "Device ID", "Action", and "Timestamp" fields into the table dimension properties, and save the graph. Create a chart named "Sensor Monitoring", select "Table" for the graph, drag the "Device ID", "Location", "Status", "Heartbeat", and "Timestamp" fields into the table dimension properties, and save the graph.

[0111] See also Figure 7 , Figure 7 The flowchart of the data visualization method based on the energy storage Internet of Things platform is exemplified, and the method specifically includes:

[0112] Step S701, obtaining the original data packets uploaded by each energy storage device.

[0113] In one embodiment, the original data packet is encapsulated according to a preset protocol, which includes a device ID field, a data type ID field, a data batch field, a data packet number field, and a reported data content field. Among them, the reported data content includes any one of an inverter monitoring indicator, a battery pack monitoring indicator, and user information, and the data type ID includes any one of an inverter ID, a battery pack ID, or a user ID. When the data type ID is an inverter ID, the reported data content is an inverter monitoring indicator, when the data type ID is a battery pack ID, the reported data content is a battery pack monitoring indicator, and when the data type ID is a user ID, the reported data content is user information.

[0114] In order to save space and transmission bandwidth, the various indicator data of the energy storage device are encoded and stored in hexadecimal format. After encoding, they are encrypted by an algorithm, encapsulated into original data packets, and uploaded to the RabbitMQ message queue under the security gateway, waiting to be consumed by downstream.

[0115] In the process of uploading data, uploading all indicators at a time will consume a lot of network bandwidth and there is a risk of upload failure. Therefore, the data of the same time batch is divided into K packages for upload, and one package is uploaded every m seconds. When all K packages of data are uploaded, the receiving end merges the data of the same time batch into a complete data.

[0116] Step S702: parse and process the original data packet to obtain original sub-packet data, and write the original sub-packet data into multiple basic data tables of the data warehouse according to their categories.

[0117] Specifically, multiple basic data tables are pre-created in the data warehouse for the energy storage Internet of Things platform, including aggregate tables and ordinary tables. Among them, a basic data in the basic data table of the aggregate table is used to store the data content of multiple original sub-package data after merging, and a basic data in the basic data table of the ordinary table is used to store the data content of an original sub-package data.

[0118] In one embodiment, the energy storage Internet of Things platform pre-creates a user basic information table and a device log aggregation table in the data warehouse. Among them, the user basic information table is a common table or an aggregation table, and the device log aggregation table is an aggregation table. The user basic information table is used to store user information, and the device log aggregation table is used to store device information, inverter monitoring indicators, and battery pack monitoring indicators. Each basic data in the device log aggregation table contains all monitoring indicators reported in a batch, and each basic data in the user basic information table contains all user information uploaded in a batch. For example, assuming that the number of all indicators reported in a batch is 200, these 200 indicators are divided into 10 original data packets and uploaded, and each original data packet contains 20 indicators. When creating the device log aggregation table, it is also necessary to create 200 fields of indicator information to receive each sub-package data in turn, and automatically merge each sub-package data into a basic data.

[0119] In one embodiment, Flink is used to subscribe to the original data packets reported in the RabbitMQ topic. Each time an original data packet is received, the data is first decrypted, and then the original data in the original data packet is parsed according to the agreed protocol. After the original data is parsed, the original data is processed in a related manner to obtain the original sub-packet data that can be written to the basic data table. Specifically, first, the diversion processing is performed according to the parsed data type ID, and the original data with the data type ID of the inverter ID or the battery pack ID is used as the first branch flow data, and the original data with the data type ID of the user ID is used as the second branch flow data. Secondly, the first branch flow data is processed in a first manner to obtain the first original sub-packet data, and then the first original sub-packet data is written into the device log aggregation table in the data warehouse; the second branch flow data is processed in a second manner to obtain the second original sub-packet data, and then the second original sub-packet data is written into the user basic information table in the data warehouse.

[0120] In one embodiment, the first processing and the second processing include data conversion processing, such as converting the parsed hexadecimal raw data into decimal raw data. In other embodiments, the first processing also includes data cleaning, such as cleaning data with empty indicators, converting units of values, filling default values ​​in designated fields, etc.

[0121] After obtaining the original sub-package data, the specific method of writing the original sub-package data into multiple basic data tables of the data warehouse according to its category is: use the Stream load micro-batch method provided by Starrocks to write the original sub-package data into the corresponding basic data table. When large quantities of data are stored in real time, the use of the Stream load micro-batch method can not only avoid putting pressure on the performance of the data warehouse, but also improve writing efficiency and stability compared to the conventional insert and batch insert methods. In addition, Stream load writes data using HTTP requests to submit file stream data to the Starrocks database. It has nothing to do with the programming language and can be used as long as the Http protocol is implemented.

[0122] In an embodiment of the present application, by creating an aggregation table in the Starrocks database, multiple pieces of original sub-package data can be automatically merged according to the aggregation key without any coding to obtain a complete piece of basic data to form the basic data of the data warehouse, which greatly saves development costs and time costs. In one embodiment, writing the original sub-package data into multiple basic data tables of the data warehouse according to its category specifically includes: merging multiple pieces of original sub-package data belonging to the same category into a piece of basic data of the aggregation table corresponding to the category based on a preset aggregation key. Figure 5As shown, it is a schematic diagram of automatic aggregation of original sub-package data. The device log aggregation table created in the Starrocks database includes a date field, a device ID field, a data batch field, and multiple indicator information fields. For each original sub-package data whose data type ID is the inverter ID or the battery pack ID, "date + device ID + data batch ID" is used as the aggregation key, and each original sub-package data whose data type ID is the inverter ID or the battery pack ID and the device ID and the data batch ID are the same are merged and written into the same basic data of the device log aggregation table. For example, assuming that the number of all indicators reported in a batch is 200, these 200 indicators are divided into 10 original data packets and uploaded, and each original data packet contains 20 indicators, then a device log aggregation table containing 200 indicator information fields is created in advance to receive data. When the first package of 20 indicators arrives, the corresponding 20 indicators in the table are filled, and the other 180 indicator values ​​in the table are empty. When the second package of 20 indicators arrives, 20 of the remaining 180 indicators will be filled. After filling, there are still 160 indicators left empty, and so on until 10 packages of data are filled. At this time, a complete basic data is formed.

[0123] In some embodiments, the device log aggregation table also includes a timestamp field. Different batches of reported data correspond to different timestamps, so that the energy storage IoT platform can extract, summarize and analyze indicator information from multiple time dimensions (minutes, hours, days, months, etc.) according to the timestamp information to obtain corresponding views.

[0124] In some embodiments, the user basic information table includes a user ID field and multiple user information fields. The energy storage Internet of Things platform parses and processes the original data packet whose data type ID is user ID, obtains the original sub-package data containing user information, and then updates the original sub-package data to the existing user basic information table. Specifically, when the user basic information table contains the basic data of the user ID in the original sub-package data, the user information field of the basic data is updated; when the user basic information table does not contain the basic data of the user ID in the original sub-package data, the original sub-package data is written into the user basic information table as a new data. In this way, the user information stored in the user basic information table can always be the latest state. When the amount of user information is small, it can be uploaded through an original data packet, and a user basic information table of ordinary table type is created, and the original sub-package data containing user information is directly updated to the user basic information table. When the amount of user information is large, it is uploaded through multiple original data packets, and a user basic information table of aggregate table type is created, and multiple original sub-package data are merged and updated to the user basic information table. When merging, "user ID" is used as the aggregation key for merging.

[0125] Step S703: Perform a first data integration based on multiple basic data tables to form multiple regular views in the data warehouse, and then perform a second data integration based on the multiple regular views to form at least one materialized view in the data warehouse.

[0126] The data warehouse in this application is mainly implemented using views, which can greatly reduce storage costs and support flexible updates of current historical data. The data warehouse of this application is divided into three logical layers to store data, namely the original data layer, the data warehouse detail layer, and the data mart layer. The implementation methods of each logical layer can be referred to the description in the energy storage Internet of Things platform embodiment, which will not be repeated here.

[0127] Step S704: Obtain corresponding fields of at least one regular view or at least one materialized view from the data warehouse based on user settings and generate a chart for display.

[0128] In one embodiment, the Superset visualization tool is used to directly connect to the data warehouse result table. The main display method of Superset is Dashboards, which contains many Charts. Each Chart can customize the chart. Select the data table in Chats, and then bind the field data in the data table to the chart for display. Among them, the information set by the user includes the selected view, the selected icon display style, the selected field, the set aggregation parameters, etc. The method of using the Superset visualization tool to connect to the data warehouse result table of this application to generate icons can be referred to the description in the energy storage Internet of Things platform embodiment, which will not be repeated here.

[0129] The energy storage Internet of Things platform provided by the embodiment of the present application solves several key problems existing in the prior art and has the following significant effects: 1) The present application innovatively adopts a data warehouse architecture with views as the core. By combining conventional views and materialized views, the storage cost is significantly reduced, and real-time data query and flexible changes of historical data are supported. This architecture breaks through the limitations of traditional physical tables, improves the response speed and scalability of the data platform, and ensures that efficient query performance can be maintained when the amount of data increases significantly; 2) By realizing zero-code automatic data merging in the data warehouse, the present application greatly reduces the development workload and complexity of the data acquisition module, and improves the development efficiency and maintenance convenience of the system. In addition, the automated data aggregation and cleaning process ensures the accuracy and integrity of the data and reduces the possibility of human errors; 3) Using the Superset visualization tool to directly connect to the data warehouse result table avoids the traditional way of developing Web icons one by one, which not only shortens the development cycle and reduces the complexity of the code, but also can quickly respond to changes in business needs, realize real-time visualization analysis of data, and effectively improve the ability of data display and decision support. In summary, this application successfully built a simple and efficient energy storage IoT data platform through highly integrated automated data processing, innovative storage architecture, and fast-response data visualization, providing powerful data management and analysis support for the energy storage industry.

[0130] The present application also provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 7 method steps.

[0131] The present application also provides a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned data visualization method based on the energy storage Internet of Things platform, for example, executing the above-described Figure 7 method steps.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An energy storage Internet of Things platform, characterized in that: It includes data reporting module, data storage module, data integration module and data visualization module, among which: The data reporting module is used to obtain the original data packets uploaded by each energy storage device and transmit the original data packets to the data storage module; The data storage module is used to pre-create multiple basic data tables, parse and process the original data packets to obtain original sub-packet data and categories of the original sub-packet data, and write the original sub-packet data into the multiple basic data tables of the data warehouse according to their categories; The data integration module is used to perform a first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse, and then perform a second data integration based on the multiple conventional views to form at least one materialized view in the data warehouse; The data visualization module is used to obtain corresponding fields of at least one conventional view or at least one materialized view from the data warehouse based on user settings to generate a chart for display; The first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse includes: Extracting a group of related field data from one of the multiple basic data tables to form a first normal view, and / or extracting a group of related field data from at least two of the multiple basic data tables respectively, and associating and combining the extracted at least two groups of related field data to form a second normal view; The original data packet includes a device ID, a data batch ID and a data type ID, the data type ID includes an inverter ID, a battery pack ID and a user ID, the basic data table includes a device log aggregation table, the device log aggregation table includes a date field, a device ID field, a data batch field and multiple indicator information fields, and the data warehousing module is also used to use "date + device ID + data batch ID" as an aggregation key to merge and write each original sub-package data whose data type ID is an inverter ID or a battery pack ID and whose device ID and data batch ID are the same into the same basic data of the device log aggregation table.

2. The energy storage Internet of Things platform according to claim 1, characterized in that: The basic data table also includes a user basic information table of the aggregate table type. The data warehousing module is also used to merge multiple original sub-package data with data type ID as user ID into a basic data of the user basic information table of the aggregate table type based on a preset aggregation key.

3. The energy storage Internet of Things platform according to claim 1, characterized in that: The basic data table also includes a user basic information table. The data storage module is further used to write the original sub-packaged data whose data type ID is the user ID into the user basic information table.

4. The energy storage Internet of Things platform according to any one of claims 1 to 3, characterized in that: The data integration module is further used to perform data conversion on the data content of some fields in the extracted set of related field data before forming the first conventional view; and / or to perform data conversion on the data content of some fields in at least one set of related field data in the at least two sets of related field data before forming the second conventional view.

5. A data visualization method based on an energy storage Internet of Things platform, characterized in that: The method comprises: Obtain the original data packets uploaded by each energy storage device; Pre-create multiple basic data tables, parse and process the original data packets to obtain original sub-packet data, and write the original sub-packet data into the multiple basic data tables of the data warehouse according to their categories; Performing a first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse, and then performing a second data integration based on the multiple conventional views to form at least one materialized view in the data warehouse; Obtain corresponding fields of at least one regular view or at least one materialized view from the data warehouse based on user settings and generate a chart for display; The first data integration based on the multiple basic data tables to form multiple conventional views in the data warehouse includes: Extracting a group of related field data from one of the multiple basic data tables to form a first conventional view, and / or extracting a group of related field data from at least two of the multiple basic data tables respectively, and associating and combining the extracted at least two groups of related field data to form a second conventional view; The original data packet includes a device ID, a data batch ID and a data type ID, the data type ID includes an inverter ID, a battery pack ID and a user ID, the basic data table includes a device log aggregation table, the device log aggregation table includes a date field, a device ID field, a data batch field and multiple indicator information fields, and the writing of the original sub-packaged data into multiple basic data tables of the data warehouse according to its category includes: Using "date+device ID+data batch ID" as the aggregation key, all original sub-packaged data whose data type ID is inverter ID or battery pack ID and whose device ID and data batch ID are the same are merged and written into the same basic data of the device log aggregation table.

6. The method according to claim 5, characterized in that The basic data table also includes a user basic information table of the aggregate table type, and the writing of the original sub-package data into the multiple basic data tables of the data warehouse according to its category includes: Based on a preset aggregation key, multiple pieces of original sub-packaged data whose data type ID is user ID are merged and written into a basic data of the user basic information table of the aggregation table type.

Images