A charging station IoT data storage method and server

Abstract

This invention relates to the field of data transmission, and particularly to a method and server for storing IoT data in charging stations. The method involves receiving IoT data sent by the charging station through a message middleware and transmitting it to a stream computing module. The stream computing module preprocesses the IoT data to ensure it conforms to preset data specifications. For the IoT data, a metadata query is performed on the database to determine if a target data table associated with the IoT data exists. If the target data table does not exist, a new data table is created in the data lake and the data is written. If the target data table exists, the metadata is used to determine if the structure of the target database table is compatible with the IoT data. If compatible, the data is written to the target data table in the data lake; otherwise, the data table structure is modified before data is written. The database provides metadata storage for the data lake. This invention ensures the validity of data writing and achieves automated management of the storage of IoT data from charging stations in the data lake.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a charging station IoT data storage method and server. Background Technology

[0002] With the rise of the new energy industry, more and more photovoltaic, energy storage and charging projects have been successfully built and put into operation at home and abroad, generating a large amount of IoT data. The operating data of these devices has great analytical value for the operation of the entire site. For example, it can provide early warnings through the status of the equipment, predict the curtailment rate of photovoltaics through the status of energy storage and photovoltaics, and analyze the health of the equipment through the power generation of the equipment, etc.

[0003] During the process of writing IoT data into the data warehouse via stream computing, each type of table requires manual schema creation and corresponding code addition before being updated online. If subsequent iterations of the corresponding module's protocol modify the schema, a manual schema modification and code correction are necessary before the data is written to the database. If the schema is not updated before new data arrives, data loss will occur, affecting data accuracy. Furthermore, the platform handles various data types, including different devices, modules, faults, and logs, requiring data to be written to the database; manual maintenance is cumbersome and inaccurate. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a charging station IoT data storage method and server to realize automated management of charging station IoT data entering and storing in a lake.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for storing IoT data in charging stations, comprising the following steps:

[0007] S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module;

[0008] S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications;

[0009] S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table that is compatible with the IoT data. If it exists, write data to the target data table; otherwise, create a new data table in the data lake and change the table structure to adapt to the IoT data.

[0010] The database provides metadata storage for the data lake.

[0011] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0012] A charging station IoT data storage service terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0013] S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module;

[0014] S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications;

[0015] S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table associated with the IoT data. If the target data table does not exist, create a new data table in the data lake and write the data. If the target data table exists, determine whether the structure of the target database table is compatible with the IoT data based on the metadata. If it is compatible, write the data to the target data table in the data lake. Otherwise, change the data table structure and write the data.

[0016] The database provides metadata storage for the data lake.

[0017] The beneficial effects of this invention are as follows: The charging station IoT data storage method and server of this invention preprocess IoT data through streaming data to make it conform to data specifications and facilitate subsequent processing; in addition, metadata is stored through a database. Before IoT data is stored in the lake, the existence of a target table and whether the target table is suitable are determined by the metadata in the database. If the target table does not exist, it is automatically created and written. If there are differences in the data structure, the data table structure is automatically modified before writing, ensuring the validity of data writing and realizing automated management of charging station IoT data storage in the lake. Attached Figure Description

[0018] Figure 1 This is a flowchart of a charging station IoT data storage method according to an embodiment of the present invention;

[0019] Figure 2 This is a structural diagram of a charging station IoT data storage service terminal according to an embodiment of the present invention;

[0020] Figure 3 This is a data flow diagram illustrating an IoT data storage method for charging stations according to an embodiment of the present invention.

[0021] Figure 4 This is an example diagram illustrating the data processing flow of a charging station IoT data storage method according to an embodiment of the present invention.

[0022] Label Explanation:

[0023] 1. A charging station IoT data storage service client; 2. Processor; 3. Memory. Detailed Implementation

[0024] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0025] Explanation of the name:

[0026]

[0027] Please refer to Figure 1 as well as Figure 3 A method for storing IoT data in charging stations, comprising the following steps:

[0028] S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module;

[0029] S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications;

[0030] S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table that is compatible with the IoT data. If it exists, write data to the target data table; otherwise, create a new data table in the data lake and change the table structure to adapt to the IoT data.

[0031] The database provides metadata storage for the data lake.

[0032] As can be seen from the above description, the beneficial effects of the present invention are as follows: The charging station IoT data storage method and server of the present invention preprocess IoT data through streaming data to make it conform to data specifications and facilitate subsequent processing; in addition, the metadata is stored through a database. Before the IoT data is stored in the lake, it is necessary to determine whether the target table exists and whether the target table is suitable through the metadata in the database. If the target table does not exist, the table is automatically created and written. If there are differences in the data structure, the data table structure is automatically modified before writing, ensuring the validity of data writing and realizing the automated management of charging station IoT data storage in the lake.

[0033] Furthermore, the preprocessing in step S2 includes the following steps:

[0034] S21. Broaden the IoT data and match it with the site information, module information and supplier information configured on the platform;

[0035] S22. Perform field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

[0036] As described above, the data preprocessing of the stream computing module includes widening the IoT data by adding site information, module information, and supplier information to the IoT data, which facilitates data processing and calculation by the real-time computing system; and performing field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

[0037] Furthermore, step S3 is followed by the following steps:

[0038] S31. The IoT data is written into the detailed data layer of the data lake through the task scheduling system, and the data is aggregated in the detailed data layer to obtain aggregated data.

[0039] S32. Write the aggregated data into the data aggregation layer of the data lake, and generate business data according to business needs;

[0040] S33. Write the business data into the application data layer;

[0041] S34. The analysis tool queries the application data layer to obtain and display the business data.

[0042] As described above, the task scheduling system generates data for different purposes based on IoT data and stores it in layers, allowing analysis tools to directly query the application data layer to obtain the corresponding business data, avoiding the need for real-time generation of business data and reducing response time.

[0043] Furthermore, the task scheduling system adopts dolphin scheduling.

[0044] As can be seen from the above description, the task scheduling system adopts dolphin scheduling as a specific embodiment of the present invention.

[0045] Furthermore, step S3 is followed by the following steps:

[0046] The metadata of the IoT data is submitted to the database and cloud storage service for storage, and the compressed file data of the IoT data is submitted to the cloud storage service for storage.

[0047] As can be seen from the above description, adopting a two-stage submission method effectively ensures data accuracy and avoids data loss.

[0048] Please refer to Figure 2 A charging station IoT data storage service terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0049] S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module;

[0050] S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications;

[0051] S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table associated with the IoT data. If the target data table does not exist, create a new data table in the data lake and write the data. If the target data table exists, determine whether the structure of the target database table is compatible with the IoT data based on the metadata. If it is compatible, write the data to the target data table in the data lake. Otherwise, change the data table structure and write the data.

[0052] The database provides metadata storage for the data lake.

[0053] As can be seen from the above description, the beneficial effects of the present invention are as follows: The charging station IoT data storage method and server of the present invention preprocess IoT data through streaming data to make it conform to data specifications and facilitate subsequent processing; in addition, the metadata is stored through a database. Before the IoT data is stored in the lake, it is necessary to determine whether the target table exists and whether the target table is suitable through the metadata in the database. If the target table does not exist, the table is automatically created and written. If there are differences in the data structure, the data table structure is automatically modified before writing, ensuring the validity of data writing and realizing the automated management of charging station IoT data storage in the lake.

[0054] Furthermore, the preprocessing in step S2 includes the following steps:

[0055] S21. Broaden the IoT data and match it with the site information, module information and supplier information configured on the platform;

[0056] S22. Perform field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

[0057] As described above, the data preprocessing of the stream computing module includes widening the IoT data by adding site information, module information, and supplier information to the IoT data, which facilitates data processing and calculation by the real-time computing system; and performing field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

[0058] Furthermore, step S3 is followed by the following steps:

[0059] S31. The IoT data is written into the detailed data layer of the data lake through the task scheduling system, and the data is aggregated in the detailed data layer to obtain aggregated data.

[0060] S32. Write the aggregated data into the data aggregation layer of the data lake, and generate business data according to business needs;

[0061] S33. Write the business data into the application data layer;

[0062] S34. The analysis tool queries the application data layer to obtain and display the business data.

[0063] As described above, the task scheduling system generates data for different purposes based on IoT data and stores it in layers, allowing analysis tools to directly query the application data layer to obtain the corresponding business data, avoiding the need for real-time generation of business data and reducing response time.

[0064] Furthermore, the task scheduling system adopts dolphin scheduling.

[0065] As can be seen from the above description, the task scheduling system adopts dolphin scheduling as a specific embodiment of the present invention.

[0066] Furthermore, step S3 is followed by the following steps:

[0067] The metadata of the IoT data is submitted to the database and cloud storage service for storage, and the compressed file data of the IoT data is submitted to the cloud storage service for storage.

[0068] As can be seen from the above description, adopting a two-stage submission method effectively ensures data accuracy and avoids data loss.

[0069] The present invention provides a charging station IoT data storage method and server, which is applicable to scenarios that require the storage of a large amount of charging station IoT data in a lake.

[0070] Please refer to Figure 1 and Figure 3 Embodiment 1 of the present invention is as follows:

[0071] A method for storing IoT data in charging stations, comprising the following steps:

[0072] S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module.

[0073] In this embodiment, Pulsar (middleware) is standardized upstream: after comparing and verifying all received IoT data according to the object model configured on the platform, it is written to the source topic and then sent to the downstream Flink (stream computing) topic.

[0074] S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications;

[0075] The preprocessing in step S2 includes the following steps:

[0076] S21. Broaden the IoT data and match it with the site information, module information and supplier information configured on the platform;

[0077] S22. Perform field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

[0078] In this embodiment, Flink (stream computing) broadens the data during data reading, matching it with platform-configured site information, module information, and supplier information. Then, this data is used to perform standardized field assignments and calculations based on the modules, such as charging and discharging amounts, ensuring the data conforms to platform-wide standards.

[0079] S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table that is compatible with the IoT data. If it exists, write data to the target data table; otherwise, create a new data table in the data lake and change the table structure to adapt to the IoT data.

[0080] The database provides metadata storage for the data lake.

[0081] In this embodiment, Hive Catalog (a big data component) is integrated when writing data to Iceberg (the data lake) to manage metadata, which is then uniformly stored in MySQL (the database). When data is written through Iceberg, it checks if the corresponding metadata information exists in MySQL (the database). If not, a new table is created. Then, it compares the schema (table structure) of the data being written with the schema (table structure) of MySQL (the database) to see if there are any changes. If not, the schema is created or modified, and then another data write is performed, thus achieving dynamic schema changes.

[0082] Step S3 is followed by the following steps:

[0083] S31. The IoT data is written into the detailed data layer of the data lake through the task scheduling system, and the data is aggregated in the detailed data layer to obtain aggregated data.

[0084] In this embodiment, the task scheduling system adopts dolphin scheduling.

[0085] S32. Write the aggregated data into the data aggregation layer of the data lake, and generate business data according to business needs;

[0086] S33. Write the business data into the application data layer;

[0087] S34. The analysis tool queries the application data layer to obtain and display the business data.

[0088] In this embodiment, after the data enters the data lake, the dolphin scheduler executes a scheduling task to first write the result data to the data lake's dwd (detailed data layer). Then, simple aggregation is performed based on the dwd (detailed data layer), such as generating data for one day or generating aggregated data between sites, and then written to the dws (data aggregation layer). Finally, the data ads (application data layer) that the application needs to display is generated according to business requirements.

[0089] Data can be viewed and analyzed using the OLAP analytics tool (Doris). Thanks to the layered data warehouse calculations, now you only need to use Doris (OLAP analytics tool) to query a single record in the charging table of the ADS (Application Data Layer) for the current day or month to obtain the charging volume and charging revenue of that site for that day or month.

[0090] Step S3 is followed by the following steps:

[0091] The metadata of the IoT data is submitted to the database and cloud storage service for storage, and the compressed file data of the IoT data is submitted to the cloud storage service for storage.

[0092] In this embodiment, to ensure data accuracy and prevent data loss, a two-phase commit (2PC) is enabled. First, metadata is committed to OBS (Huawei Cloud Storage) and MySQL. Second, compressed file data is committed to OBS (Huawei Cloud Storage) to ensure data transactionality and idempotency, thereby guaranteeing that data will not be lost and that there will be no duplicate data.

[0093] Please refer to Figure 2 Embodiment two of the present invention is as follows:

[0094] A charging station IoT data storage service terminal 1 includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, it implements the steps in the charging station IoT data storage method described in Embodiment 1 above.

[0095] In summary, the IoT data storage method and server provided by this invention preprocess IoT data through streaming data to ensure it conforms to data specifications and facilitates subsequent processing. Furthermore, metadata is stored in a database. Before IoT data is stored in the lake, the database metadata is used to determine whether a target table exists and whether the target table is compatible. If the target table does not exist, it is automatically created and written. If there are differences in the data structure, the data table structure is automatically modified before writing, ensuring the validity of data writing and achieving automated management of IoT data storage in charging stations.

[0096] This invention supports dynamic schema changes and flexible data manipulation; because data writing is compressed, it greatly saves storage space; and metadata management is more concise and objective.

[0097] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for storing IoT data in a charging station, characterized in that, Including the following steps: S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module; S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications; S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table associated with the IoT data. If the target data table does not exist, create a new data table in the data lake and write the data. If the target data table exists, determine whether the structure of the target data table is compatible with the IoT data based on the metadata. If it is compatible, write the data to the target data table in the data lake. Otherwise, change the data table structure and write the data. The database provides metadata storage for the data lake; Step S3 is followed by the following steps: S31. The IoT data is written to the detailed data layer of the data lake in real time through the task scheduling system, and the data is aggregated in the detailed data layer to obtain aggregated data. S32. Write the aggregated data into the data aggregation layer of the data lake, and generate business data according to business needs; S33. Write the business data into the application data layer; S34. When the analysis tool needs to perform data analysis, the application data layer provides the analysis tool with the pre-calculated business data.

2. The charging station IoT data storage method according to claim 1, characterized in that, The preprocessing in step S2 includes the following steps: S21. Broaden the IoT data and match it with the site information, module information and supplier information configured on the platform; S22. Perform field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

3. The charging station IoT data storage method according to claim 1, characterized in that, The task scheduling system uses dolphin scheduling.

4. The charging station IoT data storage method according to claim 1, characterized in that, Step S3 is followed by the following steps: The metadata of the IoT data is submitted to the database and cloud storage service for storage, and the compressed file data of the IoT data is submitted to the cloud storage service for storage.

5. A charging station IoT data storage service terminal, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: S1. Receive IoT data sent by the power station through the message middleware and send it to the stream computing module; S2. The IoT data is preprocessed by the stream computing module to make the IoT data conform to the preset data specifications; S3. For the IoT data, perform a metadata query on the database to determine whether there is a target data table associated with the IoT data. If the target data table does not exist, create a new data table in the data lake and write the data. If the target data table exists, determine whether the structure of the target data table is compatible with the IoT data based on the metadata. If it is compatible, write the data to the target data table in the data lake. Otherwise, change the data table structure and write the data. The database provides metadata storage for the data lake; Step S3 is followed by the following steps: S31. The IoT data is written to the detailed data layer of the data lake in real time through the task scheduling system, and the data is aggregated in the detailed data layer to obtain aggregated data. S32. Write the aggregated data into the data aggregation layer of the data lake, and generate business data according to business needs; S33. Write the business data into the application data layer; S34. When the analysis tool needs to perform data analysis, the application data layer provides the analysis tool with the pre-calculated business data.

6. The IoT data storage service terminal for charging stations according to claim 5, characterized in that, The preprocessing in step S2 includes the following steps: S21. Broaden the IoT data and match it with the site information, module information and supplier information configured on the platform; S22. Perform field assignment calculations on the IoT data to make the IoT data conform to the preset data specifications.

7. The IoT data storage service terminal for charging stations according to claim 5, characterized in that, The task scheduling system uses dolphin scheduling.

8. The IoT data storage service terminal for charging stations according to claim 5, characterized in that, Step S3 is followed by the following steps: The metadata of the IoT data is submitted to the database and cloud storage service for storage, and the compressed file data of the IoT data is submitted to the cloud storage service for storage.

Images