A method and terminal for supplementing offline data transmission of IoT platform

Abstract

The present invention discloses a method for supplementing offline data transmission of an Internet of Things platform, which monitors the MQTT message for uploading offline files to the Internet of Things platform, generates a first file attribute corresponding to the offline file; when recovering offline data, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the Internet of Things platform, a local file is obtained and a corresponding second file attribute is generated; according to the second file attribute, the corresponding local file is decompressed, and the task of recovering data to the MQTT database of the Internet of Things platform is executed based on the decompressed result, and a task abnormality recovery mechanism is established at the same time. The present invention submits files through MQTT messages to avoid occupying a large amount of network broadband resources, relies on the first file attribute and the second file attribute to realize breakpoint continuation during the writing process, combines the use of the task abnormality recovery mechanism, avoids repeated data writing and affects the online operation status of the Internet of Things platform, and quickly and completely completes offline data recovery.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to an offline data supplementary transmission method and a terminal for an Internet of Things platform. Background Art

[0002] The IoT platform refers to a comprehensive platform that uses the Internet of Things (IOT) technology to manage and optimize new energy systems. Since the station equipment often fails to report IoT data due to updates and upgrades, network fluctuations, manual offline, etc., the IoT platform is often unable to receive the full amount of data from the station, so offline data supplementation came into being. With the offline supplementation function, the new energy station can save the IoT data in local storage when it is offline and disconnected from the network; then, the operator can supplement the missing data to the IoT platform through offline supplementation at an appropriate time in the future, thereby improving the IoT platform's monitoring, analysis and prediction of offline stations.

[0003] The IoT data received by the IoT platform must be cleaned, converted, and formatted before being stored for analysis. Therefore, the data uploaded offline must be restored from the IoT platform entrance and stored based on the data flow specification. However, if the site is offline for a long time and the submitted offline file is too large, a large amount of network bandwidth will be occupied when submitting the offline file, which may occupy the channel for the online site to report data, and in serious cases, it will cause service memory overflow and downtime;

[0004] In addition, when uploading offline files, they are submitted, decompressed and written to the database of the IoT platform as a complete file. If the service fails to write due to external reasons during the submission of files or data recovery, we must re-read the file for writing, which will inevitably lead to duplicate data writing and dirty data in the database. Summary of the invention

[0005] The technical problem to be solved by the present invention is to propose an offline data retransmission method and terminal for an IoT platform, which can quickly and completely complete offline data recovery without affecting the online operation status of the IoT platform.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for supplementing offline data transmission of an IoT platform comprises the following steps:

[0008] S1. Listen to the MQTT message for uploading an offline file to the IoT platform, and generate a first file attribute corresponding to the offline file;

[0009] S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, a local file is obtained and a corresponding second file attribute is generated;

[0010] S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, the task of restoring data to the MQTT database of the Internet of Things platform is performed, and a task exception recovery mechanism is established.

[0011] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0012] An IoT platform offline data retransmission terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0013] S1. Listen to the MQTT message for uploading an offline file to the IoT platform, and generate a first file attribute corresponding to the offline file;

[0014] S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, a local file is obtained and a corresponding second file attribute is generated;

[0015] S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, the task of restoring data to the MQTT database of the Internet of Things platform is performed, and a task exception recovery mechanism is established.

[0016] The beneficial effects of the present invention are: providing an offline data retransmission method and terminal for an Internet of Things platform, submitting files through MQTT messages to avoid occupying a large amount of network broadband resources, and decoupling the process of submitting offline files to the Internet of Things platform from decompressing offline files and recovering data, relying on the first file attribute and the second file attribute to achieve breakpoint resumption during the writing process, combining with the use of a task exception recovery mechanism to avoid repeated data writing and affecting the online operation status of the Internet of Things platform, and quickly and completely completing offline data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of the steps of a method for supplementing offline data transmission of an Internet of Things platform according to the present invention;

[0018] Figure 2 The present invention relates to a system block diagram of an offline data retransmission terminal for an Internet of Things platform.

[0019] Description of labels:

[0020] 1. An offline data retransmission terminal for an Internet of Things platform; 2. A memory; 3. A processor. DETAILED DESCRIPTION

[0021] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0022] Please refer to Figure 1 , a method for supplementing offline data transmission of an IoT platform, comprising the following steps:

[0023] S1. Listen to the MQTT message for uploading an offline file to the IoT platform, and generate a first file attribute corresponding to the offline file;

[0024] S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, a local file is obtained and a corresponding second file attribute is generated;

[0025] S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, the task of restoring data to the MQTT database of the Internet of Things platform is performed, and a task exception recovery mechanism is established.

[0026] From the above description, it can be seen that the beneficial effects of the present invention are: providing a method for offline data retransmission of an Internet of Things platform, submitting files through MQTT messages to avoid occupying a large amount of network broadband resources, and decoupling the process of submitting offline files to the Internet of Things platform from decompressing offline files and recovering data, relying on the first file attribute and the second file attribute to achieve breakpoint resumption during the writing process, combined with the use of task exception recovery mechanism, to avoid repeated data writing and affecting the online operation status of the Internet of Things platform, and quickly and completely complete offline data recovery.

[0027] Furthermore, it also includes:

[0028] S4. According to the portion of the restored data corresponding to the station, correct the index data of the station.

[0029] From the above description, it can be seen that data recalculation is triggered immediately after data recovery is completed, that is, the corresponding data in the indicator data of the station is recalculated based on the recovered data, so as to eliminate the deviation of the indicator data caused by the lack of offline data and improve the monitoring, analysis and prediction of offline stations by the Internet of Things platform.

[0030] Furthermore, the step S3 further includes:

[0031] While restoring data, the writing speed of data to the MQTT database of the IoT platform is controlled to be lower than a preset speed threshold.

[0032] From the above description, it can be seen that when the data is restored offline, the flow is limited to limit the writing speed of data to the MQTT database, such as the number of messages written per second, so as to avoid large quantities of data being written to MQTT, resulting in blocking of normal message channels, and avoid excessive network bandwidth and disk IO usage, which affects the normal data reporting of online new energy stations.

[0033] Furthermore, before step S1, the following steps are also included:

[0034] S0. Storing the offline file in the cloud server object storage;

[0035] The acquiring of the offline file according to the first file attribute specifically includes:

[0036] The offline file is serially read from the object storage according to the first file attribute.

[0037] From the above description, it can be seen that the offline files submitted by the operator are stored on an object basis, which has high availability, scalability and fault tolerance. At the same time, the offline files are read serially to the local storage during recovery, and abnormal resource fluctuations will not be caused by batch submission of offline files.

[0038] Furthermore, the task abnormality recovery mechanism specifically includes:

[0039] While restoring the data, the progress of the task is recorded and a checkpoint mechanism is introduced, so that the task can be restored through the checkpoint when the task is abnormally restarted.

[0040] From the above description, we can see that the checkpoint mechanism is introduced while performing data recovery. When the task is restarted abnormally, the number of rows that have been successfully recovered can be obtained through the checkpoint to ensure that data will not be recovered repeatedly.

[0041] Please refer to Figure 2 , an IoT platform offline data retransmission terminal 1, comprising a memory 2, a processor 3 and a computer program stored in the memory 2 and executable on the processor 3, wherein the processor 3 implements the following steps when executing the computer program:

[0042] S1. Listen to the MQTT message for uploading an offline file to the IoT platform, and generate a first file attribute corresponding to the offline file;

[0043] S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, a local file is obtained and a corresponding second file attribute is generated;

[0044] S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, the task of restoring data to the MQTT database of the Internet of Things platform is performed, and a task exception recovery mechanism is established.

[0045] From the above description, it can be seen that the beneficial effects of the present invention are: providing an offline data retransmission terminal for an Internet of Things platform, submitting files through MQTT messages to avoid occupying a large amount of network broadband resources, and decoupling the process of submitting offline files to the Internet of Things platform from decompressing offline files and recovering data, relying on the first file attribute and the second file attribute to achieve breakpoint resumption during the writing process, combined with the use of task exception recovery mechanism, to avoid repeated data writing and affecting the online operation status of the Internet of Things platform, and quickly and completely complete offline data recovery.

[0046] Furthermore, it also includes:

[0047] S4. According to the portion of the restored data corresponding to the station, correct the index data of the station.

[0048] From the above description, it can be seen that data recalculation is triggered immediately after data recovery is completed, that is, the corresponding data in the indicator data of the station is recalculated based on the recovered data, so as to eliminate the deviation of the indicator data caused by the lack of offline data and improve the monitoring, analysis and prediction of offline stations by the Internet of Things platform.

[0049] Furthermore, the step S3 further includes:

[0050] While restoring data, the writing speed of data to the MQTT database of the IoT platform is controlled to be lower than a preset speed threshold.

[0051] From the above description, it can be seen that when the data is restored offline, the flow is limited to limit the writing speed of data to the MQTT database, such as the number of messages written per second, so as to avoid large quantities of data being written to MQTT, resulting in blocking of normal message channels, and avoid excessive network bandwidth and disk IO usage, which affects the normal data reporting of online new energy stations.

[0052] Furthermore, before step S1, the following steps are also included:

[0053] S0. Storing the offline file in the cloud server object storage;

[0054] The acquiring of the offline file according to the first file attribute specifically includes:

[0055] The offline file is serially read from the object storage according to the first file attribute.

[0056] From the above description, it can be seen that the offline files submitted by the operator are stored on an object basis, which has high availability, scalability and fault tolerance. At the same time, the offline files are read serially to the local storage during recovery, and abnormal resource fluctuations will not be caused by batch submission of offline files.

[0057] Furthermore, the task abnormality recovery mechanism specifically includes:

[0058] While restoring the data, the progress of the task is recorded and a checkpoint mechanism is introduced, so that the task can be restored through the checkpoint when the task is abnormally restarted.

[0059] From the above description, we can see that the checkpoint mechanism is introduced while performing data recovery. When the task is restarted abnormally, the number of rows that have been successfully recovered can be obtained through the checkpoint to ensure that data will not be recovered repeatedly.

[0060] Please refer to Figure 1 , Embodiment 1 of the present invention is:

[0061] Before the description, the technical terms involved in this embodiment are explained as follows:

[0062] MQTT, the full name of which is Message Queue Telemetry Transport, is a message protocol based on the publish / subscribe paradigm under the ISO standard. It works on the TCP / IP protocol family and is a publish / subscribe message protocol designed for remote devices with low hardware performance and poor network conditions.

[0063] MySQL CDC (Change Data Capture) is a technology used to capture data changes in the database in real time, including insert (INSERT), update (UPDATE) and delete (DELETE) operations. MySQL CDC allows applications or systems to obtain these changes immediately when data changes, which is very useful for scenarios such as data synchronization, real-time analysis, event-driven architecture and data replication.

[0064] Apache Flink's ValueState is part of the Flink State Backend, and is mainly used to save state information for each key in stream processing jobs. ValueState allows you to store a value for each key, which can be any data type that implements the serialization interface. ValueState is mainly used in scenarios where you need to maintain state between events, such as cumulative counts, latest state updates, or window operations.

[0065] Flink stream computing is an "event-triggered" computing mode, and the trigger source is the unbounded streaming data mentioned above. Once new streaming data enters the stream computing, the stream computing immediately initiates and performs a computing task, so the entire stream computing is a continuous computing.

[0066] Flink checkpoint, Apache Flink's Checkpointing mechanism is a core component of its high fault tolerance and Exactly-Once semantics. Checkpointing allows Flink to save snapshots of intermediate states in stream processing jobs, so that when a system failure occurs, the state can be restored from the most recent Checkpoint, thus avoiding re-running the entire job from scratch, greatly improving the system's elasticity and processing efficiency.

[0067] A method for supplementing offline data transmission on an IoT platform, such as Figure 1 As shown, the following steps are included:

[0068] S0. Store offline files in the cloud server object storage.

[0069] S1. Monitor an MQTT message for uploading an offline file to the IoT platform, and generate a first file attribute corresponding to the offline file.

[0070] In this embodiment, the offline files submitted by the operator will be saved in the cloud server object storage, specifically Huawei Cloud's object storage (OBS), and then trigger an MQTT message for offline file upload to the IoT platform, avoiding abnormal resource occupation caused by uploading files through HTTP requests. The cloud service of the IoT platform then monitors the MQTT message and generates the first file attribute corresponding to the offline file, which generally includes attributes such as file bucket, file path, file type, status, and number of retries. The first file attribute is only used to identify the offline file that needs to be uploaded, and does not play the role of offline file decompression identification.

[0071] S2. When performing offline data recovery, obtain the offline file according to the first file attribute, decompress the offline file and save it to the local storage of the IoT platform, obtain the local file and generate the corresponding second file attribute.

[0072] In this embodiment, two stream computing tasks (Flink) are established to be responsible for the entire offline data recovery process, one is the offline file loading task (OLinkLoad task), corresponding to step S2, and the other is the offline file decompression and reading task (OLinkParse task), corresponding to step S3;

[0073] OLinkLoad (offline file loading task) will monitor the newly written data of the database through MysqlCDC, and then read the file in the object storage through the file path in the first file attribute, and write it to the local storage; specifically, this task is to serially read the cloud file and write it to the local storage, so there will be no abnormal fluctuation of resources due to the batch submission of offline files;

[0074] In addition, when OLinkLoad decompresses the data and writes it locally, it will generate the second file attribute of the local file in the database (Mysql) through the file tag, which generally includes attributes such as file size, number of rows, and MQTT topic written; the second file attribute is used to identify the offline file to be decompressed.

[0075] S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, the task of restoring data to the MQTT database of the IoT platform is executed, and a task exception recovery mechanism is established.

[0076] In this embodiment, OLinkParse will also obtain the local file corresponding to the second file attribute by listening to MysqlCDC, and then decompress the file to restore the data to the MQTT of the IoT platform; during recovery, the writing speed of data to the MQTT database of the IoT platform is controlled to be lower than the preset speed threshold. For example, based on the stress test of the IoT platform, assuming that the number of messages processed per second (TPS) is 100,000, the upper limit of restoring data written to the MQTT topic is limited to 30,000 per second.

[0077] In addition, when restoring data, the progress of the task is recorded and a checkpoint mechanism is introduced. When the task is restarted abnormally, the task is restored through the checkpoint. Specifically, when restoring data, the number of restored rows is recorded through Flink's memory object valueState. In this way, based on Flink's Checkpoint recovery mechanism, the number of rows that have been successfully restored can be obtained through the checkpoint when the task is restarted abnormally, ensuring that data will not be restored repeatedly.

[0078] S4. Correct the index data of the station according to the portion of the recovered data corresponding to the station.

[0079] In this embodiment, while the data is being restored, it is recorded which sites have restored the data at which time, and finally the corresponding time between the sites and the data is counted and sent to the task scheduling center of the IoT platform via an MQTT message. After the task scheduling center of the IoT platform receives the message for recalculating the indicators, it will call the recalculation interface to restore the indicators.

[0080] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0081] An IoT platform offline data retransmission terminal 1 includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, an IoT platform offline data retransmission method according to embodiment 1 is implemented.

[0082] In summary, the present invention provides a method and terminal for supplementing offline data transmission of an IoT platform, which submits files through MQTT messages to avoid occupying a large amount of network broadband resources, and decouples the process of submitting offline files to the IoT platform from decompressing offline files and recovering data, and relies on the first file attribute and the second file attribute to achieve breakpoint resumption during the writing process, and combines the use of task abnormality recovery mechanism to avoid repeated data writing and affecting the online operation status of the IoT platform, and quickly and completely complete offline data recovery. The writing process is subjected to current limiting processing to prevent normal message channels from being blocked, and then the indicator data of the station is immediately corrected based on the recovered data, so as to improve the monitoring, analysis and prediction of offline stations by the IoT platform.

[0083] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for supplementing offline data transmission of an IoT platform, characterized in that: The steps include: S0, storing offline files in the cloud server object storage; S1. Listen to the MQTT message for uploading offline files to the IoT platform, and generate a first file attribute corresponding to the offline file. The first file attribute is only used to identify the offline file to be uploaded, but does not play the role of offline file decompression identification; S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, the local file is obtained and a corresponding second file attribute is generated, where the second file attribute is used to identify the offline file to be decompressed; S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, a task of restoring data to the MQTT database of the IoT platform is performed, and a task abnormality recovery mechanism is established; The acquiring of the offline file according to the first file attribute specifically includes: serially reading the offline file from the object storage according to the first file attribute; The task abnormality recovery mechanism specifically includes: While restoring data, the progress of the task is recorded and a checkpoint mechanism is introduced, so that the task can be restored through the checkpoint when the task is abnormally restarted; Create two stream computing tasks, one for loading offline files and the other for decompressing and reading offline files. Step S2 includes: The offline file loading task will monitor the newly written data in the database through MysqlCDC, and then read the file in the object storage through the file path in the first file attribute, and write it to the local storage; after the offline file loading task decompresses the data and writes it to the local, it will generate the second file attribute of the local file in the database through the file tag; Step S3 includes: The offline file decompression and reading task will also obtain the local file corresponding to the second file attribute by monitoring MysqlCDC, and then decompress the file and restore the data to the MQTT of the IoT platform.

2. The method for supplementing offline data transmission of an IoT platform according to claim 1, characterized in that: Also includes: S4. According to the portion of the restored data corresponding to the station, correct the index data of the station.

3. The method for supplementing offline data transmission of an IoT platform according to claim 1, characterized in that: The step S3 further comprises: While restoring data, the writing speed of data to the MQTT database of the IoT platform is controlled to be lower than a preset speed threshold.

4. An IoT platform offline data retransmission terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S0, storing offline files in the cloud server object storage; S1. Listen to the MQTT message for uploading offline files to the IoT platform, and generate a first file attribute corresponding to the offline file. The first file attribute is only used to identify the offline file to be uploaded, but does not play the role of offline file decompression identification; S2. When performing offline data recovery, the offline file is obtained according to the first file attribute, the offline file is decompressed and saved to the local storage of the IoT platform, the local file is obtained and a corresponding second file attribute is generated, where the second file attribute is used to identify the offline file to be decompressed; S3. According to the second file attribute, the corresponding local file is decompressed, and based on the decompression result, a task of restoring data to the MQTT database of the IoT platform is performed, and a task abnormality recovery mechanism is established; The acquiring of the offline file according to the first file attribute specifically includes: serially reading the offline file from the object storage according to the first file attribute; The task abnormality recovery mechanism specifically includes: While restoring data, the progress of the task is recorded and a checkpoint mechanism is introduced, so that the task can be restored through the checkpoint when the task is abnormally restarted; Create two stream computing tasks, one for loading offline files and the other for decompressing and reading offline files. Step S2 includes: The offline file loading task will monitor the newly written data in the database through MysqlCDC, and then read the file in the object storage through the file path in the first file attribute, and write it to the local storage; after the offline file loading task decompresses the data and writes it to the local, it will generate the second file attribute of the local file in the database through the file tag; Step S3 includes: The offline file decompression and reading task will also obtain the local file corresponding to the second file attribute by monitoring MysqlCDC, and then decompress the file and restore the data to the MQTT of the IoT platform.

5. The IoT platform offline data retransmission terminal according to claim 4, characterized in that: Also includes: S4. According to the portion of the restored data corresponding to the station, correct the index data of the station.

6. The IoT platform offline data retransmission terminal according to claim 4, characterized in that: The step S3 further comprises: While restoring data, the writing speed of data to the MQTT database of the IoT platform is controlled to be lower than a preset speed threshold.

Images