Method, system, device, medium and product for collecting and reporting data continuously transmitted after network disconnection

Abstract

The invention belongs to the technical field of data transmission, and particularly relates to an off-network continuous transmission data acquisition and reporting method, system and device, a medium and a product, and the method comprises the steps: collecting data in real time through a remote communication interface; monitoring a network connection state of the data receiving and transferring service, and when a network is disconnected, independently storing each packet of collected data as a single file named by a timestamp; maintaining a set file of single file names, and recording name prefixes of all to-be-reported single files; after the network is recovered, single files to be reported are retrieved according to the set file and are reported to a cloud service in sequence; and after the single file is reported successfully, deleting the successfully reported single file, and removing the corresponding name prefix from the set file. According to the method, all the to-be-reported data files are tracked through the file name set file, and repeated reporting or data loss is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and in particular relates to a method, system, device, medium and product for collecting and reporting data on a network-broken and resumable basis. Background Art

[0002] In modern society, intelligent data collection equipment is widely used to realize remote data collection and monitoring of power equipment in real time. Wireless IoT technology is used to realize remote monitoring, and professional algorithms and models are used to diagnose and predict faults. This targeted analysis and judgment can not only provide fault information in real time, but also provide corresponding maintenance and maintenance suggestions based on historical data and trend analysis. Through in-depth analysis and accurate judgment, the remote diagnosis system can greatly improve the speed and accuracy of power equipment troubleshooting, reduce maintenance costs and downtime. However, the premise of ensuring remote monitoring is that the collection equipment itself can be online in the cloud.

[0003] In industrial automation, the Internet of Things, and other remote monitoring systems, devices send data to cloud servers through the network. However, in practical applications, network interruptions are inevitable. Traditional solutions often focus on resending data after the network is restored, while ignoring the details of how to effectively manage and transmit data during network interruptions.

[0004] For acquisition equipment, data acquisition is normal when the equipment is powered on, but data transmission is interrupted due to abnormal network signals. The acquisition equipment needs to store real-time data during the network disconnection period. Conventional single-file storage of disconnected data has high requirements on the memory and performance of the acquisition equipment, which greatly increases the overall cost and hardware performance of the equipment. In order to solve this confusion, improve reading and writing efficiency and save hardware costs, we have proposed an efficient data monitoring and reading and writing continuation method. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for collecting and reporting data after network disconnection and resuming transmission, comprising:

[0006] Real-time data collection via remote communication interface;

[0007] Monitor the network connection status of data receiving and transfer services. When the network is disconnected, store each packet of collected data independently as a single file named with a timestamp;

[0008] Maintain a collection file of single file names, recording the name prefixes of all single files to be reported;

[0009] When the network is restored, the individual files to be reported are retrieved according to the collection file, and reported to the cloud service in sequence;

[0010] After a single file is successfully reported, delete the successfully reported single file and remove the corresponding name prefix from the collection file.

[0011] Furthermore, the remote communication interface adopts RS485 protocol.

[0012] Furthermore, the data receiving and transferring service includes: northbound EMQ service.

[0013] Furthermore, the timestamp is generated by the System.currentTimeMillis method, and the file name format is "xxx.json", where "xxx" is the timestamp of the file and ".json" is the extension of the file.

[0014] Furthermore, the monitoring data receiving and the network connection status of the transfer service include:

[0015] Record the first timestamp when data reception and relay services are disconnected to the device memory;

[0016] generating a second timestamp when it is detected that the data reception and the transit service connection are reestablished;

[0017] If the second timestamp is greater than the first timestamp, it is determined that the network is restored.

[0018] Furthermore, after the single file is successfully reported, the successfully reported single file is deleted, and the corresponding name prefix is ​​removed from the collection file, including:

[0019] First delete the reported single file;

[0020] After a single file is deleted successfully, delete the corresponding name prefix in the collection file;

[0021] If an exception occurs after a single file is deleted and the name prefix in the collection file is not deleted, a resume prompt is generated based on the collection file, and the name prefix corresponding to the single file in the collection file is automatically deleted.

[0022] The present invention provides a data collection and reporting system for network disconnection and resuming transmission, comprising:

[0023] Acquisition module: used to collect data in real time through the remote communication interface;

[0024] Monitoring module: used to monitor the network connection status of data receiving and transfer services. When the network is disconnected, each packet of collected data is stored independently as a single file named with a timestamp;

[0025] Collection module: used to maintain the collection file of single file names, recording the name prefixes of all single files to be reported;

[0026] Retrieval module: used to retrieve the individual files to be reported according to the collection file after the network is restored, and report them to the cloud service in sequence;

[0027] Reporting module: After a single file is successfully reported, it is used to delete the single file that has been successfully reported and remove the corresponding name prefix from the collection file.

[0028] The present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and is characterized in that when the computer program is executed by the processor, the processor executes the steps of the network-disconnected data collection and reporting method.

[0029] The present invention provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the processor executes the steps of the network-disconnected and resumable data collection and reporting method.

[0030] The present invention provides a computer program product, including a computer program, characterized in that the computer program is executed by a processor to execute the network-disconnected and resumable data collection and reporting method.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] The present invention uses a single file to store a single package of data, reduces the reading of a single large file and prevents performance consumption. The data collection and storage of the present invention are not affected by the network, ensuring the integrity of the data. The present invention can report data after the network is restored without manual intervention. The present invention tracks all data files to be reported through a file name collection file to avoid repeated reporting or data loss.

[0033] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flow chart of a method for collecting and reporting data after network interruption and resuming transmission provided in an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram showing a flow of a method for collecting and reporting data after network interruption and continued transmission provided in an embodiment of the present invention is shown;

[0037] Figure 3 A schematic diagram of a cache file search process of a method for collecting and reporting data during network-resumable transmission provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 and Figure 2 The embodiment of the present invention provides a method for collecting and reporting data after network disconnection and resuming transmission, comprising:

[0040] S1: real-time data collection through remote communication interface;

[0041] This embodiment uses RS485 (a remote communication interface standard, its full name is "Recommended Standard 485") for data collection operation. During the data collection process, the device is not restricted by the network signal, and as long as the device is in a normal power-on state, data collection can continue.

[0042] The RS485 communication standard uses balanced transmission and differential reception, and has the ability to suppress common-mode interference. When the communication distance is in the range of tens of meters to thousands of meters, the RS485 serial bus standard is usually widely used for communication.

[0043] S2: Monitor the network connection status of data receiving and transfer services. When the network is disconnected, each packet of collected data is stored independently as a single file named with a timestamp;

[0044] In this embodiment, when it is detected that the service connection status of the northbound EMQ (i.e., Erlang MQTT Broker) is disconnected, each data packet will be stored in a specific format. Specifically, each packet of data will be stored as an independent single file. The naming rule of the single file is determined according to the timestamp corresponding to the data collection moment. The file name is: xxx.json. In this file name structure, the "xxx" part accurately corresponds to the time information of the corresponding data collection action, and ".json" is clearly the extension of the file, so as to ensure the orderly storage and subsequent traceability of the data, and facilitate the rapid positioning and retrieval of data files collected in a specific period of time according to the timestamp information in the file name when needed.

[0045] In the timestamp generation phase, the System.currentTimeMillis() method is used to clearly define it. The value returned by this method is the number of milliseconds that have passed since 00:00:00 on January 1, 1970, Coordinated Universal Time (UTC). It should be noted that this return value itself does not contain time zone related information. It simply represents a specific point in time, that is, the number of milliseconds measured from the start of UTC time. This value is absolute and will not change due to different local time zones.

[0046] In the IoT architecture system, northbound EMQ services are provided by EMQ. Its core function is to build a data path that starts from the device terminal and extends upstream to enable data to flow smoothly to target areas such as the application layer and the cloud. It shoulders the key responsibility of ensuring upstream data transmission, so that the data generated by the device can be accurately and efficiently received by the application layer and the cloud for subsequent processing, providing strong support for the data flow and collaborative operation of the entire IoT system.

[0047] S3: Maintains a collection file of single file names, recording the name prefixes of all single files to be reported;

[0048] This embodiment constructs a collection file that covers the single file name prefixes corresponding to all data packets. This file plays a key role in the entire operation mechanism. When the network is abnormally interrupted and the northbound EMQ service is disconnected, the system generates and stores data packet files named with timestamps according to established rules. After the network returns to normal, the collection file can accurately track and identify all single files to be sent by virtue of the stored file name prefix information, ensuring the integrity and continuity of data transmission and ensuring that the data flow remains stable under network fluctuations.

[0049] S4: When the network is restored, the individual files to be reported are retrieved according to the collection file, and reported to the cloud service in sequence;

[0050] See also Figure 3 After the network connection status is restored to normal from the interruption, this embodiment relies on the pre-maintained collection file for storing the prefixes of all data packet file names, and uses the retrieval algorithm to quickly and accurately screen and determine all the single file names that need to be reported from the numerous stored data files. Then, according to the obtained single file names, under the corresponding storage path and directory, through the precise matching search operation, each corresponding single file that actually needs to be reported is located, so as to ensure that the data temporarily stored during the network failure is reported to the target receiving end in a timely and accurate manner as soon as the network is restored, so as to ensure the timeliness and integrity of data flow and maintain the stable and efficient operation of the system.

[0051] In this embodiment, the determination of whether the network has recovered is as follows: when the northbound EMQ connection is detected to be disconnected, the system immediately triggers the storage action, and stores the time information corresponding to the disconnection moment into the device memory. The disconnection time is used as one of the key benchmark data for subsequent determination of network recovery. When it is subsequently detected that the northbound EMQ connection is successfully established, the system synchronously generates a connection time representing the moment of successful connection. Start the reading program, read the previously stored disconnection time from the device memory, and compare it with the newly generated connection time. Once it is determined that the disconnection time is less than the connection time in the time series, it can be determined that the network has recovered to normal at this time, thereby triggering a series of subsequent preset operation processes related to network recovery, such as carrying out data reporting and other work based on the data storage and retrieval mechanism mentioned above, to ensure that the entire system operates stably and orderly during the dynamic change of network status.

[0052] S5: After a single file is successfully reported, the successfully reported single file is deleted, and the corresponding name prefix is ​​removed from the collection file.

[0053] When a single file successfully completes the reporting operation, the successfully reported single file will be deleted, and the corresponding file name will be accurately located in the collection file, and the file name will also be deleted to ensure that the relevant data will not be reported repeatedly, thereby maintaining the accuracy and efficiency of the data reporting process and avoiding a series of adverse consequences such as waste of system resources, data redundancy, statistical deviations, etc. caused by repeated data reporting.

[0054] When the event of successful upload of a single file is confirmed, the system will immediately start the deletion process for the single file to optimize storage space and ensure the orderliness of the data processing process. Only when the single file deletion operation is successfully completed and the system has accurate feedback, will the deletion step of the corresponding specific file name in the file name set be triggered.

[0055] However, in actual operation, if the system crashes suddenly after a single file deletion operation is completed, if the file search operation is performed again based on the file name in the file collection, the system will generate and feedback the "File has been resumed" prompt information according to the preset logic. This prompt indicates that although the previously uploaded file has experienced twists and turns, the data integrity is guaranteed and it has been identified by the system as a resumed transfer. Once this prompt is received, the system's built-in program will automatically respond and execute the deletion action for the specific file name in the file name collection, thereby ensuring the consistency and accuracy of the entire file upload, deletion and name management process, avoiding data confusion, file redundancy, and invalid storage space occupation caused by system abnormalities, and maintaining efficient and stable system operation.

[0056] In some remote mountain substations, due to the complex geographical environment, unstable network signals and inverter equipment voltage, network disconnections often occur. During a network disconnection lasting 7 days, the embodiment of the present invention continuously collects real-time power, voltage, current and other data of the inverter, and stores these data in the local memory at a frequency of one data packet per 5 minutes, storing a file, and accumulating 2016 data packet files. After the network is restored, the data is fully uploaded to the data center in the storage order, ensuring the continuity and integrity of the power data.

[0057] When the distributed wireless intelligent collection and monitoring equipment detects that the cloud service is disconnected, it stores the disconnection timestamp. After the data processing layer stores the single-packet data file, it stores the file name in the data name collection file. The data transmission frequency during the network disconnection is one packet of data every 5 minutes. When the network is detected to be restored, the network recovery time is recorded. After the network is restored, the real-time data packet is sent first. After the real-time data is sent, the data packet cached during the network disconnection period is sent according to the network disconnection time and the network recovery time. After each packet is sent, the cached single-file data is deleted first, and then the file name in the file name collection is deleted. The real-time data has a higher priority than the network disconnection data. The network disconnection data is only sent within the 5-minute interval of the real-time data. After the data is sent, the network disconnection-related data files are deleted.

[0058] The embodiment of the present invention provides a data collection and reporting system for network disconnection and resumable transmission, including:

[0059] Acquisition module: used to collect data in real time through the remote communication interface;

[0060] Monitoring module: used to monitor the network connection status of data receiving and transfer services. When the network is disconnected, each packet of collected data is stored independently as a single file named with a timestamp;

[0061] Collection module: used to maintain the collection file of single file names, recording the name prefixes of all single files to be reported;

[0062] Retrieval module: used to retrieve the individual files to be reported according to the collection file after the network is restored, and report them to the cloud service in sequence;

[0063] Reporting module: After a single file is successfully reported, it is used to delete the single file that has been successfully reported and remove the corresponding name prefix from the collection file.

[0064] An application embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a method for collecting and reporting data after network disconnection and resuming transmission.

[0065] The application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method for collecting and reporting data after network disconnection and resuming transmission.

[0066] The embodiments of the present invention further provide a computer program product corresponding to the method for collecting and reporting data after network interruption and resuming transmission provided in the above embodiments, and the computer program product includes a computer program, and the computer program is executed by a processor to implement the method provided in the above embodiments.

[0067] The above description and the accompanying drawings fully illustrate the embodiments of the present invention so that those skilled in the art can practice them. Other embodiments may include structural and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The embodiments of the present invention are not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for collecting and reporting data after network interruption and resuming transmission, characterized in that: include: Real-time data collection via remote communication interface; Monitor the network connection status of data receiving and transfer services. When the network is disconnected, store each packet of collected data independently as a single file named with a timestamp; Maintain a collection file of single file names, recording the name prefixes of all single files to be reported; When the network is restored, the individual files to be reported are retrieved according to the collection file, and reported to the cloud service in sequence; After a single file is successfully reported, delete the successfully reported single file and remove the corresponding name prefix from the collection file.

2. The method for collecting and reporting data after network disconnection and resuming transmission according to claim 1 is characterized in that: The remote communication interface adopts RS485 protocol.

3. The method for collecting and reporting data after network disconnection and resuming transmission according to claim 1 is characterized in that: The data receiving and transferring service includes: northbound EMQ service.

4. The method for collecting and reporting data after network disconnection and resuming transmission according to claim 1 is characterized in that: The timestamp is generated by the System.currentTimeMillis method, and the file name format is "xxx.json", where "xxx" is the timestamp of the file and ".json" is the file extension.

5. The method for collecting and reporting data after network disconnection and resuming transmission according to claim 1 is characterized in that: The monitoring data receiving and transferring service network connection status includes: Record the first timestamp when data reception and relay services are disconnected to the device memory; generating a second timestamp when it is detected that the data reception and the transit service connection are reestablished; If the second timestamp is greater than the first timestamp, it is determined that the network is restored.

6. The method for collecting and reporting data after network disconnection and resuming transmission according to claim 1, characterized in that: After the single file is successfully reported, the successfully reported single file is deleted, and the corresponding name prefix is ​​removed from the collection file, including: First delete the reported single file; After a single file is deleted successfully, delete the corresponding name prefix in the collection file; If an exception occurs after a single file is deleted and the name prefix in the collection file is not deleted, a resume prompt is generated based on the collection file, and the name prefix corresponding to the single file in the collection file is automatically deleted.

7. A data collection and reporting system for network disconnection and resuming transmission, characterized in that: include: Acquisition module: used to collect data in real time through the remote communication interface; Monitoring module: used to monitor the network connection status of data receiving and transfer services. When the network is disconnected, each packet of collected data is stored independently as a single file named with a timestamp; Collection module: used to maintain the collection file of single file names, recording the name prefixes of all single files to be reported; Retrieval module: used to retrieve the individual files to be reported according to the collection file after the network is restored, and report them to the cloud service in sequence; Reporting module: After a single file is successfully reported, it is used to delete the single file that has been successfully reported and remove the corresponding name prefix from the collection file.

8. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by a processor, the processor executes the steps of the method for collecting and reporting network-disconnected and resumable data transmission as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor executes the steps of the method for collecting and reporting network-disconnected and resumable data transmission as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the method for collecting and reporting network-disconnected and resumable data according to any one of claims 1 to 6.

Images