Edge cloud dual cache deduplication access system and method for smart fishery water quality monitoring

Abstract

The application discloses an edge cloud double-buffer deduplication access system and method for intelligent fishery water quality monitoring. The sensing and collecting layer is responsible for collecting a series of water quality parameters such as temperature, pH, dissolved oxygen and turbidity. The edge reliable access layer needs to analyze the data, generate a unique message identifier, perform local caching operations, and manage the state, breakpoint transmission and other matters. The cloud caching layer is mainly used to temporarily store on-site data. The business warehousing layer uses device-level pull cursors, overlapping time windows, message unique identifier verification and other methods to achieve the purpose of supplementing, deduplication and warehousing. The application display layer displays monitoring data, transmission status and missing detection results. The application can reduce the occurrence of data loss and repeated warehousing in the case of network fluctuations, backend failures and repeated transmission, thereby improving the continuity, consistency and traceability of intelligent fishery water quality monitoring data.

Description

Technical Field

[0001] This invention belongs to the fields of smart fisheries, IoT water quality monitoring, edge computing and reliable data access technology, specifically relating to an edge cloud dual-caching deduplication access system and method for smart fisheries water quality monitoring. Background Technology

[0002] With the continuous development of smart fisheries and digital agriculture, the demand for continuous monitoring of water quality parameters such as water temperature, pH, dissolved oxygen, and turbidity in aquaculture is increasing. Current fisheries monitoring systems typically collect data from the field using sensors, then upload it to a cloud platform or a self-built backend via an edge gateway, ultimately displaying the data on a web or mobile device. While these systems can replace manual inspections to some extent, they still suffer from reliability issues in long-term operation and under weak network conditions.

[0003] In actual aquaculture environments, fishponds and other similar facilities may be located in remote areas, leading to unstable network signals. Backend services may also become unavailable for short periods due to upgrades, restarts, malfunctions, or interface timeouts. Because edge gateways directly upload data to the business database or self-built backend, any interruption in the transmission link can result in the inability to promptly save existing data, potentially causing sampling time gaps.

[0004] Some fishery systems have set up local caches at the edge; however, when cached data is retransmitted, it is easy for it to overlap with data already received in the cloud, leading to duplicate entries into the business database. Other fishery systems use cloud platforms for device access, but when the backend pulls data, it lacks device-level cursors, overlapping time windows, and data pull verification mechanisms. Even so, data may still be missed due to interface failures or time boundary errors.

[0005] Furthermore, when multiple types of sensor data are stored together in the business database, the complexity of retrieval and maintenance becomes quite high when the system performs historical queries, trend statistics, or missing data detection based on a single indicator. Existing solutions typically focus only on real-time display and threshold alarms, without systematically designing data status synchronization, deduplication verification, and retransmission loops between edge caching, cloud caching, and business database entry.

[0006] It is necessary to propose a reliable data access solution for smart fishery water quality monitoring. By leveraging dual-caching collaboration between the edge and cloud sides, unique message identification, device-level retrieval cursor, overlapping time window retrieval, and deduplication mechanism for data entry, this solution addresses the issues of missing, duplicate, and untraceable data in scenarios involving network fluctuations, backend failures, and repeated data transmission. Summary of the Invention

[0007] The technical problem this invention aims to solve is that existing intelligent fishery water quality monitoring systems are prone to data loss, duplicate data entry, time-outs, and untraceable access logs during network fluctuations, backend service anomalies, interface timeouts, and data retransmission processes.

[0008] To address the aforementioned technical issues, this invention provides an edge-cloud dual-caching deduplication access system and method for intelligent fishery water quality monitoring. This system does not focus its innovation on ordinary multi-sensor monitoring, page display, or general access control, but rather on the reliable access of water quality data, constructing a closed-loop mechanism from on-site data collection, edge caching, cloud temporary storage, backend retrieval, to business data entry.

[0009] The system comprises a sensing and acquisition layer, an edge reliable access layer, a cloud caching layer, a business data entry layer, and an application display layer. The sensing and acquisition layer is used to collect water quality parameters such as temperature, pH, dissolved oxygen, and turbidity, and generate raw data frames containing device number, sensor type, sampling time, and monitoring values.

[0010] The edge reliable access layer receives raw data frames, performs integrity checks and field parsing on them, and generates a unique message identifier based on the device number, sensor type, sampling time, local incrementing sequence number, and data payload digest. The edge reliable access layer first writes the data to a local cache queue and sets an access status for each data item. Access statuses include at least: not uploaded, uploaded to the cloud, awaiting confirmation, in retransmission, and database confirmation.

[0011] The cloud caching layer receives data uploaded from the reliable edge access layer and continues to store on-site data when the business backend is unavailable, undergoes upgrades, or fails to retrieve data. The cloud caching layer can be implemented using an IoT cloud platform. In one implementation, the IoT cloud platform is the OneNet cloud platform; in other implementations, other IoT platforms with device access and data temporary storage capabilities can also be used.

[0012] The business data entry layer is used to pull data from the cloud caching layer and write it to the business database. The business data entry layer maintains an independent pull cursor for each device and sets an overlapped time window for each pull. When the last pull from the backend fails or the interface times out, the system re-pulls data for the corresponding time period based on the cursor and the overlapped time window. Before writing to the database, the business data entry layer performs deduplication verification based on the message's unique identifier. If the identifier already exists, duplicate data is skipped and duplicate events are recorded; if the identifier does not exist, the data is written to the corresponding data table according to the sensor type.

[0013] The business data ingestion layer establishes access logs and a missing data detection mechanism to ensure data continuity. The system performs continuity checks on ingested data based on device number and sampling time. If data is found to be missing for a specific time period, a data retrieval task is automatically generated to retrieve the missing data from the cloud cache layer. If the cloud cache layer still does not have the corresponding data, the system sends a retransmission request to the reliable edge access layer or marks it as missing in the logs.

[0014] The application presentation layer is used to display real-time monitoring data, historical query results, device status, retransmission status, duplicate data records, and missing data detection results. The application presentation layer can include web and mobile clients, but its function is to display the operational results of the access link and is not considered a core limiting feature of this invention. The beneficial effects of this invention are:

[0015] 1. Improve data continuity: With the help of edge local caching and cloud caching, data collected on-site can still be preserved in the event of network instability or backend service unavailability, thereby reducing the risk of sampling time breakpoints.

[0016] 2. There are several ways to reduce duplicate data entries. This can be achieved by using unique message identifiers, unique database indexes, and deduplication checks during re-transmission to prevent the same monitoring data from being written repeatedly during breakpoint re-transmissions or overlapping fetches.

[0017] 3. Improve fault recovery capabilities: By leveraging device-level fetch cursors, overlapping time windows, and missing data detection mechanisms, when the backend experiences interface timeouts, fetch failures, or service restarts, it can automatically fetch data that was not successfully entered into the database.

[0018] 4. Enhanced traceability: By tracking access status, retransmission status, duplicate events, and missing data detection logs, administrators can easily trace the access process of each piece of data from the edge to the business database.

[0019] 5. Facilitates expansion and maintenance: By storing data in separate tables or partitions according to sensor type, it is easy to add new water quality indicators in the future, and improve the maintenance efficiency of historical query and statistical analysis of single indicators. Attached Figure Description

[0020] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the data access and deduplication process of the present invention; Figure 3 A unique identifier generation and verification diagram for the message of this invention; Figure 4 This is a schematic diagram of the database storage structure of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0022] (I) System Overall Architecture As shown in Figure 1, the system of this invention comprises several parts: a sensing and acquisition layer, a reliable edge access layer, a cloud caching layer, a business data entry layer, and an application display layer. The sensing and acquisition layer consists of water temperature sensors, pH sensors, dissolved oxygen sensors, and turbidity sensors, and its function is to collect basic water quality parameters of the aquaculture water. The acquisition nodes can be implemented using Arduino, STM32, or other microcontrollers, while the reliable edge access layer can be achieved using Raspberry Pi, industrial gateways, or other edge devices with local caching and network communication capabilities.

[0023] In this embodiment, the sensing and acquisition layer utilizes four types of sensors—temperature, pH, dissolved oxygen, and turbidity—to collect relevant data. The edge reliable access layer uses a Raspberry Pi gateway as the access device, the cloud caching layer employs the OneNet IoT cloud platform, and the business data entry layer uses a self-built backend service and a relational database. It should be noted that the hardware devices and platforms mentioned above are merely one among many implementation methods. The core protection focus of this invention is on edge-cloud dual caching, deduplication access, and a closed-loop retransmission mechanism, and is not limited to a specific brand or platform.

[0024] (ii) Generation of unique message identifiers from Figure 3 It can be seen that after parsing each piece of monitoring data, the edge reliable access layer generates a unique message identifier. This unique message identifier is jointly generated by the device number, sensor type, sampling time, local incrementing sequence number, and data payload digest. The data payload digest can be calculated based on the monitored value, unit, sampling time, and verification field. By incorporating the local incrementing sequence number and data payload digest into the unique message identifier, even if the same device samples multiple times within a similar timeframe, different sampling records can be distinguished. Even after a network outage and batch retransmission, the backend can still use this identifier to determine whether the data has been stored in the database.

[0025] (III) Edge caching and state management After receiving data, the edge reliable access layer first writes the data to a local cache queue. Each cache record includes the device number, sensor type, monitored value, sampling time, unique message identifier, number of uploads, last upload time, and access status.

[0026] When the network is normal, the edge reliable access layer uploads the data in the cache queue to the cloud cache layer and updates the status from not uploaded to uploaded to the cloud. When the network is abnormal or the upload interface times out, the data is retained in the local cache queue and awaits subsequent re-upload. After the network is restored, the edge reliable access layer performs batch re-upload according to the sampling time and the locally incrementing sequence number to avoid backend statistical errors caused by out-of-order new and old data.

[0027] (iv) Cloud caching and backend fetching The cloud caching layer acts as an intermediary between the data acquisition and business processes, receiving data uploaded from the edge reliable access layer. Even when the business data entry layer is upgraded, restarted, or temporarily unavailable, the edge reliable access layer can still continue uploading data to the cloud caching layer.

[0028] The business ingestion layer maintains an independent fetch cursor for each device, recording the maximum sampling time or maximum message sequence number successfully processed by that device. When the business ingestion layer periodically fetches data from the cloud cache layer, it backs back one overlapping time window from the cursor to cover any omissions caused by previous fetch failures, incomplete API responses, or boundary time errors. Duplicate data generated within the overlapping time window is automatically filtered by a unique message identifier verification mechanism.

[0029] (v) Deduplication and partitioned storage like Figure 4 As shown, the business database can create temperature, pH, dissolved oxygen, and turbidity data tables according to sensor type, or it can be partitioned and stored according to sensor type. Each type of data table includes fields such as device number, unique message identifier, monitored value, unit, sampling time, receiving time, data quality score, and storage status.

[0030] Before writing data to the business data entry layer, the system first queries the unique index based on the message's unique identifier. If the identifier already exists, the system will not write the data again, but will only record the duplicate access event; if the identifier does not exist, the data will be written to the corresponding sensor data table, and a successful data entry log will be recorded. For data with the same message identifier but different data payload digests, the system marks it as conflicting data and puts it into the exception review queue.

[0031] (vi) Missing item detection and fill mechanism The business data entry layer periodically analyzes the sampling interval of each device within a specific time window. If the system detects a missing sampling record within a certain time period, it generates a missing data detection task. For data that can be retrieved from the cloud cache layer, the business data entry layer automatically performs a data retrieval operation. For data not present in the cloud cache layer, the system records the missing result in the access log and prompts a check of the edge cache or on-site collection nodes. This mechanism allows the system to not only detect abnormalities in water quality parameters but also the continuity of the data access process, thus providing a more reliable data foundation for subsequent water quality analysis and alarm judgment.

[0032] (vii) Application demonstration and management The application presentation layer can display real-time monitoring data, historical curves, equipment status, missing detection results, duplicate data records, and retransmission progress through a unified interface. Ordinary aquaculture personnel mainly view water quality data and equipment status, while system maintenance personnel can view access logs, retransmission queues, and conflicting data.

[0033] The specific front-end form of the application presentation layer can be a web management terminal, a WeChat mini-program terminal, or a mobile app. However, these presentation forms are optional implementations of the present invention and do not affect the technical solution of the present invention with reliable access and deduplication and retransmission as its core.

[0034] (viii) Example of access status field In one implementation, the edge cache table and access log table can be configured with the following status fields to record the entire process of data collection and storage, as shown in Table 1 below.

[0035] Table 1 Access Status Not uploaded The data has been written to the edge local cache but has not yet been successfully uploaded to the cloud cache layer. First entry into edge cache after collection Uploaded to the cloud The data has been uploaded to the cloud caching layer, but the business database has not yet confirmed its entry. Edge upload successful In the supplementary transmission Historical data that was not uploaded is being re-uploaded. Network recovery or manual triggering of retransmission Warehouse entry confirmation The business inbound layer has completed deduplication verification and written the data to the business database. Backend database entry successful Conflict pending review Messages with the same unique identifier but different data digests Abnormal duplication or suspected tampering was detected.

Claims

1. An edge-cloud dual-buffered deduplication access system for intelligent fishery water quality monitoring, characterized in that, It includes a perception and acquisition layer, a reliable edge access layer, a cloud caching layer, a business data entry layer, and an application presentation layer; The sensing and acquisition layer is used to collect water quality parameters of the aquaculture water body and form raw data frames; The edge reliable access layer is used to parse the original data frame, generate a unique message identifier, write the monitoring data into the local cache queue, and upload or re-upload it to the cloud cache layer according to the network status. The cloud caching layer is used to receive and temporarily store monitoring data uploaded by the edge reliable access layer; The business ingestion layer is used to retrieve monitoring data from the cloud cache layer according to the device-level retrieval cursor, and write the data into the business database after deduplication verification using the unique message identifier. The application presentation layer is used to display monitoring data, device status, retransmission status, duplicate data records, and missing data detection results.

2. The system according to claim 1, characterized in that, The unique identifier of the message is generated based on the device number, sensor type, sampling time, local incremental sequence number and data payload summary. The data payload summary is calculated based at least on the monitored value, unit and sampling time.

3. The system according to claim 1, characterized in that, The edge reliable access layer sets an access status for each piece of monitoring data. The access status includes at least one of the following: not uploaded, uploaded to the cloud, waiting for confirmation, in the process of re-uploading, and confirmed for entry into the database.

4. The system according to claim 1, characterized in that, The edge reliable access layer retains local cached data when the network is abnormal or upload fails, and after the network is restored, it performs batch re-upload of the unsuccessfully uploaded data according to the sampling time and the local incrementing sequence number.

5. The system according to claim 1, characterized in that, The business data entry layer maintains an independent pull cursor for each device and sets an overlapping time window when pulling data from the cloud cache layer, so that data that may have been missed can be pulled again after the backend service restarts, the interface times out, or the last pull fails.

6. The system according to claim 1, characterized in that, Before writing to the business database, the business entry layer queries the unique index based on the message's unique identifier; if the message's unique identifier already exists, the writing is skipped and a duplicate access event is recorded; if the message's unique identifier does not exist, it is written to the corresponding data table and a successful entry log is recorded.

7. The system according to claim 6, characterized in that, When messages have the same unique identifier but different data payload digests, the business data entry layer marks the data as conflicting data and writes it to the exception review queue.

8. The system according to claim 1, characterized in that, The business database establishes independent data tables or data partitions according to sensor type. Each type of data table or partition includes fields such as device number, unique message identifier, monitoring value, unit, sampling time, reception time, data quality score, and storage status.

9. The system according to claim 1, characterized in that, The business entry layer also includes a missing detection module. The missing detection module determines whether there are missing sampling records within a specified time window based on the device number, sampling time, and preset sampling period, and generates a supplementary task when a missing record is found.

10. A dual-buffered edge-cloud deduplication access method for intelligent fishery water quality monitoring, characterized in that, Includes the following steps: S1. Collect water quality parameters of the aquaculture water body and form raw data frames; S2. The edge reliable access layer parses the original data frame and generates a unique message identifier; S3, the edge reliable access layer writes the monitoring data into the local cache queue and sets the access status; S4. When the network is normal, upload the data in the local cache queue to the cloud cache layer; when the network is abnormal, retain the data that has not been uploaded and wait for the network to be restored before uploading it. S5. The business inbound layer retrieves monitoring data from the cloud cache layer according to the device-level retrieval cursor and overlapping time window. S6. The business inbound layer performs deduplication verification based on the unique identifier of the message; S7. For non-duplicate data, write it to the business database according to the sensor type; for duplicate data, skip writing and record the duplicate access event. S8. The business inbound layer performs missing detection based on the continuity of sampling time and generates a supplementary task for the missing time period.

Images