Iot edge gateway adaptive data caching and synchronization method, system, device and storage medium

Abstract

The application discloses an Internet of Things edge gateway adaptive data caching and synchronization method, system, device and storage medium, in which network quality parameters such as link bandwidth, time delay and packet loss rate between nodes are acquired, instruction conflicts are detected and solved based on a vector clock mechanism, an instruction execution sequence is generated, a local caching and incremental synchronization strategy is adopted for hot data, a machine learning algorithm is combined to dynamically predict a data access mode, and a caching update period is optimized. Future data is predicted according to data characteristics of Internet of Things applications, and the data caching update period of each node is dynamically adjusted, so that each node can maintain a relatively consistent update rhythm when caching hot data copies, which helps to ensure that hot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to methods, systems, devices and storage media for adaptive data caching and synchronization of Internet of Things (IoT) edge gateways. Background Technology

[0002] Intelligent IoT edge gateways play a crucial role in data processing and transmission. To improve system performance and response speed, adaptive data caching and synchronization methods are widely used. Edge gateways can upload or cache data based on the communication status with the IoT platform. When communication is normal, the edge gateway acquires data from the sensing layer and sends it to the IoT platform; when communication fails, the edge gateway caches the sensing layer data according to caching trigger rules to ensure data integrity. Cloud-based management of edge devices enables resource synchronization and monitoring services. Edge devices can synchronize data and collaborate with the cloud, meeting the complex and diverse needs of edge services.

[0003] Currently, IoT edge gateway technology is applied in many scenarios, such as urban transportation systems. IoT edge gateways are distributed across various urban transportation hubs, generating massive amounts of data within the urban transportation system, such as traffic flow, speed, and vehicle type data at intersections, as well as parking space occupancy and passenger flow at bus stops. This data is characterized by low latency and high concurrency. Due to dynamic changes in network topology (such as temporary adjustments to network lines due to road construction), these instructions need to be transmitted and executed promptly and accurately between various gateway nodes. Different nodes need to have a consistent understanding of the same data entities (such as the status of traffic lights and parking space availability). Furthermore, when the traffic management center issues instructions to adjust traffic light durations, all relevant nodes (adjacent intersection traffic lights, parking management systems, etc.) need to have a unified understanding of this change to ensure the coordinated operation of the entire transportation system.

[0004] Edge gateways use a distributed system for data caching and synchronization. Nodes in the distributed system refresh the data in their caches according to different update cycles. Since the cache update cycles of each node may be different, when the data from the data source changes, nodes with shorter update cycles will quickly obtain and cache the new data, while nodes with longer update cycles will retain the old data until their next update arrives. This causes the data cached by different nodes to differ at the same time, which may lead to data inconsistencies during data reading or use, affecting the accuracy of decisions or operations based on this data. Summary of the Invention

[0005] The technical problem this application aims to solve is that edge gateways use a distributed system for data caching and synchronization. The data cached by different nodes in the distributed system may differ at the same time, which may lead to data inconsistency during data reading or use, affecting the accuracy of decisions or operations based on this data.

[0006] To address the aforementioned problems, or at least partially address the aforementioned technical issues, this application provides an adaptive data caching and synchronization method, system, device, and storage medium for IoT edge gateways.

[0007] In a first aspect, the present invention discloses an adaptive data caching and synchronization method for an IoT edge gateway, which includes the following steps: Obtain the final execution sequence of instructions between each edge gateway node. Based on the final execution sequence of instructions, capture data whose local cache access frequency in the edge gateway node exceeds a preset threshold to obtain hot data. Each node caches a copy of the hot data locally. The data characteristics of IoT applications are acquired and input into a preset data access and load trend prediction model for prediction. Predicted data for a future period is obtained, and the data cache update cycle and node maintenance time interval of each data node are dynamically adjusted based on the predicted data. The data characteristics of IoT applications include data generation rate, data scale, data type, and response time threshold for service quality requirements. Acquire real-time monitoring data of network environment and node status, perform key data anomaly detection and processing on real-time monitoring data, and obtain key data anomaly results. The key data includes network link utilization and node heartbeat data. For critical data anomalies, the data transmission path is dynamically adjusted based on multi-path routing and data redundancy backup mechanisms, and instructions and data are synchronously stored on multiple available nodes.

[0008] Preferably, obtaining the final execution sequence of instructions between each edge gateway node specifically includes the following steps: Obtain the connection status parameters and network quality parameters of each edge gateway node, and judge the connectivity and communication quality between nodes by comparing them with preset thresholds; Obtain the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node, and detect instruction conflict information between nodes to obtain execution instruction conflict information. In response to instruction conflict information, based on the instruction priority hierarchy, the highest priority instruction is retained, and other conflicting instructions are discarded to generate the final execution sequence of instructions between nodes.

[0009] Preferably, the step of acquiring data characteristics of IoT applications and inputting them into a preset data access and load trend prediction model for prediction to obtain prediction data for a future period specifically includes the following steps: Acquire the data characteristics of IoT applications, including data generation rate, data scale and data type, and service quality requirements. Based on these characteristics, determine the basic requirements for data processing and the response time threshold for data processing. Based on historical data access patterns and load changes, a data access and load trend prediction model is trained using machine learning algorithms such as support vector machines to obtain prediction data for a future period.

[0010] Preferably, the step of dynamically adjusting the data cache update cycle and node maintenance time interval of each node based on the prediction results specifically includes the following steps: Based on the predicted load change trend, the time interval for data consistency maintenance is dynamically adjusted. If the predicted future load is high, the consistency maintenance interval is extended. If the predicted load is low, shorten the maintenance interval and synchronize the data in a timely manner; Obtain the computing resource utilization and storage resource capacity of each node in the Internet of Things system, and use them as an important basis for task scheduling and resource allocation; Based on node computing utilization and storage capacity, a heuristic optimization algorithm is used to optimize task scheduling strategies and resource allocation schemes, so as to reasonably allocate tasks to nodes with low load and sufficient storage space.

[0011] Preferably, the process of acquiring real-time monitoring data of the network environment and node status, performing key data anomaly detection processing on the real-time monitoring data, and obtaining key data anomaly results, wherein the key data includes network link utilization and node heartbeat data, specifically includes the following steps: Acquire real-time monitoring data of network environment and node status, and sample and quantify network link utilization and node heartbeat data; The collected time-series monitoring data is input into the time-series analysis model. By statistically modeling the historical data, it is determined whether the current monitoring indicators exceed the normal range. If the time series analysis model detects an anomaly, it triggers the anomaly detection process, which uses unsupervised learning algorithms such as isolated forests to locate and diagnose the root cause of the anomaly.

[0012] Preferably, the step of dynamically adjusting the data transmission path based on multi-path routing and data redundancy backup mechanisms for critical data anomalies, with instructions and data synchronously stored on multiple available nodes, specifically includes the following steps: Based on the anomaly detection results, the network topology is dynamically adjusted, and redundant links are planned using a multi-path routing algorithm. Based on multi-path routing, a data redundancy backup mechanism is adopted to synchronously transmit critical instructions and data to multiple available nodes; Continuously track network recovery status, assess node status and link quality, dynamically adjust data storage strategies, and prioritize migrating data to nodes with high stability. It summarizes the results of network anomaly detection and handling, generates event reports and alarm information, and presents them through a visual dashboard.

[0013] Preferably, the method further includes the following steps: The system aggregates the results of network anomaly detection and handling, and adopts a microservice architecture and containerized deployment to divide data collection, processing and synchronization into independent service components. Service-level resource configuration and load balancing are achieved through service orchestration and dynamic scaling. Data interaction between service components is achieved using homomorphic encryption and secure multi-party computation technology, and end-to-end encryption is implemented during data transmission and storage.

[0014] Secondly, the present invention discloses an adaptive data caching and synchronization system for IoT edge gateways, which includes the aforementioned adaptive data caching and synchronization method for IoT edge gateways.

[0015] Thirdly, the present invention discloses an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the IoT edge gateway adaptive data caching and synchronization method.

[0016] Fourthly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of an adaptive data caching and synchronization method for an Internet of Things edge gateway.

[0017] The technical solution provided in this application has the following advantages compared with the prior art: This application provides an adaptive data caching and synchronization method, system, device, and storage medium for IoT edge gateways. The method acquires network quality parameters such as inter-node link bandwidth, latency, and packet loss rate, detects and resolves command conflicts based on a vector clock mechanism, and generates command execution sequences. For hot data, a local caching and incremental synchronization strategy is employed, combined with machine learning algorithms to dynamically predict data access patterns and optimize the cache update cycle. Simultaneously, anomaly detection algorithms identify abnormal network states, multi-path routing and data redundancy backup ensure system availability, and a microservice architecture is used to dynamically schedule service components. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, a relatively consistent update rhythm can be maintained among nodes when caching hot data copies. This helps ensure that hot data copies from different nodes have similar accuracy and timeliness at the same point in time, thereby maintaining data consistency. This effectively improves the data synchronization efficiency between edge gateway nodes, reduces network overhead, ensures data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an adaptive data caching and synchronization method for an IoT edge gateway provided in this application. Figure 1 . Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Firstly, see Figure 1 This invention discloses an adaptive data caching and synchronization method for IoT edge gateways, which includes the following steps: Step S1: Obtain the final execution sequence of instructions between each edge gateway node. Based on the final execution sequence of instructions, capture data whose local cache access frequency exceeds a preset threshold in the edge gateway node to obtain hot data. Each node caches a copy of the hot data locally. Step S2: Obtain the data characteristics of the IoT application, input them into the preset data access and load trend prediction model for prediction, obtain the prediction data for a future period of time, and dynamically adjust the data cache update cycle and node maintenance time interval of each data node according to the prediction data; the data characteristics of the IoT application include data generation rate, data scale, data type, and response time threshold of service quality requirements. Step S3: Obtain real-time monitoring data of network environment and node status, perform key data anomaly detection processing on real-time monitoring data, and obtain key data anomaly results. The key data includes network link utilization and node heartbeat data. Step S4: For critical data anomalies, dynamically adjust the data transmission path based on multi-path routing and data redundancy backup mechanisms, and store instructions and data synchronously on multiple available nodes; Step S5: Summarize the results of network anomaly detection and processing, and use a microservice architecture and containerized deployment to divide data collection, processing and synchronization into independent service components. Service-level resource configuration and load balancing are achieved through service orchestration and dynamic scaling.

[0023] Specifically, in step S1, the final execution sequence of instructions between edge gateway nodes is first obtained. Based on this execution sequence, the local cache of the edge gateway nodes is filtered to identify data whose access frequency exceeds a preset threshold. This data is defined as hot data, and each node then caches copies of this hot data locally. By identifying hot data and caching copies locally, when a node needs this frequently accessed data again, it can directly retrieve it from the local cache, reducing data retrieval time latency and improving data access speed. Furthermore, each node determines hot data based on its own instruction execution sequence, ensuring that the local cache stores only the most frequently accessed data, improving the resource utilization of the local cache and avoiding caching large amounts of infrequently used data.

[0024] Specifically, in step S2, data characteristics of the IoT application are collected, including data generation rate, data scale, data type, and response time thresholds for service quality requirements. These data characteristics are then input into a preset data access and load trend prediction model. This model predicts relevant data for a future period and dynamically adjusts the data cache update cycle and node maintenance interval for each node based on the predicted data. By making predictions based on the data characteristics of the IoT application and dynamically adjusting the cache update cycle and node maintenance interval based on the prediction results, data management can adapt to changes in the IoT application. In addition, considering the response time thresholds for service quality requirements, by reasonably adjusting the cache update cycle and node maintenance intervals, the data processing flow is optimized while meeting response time requirements, thereby improving the overall service quality of the IoT application.

[0025] Specifically, in step S3, real-time monitoring data of the network environment and node status is collected, with a focus on detecting and processing anomalies in key data such as network link utilization and node heartbeat data. Anomalies in these key data are obtained. Through real-time monitoring and anomaly detection of key data, problems such as excessively high network link utilization or interrupted node heartbeats can be quickly identified, providing a basis for subsequent network adjustments and ensuring the normal operation of the network. In addition, timely detection of potential network risks helps to take measures in advance to avoid network failures, thereby improving network stability and ensuring the normal transmission and interaction of data in the IoT system.

[0026] Specifically, in step S4, when critical data anomalies occur, the data transmission path is dynamically adjusted using multi-path routing and data redundancy backup mechanisms. Instructions and data are synchronously stored on multiple available nodes. When network anomalies occur, multi-path routing ensures uninterrupted data transmission, and the data redundancy backup mechanism ensures that data is stored on multiple nodes. Even if some nodes fail, data can still be obtained from other available nodes, improving the reliability of data transmission. In addition, storing instructions and data synchronously on multiple nodes avoids data loss or instruction failure due to partial node failures, ensuring data integrity and helping to maintain the normal operation of the IoT system.

[0027] Specifically, in step S5, the results of network anomaly detection and processing are first summarized. Then, using a microservice architecture and containerized deployment, data collection, processing, and synchronization functions are built as independent service components. Through service orchestration and dynamic scaling mechanisms, service-level resource configuration and load balancing are implemented for each service component. This means that computing, storage, and other resources are flexibly allocated to each service component according to actual business needs and load conditions, ensuring a balanced distribution of load among the components. When data interacts between service components, homomorphic encryption and secure multi-party computation technologies are used to achieve end-to-end encryption throughout the data transmission and storage process, ensuring data security and privacy.

[0028] It is understandable that by acquiring network quality parameters such as inter-node link bandwidth, latency, and packet loss rate, and using a vector clock mechanism to detect and resolve command conflicts, a command execution sequence is generated. For hot data, a local caching and incremental synchronization strategy is employed, combined with machine learning algorithms to dynamically predict data access patterns and optimize the cache update cycle. Simultaneously, anomaly detection algorithms identify network anomalies, multi-path routing and data redundancy backup ensure system availability, and a microservice architecture enables dynamic scheduling of service components. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, a relatively consistent update rhythm can be maintained among nodes when caching hot data copies. This helps ensure that hot data copies from different nodes have similar accuracy and timeliness at the same point in time, thereby maintaining data consistency. This effectively improves the data synchronization efficiency between edge gateway nodes, reduces network overhead, ensures data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. The microservice architecture and containerized deployment method allow each function (data acquisition, processing, synchronization) to become an independent service component, improving system flexibility. Service orchestration and dynamic scaling achieve service-level resource configuration and load balancing. This effectively improves resource utilization efficiency and avoids resource idleness or overuse. During peak business load periods, resources can be dynamically allocated to service components with high loads; during low load periods, resource consumption is reduced, thereby optimizing the overall system performance and reducing costs. Homomorphic encryption, secure multi-party computation, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, protecting the security and privacy of network anomaly detection and processing results.

[0029] Step S1 specifically includes the following steps: Step S11: Obtain the connection status parameters and network quality parameters of each edge gateway node, and judge the connectivity and communication quality between nodes by comparing them with preset thresholds; Step S12: Obtain the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node, and detect instruction conflict information between nodes to obtain execution instruction conflict information. Step S13: For instruction conflict information, based on the instruction priority level, retain the instruction with the highest priority and discard other conflicting instructions to generate the final execution sequence of instructions between nodes; Step S14: According to the final execution sequence of the instructions, capture hot data. Hot data is data whose local cache access frequency in all edge gateway nodes exceeds a preset threshold. The preset threshold is dynamically adjusted according to the overall load and network bandwidth. Step S15: The hot data is compressed and copied to obtain a processed copy of the hot data. All edge gateway nodes store the processed copy of the hot data.

[0030] Specifically, the system acquires the connection status parameters and network quality parameters of each edge gateway node. By comparing these parameters with preset thresholds, it determines the connectivity and communication quality between nodes. Network quality parameters include inter-node link bandwidth, latency, and packet loss rate. By assessing the network status at each time point, connection problems can be identified and addressed promptly. The system also acquires the instruction status information of each edge gateway node, synchronizes this information according to time points, and detects instruction conflicts between nodes to obtain execution instruction conflict information. For execution instruction conflicts, based on instruction priority levels, the highest priority instruction is retained, and other conflicting instructions are discarded, generating the final execution sequence of instructions between nodes. Finally, the system acquires the access logs of the edge gateway nodes, counts the access frequency of each data point, and determines whether it exceeds a preset threshold. If it does, it is marked as hot data. For data marked as hot data, further processing is performed at the edge... A replica is created in the local cache of the gateway node. This replica is compressed and copied during caching to reduce its cache space usage. A data version management mechanism is established to record the version number of each modification. Incremental synchronization technology is used between multiple edge gateway nodes, synchronizing only the differences between the local replica and other nodes to reduce data transmission volume. Distributed locks or token mechanisms control cross-node concurrent access, preventing conflicts caused by multiple nodes modifying the same data simultaneously. During data synchronization, the version number is compared to determine if the local replica is the latest version; if not, incremental synchronization is triggered to fetch the new version data. An eventual consistency model is adopted, allowing inconsistencies between data replicas across nodes for a short period, but eventually reaching a consistent state. For data with low update frequency and low consistency requirements, the synchronization frequency can be reduced to decrease synchronization overhead; for data with high update frequency and high consistency requirements, the synchronization frequency can be increased to reach a consistent state as quickly as possible.

[0031] Understandably, the first step is to determine the network stability of each edge gateway node, providing a prerequisite for network optimization. The command status information of the nodes is synchronized sequentially, and command conflicts are detected. Conflicts are resolved based on command priority to generate the final execution sequence. The complete transmission route of the commands is obtained to ensure that commands are executed in a reasonable order, improving the accuracy of system operation. Hotspot data is identified by counting access frequency based on edge gateway node access logs. A copy is created in the local cache, and a version management mechanism is established to improve the access efficiency of hotspot data. By caching hotspot data in advance, the caching speed for users on the node can be accelerated, reducing the number of data transfers between nodes and improving caching speed. Furthermore, during data synchronization, version numbers are compared to determine whether an update is needed. An eventual consistency model is adopted, and the synchronization frequency is adjusted according to the data update frequency and consistency requirements to further optimize resource utilization and meet consistency needs.

[0032] As one example, in an urban transportation system, edge gateway nodes can be traffic signal control devices, roadside sensors, etc. Various command interactions exist between these edge gateway nodes; for example, traffic signal control devices may exchange commands to coordinate signal timing. Determining the final execution sequence of these commands between edge gateway nodes is similar to determining the order in which traffic signal control devices execute commands such as adjusting traffic light durations and switching traffic light states. Then, based on this final execution sequence, data with access frequencies exceeding a preset threshold in the local cache of the edge gateway nodes is retrieved. For example, roadside sensors may frequently access traffic flow data at specific intersections; when the access frequency of this data exceeds a set threshold, it is identified as hotspot data. Each node (such as traffic signal control devices, nearby traffic information displays, etc.) caches copies of this hotspot data locally. In other words, in urban transportation systems, hot data (such as traffic flow data at key intersections) are cached locally at various relevant nodes. When these data are needed for traffic decisions (such as traffic signal control equipment adjusting signal light durations based on traffic flow data), they can be directly retrieved from the local cache, greatly shortening the data acquisition time. This allows operations such as traffic signal adjustments to respond more quickly to changes in traffic conditions, reduce vehicle waiting time, and improve road traffic efficiency.

[0033] As one implementation, the preset threshold can be dynamically adjusted based on the current system state (such as overall load, network bandwidth, etc.). When the system load is high, the threshold is appropriately increased to prevent too much data from being marked as hot data; when the system load is low, the threshold is decreased to capture hot data more accurately. This ability to adaptively adjust the criteria for judging hot data based on the actual system operation improves the accuracy and flexibility of hot data capture, better adapting to different workloads and network conditions, and avoiding the problem of fixed thresholds being too strict or lenient in certain situations.

[0034] As one implementation, when capturing hot data, each node can randomly sample access data cached locally within a predetermined time interval. A preset access threshold is established, and the frequency of access to each data point within the sampling time is counted. When the frequency exceeds the preset access threshold, the data is identified as hot data. Furthermore, to improve the accuracy of hot data sampling, the sampling frequency and sample size can be adjusted based on the size of the data cache and the data distribution. For example, when the system load is high, the threshold can be appropriately increased to prevent too much data from being marked as hot data; when the system load is low, the threshold can be decreased to more accurately capture hot data. This method can adaptively adjust the criteria for judging hot data based on the actual system operation, improving the accuracy and flexibility of hot data capture.

[0035] As one embodiment, hot data can also be captured based on the number of nodes with cached data within the sampling time. Within a preset sampling time, the number of times a certain data is cached locally on a node is determined. A preset caching threshold is set, and the cached data is compared with the caching threshold. Data exceeding the threshold can be identified as hot data.

[0036] Step S2 specifically includes the following steps: Step S21: Obtain the data characteristics of the IoT application, including data generation rate, data scale and data type, service quality requirements, and determine the basic requirements for data processing and the response time threshold for data processing based on these characteristics; Step S22: Based on historical data access patterns and load changes, use machine learning algorithms such as support vector machines to train a data access and load trend prediction model to obtain prediction data for a future period of time. Step S23: Based on the predicted load change trend, dynamically adjust the data consistency maintenance interval. If the predicted future load is high, extend the consistency maintenance interval. Step S24: If the predicted load is low, shorten the maintenance interval and synchronize the data in a timely manner; Step S25: Obtain the computing resource utilization and storage resource capacity of each node in the IoT system, and use them as an important basis for task scheduling and resource allocation; Step S26: Based on the node utilization rate and storage capacity, use a heuristic optimization algorithm to optimize the task scheduling strategy and resource allocation scheme, and reasonably allocate tasks to nodes with low load and sufficient storage space. Step S27: Use heuristic optimization algorithms, such as genetic algorithms, to optimize task scheduling strategies and resource allocation schemes based on node computing utilization and storage capacity, so as to reasonably allocate tasks to nodes with low load and sufficient storage space, balance system load, and improve overall performance.

[0037] Specifically, the data characteristics of IoT applications are acquired, including generation rate, scale, type, and quality of service requirements. For data characteristic acquisition, sensors installed on IoT devices can be used to process the data, allowing for real-time monitoring of the devices' corresponding data. The collected data is then aggregated at edge gateway nodes for further processing. Basic data processing requirements and response time thresholds are determined, ensuring that data processing conforms to the data's inherent characteristics and quality of service requirements, guaranteeing the effectiveness of IoT application data processing. Based on historical data access patterns and load changes, algorithms such as Support Vector Machines are used to train data access and load trend prediction models, acquiring predictive data to anticipate load change trends and provide a basis for subsequent resource adjustments, improving the system's foresight. Data consistency maintenance intervals are dynamically adjusted based on predicted load trends—extending them when the load is high and shortening them when the load is low. This flexible adjustment reduces the impact of maintenance on the system during high loads and ensures timely data synchronization during low loads, improving system efficiency. The computing and storage resource status of nodes is acquired, and heuristic optimization algorithms are used to optimize task scheduling and resource allocation, assigning tasks to appropriate nodes to balance system load, improve node resource utilization, and ultimately enhance the overall performance of the IoT system.

[0038] As one example, in urban transportation systems, the data characteristics of IoT applications include the traffic flow data generation rate (e.g., the number of vehicles passing through an intersection per hour), data scale (e.g., the amount of monitoring data for the entire urban transportation network), data type (e.g., vehicle location, speed, road conditions, etc.), and service quality requirement response time thresholds (e.g., the time requirement for traffic lights to adjust based on real-time traffic flow). These data characteristics are input into a pre-defined data access and load trend prediction model. For example, it can predict the traffic flow trend of a specific road segment over a future period, or predict the arrival time and duration of peak traffic periods. Based on the predicted data, the data cache update cycle and node maintenance interval of each node in the urban transportation system are dynamically adjusted. For instance, when peak traffic periods are predicted, the data cache update cycle is extended because the system is primarily focused on handling large amounts of traffic data processing and real-time traffic control, reducing system overhead caused by frequent cache updates and node maintenance. During off-peak traffic periods, the data cache update cycle and node maintenance interval are shortened to ensure timely data updates and maintenance, guaranteeing the accuracy of traffic data and providing reliable data for subsequent traffic planning and analysis. It is understandable that by dynamically adjusting the data cache update cycle and node maintenance interval, urban transportation systems can maintain efficient operation during peak hours and reduce traffic information processing delays caused by excessive maintenance. Timely updates and maintenance of data during off-peak hours facilitate long-term planning and analysis of traffic data, providing accurate data for improving urban traffic layout, while also increasing system resource utilization and reducing operating costs.

[0039] As one implementation, when using sensors to collect data on IoT devices, a data filtering mechanism can be set up at the sensor end to filter out obviously abnormal or useless data at the source, reducing unnecessary data transmission and improving the accuracy of data feature acquisition. For example, the sensor device can attach information such as its own generation rate (e.g., the amount of data generated per second), data size (e.g., the number of bytes sent this time), and data type (e.g., temperature data sent by a temperature sensor is numerical data) when sending data. Using sensors to acquire data is the most basic and accurate method because it obtains feature information at the source of data generation, without intermediate conversion or estimation, reducing the possibility of errors. In addition, no additional processing or analysis is required to obtain data features; the device only needs to send its own feature information along with the data in a predetermined format.

[0040] As one implementation, after extracting data features, data mining algorithms can be used at the edge gateway node for classification. This preliminary classification of the acquired data features determines the data type, and analysis of historical data growth trends reveals characteristics such as data generation rate and data scale. For example, by analyzing the number and size of data packets received from a device within a certain time period, the data generation rate and scale of that device can be calculated; the data type can be determined by examining the data structure (e.g., whether it is structured JSON, numeric, or character). This method leverages the computing power of edge nodes to acquire data features, reducing the burden on the device side. This method is very effective for simple devices or devices that cannot directly report data features. Furthermore, comprehensive analysis of data from multiple devices can reveal data correlations between devices and overall data trend characteristics, contributing to a more comprehensive understanding of the data situation in IoT applications.

[0041] Step S3 specifically includes the following steps: Step S31: Obtain real-time monitoring data of network environment and node status, and sample and quantify network link utilization and node heartbeat data; Step S32: Input the collected time-series monitoring data into the time-series analysis model. By statistically modeling the historical data, determine whether the current monitoring indicators exceed the normal range. Step S33: If the time series analysis model detects an anomaly, the anomaly detection process is triggered, and unsupervised learning algorithms such as isolated forest are used to locate and diagnose the root cause of the anomaly.

[0042] Specifically, real-time monitoring data of the network environment and node status is acquired, with a focus on sampling and quantifying network link utilization and node heartbeat data to form time-series monitoring data. Each edge gateway has a self-monitoring module internally configured to detect its own status and the network link utilization between adjacent nodes. Each node's self-monitoring module connects to the monitoring center to transmit signals, and periodically reports the monitored data to the monitoring center. Network link utilization reflects the degree of network resource usage, and node heartbeat data is a crucial basis for determining node survival and healthy operation. The collected time-series monitoring data is input into a time-series analysis model. Through statistical modeling of historical data, the model determines whether monitoring indicators are abnormal. The time-series analysis model is based on historical data for statistical modeling; for example, using LSTM and other time-series analysis models, the model is built on the indicator data of the most recent hour, and the 3-sigma principle is used to determine whether the current score exceeds the normal range. If an anomaly is detected, an anomaly detection process is triggered, using unsupervised learning algorithms such as Isolation Forest to locate and diagnose the root cause of the abnormal indicators. The Isolation Forest algorithm is an isolation-based anomaly detection algorithm that effectively distinguishes between normal and anomalous data in the feature space. By analyzing the distribution of anomalous indicators in the feature space and their relationship with other normal indicators, it identifies the root causes of anomalies, significantly shortening troubleshooting time and reducing the time that networks and nodes are down or inefficient due to anomalies, thereby improving the availability and stability of networks and nodes.

[0043] As one example, in an urban transportation system, the network environment acts as a communication network between the command center and traffic nodes (traffic lights, sensors, etc.), with node status referring to the operational status of traffic lights and sensors. Network link utilization reflects the busyness of data transmission, and node heartbeats are signals indicating normal equipment operation. These data are collected and quantified at regular intervals; for example, link utilization is expressed as a percentage, a normal heartbeat is 1, and an abnormal heartbeat is 0, resulting in time-series monitoring data. This data is then input into a time-series analysis model based on historical data (traffic data transmission, equipment operational data, etc.) to determine whether monitoring indicators, such as traffic light link utilization during morning rush hour, exceed normal ranges. If an anomaly is detected, such as abnormal traffic light heartbeats or excessively high link utilization, a process is triggered to locate the root cause using the isolated forest algorithm. For example, it might be found that a new display screen shares a line with a traffic light and consumes bandwidth, leading to high traffic light link utilization. It is understandable that by monitoring, analyzing, and diagnosing the network environment and node status in an urban transportation system in this way, potential problems with traffic lights, roadside sensors, and other equipment can be detected in a timely manner. For example, if anomalies in traffic signal network links can be detected and resolved in a timely manner, it can ensure that traffic lights work properly, reduce traffic congestion caused by traffic light malfunctions, and improve the reliability and operational efficiency of the entire urban transportation system.

[0044] As one example, distributed agents are deployed at key locations in the network, such as nodes connected to multiple other nodes. These agents are responsible for collecting real-time monitoring data on the network environment and node status. The collected data is then sent to a centralized monitoring center for aggregation and analysis. The collected data is then quantified for subsequent processing. In addition, for agents deployed within nodes, the deployment location and collection frequency are adjusted according to the network topology and node distribution. The deployment location and collection frequency are continuously optimized to ensure that the monitoring data is acquired comprehensively and in a timely manner.

[0045] As one implementation, thresholds within normal ranges are set for key data such as network link utilization and node heartbeat data. When monitored data exceeds these thresholds, it is considered an anomaly. For network link utilization, a preset upper limit threshold is set; if this threshold is exceeded, network congestion is considered possible. The thresholds are periodically evaluated and adjusted, dynamically updating their values ​​based on historical network performance and changes in business needs to improve the accuracy of anomaly detection.

[0046] Step S4 specifically includes the following steps: Step S41: Based on the anomaly detection results, dynamically adjust the network topology and plan redundant links using a multi-path routing algorithm; Step S42: Based on multi-path routing, a data redundancy backup mechanism is adopted to synchronously transmit critical instructions and data to multiple available nodes; Step S43: Continuously track network recovery status, evaluate node status and link quality, dynamically adjust data storage strategy, and prioritize migrating data to nodes with high stability.

[0047] Specifically, when anomaly detection reveals problems in the network (such as link failures or node overload), the network topology is dynamically adjusted. This means changing the connection methods between nodes in the network based on the current actual network operation, and using multi-path routing algorithms to plan redundant links. Multi-path routing algorithms can find multiple available transmission paths between the source and destination nodes. These redundant links can take over data transmission tasks when the main link fails, improving network fault tolerance. Based on multi-path routing, a data redundancy backup mechanism is adopted. For critical instructions and data, they are simultaneously copied and transmitted to multiple available nodes. This is like storing multiple copies of important files in different places; even if one node fails, other nodes still retain complete data, ensuring data security and availability. Critical instructions and data are synchronously transmitted to multiple available nodes through multi-path routing. The network recovery status is continuously tracked, including monitoring whether nodes have resumed normal operation, whether links have restored normal bandwidth and low latency, etc. Node status (such as node load, operating temperature, resource utilization, etc.) and link quality (such as bandwidth, latency, packet loss rate, etc.) are evaluated. Based on the evaluation results, the data storage strategy is dynamically adjusted.

[0048] As one implementation, the quality of each link in the multi-path routing is periodically assessed. Assessment metrics may include link bandwidth, latency, and packet loss rate. Based on the assessment results, the optimal transmission path is selected for commands and data. When critical data anomalies indicate a problem with a particular link, a backup path with better quality is prioritized. A link quality assessment model is established, which continuously updates the link quality assessment results based on historical and real-time monitoring data. Furthermore, the model can adjust the path selection strategy according to different IoT application requirements (such as applications sensitive to latency or with high bandwidth requirements).

[0049] As one example, in multi-path routing, the load of each path (such as the amount of data transmitted, the current bandwidth utilization, etc.) is monitored in real time. When it is found that the load of a certain path is too high, the transmission paths of some instructions and data are adjusted to paths with lower load to achieve load balancing. A load balancing algorithm is established, such as a weighted round-robin algorithm, which assigns different weights to each path according to the load capacity of the path (such as bandwidth), so as to allocate instruction and data transmission tasks more reasonably.

[0050] As one implementation, for the synchronous storage of instructions and data across nodes, a distributed transaction protocol (such as a two-phase commit protocol, 2PC) is employed to ensure the synchronous storage of instructions and data across multiple available nodes. During the storage process, a pre-commit operation is first performed, where each node prepares its storage resources and reports whether it can commit. If all nodes report that they can commit, a formal commit operation is performed to ensure data consistency. Furthermore, to further improve the efficiency of synchronous storage, the distributed transaction protocol needs to be optimized periodically. Based on actual node parameters, such as storage capacity and the amount of data processed, the protocol is adjusted and optimized to improve the efficiency and reliability of synchronous storage and reduce synchronization failures caused by network faults or other reasons.

[0051] Step S5 specifically includes the following steps: Step S51: Obtain the exception handling result to determine the system running status. If the system encounters an exception, trigger the service orchestration mechanism of the microservice architecture to perform service scheduling and resource reallocation. Step S52: Adopt a containerized deployment approach, dividing data collection, processing, and synchronization functions into independent service components. Each component runs in an independent container, and container orchestration tools are used to achieve dynamic scaling and load balancing of services. Step S53: For the data acquisition service, design a dedicated data acquisition agent and deploy it on the node where the data source is located. This agent supports access from multiple data sources and can dynamically load data acquisition plugins through configuration files, thereby acquiring data in real time and transmitting it to the data processing service. Step S54: The data processing service uses distributed computing frameworks such as Spark or Flink to dynamically adjust computing resources based on the amount of data and processing complexity, such as adding computing nodes or adjusting the parallelism, to process and transform the collected data in real time. Step S55: The processed data is written to the target database or message queue by the data synchronization service. This synchronization service adopts a producer-consumer model and uses the message queue to buffer data to achieve peak smoothing and ensure high availability of data writing. Step S56: Design health check and failover mechanisms for each service component, periodically check the availability of service instances, and automatically remove a faulty instance from the service registry and start a new instance to take over the service. Step S57: Use adaptive load balancing algorithms such as minimum number of connections or response time to dynamically adjust the traffic distribution of service instances, and dynamically adjust the number of service instances according to the resource utilization of the service (such as CPU and memory metrics) to achieve elastic scaling of the service.

[0052] In step S51, exception handling is used to determine the system's operating status. When a system exception occurs, the service orchestration mechanism of the microservice architecture is triggered. This means that a mechanism is needed to detect system exceptions, which can be done by monitoring system logs, performance metrics, etc. Once an exception is detected, the service orchestration mechanism will intervene, reschedule services, and allocate resources to maintain the normal operation of the system. In step S52, containerized deployment breaks down data acquisition, processing, and synchronization functions into independent service components, placing each component in an independent container. Containers provide a lightweight virtualization method, allowing each service component to run independently without interference. With the help of container orchestration tools, the number of service instances can be dynamically increased or decreased according to the system load (dynamic scaling), and incoming requests can be reasonably distributed (load balancing) to optimize the utilization of system resources and improve the overall performance of the system. In step S53, the data acquisition service has a specially designed data acquisition agent deployed on the node where the data source is located. This design reduces network overhead for data transmission and improves acquisition efficiency. This agent supports multiple data source accesses, has strong versatility, and can adapt to different types of data sources, such as databases and file systems. The ability to dynamically load data acquisition plugins through configuration files greatly increases the flexibility of data acquisition. New acquisition functions can be added without rewriting code, and data can be acquired and transmitted to the data processing service in real time. In step S54, the data processing service is based on distributed computing frameworks such as Spark or Flink. These frameworks excel at handling large-scale data and can flexibly adjust computing resources according to the size and complexity of the data. When the data volume suddenly increases or the processing logic becomes more complex, computing nodes can be added to share the computing tasks, or the parallelism can be adjusted to improve processing efficiency. This data processing method can process and transform the acquired data in real time, meeting the business requirements with high data timeliness. In step S55, the data synchronization service is responsible for writing the processed data to the target database or message queue. A producer-consumer model is adopted. Producers are responsible for putting data into a message queue, and consumers retrieve data from the message queue and write it to the target database. The message queue acts as a buffer here. When the data writing speed is unstable, such as during data peaks, the message queue can temporarily store data (peak smoothing) and then write the data to the database during data troughs (valley filling), thereby ensuring high availability of data writing and avoiding data loss or system failures due to mismatched write speeds. In step S56, health checks and failover mechanisms are designed for each service component. Availability checks of service instances are performed periodically, possibly by sending heartbeat packets or checking service responses. Once a service instance is found to be faulty, it is removed from the service registry, so other services will no longer send requests to this faulty instance. At the same time, a new service instance is started to take over the work of the faulty instance, ensuring service continuity and reliability.In step S57, an adaptive load balancing algorithm based on metrics such as minimum connection count or response time is employed. This algorithm dynamically adjusts the distribution of incoming requests among various service instances based on the current number of connections or response time of each service instance. Simultaneously, it considers service resource utilization, such as CPU and memory usage, to adjust the number of service instances. If resource utilization is too high, more service instances may be added; if resource utilization is too low, fewer service instances may be added. This elastic scaling mechanism can automatically optimize resource allocation based on the actual system load, improving overall system performance and resource utilization.

[0053] Specifically, the system monitors the real-time operational status and anomaly information of each service through monitoring components. For example, if a service response time exceeds 500 milliseconds or service availability is below 99%, the system triggers the service orchestration mechanism of the microservice architecture. Orchestration tools such as Kubernetes automatically schedule faulty service instances according to preset orchestration rules, removing them from the load balancer and simultaneously launching new service instances from the resource pool and adding them to the load balancer. The data acquisition agent adopts a pluggable design, supporting multiple data source access methods such as JDBC, Kafka, and Flume, and dynamically loading the required acquisition plugins through XML configuration files. The data processing service is based on Spark Streaming, setting the data reception interval to 5 seconds and using DStream for real-time data processing. Parallelism is dynamically adjusted according to data volume and processing complexity, such as adding one processing thread for every 100MB of data, with a maximum of 20 threads. Processed data is written to the target database via Kafka, with the Kafka message queue buffer set to 64MB. The production rate is automatically adjusted according to the consumption rate to ensure that the queue utilization is maintained between 60% and 80%. The health check interval for service components is 30 seconds; three consecutive failed checks indicate a faulty instance. Load balancing employs a weighted least connections algorithm. The weight of a service instance is calculated based on its CPU, memory, and other resource utilization; for example, the weight decreases by 1 for every 10% increase in CPU utilization. The number of service instances automatically scales based on request volume and resource utilization; for example, one new service instance is added for every 1000 QPS increase in request volume, with a maximum of 10 instances. When data interacts between service components, the homomorphic encryption algorithm RSA-2048 is used to encrypt the data with a 2048-bit key, ensuring confidentiality during transmission and storage. The Secure Multi-Party Computation Protocol (SPDZ) enables multiple participants to collaboratively complete computational tasks without disclosing their individual privacy data, achieving millisecond-level computational efficiency. For encrypted data, a Role-Based Access Control List (RBAC) is used to grant appropriate data access permissions to users with different roles, and authentication is performed using a digital certificate X.509, with the authentication process taking less than one second. Based on predefined data sharing rules, the legitimacy of cross-node data sharing requests is verified. The verification process, based on the attribute encryption algorithm CP-ABE, can be completed within 100 milliseconds. For shared sensitive data, data anonymization techniques, such as masking and noise addition, are employed to conceal sensitive information while preserving the basic characteristics of the data. The anonymized data is then protected using the differential privacy algorithm ε, with a privacy budget ε set to 1. This effectively prevents privacy leaks while ensuring data availability.A distributed data storage system is built using blockchain technology. Leveraging the Merkle tree structure and proof-of-work consensus mechanism of blockchain, data integrity and traceability are guaranteed, with each block generation time controlled within 10 minutes. The zero-knowledge proof protocol zk-SNARK is employed to verify the authenticity and compliance of data without disclosing the original data. Proof generation time is less than 1 second, protecting data privacy while enabling trusted data sharing.

[0054] As one example, regarding resource adjustment strategies for data processing services, a more intelligent resource prediction module is added to the data processing service. This module can predict the data volume and processing complexity over a future period based on historical data and real-time data flow trends. It can proactively respond to changes in data volume, avoiding processing delays caused by sudden increases in data volume, and improving the efficiency and stability of data processing. Based on the prediction results, computing resources are adjusted in advance, not just based on the current data volume and processing complexity. If a significant increase in data volume is predicted in the next few hours, more computing resources can be requested in advance. In other words, through a feedback mechanism, the resource adjustment strategy can be continuously optimized to better adapt to the actual operation of the system and improve resource utilization. A feedback mechanism for resource usage is established to monitor the effect of adjusted resource usage in real time. If the adjusted resource allocation is found to be unreasonable, a secondary adjustment is made promptly. Predictive resource adjustments can better coordinate with other services, such as better matching the rhythm of data acquisition and data synchronization services, improving the overall system synergy.

[0055] As one implementation, the health check and failover mechanism incorporates checks on key internal service indicators, such as database connection pool usage and memory leaks. This involves employing multiple health check methods; the combination of these methods provides a more comprehensive assessment of service health, enabling the timely detection of deeply hidden faults and improving system reliability. The health check cycle is shortened from longer intervals to more frequent checks, allowing for faster detection of potential faults. More frequent health checks can identify problems at an earlier stage, reducing the scope and duration of the fault's impact on the system. During failover, not only is a new instance started to take over the service, but some status data from the failed instance is also synchronized to the new instance. This allows the new instance to recover services more quickly, reducing service interruption time and enabling it to be operational more rapidly, further shortening service interruption time and improving user experience.

[0056] Secondly, the present invention discloses an adaptive data caching and synchronization system for IoT edge gateways, which includes the aforementioned adaptive data caching and synchronization method for IoT edge gateways.

[0057] Specifically, the system implements a method that acquires network quality parameters such as inter-node link bandwidth, latency, and packet loss rate. It then uses a vector clock mechanism to detect and resolve command conflicts, generating a command execution sequence. For hot data, a local caching and incremental synchronization strategy is employed, combined with machine learning algorithms to dynamically predict data access patterns and optimize cache update cycles. Simultaneously, anomaly detection algorithms identify network anomalies, multi-path routing and data redundancy backup ensure system availability, and a microservice architecture enables dynamic scheduling of service components. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, a relatively consistent update rhythm can be maintained among nodes when caching hot data copies. This helps ensure that hot data copies from different nodes have similar accuracy and timeliness at the same point in time, thus maintaining data consistency. This effectively improves data synchronization efficiency between edge gateway nodes, reduces network overhead, ensures data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. The microservice architecture and containerized deployment allow each function (data acquisition, processing, synchronization) to become an independent service component, improving system flexibility. Service orchestration and dynamic scaling achieve service-level resource configuration and load balancing. This effectively improves resource utilization efficiency and avoids resource idleness or overuse. During peak business load periods, resources can be dynamically allocated to service components with high loads; during low load periods, resource consumption is reduced, thereby optimizing the overall system performance and reducing costs. Homomorphic encryption, secure multi-party computation, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, protecting the security and privacy of network anomaly detection and processing results.

[0058] Thirdly, the present invention discloses an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the IoT edge gateway adaptive data caching and synchronization method.

[0059] Specifically, the processor of the electronic device implements the program stored in memory. This program implements the steps of an adaptive data caching and synchronization method for IoT edge gateways. This method acquires network quality parameters such as inter-node link bandwidth, latency, and packet loss rate, detects and resolves instruction conflicts based on a vector clock mechanism, and generates an instruction execution sequence. For hot data, a local caching and incremental synchronization strategy is adopted, combined with machine learning algorithms to dynamically predict data access patterns and optimize the cache update cycle. Simultaneously, anomaly detection algorithms identify network anomalies, multi-path routing and data redundancy backup ensure system availability, and a microservice architecture is used to dynamically schedule service components. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, a relatively consistent update rhythm can be maintained among nodes when caching hot data copies. This helps ensure that hot data copies from different nodes have similar accuracy and timeliness at the same point in time, thereby maintaining data consistency. This effectively improves the data synchronization efficiency between edge gateway nodes, reduces network overhead, ensures data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. Microservice architecture and containerized deployment allow each function (data acquisition, processing, synchronization) to become an independent service component, improving system flexibility. Service orchestration and dynamic scaling enable service-level resource configuration and load balancing. This effectively improves resource utilization efficiency and avoids resource idleness or overuse. During peak business loads, resources can be dynamically allocated to heavily loaded service components; during low loads, resource consumption is reduced, thereby optimizing overall system performance and reducing costs. Homomorphic encryption, secure multi-party computation, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, protecting the security and privacy of network anomaly detection and processing results.

[0060] Fourthly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of an adaptive data caching and synchronization method for an Internet of Things edge gateway.

[0061] Specifically, a computer-readable storage medium stores a computer program that implements the steps of an adaptive data caching and synchronization method for IoT edge gateways. This method acquires network quality parameters such as inter-node link bandwidth, latency, and packet loss rate, detects and resolves instruction conflicts based on a vector clock mechanism, and generates an instruction execution sequence. For hot data, a local caching and incremental synchronization strategy is adopted, combined with machine learning algorithms to dynamically predict data access patterns and optimize the cache update cycle. Simultaneously, anomaly detection algorithms identify abnormal network states, multi-path routing and data redundancy backup ensure system availability, and a microservice architecture is used to dynamically schedule service components. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, a relatively consistent update rhythm can be maintained among nodes when caching hot data copies. This helps ensure that hot data copies from different nodes have similar accuracy and timeliness at the same point in time, thereby maintaining data consistency. This effectively improves the data synchronization efficiency between edge gateway nodes, reduces network overhead, ensures data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. Microservice architecture and containerized deployment allow each function (data acquisition, processing, synchronization) to become an independent service component, improving system flexibility. Service orchestration and dynamic scaling enable service-level resource configuration and load balancing. This effectively improves resource utilization efficiency and avoids resource idleness or overuse. During peak business loads, resources can be dynamically allocated to heavily loaded service components; during low loads, resource consumption is reduced, thereby optimizing overall system performance and reducing costs. Homomorphic encryption, secure multi-party computation, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, protecting the security and privacy of network anomaly detection and processing results.

[0062] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0063] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0064] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0065] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0066] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0067] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0068] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0069] The above description describes specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive data caching and synchronization method for an IoT edge gateway, characterized in that, Includes the following steps, Obtain the final execution sequence of instructions between each edge gateway node. Based on the final execution sequence of instructions, capture data whose local cache access frequency exceeds a preset threshold in each edge gateway node to obtain hot data. Each node caches a compressed copy of the hot data locally. The data characteristics of IoT applications are acquired and input into a preset data access and load trend prediction model for prediction. Predicted data for a future period is obtained, and the data cache update cycle and node maintenance time interval of each data node are dynamically adjusted based on the predicted data. The data characteristics of the IoT applications include data generation rate, data scale, data type, and response time threshold for service quality requirements. The process of dynamically adjusting the data cache update cycle and node maintenance time interval of each data node based on the predicted data specifically includes the following steps: Based on the predicted load change trend, the time interval for data consistency maintenance is dynamically adjusted. If the predicted future load is high, the consistency maintenance interval is extended. If the predicted load is low, shorten the maintenance interval and synchronize the data in a timely manner; Obtain the computing resource utilization and storage resource capacity of each node in the Internet of Things system, and use them as an important basis for task scheduling and resource allocation; Based on node computing utilization and storage capacity, a heuristic optimization algorithm is used to optimize task scheduling strategy and resource allocation scheme, so as to reasonably allocate tasks to nodes with low load and sufficient storage space. Acquire real-time monitoring data of network environment and node status, perform key data anomaly detection and processing on real-time monitoring data, and obtain key data anomaly results. The key data includes network link utilization and node heartbeat data. For critical data anomalies, the data transmission path is dynamically adjusted based on multi-path routing and data redundancy backup mechanisms, and instructions and data are synchronously stored on multiple available nodes; and corresponding anomaly handling results are generated. The results of anomaly handling are summarized, including network anomaly detection and handling results. A microservice architecture and containerized deployment are used to divide data collection, processing and synchronization into independent service components. Service-level resource configuration and load balancing are achieved through service orchestration and dynamic scaling. Data interaction between service components is carried out using homomorphic encryption and secure multi-party computation technology. End-to-end encryption is implemented during data transmission and storage.

2. The method according to claim 1, characterized in that, The process of obtaining the final execution sequence of instructions between various edge gateway nodes, and based on the final execution sequence, capturing data whose local cache access frequency exceeds a preset threshold in each edge gateway node to obtain hot data, and having each node cache a copy of the hot data locally, specifically includes the following steps: Obtain the connection status parameters and network quality parameters of each edge gateway node, and judge the connectivity and communication quality between nodes by comparing them with preset thresholds; Obtain the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node, and detect instruction conflict information between nodes to obtain execution instruction conflict information. In response to instruction conflict information, based on the instruction priority level, the highest priority instruction is retained and other conflicting instructions are discarded to generate the final execution sequence of instructions between nodes; According to the final execution sequence of the instructions, hot data is captured. Hot data is data whose local cache access frequency in all edge gateway nodes exceeds a preset threshold. The preset threshold is dynamically adjusted according to the overall load and network bandwidth. Hotspot data is compressed and copied to obtain processed hotspot data copies, which are then stored on all edge gateway nodes.

3. The method according to claim 1, characterized in that, The process of acquiring data characteristics from IoT applications and inputting them into a preset data access and load trend prediction model to obtain predicted data for a future period includes the following steps: Acquire the data characteristics of IoT applications, including data generation rate, data scale and data type, and service quality requirements. Based on these characteristics, determine the basic requirements for data processing and the response time threshold for data processing. Based on historical data access patterns and load changes, a data access and load trend prediction model is trained using machine learning algorithms to obtain prediction data for a future period.

4. The method according to claim 1, characterized in that, The process of acquiring real-time monitoring data of the network environment and node status, performing key data anomaly detection processing on the real-time monitoring data, and obtaining key data anomaly results, including network link utilization and node heartbeat data, specifically includes the following steps: Acquire real-time monitoring data of network environment and node status, and sample and quantify network link utilization and node heartbeat data; The collected time-series monitoring data is input into the time-series analysis model. By statistically modeling the historical data, it is determined whether the current monitoring indicators exceed the normal range. If the time series analysis model detects an anomaly, it triggers the anomaly detection process, which uses an unsupervised learning algorithm to locate and diagnose the root cause of the anomaly.

5. The method according to claim 1, characterized in that, Regarding the handling of critical data anomalies, based on multi-path routing and data redundancy backup mechanisms, the data transmission path is dynamically adjusted, and instructions and data are synchronously stored on multiple available nodes. Specifically, this includes the following steps: Based on the anomaly detection results, the network topology is dynamically adjusted, and redundant links are planned using a multi-path routing algorithm. Based on multi-path routing, a data redundancy backup mechanism is adopted to synchronously transmit critical instructions and data to multiple available nodes; Continuously track network recovery, assess node status and link quality, dynamically adjust data storage strategies, and prioritize migrating data to nodes with high stability.

6. The method according to claim 1, characterized in that, The summary of anomaly handling results, using a microservice architecture and containerized deployment, divides data collection, processing, and synchronization into independent service components. Service-level resource configuration and load balancing are achieved through service orchestration and dynamic scaling. The process also includes the following steps: Obtain the anomaly handling results, which include network anomaly detection and handling results data. Determine the system operating status based on the anomaly handling results data. If an anomaly occurs in the system, trigger the service orchestration mechanism of the microservice architecture to perform service scheduling and resource reallocation. By adopting a containerized deployment approach, the data collection, processing, and synchronization functions are divided into independent service components, each of which runs in an independent container. Container orchestration tools are used to achieve dynamic scaling and load balancing of services. For the data acquisition service, a dedicated data acquisition agent is designed and deployed on the node where the data source is located. This agent supports access from multiple data sources and can dynamically load data acquisition plugins through configuration files, thereby acquiring data in real time and transmitting it to the data processing service. The data processing service utilizes a distributed computing framework to dynamically adjust computing resources based on the amount of data and processing complexity. The processed data is written to the target database or message queue by the data synchronization service. This synchronization service adopts a producer-consumer model and uses the message queue to buffer data in order to achieve peak smoothing and valley filling. Health checks and failover mechanisms are designed for each service component. The availability of service instances is checked regularly. Once a faulty instance is detected, it is automatically removed from the service registry and a new instance is started to take over the service. An adaptive load balancing algorithm is used to dynamically adjust the traffic allocation of service instances, and the number of service instances is dynamically adjusted according to the resource utilization of the service to achieve elastic scaling of the service.

7. The method according to claim 1, characterized in that, The summary of anomaly handling results adopts a microservice architecture and containerized deployment approach, dividing data collection, processing, and synchronization into independent service components. Service-level resource configuration and load balancing are achieved through service orchestration and dynamic scaling. Data interaction between service components employs homomorphic encryption and secure multi-party computation technology, and end-to-end encryption is implemented during data transmission and storage. The process also includes the following steps: By constructing a distributed data storage system using blockchain technology, and utilizing the Merkle tree structure and proof-of-work consensus mechanism of blockchain, the integrity and traceability of data can be guaranteed. A zero-knowledge proof protocol is used to verify the authenticity and compliance of data without disclosing the original data.

8. An adaptive data caching and synchronization system for an IoT edge gateway, characterized in that, The IoT edge gateway adaptive data caching and synchronization method includes any one of claims 1-7 above.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the adaptive data caching and synchronization method for an IoT edge gateway as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the IoT edge gateway adaptive data caching and synchronization method as described in any one of claims 1-7 above.

Images