Internet of Things edge gateway adaptive data caching and synchronizing method, system and device and storage medium

Abstract

The invention discloses a self-adaptive data caching and synchronizing method, system and device for an edge gateway of the Internet of Things and a storage medium, and the method comprises the steps: obtaining network quality parameters, such as link bandwidth, time delay and packet loss rate, between nodes, detecting and solving instruction conflicts based on a vector clock mechanism, generating an instruction execution sequence, and carrying out the self-adaptive data caching and synchronizing of the edge gateway of the Internet of Things; local caching and incremental synchronization strategies are adopted for hotspot data, a data access mode is dynamically predicted in combination with a machine learning algorithm, and a cache updating period is optimized. The future data is predicted according to the data characteristics of the Internet of Things application, and the data cache updating period of each node is dynamically adjusted, so that each node can keep a relatively consistent updating rhythm when caching a hotspot data copy; therefore, the hot spot data copies of different nodes can be ensured to have similar accuracy and timeliness at the same time point, so that the data consistency is maintained.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an adaptive data caching and synchronization method, system, device and storage medium for an edge gateway of the Internet of Things. Background Art

[0002] Smart IoT edge gateways play a vital role in data processing and transmission. In order to improve the performance and response speed of the system, adaptive data caching and synchronization methods are widely used. The edge gateway can upload or cache data according to the communication status with the IoT platform. When the communication is normal, the edge gateway obtains the perception layer data and sends it to the IoT platform; when the communication fails, the edge gateway caches the perception layer data according to the cache trigger rules to ensure the integrity of the data. Through the cloud management of edge devices, resource synchronization and monitoring services are realized. Edge devices can synchronize data and collaborate with the cloud to meet complex and diverse edge business needs.

[0003] At present, IoT edge gateway technology is applied in many scenarios, such as urban transportation systems. IoT edge gateways are distributed in various transportation hubs in the city, and a large amount of data will be generated in the urban transportation system, such as traffic flow, speed, vehicle type, etc. at each intersection, as well as parking space occupancy in parking lots, passenger flow at bus stops, etc. These data have the characteristics of low latency and high concurrency. Due to the dynamic changes in the network topology (such as temporary adjustments to network lines due to construction of certain sections of roads), these instructions need to be transmitted and executed between various gateway nodes in a timely and accurate manner. Different nodes have a consistent understanding of the same data entity (such as the status of traffic lights, parking space information, etc.). In addition, when the traffic management center issues an instruction to adjust the duration of traffic lights, each related node (adjacent intersection lights, parking lot management system, etc.) needs to have a unified understanding of this change to ensure the coordinated operation of the entire transportation system.

[0004] The edge gateway uses a distributed system for data caching and synchronization. The nodes in the distributed system refresh the data in the cache according to different update cycles. Since the cache update cycles of each node may be different, when the data of the data source changes, the nodes with short update cycles will obtain and cache the new data faster, while the nodes with long update cycles will retain the old data until the next update. This makes the data cached by different nodes different at the same time, which may lead to data inconsistency during data reading or use, affecting the accuracy of decisions or operations based on these data. Summary of the invention

[0005] The technical problem to be solved by this application is that the edge gateway adopts a distributed system for data caching and synchronization. The data cached by different nodes in the distributed system may differ at the same time, which may lead to data inconsistency during data reading or use, affecting the accuracy of decisions or operations based on these data.

[0006] In order to solve the above problems, in order to solve the above technical problems or at least partially solve the above technical problems, the present application provides an IoT edge gateway adaptive data caching and synchronization method, system, device and storage medium.

[0007] In a first aspect, the present invention discloses an adaptive data caching and synchronization method for an edge gateway of the Internet of Things, which comprises the following steps:

[0008] Obtain the final execution sequence of instructions between each edge gateway node, and based on the final execution sequence of instructions, capture the data whose access frequency of the local cache in the edge gateway node exceeds the preset threshold, obtain the hotspot data, and each node caches a copy of the hotspot data locally;

[0009] Obtain data characteristics of IoT applications, input them into a preset data access and load trend prediction model for prediction, obtain prediction data for a period of time in the future, and dynamically adjust the data cache update cycle and node maintenance time interval of each data node according to the prediction data; the data characteristics of IoT applications include data generation rate, data scale, data type, and response time threshold required by quality of service;

[0010] Acquire real-time monitoring data of network environment and node status, perform key data anomaly detection and processing on the real-time monitoring data, and obtain key data anomaly results, wherein the key data includes network link utilization and node heartbeat data;

[0011] In response to abnormal results of key data, the data transmission path is dynamically adjusted based on multi-path routing and data redundant backup mechanism, and instructions and data are synchronously stored in multiple available nodes.

[0012] Preferably, the obtaining of the final execution sequence of instructions between the edge gateway nodes specifically includes the following steps:

[0013] Obtain the connection status parameters and network quality parameters of each edge gateway node, and judge the connectivity and communication quality between nodes by comparing with preset thresholds;

[0014] Obtain the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node and detect the instruction conflict information between the nodes to obtain the execution instruction conflict information;

[0015] For the information of conflicting execution instructions, according to the limited level of instructions, the instruction with the highest priority is retained, and other conflicting instructions are discarded to generate the final execution sequence of instructions between nodes.

[0016] Preferably, the data characteristics of the IoT application are obtained and input into a preset data access and load trend prediction model for prediction to obtain prediction data for a period of time in the future, specifically including the following steps:

[0017] Obtain the data characteristics of IoT applications, including data generation rate, data size and data type, and service quality requirements, and determine the basic requirements for data processing and the response time threshold for data processing based on these characteristics;

[0018] According to the historical data access patterns and load changes, machine learning algorithms such as support vector machines are used to train the data access and load trend prediction model to obtain the prediction data for a period of time in the future.

[0019] Preferably, dynamically adjusting the data cache update period and node maintenance time interval of each node according to the prediction result specifically includes the following steps:

[0020] Dynamically adjust the interval for data consistency maintenance based on the predicted load change trend. If the future load is predicted to be high, extend the consistency maintenance interval.

[0021] If the predicted load is low, shorten the maintenance interval and synchronize data in time;

[0022] Obtain the computing resource utilization and storage resource capacity of each node in the IoT system, and use them as an important basis for task scheduling and resource allocation;

[0023] According to the computing utilization and storage capacity of nodes, a heuristic optimization algorithm is used to optimize the task scheduling strategy and resource allocation plan, and tasks are reasonably allocated to nodes with low load and sufficient storage space.

[0024] Preferably, the real-time monitoring data of the network environment and node status is acquired, and key data abnormality detection and processing is performed on the real-time monitoring data to obtain key data abnormality results, wherein the key data includes network link utilization and node heartbeat data, and specifically includes the following steps:

[0025] Obtain real-time monitoring data of network environment and node status, and sample and quantify network link utilization and node heartbeat data;

[0026] Input the collected time series monitoring data into the time series analysis model, and determine whether the current monitoring indicators are beyond the normal range through statistical modeling of historical data;

[0027] If the time series analysis model detects an anomaly, the anomaly detection process is triggered, and unsupervised learning algorithms such as isolation forests are used to locate and diagnose the root cause of the abnormal indicator.

[0028] Preferably, the method for abnormal results of key data dynamically adjusts the data transmission path based on multi-path routing and data redundancy backup mechanism, and synchronously stores instructions and data in multiple available nodes, specifically including the following steps:

[0029] According to the anomaly detection results, the network topology is dynamically adjusted and redundant links are planned through multi-path routing algorithms;

[0030] Based on multi-path routing, a data redundancy backup mechanism is adopted to synchronously transmit key instructions and data to multiple available nodes;

[0031] Continuously track network recovery, evaluate node status and link quality, dynamically adjust data storage strategies, and prioritize data migration to nodes with high stability;

[0032] Summarize the result data of network anomaly detection and processing, generate event reports and alarm information, and present them through a visual dashboard.

[0033] Preferably, the method further comprises the following steps:

[0034] Summarize the result data of network anomaly detection and processing, use microservice architecture and containerized deployment to divide data collection, processing, and synchronization into independent service components, implement service-level resource allocation and load balancing through service orchestration and dynamic scaling, and use homomorphic encryption and secure multi-party computing technology to interact with data between service components, and implement end-to-end encryption during data transmission and storage.

[0035] In a second aspect, the present invention discloses an adaptive data caching and synchronization system for an Internet of Things edge gateway, which includes the above-mentioned adaptive data caching and synchronization method for an Internet of Things edge gateway.

[0036] In a third aspect, the present invention discloses an electronic device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0037] Memory, used to store computer programs;

[0038] The processor is used to implement the steps of the method for adaptive data caching and synchronization of the edge gateway of the Internet of Things when executing the program stored in the memory.

[0039] In a fourth aspect, the present invention discloses a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a method for adaptive data caching and synchronization of an edge gateway of the Internet of Things.

[0040] The above technical solution provided by this application has the following advantages compared with the prior art:

[0041] The present application provides an adaptive data caching and synchronization method, system, device and storage medium for an edge gateway of the Internet of Things. In this method, by obtaining network quality parameters such as link bandwidth, delay, and packet loss rate between nodes, instruction conflicts are detected and resolved based on a vector clock mechanism to generate an instruction execution sequence. For hot data, a local cache and incremental synchronization strategy is adopted, and the data access mode is dynamically predicted in combination with a machine learning algorithm to optimize the cache update cycle. At the same time, the abnormal state of the network is identified through an anomaly detection algorithm, multi-path routing and data redundancy backup are used to ensure system availability, and the dynamic scheduling of service components is realized using a microservice architecture. By predicting future data based on the data characteristics of the Internet of Things application and dynamically adjusting the data cache update cycle of each node, each node can maintain a relatively consistent update rhythm when caching a copy of the hot data, which helps to ensure that the hot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency. It can effectively improve the data synchronization efficiency between edge gateway nodes, reduce network overhead, ensure data consistency and system availability, and is suitable for distributed data management in Internet of Things application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a method for adaptive data caching and synchronization of an IoT edge gateway provided in this application Figure 1 . DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] First, see Figure 1 The present invention discloses an adaptive data caching and synchronization method for an edge gateway of the Internet of Things, which comprises the following steps:

[0047] Step S1: Obtain the final execution sequence of instructions between each edge gateway node, and according to the final execution sequence of instructions, capture the data whose local cache access frequency in the edge gateway node exceeds a preset threshold to obtain hotspot data, and each node caches a copy of the hotspot data locally;

[0048] Step S2: Obtain data characteristics of IoT applications, input them into a preset data access and load trend prediction model for prediction, obtain prediction data for a period of time in the future, and dynamically adjust the data cache update cycle and node maintenance time interval of each data node according to the prediction data; the data characteristics of IoT applications include data generation rate, data scale, data type, and response time threshold required by quality of service;

[0049] Step S3: Acquire real-time monitoring data of the network environment and node status, perform key data abnormality detection and processing on the real-time monitoring data, and obtain key data abnormality results, wherein the key data includes network link utilization and node heartbeat data;

[0050] Step S4: For abnormal results of key data, based on multi-path routing and data redundancy backup mechanism, the data transmission path is dynamically adjusted, and the instructions and data are synchronously stored in multiple available nodes;

[0051] Step S5: Summarize the result data of network anomaly detection and processing, and use microservice architecture and containerized deployment to divide data collection, processing, and synchronization into independent service components, and implement service-level resource allocation and load balancing through service orchestration and dynamic scaling.

[0052] Specifically, in step S1, the final execution sequence of instructions between edge gateway nodes is first obtained, and based on the execution sequence, the local cache of the edge gateway node is screened to find data whose access frequency exceeds the preset threshold, which is defined as hot data, and then each node is allowed to cache a copy of these hot data locally. By identifying hot data and caching a copy locally, when the node needs these frequently accessed data again, it can be directly obtained from the local cache, which reduces the time delay of data acquisition and improves the speed of data access. In addition, each node determines the hot data according to its own instruction execution sequence, so that the local cache stores the most frequently accessed data, which improves the resource utilization of the local cache and avoids caching a large amount of infrequently used data.

[0053] Specifically, in step S2, data characteristics of IoT applications are collected, including data generation rate, data scale, data type, and response time threshold required by service quality, etc., and these data characteristics are input into the preset data access and load trend prediction model. The model predicts relevant data for a period of time in the future, and finally dynamically adjusts the data cache update cycle and node maintenance time interval of each node according to the predicted data. According to the data characteristics of IoT applications, predictions are made, and the cache update cycle and node maintenance time interval are dynamically adjusted based on the prediction results, so that data management can adapt to changes in IoT applications. In addition, considering the response time threshold required by service quality, by reasonably adjusting the cache update cycle and node maintenance time interval, it is ensured that the data processing process is optimized while meeting the response time requirements, thereby improving the overall service quality of IoT applications.

[0054] Specifically, in step S3, real-time monitoring data of the network environment and node status are collected, focusing on abnormal situation detection and processing of key data such as network link utilization and node heartbeat data, and obtaining abnormal results about these key data. Through real-time monitoring and abnormal detection of key data, problems such as excessive network link utilization or node heartbeat interruption can be quickly discovered, providing a basis for subsequent network adjustments and ensuring the normal operation of the network. In addition, timely discovery of potential network risks helps to take measures in advance to avoid network failures, thereby improving network stability and ensuring the normal transmission and interaction of data in the Internet of Things system.

[0055] Specifically, in step S4, when abnormal results of key data occur, multi-path routing and data redundancy backup mechanism are used to dynamically adjust the data transmission path, and the instructions and data are synchronously stored in multiple available nodes. When an abnormality occurs in the network, multi-path routing can ensure that data transmission is not interrupted, and the data redundancy backup mechanism ensures that the data is stored in multiple nodes. Even if some nodes fail, the data can still be obtained from other available nodes, thereby improving the reliability of data transmission. In addition, the instructions and data are synchronously stored in multiple nodes, avoiding data loss or inability to execute instructions due to failure of some nodes, ensuring data integrity, and helping to maintain the normal operation of the Internet of Things system.

[0056] Specifically, in step S5, the result data of network anomaly detection and processing are first summarized. Then, the microservice architecture and containerized deployment method are adopted to build data collection, processing, synchronization and other functions into independent service components, and service-level resource configuration and load balancing are implemented for each service component through service orchestration and dynamic scaling mechanism. This means that according to actual business needs and load conditions, computing, storage and other resources are flexibly allocated to each service component, and the load is balanced among the components. When data is exchanged between service components, homomorphic encryption and secure multi-party computing technology are used to achieve end-to-end encryption throughout the process of data transmission and storage to ensure data security and privacy.

[0057] It can be understood that by obtaining network quality parameters such as link bandwidth, latency, and packet loss rate between nodes, instruction conflicts are detected and resolved based on the vector clock mechanism, and instruction execution sequences are generated. For hot data, local cache and incremental synchronization strategies are adopted, and machine learning algorithms are combined to dynamically predict data access patterns and optimize cache update cycles. At the same time, the abnormal state of the network is identified through anomaly detection algorithms, multi-path routing and data redundancy backup are used to ensure system availability, and the dynamic scheduling of service components is realized by using microservice architecture. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, each node can maintain a relatively consistent update rhythm when caching hot data copies. This helps to ensure that hot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency. It can effectively improve the data synchronization efficiency between edge gateway nodes, reduce network overhead, ensure data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. The microservice architecture and containerized deployment method make each function (data collection, processing, synchronization) independent service components, improve the flexibility of the system, and service orchestration and dynamic scaling realize service-level resource configuration and load balancing. This can effectively improve resource utilization efficiency and avoid idle or overused resources. When the business load is at a peak, resources can be dynamically increased and allocated to service components with high load; when the load is low, resource usage can be reduced, thereby optimizing the performance of the entire system and reducing costs. Homomorphic encryption and secure multi-party computing technology as well as end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, and protects the security and privacy of network anomaly detection and processing result data.

[0058] The step S1 specifically includes the following steps:

[0059] Step S11: obtaining connection status parameters and network quality parameters of each edge gateway node, and judging the connectivity and communication quality between nodes by comparing with preset thresholds;

[0060] Step S12: Acquire the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node and detect the instruction conflict information between the nodes to obtain the execution instruction conflict information;

[0061] Step S13: for the information on conflicting execution instructions, according to the limited level of instructions, retain the instruction with the highest priority, discard other conflicting instructions, and generate the final execution sequence of instructions between nodes;

[0062] Step S14: According to the final execution sequence of the instruction, hotspot data is captured, where the hotspot data is data whose local cache access frequency in all edge gateway nodes exceeds a preset threshold; the preset threshold is dynamically adjusted according to the overall load and network bandwidth;

[0063] Step S15: The hotspot data is compressed and copied to obtain a processed hotspot data copy, and all edge gateway nodes store the processed hotspot data copy.

[0064] Specifically, the connection status parameters and network quality parameters of each edge gateway node are obtained, and the connectivity between nodes and the communication quality between nodes are judged by comparing with preset thresholds, wherein the network quality parameters include the link bandwidth, delay and packet loss rate between nodes. By judging the network status at each time node in time, if there is a connection problem between certain nodes, it can be adjusted in time; the instruction status information of each edge gateway node is obtained, the instruction status information of each node is synchronized according to the time node and the instruction conflict information between nodes is detected to obtain the execution instruction conflict information; for the execution instruction conflict information, according to the limited level of instructions, the highest priority instruction is retained, and other conflicting instructions are discarded to generate the final execution sequence of instructions between nodes; the access log of the edge gateway node is obtained, and the number of accesses of each data is counted according to the access frequency to determine whether it exceeds the preset threshold. If it exceeds, it is marked as hot data; for the data marked as hot data, at the edge A copy is created in the local cache of the gateway node. The copy is compressed and copied when cached to reduce the cache space it occupies, and a data version management mechanism is established to record the version number of each modification. Incremental synchronization technology is used between multiple edge gateway nodes to synchronize only the differentiated parts of the local copy relative to other nodes to reduce the amount of data transmission. Through distributed locks or token mechanisms, cross-node concurrent access is controlled to avoid conflicts caused by multiple nodes modifying the same data at the same time. During the data synchronization process, the local copy is judged to be the latest version by comparing the version number. If not, incremental synchronization is triggered to pull the new version data. The eventual consistency model is adopted to allow inconsistencies in data copies across nodes for a short period of time, but a consistent state will be reached in the end. For data with low update frequency and low consistency requirements, the synchronization frequency can be reduced to reduce synchronization overhead. For data with high update frequency and high consistency requirements, the synchronization frequency can be increased to reach a consistent state as soon as possible.

[0065] It can be understood that the network stability of each edge gateway node is first determined to provide the optimization premise for network optimization, the instruction status information of the node is synchronized in time sequence and instruction conflicts are detected, conflicts are resolved according to instruction priority to generate the final execution sequence, and the complete transmission route of the instruction is obtained to ensure that the instructions are executed in a reasonable order, improve the accuracy of system operation, and determine the hot data based on the number of accesses counted in the edge gateway node access log, create a copy in the local cache and establish a version management mechanism to improve the access efficiency of hot data. By caching the hot data in advance, the caching speed of the hot data for users in the node can be accelerated, the number of data transfers between nodes can be reduced, and the caching speed can be improved. In addition, when synchronizing data, the version number is compared to determine whether it is updated, and the final consistency model is adopted to adjust the synchronization frequency according to the data update frequency and consistency requirements, so as to further optimize resource utilization and meet consistency requirements.

[0066] As an embodiment, in an urban traffic system, the edge gateway node may be a traffic signal control device, a roadside sensor, etc. There are various command interactions between the edge gateway nodes. For example, there may be commands to coordinate signal timing between traffic signal control devices. By determining the final execution sequence of the commands between these edge gateway nodes, it is like determining the order in which the traffic signal control device executes commands such as adjusting the duration of the signal light and switching the status of the signal light. Then, according to the final execution sequence of the command, the data whose access frequency exceeds the preset threshold in the local cache of the edge gateway node is captured. For example, a roadside sensor may frequently access traffic flow data at a specific intersection. When the access frequency of this data exceeds the set threshold, it is determined as hotspot data. Each node (such as a traffic signal control device, a nearby traffic information display board, etc.) caches a copy of these hotspot data locally. That is, in the urban traffic system, after the hot spot data (such as traffic flow data at key intersections) are cached locally at each relevant node, when this data is needed to make traffic decisions (such as traffic signal control equipment adjusting the duration of traffic lights based on flow data), it can be directly obtained from the local cache, greatly shortening the data acquisition time, allowing operations such as traffic signal adjustment to respond more quickly to changes in traffic conditions, reducing vehicle waiting time, and improving road traffic efficiency.

[0067] As an embodiment, the preset threshold value can be adjusted dynamically, and the threshold value of hotspot data can be adjusted dynamically according to the current state of the system (such as overall load, network bandwidth, etc.). When the system load is high, the threshold value is appropriately increased to avoid too much data being marked as hotspot data; when the system load is low, the threshold value is lowered to capture hotspot data more accurately. The judgment criteria of hotspot data can be adaptively adjusted according to the actual operation of the system, which improves the accuracy and flexibility of hotspot data capture, can better adapt to different workloads and network conditions, and avoid the problem of fixed thresholds being too strict or loose in some cases.

[0068] As an embodiment, when capturing hotspot data, each node can randomly sample the access data in the node's local cache within a predetermined time interval, preset an access threshold, and count the frequency of each data being accessed within the sampling time end. When the frequency exceeds the preset access threshold, the hotspot data is judged. In addition, in order to make the sampling of hotspot data more accurate, the sampling frequency and sample quantity can be adjusted according to the size of the data cache and the distribution of the data to improve the accuracy of hotspot data judgment. For example, when the system load is high, the threshold is appropriately increased to avoid too much data being marked as hotspot data; when the system load is low, the threshold is lowered to capture hotspot data more accurately. This method can adaptively adjust the judgment criteria of hotspot data according to the actual operation of the system, improving the accuracy and flexibility of hotspot data capture.

[0069] As an embodiment, hotspot data can also be captured based on the number of nodes that cache data within the sampling time. Within the preset sampling time, the number of times a certain data is locally cached in the node is determined, a cache count threshold is preset, and the cached data is compared with the cache count threshold. Data that exceeds the threshold can be determined as hotspot data.

[0070] The step S2 specifically includes the following steps:

[0071] Step S21: obtaining data characteristics of IoT applications, including data generation rate, data size and data type, and service quality requirements, and determining basic requirements for data processing and a response time threshold for data processing based on these characteristics;

[0072] Step S22: Based on the historical data access pattern and load change, a machine learning algorithm such as a support vector machine is used to train a data access and load trend prediction model to obtain prediction data for a period of time in the future;

[0073] Step S23: dynamically adjust the time interval for data consistency maintenance according to the predicted load change trend. If the future load is predicted to be high, extend the consistency maintenance interval.

[0074] Step S24: if the predicted load is low, shorten the maintenance interval and synchronize data in time;

[0075] Step S25: Obtain the computing resource utilization and storage resource capacity of each node in the IoT system, and use them as an important basis for task scheduling and resource allocation;

[0076] Step S26: Based on the node computing utilization and storage capacity, a heuristic optimization algorithm is used to optimize the task scheduling strategy and resource allocation scheme, and tasks are reasonably allocated to nodes with low load and sufficient storage space;

[0077] Step S27: Use heuristic optimization algorithms, such as genetic algorithms, to optimize task scheduling strategies and resource allocation plans based on node computing utilization and storage capacity, and reasonably allocate tasks to nodes with lower loads and sufficient storage space to balance system loads and improve overall performance.

[0078] Specifically, the data characteristics of IoT applications are obtained, including generation rate, scale, type, and service quality requirements. For the acquisition of data characteristics, sensors can be installed on IoT devices for processing to detect the corresponding data of the devices in real time, and the collected data is aggregated to the edge gateway node for further processing. The basic requirements and response time thresholds for data processing are determined, and the data processing meets the data characteristics and service quality requirements, to ensure the effectiveness of IoT application data processing. Based on the historical data access mode and load changes, support vector machine and other algorithms are used to train data access and load trend prediction models, obtain predicted data, grasp the load change trend in advance, and provide a basis for subsequent resource adjustment and other operations, improve the forward-looking nature of the system, dynamically adjust the data consistency maintenance interval according to the predicted load trend, extend it when the load is high, and shorten it when the load is low, flexibly adjust the consistency maintenance interval, reduce the impact of maintenance on the system when the load is high, and synchronize data in time when the load is low, improve system efficiency, obtain node computing and storage resource conditions, use heuristic optimization algorithms to optimize task scheduling and resource allocation, assign tasks to appropriate nodes, balance system load, improve node resource utilization, and thus improve the overall performance of the IoT system.

[0079] As an embodiment, in an urban traffic system, the data characteristics of the Internet of Things application include the rate of traffic flow data generation (such as the number of vehicles passing through a certain intersection per hour), data scale (such as the amount of monitoring data of the entire urban traffic network), data type (such as vehicle location, speed, road conditions, etc.) and the response time threshold required by the quality of service (such as the time requirement for traffic lights to adjust according to real-time traffic flow). These data characteristics are input into the preset data access and load trend prediction model. For example, the traffic flow change trend of a specific road section in the future period can be predicted, or the arrival time and duration of the traffic peak can be predicted. According to the predicted data, the data cache update cycle and node maintenance time interval of each node in the urban traffic system are dynamically adjusted. For example, when the traffic peak is predicted, the data cache update cycle is extended, because at this time the system is mainly focused on dealing with a large amount of traffic data processing and real-time traffic control, reducing the system overhead caused by frequent updates of cache and maintenance nodes; during the traffic trough period, the data cache update cycle and node maintenance time interval are shortened, and the data is updated and maintained in time to ensure the accuracy of traffic data and provide reliable data for subsequent traffic planning and analysis. It can be understood that by dynamically adjusting the data cache update cycle and node maintenance time interval, the urban traffic system can maintain efficient operation during peak hours and reduce traffic information processing delays caused by excessive maintenance. Timely updating and maintaining data during off-peak hours helps with long-term planning and analysis of traffic data, provides an accurate basis for improving urban traffic layout, and improves the utilization of system resources and reduces operating costs.

[0080] As an embodiment, when using sensors to collect data on IoT devices, a data filtering mechanism can be set at the sensor end to filter obviously abnormal or useless data at the source, reduce unnecessary data transmission, and improve the accuracy of data feature acquisition. For example, the sensor device can attach its own generation rate (such as the amount of data generated per second), data size (such as the number of bytes of data sent this time) and data type (such as the temperature data sent by the temperature sensor is numerical data) and other information while sending data. The method of acquiring by sensor is the most original and accurate, because the feature information is obtained at the source of data generation, without conversion or estimation in the middle link, which reduces the possibility of error. In addition, no additional processing or analysis is required to obtain data features. The device only needs to send its own feature information together with the data in a predetermined format.

[0081] As an embodiment, after extracting the data features, data mining algorithms can be used at the edge gateway node to classify the acquired data features, preliminarily classify the data types, and obtain features such as data generation rate and data scale by analyzing the growth trend of historical data. For example, by analyzing the number and size of data packets received from a device within a certain period of time, the data generation rate and scale of the device can be calculated; the data type can be determined by checking the structure of the data (such as whether it is a structured JSON format, a numeric type, or a character type, etc.). This method can utilize the computing power of the edge node to acquire data features, reducing the burden on the device side. This method is very effective for some simple devices or devices that cannot directly report data features; in addition, it can comprehensively analyze the data of multiple devices, and may find data associations between some devices and overall data feature trends, which helps to more fully understand the data situation of IoT applications.

[0082] The step S3 specifically comprises the following steps:

[0083] Step S31: Acquire real-time monitoring data of the network environment and node status, and sample and quantify network link utilization and node heartbeat data;

[0084] Step S32: input the collected time series monitoring data into the time series analysis model, and determine whether the current monitoring index exceeds the normal range by statistical modeling of historical data;

[0085] Step S33: If the time series analysis model detects an anomaly, the anomaly detection process is triggered, and unsupervised learning algorithms such as isolation forests are used to locate and diagnose the root cause of the abnormal indicator.

[0086] Specifically, real-time monitoring data of the network environment and node status are obtained, and the network link utilization and node heartbeat data are sampled and quantified to form time series monitoring data. Self-monitoring modules are set up inside each edge gateway to detect the node's own status and the network link utilization between adjacent nodes. The self-monitoring modules of each node are connected to the monitoring center to transmit signals. The self-monitoring modules of each node regularly report the monitored data to the monitoring center. Among them, the network link utilization reflects the degree of use of network resources, and the node heartbeat data is an important basis for judging whether the node is alive and running healthily. The collected time series monitoring data is input into the time series analysis model, and the monitoring indicators are judged to be abnormal through statistical modeling of historical data. The time series analysis model is based on statistical modeling of historical data. For example, the indicator data of the last hour is modeled using time series analysis models such as LSTM, and the current score is judged to be beyond the normal range through the 3-sigma principle. If an abnormality is detected, the anomaly detection process is triggered, and unsupervised learning algorithms such as isolation forests are used to locate and diagnose the root cause of the abnormal indicator. The Isolation Forest Algorithm is an isolation-based anomaly detection algorithm that can effectively distinguish normal data from abnormal data in the feature space. By analyzing the distribution of abnormal indicators in the feature space and their relationship with other normal indicators, the root cause of the anomaly can be found, which greatly shortens the troubleshooting time, reduces the time that the network and nodes are shut down or operate inefficiently due to abnormal problems, and improves the availability and stability of the network and nodes.

[0087] As an embodiment, in an urban traffic system, the network environment is like a communication network between a command center and traffic nodes (signal lights, sensors, etc.), and the node status refers to the working status of signal lights and sensors. The network link utilization reflects the busyness of data transmission, and the node heartbeat is a normal working signal of the equipment. These data are collected and quantified at a certain time interval, such as the link utilization expressed as a percentage, the heartbeat is 1 for normal and 0 for abnormality, and the time series monitoring data is obtained. The data is input into a time series analysis model based on statistical modeling of historical data (traffic data transmission, equipment working data, etc.), and it is determined whether monitoring indicators such as the link utilization of the signal light during the morning rush hour exceed the normal range. If an abnormality is detected, such as an abnormal heartbeat of the signal light or an excessively high link utilization, the triggering process uses the isolation forest algorithm to locate the root cause, such as finding that the new display screen shares a line with the signal light and occupies bandwidth, resulting in a high link utilization of the signal light. It can be understood that by monitoring, analyzing and diagnosing the network environment and node status in the urban traffic system in this way, problems that may occur in equipment such as traffic lights and roadside sensors can be discovered in a timely manner. For example, if the anomalies of traffic light network links can be discovered and resolved in a timely manner, the normal operation of the traffic lights can be ensured, traffic congestion caused by traffic light failures can be reduced, and the reliability and operating efficiency of the entire urban transportation system can be improved.

[0088] As an embodiment, distributed agents are deployed at key locations in the network. For example, some nodes that are connected to multiple other nodes can be deployed with agents responsible for collecting real-time monitoring data on the network environment and node status. These agents send the collected data to a centralized monitoring center for aggregation and analysis, and then quantify the collected data for subsequent processing. In addition, for agents deployed in nodes, the deployment location and collection frequency of the agents are adjusted according to the network topology and node distribution, and the deployment location and collection frequency are continuously optimized to ensure that the monitoring data is acquired in a sufficiently comprehensive and timely manner.

[0089] As an embodiment, for key data such as network link utilization and node heartbeat data, thresholds of normal range are set. When the monitoring data exceeds these thresholds, it is determined to be an abnormal situation. For network link utilization, an upper threshold is preset. If the threshold is exceeded, it is considered that the network may be congested. The threshold is evaluated and adjusted regularly, and the size of the threshold is dynamically updated according to the historical operation of the network and changes in business needs to improve the accuracy of anomaly detection.

[0090] The step S4 specifically comprises the following steps:

[0091] Step S41: dynamically adjust the network topology according to the abnormality detection result, and plan redundant links through a multi-path routing algorithm;

[0092] Step S42: Based on multi-path routing, a data redundancy backup mechanism is used to synchronously transmit key instructions and data to multiple available nodes;

[0093] Step S43: Continuously track network recovery, evaluate node status and link quality, dynamically adjust data storage strategies, and prioritize data migration to nodes with high stability.

[0094] Specifically, when anomaly detection finds problems in the network (such as link failure, node overload, etc.), the network topology is adjusted dynamically. This means changing the connection mode between nodes in the network according to the actual operation status of the current network, and using the multi-path routing algorithm to plan redundant links. The multi-path routing algorithm can find multiple available transmission paths between the source node and the destination node. These redundant links can take over the data transmission task when the main link fails, improving the fault tolerance of the network. On the basis of multi-path routing, a data redundancy backup mechanism is adopted. For key instructions and data, they are copied and transmitted to multiple available nodes at the same time, which is like saving multiple copies of important files in different places. Even if a node has a problem, other nodes still save the complete data, ensuring the security and availability of the data. The key instructions and data are synchronously transmitted to multiple available nodes through multi-path routing. The network recovery is continuously tracked, including monitoring whether the node is working normally again, whether the link has restored normal bandwidth and low latency, etc., and evaluating the node status (such as node load, operating temperature, resource utilization, etc.) and link quality (such as bandwidth, latency, packet loss rate, etc.). According to the evaluation results, the data storage strategy is dynamically adjusted.

[0095] As an embodiment, the quality of each link in the multi-path routing is regularly evaluated, and the evaluation indicators may include link bandwidth, delay, packet loss rate, etc. According to the evaluation results, the optimal transmission path is selected for instructions and data. When the abnormal results of key data indicate that a certain link has a problem, a backup path with better quality is selected first, and a link quality evaluation model is established. The model can continuously update the evaluation results of link quality based on historical data and real-time monitoring data, and can adjust the path selection strategy according to different IoT application requirements (such as applications that are sensitive to delay or have high bandwidth requirements).

[0096] As an embodiment, in multi-path routing, the load of each path (such as the amount of data transmitted, the current bandwidth occupancy rate, etc.) is monitored in real time. When it is found that a certain path is overloaded, the transmission path of some instructions and data is adjusted to a path with a lower load to achieve load balancing; a load balancing algorithm is established, such as a weighted polling algorithm, to assign different weights to each path according to the load capacity of the path (such as the bandwidth size), so as to more reasonably distribute the transmission tasks of instructions and data.

[0097] As an embodiment, for synchronous storage of instructions and data in nodes, a distributed transaction protocol (such as a two-phase commit protocol, 2PC) is used to ensure the synchronous storage of instructions and data in multiple available nodes. During the storage process, a pre-commit operation is first performed, and each node prepares storage resources and feedbacks whether it can be submitted. If all nodes feedback that they can be submitted, a formal submission operation is performed to ensure data consistency. In addition, in order to better improve the efficiency of synchronous storage, it is necessary to optimize the distributed transaction protocol regularly, and adjust the optimization protocol based on actual node parameters, such as storage capacity and amount of data processed, to improve the efficiency and reliability of synchronous storage and reduce synchronization failures caused by network failures and other reasons.

[0098] Step S5 specifically includes the following steps:

[0099] Step S51: Obtain the exception handling result to determine the system operation status. If the system is abnormal, the service orchestration mechanism of the microservice architecture is triggered to perform service scheduling and resource reallocation;

[0100] Step S52: Using containerized deployment, data collection, processing, and synchronization functions are divided into independent service components. Each component runs in an independent container, and dynamic scaling and load balancing of services are achieved with the help of container orchestration tools.

[0101] Step S53: Design a special data collection agent for the data collection service and deploy it on the node where the data source is located. The agent supports access to multiple data sources and can dynamically load data collection plug-ins through configuration files, thereby collecting data in real time and transmitting it to the data processing service;

[0102] Step S54: The data processing service uses a distributed computing framework such as Spark or Flink to dynamically adjust computing resources according to the data volume and processing complexity, such as adding computing nodes or adjusting parallelism, to process and convert the collected data in real time;

[0103] Step S55: The processed data is written into the target database or message queue by the data synchronization service. This synchronization service adopts a producer-consumer model and uses the message queue to buffer data to achieve peak load shaving and valley load filling, thereby ensuring high availability of data writing.

[0104] Step S56: Design health check and failover mechanisms for each service component, regularly check the availability of service instances, and automatically remove a faulty instance from the service registry once it is found, and start a new instance to take over the service;

[0105] Step S57: Adopt adaptive load balancing algorithms such as minimum number of connections or response time to dynamically adjust the traffic distribution of service instances, and dynamically adjust the number of service instances according to the resource utilization of the service (such as indicators such as CPU and memory) to achieve elastic scaling of the service.

[0106] In step S51, exception handling is used to determine the operating status of the system. When the system is abnormal, the service orchestration mechanism of the microservice architecture is triggered. This means that a set of mechanisms is needed to obtain the abnormal situation of the system. It can be detected by monitoring system logs, performance indicators, etc. Once an abnormality is detected, the service orchestration mechanism will intervene, reschedule services and allocate resources to maintain the normal operation of the system. In step S52, containerized deployment splits the data collection, processing and synchronization functions into independent service components, and places each component in an independent container. The container provides a lightweight virtualization method so that each service component can run independently without interfering with each other. With the help of container orchestration tools, the number of service instances can be dynamically increased or decreased according to the load of the system (dynamic scaling), and the incoming requests can be reasonably allocated (load balancing) to optimize the utilization of system resources and improve the overall performance of the system. In step S53, the data collection service has a specially designed data collection agent, which is deployed on the node where the data source is located. This design reduces the network overhead of data transmission and improves the collection efficiency; the agent supports access to multiple data sources and has strong versatility. It can adapt to different types of data sources, such as databases, file systems, etc. The function of dynamically loading data collection plug-ins through configuration files greatly increases the flexibility of data collection. New collection functions can be added without rewriting the code, and data can be collected in real time and transmitted to the data processing service. In step S54, the data processing service is based on distributed computing frameworks such as Spark or Flink. These frameworks are good at processing large-scale data and can flexibly adjust computing resources according to the size of the data and the complexity of processing; when the amount of data suddenly increases or the processing logic becomes complicated, computing nodes can be added to share the computing tasks, or the degree of parallelism can be adjusted to improve processing efficiency. This data processing method can process and convert the collected data in real time to meet business needs with high requirements for data timeliness. In step S55, the data synchronization service is responsible for writing the processed data to the target database or message queue. The producer-consumer model is adopted. The producer is responsible for putting data into the message queue, and the consumer takes the data from the message queue and writes it into the target database. The message queue plays a buffering role here. When the speed of writing data is unstable, such as when there is a data peak, the message queue can temporarily store the data (peak shaving), and then write the data to the database (valley filling) when the data is low, thereby ensuring the high availability of data writing and avoiding data loss or system failure caused by mismatched writing speeds. In step S56, a health check and failover mechanism is designed for each service component, and the service instance is regularly checked for availability, which may be done by sending heartbeat packets, checking service responses, etc. Once a service instance is found to be faulty, it will be removed from the service registration center, so that other services will no longer send requests to this faulty instance. At the same time, a new service instance is started to take over the work of the faulty instance to ensure the continuity and reliability of the service.In step S57, an adaptive load balancing algorithm based on indicators such as the minimum number of connections or response time is used to dynamically adjust the distribution of incoming requests between various service instances according to the number of connections or response time of the current service instance. At the same time, the number of service instances is adjusted in combination with the resource utilization of the service, such as the use of CPU and memory. If the resource utilization is too high, the service instance may be increased; if the resource utilization is too low, the service instance may be reduced. This elastic scaling mechanism can automatically optimize resource allocation according to the actual load of the system and improve the overall performance and resource utilization of the system.

[0107] Specifically, the system obtains the running status and abnormal information of each service in real time through the monitoring component, such as the service response time exceeds 500 milliseconds, the service availability is less than 99%, etc., triggering the service orchestration mechanism of the microservice architecture. Orchestration tools such as Kubernetes automatically schedule faulty service instances according to preset orchestration rules, remove them from the load balancing, and start new service instances from the resource pool and add them to the load balancing. The data collection agent adopts a plug-in design, supports multiple data source access methods such as JDBC, Kafka, Flume, and dynamically loads the required collection plug-ins through XML configuration files. The data processing service is based on Spark Streaming, sets the data reception interval to 5 seconds, and uses DStream to process data in real time. According to the data volume and processing complexity, the parallelism is dynamically adjusted, such as adding a processing thread for every 100MB of data, and the maximum number of threads is 20. The processed data is written to the target database through Kafka. The Kafka message queue buffer is set to 64MB. The production rate is automatically adjusted according to the consumption rate to ensure that the queue utilization rate is maintained between 60% and 80%. The health check interval of the service component is 30 seconds. If the check fails for three consecutive times, it is considered a faulty instance. The load balancing uses the weighted minimum connection number algorithm. The weight of the service instance is calculated based on its CPU, memory and other resource utilization. For example, for every 10% increase in CPU utilization, the weight decreases by 1. The number of service instances is automatically scaled according to the request volume and resource utilization. For example, for every 1000QPS increase in the request volume, a new service instance is added, and the maximum number of instances is 10. When data is exchanged between service components, the homomorphic encryption algorithm RSA-2048 is used to encrypt the data. The key length is 2048 bits, which can ensure the confidentiality of the data during transmission and storage. Through the secure multi-party computing protocol SPDZ, multiple participants can jointly complete the computing task without leaking their own privacy data, and the computing efficiency can reach the millisecond level. For the encrypted data, the role-based access control list RBAC is used to grant corresponding data access rights to users of different roles, and the identity authentication is performed through the digital certificate X. 509. The authentication process takes less than 1 second. According to the pre-defined data sharing rules, the legitimacy of cross-node data sharing requests is verified. The verification process is based on the attribute encryption algorithm CP-ABE and can be completed within 100 milliseconds. For shared sensitive data, data desensitization techniques such as masking and noise addition are used to hide sensitive information while retaining the basic characteristics of the data. The desensitized data is protected by the differential privacy algorithm ε-differential privacy. The privacy budget ε is set to 1, which can effectively prevent privacy leakage while ensuring data availability.The distributed data storage system is built through blockchain technology, and the Merkle tree structure and proof-of-work consensus mechanism of the blockchain are used to ensure the integrity and traceability of the data. The generation time of each block is controlled within 10 minutes. The zero-knowledge proof protocol zk-SNARK is used to verify the authenticity and compliance of the data without disclosing the original data. The proof generation time is less than 1 second, which protects data privacy while realizing trusted data sharing.

[0108] As an embodiment, for the resource adjustment strategy of the data processing service, a more intelligent resource prediction module is added to the data processing service. This module can predict the data volume and processing complexity in the future based on historical data and real-time data flow trends, and can more actively respond to changes in data volume, avoid processing delays caused by a sudden increase in data volume, and improve the efficiency and stability of data processing. According to the prediction results, the computing resources are adjusted in advance, not just based on the current data volume and processing complexity. If it is predicted that the data volume will increase significantly in the next few hours, more computing resources can be applied in advance. That is, the resource adjustment strategy can be continuously optimized through the feedback mechanism to make it more adaptable to the actual operation of the system and improve resource utilization. A feedback mechanism for resource use is established to monitor the adjusted resource use effect in real time. If it is found that the adjusted resource allocation is unreasonable, a secondary adjustment is made in time. The predicted resource adjustment can better coordinate with other services, such as matching the rhythm of data collection services and data synchronization services, and improving the synergy of the entire system.

[0109] As an embodiment, with respect to the health check and failover mechanism, additional checks are performed on key indicators within the service, such as the usage of the database connection pool, memory leaks, etc., that is, multiple health check methods are used. The combination of multiple detection methods can more comprehensively detect the health status of the service, timely discover hidden fault hazards, and improve the reliability of the system. The health check cycle is shortened, and the original longer time interval inspection is changed to more frequent inspections to detect potential faults more quickly. More frequent health checks can detect problems in the early stages of a fault and reduce the scope and time of the fault's impact on the system. During a failover, not only is a new instance started to take over the service, but part of the status data of the failed instance can also be synchronized to the new instance, so that the new instance can resume service faster, reduce the time of service interruption, and enable the new instance to be put into work faster, further shortening the time of service interruption and improving user experience.

[0110] In a second aspect, the present invention discloses an adaptive data caching and synchronization system for an Internet of Things edge gateway, which includes the above-mentioned adaptive data caching and synchronization method for an Internet of Things edge gateway.

[0111] Specifically, the system is used to implement a method, in which network quality parameters such as link bandwidth, latency, and packet loss rate between nodes are obtained, instruction conflicts are detected and resolved based on a vector clock mechanism, and instruction execution sequences are generated. Local cache and incremental synchronization strategies are adopted for hotspot data, and data access patterns are dynamically predicted in combination with machine learning algorithms to optimize cache update cycles. At the same time, abnormal network states are identified through anomaly detection algorithms, multi-path routing and data redundancy backup are adopted to ensure system availability, and dynamic scheduling of service components is realized using a microservice architecture. By predicting future data based on the data characteristics of IoT applications and dynamically adjusting the data cache update cycle of each node, each node can maintain a relatively consistent update rhythm when caching hotspot data copies. This helps ensure that hotspot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency. It can effectively improve the data synchronization efficiency between edge gateway nodes, reduce network overhead, ensure data consistency and system availability, and is suitable for distributed data management in IoT application scenarios. The microservice architecture and containerized deployment method make each function (data collection, processing, synchronization) independent service components, improve the flexibility of the system, and service orchestration and dynamic scaling realize service-level resource configuration and load balancing. This can effectively improve resource utilization efficiency and avoid idle or overused resources. When the business load is at a peak, resources can be dynamically increased and allocated to service components with high load; when the load is low, resource usage can be reduced, thereby optimizing the performance of the entire system and reducing costs. Homomorphic encryption and secure multi-party computing technology as well as end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, and protects the security and privacy of network anomaly detection and processing result data.

[0112] In a third aspect, the present invention discloses an electronic device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0113] Memory, used to store computer programs;

[0114] The processor is used to implement the steps of the method for adaptive data caching and synchronization of the edge gateway of the Internet of Things when executing the program stored in the memory.

[0115] Specifically, the processor of the electronic device implements the program stored in the memory, and the program in the memory is used to implement the steps of the method for adaptive data caching and synchronization of the edge gateway of the Internet of Things. In this method, by obtaining network quality parameters such as link bandwidth, delay, and packet loss rate between nodes, the instruction conflict is detected and resolved based on the vector clock mechanism, and the instruction execution sequence is generated. For hot data, a local cache and incremental synchronization strategy is adopted, and the data access mode is dynamically predicted in combination with a machine learning algorithm to optimize the cache update cycle. At the same time, the abnormal state of the network is identified through an anomaly detection algorithm, multi-path routing and data redundancy backup are adopted to ensure system availability, and the dynamic scheduling of service components is realized by using a microservice architecture. By predicting future data according to the data characteristics of the Internet of Things application and dynamically adjusting the data cache update cycle of each node, each node can maintain a relatively consistent update rhythm when caching hot data copies. This helps to ensure that the hot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency. It can effectively improve the data synchronization efficiency between edge gateway nodes, reduce network overhead, ensure data consistency and system availability, and is suitable for distributed data management in Internet of Things application scenarios. Microservice architecture and containerized deployment methods make each function (data collection, processing, synchronization) independent service components, improving the flexibility of the system. Service orchestration and dynamic scaling realize service-level resource allocation and load balancing. This can effectively improve resource utilization efficiency and avoid idle or overused resources. When the business load is peak, resources can be dynamically increased and allocated to service components with high load; when the load is low, resource occupation can be reduced, thereby optimizing the performance of the entire system and reducing costs. Homomorphic encryption, secure multi-party computing technology, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, and protects the security and privacy of network anomaly detection and processing result data.

[0116] In a fourth aspect, the present invention discloses a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a method for adaptive data caching and synchronization of an edge gateway of the Internet of Things.

[0117] Specifically, a computer program is stored in a storage medium, and the program is used to implement the steps of an adaptive data caching and synchronization method for an edge gateway of the Internet of Things. In this method, by obtaining network quality parameters such as link bandwidth, delay, and packet loss rate between nodes, instruction conflicts are detected and resolved based on a vector clock mechanism, and an instruction execution sequence is generated. For hot data, a local cache and incremental synchronization strategy is adopted, and a machine learning algorithm is combined to dynamically predict data access patterns and optimize the cache update cycle. At the same time, an abnormal network state is identified through an anomaly detection algorithm, multi-path routing and data redundancy backup are used to ensure system availability, and a microservice architecture is used to implement dynamic scheduling of service components. By predicting future data based on the data characteristics of the Internet of Things application and dynamically adjusting the data cache update cycle of each node, each node can maintain a relatively consistent update rhythm when caching hot data copies. This helps to ensure that the hot data copies of different nodes have similar accuracy and timeliness at the same time point, thereby maintaining data consistency. It can effectively improve the data synchronization efficiency between edge gateway nodes, reduce network overhead, ensure data consistency and system availability, and is suitable for distributed data management in Internet of Things application scenarios. Microservice architecture and containerized deployment methods make each function (data collection, processing, synchronization) independent service components, improving the flexibility of the system. Service orchestration and dynamic scaling realize service-level resource allocation and load balancing. This can effectively improve resource utilization efficiency and avoid idle or overused resources. When the business load is peak, resources can be dynamically increased and allocated to service components with high load; when the load is low, resource occupation can be reduced, thereby optimizing the performance of the entire system and reducing costs. Homomorphic encryption, secure multi-party computing technology, and end-to-end encryption ensure the security of data interaction and storage. End-to-end encryption prevents data leakage and tampering during transmission and storage, and protects the security and privacy of network anomaly detection and processing result data.

[0118] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0121] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be connected, detachably connected, or integrated; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0122] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0125] The above is a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An adaptive data caching and synchronization method for an Internet of Things edge gateway, characterized in that: The following steps are included: Obtain the final execution sequence of instructions between each edge gateway node. According to the final execution sequence of instructions, capture the data whose access frequency of local cache in each edge gateway node exceeds the preset threshold, obtain the hotspot data, and each node caches a compressed copy of the hotspot data locally. Obtain data characteristics of IoT applications, input them into a preset data access and load trend prediction model for prediction, obtain prediction data for a period of time in the future, and dynamically adjust the data cache update cycle and node maintenance time interval of each data node according to the prediction data; the data characteristics of IoT applications include data generation rate, data scale, data type, and response time threshold required by quality of service; Acquire real-time monitoring data of network environment and node status, perform key data anomaly detection and processing on the real-time monitoring data, and obtain key data anomaly results, wherein the key data includes network link utilization and node heartbeat data; In response to abnormal results of key data, the data transmission path is dynamically adjusted based on multi-path routing and data redundancy backup mechanism, and instructions and data are synchronously stored in multiple available nodes; And generate corresponding exception handling results; Summarize the exception handling results, which include the result data of network anomaly detection and processing. Use microservice architecture and containerized deployment to divide data collection, processing, and synchronization into independent service components. Implement service-level resource allocation and load balancing through service orchestration and dynamic scaling.

2. The method according to claim 1, characterized in that: The method of obtaining the final execution sequence of instructions between each edge gateway node, and capturing data whose access frequency of local cache in each edge gateway node exceeds a preset threshold according to the final execution sequence of instructions, obtaining hotspot data, and each node locally caching a copy of the hotspot data, specifically includes the following steps: Obtain the connection status parameters and network quality parameters of each edge gateway node, and judge the connectivity and communication quality between nodes by comparing with preset thresholds; Obtain the instruction status information of each edge gateway node, synchronize the instruction status information of each node according to the time node and detect the instruction conflict information between the nodes to obtain the execution instruction conflict information; For the information of conflicting execution instructions, according to the limited level of instructions, the instruction with the highest priority is retained, and other conflicting instructions are discarded to generate the final execution sequence of instructions between nodes; According to the final execution sequence of the instructions, hotspot data is captured, where the local cache access frequency of all edge gateway nodes exceeds a preset threshold; the preset threshold is dynamically adjusted according to the overall load and network bandwidth; The hotspot data is compressed and copied to obtain a processed hotspot data copy, and all edge gateway nodes store the processed hotspot data copy.

3. The method according to claim 1, characterized in that: The data characteristics of the IoT application are obtained and input into a preset data access and load trend prediction model for prediction, so as to obtain prediction data for a period of time in the future, specifically including the following steps: Obtain the data characteristics of IoT applications, including data generation rate, data size and data type, and service quality requirements, and determine the basic requirements for data processing and the response time threshold for data processing based on these characteristics; According to the historical data access patterns and load changes, machine learning algorithms such as support vector machines are used to train the data access and load trend prediction model to obtain the prediction data for a period of time in the future.

4. The method according to claim 1, characterized in that: The method of dynamically adjusting the data cache update cycle and the node maintenance time interval of each node according to the prediction result specifically includes the following steps: Dynamically adjust the interval for data consistency maintenance based on the predicted load change trend. If the future load is predicted to be high, extend the consistency maintenance interval. If the predicted load is low, shorten the maintenance interval and synchronize data in time; Obtain the computing resource utilization and storage resource capacity of each node in the IoT system, and use them as an important basis for task scheduling and resource allocation; According to the computing utilization and storage capacity of nodes, a heuristic optimization algorithm is used to optimize the task scheduling strategy and resource allocation plan, and tasks are reasonably allocated to nodes with low load and sufficient storage space.

5. The method according to claim 1, characterized in that: The method of acquiring real-time monitoring data of the network environment and node status, performing key data abnormality detection and processing on the real-time monitoring data, and obtaining key data abnormality results, wherein the key data includes network link utilization and node heartbeat data, specifically includes the following steps: Obtain real-time monitoring data of network environment and node status, and sample and quantify network link utilization and node heartbeat data; Input the collected time series monitoring data into the time series analysis model, and determine whether the current monitoring indicators are beyond the normal range through statistical modeling of historical data; If the time series analysis model detects an anomaly, the anomaly detection process is triggered, and unsupervised learning algorithms such as isolation forests are used to locate and diagnose the root cause of the abnormal indicator.

6. The method according to claim 1, characterized in that: The method aims at abnormal results of key data, dynamically adjusts the data transmission path based on multi-path routing and data redundancy backup mechanism, and synchronously stores instructions and data in multiple available nodes, specifically including the following steps: According to the anomaly detection results, the network topology is dynamically adjusted and redundant links are planned through multi-path routing algorithms; Based on multi-path routing, a data redundancy backup mechanism is adopted to synchronously transmit key instructions and data to multiple available nodes; Continuously track network recovery, evaluate node status and link quality, dynamically adjust data storage strategies, and prioritize data migration to nodes with high stability.

7. The method according to claim 1, characterized in that: The summary of exception handling results, using microservice architecture and containerized deployment to divide data collection, processing, and synchronization into independent service components, and realizing service-level resource configuration and load balancing through service orchestration and dynamic scaling, also includes the following steps: Obtain exception handling results, which include the result data of network anomaly detection and processing. The system operation status is determined based on the exception handling result data. If an exception occurs in the system, the service orchestration mechanism of the microservice architecture is triggered to perform service scheduling and resource reallocation. Adopt containerized deployment mode, divide data collection, processing and synchronization functions into independent service components, each component runs in an independent container, and realizes dynamic scaling and load balancing of services with the help of container orchestration tools; For data collection services, a special data collection agent is designed and deployed on the node where the data source is located. The agent supports access to multiple data sources and can dynamically load data collection plug-ins through configuration files, thereby collecting data in real time and transmitting it to the data processing service; Data processing services use a distributed computing framework to dynamically adjust computing resources based on data volume and processing complexity; The processed data is written to the target database or message queue by the data synchronization service. This synchronization service adopts the producer-consumer model and uses the message queue to buffer data to achieve peak load shaving and valley load filling. Design health check and failover mechanisms for each service component to regularly check the availability of service instances. Once a faulty instance is found, it will be automatically removed from the service registry and a new instance will be started to take over the service. An adaptive load balancing algorithm is used to dynamically adjust the traffic distribution of service instances, and the number of service instances is dynamically adjusted according to the resource utilization of the service to achieve elastic scaling of the service.

8. An IoT edge gateway adaptive data caching and synchronization system, characterized in that: The method comprises the adaptive data caching and synchronization method of the edge gateway of the Internet of Things as described in any one of claims 1 to 7 above.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the method for adaptive data caching and synchronization of an edge gateway of the Internet of Things as described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for adaptive data caching and synchronization of an Internet of Things edge gateway as described in any one of claims 1 to 7 are implemented.

Images