A data acquisition method, device and medium for an Internet of Things (IoT) edge gateway

Abstract

This specification discloses a data acquisition method, device, and medium for an IoT edge gateway, relating to the field of data acquisition technology. The method includes: determining at least one monitoring process and multiple corresponding acquisition processes pre-deployed in each server of the edge gateway cluster; allocating acquisition tasks among the multiple acquisition processes according to a preset dynamic load balancing algorithm through the monitoring process to determine multiple online acquisition processes corresponding to the acquisition tasks; executing acquisition tasks through the multiple online acquisition processes and detecting the health status between the monitoring process and the multiple online acquisition processes to determine real-time health status data; when the real-time health status data indicates that a specified monitoring process and / or a specified online acquisition process is abnormal, determining a takeover target in the edge gateway cluster to process the acquisition tasks of the specified monitoring process and / or the specified online acquisition process through the takeover target, thereby realizing data acquisition of IoT devices.

Description

Technical Field

[0001] This specification relates to the field of data acquisition technology, and in particular to a data acquisition method, device and medium for an Internet of Things (IoT) edge gateway. Background Technology

[0002] With the rapid development of IoT technology, an increasing number of smart devices are being connected to the network, forming a vast interconnected network. The amount of data generated by these devices is increasing dramatically, placing higher demands on data collection, processing, and analysis. Traditional centralized data processing models, due to their inherent limitations such as bandwidth constraints, latency issues, and the risk of single points of failure at central nodes, are no longer sufficient to meet the needs of modern IoT applications. Especially in the field of edge computing, edge gateways, as key nodes connecting devices and cloud platforms, undertake the initial processing and forwarding of data.

[0003] However, when edge gateways face the challenge of collecting large-scale device data, the processing capacity of a single node is often insufficient. On the one hand, the limited processing power of a single node makes it difficult to handle massive concurrent data, resulting in low data collection efficiency. On the other hand, if a single node fails, data collection from surrounding devices will be interrupted, severely impacting system stability and reliability. Traditional data collection methods employ a distributed architecture, but process monitoring typically relies on a single heartbeat detection mechanism, leading to false positives or false negatives. Furthermore, in the event of anomalies, task migration requires manual intervention, with takeover delays reaching minutes, failing to meet the demands of scenarios requiring high real-time performance. Therefore, in edge gateway scenarios involving the collection of large-scale device data, traditional data collection methods suffer from low efficiency, and task migration relies on manual intervention with long takeover delays, failing to meet the requirements of scenarios demanding high real-time performance. Summary of the Invention

[0004] This specification provides one or more embodiments of a data acquisition method, device, and medium for an Internet of Things (IoT) edge gateway, which addresses the following technical problem: In scenarios where edge gateways face the collection of large-scale device data, traditional data acquisition methods are inefficient, and task migration relies on manual intervention, resulting in long takeover delays and failing to meet the requirements of scenarios with high real-time requirements.

[0005] One or more embodiments of this specification employ the following technical solutions:

[0006] This specification provides one or more embodiments of a data acquisition method for an Internet of Things (IoT) edge gateway. The method includes: determining at least one monitoring process and a corresponding plurality of acquisition processes pre-deployed in each server of an edge gateway cluster; obtaining pre-configured acquisition tasks; allocating the acquisition tasks among the plurality of acquisition processes through the monitoring process according to a preset dynamic load balancing algorithm to determine a plurality of online acquisition processes corresponding to the acquisition tasks; executing the acquisition tasks through the plurality of online acquisition processes and detecting the health status between the monitoring process and the plurality of online acquisition processes to determine real-time health status data; when the real-time health status data indicates that a specified monitoring process and / or a specified online acquisition process is abnormal, determining a takeover target in the edge gateway cluster to process the acquisition tasks of the specified monitoring process and / or the specified online acquisition process through the takeover target, thereby realizing data acquisition from IoT devices.

[0007] This specification provides one or more embodiments of a data acquisition device for an Internet of Things (IoT) edge gateway, comprising:

[0008] At least one processor; and,

[0009] A memory communicatively connected to the at least one processor; wherein,

[0010] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0011] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0012] The above-mentioned at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: Through the technical solutions of the embodiments of this specification, the traditional centralized data processing mode is limited by bandwidth and latency, and is inadequate for large-scale data collection. However, the embodiments of this specification, through a preset dynamic load balancing algorithm, can reasonably distribute the collection tasks to multiple online collection processes according to the real-time load status of the collection process, such as CPU utilization, memory utilization, and network bandwidth utilization. This effectively solves the problem of insufficient processing capacity of a single node, avoids low collection efficiency due to excessive data volume, greatly improves the speed of data collection, and can quickly process massive concurrent data, meeting the requirements of modern Internet of Things for high-efficiency data collection. In the traditional mode, a single point of failure of the central node will cause data collection to be interrupted, and the traditional distributed architecture relies on a single heartbeat detection, which is prone to misjudgment and missed judgment. The embodiments of this specification accurately identify abnormal processes through comprehensive health status detection of monitoring processes and online collection processes. Once a specified monitoring process or a specified online collection process is found to be abnormal, the takeover target can be quickly determined in the edge gateway cluster, and the task takeover can be automatically completed without manual intervention, greatly reducing the risk of data collection interruption and ensuring the stability and reliability of the system. Traditional methods rely on manual intervention for task migration in abnormal situations, resulting in takeover delays of up to minutes, which cannot meet high real-time requirements. The embodiments described in this specification can monitor process status in real time. When real-time health status data shows anomalies, it can quickly identify the takeover target and complete task migration, achieving seamless task transition with extremely short delays. This ensures timely data collection in scenarios with extremely high real-time requirements, such as industrial automation and intelligent transportation. The scale of IoT devices and data traffic are constantly changing dynamically, making traditional methods difficult to adapt. The embodiments described in this specification are based on an edge gateway cluster architecture, which can flexibly handle situations such as increased data volume due to new device access, device offline, and process anomalies. The dynamic load balancing algorithm can adjust task allocation at any time, automatically detect and handle abnormal processes, and achieve system self-optimization and adjustment. It can operate stably in complex and ever-changing IoT environments, possessing strong adaptability and scalability. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0014] Figure 1 A flowchart illustrating a data acquisition method for an IoT edge gateway provided in an embodiment of this specification;

[0015] Figure 2 This specification provides a schematic diagram of a system architecture for an application scenario.

[0016] Figure 3 This is a schematic diagram illustrating the fault and recovery process of a monitoring process, provided as an embodiment of this specification.

[0017] Figure 4 This is a flowchart illustrating a fault and recovery process for a data acquisition process, provided as an embodiment of this specification.

[0018] Figure 5 This is a schematic diagram of the structure of a data acquisition device for an Internet of Things (IoT) edge gateway, provided as an embodiment of this specification. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0020] This specification provides a data acquisition method for an IoT edge gateway. It should be noted that the execution entity in this specification can be a server or any device with data processing capabilities. Figure 1 This is a flowchart illustrating a data acquisition method for an IoT edge gateway provided in an embodiment of this specification, as shown below. Figure 1 As shown, the main steps include the following:

[0021] Step S101: Determine at least one monitoring process and a corresponding plurality of data collection processes pre-deployed on each server in the edge gateway cluster.

[0022] In one embodiment of this specification, the edge gateway system adopts a layered architecture, which includes two programs: a monitoring process and a data acquisition process. Through division of labor and cooperation, resource allocation is optimized and the overall performance of the system is improved. Figure 2 This is a schematic diagram of a system architecture for an application scenario provided in an embodiment of this specification, such as... Figure 2As shown, the edge gateway cluster includes a monitoring process and multiple corresponding acquisition processes. The monitoring process is used for inter-process synchronization and coordination, as well as emergency handling and failover in case of failure. Its main functions include monitoring, failover, and acquisition task scheduling. The monitoring function continuously monitors the running status of the acquisition processes to ensure their normal operation. The failover mechanism allows for rapid reassignment of tasks to other normally operating acquisition processes when a failure is detected. Similarly, if a monitoring process on a server fails, the system can intelligently transfer all acquisition tasks from that server to other healthy servers, ensuring the continuity and stability of data acquisition. The acquisition task scheduling function intelligently allocates and schedules tasks based on the real-time load of each acquisition process to achieve load balancing and improve system efficiency. The acquisition processes execute data acquisition tasks, sharing the data acquisition burden and improving the system's data processing capabilities through collaboration with the monitoring process.

[0023] In one embodiment of this specification, a monitoring process and multiple data acquisition processes are pre-deployed on each server of the edge gateway cluster, fully utilizing the server's computing resources while ensuring high availability and load balancing of the system. The monitoring processes on different servers exchange information through an efficient communication mechanism, achieving data synchronization and status sharing. Simultaneously, the monitoring process is responsible for scheduling the data acquisition tasks of the data acquisition processes, ensuring the orderly progress of data acquisition.

[0024] Step S102: Obtain the pre-configured acquisition tasks, and allocate the acquisition tasks among multiple acquisition processes according to the preset dynamic load balancing algorithm through the monitoring process to determine the multiple online acquisition processes corresponding to the acquisition tasks.

[0025] In the field of IoT data acquisition, traditional task allocation methods heavily rely on static device lists. These lists are based on information such as device access status and data transmission requirements at specific time points, and are relatively stable within a certain period. However, in actual IoT operating environments, device scale and data traffic are constantly changing. For example, in scenarios involving sudden device access, task allocation strategies often fail to consider these suddenly accessing devices, leading to data backlogs as the acquisition process cannot collect and process data from these devices in a timely manner. Furthermore, in complex IoT environments, devices may go offline for various reasons, such as hardware failures or unstable network connections. When devices are offline, task allocation based on static device lists cannot adjust acquisition tasks in a timely manner. Acquisition resources originally allocated to offline devices become idle, while other normally operating devices with large data volumes cannot access these idle resources, resulting in low resource utilization efficiency and an inability to achieve reasonable resource allocation. In summary, traditional task allocation methods relying on static device lists lack effective mechanisms to cope with dynamic changes in device scale and data traffic, such as sudden device access and offline events, leading to uneven resource utilization.

[0026] This monitoring process allocates the acquisition task among multiple acquisition processes according to a preset dynamic load balancing algorithm to determine the multiple online acquisition processes corresponding to the acquisition task. Specifically, this includes: acquiring real-time load metric data for the acquisition process, including CPU utilization, memory utilization, and network bandwidth utilization; determining the comprehensive load coefficient for the acquisition process based on the real-time load metric data, and then determining the dynamic weight of each acquisition process based on this comprehensive load coefficient; mapping the number of virtual slots using the dynamic weight of the acquisition process to determine the number of virtual slots corresponding to each acquisition process; and performing a weighted round-robin allocation among the multiple acquisition processes based on the number of virtual slots corresponding to each acquisition process to determine the multiple online acquisition processes corresponding to the acquisition task.

[0027] In one embodiment of this specification, the monitoring process collects three-dimensional resource metrics of each process at preset intervals (e.g., 200ms) using an embedded probe: CPU utilization Ci, memory utilization Mi, and network bandwidth utilization Ni. Simultaneously, it calculates the resource fluctuation variance over past time periods; for example, the resource fluctuation variances corresponding to CPU utilization, memory utilization, and network bandwidth utilization over the past 5 minutes are respectively... , , According to the comprehensive load factor ,in A value of 0.1 is used as the historical fluctuation penalty coefficient to avoid nodes with load oscillations. Setting the historical fluctuation penalty coefficient can prevent nodes with load oscillations from participating in task allocation and prevent the system stability from being affected by frequent and large fluctuations in node load. To prevent small constants from having a denominator of 0, α, β, and γ can be set according to requirements, for example, to 0.4, 0.3, and 0.3, and dynamically adjusted according to the business scenario. The obtained comprehensive load coefficient is used to calculate the dynamic weight corresponding to the data acquisition process using the formula Wi = 1 / Li.

[0028] First, nodes that are offline or awaiting restart (referring to data collection processes) are removed from the current allocation pool. Based on the product of each node's weight and a preset fixed total number of slots, a virtual slot count is generated for each data collection process. For ease of probability mapping, the fixed total number of slots can be set to 100. Taking 100 as an example, assuming W1 is 0.6, W2 is 0.3, and W3 is 0.1, corresponding to 60, 30, and 10 slots respectively, a virtual slot sequence is constructed, dividing the slot range of 0-99 proportionally into P1: [0-59], P2: [60-89], and P3: [90-99]. Each time a task is allocated, the corresponding data collection process is selected according to the slot order. In the example above, the first data collection process has the highest weight, occupying 60% of the slots, meaning it has a 60% probability of being selected; that is, the node with the lightest load undertakes the most tasks. As tasks are assigned, as the load on the first acquisition process gradually increases, its weight may drop to 0.4, the slot may be adjusted to 40, and subsequent task assignments will automatically reduce the frequency to that node.

[0029] By comprehensively considering three-dimensional resource indicators such as CPU utilization, memory utilization, and network bandwidth utilization, as well as the resource fluctuation variance over past time periods, the above technical solution calculates a comprehensive load coefficient and dynamic weights using complex formulas. This allows for a more accurate reflection of the actual load capacity of the data acquisition process. Virtual slots are allocated according to weight ratios, and data acquisition processes are selected in slot order, ensuring that the lightest-loaded nodes handle the most tasks. Furthermore, the task allocation frequency is automatically adjusted as node load changes, achieving dynamic and reasonable task allocation. Reasonable task allocation ensures resources are more effectively distributed to each data acquisition process, avoiding situations where some processes are overloaded while others are idle. α, β, and γ can be dynamically adjusted according to business scenarios, meeting the different priorities of CPU, memory, and network resources in various IoT application scenarios. In scenarios with high network transmission requirements, the value of γ can be appropriately increased to favor data acquisition processes with high network bandwidth utilization, improving system efficiency in specific scenarios.

[0030] Step S103: The data acquisition task is executed through multiple online acquisition processes, and the health status between the monitoring process and the multiple online acquisition processes is detected to determine the real-time health status data.

[0031] The health status of the monitoring process and the multiple online data acquisition processes is detected to determine real-time health status data. Specifically, this includes: through information interaction between the multiple monitoring processes in the gateway cluster, and according to a preset majority decision-making mechanism, detecting the health status of the monitoring process to determine the first health status data corresponding to the monitoring process; and through the monitoring process, detecting the health status of the multiple online data acquisition processes to determine the second health status data corresponding to the online data acquisition process.

[0032] In one embodiment of this specification, when the monitoring processes of different servers start, they broadcast their own server's data collection configuration information to all monitoring processes. Based on the configuration information, the monitoring processes start the necessary data collection processes, ensuring that the data collection processes are ready. Through information exchange among multiple monitoring processes within the gateway cluster, and according to a preset majority decision-making mechanism, a health status check is performed on the monitoring process to determine the first health status data corresponding to the monitoring process. Figure 3 This is a flowchart illustrating the fault and recovery process of a monitoring process, as provided in the embodiments of this specification. Figure 3 As shown, the monitoring process records all received collection configurations to ensure information integrity. Monitoring processes perform health checks by sending ping messages, marking processes that respond normally as online, and those that fail to respond after multiple retries as pending confirmation. When a majority of monitoring processes mark a particular process as pending confirmation, that process is ultimately marked as offline, and other processes are notified.

[0033] Figure 4 This is a flowchart illustrating a fault and recovery process for a data acquisition process, as provided in an embodiment of this specification. Figure 4 As shown, after assigning the devices to be collected to each collection process, a notification to start collection is sent to the collection process to officially initiate the data collection work. The monitoring process checks the health status of these multiple online collection processes to determine the corresponding secondary health status data. Specifically, the monitoring process periodically sends ping messages to the collection processes, and the collection processes reply with pong messages to confirm their online status. If a collection process does not reply with a pong message within a specified time, the monitoring process will retry sending the ping message. If no reply is received three times consecutively, the process is marked as pending restart. The monitoring process attempts to restart the collection processes in the pending restart state. If the collection process still cannot respond after restarting, it is marked as offline.

[0034] Through information interaction among multiple monitoring processes within the gateway cluster, and based on a preset majority decision-making mechanism, a health status check is performed on the monitoring process to determine the first health status data corresponding to the monitoring process. Specifically, this includes: periodically collecting the first real-time resource data corresponding to the monitoring process according to a preset period, and broadcasting it to other monitoring processes within the gateway cluster. The first real-time resource data includes real-time CPU data, real-time memory data, and real-time thread count. Other monitoring processes within the gateway cluster perform hash calculations on the first real-time resource data to determine the real-time resource fingerprint data corresponding to the monitoring process. Based on the real-time resource fingerprint data and the majority decision-making mechanism, a health status check is performed on the monitoring process to determine the first health status data corresponding to the monitoring process.

[0035] Based on the real-time resource fingerprint data and the majority decision-making mechanism, a health status check is performed on the monitoring process to determine the first health status data corresponding to the monitoring process. Specifically, this includes: if the deviation between the real-time resource fingerprint data received by any other monitoring process and the first fingerprint of the monitoring process is not greater than a preset threshold, the monitoring process is judged to be in a healthy state; if the first fingerprint deviation exceeds the threshold N times consecutively, the resource fingerprint sequence of the monitoring process in the historical time period is collected to predict the predicted resource fingerprint in the next time period; when the second fingerprint deviation between the predicted resource fingerprint and the real-time resource fingerprint data exceeds a preset threshold, the monitoring process is determined to be a monitoring process to be confirmed, a voting request is initiated to the gateway cluster, and the first health status data corresponding to the monitoring process is determined based on the voting result.

[0036] In one embodiment of this specification, real-time resource data, such as real-time CPU data, real-time memory data, and real-time thread count, of the monitoring process are collected at preset intervals and broadcast to other monitoring processes within the gateway cluster to achieve information sharing. Upon receiving the broadcast data, other monitoring processes perform hash calculations to transform the complex resource data into representative real-time resource fingerprint data, facilitating subsequent comparison and analysis. The real-time resource fingerprint data received by each monitoring process is compared with a preset local cluster average to calculate the first fingerprint deviation. If the deviation is not greater than a preset threshold, the monitoring process is determined to be in a healthy state; if the first fingerprint deviation exceeds the threshold N times consecutively, the process proceeds to the next predictive analysis step. Resource fingerprint sequences within the historical time period of the monitoring process are collected to predict the predicted resource fingerprint for the next time period. If the second fingerprint deviation between the predicted resource fingerprint and the real-time resource fingerprint data exceeds a preset threshold, the monitoring process is determined to be a monitoring process awaiting confirmation. For the monitoring process awaiting confirmation, a voting request is initiated to the gateway cluster. By combining the voting results of other monitoring processes within the cluster, the first health status data of the monitoring process is determined, and the final health status conclusion is obtained.

[0037] For example, each monitoring process periodically (e.g., every 10 seconds) reports its own resource fingerprint (CPU / memory / thread count) hash value: Fingerprint = Hash(CPU% | Memory% | ThreadCount). If the received fingerprints from other processes deviate from the local cluster average by less than 5%, it is considered a normal response. If the fingerprint deviation exceeds the threshold for three consecutive times, time series analysis is initiated: the fingerprint sequence of the process over the past 5 minutes is collected, and the fingerprint value for the next period is predicted using a predictive model, such as a Long Short-Term Memory (LSTM) model. If the difference between the actual value and the predicted value exceeds 20%, it is determined to be an abnormal process. The monitoring process initiates a voting request to the cluster. If more than (N+1) / 2 monitoring processes (rounded up) agree, it is marked as an offline node, triggering task takeover. Here, N is the number of monitoring processes.

[0038] The above technical solution involves multiple monitoring processes and a majority decision-making mechanism, avoiding the bias of judgments from a single monitoring process. By combining real-time data with historical data predictive analysis, the health status of monitoring processes is comprehensively evaluated from multiple dimensions, reducing the risk of misjudgment and providing more accurate health status information. Pre-set data collection cycles and dynamic adjustment of judgment strategies can adapt to the dynamic changes in resource usage of monitoring processes in the IoT environment, effectively responding to resource fluctuations and ensuring the timeliness and effectiveness of health status detection. Through the health status detection mechanism, potential problems can be identified and addressed in a timely manner, reducing data collection interruptions or errors caused by monitoring process failures and improving the reliability of the entire IoT edge gateway data collection system.

[0039] The monitoring process detects the health status of multiple online acquisition processes to determine the second health status data corresponding to each online acquisition process. Specifically, this includes: sending a first number of ping commands to the online acquisition process through the monitoring process; when no ping command is received from the online acquisition process for a first preset threshold number of consecutive times, acquiring the real-time resource data of the online acquisition process; performing a secondary status detection on the online acquisition process based on the real-time resource data and pre-set historical baseline data; and when the real-time resource data and the historical baseline data meet preset conditions, sending a simulated acquisition command to determine the execution result data of the online acquisition process, and determining the second health status data corresponding to the online acquisition process based on the execution result data.

[0040] In one embodiment of this specification, the monitoring process sends a first number of ping commands to the online acquisition process, and determines whether the online acquisition process is operating normally by using the pong commands returned by the online acquisition process. If a pong command is received in a timely manner, it can be preliminarily considered that the acquisition process is in a normal working state. When no pong command is received for a first preset threshold number of consecutive times, it indicates that the acquisition process may have a problem. At this time, relying solely on ping commands is insufficient to accurately determine its status. Therefore, the monitoring process will obtain the real-time resource data of the online acquisition process for more in-depth detection.

[0041] The acquired real-time resource data is compared and analyzed with pre-set historical baseline data. Historical baseline data is accumulated over a long period based on the resource usage of the acquisition process under normal operating conditions. By comparing the differences between the two, the status of the acquisition process is judged to be abnormal according to preset conditions. Preset conditions can be such as the deviation range of resource data; for example, when the real-time CPU utilization exceeds a certain percentage of the historical baseline data, it is considered abnormal. When the real-time resource data and historical baseline data meet the preset conditions, i.e., there is a situation that may affect the normal operation of the acquisition process, the monitoring process sends a simulated acquisition command. The simulated acquisition command simulates a real data acquisition task, and the acquisition process returns execution result data after executing the command. Based on this execution result data, the monitoring process ultimately determines whether the online acquisition process is offline. If the execution result data shows that the execution verification failed, or the execution result data is empty, the online acquisition process is determined to be offline.

[0042] For example, the monitoring process sends a 1-byte ping command per second, and the acquisition process immediately replies with a pong. If no pong is received twice consecutively, the process switches to resource probe mode, and the acquisition process needs to return its current CPU / memory usage. If the resource data deviates too much from the historical baseline (e.g., CPU suddenly increases to 90%), a simulated acquisition command is sent (e.g., reading the device's 0x00 register) to verify the actual execution capability. Based on this execution result data, it is finally determined whether the online acquisition process is offline.

[0043] The above technical solution goes beyond a simple ping-pong mechanism. Instead, it combines real-time resource data with historical baseline data and simulated acquisition command execution results for comprehensive judgment. This avoids misjudgments caused by accidental factors such as network fluctuations, and can more accurately determine the health status of the online acquisition process, providing a reliable basis for subsequent decision-making. Through multi-layered detection methods, it can better adapt to various complex situations. Whether it is hardware failure, software anomaly, or network fluctuation, it can effectively detect changes in the health status of the acquisition process, ensuring that the system can operate stably even in complex environments.

[0044] Step S104: When the real-time health status data contains anomalies in the specified monitoring process and / or the specified online acquisition process, a takeover target is determined in the edge gateway cluster so as to process the acquisition tasks of the specified monitoring process and / or the specified online acquisition process through the takeover target, thereby realizing the data acquisition of IoT devices.

[0045] In this edge gateway cluster, the takeover target is determined by: when the specified online acquisition process is abnormal, filtering among multiple other online acquisition processes on the server through the monitoring process to determine the acquisition process to take over; processing the acquisition tasks of the specified online acquisition process through the takeover acquisition process and monitoring the health status of the specified online acquisition process in real time; and restoring the acquisition task to the specified online acquisition process when the health status of the specified online acquisition process returns to normal.

[0046] In one embodiment of this specification, in an IoT edge gateway data acquisition system, a monitoring process plays a crucial role when a designated online acquisition process malfunctions. The monitoring process filters out a takeover acquisition process from among multiple other online acquisition processes on the same server. This filtering process can be achieved through the load balancing algorithm in step S102 combined with the current load of other acquisition processes, aiming to select the most suitable process to take over the tasks of the malfunctioning acquisition process. After determining the takeover acquisition process, the acquisition task of the malfunctioning acquisition process is immediately executed to ensure uninterrupted data acquisition. Simultaneously, while the takeover acquisition process is processing the acquisition tasks of the designated online acquisition process, the monitoring process monitors the health status of the designated online acquisition process in real time. Once the offline acquisition process returns to online status, the monitoring process reallocates the previously transferred tasks back to that process.

[0047] By employing the aforementioned technical solutions, when an anomaly occurs in the data acquisition process, a takeover process can be quickly identified to continue the acquisition task, preventing data acquisition interruptions due to process anomalies. This ensures that data generated by the device can be continuously collected, guaranteeing that relevant business processes are not affected, such as ensuring that data from scenarios like industrial production monitoring and intelligent environmental monitoring is not lost. By monitoring the health status of abnormal acquisition processes in real time and resuming tasks when they return to normal, system resources are fully utilized. Even if some processes experience temporary failures, stable operation can be maintained, enhancing the reliability and fault tolerance of the entire edge gateway data acquisition system and reducing the risk of serious system impact caused by the failure of a single acquisition process.

[0048] In this edge gateway cluster, the takeover target is determined, specifically including: when the specified monitoring process is abnormal, the collection tasks of multiple specified collections corresponding to the specified monitoring process are suspended, and the target server with the lowest load is selected in the edge gateway cluster; the takeover target is determined based on the target server, and the collection tasks under the specified monitoring process are taken over.

[0049] In one embodiment of this specification, within the IoT edge gateway data acquisition architecture, when an anomaly is detected in a designated monitoring process, a series of operations are executed sequentially to ensure the continuity of data acquisition. First, multiple acquisition tasks scheduled and managed by the designated monitoring process are immediately suspended. This is to prevent confusion or erroneous execution of acquisition tasks when the monitoring process is abnormal, thus preventing data acquisition errors or incompleteness. Next, the entire edge gateway cluster is filtered to find the target server with the lowest load. This requires real-time acquisition of the load information of each server in the cluster, and comparison to determine the server with the lightest load as the target. Finally, the selected target server is designated as the takeover target to take over the acquisition tasks originally handled by the abnormal monitoring process. Even if a monitoring process becomes abnormal, data acquisition tasks can be quickly transferred to other servers for continued execution. Once the offline monitoring process returns to normal, the previously transferred tasks will be redistributed back to the acquisition processes under the original monitoring process.

[0050] The above technical solutions enable timely termination of data collection tasks when monitoring processes malfunction, preventing data collection from being run under incorrect scheduling or management due to monitoring process failures. This prevents the collection of erroneous, duplicate, or incomplete data, ensuring data quality. By selecting the server with the lowest load in the edge gateway cluster for task takeover, the system can effectively handle sudden situations where monitoring processes malfunction. Even if some monitoring processes fail, the entire system can continue to operate without completely halting data collection due to a problem with a single monitoring process. Selecting the server with the lowest load as the takeover target ensures the smooth progress of data collection tasks while minimizing additional consumption of cluster resources. This avoids assigning tasks to servers with excessive loads, which could lead to performance degradation or even new failures. This achieves reasonable resource allocation and efficient utilization, improving the overall operating efficiency of the edge gateway cluster.

[0051] The technical solutions in the embodiments of this specification address the limitations of traditional centralized data processing models, which are constrained by bandwidth and latency, making them inadequate for large-scale data acquisition. In contrast, the embodiments of this specification, through a preset dynamic load balancing algorithm, can rationally distribute acquisition tasks across multiple online acquisition processes based on real-time load conditions such as CPU utilization, memory utilization, and network bandwidth utilization. This effectively solves the problem of insufficient processing capacity of a single node, avoids low acquisition efficiency due to excessive data volume, and significantly improves data acquisition speed. It can quickly process massive concurrent data, meeting the high-efficiency requirements of modern IoT data acquisition. In traditional models, a single point of failure at the central node can interrupt data acquisition, and traditional distributed architectures relying on single heartbeat detection are prone to misjudgments and omissions. The embodiments of this specification accurately identify abnormal processes through comprehensive health status detection of monitoring and online acquisition processes. Once an abnormality is detected in a designated monitoring or online acquisition process, a takeover target can be quickly determined in the edge gateway cluster, automatically completing task takeover without manual intervention. This greatly reduces the risk of data acquisition interruption and ensures system stability and reliability. Traditional data acquisition methods… Traditional methods rely on manual intervention for task migration in abnormal situations, resulting in takeover delays of up to minutes, which cannot meet high real-time requirements. The embodiments described in this specification can monitor process status in real time. When real-time health status data shows anomalies, it can quickly identify the takeover target and complete task migration, achieving seamless task transition with extremely short delays. This ensures timely data collection in scenarios with extremely high real-time requirements, such as industrial automation and intelligent transportation. The scale of IoT devices and data traffic are constantly changing dynamically, making traditional methods difficult to adapt. The embodiments described in this specification are based on an edge gateway cluster architecture, which can flexibly handle situations such as increased data volume due to new device access, device offline, and process anomalies. The dynamic load balancing algorithm can adjust task allocation at any time, automatically detect and handle abnormal processes, and achieve system self-optimization and adjustment. It can operate stably in complex and ever-changing IoT environments, possessing strong adaptability and scalability.

[0052] This specification also provides an embodiment of a data acquisition device for an IoT edge gateway, such as... Figure 5 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0053] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0054] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0056] The devices, media, and methods provided in the embodiments of this specification are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0057] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0062] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0063] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0064] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A data acquisition method for an IoT edge gateway, characterized in that, The method includes: Identify at least one monitoring process and corresponding multiple data collection processes pre-deployed on each server in the edge gateway cluster; The pre-configured collection tasks are obtained, and the collection tasks are allocated among the multiple collection processes according to the preset dynamic load balancing algorithm through the monitoring process to determine the multiple online collection processes corresponding to the collection tasks. The collection task is executed through the multiple online collection processes, and the health status between the monitoring process and the multiple online collection processes is detected to determine real-time health status data. When the real-time health status data contains anomalies in a specified monitoring process and / or a specified online data acquisition process, a takeover target is determined in the edge gateway cluster. The data acquisition tasks of the specified monitoring process and / or the specified online data acquisition process are then processed through the takeover target to realize data acquisition from IoT devices. Through the monitoring process, the data acquisition tasks are allocated among the multiple data acquisition processes according to a preset dynamic load balancing algorithm, thereby determining the multiple online data acquisition processes corresponding to each data acquisition task, specifically including: Obtain real-time load metrics data of the acquisition process, wherein the real-time load metrics data includes CPU utilization, memory utilization, and network bandwidth utilization. Calculate the resource fluctuation variance over past time periods to determine the resource fluctuation variance corresponding to CPU utilization. Resource fluctuation variance corresponding to memory utilization Resource fluctuation variance corresponding to network bandwidth utilization ; Based on the real-time load index data and the resource fluctuation variance, the comprehensive load coefficient corresponding to the acquisition process is determined, and the dynamic weight of each acquisition process is determined based on the comprehensive load coefficient. Wherein, the comprehensive load factor Ci represents CPU utilization, Mi represents memory utilization, and Ni represents network bandwidth utilization. A constant used to prevent the denominator from being zero. α represents the historical fluctuation penalty coefficient, and β and γ are the indicator demand weights corresponding to CPU utilization, memory utilization, and network bandwidth utilization, respectively. The number of virtual slots is mapped by the dynamic weight of the acquisition process to determine the number of virtual slots corresponding to each acquisition process. Based on the number of virtual slots corresponding to each acquisition process, a weighted round-robin allocation is performed among the multiple acquisition processes to determine the multiple online acquisition processes corresponding to the acquisition task.

2. The data acquisition method for an IoT edge gateway according to claim 1, characterized in that, The health status of the monitoring process and the multiple online data acquisition processes is detected to determine real-time health status data, specifically including: Through information interaction among multiple monitoring processes within the gateway cluster, and based on a preset majority decision-making mechanism, the health status of the monitoring processes is detected to determine the first health status data corresponding to the monitoring process. The monitoring process detects the health status of the multiple online data acquisition processes to determine the second health status data corresponding to each online data acquisition process.

3. The data acquisition method for an IoT edge gateway according to claim 2, characterized in that, Through information exchange among multiple monitoring processes within the gateway cluster, and based on a preset majority decision-making mechanism, the health status of the monitoring processes is checked to determine the first health status data corresponding to the monitoring process, specifically including: According to a preset period, the first real-time resource data corresponding to the monitoring process is periodically collected and broadcast to other monitoring processes in the gateway cluster. The first real-time resource data includes real-time CPU data, real-time memory data and real-time thread count. The first real-time resource data is hashed by other monitoring processes within the gateway cluster to determine the real-time resource fingerprint data corresponding to the monitoring process. Based on the real-time resource fingerprint data and the majority decision-making mechanism, the monitoring process is subjected to health status detection to determine the first health status data corresponding to the monitoring process.

4. The data acquisition method for an IoT edge gateway according to claim 2, characterized in that, The monitoring process detects the health status of the multiple online data acquisition processes to determine the second health status data corresponding to each online data acquisition process, specifically including: The monitoring process sends a first number of ping commands to the online data acquisition process. If the online acquisition process fails to receive a pong instruction from the online acquisition process for a first preset number of consecutive thresholds, the real-time resource data of the online acquisition process is obtained. Based on the real-time resource data and the pre-set historical baseline data, a secondary status detection is performed on the online acquisition process; When the real-time resource data and the historical baseline data meet preset conditions, a simulated acquisition command is sent to determine the execution result data of the online acquisition process, so as to determine the second health status data corresponding to the online acquisition process based on the execution result data.

5. The data acquisition method for an IoT edge gateway according to claim 3, characterized in that, Based on the real-time resource fingerprint data and the majority decision-making mechanism, a health status check is performed on the monitoring process to determine the first health status data corresponding to the monitoring process, specifically including: If the deviation between the real-time resource fingerprint data of the monitoring process received by any other monitoring process and the first fingerprint of the preset local cluster average value is not greater than a preset threshold, the monitoring process is determined to be in a healthy state. If the first fingerprint deviation exceeds the threshold N times consecutively, the resource fingerprint sequence of the monitoring process in the historical time period is collected to predict the predicted resource fingerprint in the next time period. When the second fingerprint deviation between the predicted resource fingerprint and the real-time resource fingerprint data exceeds a preset threshold, the monitoring process is determined to be a monitoring process to be confirmed, a voting request is sent to the gateway cluster, and the first health status data corresponding to the monitoring process is determined based on the voting result.

6. The data acquisition method for an IoT edge gateway according to claim 1, characterized in that, Determining the takeover target within the edge gateway cluster specifically includes: When the specified online data collection process is abnormal, the monitoring process filters among multiple other online data collection processes in the server to determine which data collection process will take over. The acquisition process is taken over to process the acquisition tasks of the designated online acquisition process and to monitor the health status of the designated online acquisition process in real time. When the health status of the designated online acquisition process returns to normal, the acquisition task will be restored to the designated online acquisition process.

7. The data acquisition method for an IoT edge gateway according to claim 1, characterized in that, Determining the takeover target within the edge gateway cluster specifically includes: When the specified monitoring process is abnormal, the collection tasks of multiple specified collections corresponding to the specified monitoring process are suspended, and the target server with the least load is selected in the edge gateway cluster. The target server is used to determine the takeover target, and the data collection tasks under the specified monitoring process are taken over.

8. A data acquisition device for an Internet of Things (IoT) edge gateway, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-7.

Images