Data analysis distributed processing method for large-scale Internet of Things network

Abstract

The invention belongs to the technical field of Internet of Things big data, and particularly relates to a data analysis distributed processing method oriented to a large-scale Internet of Things network, which comprises the following steps: S1, dividing each node at the edge end of the Internet of Things network into a plurality of partitions; s2, judging whether a data processing task belongs to a complex task or not; if yes, sending to the cloud for processing; if not, turning to S3; s3, judging whether the node can process the task at present or not; if yes, processing and sending a processing result to the cloud; if not, turning to S4; s4, judging whether other nodes capable of processing the task currently exist in the partition where the node is located or not, if so, selecting a target node from the other nodes, and turning to S5; if not, selecting a target node from other partitions, and turning to S5; and S5, sending the task to the target node for processing, and sending a processing result to the cloud. According to the method, edge computing resources can be efficiently utilized, the method adapts to the characteristics of a large-scale Internet of Things network, and efficient and low-delay data processing and analysis are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things big data, and in particular relates to a data analysis distributed processing method for large-scale Internet of Things networks. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, the number of IoT devices around the world is growing at an unprecedented rate. According to Statista data, by 2023, the number of IoT devices deployed worldwide has exceeded 25 billion, and this number is expected to continue to rise in the next few years. The widespread application of IoT devices, such as smart cities, Industry 4.0, smart agriculture, and medical health, has not only greatly enriched people's lives and work methods, but also generated massive amounts of data. These data have posed unprecedented challenges to data storage, processing, and analysis capabilities.

[0003] When processing IoT data, two main computing architectures are currently used: centralized computing architecture and edge computing architecture. The centralized computing architecture transmits the data generated by IoT devices to the cloud through the network for storage and analysis, such as the AWS IoT platform and the Microsoft Azure IoT platform. This architecture can handle complex data analysis tasks with the powerful computing and storage capabilities of the cloud. However, with the surge in the number of IoT devices, the centralized computing architecture has gradually exposed problems such as data transmission delays, increased network bandwidth pressure, and increased security risks. In particular, in application scenarios with high real-time requirements, the centralized computing architecture is difficult to meet the needs of low latency and high timeliness.

[0004] In order to overcome the limitations of centralized computing architecture, edge computing architecture came into being. Edge computing delegates some data processing tasks to edge devices close to the data source, such as fog computing and edge computing nodes. This architecture effectively reduces latency by reducing the distance and time for data to be transmitted to the cloud, and reduces the computing pressure and network transmission burden on the cloud. However, edge computing architecture also faces many challenges. Due to the limited computing and storage capabilities of edge devices, they are often unable to handle complex data analysis tasks. In addition, the management complexity of edge computing nodes is high, and the load balancing problem also needs to be solved urgently.

[0005] In summary, centralized computing architecture and edge computing architecture each have their own advantages and disadvantages when processing large-scale IoT data. Although the centralized architecture has powerful computing and storage capabilities, it is difficult to meet the requirements of low latency and high timeliness; while the edge computing architecture can reduce latency and reduce cloud load, it is insufficient to handle complex data analysis tasks in a resource-limited environment. This situation has seriously affected the efficiency and accuracy of IoT data processing and analysis. The main problem with the existing technology is how to efficiently process large-scale IoT data in a resource-limited edge computing environment while meeting the requirements of low latency and high timeliness. The main reason for this problem is the limited computing and storage resources of edge devices, as well as the complexity of node management and load balancing. The difficulty in solving this problem lies in the fact that on the one hand, it is necessary to make full use of the computing resources of edge devices to improve data processing efficiency; on the other hand, it is necessary to ensure the accuracy and security of data processing to avoid risks such as data loss or leakage.

[0006] Therefore, how to efficiently utilize edge computing resources while adapting to the characteristics of large-scale IoT networks and achieving efficient, low-latency data processing and analysis has become a problem that needs to be solved urgently. Summary of the invention

[0007] In view of the shortcomings of the above-mentioned prior art, the present invention provides a distributed processing method for data analysis for large-scale Internet of Things networks, which can adapt to the characteristics of large-scale Internet of Things networks while efficiently utilizing edge computing resources to achieve efficient and low-latency data processing and analysis.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A distributed processing method for data analysis in a large-scale Internet of Things network comprises the following steps:

[0010] S1. Divide each node at the edge of the IoT network into multiple partitions according to a preset partitioning method;

[0011] S2, after obtaining the collected data of the corresponding device layer through the node, determine whether the data processing task belongs to the preset complex task; if it is a complex task, send the task to the cloud for processing; if it is not a complex task, go to S3;

[0012] S3: Determine whether the node can currently process the task based on the node load; if so, the node processes the task and sends the processing result to the cloud; if not, go to S4;

[0013] S4: Determine whether there are other nodes in the partition where the node is located that can currently process the task. If so, select them and select other nodes with the best transmission path as the target node, and go to S5; if not, select other nodes that can currently process the task from other partitions, select other nodes with the best path as the target node, and go to S5;

[0014] S5. Send the task to the target node for processing, and send the processing result to the cloud.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. This method can efficiently utilize edge computing resources and optimize resource allocation through partition management. Through the preset partitioning method, each node at the edge of the IoT network is divided into multiple partitions, making resource management and task allocation more flexible and efficient. This partitioning strategy helps to reasonably allocate nodes based on factors such as their geographical location and computing power, thereby maximizing the use of computing resources on edge devices.

[0017] 2. This method can improve data processing efficiency. Task hierarchical processing: According to the complexity of data processing tasks, tasks are divided into complex tasks and non-complex tasks. Complex tasks are sent directly to the cloud for processing to ensure the accuracy and security of the processing results; non-complex tasks are processed by edge nodes, which reduces data transmission delays and bandwidth consumption and improves processing efficiency. Load balancing mechanism: When a node cannot process a task, it determines whether there are other nodes in the partition where the node is located or other partitions that can process the task, and selects the node with the optimal transmission path as the target node, thereby realizing dynamic allocation and load balancing of tasks. This helps to avoid overloading of a single node and improve the stability of the entire system and data processing efficiency.

[0018] 3. This method can ensure the accuracy and security of data processing. The cloud processes complex tasks; for complex tasks, they are sent to the cloud for processing, and the powerful computing power of the cloud is used to ensure the accuracy of the processing results. At the same time, the cloud also has higher security protection capabilities, which helps to avoid risks such as data leakage. The edge node processes non-complex tasks; for non-complex tasks, the edge node directly processes and sends the processing results to the cloud. This processing method reduces the links of data transmission, reduces the risk of data loss, and also helps to improve the timeliness of data processing.

[0019] In summary, this method can adapt to the characteristics of large-scale IoT networks while making efficient use of edge computing resources, and achieve efficient and low-latency data processing and analysis. Through partition management to optimize resource allocation, task hierarchical processing, load balancing mechanism, and the collaborative work of cloud and edge nodes, this method not only improves data processing efficiency, but also ensures the accuracy and security of data processing.

[0020] Preferably, in S4, the optimal transmission path is a data transmission path with the lowest transmission delay.

[0021] Such a setting can significantly reduce the delay time of data during transmission by selecting the data transmission path with the lowest transmission delay. This is especially important for application scenarios that require real-time processing or low-latency response, such as intelligent transportation systems and telemedicine. Selecting the optimal transmission path can avoid the waste of resources caused by poor paths during data transmission. For example, it can reduce the number of transmission failures or retransmissions caused by network congestion and reduce unnecessary consumption of network resources.

[0022] Preferably, in S4, when selecting the optimal transmission path, if an equivalent optimal path exists, then other nodes with the highest node weights are selected as target nodes from the equivalent optimal path and other nodes of the transmission path with the lowest delay; the equivalent optimal path is a transmission path whose delay difference with the transmission path with the lowest delay is less than a preset difference.

[0023] Such a setting, 1. Expand the selection range and improve flexibility. By introducing the concept of equivalent optimal path, the selection of the optimal transmission path is no longer limited to a single path with the lowest delay, but can be selected from multiple equivalent paths. This increases the flexibility of selection and helps to find a more suitable transmission path in specific situations (such as network congestion, node failure, etc.).

[0024] 2. Optimize node selection and improve system performance. When selecting the target node, in addition to considering the delay of the transmission path, the node weight is also combined. The node weight may be based on multiple factors such as the node's computing power, stability, and historical performance. By selecting the node with the highest weight, the efficiency of data processing and transmission can be further optimized, and the performance of the entire system can be improved.

[0025] 3. Enhance system robustness. When there is an equivalent optimal path, selecting the node with the highest weight as the target node helps to quickly switch to nodes on other equivalent paths when a node or path fails, thereby maintaining the continuity and stability of data transmission. This enhances the robustness and fault tolerance of the system.

[0026] 4. Balance latency and resource utilization. Although latency is an important consideration in selecting a transmission path, in some cases, a certain amount of latency may need to be sacrificed in order to optimize resource utilization or improve processing efficiency. By introducing the concepts of equivalent optimal paths and node weights, it is possible to better balance resource utilization and processing efficiency while maintaining lower latency.

[0027] Preferably, the calculation formula of the weight Wi of node i is:

[0028] W i =αC i +βM i +γB i ;

[0029] Where α, β, and γ are adjustment parameters, and Ci, Mi, and Bi represent the computing power, storage capacity, and transmission rate of node i, respectively.

[0030] Such a setting, 1. By comprehensively considering the computing power, storage capacity and transmission rate of the node, the calculation formula can comprehensively evaluate the comprehensive performance of the node. This helps to more accurately judge the processing power and transmission efficiency of the node when selecting the optimal transmission path and target node, so as to make more reasonable decisions. Considering the computing power, storage capacity and transmission rate of the node at the same time, this helps to achieve more reasonable resource allocation in the resource-constrained IoT environment. By selecting a node with a higher weight as the target node, its computing resources and storage capacity can be fully utilized to improve the efficiency of data processing and transmission.

[0031] 2. Flexible adjustment of weight coefficients. The adjustment parameters α, β, and γ can be flexibly adjusted according to the actual application scenarios and requirements. For example, in scenarios where high-speed data transmission is required, the weight coefficient γ of the transmission rate B_i can be increased; in scenarios where a large amount of data needs to be processed, the weight coefficient α of the computing power C_i can be increased. This flexibility makes the calculation formula applicable to different IoT application scenarios.

[0032] 3. Improve system reliability and stability. The calculation of node weights takes multiple factors into account, so that the performance and stability of the nodes can be more comprehensively evaluated when selecting target nodes. This helps to quickly switch to other nodes with better performance when a node or path fails, thereby maintaining the continuity and stability of data transmission and improving the reliability and stability of the entire system.

[0033] Preferably, in S4, when calculating the delay of the transmission path, the path P from node i to node j is ij The delay T ij for:

[0034]

[0035] Where, L ij For path P ij The physical distance or basic delay between nodes; w is the bandwidth requirement coefficient of the task data processing type;

[0036] Ui(1-ui) and Dj(1-dj) are the remaining effective bandwidths of nodes i and j respectively; Ui represents the uplink bandwidth of node i, ui represents the proportion of uplink bandwidth currently occupied by node i, and u i = Uplink data volume / U i ; Dj represents the downlink bandwidth of node j, dj represents the proportion of downlink bandwidth of node j currently occupied, d j = Downlink data volume / D i .

[0037] With this setup, 1. Accurately calculate the path delay. By introducing the physical distance (or basic delay) Lij, the bandwidth requirement coefficient w, and the node's remaining effective bandwidth Ui (1-ui) and Dj (1-dj), the formula can more accurately calculate the delay Tij of the path Pij. This helps to more accurately evaluate the transmission performance of different paths when selecting the optimal transmission path, thereby making more reasonable decisions.

[0038] 2. Consider bandwidth requirements and node load. The bandwidth requirement coefficient w in the formula reflects the bandwidth requirement of the task data processing type, while the node remaining effective bandwidth Ui (1-ui) and Dj (1-dj) take into account the current load of the node. This design allows the bandwidth requirements of the task and the processing capacity of the node to be considered simultaneously when selecting the transmission path, avoiding the selection of overloaded or insufficiently bandwidthd nodes, thereby improving the reliability and efficiency of data transmission.

[0039] 3. Adapt to dynamic network environments. Since the uplink and downlink bandwidth usage of nodes changes over time, this formula can reflect the load status of nodes in real time. This helps to adjust the transmission path according to the real-time load of nodes in a dynamically changing IoT environment, ensuring the continuity and stability of data transmission.

[0040] 4. Optimize resource utilization. By comprehensively considering the bandwidth resources and load conditions of the nodes, this formula helps to select nodes with sufficient remaining bandwidth and light load when selecting the transmission path, thereby optimizing resource utilization and improving the transmission performance of the entire IoT network.

[0041] 5. Improve system scalability. The design of this formula takes into account the dynamic changes in node bandwidth, allowing the system to adapt to IoT networks of different sizes and complexities. As the network scale expands and the number of nodes increases, the formula can still accurately calculate the path delay, providing strong support for selecting the optimal transmission path.

[0042] Preferably, if the task is data-intensive, its bandwidth requirement coefficient w is 1.5; if the task is computationally intensive, its bandwidth requirement coefficient w is 0.5; if the task is balanced, its bandwidth requirement coefficient w is 1.

[0043] This setting can 1. accurately match task requirements. By assigning different bandwidth requirement coefficients w to different types of tasks, the actual bandwidth requirements of the tasks can be more accurately reflected. Data-intensive tasks require higher bandwidth to transmit large amounts of data, so they are assigned a higher coefficient (1.5); compute-intensive tasks focus more on computing performance and have relatively low bandwidth requirements, so they are assigned a lower coefficient (0.5); balanced tasks are between the two and are assigned a medium coefficient (1). This design helps to more accurately consider task requirements when selecting a transmission path, thereby optimizing resource allocation.

[0044] 2. Optimize resource utilization. Adjusting the bandwidth demand coefficient w according to the task type helps achieve more reasonable resource utilization in resource-constrained IoT environments. For data-intensive tasks, selecting a transmission path with high bandwidth can ensure fast and complete data transmission; for computation-intensive tasks, more consideration can be given to the node's computing power and remaining bandwidth to avoid unnecessary bandwidth waste. This strategy helps maximize resource utilization efficiency while meeting task requirements.

[0045] Preferably, in S4, when other nodes that can currently process the task are screened out from other partitions, if there are multiple other partitions that have other nodes that meet the conditions, the other partition with the smallest global average deviation is selected first, and then other nodes are selected from the selected other partitions.

[0046] This setting, 1. Optimize load balancing. By selecting the partition with the smallest global average deviation for task transfer, the system can ensure that the task load between partitions remains relatively balanced. This helps to avoid performance degradation due to task overload in some partitions, while preventing other partitions from wasting resources due to insufficient tasks. Balanced task load can improve resource utilization and response speed of the entire system.

[0047] 2. Improve system stability. Balanced task load distribution helps reduce system uncertainty and volatility. When a partition is overloaded, it may cause performance bottlenecks, data loss, or transmission delays. By optimizing load balancing, the system can better cope with these potential risks, thereby improving overall stability and reliability.

[0048] Preferably, the global average deviation is calculated as:

[0049]

[0050] In the formula, ΔK k represents the global average deviation of partition k, represents the average load of each node in partition k; L total represents the global load of each edge node; N represents the number of edge nodes;

[0051] Among them, the global load L total The calculation formula is:

[0052]

[0053] Where Li represents the load of node i;

[0054] L i =f(Ci,Mi,Bi,Di);

[0055] In the formula, Ci represents CPU usage data; Mi represents memory usage data; Bi represents network bandwidth occupancy data; Di represents node geographic location or physical distance data; f() represents a comprehensive function that integrates various factors.

[0056] This setting can 1. Accurately quantify the load balancing status. Through the global average deviation ΔK k The calculation of can accurately quantify the degree of deviation of each partition relative to the global average load. This helps system administrators or algorithms to more accurately judge the load status of each partition when allocating and scheduling tasks, so as to make more reasonable decisions.

[0057] 2. Comprehensive evaluation of node load. The calculation of node load Li takes into account multiple factors such as CPU usage data, memory usage data, network bandwidth usage data, and node geographic location or physical distance data. This comprehensive evaluation method can more comprehensively reflect the actual load status of the node and avoid the one-sidedness of a single indicator evaluation.

[0058] Preferably, in S1, important data is also backed up between different nodes; if a node fails, the tasks of the node are automatically transferred to other nodes that back up its important data;

[0059] The requirement for the number of backups of important quantities is that the data reliability is greater than a preset reference value when the backup quantity is used; the calculation formula for data reliability is:

[0060] R=1-(1-p) k ;

[0061] In the formula, R represents data reliability; p is the probability that the node works normally; k is the number of redundant backups of data.

[0062] Such a setting can 1. Improve data reliability. Increasing the k value, that is, increasing the number of redundant data backups, can significantly improve data reliability. According to the data reliability calculation formula R = 1-(1-p)k, when p (the probability of a node working properly) is fixed, the larger the k value, the closer R (data reliability) is to 1, that is, the lower the risk of data loss. This is crucial for protecting critical business data and ensuring high availability of the system.

[0063] 2. Optimize storage overhead. Increasing the k value will also bring additional storage overhead. Each backup requires a certain amount of storage space, so the larger the k value, the more storage space is required. By properly adjusting the k value, the system can avoid unnecessary waste of storage resources while ensuring data reliability. For example, when the data reliability requirement is not particularly high, the k value can be appropriately reduced to reduce storage overhead; in scenarios where the data reliability requirement is extremely high, the k value can be appropriately increased to ensure absolute data security.

[0064] 3. Achieve flexible configuration. This strategy allows the system to flexibly adjust the k value according to actual needs, thereby achieving the best balance between reliability and storage overhead in different scenarios. This helps the system maintain efficient and stable operation under different loads, different data importance, and different resource constraints.

[0065] 4. Enhance system fault tolerance. By backing up important data between different nodes, the system has stronger fault tolerance. Even if a node fails, the task can be quickly transferred to other nodes with backed-up data, thus ensuring the continuity and stability of the system. This fault tolerance is of great significance to improving the overall performance of the system and user experience.

[0066] In summary, by adjusting the k value, the system can achieve a significant balance between data reliability and storage overhead. This strategy not only improves data reliability, but also optimizes the utilization of storage resources, enhances the system's fault tolerance, and provides strong support for the stable operation of the IoT system.

[0067] Preferably, in S1, the preset partitioning method is to divide nodes that are physically or network-close into one partition;

[0068] In S2, when the computational complexity of the data processing task is greater than a preset threshold, it is determined to be a complex task.

[0069] Such a setting, 1. Dividing nodes that are physically or network-close to each other into the same partition can reduce data transmission delays and costs. Because the communication links between these nodes are relatively short, data transmission speeds are faster and are not easily affected by network congestion and delays. In addition, dividing similar nodes into the same partition can simplify system management and maintenance. Administrators can more easily monitor, configure, and troubleshoot these nodes, thereby improving management efficiency.

[0070] 2. By setting the computational threshold, the system can accurately identify which tasks are complex and which are simple. This helps the system to process and optimize different types of tasks in a targeted manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0072] Figure 1 The system architecture of the method in the embodiment and the interaction between the layers are shown;

[0073] Figure 2 Flowchart of this method. DETAILED DESCRIPTION

[0074] The following is a further detailed description through specific implementation methods:

[0075] Embodiment 1

[0076] This embodiment discloses a distributed processing method for data analysis for a large-scale Internet of Things network.

[0077] In order to better understand this method, the system architecture of the distributed computing method for large-scale IoT network data is briefly described. When the method is implemented, its architecture mainly includes three main layers: IoT device layer (i.e. data acquisition end), edge computing layer (i.e. each node at the edge) and cloud computing center layer (i.e. cloud). Figure 1 The system architecture and the interactions between various layers are shown.

[0078] IoT device layer: contains multiple sensors and devices responsible for real-time data collection. Each device communicates with the edge node through a wireless network (such as Wi-Fi, LoRa, 5G) to upload data for further processing. IoT devices have low computing power and only perform basic data collection tasks.

[0079] Edge computing layer: The edge computing layer consists of multiple edge nodes, which are usually gateways or servers close to the data source. These nodes perform preliminary processing on the data from the device through distributed computing methods, which reduces the amount of data transmitted to the cloud and reduces system latency. The edge nodes are connected to each other through a local area network (LAN) or high-speed Internet, and maintain communication with the cloud computing center.

[0080] Cloud computing center layer: The cloud computing center provides global computing and storage functions, handling complex tasks that edge nodes cannot complete. This layer is also responsible for global scheduling, load balancing and data management to ensure efficient operation of the system.

[0081] By using multi-level scheduling, weight adjustment and distributed node state synchronization, an algorithm with a certain degree of complexity is formed. This algorithm can accurately measure and allocate system resources through dynamic real-time monitoring, partitioned balancing strategy and adaptive adjustment function.

[0082] In the specific implementation, in the hardware architecture of the edge node, the circuit design needs to support low power consumption and high concurrency processing capabilities. The core circuit design of the edge node includes the following functional modules:

[0083] Data receiving module: used to receive data sent by IoT devices, using wireless communication chips (such as LoRa or 5G chips) to achieve high-speed data transmission;

[0084] Processing unit module: contains multiple processor cores, responsible for preprocessing and analyzing received data. This module is connected to memory and storage devices through a high-speed data bus;

[0085] Communication module: Transmit data with the cloud computing center through high-speed Ethernet or wireless network to ensure fast communication between the cloud and edge nodes;

[0086] Storage module: used to store processed data or redundant backup data. The storage module can be a high-speed solid-state drive or RAM, depending on the data processing requirements.

[0087] The modules are interconnected via a standard data bus (such as PCIe or USB), ensuring high-speed and stable data transmission between system components.

[0088] like Figure 2 As shown, the method comprises the following steps:

[0089] S1. According to the preset division method, each node at the edge of the Internet of Things network is divided into multiple partitions.

[0090] In specific implementation, nodes that are physically or network-close can be grouped into one partition. In this way, grouping nodes that are physically or network-close into the same partition can reduce data transmission delays and costs. Because the communication links between these nodes are relatively short, data transmission speeds are faster and are not easily affected by network congestion and delays. In addition, grouping similar nodes into the same partition can simplify system management and maintenance. Administrators can more easily monitor, configure, and troubleshoot these nodes, thereby improving management efficiency.

[0091] And back up important data between different nodes; if a node fails, the tasks of the node will be automatically transferred to other nodes that back up its important data;

[0092] The requirement for the number of backups of important quantities is that the data reliability is greater than a preset reference value when the backup quantity is used; the calculation formula for data reliability is:

[0093] R=1-(1-p) k ;

[0094] In the formula, R represents data reliability; p is the probability that the node works normally; k is the number of redundant backups of data.

[0095] In this way, data reliability is improved. Increasing the k value, that is, increasing the number of redundant data backups, can significantly improve data reliability. According to the data reliability calculation formula R = 1-(1-p)k, when p (the probability of a node working properly) is fixed, the larger the k value, the closer R (data reliability) is to 1, that is, the lower the risk of data loss. This is crucial for protecting critical business data and ensuring high availability of the system. Storage overhead can also be optimized. Increasing the k value will also bring additional storage overhead. Each backup requires a certain amount of storage space, so the larger the k value, the larger the storage space required. By reasonably adjusting the k value, the system can avoid unnecessary waste of storage resources while ensuring data reliability. For example, when the data reliability requirement is not particularly high, the k value can be appropriately reduced to reduce storage overhead; in scenarios where the data reliability requirement is extremely high, the k value can be appropriately increased to ensure the absolute security of the data.

[0096] In addition, this strategy allows the system to flexibly adjust the k value according to actual needs, so as to achieve the best balance between reliability and storage overhead in different scenarios. This helps the system maintain efficient and stable operation under different loads, different data importance and different resource constraints. In addition, by backing up important data between different nodes, the system has stronger fault tolerance. Even if a node fails, the task can be quickly transferred to other nodes that have backed up the data, thereby ensuring the continuity and stability of the system. This fault tolerance is of great significance to improving the overall performance of the system and user experience. By adjusting the k value, the system can achieve a significant balance between data reliability and storage overhead. This strategy not only improves the reliability of data, but also optimizes the utilization of storage resources, enhances the fault tolerance of the system, and provides strong support for the stable operation of the Internet of Things system.

[0097] S2. After obtaining the collected data of the corresponding device layer through the node, determine whether the data processing task belongs to the preset complex task; if it is a complex task, send the task to the cloud for processing; if it is not a complex task, go to S3.

[0098] In specific implementation, edge nodes can also pre-process the received data, such as data denoising and compression, to reduce the amount of transmitted data and processing time.

[0099] In specific implementation, when the computational workload of a data processing task is greater than a preset threshold, it is judged as a complex task. By setting the computational workload threshold, the system can accurately identify which tasks are complex and which tasks are simple. This helps the system to process and optimize different types of tasks in a targeted manner.

[0100] S3. Determine whether the node can currently process the task based on the node load; if so, the node processes the task and sends the processing result to the cloud; if not, go to S4.

[0101] In specific implementation, node Ei needs to meet the following conditions:

[0102]

[0103] Ri represents the total amount of resources (such as CPU, memory, etc.) of edge node i. ij represents the resources required by task Tj at node i; n is the number of tasks at node i. This inequality ensures that task allocation does not overload the resources of the node.

[0104] S4. Determine whether there are other nodes in the partition where the node is located that can currently process the task. If so, filter them out and select other nodes with the optimal transmission path as the target node, and go to S5. If not, filter out other nodes that can currently process the task from other partitions, select other nodes with the optimal path as the target node, and go to S5.

[0105] Among them, the optimal transmission path is the data transmission path with the lowest transmission delay. In this way, by selecting the data transmission path with the lowest transmission delay, the delay time of data during transmission can be significantly reduced. This is especially important for application scenarios that require real-time processing or low-latency response, such as intelligent transportation systems and telemedicine. Selecting the optimal transmission path can avoid resource waste caused by poor paths during data transmission. For example, it can reduce the number of transmission failures or retransmissions caused by network congestion and reduce unnecessary consumption of network resources.

[0106] In specific implementation, when calculating the delay of the transmission path, the path P from node i to node j ij The delay T ij for:

[0107]

[0108] Where, L ij For path P ij The physical distance or basic delay between nodes; w is the bandwidth requirement coefficient of the task data processing type;

[0109] Ui(1-ui) and Dj(1-dj) are the remaining effective bandwidths of nodes i and j respectively; Ui represents the uplink bandwidth of node i, ui represents the proportion of uplink bandwidth currently occupied by node i, and u i = Uplink data volume / U i ; Dj represents the downlink bandwidth of node j, dj represents the proportion of downlink bandwidth of node j currently occupied, d j = Downlink data volume / D i .

[0110] Among them, if the task is data-intensive, its bandwidth demand coefficient w is 1.5; if the task is computation-intensive, its bandwidth demand coefficient w is 0.5; if the task is balanced, its bandwidth demand coefficient w is 1. In this way, by assigning different bandwidth demand coefficients w to different types of tasks, the actual bandwidth demand of the task can be more accurately reflected. Data-intensive tasks require higher bandwidth to transmit large amounts of data, so they are assigned a higher coefficient (1.5); computation-intensive tasks focus more on computing performance and have relatively low bandwidth requirements, so they are assigned a lower coefficient (0.5); balanced tasks are between the two and are assigned a medium coefficient (1). This design helps to more accurately consider task requirements when selecting a transmission path, thereby optimizing resource allocation. Adjusting the bandwidth demand coefficient w according to the task type helps to achieve more reasonable resource utilization in a resource-constrained IoT environment. For data-intensive tasks, selecting a transmission path with high bandwidth can ensure fast and complete data transmission; for computation-intensive tasks, more consideration can be given to the computing power and remaining bandwidth of the node to avoid unnecessary bandwidth waste. This strategy helps to maximize resource utilization efficiency while meeting task requirements.

[0111] With this setting, by introducing the physical distance (or basic delay) Lij, the bandwidth requirement coefficient w, and the remaining effective bandwidth of the node Ui (1-ui) and Dj (1-dj), the formula can more accurately calculate the delay Tij of the path Pij. This helps to more accurately evaluate the transmission performance of different paths when selecting the optimal transmission path, so as to make more reasonable decisions. The bandwidth requirement coefficient w in the formula reflects the bandwidth requirement of the task data processing type, while the remaining effective bandwidth of the node Ui (1-ui) and Dj (1-dj) take into account the current load of the node. This design enables the bandwidth requirement of the task and the processing capacity of the node to be considered simultaneously when selecting the transmission path, avoiding the selection of overloaded or insufficiently bandwidthd nodes, thereby improving the reliability and efficiency of data transmission. In addition, since the uplink and downlink bandwidth usage of the node changes over time, the formula can reflect the load status of the node in real time. This helps to adjust the transmission path according to the real-time load of the node in a dynamically changing IoT environment to ensure the continuity and stability of data transmission. In addition, by comprehensively considering the bandwidth resources and load conditions of the nodes, this formula helps to select nodes with sufficient remaining bandwidth and light load when selecting the transmission path, thereby optimizing resource utilization and improving the transmission performance of the entire IoT network. In addition, the design of this formula takes into account the dynamic changes in node bandwidth, allowing the system to adapt to IoT networks of different sizes and complexities. As the network scale expands and the number of nodes increases, the formula can still accurately calculate the path delay, providing strong support for selecting the optimal transmission path.

[0112] It should be noted that during the operation of the system, the bandwidth occupancy and load of the nodes will change dynamically, so it is necessary to regularly update the bandwidth occupancy of each node, the task data processing type and other parameters. For this purpose, an update cycle can be set to recalculate the path delay in each cycle to ensure the real-time performance of the transmission path.

[0113] In specific implementation, when filtering out other nodes that can currently process the task from other partitions, if there are multiple other partitions that have other nodes that meet the conditions, first select the other partition with the smallest global average deviation, and then select other nodes from the selected other partitions. In this way, by selecting the partition with the smallest global average deviation for task transfer, the system can ensure that the task load between the partitions remains relatively balanced. This helps to avoid performance degradation due to task overload in some partitions, and prevents other partitions from wasting resources due to insufficient tasks. Balanced task load can improve the resource utilization and response speed of the entire system. In addition, balanced task load distribution helps reduce the uncertainty and volatility of the system. When a partition is overloaded with tasks, it may cause performance bottlenecks, data loss, or transmission delays. By optimizing load balancing, the system can better deal with these potential risks, thereby improving overall stability and reliability.

[0114] The calculation formula of the global average deviation is:

[0115]

[0116] In the formula, ΔK k represents the global average deviation of partition k, represents the average load of each node in partition k; L total represents the global load of each edge node; N represents the number of edge nodes;

[0117] Among them, the global load L total The calculation formula is:

[0118]

[0119] Where Li represents the load of node i;

[0120] L i =f(Ci,Mi,Bi,Di);

[0121] In the formula, Ci represents CPU usage data; Mi represents memory usage data; Bi represents network bandwidth occupancy data; Di represents node geographic location or physical distance data; f() represents a comprehensive function that integrates various factors.

[0122] In this way, the global average deviation ΔK kThe calculation of can accurately quantify the degree of deviation of each partition relative to the global average load. This helps system administrators or algorithms to more accurately judge the load status of each partition when allocating and scheduling tasks, so as to make more reasonable decisions. In addition, the calculation of node load Li takes into account multiple factors such as CPU usage data, memory usage data, network bandwidth usage data, and node geographic location or physical distance data. This comprehensive evaluation method can more comprehensively reflect the actual load status of the node and avoid the one-sidedness of single indicator evaluation.

[0123] S5. Send the task to the target node for processing, and send the processing result to the cloud.

[0124] This method can efficiently utilize edge computing resources, and optimize resource allocation through partition management. Through a preset partitioning method, each node at the edge of the IoT network is divided into multiple partitions, making resource management and task allocation more flexible and efficient. This partitioning strategy helps to reasonably allocate nodes according to factors such as their geographical location and computing power, thereby maximizing the use of computing resources on edge devices. In addition, this method can improve data processing efficiency. Task hierarchical processing: tasks are divided into complex tasks and non-complex tasks according to the complexity of data processing tasks. Complex tasks are sent directly to the cloud for processing to ensure the accuracy and security of processing results; non-complex tasks are processed by edge nodes, which reduces data transmission delays and bandwidth consumption and improves processing efficiency. Load balancing mechanism: when a node cannot process a task, it determines whether there are other nodes in the partition where the node is located or in other partitions that can process the task, and selects the node with the optimal transmission path as the target node, thereby achieving dynamic task allocation and load balancing. This helps to avoid overloading of a single node and improve the stability and data processing efficiency of the entire system. In addition, this method can ensure the accuracy and security of data processing. Complex tasks are processed in the cloud: for complex tasks, they are sent to the cloud for processing, and the powerful computing power of the cloud is used to ensure the accuracy of processing results. At the same time, the cloud also has higher security protection capabilities, which helps to avoid risks such as data leakage. Edge nodes handle non-complex tasks; for non-complex tasks, edge nodes directly process and send processing results to the cloud. This processing method reduces the links of data transmission, reduces the risk of data loss, and also helps to improve the timeliness of data processing.

[0125] This method can efficiently utilize edge computing resources while adapting to the characteristics of large-scale IoT networks to achieve efficient and low-latency data processing and analysis. Through partition management to optimize resource allocation, task hierarchical processing, load balancing mechanism, and the collaborative work of cloud and edge nodes, this method not only improves data processing efficiency, but also ensures the accuracy and security of data processing.

[0126] Embodiment 2

[0127] Different from the first embodiment, in S4 of this embodiment, when selecting the optimal transmission path, if an equivalent optimal path exists, other nodes with the highest node weights are selected as target nodes from the equivalent optimal path and other nodes of the transmission path with the lowest delay; the equivalent optimal path is a transmission path whose delay difference with the transmission path with the lowest delay is less than a preset difference.

[0128] By introducing the concept of equivalent optimal path, the selection of the optimal transmission path is no longer limited to a single path with the lowest delay, but can be selected from multiple equivalent paths. This increases the flexibility of selection and helps to find a more suitable transmission path in specific situations (such as network congestion, node failure, etc.). When selecting the target node, in addition to considering the delay of the transmission path, the weight of the node is also combined. The node weight may be based on multiple factors such as the computing power, stability, and historical performance of the node. By selecting the node with the highest weight, the efficiency of data processing and transmission can be further optimized, and the performance of the entire system can be improved. In addition, in the case of an equivalent optimal path, selecting the node with the highest weight as the target node helps to quickly switch to nodes on other equivalent paths when a node or path fails, thereby maintaining the continuity and stability of data transmission. This enhances the robustness and fault tolerance of the system. In addition, although delay is an important consideration in selecting a transmission path, in some cases, a certain delay may need to be sacrificed in order to optimize resource utilization or improve processing efficiency. By introducing the concepts of equivalent optimal path and node weight, resource utilization and processing efficiency can be better balanced while maintaining low latency.

[0129] In specific implementation, the calculation formula of the weight Wi of node i is:

[0130] W i =αC i +βM i +γB i ;

[0131] Where α, β, and γ are adjustment parameters, and Ci, Mi, and Bi represent the computing power, storage capacity, and transmission rate of node i, respectively.

[0132] In this way, by comprehensively considering the computing power, storage capacity and transmission rate of the node, the calculation formula can comprehensively evaluate the comprehensive performance of the node. This helps to more accurately judge the processing power and transmission efficiency of the node when selecting the optimal transmission path and target node, so as to make more reasonable decisions. Considering the computing power, storage capacity and transmission rate of the node at the same time helps to achieve more reasonable resource allocation in a resource-constrained IoT environment. By selecting nodes with higher weights as target nodes, their computing resources and storage capacity can be fully utilized to improve the efficiency of data processing and transmission. In addition, the adjustment parameters α, β, and γ can be flexibly adjusted according to actual application scenarios and requirements. For example, in scenarios where high-speed data transmission is required, the weight coefficient γ of the transmission rate B_i can be increased; in scenarios where a large amount of data needs to be processed, the weight coefficient α of the computing power C_i can be increased. This flexibility makes the calculation formula applicable to different IoT application scenarios. In addition, the calculation of node weights takes into account multiple factors, so that the performance and stability of the node can be more comprehensively evaluated when selecting the target node. This helps to quickly switch to other nodes with better performance when a node or path fails, thereby maintaining the continuity and stability of data transmission and improving the reliability and stability of the entire system.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A distributed processing method for data analysis in a large-scale Internet of Things network, characterized in that: The following steps are involved: S1. Divide each node at the edge of the IoT network into multiple partitions according to a preset partitioning method; S2, after obtaining the collected data of the corresponding device layer through the node, determine whether the data processing task belongs to the preset complex task; if it is a complex task, send the task to the cloud for processing; if it is not a complex task, go to S3; S3: Determine whether the node can currently process the task based on the node load; if so, the node processes the task and sends the processing result to the cloud; if not, go to S4; S4, determine whether there are other nodes in the partition where the node is located that can currently process the task, if so, filter them out, and select other nodes with the best transmission path as the target node, and go to S5; If not, select other nodes that can currently process the task from other partitions, select other nodes on the optimal path as the target node, and go to S5; S5. Send the task to the target node for processing, and send the processing result to the cloud.

2. The data analysis distributed processing method for large-scale Internet of Things networks according to claim 1, characterized in that: In S4, the optimal transmission path is a data transmission path with the lowest transmission delay.

3. The data analysis distributed processing method for large-scale Internet of Things networks as claimed in claim 2, characterized in that: In S4, when selecting the optimal transmission path, if an equivalent optimal path exists, then other nodes with the highest node weight are selected as target nodes from the equivalent optimal path and other nodes of the transmission path with the lowest delay; the equivalent optimal path is a transmission path whose delay difference with the transmission path with the lowest delay is less than a preset difference.

4. The data analysis distributed processing method for large-scale Internet of Things networks as claimed in claim 3, characterized in that: The calculation formula of the weight Wi of node i is: W i =αC i +βM i +γB i ; Where α, β, and γ are adjustment parameters, and Ci, Mi, and Bi represent the computing power, storage capacity, and transmission rate of node i, respectively.

5. The data analysis distributed processing method for large-scale Internet of Things networks as claimed in claim 2, characterized in that: In S4, when calculating the delay of the transmission path, the path P from node i to node j ij The delay T ij for: Where, L ij For path P ij The physical distance or basic delay between nodes; w is the bandwidth requirement coefficient of the task data processing type; Ui(1-ui) and Dj(1-dj) are the remaining effective bandwidths of nodes i and j respectively; Ui represents the uplink bandwidth of node i, ui represents the proportion of uplink bandwidth currently occupied by node i, and u i = Uplink data volume / U i ; Dj represents the downlink bandwidth of node j, dj represents the proportion of downlink bandwidth of node j currently occupied, d j = Downlink data volume / D i .

6. The data analysis distributed processing method for large-scale Internet of Things networks as claimed in claim 5, characterized in that: If the task is data-intensive, its bandwidth requirement coefficient w is 1.5; if the task is compute-intensive, its bandwidth requirement coefficient w is 0.5; if the task is balanced, its bandwidth requirement coefficient w is 1.

7. The data analysis distributed processing method for large-scale Internet of Things networks according to claim 1, characterized in that: In S4, when filtering out other nodes that can currently process the task from other partitions, if there are multiple other partitions that have other nodes that meet the conditions, first select the other partition with the smallest global average deviation, and then select other nodes from the selected other partitions.

8. The data analysis distributed processing method for a large-scale Internet of Things network as claimed in claim 7, characterized in that: The global average deviation is calculated as: In the formula, ΔK k represents the global average deviation of partition k, represents the average load of each node in partition k; L total represents the global load of each edge node; N represents the number of edge nodes; Among them, the global load L total The calculation formula is: Where Li represents the load of node i; L i =f(Ci,Mi,Bi,Di); In the formula, Ci represents CPU usage data; Mi represents memory usage data; Bi represents network bandwidth occupancy data; Di represents node geographic location or physical distance data; f() represents a comprehensive function that integrates various factors.

9. The data analysis distributed processing method for large-scale Internet of Things networks according to claim 1, characterized in that: In S1, important data is also backed up between different nodes; If a node fails, the tasks of that node are automatically transferred to other nodes that back up its important data; The requirement for the number of backups of important quantities is that the data reliability is greater than a preset reference value when the backup quantity is used; the calculation formula for data reliability is: R=1-(1-p) k ; In the formula, R represents data reliability; p is the probability that the node works normally; k is the number of redundant backups of data.

10. The data analysis distributed processing method for large-scale Internet of Things networks according to claim 1, characterized in that: In S1, the preset partitioning method is to divide nodes that are physically or network-close into one partition; In S2, when the computational complexity of the data processing task is greater than a preset threshold, it is determined to be a complex task.

Images