Internet of Things data processing method based on distributed computing power scheduling

Abstract

The invention relates to the field of Internet of Things data processing, in particular to an Internet of Things data processing method based on distributed computing power scheduling, which comprises the steps of selecting Internet of Things node communication connection, dividing computing task priorities to generate a task sequence, executing tasks item by item to obtain individual and overall computing results, and analyzing load balancing to obtain a resource occupancy rate. And optimizing a task sequence and differentially setting a calculation path, and finally completing a data processing operation. Through the technical means of the hierarchical topological structure, genetic algorithm optimization, a path redundancy mechanism and the like, the distributed computing power scheduling efficiency and the system fault-tolerant capability are improved, and the requirement of a complex Internet of Things scene for efficient data processing is met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of Internet of Things and distributed computing, specifically a method for processing data of Internet of Things based on distributed computing power scheduling. BACKGROUND

[0002] The rapid development of Internet of Things technology has promoted the application of distributed computing power scheduling in the field of data processing. By reasonably allocating computing resources, the response speed, data processing efficiency and overall performance of the Internet of Things system can be significantly improved. However, the existing Internet of Things system still has certain limitations in computing power scheduling and data processing, especially in complex scenarios, efficient processing requirements pose higher requirements on the system.

[0003] After searching, a patent document with publication number CN112543223B was found, which was published on May 10, 2024. The system is deployed in a public network address, including an Internet of Things device access unit, a PaaS platform access unit and a customer access processing unit. The data uploaded by the Internet of Things device is preprocessed and sent to the PaaS platform for further processing, which improves the security of the system and reduces the operating cost. However, this technical solution mainly relies on a centralized PaaS platform for data processing, and the distributed computing power is not fully utilized, which may lead to high system load in high concurrency scenarios, affecting the real-time performance and stability of data processing. In addition, this scheme does not involve dynamic computing power scheduling strategies for different devices or tasks, which may be difficult to adapt to the needs of diversified Internet of Things application scenarios.

[0004] Another patent document with publication number CN111416870B was found, which was published on March 24, 2023, which proposes a distributed architecture based on multiple personalized cloud platforms. The system acquires data through a communication module, and the received data is matched and operated by an expert system, and finally stored as an event database. Through the distributed cloud platform cluster, this scheme reduces the size of the single cloud database, thereby reducing the downtime risk and operation and maintenance cost caused by data problems. However, the computing power scheduling of this technical solution mainly focuses on the data storage and event management level, and does not fully consider the dynamic allocation of computing resources in the data processing process. In actual application, especially in multi-task parallel processing, there may be an uneven distribution of computing power, and the heavy load of some nodes may affect the overall system efficiency.

[0005] The above problems show that the existing Internet of Things data processing system still has room for improvement in terms of distributed computing power scheduling, dynamic resource allocation, and processing capacity in high-concurrency scenarios. Therefore, the present application provides an Internet of Things data processing method based on distributed computing power scheduling, aiming to optimize the utilization of computing resources through intelligent computing power scheduling strategies, improve the real-time performance of data processing and system stability, and better meet the needs of complex Internet of Things scenarios for efficient data processing. SUMMARY

[0006] The present application relates to the field of Internet of Things data processing, and particularly relates to an Internet of Things data processing method based on distributed computing power scheduling. The method comprises:

[0007] S101, selecting a plurality of Internet of Things nodes for communication connection, and constructing a hierarchical topology structure through short-distance communication links and long-distance communication links;

[0008] S102, performing priority division on the computing tasks to obtain a task sequence;

[0009]

[0009] S103, executing the computing tasks on each Internet of Things node item by item according to the task sequence to obtain individual computing results of each Internet of Things node;

[0010] S104, executing the computing tasks on all Internet of Things nodes item by item according to the task sequence to obtain an overall computing result;

[0011] S105, performing load balancing analysis on each computing task according to the overall computing result and the individual computing result of each Internet of Things node to obtain the resource occupation rate of each computing task;

[0012] S106, adjusting the sorting of the computing tasks in the task sequence according to the resource occupation rate of each computing task to generate a plurality of optimized task sequences;

[0013] S107, differentially setting the computing paths according to the optimized task sequences;

[0014] S108, performing data processing operations on the Internet of Things nodes using the computing paths.

[0015] The present application achieves the above-mentioned purposes through the following specific technical means:

[0016] First, select several Internet of Things nodes for communication connection. These nodes can be edge computing devices, gateway devices or cloud servers distributed in different geographical locations. Each node is connected through wireless communication protocols such as 5G, Wi-Fi6 or wired communication protocols such as Ethernet. To reduce communication delay, the connection between nodes adopts a hierarchical topology structure, that is, the nodes close to the data source prefer to establish short-distance communication links with adjacent nodes, while the nodes far from the data source establish long-distance communication links with the center node through the backbone network. This hierarchical topology structure can effectively reduce the congestion risk of communication links and improve the stability of data transmission.

[0017] Second, the priority of the computing task is divided to obtain the task sequence. Computing tasks include data collection, data preprocessing, data analysis and data storage, etc. Priority division is based on the real-time requirement, computational complexity and data size of the task. For example, tasks with high real-time requirements have higher priority, while tasks with lower computational complexity also have higher priority. In this way, the task sequence can reflect the execution order of different types of tasks, thereby providing a basis for subsequent distributed computing power scheduling.

[0018] Next, according to the task sequence, each Internet of Things node is executed in turn to obtain the individual computing result of each Internet of Things node. In this process, each node independently completes the computing task assigned to it according to its hardware performance (such as CPU frequency, memory capacity, storage speed) and current load. To avoid task failure due to overload of a single node, the system monitors the resource usage of each node in real time and automatically suspends the allocation of new tasks when the resource approaches the critical value. In addition, the communication between nodes adopts an asynchronous message queue mechanism to ensure efficient data transmission during task execution, while avoiding task blocking caused by communication delay.

[0019] Then, according to the task sequence, all Internet of Things nodes are executed in turn to obtain the overall computing result. In this stage, the system coordinates the computing process of multiple nodes through a distributed collaborative algorithm. Based on the principle of consistent hashing, the algorithm distributes computing tasks evenly to each node, thereby avoiding the situation where some nodes are overloaded while others are idle. At the same time, the system introduces a redundant computing mechanism, that is, some key tasks will be executed repeatedly on multiple nodes to improve the reliability of the computing result. Finally, by aggregating and verifying the computing results of all nodes, the system generates the overall computing result.

[0020] Subsequently, load balancing analysis is performed on each computing task based on the overall calculation results and individual calculation results of each Internet of Things node, and the resource occupancy rate of each computing task is obtained. The core of load balancing analysis lies in comparing the difference between the actual computing capacity and the theoretical computing capacity of each node. For example, if the theoretical computing capacity of a node is 100%, but the actual computing capacity is only 70%, it indicates that the node has a resource bottleneck. By counting the resource occupancy rate of each node, the system identifies computing tasks with uneven resource allocation and marks them as tasks that need to be optimized.

[0021] Next, the ordering of computing tasks in the task sequence is adjusted based on the resource occupancy rate of each computing task, generating several optimized task sequences. The optimization process is based on genetic algorithms, that is, continuously iterating the task sequence by simulating natural selection until the optimal solution is found. Specifically, the system first randomly generates multiple groups of task sequences as the initial population, and then calculates the fitness value based on the resource occupancy rate of each sequence. The higher the fitness value, the more reasonable the resource allocation of the sequence. After multiple rounds of crossover and mutation operations, the system finally generates several optimized task sequences, which can maximize the reduction of resource waste while ensuring computing efficiency.

[0022] After that, the computing path is set differently according to the optimized task sequence. The computing path refers to the transmission path of data from the source node to the target node. The core of differential setting lies in dynamically adjusting the priority of the path according to the needs of different tasks. For example, for tasks with high real-time requirements, the system will choose the shortest path to reduce transmission delay; while for tasks with large data volume, the system will choose a path with higher bandwidth to improve transmission efficiency. In addition, the system also introduces a path redundancy mechanism, that is, setting multiple backup paths for critical tasks to cope with the failure of the main path.

[0023] Finally, the Internet of Things nodes are processed using the computing path. In this stage, the system controls the task execution process of each node through a dynamic scheduling module. Based on the optimized task sequence and the differential setting of the computing path, the module allocates computing tasks in real time and monitors the task execution status. If a node appears abnormal (such as hardware failure or network interruption), the system will immediately redistribute the tasks on that node to other available nodes, thereby ensuring the continuity of the data processing process.

[0024] The present application solves the problem of insufficient efficiency in distributed computing power scheduling and dynamic resource allocation in existing Internet of Things systems through the above technical means. Specifically, the hierarchical topology and asynchronous message queue mechanism improve communication efficiency, the genetic algorithm optimizes resource allocation of the task sequence, and the path redundancy mechanism enhances the fault tolerance of the system. These technical means work together to enable the Internet of Things system to maintain high computing power and stable operation in high concurrency scenarios, thereby meeting the demand for efficient data processing in complex Internet of Things scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A schematic diagram of the hierarchical topology of the communication connection of the Internet of Things nodes in the embodiment of the present application;

[0026] Figure 2 A flowchart of the task sequence optimization process in the embodiment of the present application;

[0027] Figure 3 A schematic diagram of the differential computing path setting in the embodiment of the present application.

[0028] The reference signs are as follows: 1, short-distance communication link; 2, long-distance communication link; 3, Internet of Things node; 4, initial population; 5, fitness value calculation module; 6, crossover operation module; 7, mutation operation module; 8, shortest path; 9, high-bandwidth path; 10, backup path. DETAILED DESCRIPTION

[0029] The present application provides an Internet of Things data processing method based on distributed computing power scheduling, and the specific implementation is as follows. The drawings shown in Figure 1 , Figure 2 and Figure 3 are described in detail in combination with the reference signs thereof.

[0030] First, in the establishment process of the communication connection of the Internet of Things nodes, a number of Internet of Things nodes are selected and a hierarchical topology is constructed. As shown in Figure 1 , the nodes close to the data source establish connections with adjacent nodes through short-distance communication links 1, and the nodes far from the data source establish connections with Internet of Things nodes 3 through long-distance communication links 2. The short-distance communication link 1 usually adopts a low-power wireless communication protocol (such as Wi-Fi6 or Zigbee), which is suitable for communication between nodes in a small range; the long-distance communication link 2 adopts a high-bandwidth wired or wireless communication protocol (such as 5G or optical fiber) to ensure the stability of long-distance data transmission. The design of this hierarchical topology can effectively reduce the congestion risk of the communication link. In specific implementation, the system will dynamically adjust the configuration parameters of the short-distance communication link 1 and the long-distance communication link 2, such as signal strength, transmission rate, and retransmission mechanism, according to the geographical location and communication requirements of the nodes, so as to realize efficient data transmission.

[0031] Secondly, the computing tasks are prioritized to generate a task sequence. The computing tasks include data collection, data preprocessing, data analysis, and data storage, etc. The priority division is based on a comprehensive evaluation of the real-time requirement, computational complexity, and data size of the task. For example, tasks with high real-time requirements have higher priority, while tasks with lower computational complexity also have higher priority. In actual operation, the system will assign each task a priority weight value, which is calculated by weighting the quantitative indicators of the above three dimensions. The higher the weight value, the higher the task in the task sequence. The specific implementation of this process is completed by the task management module, which generates an initial task sequence according to the priority weight value of the task and distributes the task sequence to each Internet of Things node.

[0032] Next, each Internet of Things node is executed in turn according to the task sequence to obtain the individual computing result of each Internet of Things node. In this process, each node independently completes the computing task assigned to it according to its hardware performance (such as CPU frequency, memory capacity, storage speed) and current load. To avoid task failure due to overload of a single node, the system will monitor the resource usage of each node in real time, including CPU occupancy, memory occupancy, and network bandwidth occupancy, etc. When the resource usage of a certain node approaches the critical value, the system will automatically suspend the allocation of new tasks and reassign the unfinished tasks to other available nodes. The communication between nodes uses an asynchronous message queue mechanism to ensure efficient data transmission during task execution, while avoiding task blocking due to communication delay. In specific implementation, the system will deploy a resource monitoring module on each node, which is responsible for real-time collection of node resource usage data and uploading these data to the Internet of Things node 3 for analysis and processing.

[0033] Then, all Internet of Things nodes are executed in turn according to the task sequence to obtain the overall computing result. In this stage, the system coordinates the computing process of multiple nodes through a distributed collaborative algorithm. Based on the principle of consistent hashing, the algorithm distributes computing tasks evenly to each node, thereby avoiding the situation where some nodes are overloaded while others are idle. For example, for a computing task that needs to process a large amount of data, the system will split it into multiple subtasks and distribute these subtasks to different nodes for parallel execution. At the same time, the system introduces a redundant computing mechanism, i.e. some key tasks will be executed repeatedly on multiple nodes to improve the reliability of the computing result. Finally, by aggregating and verifying the computing results of all nodes, the system generates the overall computing result. In specific implementation, the system will deploy a collaborative computing module on each node, which is responsible for receiving task allocation instructions from the Internet of Things node 3 and collaborating with other nodes to complete the computing task.

[0034] Subsequently, load balancing analysis is performed on each computing task based on the overall calculation results and individual calculation results of each Internet of Things node, and the resource occupancy rate of each computing task is obtained. The core of load balancing analysis lies in comparing the difference between the actual computing capacity and the theoretical computing capacity of each node. For example, if the theoretical computing capacity of a node is 100%, but the actual computing capacity is only 70%, it indicates that the node has a resource bottleneck. By counting the resource occupancy rate of each node, the system identifies computing tasks with uneven resource allocation and marks them as tasks that need to be optimized. In specific implementation, the system will deploy a load balancing analysis module on the Internet of Things node 3, which is responsible for collecting resource usage data of each node and calculating the resource occupancy rate of each computing task based on these data.

[0035] Next, the ordering of computing tasks in the task sequence is adjusted based on the resource occupancy rate of each computing task, generating several optimized task sequences. The optimization process is based on genetic algorithm, that is, continuously iterating the task sequence by simulating natural selection until the optimal solution is found. As shown in Figure 2 , the system first randomly generates multiple groups of task sequences as the initial population 4, and then calculates the fitness value based on the resource occupancy rate of each sequence. The higher the fitness value, the more reasonable the resource allocation of the sequence. After multiple rounds of operation of the crossover operation module 6 and the mutation operation module 7, the system finally generates several optimized task sequences. In specific implementation, the system will deploy a genetic algorithm optimization module on the Internet of Things node 3, which is responsible for the generation of the initial population 4, the calculation of the fitness value calculation module 5, and the operation of the crossover operation module 6 and the mutation operation module 7.

[0036] After that, the computing path is set differently based on the optimized task sequence. As shown in Figure 3 , the system dynamically selects the shortest path 8 or high-bandwidth path 9 based on task requirements, and sets multiple backup paths 10 for critical tasks. For example, for tasks with high real-time requirements, the system will select the shortest path 8 to reduce transmission delay; while for tasks with large data volume, the system will select the high-bandwidth path 9 to improve transmission efficiency. In addition, the system also introduces a path redundancy mechanism, that is, multiple backup paths 10 are set for critical tasks to cope with the failure of the main path. In specific implementation, the system will deploy a path management module on each node, which is responsible for dynamically adjusting the configuration parameters of the computing path based on task requirements, and monitoring the status of the path in real time.

[0037] Finally, the computing path is used to perform data processing operations on the IoT nodes. In this stage, the system controls the task execution process of each node through the dynamic scheduling module. According to the optimized task sequence and the differentiated computing path, the module allocates computing tasks in real time and monitors the task execution status. If an exception occurs on a node (such as hardware failure or network interruption), the system will immediately reassign the tasks on that node to other available nodes, thereby ensuring the continuity of the data processing process. In specific implementation, the system will deploy a dynamic scheduling module on each node, which is responsible for receiving task allocation instructions from the IoT node 3 and performing data processing operations according to the optimized task sequence and the differentiated computing path.

[0038] The above implementation improves communication efficiency through hierarchical topology and asynchronous message queue mechanism, optimizes resource allocation of task sequence through genetic algorithm, and enhances fault tolerance of the system through path redundancy mechanism. These technical means work together to enable the IoT system to maintain high computing capacity and stable running state in high-concurrency scenarios, thereby meeting the demand for efficient data processing in complex IoT scenarios.

[0039] In order to better enable relevant persons in the technical field to fully understand and implement the present application, the specific implementation principles of the present application are further supplemented below in conjunction with a specific application scenario.

[0040] In the traffic management system of smart city, IoT nodes are widely deployed on devices such as road monitoring cameras, traffic signal controller, vehicle detection sensors, etc. These nodes establish connections with neighboring nodes through short-range communication link 1 (such as Wi-Fi 6), and nodes close to the data source prefer to complete low-latency data transmission tasks; while nodes far from the data source establish connections with IoT node 3 through long-range communication link 2 (such as 5G or optical fiber) to ensure the stability of long-distance data transmission. For example, when a monitoring camera on a certain road section captures abnormal traffic flow, its data will be quickly transmitted to the neighboring data processing node through the short-range communication link 1 for preliminary analysis, and then the processing result will be uploaded to the IoT node 3 through the long-range communication link 2 for global optimization scheduling.

[0041] In the task priority classification stage, the system evaluates tasks based on real-time requirements, computational complexity, and data size. For example, for a traffic signal control task with high real-time requirements, the system assigns it a high priority weight value and places it at the front of the task sequence. For a historical data analysis task with large data size, the system assigns it a low priority weight value and places it at the back of the task sequence. This priority classification ensures that critical tasks are executed first. Specifically, the system generates an initial task sequence through the task management module and distributes the task sequence to each IoT node. Each node independently completes the assigned tasks based on its hardware performance and current load, while the resource monitoring module collects data such as CPU and memory usage and uploads them to the IoT node 3 for analysis.

[0042] In the collaborative computing stage, the system uses a distributed collaborative algorithm to coordinate the computing processes of multiple nodes. For example, when predicting traffic flow in a certain area, the system splits the task into multiple subtasks and distributes them evenly to each node using a consistent hashing algorithm. For critical tasks such as traffic accident warning, the system introduces a redundant computing mechanism, i.e., the same task is executed repeatedly on multiple nodes to improve the reliability of the computing result. Finally, the overall computing result is generated by aggregating and verifying the results of all nodes. In this process, the collaborative computing module is responsible for receiving task allocation instructions from the IoT node 3 and collaborating with other nodes to complete the computing task.

[0043] In the load balancing analysis stage, the system identifies computing tasks with uneven resource allocation by comparing the actual computing capacity of each node with its theoretical computing capacity. For example, if a node's theoretical computing capacity is 100% but its actual computing capacity is only 70%, it indicates that the node has a resource bottleneck. At this time, the load balancing analysis module marks the related tasks on this node as tasks that need to be optimized and reassigns them to other available nodes. This dynamic adjustment mechanism ensures the overall efficiency of the system.

[0044] In the task sequence optimization stage, the system iteratively optimizes the task sequence based on genetic algorithms. For example, the system first randomly generates multiple sets of task sequences as initial populations and calculates the fitness values based on the resource occupancy of each sequence. After the operations of the crossover operation module 6 and the mutation operation module 7, the system finally generates several optimized task sequences. These optimized task sequences can maximize resource waste reduction while ensuring computing efficiency.

[0045] In the differentiated path setting stage, the system dynamically selects the shortest path 8 or high-bandwidth path 9 according to task requirements, and sets multiple backup paths 10 for critical tasks. For example, for traffic signal control tasks with high real-time requirements, the system selects the shortest path 8 to reduce transmission delay; for historical data analysis tasks with large data volume, the system selects the high-bandwidth path 9 to improve transmission efficiency. In addition, the system introduces a path redundancy mechanism, i.e., sets multiple backup paths 10 for critical tasks to cope with the failure of the main path. This differentiated path setting method significantly improves the fault tolerance and data transmission efficiency of the system.

[0046] Finally, in the dynamic scheduling stage, the system controls the task execution process of each node through the dynamic scheduling module. For example, when a monitoring camera on a certain road segment fails due to hardware failure, the system will immediately redistribute the tasks on this node to other available nodes, ensuring the continuity of data processing. The dynamic scheduling module allocates computing tasks in real time and monitors the task execution status according to the optimized task sequence and differentiated computing path, ensuring the efficient operation of the entire system.

[0047] The above embodiments improve communication efficiency through hierarchical topology and asynchronous message queue mechanism, optimize resource allocation of task sequence through genetic algorithm, and enhance fault tolerance of the system through path redundancy mechanism. In the specific application scenario of intelligent city traffic management system, these technical means work together to enable the Internet of Things system to maintain high computing capacity and stable operation state in high-concurrency scenarios, thereby meeting the demand for efficient data processing in complex Internet of Things scenarios.

Claims

1. A method for processing Internet of Things data based on distributed computing power scheduling, characterized in that: The method comprises: S101, selecting a plurality of IoT nodes for communication connection, wherein the communication connection constructs a hierarchical topology structure through a short-distance communication link (1) and a long-distance communication link (2); S102, prioritize the computing tasks to obtain a task sequence; S103, executing the computing tasks for each IoT node (3) one by one according to the task sequence, and obtaining the individual computing results of each IoT node; S104, executing the computing tasks one by one on all IoT nodes (3) according to the task sequence, and obtaining the overall computing result; S105, performing load balancing analysis on each computing task based on the overall computing result and the individual computing result of each IoT node (3), and obtaining the resource occupancy rate of each computing task; S106, adjusting the order of the computing tasks in the task sequence according to the resource occupancy rate of each computing task to generate a plurality of optimized task sequences; S107, performing differentiated setting of the calculation path according to the optimized task sequence; S108, performing data processing operations on the Internet of Things node (3) using the calculation path.

2. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The priority division of computing tasks to obtain a task sequence includes: S201, performing a comprehensive evaluation based on the three dimensions of real-time requirements, computing complexity and data size of the computing tasks, and assigning priority weight values; S202, arranging the priority weight values ​​in order from high to low to generate a task sequence.

3. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The method of executing computing tasks on each IoT node one by one according to the task sequence to obtain individual computing results for each IoT node includes: S301, allocating computing tasks according to the hardware performance and current load of each IoT node; S302, monitoring the resource utilization rate of each IoT node in real time, and suspending the allocation of new tasks when the resource utilization rate approaches a critical value; S303, reallocating unfinished tasks to other available IoT nodes until all computing tasks are completed.

4. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The method of executing computing tasks on all IoT nodes one by one according to the task sequence to obtain an overall computing result includes: S401, using the consistent hashing principle to evenly distribute computing tasks to each IoT node; S402, repeatedly executing some key tasks on multiple IoT nodes to improve reliability; S403, summarizing and verifying the computing results of all IoT nodes to generate an overall computing result.

5. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The method adjusts the order of computing tasks in the task sequence according to the resource occupancy rate of each computing task to generate a plurality of optimized task sequences, including: S501, randomly generating multiple groups of task sequences as an initial population (4); S502, calculating the fitness value according to the resource occupancy rate of each group of task sequences; S503, performing multiple rounds of iterative optimization on the task sequence through a crossover operation module (6) and a mutation operation module (7), to generate a plurality of optimized task sequences.

6. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The differentiated setting of computing paths according to the optimized task sequence includes: S601, selecting the shortest path (8) according to the real-time requirements of the computing task to reduce transmission delay; S602, selecting a high-bandwidth path (9) according to the data volume of the computing task to improve transmission efficiency; S603, setting multiple backup paths (10) for critical tasks to cope with the situation of primary path failure.

7. The method for processing Internet of Things data based on distributed computing power scheduling according to claim 1, characterized in that: The data processing operation on the IoT node using the computing path includes: S701, the dynamic scheduling module allocates computing tasks in real time according to the optimized task sequence and the differentiated computing path; S702, monitoring the task execution status of each IoT node, and when an IoT node has an abnormality, reallocating the tasks on it to other available IoT nodes.

Images