Deep reinforcement learning scheduling method and system for industrial Internet of Things streaming task

Abstract

The invention provides a deep reinforcement learning scheduling method and system for an industrial Internet of Things streaming task. The method comprises the steps of obtaining a current to-be-processed task, constructing an action space based on the current to-be-processed task and a computing node, and constructing a state space based on computing resources and network resources of the industrial Internet of Things; a reinforcement learning algorithm is adopted to carry out iterative calculation of a plurality of processing rounds, and in each processing round, the state vector of the current state is input into the deep reinforcement learning model; the deep reinforcement learning model comprises a plurality of processing units which are arranged in sequence, each processing unit comprises a hidden layer and an early-backing processing branch which are connected in sequence, a process tensor output by the hidden layer is input into the early-backing processing branch, and the early-backing processing branch outputs a process action; and early-departure condition judgment is carried out based on the output of the early-departure processing branch, the early-departure condition judgment comprises time delay early-departure condition judgment, and if the time delay early-departure condition is met, the process action is taken as the final action.

Description

A Deep Reinforcement Learning Scheduling Method and System for Streaming Tasks in the Industrial Internet of Things Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a deep reinforcement learning scheduling method and system for streaming tasks in the industrial IoT. Background Technology

[0002] The Industrial Internet of Things (IIoT), as a crucial application of next-generation information and communication technologies (ICT) in the industrial sector, integrates technologies such as artificial intelligence, cloud computing, and edge computing to construct a networked production system that deeply interconnects people, machines, and things, providing core support for achieving industrial digitalization, networking, and intelligence. Essentially, it uses a large number of sensors, controllers, and edge devices deployed in the Operations Technology (OT) domain to collect multi-dimensional data in real time, including production status, equipment operation, and environmental monitoring. This data is then combined with production plans, inventory, and order information from the Information Technology (IT) domain to support intelligent decision-making and dynamic scheduling in industrial production. In this context, IIoT not only brings tremendous innovation opportunities to the industrial sector but also becomes a vital support for promoting intelligent manufacturing and improving production efficiency.

[0003] In industrial IoT-supported production scenarios, information-based production equipment with sensing capabilities continuously generates massive amounts of data, such as facial data captured by cameras in key areas and anomaly detection data from robotic arms on production lines. The computing tasks and resources in such scenarios have the following characteristics: First, task processing requires high timeliness. Computational tasks generated by terminal devices typically have strict time constraints; failure to complete them in a timely manner will severely impact production efficiency and may even cause production failures. Second, some tasks have high demands on computing resources. In addition to timeliness, some data-intensive or computationally complex tasks require significant computing resources. Therefore, scheduling algorithms must rationally allocate tasks to appropriate nodes while also considering the overall state to avoid overall performance degradation.

[0004] Furthermore, industrial IoT environments often see a large number of latency-sensitive, dynamically changing, and unpredictable streaming tasks. These tasks have high requirements for real-time performance and low latency. Delayed scheduling responses can not only reduce production efficiency but also potentially lead to safety hazards. For example, in intelligent assembly lines, video monitoring and quality inspection can suffer from delayed streaming data processing, causing delayed feedback and potentially allowing defective products to enter subsequent stages, resulting in quality and safety risks. Therefore, efficient and reasonable task scheduling is crucial for ensuring system efficiency and optimal resource utilization in IIoT systems.

[0005] In this field, deep reinforcement learning models are commonly used for decision-making. However, the reasoning and decision-making process of deep reinforcement learning models often consumes a certain amount of time, which means that when the deep reinforcement learning model outputs the action corresponding to the task, it can no longer meet the time-latency sensitive characteristics of the task. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things (IIoT) to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things (IIoT). The method is applied to the IIoT, which includes multiple computing nodes. The steps of the method include: acquiring the currently pending tasks in the IIoT; constructing an action space based on the currently pending tasks and computing nodes; constructing a state space based on the computing and network resources of the IIoT; employing a reinforcement learning algorithm to perform iterative calculations for multiple processing rounds; in each iterative calculation round, inputting the state vector of the current state into a deep reinforcement learning model, the deep reinforcement learning model including multiple sequentially arranged processing units; each processing unit of the deep reinforcement learning model including a sequentially connected hidden layer and an early termination processing branch; inputting the process tensor output by the hidden layer into the early termination processing branch, the early termination processing branch outputting a process action; determining early termination conditions based on the output of the early termination processing branch, the early termination condition determination including a delay early termination condition determination; if the delay early termination condition is met, then the process action is taken as the final action; and determining the correspondence between the pending tasks and computing nodes based on the final action.

[0008] The above-mentioned scheme first employs a deep reinforcement learning model for task allocation decisions. Through iterative processing of the deep reinforcement learning model, the rationality of task allocation is ensured. Furthermore, considering the high latency sensitivity of the Industrial Internet of Things (IIoT), a deep reinforcement learning model structure distinct from existing technologies is constructed. This scheme fully considers the time consumed by the decision-making process of the deep reinforcement learning model. Each hidden layer in the deep reinforcement learning model structure of this scheme has an early termination processing branch. This branch can output process actions and determine whether the latency early termination condition is met based on the current process action. Based on the latency early termination condition, the final action can be output during one iteration of the deep reinforcement learning model, further meeting the latency requirements of the latency-sensitive dynamic flow tasks in the IIoT.

[0009] In some embodiments of the present invention, in the step of taking the process action as the final action if the early termination condition is met, the early termination condition is determined based on the current time, the task arrival time of each pending task, the expected processing time of each pending task, and the task tolerance delay.

[0010] In some embodiments of the present invention, in the step of determining the early termination condition based on the current time, the arrival time of each pending task, the estimated processing time of each pending task, and the task tolerance delay, the following formula is used to determine the early termination condition: in, Indicates the current time. This indicates the arrival time of task i to be processed. This represents the task tolerance latency for task i to be processed. This indicates the estimated processing time for assigning task i to computing node j.

[0011] In some embodiments of the present invention, in the step of inputting the process tensor output by the hidden layer into the early termination processing branch, and the early termination processing branch outputting the process action, the early termination processing branch adopts an Actor-Critic network structure, the early termination processing branch includes an Actor network and a Critic network, the Actor network outputs the process action based on the process tensor; the Critic network determines the process state based on the process action, and calculates the state value vector based on the process state.

[0012] In some embodiments of the present invention, the method further includes: determining the value early termination condition based on the current state value vector and the state value vector calculated by the previous early termination processing branch; if the value early termination condition is met, then the process action is taken as the final action.

[0013] In some embodiments of the present invention, in the step of determining the value early termination condition based on the current state value vector and the state value vector calculated from the previous early termination processing branch, the following formula is used to determine the value early termination condition: in, Represents the current state value vector. This represents the state value vector calculated in the previous early departure processing branch. This indicates the preset judgment threshold.

[0014] The above scheme incorporates a value early termination condition determination into the processing of the deep reinforcement learning model. The value early termination condition determination is used to determine the update magnitude of the state value vector between the two hidden layers. If the update magnitude is small, it indicates that the deep reinforcement learning model has been utilized to the maximum extent. Further processing would consume too much time but the optimization magnitude is small, thus further ensuring the requirement of latency sensitivity.

[0015] In some embodiments of the present invention, in the step of determining the early termination condition based on the output of the early termination processing branch, if the output of the early termination processing branch does not meet the delay early termination condition and does not meet the value early termination condition, it is determined whether the current processing unit is the last processing unit. If so, the current process action is used as the output of this processing round; if not, the process tensor output by the hidden layer of the current processing unit is input to the next processing unit.

[0016] In some embodiments of the present invention, in the iterative calculation of each processing round, an update state is determined based on the process action output in the current processing round, a reward function is calculated based on the update state, and the deep reinforcement learning model is trained based on the reward function.

[0017] In some embodiments of the present invention, the reward function is calculated using the following formula in the step of calculating the reward function based on the updated state: in, This represents the value of the reward function in the t-th processing round. This represents the latest completion time of all pending tasks in the t-th processing round. This indicates the completion time of task i to be processed. This represents the task tolerance latency for task i to be processed. and All of these are preset calculation parameters. Indicates an indicator function, .

[0018] A second aspect of the present invention also provides a deep reinforcement learning scheduling system for streaming tasks in the Industrial Internet of Things (IIoT). The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0019] A third aspect of the invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things.

[0020] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0021] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0022] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0023] Figure 1 is a schematic diagram of one embodiment of the deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to the present invention; Figure 2 is a schematic diagram of the Markov decision process for industrial task scheduling; Figure 3 is a schematic diagram of the overall architecture of the present solution; Figure 4 is a schematic diagram of the architecture of the deep reinforcement learning model of the present solution; Figure 5 is a schematic diagram of the hyperparameter sensitivity analysis of the present solution; Figure 6 is a schematic diagram comparing the timeout rate and computation latency of different methods with the present solution; Figure 7 is a schematic diagram comparing the timeout rate of each method with the present solution under different computation task scales; Figure 8 is a schematic diagram comparing the completion time of each method with the present solution under different computation task scales. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0025] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0026] In existing technologies, traditional batch processing tasks differ in task type and scheduling requirements. Batch processing tasks typically perform centralized optimization under the condition that the task set is known and the data is bounded, aiming to maximize throughput and globally optimal scheduling; while streaming tasks are oriented towards continuously arriving dynamic task flows, emphasizing rapid system response within the task tolerance latency. This difference leads to two limitations of traditional batch processing models in IIoT environments: first, the inherent data accumulation and waiting mechanisms of batch processing result in high latency, making it difficult to meet the stringent real-time requirements of scenarios such as device control and status monitoring; second, it is usually based on the assumption that the entire task set is known and relies on predefined strategies for batch scheduling, making it difficult to adapt to the uncertainty of task arrival and dynamic fluctuations of resources in IIoT, thus leading to lag in scheduling response and decreased resource utilization. Therefore, researching scheduling mechanisms that can adapt to the dynamic characteristics of task flows and have real-time response capabilities has become a key issue in improving the operating efficiency and intelligence level of IIoT systems.

[0027] Traditional IIoT task scheduling methods often employ heuristic algorithms (such as shortest job first, priority scheduling) or metaheuristic algorithms (such as genetic algorithms). These methods typically perform global optimization based on a static task set and preset rules, assuming that task attributes and resource states are known before scheduling. While they can quickly generate scheduling schemes, their static mechanism, which relies on manually configured rules, is ill-suited to the dynamic, aperiodic, and latency-sensitive streaming tasks in IIoT. On the one hand, fixed rules struggle to respond promptly to sudden task arrivals and resource state fluctuations; on the other hand, such strategies often fail to achieve an effective balance between real-time performance, flexibility, and overall scheduling performance, making it difficult to meet the high real-time and adaptive requirements of IIoT.

[0028] Deep reinforcement learning (DRL), as a method with adaptive decision-making capabilities, has shown significant potential in task scheduling problems in dynamic environments. DRL learns scheduling strategies through continuous interaction with the environment, enabling efficient decision-making in the face of dynamic task flows and complex system conditions. However, existing DRL models are typically structurally complex, requiring all network forward inference to output scheduling results, leading to high decision latency and making it difficult to meet the low-latency requirements of streaming tasks.

[0029] To address the aforementioned challenges, this study introduces an early exit mechanism. This mechanism interrupts the complete inference process of the current deep model upon task arrival and generates a scheduling strategy based on the intermediate layer output, effectively resolving the conflict between dynamism and real-time performance in IIoT streaming task scheduling. This approach aims to ensure the system can make scheduling decisions quickly upon task arrival and strives to minimize task completion time and improve task completion rate within tolerable latency. By designing a feature modeling method capable of real-time task complexity awareness and selecting an appropriate backbone network, combined with a multi-level early exit mechanism, this study achieves dynamic truncation of the task processing during model inference to reduce scheduling latency. Simultaneously, leveraging the accurate modeling capabilities of deep networks for complex tasks, an effective balance between scheduling efficiency and decision quality is achieved.

[0030] Introduction to Existing Technologies Existing Technologies I. In recent years, with the development of artificial intelligence technology, especially Deep Reinforcement Learning (DRL), new solutions have been provided for real-time scheduling problems in IIoT scenarios. DRL combines the powerful feature extraction capabilities of Deep Neural Networks (DNNs) with the decision-making framework of traditional Reinforcement Learning (RL), greatly expanding RL's ability to handle complex, high-dimensional state spaces. Deep Q-Learning proposed by Mnih et al. is an early and typical work combining deep learning with Q-Learning methods, significantly improving the ability of reinforcement learning to handle high-dimensional state spaces and initiating a research boom in DRL. Subsequently, algorithms such as Double DQN, Dueling DQN, DDPG, and PPO have been proposed, further optimizing the learning efficiency and stability of the models, enabling them to achieve breakthroughs in complex decision-making fields such as games and robot control.

[0031] DRL technology has also been widely used in task scheduling and resource allocation in IIoT and mobile edge computing (MEC). For example, a DRL-based distributed task offloading algorithm, by combining LSTM and DQN technologies, effectively addresses the dynamic changes in edge node load and significantly reduces latency and task drop rate; a DRL-based resource scheduling method has achieved significant results in reducing the average weighted turnaround time of tasks.

[0032] While the deep reinforcement learning-based method in Existing Technology 1 overcomes the shortcomings of traditional heuristic algorithms that rely on manual rules to some extent by introducing deep neural networks to model the system state in a high dimension and using policy networks to adaptively generate scheduling decisions, its specific technical solution still has the following inherent defects: First, existing DRL scheduling methods generally adopt a "single, complete forward inference" decision-making mode. That is, in each scheduling decision, the current system state needs to be input into the deep reinforcement learning model and complete the complete forward propagation process from the input layer to the output layer before the final scheduling action can be obtained. This technical feature makes the model inference latency significantly positively correlated with network depth, parameter size, and state dimension. In the streaming task scenario of Industrial Internet of Things, when tasks arrive frequently and the system state update rate is high, the scheduler needs to make multiple decisions continuously within a very short time window. The above-mentioned complete inference mechanism will inevitably introduce a large decision computation overhead, thereby compressing the remaining time available for transmission and execution of the task itself, resulting in an increase in the overall task completion latency.

[0033] Secondly, existing DRL models typically assume that the impact of decision time on scheduling objectives is negligible, failing to explicitly incorporate "model inference latency" into the scheduling optimization process. In these technical solutions, reward functions or optimization objectives are often focused on metrics such as task completion time, resource utilization, or load balancing, while the model's own inference time is not modeled as a state or penalty. This design means that the scheduling strategy is unaware of the "time cost of deep inference" during training. Consequently, in actual deployment, even when the system faces strict latency constraints, the model tends to perform complete inference, making it difficult to output a usable suboptimal scheduling solution in a timely manner.

[0034] Furthermore, existing DRL scheduling methods typically rely on neural networks with fixed structures and uniform depths for decision-making, and their network inference paths lack the ability to differentiate between tasks of varying complexity and urgency. Specifically, regardless of whether the arriving task is an urgent task with low computational load and short latency tolerance, or a regular task with high computational complexity and relatively relaxed latency constraints, the model uses the same network depth and inference process. This "one-size-fits-all" inference mechanism prevents the model from dynamically adjusting the computational intensity of decision-making based on the real-time requirements of the task, further limiting its applicability in latency-sensitive industrial scenarios.

[0035] Existing technology two addresses the problem of long inference time in deep learning models. Early Exit (EE) is a novel method proposed in recent years that accelerates computation by optimizing the network inference path. The core idea of ​​the EE mechanism is to set internal classifiers (ICs) or exit points in the intermediate layers of the deep neural network, allowing the model to terminate the computation early when the prediction result reaches sufficient confidence, thereby avoiding the computational overhead of full network inference.

[0036] Early exit strategies primarily focused on determining exit policies: early exit based on confidence (Entropy / Confidence). Existing techniques added early exit thresholds at specific locations within the deep neural network and determined whether to exit prematurely based on the entropy value of the prediction result. Current techniques have improved early exit strategies by directly measuring the reliability of the prediction result through a confidence threshold, achieving more efficient exit decisions.

[0037] Predictive early termination model: This method emphasizes real-time performance and low latency. By adding prediction heads to different layers, the model can output an effective prediction result at any point in the calculation process, allowing the model to gradually improve its prediction over time and stop when it reaches a sufficiently accurate result.

[0038] While Existing Technology 2 reduces unnecessary computational overhead to some extent by introducing a model early termination mechanism into deep neural networks, its specific implementation still has significant limitations, mainly in the following aspects: First, most existing early termination mechanisms are based on fixed confidence thresholds or preset entropy values ​​as exit criteria. In this type of technical solution, whether the model terminates inference early mainly depends on whether the output of the intermediate layer meets a pre-set uniform threshold, which usually remains unchanged after the model is deployed. This technical characteristic means that the early termination decision is only related to the current network output distribution, and is unrelated to external environmental factors such as the latency constraints of the input task, the system load status, and the urgency of scheduling, thus making the exit behavior lack dynamic perception capability.

[0039] Secondly, existing early termination methods are mainly geared towards classification tasks and inference accuracy optimization scenarios. These methods primarily focus on reducing model computation while ensuring that prediction accuracy does not significantly decrease, without incorporating system-level indicators such as "scheduling decision quality," "task completion delay," or "task timeout risk" into the exit decision conditions. Therefore, in industrial IoT scheduling scenarios, even if the prediction confidence of a certain intermediate layer is high, it may still produce scheduling results that are detrimental to the overall system performance by ignoring the remaining tolerance time of the current task or the queuing status of nodes.

[0040] As shown in Figure 1, this invention proposes a deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things (IIoT). The method is applied to the IIoT, which includes multiple computing nodes. The steps of the method include: Step S100, obtaining the currently pending tasks in the IIoT, constructing an action space based on the currently pending tasks and computing nodes, and constructing a state space based on the computing and network resources of the IIoT. In some embodiments of this invention, the resource scheduling problem is formalized as a finite-stage Markov decision process (MDP). Because the arrival and scheduling decisions of industrial task flows depend only on the current system state and are independent of historical scheduling trajectories, this perfectly matches the characteristics of a Markov decision process. Based on this, deep reinforcement learning is used to approximate its optimal strategy. The design of the MDP is shown in Figure 2.

[0041] The scheduling system observes the state at each time step, selects an action, and transitions to the next state based on the reward, repeating this process until the model converges or prematurely terminates. MDP is specifically represented as follows: State Space Composed of feature vectors learned by the neural network from all tasks and computing nodes, it reflects the current status of tasks to be processed and global information such as computing resources and network resources. The state represents the agent's observation of the current environment; the set of all task matrices, computing node resources, and network resource nodes embedded in the system constitutes the state of the Markov decision process. (State at a certain moment) Includes a real-time snapshot of the current environment, encoded as a tuple. ,in: This indicates the status of all tasks.

[0042] This indicates the status of all compute nodes.

[0043] Action space This indicates that a specific task is assigned to a specific computing node. Based on the logic of the industrial scheduling problem, this scheme sequentially assigns tasks to their corresponding nodes until all tasks are scheduled to their respective resource nodes. Each action represents an immediate decision made by the scheduler within a specified time window. When a latency-sensitive industrial flow task arrives, the scheduler must determine which computing node the task should be assigned to based on conditions such as status and rewards. The completion of each action marks the task's transition from the pending scheduling state to the execution state.

[0044] Actions are defined as task-node mapping matrices: Since a task can only correspond to a unique computation node in the mapping matrix of scheduling actions, the mapping matrix must satisfy the following constraint: At the same time, after a task is scheduled to a node, its total computational demand cannot exceed the available computing power of that node (considering the current load): according to and The environment will definitively transition to a new state. This refers to the new system state after task scheduling. Once an action is completed, i.e., a task is assigned to a unique, specific node, tasks are immediately inserted into the tail of the target node's local waiting queue according to the "First-Come, First-Served" (FCFS) principle; the node then executes tasks in sequence, and the remaining computation time and queue length of already running tasks are updated accordingly. Simultaneously, new tasks can... → Arrivals are random, ranging from +1, and node CPU / memory usage varies depending on task completion or new task injection. Therefore, in this scheme, the two different states are distinguished by the joint difference of the waiting queue length, the remaining computational load of tasks currently being executed by each node, and the triplet of the attribute matrix of the newly arrived task. The system is considered to have entered a new discrete state only if any one of these dimensions changes numerically. This ensures that the Markov property holds true.

[0045] Step S200: A reinforcement learning algorithm is used to perform iterative calculations for multiple processing rounds. In each processing round, the state vector of the current state is input into the deep reinforcement learning model, which includes multiple sequentially arranged processing units. Step S300: Each processing unit of the deep reinforcement learning model includes a sequentially connected hidden layer and an early termination processing branch. The process tensor output by the hidden layer is input into the early termination processing branch, and the early termination processing branch outputs the process action. In specific implementation, each hidden layer is equipped with a fully connected layer, and the early termination processing branch adopts the existing Actor-Critic network structure. Both the Actor network and the Critic network are equipped with multiple fully connected layers.

[0046] Step S400: Based on the output of the early termination processing branch, the early termination condition is determined. The early termination condition determination includes the delay early termination condition determination. If the delay early termination condition is met, the process action is taken as the final action. In specific implementation, this solution overcomes the shortcomings of traditional DRL inference (slow inference) and fixed early termination threshold (lack of adaptive capability) under the conditions of sudden arrival of IIoT latency-sensitive streaming tasks, extremely short latency tolerance, and drastic load fluctuations. It provides an efficient "decision-reasoning" integrated scheduling method, enabling the scheduler to output high-quality actions in advance at any intermediate layer according to the urgency of the task and the system status, thereby achieving multi-objective synchronous optimization with the lowest possible task timeout rate, the lowest possible average completion time, and the lowest possible decision delay.

[0047] Step S500: Determine the correspondence between the task to be processed and the computing node based on the final action.

[0048] In the specific implementation process, in the step of determining the correspondence between the task to be processed and the computing node based on the final action, the task to be processed is scheduled to the corresponding computing node based on the correspondence.

[0049] The above-mentioned scheme first employs a deep reinforcement learning model for task allocation decisions. Through iterative processing of the deep reinforcement learning model, the rationality of task allocation is ensured. Furthermore, considering the high latency sensitivity of the Industrial Internet of Things (IIoT), a deep reinforcement learning model structure distinct from existing technologies is constructed. This scheme fully considers the time consumed by the decision-making process of the deep reinforcement learning model. Each hidden layer in the deep reinforcement learning model structure of this scheme has an early termination processing branch. This branch can output process actions and determine whether the latency early termination condition is met based on the current process action. Based on the latency early termination condition, the final action can be output during one iteration of the deep reinforcement learning model, further satisfying the high latency sensitivity requirements of IIoT tasks.

[0050] In some embodiments of the present invention, in the step of taking the process action as the final action if the early termination condition is met, the early termination condition is determined based on the current time, the task arrival time of each pending task, the task arrival time of each pending task, and the task tolerance delay.

[0051] In some embodiments of the present invention, in the step of determining the early termination condition based on the current time, the arrival time of each pending task, the estimated processing time of each pending task, and the task tolerance delay, the following formula is used to determine the early termination condition: in, Indicates the current time. This indicates the arrival time of task i to be processed. This represents the task tolerance latency for task i to be processed. This indicates the estimated processing time for assigning task i to computing node j.

[0052] In some embodiments of the present invention, in the step of inputting the process tensor output by the hidden layer into the early termination processing branch, and the early termination processing branch outputting the process action, the early termination processing branch adopts an Actor-Critic network structure, the early termination processing branch includes an Actor network and a Critic network, the Actor network outputs the process action based on the process tensor; the Critic network determines the process state based on the process action, and calculates the state value vector based on the process state.

[0053] Specifically, the scheduling model in this solution is based on a centralized, globally managed industrial IoT workshop architecture. In this architecture, a high-performance central server (which can be considered a cloud or high-performance edge server) is responsible for aggregating all streaming task demands and real-time status information of computing nodes (such as edge gateways and industrial control computers) throughout the workshop, thereby obtaining global information. The deep reinforcement learning model in this solution is trained on this central server and ultimately deployed there to generate scheduling decisions in real time. This centralized design allows the scheduler to obtain task cluster demands and computing node statuses from a single location, enabling it to match tasks using globally optimal strategies.

[0054] To address the high dynamism and latency sensitivity of task flows in Industrial Internet of Things (IIoT) scenarios, this solution designs a time-sensitive task scheduling system architecture that combines deep reinforcement learning with an early exit mechanism. The deep reinforcement learning model employs a modified Actor-Critic (A2C) main network PPO optimization framework, deeply integrated with multi-level early exit branches, achieving a dynamic trade-off between scheduling quality and decision-making timeliness. The overall architecture is shown in Figure 3.

[0055] To address the characteristics of task modeling, this solution optimizes the traditional Actor-Critic main network in two ways: First, it adopts a shared-parameter A2C network design. This means the Actor (policy network) and Critic (value network) share the same set of underlying parameters to extract global representations of tasks and nodes. The core motivations for this design include: 1. Improved sampling efficiency and shared parameter learning. In traditional A2C, the Actor and Critic networks encode states separately, resulting in feature redundancy and wasted samples. This solution, by sharing the underlying feature extraction layer, allows the Actor and Critic to learn state representations in the same latent space. The Critic network provides meaningful gradient information to the Actor while learning the state-value mapping, thus improving sampling efficiency and training stability. 2. Consistency and interpretability of feature representations. In industrial task flow scheduling, task features and node features are strongly correlated. The shared network, through joint training, enables the model to simultaneously capture the interaction between the two, ensuring consistency in the Actor and Critic's understanding of the same state. This not only improves the stability of policy evaluation but also gives the network better generalization ability.

[0056] Therefore, the backbone of the model is constructed as a Shared Encoder, whose parameters are simultaneously used by the Actor and Critic subnetworks, outputting a shared high-dimensional state representation. Based on this, the Actor output scheduling strategy is distributed. Critic output state value estimation .

[0057] Second, the feature extraction layer design. Addressing the issue of inconsistent feature dimensions and magnitudes between tasks and computing nodes in an Industrial Internet of Things (IIoT) environment, this solution incorporates a feature extraction layer at the input layer front-end. This layer normalizes, reduces dimensionality, and embeds the original features. Because... and The values ​​of features in streaming tasks are dynamically changing, and direct processing can lead to numerical instability and convergence difficulties during training. The introduction of a feature extraction layer can normalize the original features. Each element in this feature set ( Each of these is a high-dimensional feature vector, representing the task. Assigned to node The potential adaptability at that time. × The high-dimensional feature set is then flattened or recombined and input into the backbone network.

[0058] To meet the latency-sensitive real-time inference requirements, this solution introduces an early termination mechanism in the hidden layer of the main network and adaptively improves it according to the characteristics of industrial scheduling tasks. This mechanism allows the model to output decision results in the intermediate layers during the inference process, thereby reducing inference time while ensuring accuracy.

[0059] As shown in Figure 4, after each hidden layer of the Shared network, an independent early exit branch is set, containing a set of lightweight Actor headers and Critic headers: the Actor header outputs the task-node scheduling probability matrix of the current layer. Critic head output state value This is used to estimate the global score of the current state. Each early exit branch constitutes a local "mini A2C unit," capable of generating a scheduling scheme without completing full network inference. During the training phase, a unique scheduling action matrix is ​​obtained by sampling the probability matrix output by the Actor network through policy sampling.

[0060] In some embodiments of the present invention, the method further includes: determining the value early termination condition based on the current state value vector and the state value vector calculated by the previous early termination processing branch; if the value early termination condition is met, then the process action is taken as the final action.

[0061] In some embodiments of the present invention, in the step of determining the value early termination condition based on the current state value vector and the state value vector calculated from the previous early termination processing branch, the following formula is used to determine the value early termination condition: in, Represents the current state value vector. This represents the state value vector calculated in the previous early departure processing branch. This indicates the preset judgment threshold.

[0062] Based on two early exit conditions, this scheme immediately exits the model and outputs the current decision result when the remaining decision time is insufficient or the change in state value tends to stabilize (reward gradient approaches 0). This dual-condition exit mechanism based on time constraints and value convergence ensures that the model can respond to extreme real-time requirements while continuing deep inference to improve accuracy when computational resources are sufficient.

[0063] The above scheme incorporates a value early termination condition determination into the processing of the deep reinforcement learning model. The value early termination condition determination is used to determine the update magnitude of the state value vector between the two hidden layers. If the update magnitude is small, it indicates that the deep reinforcement learning model has been utilized to the maximum extent. Further processing would consume too much time but the optimization magnitude is small, thus further ensuring the requirement of latency sensitivity.

[0064] In some embodiments of the present invention, in the step of determining the early termination condition based on the output of the early termination processing branch, if the output of the early termination processing branch does not meet the delay early termination condition and does not meet the value early termination condition, it is determined whether the current processing unit is the last processing unit. If so, the current process action is used as the output of this processing round; if not, the process tensor output by the hidden layer of the current processing unit is input to the next processing unit.

[0065] In some embodiments of the present invention, in the iterative calculation of each processing round, an update state is determined based on the process action output in the current processing round, a reward function is calculated based on the update state, and the deep reinforcement learning model is trained based on the reward function.

[0066] The reward reflects the quality of the agent's actions in the current environmental state. As mentioned earlier, this relates to the completion time of each task. This solution comprises four parts: model computation latency, transmission latency, queuing latency, and node processing latency. The goal is to minimize the maximum task completion time (makespan), i.e.: At the same time, consider the task timeout penalty. We assign weights to the task completion time and timeout penalty, and the optimization objective can be expressed as: , Therefore, in the step of calculating the reward function based on the updated state, the reward function is calculated using the following formula: in, This represents the value of the reward function in the t-th processing round. This represents the latest completion time of all pending tasks in the t-th processing round. This indicates the completion time of task i to be processed. This represents the task tolerance latency for task i to be processed. and All of these are preset calculation parameters. .

[0067] In practical implementation, this solution represents the tasks to be processed in the current industrial IoT scenario as follows: ,in ,here, This represents the total number of tasks pending processing at the current moment. Indicates the time of task arrival, Indicates task size, Indicates the amount of computation required for the task. This represents the task's tolerance latency. Similarly, the computing node is represented as... ,in ,here This represents the total number of computing nodes. Represents computing resources, Indicates network resources, Indicates the current queue length, This indicates the current load ratio.

[0068] Based on the task scheduling scenario of Industrial Internet of Things, this solution makes the following settings: (1) There is no data dependency or control dependency between any two computing tasks. (2) The waiting latency between the generation of a streaming task and its computation at the edge server is negligible. (3) The scheduling decision of each task will correspond to only one computing resource node, and task fragmentation is not allowed until the task is completed. (4) Tasks follow the First-Come, First-Served (FCFS) order in the local queue of the computing node, and the execution process cannot be preempted or migrated.

[0069] Task Scheduled to node Total completion time (Completion Time) includes model computation latency. Transmission delay Queuing delay Node processing latency The system consists of four parts. Model computation latency represents the time required for scheduling decision-making and model inference on the edge server, and is related to the model's performance and early termination settings. Transmission latency refers to the task... From end device scheduling to target node The required time is calculated as follows: Similarly, queuing delay represents the task. At the target node The time it takes for execution to begin. It is mainly determined by the node. The total load of currently waiting and executing tasks is determined by the following formula: in It is a very small positive number (such as 0.01) to prevent the denominator from being zero.

[0070] Node processing latency representation task At the target node The pure computation time required for execution is determined by the computational load of the task and the processing capacity of the node, and the calculation formula is as follows: The total task completion delay can be expressed as: = + + In the specific implementation process , and All of these can be obtained through simulation using pre-built simulation models.

[0071] Using the above scheme, this method formalizes the time-sensitive industrial task flow scheduling problem into a Markov Decision Process (MDP) with real-time decision constraints. This scheme comprehensively considers the current state of the tasks to be processed, computing resources, and network resources, and provides the optimal scheduling decision based on the current state, aiming to minimize the total computational latency of the tasks.

[0072] The training process for the Actor-Critic network with early termination is shown in Table 1 below: Table 1 While traditional Actor-Critic (A2C) algorithms can handle discrete action spaces, policy gradient training methods typically suffer from high variance, training instability, and low sample efficiency, making them unsuitable for IIoT scenarios requiring fast convergence and stable decision-making. To address the inherent shortcomings of A2C policy gradient training, in some embodiments of this invention, the Proximal Policy Optimization (PPO) algorithm is used to train the Actor-Critic main network. The PPO algorithm is one of the most widely used and stable policy gradient algorithms. By introducing importance sampling and policy clipping mechanisms, it restricts policy updates to a reasonable range, effectively balancing policy exploration and stability.

[0073] PPO not only ensures the stability of scheduling policy learning but also provides a unified training paradigm for multi-layered early termination structures. During PPO training, the model interacts with the environment, collecting trajectory data on states, actions, rewards, and the next state. This data is then used to calculate the advantage function and value loss, and the Actor network parameters are updated using a pruning objective function, while the Critic network parameters are updated to minimize value prediction error. The early termination mechanism also plays a role during training; the model can simulate early termination behavior during inference based on preset early termination conditions, enabling the model to learn effective decision-making capabilities at different depths. This training method ensures that, in actual deployment, the model can provide a relatively reasonable scheduling scheme regardless of when early termination is triggered.

[0074] During the training phase, the Actor-Critic module for all early-retreating branches is optimized simultaneously. For each branch, the model calculates the corresponding Actor loss and Critic loss, and adds a decision-time penalty term. in and where K represents the weighting coefficients. This design enables the model to actively learn the optimal balance between inference time and accuracy during training, where K is the total number of branches handling early termination. This represents the loss of the Actor network for the early termination processing branch k. This represents the loss of the Critic network for the early termination branch k. This represents the calculation decision time penalty term for the preset early departure processing branch k.

[0075] In summary, the PPO training framework mentioned in this section provides a stable and efficient learning paradigm for the complex multi-layered early-retreat network structure in this study through its inherent policy update pruning mechanism. This training process not only optimizes the global scheduling policy, but more importantly, it guides the model to actively learn the dynamic trade-off between scheduling decision quality and inference speed by jointly optimizing all early-retreat branches and introducing decision-time penalties.

[0076] Experimental Example: This experimental example constructs simulation platforms of different scales to simulate a cloud-edge-device hierarchical architecture. The experiment includes three different scenarios, each representing a different production scale: 10 nodes × 100 tasks, 20 nodes × 200 tasks, and 40 nodes × 400 tasks.

[0077] To demonstrate the efficiency of the industrial streaming task scheduling method proposed in this scheme, the model is compared with some representative task scheduling algorithms: Heuristic Algorithm: This scheme selects the First-In-First-Out (FIFO) strategy, the most classic and widely used strategy in industrial task scheduling, as a representative heuristic algorithm for comparison. The FIFO algorithm schedules tasks sequentially according to their arrival order, and is characterized by its simplicity and extremely low scheduling overhead, making it particularly suitable for scenarios where task arrival is stable and scheduling resources are sufficient. This scheme uses it as a comparison method to verify the advantages of the proposed method over traditional heuristic methods in terms of scheduling efficiency, task completion rate, and real-time performance when facing complex, dynamic, and latency-sensitive industrial task flows.

[0078] Metaheuristic Algorithm: This scheme selects the classic Genetic Algorithm (GA) as the baseline for its algorithm. The GA generates more solutions by mimicking the crossover and mutation of genes during biological reproduction, and simulates the natural selection process to eliminate inferior solutions. For the task scheduling problem, the GA initially generates an initial population using a greedy algorithm, where each genotype represents the scheduling and allocation results for all tasks. Then, the GA uses crossover and mutation to randomly propagate genes within the population. Finally, the gene sets of the next generation and the current generation are sorted, eliminating inferior genes. This process continues iteratively until the expected iteration limit is reached or all genotypes in the population are identical.

[0079] General Reinforcement Learning Algorithms: To demonstrate the efficiency of this approach, general reinforcement learning algorithms are selected as the baseline. This method avoids early model regression and main network optimization, and directly uses the policy gradient algorithm for training. Selecting a general deep reinforcement learning model as the experimental baseline serves two purposes: firstly, it demonstrates the performance improvement of the proposed method; secondly, using deep reinforcement learning as the baseline allows for observation of the performance comparison between deep learning methods and metaheuristic algorithms in handling task scheduling problems.

[0080] Through a series of extensive experiments and a comprehensive analysis of the performance of this method in a range of industrial scenarios, this approach has been meticulously evaluated for its scheduling efficiency and generalization ability. This rigorous evaluation once again clearly confirms the effectiveness of the model.

[0081] The parameters used in this scheme are mainly divided into two parts: industrial production parameters and model hyperparameters.

[0082] Industrial production parameters include computational task parameters, computational resource parameters, and network connection parameters. Computational tasks have parameters such as data volume, maximum tolerable latency, processing time, and scheduling status; computational resources have parameters such as task queue queuing time and the domain node ID; and the network has bandwidth parameters.

[0083] The model hyperparameters mainly include learning rate, batch size, and embedding dimension. The learning rate is between 0.0001 and 0.01 and is responsible for controlling the model's convergence speed. The batch size is between 64 and 256 and is responsible for defining the batch size. The embedding dimension is between 8 and 32 and is responsible for representing the comprehensive information of the nodes.

[0084] First, to explore the impact of hyperparameters on model performance, this approach conducted experiments with different batch sizes, embedding dimensions, and learning rates, identifying the optimal hyperparameter combination to maximize model accuracy and efficiency. The results are shown in Figure 5.

[0085] As shown in Figure 5(a), in the experiment regarding batch size, this scheme tested different batch sizes from 64 to 256. The experimental results show that the model performs best when the batch size is 128. Based on the experimental results, this scheme believes that smaller batch sizes often lead to insufficient regularization, while larger batch sizes increase memory requirements and prolong training time, thereby reducing model efficiency and performance. As shown in Figure 5(b), in the experiment regarding embedding dimension, this scheme tested different embedding dimensions from 8 to 32. The experimental results show that the model performs best when the embedding dimension is 16. Based on the experimental results, this scheme believes that if the embedding dimension is too small, the embedding representation is too poor and cannot effectively capture all the information within the node; while if the embedding dimension is too large, the required information becomes too sparse, making it more difficult for the model to capture the most important information contained in the embedding. Finally, as shown in Figure 5(c), in the study on learning rate, this approach experimented with learning rates from 0.1 to 0.0001 and found that the best performance was achieved when the learning rate was 0.0001. However, this approach also found that the change in learning rate had a less significant impact on the model's performance compared to the batch size and embedding dimension.

[0086] In summary, this approach not only obtained ideal hyperparameter values ​​through hyperparameter sensitivity experiments, but also clarified the impact of these parameters on model performance.

[0087] To verify the effectiveness of the proposed scheduling model in typical IIoT scenarios, we compared the proposed method (A2C+PPO+Early Exit) with First-Come, First-Served (FIFO), Genetic Algorithm (GA), and a general deep reinforcement learning (DRL) model without an early exit mechanism, based on initial settings and a relatively balanced configuration of computing power and network resources. The evaluation metrics selected were timeout rate and average computation latency. The timeout rate was defined as the percentage of tasks whose completion time exceeded their tolerance latency in a single scheduling cycle, while the average computation latency was the arithmetic mean of the completion times of all tasks in that batch.

[0088] As shown in Figure 6, while the proposed scheme achieves a slightly higher timeout rate than the traditional DRL method in small-scale scenarios, it achieves the lowest overall timeout rate across all three task scales, especially in medium-to-large-scale scenarios, where the timeout rate remains below 19%. This indicates that the proposed scheme can more effectively perceive task latency constraints and avoid task timeouts through dynamic scheduling strategies. The proposed scheme also performs best in terms of average computation latency. As the number of computing nodes and tasks increases, the task cluster size that the scheduling decision model on each node needs to handle increases, thus increasing the decision latency itself. In this scenario, the advantage of early-retirement branches becomes more apparent. In summary, in typical resource-balanced scenarios, the proposed deep reinforcement learning scheduling method based on an early-retirement mechanism outperforms traditional heuristic, metaheuristic, and general deep reinforcement learning methods in both task completion timeliness and system processing efficiency, validating its effectiveness and advancement in IIoT streaming task scheduling scenarios.

[0089] To evaluate the performance and scalability of the proposed scheduling model under different load conditions, we designed a dynamic load experiment. This experiment uses a fixed 5-minute time window and simulates three different system load scenarios—low, medium, and high—by varying the total number of randomly arriving streaming tasks within that time period. Specifically, these scenarios include: Scenario A (200 tasks), Scenario B (500 tasks), and Scenario C (1000 tasks). We also used timeout rate and average computation latency as core evaluation metrics to observe the performance of our proposed solution and the baseline under different loads. The experimental results are shown in Figures 7 and 8.

[0090] As shown in Figures 7 and 8, it can be observed that with a significant increase in system load, the timeout rate and average computation latency of all scheduling methods show an upward trend. In low-load scenarios (200 tasks), resources are relatively abundant, and the performance differences between methods are not significant. However, when the load rises to a medium level (500 tasks), the timeout rates of traditional FIFO and Genetic Algorithm (GA) climb rapidly, indicating that traditional methods based on static rules or global optimization are difficult to cope with dynamically increasing load pressure. Although the general DRL method is superior to traditional methods, its timeout rate and computation latency growth slope are also quite significant, showing that its decision-making ability is limited under sudden high loads.

[0091] In contrast, the DRL scheduling model based on the early-retreat mechanism proposed in this invention maintains optimal or suboptimal performance in all load scenarios, or even in some scale scenarios. Its advantages are particularly prominent in high-load and large-scale scenarios: its timeout rate is significantly lower than other comparative algorithms, and the growth of average computation latency is the most gradual. This indicates that although the decision-making difficulty increases exponentially under large-scale clusters, the early-retreat mechanism introduced in this scheme plays a crucial role at this scale. By dynamically exiting the inference process, it avoids the problem of insufficient remaining processing time or even timeouts caused by long decision-making delays. This demonstrates that the early-retreat mechanism in the model, by dynamically adjusting the inference depth, effectively strikes a favorable balance between decision quality and decision time under the pressure of high load and high real-time requirements, effectively avoiding system performance degradation caused by the scheduler's own decision-making delay. The experimental results also fully verify that this model has good scalability and robustness when facing unpredictable load fluctuations in IIoT scenarios.

[0092] This solution addresses the real-time scheduling challenges posed by latency-sensitive, non-periodic streaming tasks in Industrial Internet of Things (IIoT) scenarios. By modeling the scheduling problem as a finite-stage Markov decision process and explicitly introducing model inference latency constraints into a deep reinforcement learning scheduling framework, the scheduling strategy can simultaneously balance task execution latency, transmission latency, and decision computation overhead during the generation process. This fundamentally overcomes the problem that existing scheduling methods generally ignore the time cost of scheduling decisions themselves, thereby improving the scheduling model's ability to characterize and its applicability to real industrial operating environments.

[0093] Furthermore, this solution introduces a learnable model early termination mechanism into the Actor-Critic scheduling network, enabling the scheduling model to dynamically adjust its inference depth based on system state and time budget, thus achieving an adaptive balance between scheduling decision quality and real-time response capability. This technical solution avoids the latency amplification problem caused by the fixed inference process in traditional methods. In industrial IoT scenarios with limited computing power and bursty tasks, it can generate executable scheduling schemes earlier, thereby improving the system's stability and engineering deployability under dynamic load conditions. This provides a technical path with promotional value for time-sensitive streaming task scheduling in industrial IoT scenarios.

[0094] The beneficial effects of this scheme include: 1. A shared-encoding Actor-Critic backbone network, with lightweight Actor / Critic heads embedded in the hidden layers to form a "one master, multiple early exits" architecture, enabling the same model to have multiple schedulable inference depths.

[0095] 2. The timing of early termination is determined in parallel by two conditions: "remaining tolerance time of the task ≤ expected processing time" or "value change of adjacent layers < ε", so as to achieve a dynamic optimal trade-off between decision quality and inference delay.

[0096] 3. Explicitly add a decision time penalty term to the PPO joint loss, train all early termination branches end-to-end, and ensure that any exit point can output a feasible and optimized task-node mapping policy.

[0097] This invention also provides a deep reinforcement learning scheduling system for streaming tasks in the Industrial Internet of Things (IIoT). The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0098] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things (IIoT). The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0099] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0100] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0101] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things (IIoT), characterized in that, The method is applied to the Industrial Internet of Things (IIoT), which includes multiple computing nodes. The steps of the method include: acquiring the current pending task in the IIoT; constructing an action space based on the current pending task and computing nodes; constructing a state space based on the computing and network resources of the IIoT; employing a reinforcement learning algorithm to perform iterative calculations for multiple processing rounds; in each iterative calculation round, inputting the state vector of the current state into a deep reinforcement learning model, which includes multiple sequentially arranged processing units; each processing unit of the deep reinforcement learning model includes a sequentially connected hidden layer and an early termination processing branch; inputting the process tensor output by the hidden layer into the early termination processing branch, which outputs a process action; determining early termination conditions based on the output of the early termination processing branch, including a delay early termination condition determination; if the delay early termination condition is met, then the process action is taken as the final action; and determining the correspondence between the pending task and the computing node based on the final action.

2. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 1, characterized in that, In the step where the process action is taken as the final action if the early termination condition is met, the early termination condition is determined based on the current time, the arrival time of each pending task, the expected processing time of each pending task, and the task's tolerance delay.

3. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 2, characterized in that, In the step of determining the early termination condition based on the current time, the arrival time of each pending task, the estimated processing time of each pending task, and the task's tolerance latency, the following formula is used for determining the early termination condition: in, Indicates the current time. This indicates the arrival time of task i to be processed. This represents the task tolerance latency for task i to be processed. This indicates the estimated processing time for assigning task i to computing node j.

4. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 1, characterized in that, In the step of inputting the process tensor output by the hidden layer into the early termination processing branch, and the early termination processing branch outputting the process action, the early termination processing branch adopts an Actor-Critic network structure, the early termination processing branch includes an Actor network and a Critic network, and the Actor network outputs the process action based on the process tensor; The Critic network determines the process state based on process actions and calculates the state value vector based on the process state.

5. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 2, characterized in that, The method further includes the following steps: determining the value early termination condition based on the current state value vector and the state value vector calculated by the previous early termination processing branch; if the value early termination condition is met, then the process action is taken as the final action.

6. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 5, characterized in that, In the step of determining the early termination condition based on the current state value vector and the state value vector calculated from the previous early termination processing branch, the following formula is used for determining the early termination condition: in, Represents the current state value vector. This represents the state value vector calculated in the previous early departure processing branch. This indicates the preset judgment threshold.

7. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 5, characterized in that, In the step of determining the early termination condition based on the output of the early termination processing branch, if the output of the early termination processing branch does not meet the delay early termination condition and does not meet the value early termination condition, it is determined whether the current processing unit is the last processing unit. If so, the current process action is used as the output of this processing round; otherwise, the process tensor of the hidden layer output of the current processing unit is input to the next processing unit.

8. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to any one of claims 1 to 7, characterized in that, In each processing round of iterative calculation, the updated state is determined based on the process action output in the current processing round, the reward function is calculated based on the updated state, and the deep reinforcement learning model is trained based on the reward function.

9. The deep reinforcement learning scheduling method for streaming tasks in the Industrial Internet of Things according to claim 8, characterized in that, In the step of calculating the reward function based on the updated state, the reward function is calculated using the following formula: in, This represents the value of the reward function in the t-th processing round. This represents the latest completion time of all pending tasks in the t-th processing round. This indicates the completion time of task i to be processed. This represents the task tolerance latency for task i to be processed. and All of these are preset calculation parameters. 。 10. A deep reinforcement learning scheduling system for streaming tasks in the Industrial Internet of Things, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 9.

Images