A method and related device for edge-side collaborative optimization of a multi-output DNN model

Abstract

The application discloses a kind of edge-end collaborative optimization methods for multi-outlet DNN model and related equipment, it is related to edge intelligence field.In offline stage, DNN with multiple exit points is constructed and strategy model is trained;Online time slot bandwidth, EN / ES calculation and EN→ES transmission delay and queuing state are acquired, and strategy model outputs joint decision synchronous selection target exit point and partition point;In EN, intermediate characteristics are generated to partition point, and sent to ES through transmission queue and inference output is completed at exit point;According to the time of entering / leaving queue, update each stage waiting and delay, and roll collaborative inference.Through the precision-delay trade-off of bandwidth self-adaption, on the premise of meeting the maximum tolerable delay, low-latency, high-efficiency online inference is realized.

Description

Technical Field

[0001] This invention relates to the field of edge intelligence, and more specifically to an edge-end collaborative optimization method for multi-exit DNN models. Background Technology

[0002] With the development of artificial intelligence technology, deep neural networks (DNNs) have achieved remarkable application results in many fields such as computer vision, natural language processing, and speech recognition. However, as the complexity of tasks increases, the computational requirements of DNN models also increase significantly. Especially in embedded devices or edge computing scenarios, the limitations of computing power and bandwidth resources cause traditional cloud inference architectures to face huge performance bottlenecks.

[0003] To address this issue, edge computing has emerged as a new computing architecture. It reduces data transmission latency and alleviates computing pressure on the cloud by distributing computational tasks between edge nodes (such as smartphones and IoT devices) and edge servers located close to the data source. A typical edge-to-device collaborative inference architecture optimizes computational load, improves inference efficiency, and reduces latency by offloading some computational tasks to nodes while retaining some computation at the edge.

[0004] However, in practical applications, edge-to-edge collaborative inference still faces several technical challenges. First, the computing power of edge computing and edge devices is usually limited, especially when the device load is high. How to reasonably distribute tasks between nodes and the edge becomes a key factor affecting performance. Existing methods are usually based on static partitioning of single tasks. In a continuous DNN task flow, these methods often cannot fully utilize the dynamic resources of edge computing and edge nodes.

[0005] Secondly, network bandwidth fluctuations are another major factor affecting edge-to-edge collaborative inference performance. Unstable network bandwidth leads to significant latency in data transmission, thus impacting the completion time of inference tasks. Therefore, dynamically adjusting computational allocation under unstable bandwidth conditions to ensure timely task completion is a pressing issue.

[0006] Furthermore, most existing edge-to-edge collaborative inference frameworks employ fixed exit points and partition points, meaning the model switches computations at specific levels. However, this approach does not consider the impact of factors such as network bandwidth and device status on inference performance, resulting in inference performance failing to meet expectations under conditions of limited bandwidth or strained device resources. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to flexibly optimize task allocation based on dynamic network bandwidth and device computing resources in an edge computing environment, and improve the efficiency and accuracy of edge-end collaborative inference. The purpose is to provide an edge-end collaborative optimization method and related equipment for multi-exit DNN models, which solves the problems of existing static task allocation methods failing to fully utilize resources, bandwidth fluctuations leading to performance degradation and excessive inference latency.

[0008] This invention is achieved through the following technical solution:

[0009] An edge-to-edge collaborative optimization method for multi-exit DNN models is applied to a network composed of terminal devices, edge nodes, and edge servers. Its key feature is that, for each task in the inference task flow, a combination of exit points and partition points is selected based on network bandwidth and queue status. Logically, the method includes a model configuration phase, an offline training phase, and an online collaborative inference phase, wherein:

[0010] During the model configuration phase, a DNN model with multiple exits is constructed, and the execution latency parameters of each network layer of the DNN model on the terminal device side and the edge server side are obtained. At the same time, a queue model including a terminal computing queue, a transmission queue, and a server computing queue is constructed according to the inference task flow. The corresponding exit point and partition point combination are jointly determined for each task. The partition optimization problem is established with the goal of minimizing the average execution latency of all tasks and meeting the preset completion time threshold and inference accuracy requirements.

[0011] During the offline training phase, the partitioning optimization problem is formalized as a Markov decision process, where a system state including network bandwidth and queue state is defined, and the action of selecting an exit point and partition point for the current task is defined. A reward function is defined that gives a positive reward when the task meets the completion time threshold and accuracy requirements, and otherwise gives a penalty. Based on the Markov decision process, a deep reinforcement learning algorithm is used to train a partitioning scheduling policy network to obtain a partitioning scheduler that maps the system state to exit point and partition point decisions.

[0012] During the online collaborative reasoning phase, the currently detected network bandwidth and the queue status are input into the partition scheduler to obtain the exit point and partition point for the current task. The network layer before the partition point is executed on the terminal device side, and the network layer after the partition point is executed on the edge server. When the network reasoning reaches the exit point, the reasoning is terminated and the reasoning result is output. In the continuous task flow, the exit point and partition point of different tasks are dynamically adjusted according to the decision output by the partition scheduler.

[0013] During the online collaborative inference phase, the currently detected network bandwidth and the queue status are input into the partition scheduler to obtain the exit point and partition point for the current task. The network layer before the partition point is executed on the terminal device side, and the network layer after the partition point is executed on the edge server. When the network inference reaches the exit point, the inference is terminated and the inference result is output. In the continuous task flow, the exit point and partition point of different tasks are dynamically adjusted according to the decision output by the partition scheduler.

[0014] Furthermore, the model configuration stage includes: periodically collecting images, videos and / or sensor signals from the manufacturing site on the terminal device side based on a preset production cycle, encapsulating the collected data corresponding to each production cycle into a task to be analyzed, and forming a continuous DNN model inference task flow.

[0015] Furthermore, the establishment of the partitioning optimization problem includes: numbering the task flow according to discrete time slots, defining the arrival time, preset completion time threshold and corresponding inference accuracy requirements for each task, and recording the timestamps of each task entering and leaving the terminal computing queue, transmission queue and server computing queue in the queue model, taking the total task execution latency as the sum of the terminal computing latency, transmission latency and server computing latency, and taking the minimum average execution latency of all tasks and not exceeding their respective completion time thresholds as the optimization objective.

[0016] Furthermore, the queue model includes:

[0017] Terminal computing queue is used to describe the queuing and processing process of each task at the network layer before it waits for and executes its partition point at the edge node on the terminal device side;

[0018] Transmission queues are used to describe the queuing and transmission process of tasks waiting in a network link and completing intermediate feature transmissions.

[0019] The server computation queue is used to describe the queuing and processing of tasks at the network layer after each task waits and executes its partition point on the edge server.

[0020] The queue length and the estimated completion time of the task at the head of each queue are used to characterize the queue status.

[0021] Furthermore, the system status includes at least: the current network bandwidth, the queue length and / or estimated completion time of the terminal computing queue, transmission queue and server computing queue, the arrival time of the current task and the preset completion time threshold, and the model accuracy of the currently used exit point.

[0022] Furthermore, the reward function adopts a weighted form that simultaneously considers task inference accuracy and execution latency, wherein:

[0023] When the task execution delay does not exceed its preset completion time threshold and the inference accuracy meets the preset accuracy requirement, a corresponding reward value is set; when the task execution delay exceeds the preset completion time threshold or the inference accuracy is lower than the preset accuracy requirement, a corresponding penalty value is set, and the penalty value increases as the difference between the task execution delay and the preset completion time threshold increases.

[0024] Furthermore, the model configuration phase also includes:

[0025] For different types of network layers, hierarchical latency prediction models are constructed respectively. Historical execution samples of each network layer running on the edge nodes and edge servers of the terminal device are obtained. The samples include layer type, input and output feature size, convolution kernel size, number of convolution kernels, stride, number of channels and corresponding measurement latency.

[0026] A random forest regression model is trained based on the samples to predict the execution latency of each network layer on different computing nodes under a given exit point and partition point configuration during the offline training and online collaborative inference phases, thereby providing latency estimation for the partition optimization problem.

[0027] This invention also provides an edge-to-edge collaborative inference system for multi-exit DNN models, used to implement the aforementioned edge-to-edge collaborative optimization method for multi-exit DNN models, applied to a network composed of terminal devices, edge nodes, and edge servers, including:

[0028] Terminal equipment is used to collect on-site business data and generate DNN data streams to be analyzed.

[0029] Edge nodes, which are communicatively connected to the terminal device, are used to execute the portion of the deep neural network before the partitioning point;

[0030] An edge server, communicatively connected to the edge nodes, is used to execute the portion of the deep neural network after the partitioning point;

[0031] An optimization control device is communicatively connected to the terminal device, edge node, and edge server. The optimization control device includes:

[0032] The task generation module is used to generate a continuous deep neural network inference task flow based on the business data collected by the terminal device.

[0033] The queue modeling module is used to establish a queue model including a terminal computing queue, a transmission queue, and a server computing queue, and to determine the execution latency of each task in the terminal computing stage, the transmission stage, and the server computing stage based on the arrival time of each task, the queue status, and the execution latency parameters of each network layer on the terminal device side and the edge server.

[0034] The partition optimization modeling module is used to jointly determine the set of exit points and the set of partition points for each inference task in a deep neural network with multiple exit points. Based on the above queue model, a partition optimization problem is established with the goal of minimizing the average execution latency of all tasks and meeting the preset completion time threshold and inference accuracy requirements.

[0035] The deep reinforcement learning training module is used to formalize the partition optimization problem into a Markov decision process and train the partition scheduling policy network based on the deep reinforcement learning algorithm to obtain a partition scheduler that maps the system state to exit point and partition point decision.

[0036] The partitioned collaborative inference module is used to input the currently detected network bandwidth and the queue status into the partition scheduler during the online collaborative inference phase, obtain the exit point and partition point for the current task, execute the network layer before the partition point on the edge node, and execute the network layer after the partition point on the edge server. When the network inference reaches the exit point, the inference is terminated and the inference result is output. In the continuous task flow, the exit point and partition point of different tasks are dynamically adjusted according to the decision output by the partition scheduler to optimize the average execution latency of all tasks.

[0037] The present invention also provides an electronic device, including a memory, a processor, a communication module, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the edge-to-edge collaborative optimization method for multi-exit DNN models as described above.

[0038] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the edge-to-edge collaborative optimization method for multi-exit DNN models as described above.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] By constructing a queue model that includes terminal computing queues, transmission queues, and server computing queues, and taking "minimizing the average execution latency of all tasks while satisfying completion time thresholds and accuracy constraints" as the optimization objective under continuous task flow, the exit point and partition point are modeled as joint decision quantities. Compared with segmentation methods that only target single inference or static tasks, this method can more finely characterize latency bottlenecks and resource constraints in continuous task scenarios driven by industrial production cycle time, thereby improving overall resource utilization and task throughput.

[0041] The partitioning optimization problem is formalized as a Markov decision process, and a partition scheduler is trained based on deep reinforcement learning. This enables the partitioning strategy to adaptively learn the statistical laws of network bandwidth fluctuations and queue dynamic changes from historical interaction data. Compared with fixed partitioning strategies that rely on human experience or analytical optimization, it has stronger modeling and adaptive capabilities for complex nonlinear systems, which helps to stably meet the task completion time and accuracy requirements in a variable network environment.

[0042] During the online collaborative reasoning phase, the partition scheduler dynamically selects the exit point and partition point for each task based on real-time bandwidth and queue status, realizing adaptive computational division of labor between the terminal side and the edge server. Compared with fixed partition points or simple early exit strategies that only consider bandwidth, it can effectively reduce the probability of task timeouts and queuing congestion under high load or weak network conditions, and optimize the overall latency performance of continuous task streams.

[0043] The method and system of the present invention only require the addition of a partition scheduling and status monitoring module to the existing terminal equipment, edge node and edge server architecture. The changes to the existing deep neural network structure and business process are small, and it has good engineering feasibility and versatility. It can be adapted to a variety of multi-exit deep neural network models and different types of edge-cloud collaborative inference application scenarios. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0045] Figure 1 Here is a flowchart of the edge-to-edge collaborative optimization method for a multi-exit DNN model in Example 1;

[0046] Figure 2 This is a structural framework diagram of the edge-to-edge collaborative optimization method for multi-exit DNN models in Example 1;

[0047] Figure 3 This is a structural diagram of the DNN model with multiple exit points in Example 1;

[0048] Figure 4 End-to-end delay graphs for characterizing DNN models with multiple exit points when different partitioning points are selected;

[0049] Figure 5 To characterize the delay and output data size of each layer of a DNN model with multiple exit points on JetsonTX2;

[0050] Figure 6 To characterize the impact of different partition points on DNN execution latency under network changes;

[0051] Figure 7 To characterize the arrival of consecutive tasks, (a) all queues are waiting, (b) no waiting, and (c) some are waiting;

[0052] Figure 8 A flowchart for training a partition scheduler for Proximity Policy Optimization (PPO). Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0054] Example 1

[0055] An edge-to-edge collaborative optimization method for multi-exit DNN models, such as Figure 1 As shown, this method is applied to a network composed of terminal devices, edge nodes, and edge servers. Its characteristic is that, for each task in the inference task flow, a combination of exit points and partition points is selected based on network bandwidth and queue status. Logically, the method includes a model configuration phase, an offline training phase, and an online collaborative inference phase, wherein:

[0056] Step S1: Construct a DNN model with multiple exits and obtain the execution latency parameters of each network layer of the DNN model on the terminal device side and the edge server side; at the same time, construct a queue model including terminal computing queue, transmission queue and server computing queue according to the inference task flow, jointly determine the corresponding exit point and partition point combination for each task, and establish a partition optimization problem with the goal of minimizing the average execution latency of all tasks and meeting the preset completion time threshold and inference accuracy requirements;

[0057] Step S2: The partitioning optimization problem is formalized as a Markov decision process. The system state, including network bandwidth and queue state, is defined. The action is to select the exit point and partition point for the current task. A reward function is defined to give a positive reward when the task meets the completion time threshold and accuracy requirements, and otherwise to give a penalty.

[0058] Step S3: Based on the Markov decision process, a deep reinforcement learning algorithm is used to train the partition scheduling policy network to obtain a partition scheduler that maps the system state to exit point and partition point decisions.

[0059] Step S4: Input the currently detected network bandwidth and the queue status into the partition scheduler to obtain the exit point and partition point for the current task. Execute the network layer before the partition point on the terminal device side and the network layer after the partition point on the edge server. When network inference reaches the exit point, terminate the inference and output the inference result.

[0060] Step S5: In the continuous task flow, dynamically adjust the exit point and partition point of different tasks according to the decision output by the partition scheduler.

[0061] As an important middleware support between terminal devices and the cloud, the edge layer is divided into far-edge devices (such as sensors), mid-edge devices (such as single-board computing Jetson), and near-edge devices (such as fog servers) based on computing and storage capacity. Mid-edge devices are called edge nodes (EN), while near-edge devices are called edge servers (ES).

[0062] like Figure 2 As shown, the Internet of Things (IoT) terminal device (IoTD) is deployed in the workshop or production line to sense key data (images, videos, or sensor signals) of the manufacturing process. The IoTD is equipped with a single-board computer with processing, storage, and communication functions as an edge node (EN), and performs task analysis in collaboration with the nearest edge server (ES).

[0063] In application scenarios where edge nodes (EN) and nearby edge servers (ES) collaborate, the tasks to be analyzed rely on multi-exit deep neural network (DNN) models for feature extraction and real-time inference. Therefore, two types of component configuration files are constructed during the initial deployment phase:

[0064] Model Profile: This profile defines the multi-branch / multi-exit DNN architecture used, labels the layer index and computation depth corresponding to each exit, and pre-trains the model to ensure usable intermediate outputs at each exit. Different exits correspond to different computation depths, reflecting the trade-off between accuracy and latency, and providing a structured and accuracy-based basis for subsequent exit point selection.

[0065] Layer Profile: Under unified benchmark data and operating conditions, benchmark execution and time measurement are performed on each layer of the model on EN and ES respectively, recording the execution time of each layer on EN / ES and related input / output tensor scales and other information; the above measurement results serve as the raw data for the layer latency prediction model, which is used to build the DRL interactive environment during the offline training phase and support subsequent joint decision-making (exit point and partition point).

[0066] The two types of configuration files mentioned above serve as inputs for the offline training phase, supporting subsequent layer latency prediction modeling and policy training, and thus providing a basis for model partitioning and exit point selection for EN→ES in the online collaboration phase.

[0067] During the model configuration phase, images, videos, and / or sensor signals from the manufacturing site are periodically collected on the terminal device side based on a preset production cycle. The collected data corresponding to each production cycle is encapsulated into a task to be analyzed, forming a continuous DNN model inference task flow.

[0068] In offline training, a regression model (e.g., random forest) for layer latency prediction is trained based on layer execution latency samples of EN and ES collected from layer configuration files. This regression model, along with the already constructed multi-branch DNN model configuration, forms a deep reinforcement learning (DRL) interactive environment. The policy network is trained in this environment to obtain scheduler parameters for jointly selecting exit points and partition points. During training, a bandwidth-adaptive accuracy-latency tradeoff design is employed, enabling the policy to dynamically balance end-to-end inference time and inference accuracy under different bandwidth conditions. After training, the scheduler is deployed online as a queue model for real-time invocation.

[0069] The queue model includes:

[0070] Terminal computing queue is used to describe the queuing and processing process of each task at the network layer before it waits for and executes its partition point at the edge node on the terminal device side;

[0071] Transmission queues are used to describe the queuing and transmission process of tasks waiting in a network link and completing intermediate feature transmissions.

[0072] The server computation queue is used to describe the queuing and processing of tasks at the network layer after each task waits and executes its partition point on the edge server.

[0073] The queue length and the estimated completion time of the task at the head of each queue are used to characterize the queue status.

[0074] In each time slot, the queue model reads the system state from the current environment (including network bandwidth, EN computation stage latency, EN→ES transmission stage latency, ES computation stage latency, and corresponding queuing waiting time), and outputs a joint decision {target exit point ep, target partition point pp}. When partitioning the model, the selected branch divides the DNN into two segments according to the layer boundaries of the hierarchical index. The model structure parameters of the shallow part are deployed in EN, and the remaining part is deployed in ES. Subsequently, EN executes up to pp to obtain intermediate features, which are sent to ES via the EN→ES transmission queue. ES continues execution from pp to ep, obtaining the inference output of the task at ep. The enqueue / dequeue times of the task and intermediate features are used to update the queue waiting time and stage latency, and then proceed to the next time slot for rolling processing.

[0075] In this embodiment, each task in the task flow includes information on the input data size, maximum tolerable latency, and the number of layers to be processed; the tasks generated by IoTD are represented as follows: ,in Indicates the size of the input data. This represents the operation layer for task processing driven by the DNN, where m is the maximum number of layers. This indicates the maximum tolerable latency for the task; the task's response time should not exceed [a certain threshold]. In this embodiment, the preferred DNN model is AlexNet, which serves as the main network structure as follows: Figure 3 As shown in the MainBranch, the DNN model contains Layers. DNN model inference is performed layer by layer in sequence. To give the final result.

[0076] For a given In a layered DNN model, task processing is modeled as a set of subtasks with layer operations as the smallest unit. In forward reasoning, The output of the layer as The input to each layer is used to maintain the sequential execution strategy of the deep neural network. The computational latency of the DNN model can be estimated based on the floating operations of each layer of the neural network.

[0077] Input data dimensions are The output data dimension is The kernel size is The floating-point operands of convolution operations and fully connected operations are given in the following formula:

[0078]

[0079]

[0080] Figure 4This paper illustrates the relationship between latency and output data size across different layers in a DNN model. The execution latency of these layers exhibits significant heterogeneity. For example, higher-latency layers such as convolutional layers (Conv-2) and fully connected layers (FC-1, FC-2) typically do not generate large amounts of output data, while shallower layers (such as convolutional layers) generate large amounts of data and increase computational cost. During computation, pooling layers reduce data size by merging feature maps, while fully connected layers transform features into one-dimensional vectors through non-linear mapping. In this way, the computational latency of deeper layers may gradually increase, while the output data of subsequent layers gradually decreases. Therefore, this invention dynamically optimizes collaborative inference performance based on this data characteristic. This optimization strategy optimizes the allocation and computation process of inference tasks by combining the latency and data size characteristics of each layer.

[0081] The system status includes at least: the current network bandwidth, the queue length and / or estimated completion time of the terminal computing queue, transmission queue and server computing queue, the arrival time of the current task and the preset completion time threshold, and the model accuracy of the currently used exit point.

[0082] Based on the system state, a deep reinforcement learning algorithm (such as PPO) is used to train a partition scheduling policy network, resulting in a partition scheduler that maps the system state to exit point and partition point decisions. This trained partition scheduler is then used to schedule partition decisions for continuous task flows in real time. Specifically, the task modeling is as follows.

[0083] At time t, the inference task generated by the IoTD is represented as follows:

[0084]

[0085] The extra tuples Indicates the time required to complete the task;

[0086] The set of exit point selections is represented as

[0087]

[0088] For the number of branches, when When that happens, the model ends its inference from the current exit point.

[0089] The partition point selection set is represented as ,

[0090] For the first The number of layers in a branch model.

[0091] Task Collaborative reasoning strategies Defined as

[0092]

[0093] and The choice of exit point and partition point, both of which can lead to execution delays.

[0094] Once a task is generated, Elasticsearch (ES) formulates an unloading strategy based on the current environment and task information, requiring an accurate estimated execution time. As a layer delay function, it represents the current unloading strategy. Next, the Layer execution time, For input data.

[0095] when hour, This indicates the estimated processing time on the terminal device. At that time, task computation is performed on an edge server.

[0096] During implementation, the impact of the DNN's partitioning decisions on the overall inference latency must also be considered. Each layer of the DNN model can serve as a potential partitioning point. For example... Figure 5 It is known that optimal partitioning can significantly reduce inference latency. In the AlexNet model, depending on the partitioning point, the execution of the first few layers will be completed on the edge device (EN), while subsequent layers will be offloaded to the edge server (ES) for processing. The optimal partitioning point is usually at Pool-3 layer; however, incorrect partitioning can lead to increased inference latency and even unnecessary performance bottlenecks. Therefore, reasonable partitioning decisions need to consider dynamic factors such as the computing power of edge nodes and network bandwidth.

[0097] In collaborative reasoning, the impact of unstable transmissions on performance must be considered. Figure 6 It is known that in dynamic network environments, the optimal partitioning layer of AlexNet changes with increasing bandwidth. When the bandwidth is below 5 Mbps, the optimal partitioning is at the FC-3 layer. However, when the bandwidth is between 5-15 Mbps and above 15 Mbps, the optimal decision changes to the FC-1 and Pool-3 layers. Therefore, dynamic environments pose challenges to partitioning decisions due to network variations and the unpredictable impact of different layer structures. Even in continuous task flows, the influence between adjacent tasks must be considered. This necessitates a unified partitioning optimization model to model the three stages of EN computation, transmission, and ES computation.

[0098] like Figure 7 As shown, Once determined and sent to EN, calculations can be performed. EN computation latency (node ​​computation phase latency) Transmission delay (transmission phase delay) and ES computation latency (Latency during edge computing). Among them,

[0099] Since the final inference result of ES is represented in a fixed text format and the amount of data transmitted is minimal, the feedback latency from ES to EN is ignored. In the continuous task optimization scenario, the interval between generating DNN inference tasks at the beginning of each time slot is... The arrival time relationship between consecutive adjacent tasks is: When there are incomplete tasks in the current queue, the overall processing delay of the current task needs to take into account the queuing time, thus affecting the overall computational delay of the task in EN. Represented as:

[0100]

[0101] in, Calculate the end time of the queue for EN.

[0102] After EN completes local computation, the intermediate data produced will be transmitted to an edge server to complete subsequent inference. Indicates the first The output data size of the layer, This represents the transmission rate of edge nodes and edge servers. In large-scale manufacturing industries, the EN (electron frequency) is basically fixed, and the channel is stable during the task. To simplify the network model, communication interference and congestion are ignored. If the bandwidth between EN and ES is expressed as , then the transmission delay is expressed as . The overall transmission delay of the task after considering queue information. Represented as:

[0103]

[0104] Once the edge server receives the intermediate data uploaded by the device, it enters the maintenance queue and uses pre-isolated resources to complete the remaining inference. At this point, the overall computation of the task in Elasticsearch... The delay is represented as:

[0105]

[0106] Therefore, the overall execution delay of the task can be expressed as the sum of the delays of the three stages:

[0107]

[0108] The establishment of the partition optimization problem includes: numbering the task flow according to discrete time slots, defining the arrival time, preset completion time threshold and corresponding inference accuracy requirements for each task, and recording the timestamps of each task entering and leaving the terminal computing queue, transmission queue and server computing queue in the queue model, taking the total task execution latency as the sum of terminal computing latency, transmission latency and server computing latency, and taking the minimum average execution latency of all tasks and not exceeding their respective completion time thresholds as the optimization objective.

[0109] The objective of the strategy model is the average latency of all tasks, and its optimization problem P1 is expressed as:

[0110]

[0111]

[0112]

[0113]

[0114] Where B is the set of all exit points, The constraint states that the task arrives at a fixed time interval. Constraints and The constraints indicate the decision range for exit points and partition points. Based on the system state, the end-to-end inference time of the current task is estimated, and the maximum tolerable latency of the task is used as a hard constraint. Any binary combination whose estimated value exceeds this latency threshold is eliminated from the feasible region, and the joint decision is only output from the remaining feasible combinations by the policy model.

[0115] Choose the optimal exit from the M exits, and then... The decision is made using partition points, and its search space is... .

[0116] The reward function adopts a weighted form that simultaneously considers task inference accuracy and execution latency, wherein:

[0117] When the task execution delay does not exceed its preset completion time threshold and the inference accuracy meets the preset accuracy requirement, a corresponding reward value is set; when the task execution delay exceeds the preset completion time threshold or the inference accuracy is lower than the preset accuracy requirement, a corresponding penalty value is set, and the penalty value increases as the difference between the task execution delay and the preset completion time threshold increases.

[0118] The partitioning strategy pi(t) is dynamically optimized, and the partition scheduler interacts with the IIoT environment. Different partitioning actions are executed to complete state transitions and receive rewards from the environment. The optimal offloading strategy is obtained by maximizing the expected reward. This process mainly consists of a state space, an action space, and a reward function.

[0119] The state space reflects the information the scheduler observes from the environment at time slot t. S(t) is defined as the state space of the perceived environment at time t, composed of current environment settings, delay information generated through state-action pairs, and queue queuing information, and is expressed as:

[0120]

[0121] in The network bandwidth at the current moment. For model accuracy, and These represent the sets of delays and queueing times at different stages of the task, respectively. Because there are continuous variables, the state space is infinite.

[0122] At each decision, the DRL scheduler selects an action based on the state observed from the current environment. Actions are set as branch and partition point selections associated with the model; the partitioning strategy action is represented as follows:

[0123]

[0124] in, and These represent the exit point and partition point selection at the current moment, respectively. Let ... .

[0125] To dynamically optimize the partitioning strategy pi(t), the partition scheduler interacts with the IIoT environment. It executes different partitioning actions to complete state transitions and receives rewards from the environment. The optimal offloading strategy is obtained by maximizing the expected reward. This process mainly consists of a state space, an action space, and a reward function.

[0126] The state space reflects the information the scheduler observes from the environment at time slot t. S(t) is defined as the state space of the perceived environment at time t, composed of current environment settings, delay information generated through state-action pairs, and queue queuing information, and is expressed as:

[0127]

[0128] in The network bandwidth at the current moment. For model accuracy, and These represent the sets of delays and queueing times at different stages of the task, respectively. Because there are continuous variables, the state space is infinite.

[0129] At each decision, the DRL scheduler selects an action based on the state observed from the current environment. Actions are set as branch and partition point selections associated with the model; the partitioning strategy action is represented as follows:

[0130]

[0131] in, and These represent the exit point and partition point selection at the current moment, respectively. Let ... .

[0132] Due to the high cost of optimal decision-making in continuous scenarios, setting a fixed target score during training is impractical. Therefore, a reward function is used to guide the policy. The decision-making process evolves towards the optimization goal of reducing the total latency of all tasks. In particular, the design of the reward function incorporates two parameters: accuracy... and predefined completion time threshold , is represented as:

[0133]

[0134] When making decisions, the scheduler selects actions based on environmental reward values ​​to maximize long-term discount benefits.

[0135] The higher the accuracy, the greater the reward; conversely, the lower the accuracy, the greater the task execution delay. and The larger the difference, the smaller the reward. λ1 and λ2 are set as penalty indicators for dynamically adjusting two hyperparameter factors. They automatically fine-tune the objective based on the time-varying network environment, and are expressed as:

[0136]

[0137] in The logarithmic form representing bandwidth is calculated as follows: .

[0138] when That is, when the bandwidth is at its minimum, =ln(1); When the bandwidth reaches its maximum, =ln(2). Set to 1- By balancing accuracy and execution latency using a reward function, the scheduler's adaptive partitioning decision-making performance in continuous dynamic environments can be maximized.

[0139] The model configuration phase also includes:

[0140] For different types of network layers, hierarchical latency prediction models are constructed respectively. Historical execution samples of each network layer running on the edge nodes and edge servers of the terminal device are obtained. The samples include layer type, input and output feature size, convolution kernel size, number of convolution kernels, stride, number of channels and corresponding measurement latency.

[0141] A random forest regression model is trained based on the samples to predict the execution latency of each network layer on different computing nodes under a given exit point and partition point configuration during the offline training and online collaborative inference phases. Based on this latency prediction, a deep reinforcement learning-based partition scheduler is trained to provide latency estimation for the partition optimization problem.

[0142] Before performing DRL offline training in the partition scheduler, the layer execution latency of arbitrary neural network architectures was modeled. This approach allows the system to estimate the latency of the DNN constituent layers for DRL policy training without executing the DNN itself, when the DNN layer is used as the smallest granularity of the partition. For each layer type, diverse layer configurations (regarding input data, kernel size, etc.) and hardware resources determine the latency variation, making it difficult to obtain the latency of each layer on each computation node. To uniformly estimate the inference time of the DNN, a regression function based on Random Forest (RF) was designed.

[0143] Based on the layer delay function, under the existing partitioning strategy, the delay of each layer at EN or ES is obtained. Common DNN models typically contain five layer types, and the layer delay function and its variables for each type are shown in Table 1. The delay impact parameters for convolutional layers, local normalization layers, and pooling layers include the number of input feature maps (cin), the kernel size (ksize), the number of kernels (knumber), and the stride. The input feature size for their RF regression model is 2. The regression model for convolutional layers is based on two variables: cin, and (ksize / stride)2×knumber, representing the computational cost per pixel in the input feature map. For pooling layers and fully connected layers, the input data features (Hin, win, cin) and output data features (Hout, wout, cout) are used as regression model variables. The regression models for other layers, such as the ReLU activation layer and the local normalization layer, have an input feature size of 1, and the input data size is used as their delay impact variable. Furthermore, residual structures in networks such as ResNet can be viewed as ensemble modules composed of a series of convolutional and normalization layers.

[0144] Table 1 Layer Delay and its Influencing Variables

[0145]

[0146] The input size of the RF regression model for all layers is determined according to the layer type, and the output is the predicted layer latency, using mean squared error as the loss function. The latency dataset is collected from the actual execution latency of different layers of various DNN models on EN and ES platforms, and layer latency functions for different layer types are fitted using historical data as components for subsequent DRL scheduler training.

[0147] In this embodiment, as Figure 8 As shown, a policy-based DRL algorithm, Proximal Policy Optimization (PPO), is used to train the partition scheduler. The cumulative reward obtained by the scheduler from the environment is defined as:

[0148]

[0149] in, Let be the discount factor. Assume K iterations are performed, and the strategy... The goal is to maximize the expected cumulative reward. Represented as

[0150]

[0151] in Represents the strategy trajectory The probability of the state. Next, the state value is evaluated using a critic. It is a strategy In state The expected cumulative return obtained at that time. We introduce As The baseline is set to avoid situations where all rewards are positive, allowing the scheduler to focus on more important actions when updating the policy. The advantage function and the new policy objective can be written as follows:

[0152]

[0153]

[0154] An older actor network was used to interact with the environment to obtain trajectories and applied gradient ascent to update policy parameters. It approximates the distribution of the target actor. Let... and Let represent the action probabilities of the objective and the old policy, respectively. Then the policy gradient can be expressed as:

[0155]

[0156] The strategy can be improved by increasing the probability of actions that yield greater rewards. Let Compared to other policy networks, the PPO algorithm uses pruning parameters to limit the changes in each network update. It allows for mini-batch training over multiple epochs using the experience gathered in the current iteration. Therefore, the loss function of the Actor network is expressed as...

[0157]

[0158] in,

[0159]

[0160] Constrained and The difference, to prevent strategy divergence, It can prevent parameters The update step size is too large, which hinders policy stability. Due to the adoption of a parameter-sharing DNN architecture, the overall loss function is a combination of the error terms of the policy network (Actor) and the value network (critic). We also add entropy gain to further enhance the target, allowing the algorithm to explore more fully. The overall loss function can be written as...

[0161]

[0162] in,

[0163]

[0164] , It is the loss coefficient. Entropy gain, To calculate the network loss, mean squared error is used. Based on the terms of the loss function in equation (19), in each iteration, Updated via the Adam optimizer to minimize And eventually converge to the optimal partitioning strategy.

[0165] Example 2

[0166] An edge-to-edge collaborative inference system for multi-exit DNN models is provided to implement the edge-to-edge collaborative optimization method for multi-exit DNN models as described in Example 1 above. The system is applied to a network consisting of terminal devices, edge nodes, and edge servers, and includes:

[0167] Terminal equipment is used to collect on-site business data and generate DNN data streams to be analyzed.

[0168] Edge nodes, which are communicatively connected to the terminal device, are used to execute the portion of the deep neural network before the partitioning point;

[0169] An edge server, communicatively connected to the edge nodes, is used to execute the portion of the deep neural network after the partitioning point;

[0170] An optimization control device is communicatively connected to the terminal device, edge node, and edge server. The optimization control device includes:

[0171] The task generation module is used to generate a continuous deep neural network inference task flow based on the business data collected by the terminal device.

[0172] The queue modeling module is used to establish a queue model including a terminal computing queue, a transmission queue, and a server computing queue, and to determine the execution latency of each task in the terminal computing stage, the transmission stage, and the server computing stage based on the arrival time of each task, the queue status, and the execution latency parameters of each network layer on the terminal device side and the edge server.

[0173] The partition optimization modeling module is used to jointly determine the set of exit points and the set of partition points for each inference task in a deep neural network with multiple exit points. Based on the above queue model, a partition optimization problem is established with the goal of minimizing the average execution latency of all tasks and meeting the preset completion time threshold and inference accuracy requirements.

[0174] The deep reinforcement learning training module is used to formalize the partition optimization problem into a Markov decision process and train the partition scheduling policy network based on the deep reinforcement learning algorithm to obtain a partition scheduler that maps the system state to exit point and partition point decision.

[0175] The partitioned collaborative inference module is used to input the currently detected network bandwidth and the queue status into the partition scheduler during the online collaborative inference phase, obtain the exit point and partition point for the current task, execute the network layer before the partition point on the edge node, and execute the network layer after the partition point on the edge server. When the network inference reaches the exit point, the inference is terminated and the inference result is output. In the continuous task flow, the exit point and partition point of different tasks are dynamically adjusted according to the decision output by the partition scheduler to optimize the average execution latency of all tasks.

[0176] Example 3

[0177] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the edge-to-edge collaborative optimization method for a multi-exit DNN model as described in Embodiment 1.

[0178] Example 4

[0179] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the edge-to-edge collaborative optimization method for a multi-exit DNN model as described in Example 1.

[0180] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0185] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An edge-end collaborative optimization method for multi-exit DNN models, applied to a network composed of terminal devices, edge nodes, and edge servers, characterized in that, For each task in the inference task flow, an exit point and a partition point combination are selected according to the network bandwidth and the queue state, and the method logically comprises a model configuration stage, an offline training stage and an online collaborative inference stage, wherein: In the model configuration stage, a DNN model with multiple exits is constructed, and the execution delay parameters of each network layer of the DNN model on the terminal device side and the edge server side are obtained; at the same time, a queue model including a terminal computing queue, a transmission queue and a server computing queue is constructed according to the inference task flow, and the corresponding exit point and partition point combination for each task are jointly determined, and a partition optimization problem is established with the goal of minimizing the average execution delay of all tasks and meeting the preset completion time threshold and inference accuracy requirements; In the offline training stage, the partition optimization problem is formalized as a Markov decision process, wherein the system state including the network bandwidth and the queue state is defined, the exit point and the partition point for the current task are selected as actions, and a reward function is defined to give a positive reward when the task meets the completion time threshold and the accuracy requirement, otherwise a penalty is given; based on the Markov decision process, a deep reinforcement learning algorithm is used to train a partition scheduling policy network to obtain a partition scheduler that maps the system state to the exit point and the partition point decision; In the online collaborative inference stage, the currently detected network bandwidth and the queue state are input into the partition scheduler to obtain the exit point and the partition point for the current task, the network layers before the partition point are executed on the terminal device side, and the network layers after the partition point are executed on the edge server, the inference is terminated when the network inference reaches the exit point, and the inference result is output, and the exit point and the partition point of different tasks are dynamically adjusted according to the decision output by the partition scheduler in the continuous task flow.

2. The method of claim 1, wherein, The model configuration stage includes: periodically collecting images, videos and / or sensor signals of the manufacturing site on the terminal device side based on a preset production cycle, encapsulating the collected data corresponding to each production cycle as a task to be analyzed to form a continuous DNN model inference task flow.

3. The method of claim 2, wherein, The establishment of the partition optimization problem includes: numbering the task flow by discrete time slots, defining the arrival time, the preset completion time threshold and the corresponding inference accuracy requirement for each task, and recording the timestamps of each task entering and leaving the terminal computing queue, the transmission queue and the server computing queue in the queue model, taking the total execution delay of the task as the sum of the terminal computing delay, the transmission delay and the server computing delay, and taking the minimum average execution delay of all tasks and not exceeding the respective completion time threshold as the optimization goal.

4. The method of claim 1, wherein, The queue model includes: a terminal computing queue for describing the queuing and processing of network layers before the partition point of each task on the terminal device side edge node; a transmission queue for describing the queuing and transmission of intermediate features in the network link; a server computing queue for describing the queuing and processing of network layers after the partition point of each task on the edge server; wherein the queue length and the expected completion time of the head task of each queue are used to represent the queue state.

5. The method of claim 1, wherein, The system state at least includes: current network bandwidth, queue length and / or estimated completion time of the terminal computing queue, transmission queue and server computing queue, arrival time of the current task and preset completion time threshold, and model accuracy of the current adopted exit point.

6. The method of claim 2, wherein, The reward function adopts a weighted form considering task reasoning accuracy and execution delay, wherein: When the task execution delay does not exceed the preset completion time threshold and the reasoning accuracy meets the preset accuracy requirement, the corresponding reward value is set; when the task execution delay exceeds the preset completion time threshold or the reasoning accuracy is lower than the preset accuracy requirement, the corresponding penalty value is set, and the difference between the task execution delay and the preset completion time threshold increases.

7. The method of claim 1, wherein, In the model configuration stage, further comprising: For different types of network layers, a hierarchical delay prediction model is constructed, historical execution samples of each network layer running on the terminal device side edge node and edge server are obtained, the samples include layer type, input and output feature size, convolution kernel size, convolution kernel number, step, channel number and corresponding measured delay; Based on the samples, a random forest regression model is trained, which is used to predict the execution delay of each network layer on different computing nodes under the given exit point and partition point configuration in the offline training stage and online collaborative reasoning stage.

8. A system for edge-server collaborative optimization for multi-output DNN model, for implementing the method of any one of claims 1-7. Applied to a network composed of a terminal device, an edge node and an edge server, comprising: The terminal device is used for collecting field business data and generating DNN data stream to be analyzed; The edge node is in communication connection with the terminal device, and is used for executing part of the deep neural network before the partition point; The edge server is in communication connection with the edge node, and is used for executing part of the deep neural network after the partition point; The optimization control device is in communication connection with the terminal device, the edge node and the edge server, and the optimization control device comprises: The task generation module is used for generating a continuous deep neural network reasoning task stream based on the business data collected by the terminal device; The queue modeling module is used for establishing a queue model including a terminal computing queue, a transmission queue and a server computing queue, and determining the execution delay of each task in the terminal computing stage, the transmission stage and the server computing stage according to the arrival time of each task, the queue state and the execution delay parameters of each network layer on the terminal device side and the edge server; The partition optimization modeling module is used for jointly determining an exit point set and a partition point set for each reasoning task for the deep neural network with multiple exit points, and establishing a partition optimization problem based on the above queue model and taking the minimum average execution delay of all tasks and meeting the preset completion time threshold and reasoning accuracy requirement as the target; The deep reinforcement learning training module is used for formalizing the partition optimization problem into a Markov decision process, and training a partition scheduling strategy network based on a deep reinforcement learning algorithm to obtain a partition scheduler mapping the system state to exit point and partition point decisions. A partition cooperative reasoning module is configured to input the current detected network bandwidth and the queue state into the partition scheduler in an online cooperative reasoning stage, obtain an exit point and a partition point for a current task, execute a network layer before the partition point at the edge node, execute a network layer after the partition point at the edge server, terminate reasoning and output a reasoning result when network reasoning reaches the exit point, and dynamically adjust the exit point and the partition point of different tasks according to the decision output by the partition scheduler in a continuous task flow.

9. An electronic device comprising a memory, a processor, a communication module, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the edge-end cooperative optimization method for the multi-outlet DNN model according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the edge-end cooperative optimization method for the multi-outlet DNN model according to any one of claims 1 to 7.

Images