Resource-aware deep neural network distributed inference method and system

Abstract

The application discloses a resource perception-based deep neural network distributed inference method and system, when a master control device starts a distributed DNN inference task, a model intelligent agent generates a plurality of candidate model division strategies according to layer configuration information and network bandwidth of a current DNN model; a time delay prediction intelligent agent performs rapid time delay evaluation on the candidate model division strategies, and selects an optimal model division strategy; a work load intelligent agent comprehensively generates a final work load distribution strategy according to the optimal model division strategy; the master control device distributes different work loads to various heterogeneous terminal devices according to the work load distribution strategy, each device performs parallel distributed calculation according to the model division strategy, and finally outputs an inference result. In the training stage, the parameters of the model intelligent agent are updated reversely by using the time sequence difference error generated by the work load intelligent agent, so that the generation of the model division strategy can fully perceive the influence of the work load distribution, thereby continuously improving the overall strategy quality.

Description

Technical Field

[0001] This invention belongs to the field of deep neural networks, specifically relating to a resource-aware deep neural network distributed inference method and system. Background Technology

[0002] In recent years, with the popularization of artificial intelligence technology, deep neural networks (DNNs) have demonstrated outstanding performance in complex tasks such as image recognition, video analysis, and natural language processing. However, DNN-based applications are generally characterized by computational intensity and high resource consumption, posing significant challenges to directly deploying complete DNN models on terminal devices with limited computing power. Therefore, distributed DNN inference technology for terminals has attracted widespread attention—this technology effectively reduces inference latency, power consumption, and enhances user data privacy protection by splitting the DNN model and distributing it across multiple terminal devices for collaborative inference.

[0003] Distributed DNN inference comprises two key phases: model partitioning and workload allocation. Model partitioning refers to the process of breaking down the DNN model into multiple model segments; workload allocation involves distributing these model segments to various terminal devices, allowing each device to collaboratively complete the inference computation. Traditional model partitioning methods divide the model layer by layer and transmit intermediate feature maps from one device to another, resulting in significant communication overhead. To address this issue, recent research has focused on partitioning feature maps within each layer, allowing parallel execution across multiple devices. These methods are primarily categorized into two main types: transmission-based methods and redundancy-based methods.

[0004] However, existing distributed DNN inference methods lack comprehensive awareness of environmental resources in their model partitioning strategies, making it difficult to consistently achieve acceleration effects under different network bandwidths, layer configurations, and device configurations. Transmission-based methods experience a significant increase in cross-device transmission overhead of overlapping feature maps in low-bandwidth environments, becoming a communication bottleneck that restricts inference acceleration. Redundancy-based methods experience a sharp increase in redundant computation in layers with large amounts of overlapping feature map data, resulting in a significant decrease in acceleration. Furthermore, as the number of devices participating in inference increases, the amount of redundant computation that each device must bear further increases, leading to an overall increase in latency. Therefore, how to dynamically select the optimal model partitioning strategy based on network bandwidth, layer configuration, and device configuration is a pressing problem to be solved in the field of distributed DNN inference.

[0005] Existing workload allocation methods struggle to balance adaptability to dynamic environments with low operational overhead. Most methods optimize only for static network bandwidth, failing to adjust allocation strategies promptly when network bandwidth changes dynamically, making them ill-suited for network fluctuations in real-world terminal environments. While some research can address dynamic network environments, their algorithms are highly complex, requiring tens of seconds or even minutes of additional time to regenerate allocation strategies after each bandwidth change. This severely undermines the low-latency response requirements of real-time terminal inference, hindering the application of distributed inference in practical dynamic scenarios. Summary of the Invention

[0006] To address the shortcomings of existing distributed DNN inference methods, such as insufficient consideration of network bandwidth, layer configuration, and device configuration in model partitioning strategies leading to poor acceleration, and the difficulty of balancing dynamic environment adaptability and low operating overhead in existing workload allocation methods, this paper proposes a resource-aware deep neural network distributed inference method and system. This method aims to achieve rapid and accurate evaluation of model partitioning and workload allocation strategies by using a latency prediction agent to comprehensively model layer configuration, dynamic network bandwidth, and heterogeneous devices. Furthermore, by decomposing the high-dimensional decision space through a multi-agent deep reinforcement learning mechanism, the generation of model partitioning and workload allocation strategies can fully perceive dynamic network and heterogeneous device resources, significantly reducing the operating overhead of strategy replanning in dynamic environments while ensuring inference acceleration.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A resource-aware deep neural network-based distributed inference method includes:

[0009] Step S1: Obtain the image to be inferred and the current environmental resource information, wherein the current environmental resource information includes a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network And the layer configuration sequence of the DNN model used for inference on the image to be inferred. ,in, , Indicates the total number of devices. , Indicates the first The device and the first Current network bandwidth between devices , Indicates the first Layer configuration, , This represents the total number of layers in the DNN model;

[0010] Step S2: Configure the layer sequence and dynamic network bandwidth vector Inputting the data into the trained model agent generates a set of n candidate model partitioning strategies. In this context, each candidate model partitioning strategy is used to determine whether each layer of the DNN model should be processed in a transmission-based or redundant computation-based manner to handle overlapping feature maps between devices. The overlapping feature maps are data regions that need to be repeatedly processed at the device boundaries due to the receptive field of the convolution kernel when the feature maps are split into rows within the layer and distributed to different devices for parallel computation.

[0011] Step S3: Divide the n candidate models into a strategy and a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network Layer configuration sequence and initial workload allocation strategy The input is fed into the latency prediction agent, which performs inference latency prediction on each candidate model partitioning strategy, obtains the predicted latency corresponding to each candidate model partitioning strategy, and selects the strategy with the minimum predicted latency as the optimal model partitioning strategy. Among them, the initial workload allocation strategy Generated based on the computing power of each device;

[0012] Step S4: Apply the optimal model partitioning strategy heterogeneous equipment collection The computing power of each device and the dynamic network bandwidth vector The input is fed into the trained workload agent, which then generates the final workload allocation strategy. The workload allocation strategy Determine the number of rows of images to be inferred assigned to each device;

[0013] Step S5: Partitioning strategy based on the optimal model and the final workload allocation strategy The image to be inferred is distributed according to the final workload allocation strategy. After being segmented, they are distributed to heterogeneous device sets. Each device in the process is partitioned according to the optimal model strategy. At each layer, an adaptive selection is made between transmission-based or redundancy-based processing methods. Each device executes the computation of each layer of the DNN model in parallel, thereby outputting the final inference result.

[0014] Furthermore, the layer configuration for each layer includes the feature map height, feature map width, convolution kernel size, number of input channels, number of output channels, stride, and padding size for that layer.

[0015] Furthermore, the method also includes jointly training the model agent and the workload agent through a dual-agent co-evolutionary mechanism, wherein the model agent and the workload agent employ a deep reinforcement learning framework, and the training process includes:

[0016] Step T1: Obtain the training dataset, which includes multiple training images and corresponding environmental resource information;

[0017] Step T2: Initialize the policy parameters of the model agent. Network weights of workload agents ;

[0018] Step T3: Select the first [item] from the training dataset. The training image is the current training image, and the current state is denoted as . The current training image and the corresponding current environmental resource information are input into the model agent, which generates a set of candidate model partitioning strategies, and the latency prediction agent selects the optimal model partitioning strategy from them.

[0019] Step T4: Input the optimal model partitioning strategy and the corresponding current environmental resource information into the workload agent. The workload agent then determines the appropriate partitioning strategy based on its current state. Generate the current workload allocation strategy;

[0020] Step T5: Based on the optimal model partitioning strategy and workload allocation strategy, perform a distributed end-to-end inference task on the current training image to obtain immediate rewards. ;

[0021] Step T6: Construct the next state based on the next training image and its corresponding environmental resource information. The value function is calculated by the value network of the workload agents respectively. and Then, the workload agent calculates the timing difference error. Where γ is the discount factor, and These represent the workload agent's current state. and the next state The value estimate, This indicates that the workload agent is in its current state. Perform the action and transition to the next state. Instant rewards obtained during the process;

[0022] Step T7: The workload agent determines the timing difference error based on the time difference error. Update its own network weight ;

[0023] Step T8: Convert the timing difference error The data is then passed back to the model agent to update its policy parameters. ;

[0024] Step T9: Repeat steps T3 to T8 until the training termination condition is met.

[0025] Furthermore, the model agent decomposes the DNN model's task into L stages of layer-by-layer decision-making. The model agent in the first stage... The state of a layer is defined as follows: ,in, , For the first Layer configuration, This represents the network bandwidth vector between devices. For the front The cumulative transmission delay of the layers is summed with the delay calculated from the overlapping feature maps. For the first The number of feature map rows transmitted or overlapped in the layer; the first Layer action definition ,in, In the DNN model, the first... Layer-to-heterogeneous device collection equipment in The model partitioning decision has a value of 0 indicating that redundant computation is used to process overlapping feature maps, and a value of 1 indicating that transmission is used to process overlapping feature maps; the reward function of the model agent is defined as: ,in, This indicates that the model agent is in a state. Next action They then received a reward.

[0026] Furthermore, during the training phase, the workload allocation decision-making process of the workload agent is modeled as an MDP process, and the state of the workload agent is defined as... That is, the first The states corresponding to the training images, where... Configure each device at the DNN model layer The computational capability vector below, This represents a regression prediction model used for predicting devices. Configure in layer The complete latency of executing this layer. This represents a dynamic network bandwidth vector composed of the current network bandwidth of each adjacent device. This indicates the sequence from the first training image to the... The cumulative system inference latency for the training images; for the current training image... The workload agent outputs a workload allocation strategy based on the training images. Workload distribution strategy Each element in the table represents the number of devices that should be allocated. The number of rows in the training images; the reward function for the workload agent is defined as follows: ,in, This refers to the overall reasoning latency of the system.

[0027] Furthermore, the overall system inference latency is:

[0028] ,

[0029] in,

[0030] The first DNN model represents the... Layers are assigned to devices Inference delay;

[0031] The first DNN model represents the... Layers are assigned to devices The computational delay of the time-overlapping feature map;

[0032] The first DNN model represents the... Layers are assigned to devices Transmission delay of time-overlapping feature maps.

[0033] Furthermore,

[0034] , ,

[0035] Indicates the first Total number of rows in the layer feature map;

[0036] Indicates the first Layer feature maps are assigned to the device The number of rows in the feature map;

[0037] Indicates device In the The number of overlapping feature map rows that need to be calculated in the layer feature map;

[0038] Indicates the first Layer feature maps need to be in the device and equipment The number of data lines transmitted between them;

[0039] This indicates the workload allocation strategy that assigns tasks to devices. The number of rows in the input image;

[0040] The width of the input image;

[0041] Main control device M and equipment Network bandwidth between.

[0042] Furthermore, in step T7, the workload agent determines the timing difference error based on the time difference error. Update its own network weight This can be expressed as a formula:

[0043] ,

[0044] in, For learning rate, and These represent the network weights of the working agents before and after the update, respectively. It is the gradient operator, representing the gradient with respect to the gradient. Differentiate, The value network representing the workload agent is in Next state The value function, i.e. .

[0045] Furthermore, in step T8, the timing difference error is... The data is then passed back to the model agent to update its policy parameters. This can be expressed as a formula:

[0046] ,

[0047] in, and This represents the policy parameters of the model agent before and after the update. For learning rate, It is the gradient operator, representing the gradient with respect to the gradient. Differentiate, This indicates that the model agent, under the policy network, in the model agent's... Layer state Action division by selection model The probability of.

[0048] This invention also provides a resource-aware deep neural network-based distributed inference system, comprising:

[0049] Memory, configured to store computer programs;

[0050] The processor is configured to execute the computer program to implement the resource-aware deep neural network-based distributed inference method as described above.

[0051] The beneficial effects of this invention are:

[0052] 1. This invention proposes a latency prediction agent, which performs comprehensive modeling of layer configuration, dynamic network bandwidth and heterogeneous devices, and completes inference latency prediction by traversing the DNN model in a single pass. Compared with the method of actually running the DNN model on real devices for testing, this invention can quickly and accurately evaluate the inference latency of arbitrary model partitioning and workload allocation strategies in linear time, which greatly reduces the time cost of strategy evaluation and device energy consumption.

[0053] 2. This invention delegates model partitioning and workload allocation to the model agent and workload agent respectively. By decomposing the high-dimensional joint decision space into two relatively independent low-dimensional decision subspaces, the exploration difficulty of deep reinforcement learning is effectively reduced. Compared with the method of merging model partitioning and workload allocation into a single agent decision, this invention significantly improves the efficiency and quality of policy generation.

[0054] 3. The model agent of this invention perceives the layer configuration and dynamic network bandwidth layer by layer, and adaptively selects the optimal processing method based on transmission or redundant computation for each layer. This overcomes the shortcomings of existing methods that use a fixed single partitioning strategy and have large performance fluctuations under different bandwidths and layer configurations, so that distributed inference can stably play an acceleration role in various network bandwidth environments of high, medium and low.

[0055] 4. The workload agent of this invention integrates the computing power of heterogeneous devices and the current network bandwidth to dynamically generate a workload allocation strategy, which can adapt to the dynamic changes in network bandwidth in real time. Compared with the existing methods that only consider static bandwidth, this invention does not need to rerun high-complexity algorithms when the network environment changes, avoiding the extra replanning overhead of tens of seconds or even minutes, and meeting the terminal's real-time inference requirements for low latency response.

[0056] 5. This invention proposes a dual-agent co-evolution mechanism, which uses the temporal difference error generated by the workload agent to update the policy parameters of the model agent. This enables the generation of the model partitioning policy to fully perceive the impact of workload allocation. This co-training mechanism allows the two agents to promote each other and optimize together. Compared with the scheme of independent training of model partitioning and workload allocation, it can further improve the overall policy quality and thus obtain lower system inference latency.

[0057] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0059] Figure 1 This is a schematic diagram of DNN distributed inference;

[0060] Figure 2 This is a comparison between existing DNN distribution inference methods and the method of this invention;

[0061] Figure 3 These are the performance results of existing DNN distributed inference methods under different resource environments;

[0062] Figure 4 This is a schematic diagram of the resource-aware DNN distributed inference method framework of the present invention. Detailed Implementation

[0063] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0064] Figure 1 This is a schematic diagram of distributed inference in a DNN. For example... Figure 1 As shown, in Figure 1 The upper part of the model uses an image of a cat as input, which is then fed into the DNN model for inference. After model partitioning, the complete DNN model is decomposed into three consecutive model segments: Model Segment 1, Model Segment 2, and Model Segment 3. Figure 1 The lower half illustrates the workload allocation phase: Model Segment 1, Model Segment 2, and Model Segment 3 are allocated to three heterogeneous terminal devices: Device 1 (smart camera terminal), Device 2 (desktop computer), and Device 3 (smartphone), respectively. After Device 1 completes the inference calculation of Model Segment 1, it transmits the intermediate feature map to Device 2, as indicated by the dashed arrow. Device 2 then transmits the intermediate feature map to Device 3 after completing the inference of Model Segment 2. Device 3 completes the final inference and outputs the result "cat," with the dashed arrow indicating the return path of the final result. This demonstrates the complete process of multi-device collaboration in completing distributed inference.

[0065] Figure 2 This is a comparison between existing DNN distributed inference methods and the method of this invention.

[0066] Figure 2 Figure (a) shows the traditional method. In this method, the complete DNN model is divided into several model segments after model partitioning, and then distributed to device A and device B respectively through workload allocation. Intermediate feature maps are passed between devices in layer order to illustrate the basic principle of this method of layer partitioning and layer-by-layer transmission.

[0067] Figure 2 Figure (b) illustrates the working principle of the CoEdge method. In this method, after the feature map is divided within a layer, it is assigned to device A and device B to process corresponding regions. The overlapping feature map at the boundary (the overlapping feature map is the data region that needs to be processed repeatedly at the device boundary due to the receptive field of the convolution kernel when the feature map generated by each layer of the DNN model is divided into rows within the layer and distributed to different devices for parallel computing) is transmitted between the devices through the network. The figure uses a numerical matrix to illustrate the data content and transmission process of the overlapping area, and the arrows indicate the transmission direction.

[0068] Figure 2 Figure (c) illustrates the working principle of the Alternative Optimization Based Fluidity (AOFL) method. In this method, feature maps are also partitioned within the layer and assigned to devices A and B. However, both devices A and B perform redundancy computation (i.e., computation of overlapping regions) locally to obtain feature maps of overlapping regions, eliminating the need for cross-device transmission. The figure uses a numerical matrix to represent the data range of local redundancy computation for each device.

[0069] Figure 2 Figure (d) illustrates the overall framework of the method of this invention. The input feature map is processed by the model agent to generate multiple candidate model partitioning strategies. A latency agent (i.e., a "latency prediction agent") performs rapid latency evaluation on each candidate strategy and selects the optimal strategy. The optimal strategy is then passed to the workload agent, which integrates the computing and network resources of the current device cluster to generate the final workload allocation strategy. The loss value generated by the workload agent is used to update the parameters of the model agent. Arrows in the figure indicate the data flow and parameter update paths between the modules.

[0070] Figure 3 The figure shows the experimental results comparing the acceleration effects of two existing methods under different environmental conditions. It contains three sub-figures, labeled as (a), (b) and (c).

[0071] Figure 3Figure (a) shows the speedup ratio curves of the transmission-based method (CoEdge) under different network bandwidths (10Mbps, 4Mbps, 1Mbps) as a function of the number of devices. The horizontal axis represents the number of devices (values ​​from 2 to 8), and three broken lines show the trend of speedup ratio under different bandwidth conditions. The figure illustrates the limitation of this method's speedup effect in low-bandwidth environments.

[0072] Figure 3 Figure (b) shows the speedup ratio and the change in the amount of overlapping feature maps for each convolutional layer (conv1 to conv13) under different numbers of devices (2, 4, 6, and 8 devices) and overlapping feature map data. The horizontal axis represents the layer name, and multiple broken lines show the changes in speedup ratio and the distribution of overlapping feature map data under different numbers of devices. It can be seen from the figure that the method involves greater redundant computation in layers with a larger amount of overlapping feature maps, and its speedup effect is not significant.

[0073] Figure 3 Figure (c) shows the percentage of redundant computation time (i.e., the proportion of total inference latency) for the Alternative Optimal Computation (AOFL) method under different numbers of devices (2, 4, 8, and 12 devices). The horizontal axis represents the number of devices, and multiple broken lines illustrate the increasing trend of redundant computation time overhead as the number of devices increases. AOFL-6, AOFL-9, and AOFL-12 in the figure represent the number of fusion layers (i.e., the number of consecutive layers processed in one batch) when using the redundancy-based computation method, which are 6, 9, and 12 layers respectively. This figure illustrates the limitation of the method's overall performance degradation as the number of devices increases.

[0074] Figure 4 This is a schematic diagram of a resource-aware deep neural network-based distributed inference method. (For example...) Figure 4As shown, the overall framework of this method consists of three core modules: a latency prediction agent module, a model agent module, and a workload agent module, as well as a dual-agent co-evolution module that runs through the inference and training phases. When the master device (i.e., the device that initiates the collaborative inference request, coordinates the workload allocation of each device, and aggregates the final inference results, and is also one of the devices in the heterogeneous device set) starts the distributed DNN inference task, the model agent generates several candidate model partitioning strategies based on the current DNN model's layer configuration information and network bandwidth; the latency prediction agent performs a rapid latency evaluation on each candidate model partitioning strategy and selects the optimal model partitioning strategy with the minimum prediction latency; the workload agent integrates the optimal model partitioning strategy, the computing power of the heterogeneous devices, and the current network bandwidth to generate the final workload allocation strategy; the master device allocates different workloads to each heterogeneous terminal device according to the workload allocation strategy, and each device performs convolution, pooling, and other operations in parallel according to the model partitioning strategy, and aggregates the feature maps to the device with the richest resources (e.g., the device with the strongest computing power), which then performs fully connected layer computation and outputs the final inference result. Figure 4 The lower center area illustrates the distributed inference execution phase. This diagram shows three heterogeneous terminal devices (represented by different icons) executing inference in parallel after receiving the workload allocation strategy, arranged in three rows. Each row corresponds to one device, and each device sequentially processes the allocated model segment feature map (represented by stacked rectangles). Upward arrows indicate the direction of feature map transmission and convergence between adjacent devices, demonstrating the collaborative inference process where each device performs convolutional layer calculations in parallel and converges intermediate feature maps to the device with the most resources. After inference is complete, the right side displays the output probability vector of the final classification result in a numerical list (0.85, 0.03, 0.04, 0.08), representing the complete output of the inference task.

[0075] Specifically, the resource-aware deep neural network distributed reasoning method may include the following steps:

[0076] Step S1: Obtain the image to be inferred and the current environmental resource information, wherein the current environmental resource information includes a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network And the layer configuration sequence of the DNN model used for inference on the image to be inferred. ,in, , Indicates the total number of devices. , Indicates the first The device and the first Current network bandwidth between devices , Indicates the first Layer configuration, , This indicates the total number of layers in the DNN model. The configuration of each layer includes the feature map height, feature map width, kernel size, number of input channels, number of output channels, stride, and padding size.

[0077] Step S2: Configure the layer sequence and dynamic network bandwidth vector Inputting the data into the trained model agent generates a set of n candidate model partitioning strategies. The partitioning strategy for each candidate model ( , This is used to determine whether each layer of the DNN model processes overlapping feature maps between devices using a transmission-based approach or a redundant computation approach;

[0078] Step S3: Divide the n candidate models into a strategy and a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network Layer configuration sequence and initial workload allocation strategy The input is fed into the latency prediction agent, which performs inference latency prediction on each candidate model partitioning strategy, obtains the predicted latency corresponding to each candidate model partitioning strategy, and selects the strategy with the minimum predicted latency as the optimal model partitioning strategy. Among them, the initial workload allocation strategy The inference latency is generated based on the computing power of each device. It represents the time that the DNN model takes to process the input data and output the results on a specific hardware device. It is the core indicator for measuring the performance of a distributed inference system.

[0079] Step S4: Apply the optimal model partitioning strategy heterogeneous equipment collection The computing power of each device and the dynamic network bandwidth vector The input is fed into the trained workload agent, which then generates the final workload allocation strategy. The workload allocation strategy Determine the number of rows of images to be inferred assigned to each device;

[0080] Step S5: Partitioning strategy based on the optimal model and the final workload allocation strategy The image to be inferred is distributed according to the final workload allocation strategy. After being segmented, they are distributed to heterogeneous device sets. Each device in the process is partitioned according to the optimal model strategy. At each layer, an adaptive selection is made between transmission-based or redundancy-based processing methods. Each device executes the computation of each layer of the DNN model in parallel, thereby outputting the final inference result.

[0081] The method further includes jointly training the model agent and the workload agent through a dual-agent co-evolution mechanism, wherein the model agent and the workload agent employ a deep reinforcement learning framework, and the training process includes:

[0082] Step T1: Obtain the training dataset, which includes multiple training images and corresponding environmental resource information;

[0083] Step T2: Initialize the policy parameters of the model agent. Network weights of workload agents ;

[0084] Step T3: Select the first [item] from the training dataset. The training image is the current training image, and the current state is denoted as . The current training image and the corresponding current environmental resource information are input into the model agent, which generates a set of candidate model partitioning strategies, and the latency prediction agent selects the optimal model partitioning strategy from them.

[0085] Step T4: Input the optimal model partitioning strategy and the corresponding current environmental resource information into the workload agent. The workload agent generates the current workload allocation strategy based on its current state.

[0086] Step T5: Based on the optimal model partitioning strategy and workload allocation strategy, perform a distributed end-to-end inference task on the current training image to obtain immediate rewards. ( );

[0087] Step T6: Construct the next state based on the next training image and its corresponding environmental resource information. The value function is calculated by the value network of the workload agents respectively. and Then, the workload agent calculates the timing difference error. Where γ is the discount factor, and These represent the workload agent's current state. and the next state The value estimate, This indicates that the workload agent is in its current state. Perform the action and transition to the next state. Instant rewards obtained during the process;

[0088] Step T7: The workload agent determines the timing difference error based on the time difference error. Update its own network weight ;

[0089] Step T8: Convert the timing difference error The data is then passed back to the model agent to update its policy parameters. ;

[0090] Step T9: Repeat step T3 (when returning to step T3, the first step will be repeated). (Using the current image as T8) until the training termination condition is met.

[0091] Among them, it is responsible for generating the model partitioning strategy, which determines whether each layer of the DNN model adopts a transmission-based or redundant computation-based processing method between devices.

[0092] The model agent decomposes the DNN model's model partitioning task into L stages of layer-by-layer decision-making (the DNN model has L layers in total) to reduce the agent's exploration space. In one embodiment of the invention, the model agent employs a deep reinforcement learning (DRL) framework, with its deep learning component being a lightweight four-layer perceptron, adapted to the resource-constrained characteristics of terminal devices.

[0093] The model agent in the first The state of a layer is defined as follows: ,in, , For the first Layer configuration, Network bandwidth vector between devices (i.e.) ), For the front The cumulative transmission delay of the layers is summed with the delay calculated from the overlapping feature map (initially 0). For the first The number of feature map rows in layer transmission or overlap. and All are determined by the state and decisions of the previous layer, satisfying the Markov property.

[0094] Model agent Layer action definition ,in, In the DNN model, the first... Layer-to-heterogeneous device collection equipment in The model partitioning decision has a value of 0, which indicates that redundant computation is used to process overlapping feature maps, and a value of 1 indicates that transmission is used to process overlapping feature maps. The action space is discrete.

[0095] The model agent in the state Next action Later received a reward The reward function of the model agent can be defined as: Maximizing the long-term cumulative reward of the model agent is equivalent to minimizing the cumulative computational latency during the DNN inference process.

[0096] The workload agent module is responsible for receiving the optimal model partitioning strategy output by the model agent, and, by combining the computing power and network bandwidth of the current device cluster, generating the final workload allocation strategy to determine the allocation scheme of each row of the input image among the devices. In one embodiment of the present invention, the workload agent also adopts a deep reinforcement learning framework, and its deep learning component is a lightweight four-layer perceptron.

[0097] During the training phase, the workload allocation decision-making process of the workload agent is modeled as a Markov Decision Process (MDP). The state of the workload agent is defined as... (The states of the model agent and the workload agent are indicated by subscripts) and To distinguish), that is, the first The states corresponding to the training images, where... Configure each device at the DNN model layer The computational capability vector below, This represents a regression prediction model used for predicting devices. Configure in layer The complete latency of executing this layer. This represents the network bandwidth of each adjacent device (i.e., the first...). The dynamic network bandwidth vector is composed of the network bandwidth when training images are input into the DNN model. This indicates the sequence from the first training image to the... The cumulative system inference delay of the training images (initially 0) satisfies the Markov property.

[0098] For the current number Zhang training images, the workload agent based on the current state It can output workload distribution strategies. This refers to the action space of the working agent. Workload allocation strategy. Each element in the table represents the number of devices that should be allocated. The number of rows in the training images represents the continuous action space. This invention employs a softmax strategy to map action values ​​to integers between 0 and H, where H is the height of the input image.

[0099] The reward function of the workload agent is defined as follows: (Right now ),in, The overall system inference latency is considered. Maximizing the reward for the workload agent is equivalent to minimizing the overall system inference latency.

[0100] The overall inference latency of the system is determined by the latency prediction agent. The latency prediction agent is responsible for quickly and accurately evaluating the inference latency of the given model partitioning strategy and workload allocation strategy, providing a latency basis for subsequent strategy selection, while avoiding the high cost and high energy consumption caused by directly running the DNN model for testing on resource-constrained terminal devices.

[0101] The input to the latency prediction agent consists of five parts: n candidate model partitioning strategies (for each input image, there are n corresponding candidate model partitioning strategies), a set of heterogeneous devices, and a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network Layer configuration sequence and initial workload allocation strategy Among them, the initial workload allocation strategy The number of image rows is generated based on the computing power of each device. Devices with stronger computing power are allocated more image rows initially, while devices with weaker computing power are allocated fewer image rows initially.

[0102] In this invention, the overall system inference latency is modeled as the sum of the latencies of each individual layer. Each individual layer latency is further divided into: device inference latency, overlapping feature map computation latency (based on redundancy computation), and overlapping feature map transmission latency (based on transmission).

[0103] For a given layer configuration and device, the computation latency for a feature map is proportional to the size of the feature map; therefore, the device inference latency is... (i.e., the first DNN model) Layers are assigned to devices The inference delay at that time can be expressed as:

[0104] ,

[0105] in, Indicates the first Total number of rows in the layer feature map; Indicates the first Layer feature maps are assigned to the device The number of rows in the feature map; This represents a regression prediction model used for predicting devices. Configure in layer The complete latency of executing this layer.

[0106] Because the feature map partitioning within a layer creates overlapping regions at device boundaries, each device needs to perform additional calculations on the overlapping rows. Therefore, the calculation delay of the overlapping feature map is increased. (i.e., the first DNN model) Layers are assigned to devices The computation delay of the time-overlapping feature map can be expressed as:

[0107] ,

[0108] in, Indicates device In the The number of overlapping feature map rows that need to be calculated in the layer feature map is determined by the model partitioning strategy and layer configuration.

[0109] Overlapping feature map transmission delay (The first DNN model) Layers are assigned to devices The transmission delay of the time-overlapping feature map is divided into two cases depending on the layer number:

[0110]

[0111] in, Indicates the first Layer feature maps need to be in the device and equipment The number of data lines transmitted between them; This indicates the workload allocation strategy that assigns tasks to devices. The number of rows in the input image; The width of the input image; Main control device M and equipment Network bandwidth between.

[0112] It should be noted that, in this invention, the workload allocation strategy must satisfy the following constraints: , and is an integer, where H represents the height of the input image.

[0113] For all layers of the DNN model, the sum of the maximum values ​​of the completion latencies of all devices in each layer is taken as the overall system inference latency. That is, the overall system inference latency is:

[0114] .

[0115] The delay prediction agent F completes the delay prediction by traversing the DNN model once, based on the above formula. The time complexity is O(D×L), where D is the number of devices and L is the number of layers in the DNN model, which achieves fast evaluation in linear time.

[0116] As mentioned above, in this invention, the dual-agent co-evolution module (hereinafter referred to as the "co-evolution module") runs through the inference and training phases, and is responsible for coordinating the collaborative decision-making and joint training of the model agent and the workload agent.

[0117] During the inference phase, the co-evolutionary module automatically generates an initial workload allocation strategy based on the computing power of each device. (The more powerful the device, the more tasks it is assigned); Next, the co-evolution module traverses the set of candidate model partitioning strategies output by the model agent. Candidate strategies are extracted sequentially. Then, the co-evolution module will Together with current network bandwidth, computing power information of heterogeneous devices, layer configuration, and initial workload allocation strategy Input the latency prediction agent together to obtain the predicted latency of the candidate policy; then, select the optimal model partitioning policy from all candidate policies with the minimum predicted latency. As shown in the following formula:

[0118] ,

[0119] Optimal model partitioning strategy The input matrix is ​​constructed together with information on heterogeneous devices and the current network bandwidth, and then passed to the workload agent to generate the final workload allocation strategy.

[0120] During the training phase, the workload agent and the model agent employ a collaborative training mechanism, updating together. The workload agent first calculates the temporal difference error. :

[0121] ,

[0122] Where γ is the discount factor. and These represent the workload agent's current state. and the next state The value estimate, The immediate reward for the next state is represented by the temporal difference error, a key metric used in deep reinforcement learning training to measure the quality of the current policy. It is defined as the difference between the sum of the immediate reward and the discounted next state value estimate and the current state value estimate. In this invention, the temporal difference error generated by the workload agent is simultaneously used to update the parameters of the model agent, achieving co-evolution of the two agents.

[0123] The workload agent based on the timing difference error Update the network weights using the mean squared error loss function. This can be expressed as a formula:

[0124] ,

[0125] in, For learning rate, and These represent the network weights of the working agents before and after the update, respectively. It is the gradient operator, representing the gradient with respect to the gradient. (Right now Differentiate, The value network representing the workload agent is in (Right now ) under state The output value function, i.e. .

[0126] Subsequently, the timing difference error generated by the workload agent The data is then passed back to the model agent to update its policy parameters. This can be expressed as a formula:

[0127] ,

[0128] in, and This represents the policy parameters of the model agent before and after the update. For learning rate, It is the gradient operator, representing the gradient with respect to the gradient. Differentiate, This indicates that the model agent, under the policy network, in the model agent's... Layer state Action division by selection model The probability of.

[0129] The aforementioned collaborative training mechanism enables the model agent to perceive the impact of workload allocation when generating model partitioning strategies. The two agents promote each other and continuously improve the overall strategy quality.

[0130] This invention supports not only Convolutional Neural Networks (CNNs) but also distributed inference acceleration for general DNN models, such as the Transformer model. For multi-head self-attention layers, this invention breaks down the multi-head self-attention inference task into several independent single-head self-attention inference subtasks, which are executed in parallel by each device and then merged. For feedforward network layers, this invention divides activation values ​​into blocks according to the number of devices, with each device computing its assigned activation block in parallel without communication between devices. After computation, the results of each block are merged to obtain the complete feedforward network layer output.

[0131] This invention also provides a resource-aware deep neural network-based distributed inference system, comprising:

[0132] Memory, configured to store computer programs;

[0133] The processor is configured to execute the computer program to implement the resource-aware deep neural network-based distributed inference method as described above.

[0134] The following specific examples illustrate the complete execution process of this invention.

[0135] Assume the heterogeneous set of terminal devices participating in distributed inference consists of three devices: ={Device 1 (smart camera, relatively weak computing power), Device 2 (desktop computer, relatively strong computing power, main control device), Device 3 (smartphone, moderate computing power)}, the DNN model to be inferred is ResNet-50, the input image size is 224×224, and the current network bandwidth vector... Both the model agent and the workload agent have been trained, and their policy parameters and network weights have been fixed, making them suitable for online decision-making in current distributed inference tasks.

[0136] Initialize system parameters: The device set contains three heterogeneous devices, and the computing power of each device is determined by a regression model. Pre-determined; network bandwidth vector Obtained through real-time measurements; ResNet-50 contains approximately 50 operator layers, layer configuration sequence It covers parameters such as kernel size, number of channels, and stride for each layer, with an input image height of H=224.

[0137] Constructing a latency prediction agent: Taking ResNet-50 layer 1 (3×3 convolutional layer, 3 input channels, 64 output channels) as an example, let the current workload allocation strategy (initial allocation strategy) be: allocate 80 rows out of the 224 rows of images to device 1, 96 rows to device 2, and 48 rows to device 3 (i.e., ... , , The model partitioning strategy in layer 1 is based on the transmission method. ).

[0138] Device inference latency of device 1: ;

[0139] Transmission delay of overlapping feature map of device 1: ;

[0140] After calculating each delay component for all devices, the maximum value of the completion delay of each device in the first layer is taken and accumulated to the overall system delay. Similarly, the time delay prediction is completed for all 50 layers of ResNet-50. The entire process has a time complexity of O(3 × 50) = O(150), which is completed in linear time.

[0141] The model agent generates candidate model partitioning strategies layer by layer: Taking the 3rd layer (3×3 convolution, 64 channels) as an example, the current state of the model agent... ,in, This refers to the network bandwidth vector between the current devices. , The cumulative transmission and overlap calculation delay of the first two layers, This represents the number of feature map rows transmitted or overlapped in Layer 2 (i.e., the layer above). The model agent takes the current state as input and outputs the partitioning actions of each device in Layer 3 through a lightweight four-layer perceptron. Due to the current network bandwidth... = 10Mbps (medium bandwidth). After training, the agent tends to choose the transmission-based method at this layer to avoid redundant computational overhead. The model agent makes decisions layer by layer on all 50 layers of ResNet-50, generating a total of n=5 candidate model partitioning strategies. .

[0142] The dual-agent co-evolutionary module performs time-latency evaluation on five candidate policies: based on the candidate policies For example, the co-evolution module combines it with the initial workload allocation strategy. heterogeneous equipment collection The computing power and dynamic network bandwidth vector of each device in the network and layer configuration The latency prediction agent was input together, and the predicted latency was 47.3ms. Similarly, the other four candidate policies were evaluated separately, and the predicted latency was 45.8ms, 49.1ms, 46.5ms, and 48.2ms, respectively. The policy with the lowest latency was selected according to the screening formula. (Prediction latency 45.8ms) is used as the optimal model partitioning strategy and passed to the workload agent. The workload agent uses... The computing power vector of each device and the network bandwidth between adjacent devices are used as state inputs. The final workload allocation scheme is output through a softmax strategy. , , (The more powerful device 2 will be allocated more rows), and the overall inference latency is expected to be T = 43.2ms.

[0143] According to the output strategy, the master control device (i.e., device 2) sends 64 rows of image data to device 1, 48 rows of image data to device 3, and retains 112 rows itself. Based on the optimal model partitioning strategy, the three devices adaptively select either transmission-based or redundancy-based processing methods at each layer to perform convolution, pooling, and other calculations in parallel. After each layer's calculation, the intermediate feature maps are aggregated to device 2. After completing the fully connected layer calculation, device 2 outputs the final classification probability vector (e.g., cat: 0.85, dog: 0.03, other categories: 0.12), completing the overall inference task. The actual inference latency and the error between the predicted value and the actual inference value are approximately 3.1%.

[0144] In summary, this invention uses multi-agent deep reinforcement learning as its core framework. It achieves rapid policy evaluation in linear time through a time-delay prediction agent, decomposes the high-dimensional decision space through a model agent and a workload agent, and achieves joint training and mutual promotion through a dual-agent co-evolution mechanism. While ensuring the acceleration of inference, it significantly reduces the operational overhead of policy replanning in dynamic environments, and has good engineering application value.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A resource-aware deep neural network-based distributed inference method, characterized in that, include: Step S1: Obtain the image to be inferred and the current environmental resource information, wherein the current environmental resource information includes a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network And the layer configuration sequence of the DNN model used for inference on the image to be inferred. ,in, , Indicates the total number of devices. , Indicates the first The device and the first Current network bandwidth between devices , Indicates the first Layer configuration, , This represents the total number of layers in the DNN model; Step S2: Configure the layer sequence and dynamic network bandwidth vector Inputting the data into the trained model agent generates a set of n candidate model partitioning strategies. In this context, each candidate model partitioning strategy is used to determine whether each layer of the DNN model should be processed in a transmission-based or redundant computation-based manner to handle overlapping feature maps between devices. The overlapping feature maps are data regions that need to be repeatedly processed at the device boundaries due to the receptive field of the convolution kernel when the feature maps are split into rows within the layer and distributed to different devices for parallel computation. Step S3: Divide the n candidate models into a strategy and a set of heterogeneous devices. The computing power and dynamic network bandwidth vector of each device in the network Layer configuration sequence and initial workload allocation strategy The input is fed into the latency prediction agent, which performs inference latency prediction on each candidate model partitioning strategy, obtains the predicted latency corresponding to each candidate model partitioning strategy, and selects the strategy with the minimum predicted latency as the optimal model partitioning strategy. Among them, the initial workload allocation strategy Generated based on the computing power of each device; Step S4: Apply the optimal model partitioning strategy heterogeneous equipment collection The computing power of each device and the dynamic network bandwidth vector The input is fed into the trained workload agent, which then generates the final workload allocation strategy. The workload allocation strategy Determine the number of rows of images to be inferred assigned to each device; Step S5: Partitioning strategy based on the optimal model and the final workload allocation strategy The image to be inferred is distributed according to the final workload allocation strategy. After being segmented, they are distributed to heterogeneous device sets. Each device in the process is partitioned according to the optimal model strategy. At each layer, an adaptive selection is made between transmission-based or redundancy-based processing methods. Each device executes the computation of each layer of the DNN model in parallel, thereby outputting the final inference result.

2. The method according to claim 1, characterized in that, The layer configuration for each layer includes the feature map height, feature map width, convolution kernel size, number of input channels, number of output channels, stride, and padding size.

3. The method according to claim 2, characterized in that, The method further includes jointly training the model agent and the workload agent through a dual-agent co-evolution mechanism, wherein the model agent and the workload agent employ a deep reinforcement learning framework, and the training process includes: Step T1: Obtain the training dataset, which includes multiple training images and corresponding environmental resource information; Step T2: Initialize the policy parameters of the model agent. Network weights of workload agents ; Step T3: Select the first [item] from the training dataset. The training image is the current training image, and the current state is denoted as . The current training image and the corresponding current environmental resource information are input into the model agent, which generates a set of candidate model partitioning strategies, and the latency prediction agent selects the optimal model partitioning strategy from them. Step T4: Input the optimal model partitioning strategy and the corresponding current environmental resource information into the workload agent. The workload agent then determines the appropriate partitioning strategy based on its current state. Generate the current workload allocation strategy; Step T5: Based on the optimal model partitioning strategy and workload allocation strategy, perform a distributed end-to-end inference task on the current training image to obtain immediate rewards. ; Step T6: Construct the next state based on the next training image and its corresponding environmental resource information. The value function is calculated by the value network of the workload agents respectively. and Then, the workload agent calculates the timing difference error. Where γ is the discount factor, and These represent the workload agent's current state. and the next state The value estimate, This indicates that the workload agent is in its current state. Perform the action and transition to the next state. Instant rewards obtained during the process; Step T7: The workload agent determines the timing difference error based on the time difference error. Update its own network weight ; Step T8: Convert the timing difference error The data is then passed back to the model agent to update its policy parameters. ; Step T9: Repeat steps T3 to T8 until the training termination condition is met.

4. The method according to claim 3, characterized in that, The model agent decomposes the DNN model's task into L stages of layer-by-layer decision-making. The model agent in the Lth stage... The state of a layer is defined as follows: ,in, , For the first Layer configuration, This represents the network bandwidth vector between devices. For the front The cumulative transmission delay of the layers is summed with the delay calculated from the overlapping feature maps. For the first The number of feature map rows transmitted or overlapped in the layer; the first Layer action definition ,in, In the DNN model, the first... Layer-to-heterogeneous device collection equipment in The model partitioning decision has a value of 0 indicating that redundant computation is used to process overlapping feature maps, and a value of 1 indicating that transmission is used to process overlapping feature maps; the reward function of the model agent is defined as: ,in, This indicates that the model agent is in a state. Next action They then received a reward.

5. The method according to claim 3, characterized in that, During the training phase, the workload allocation decision-making process of the workload agent is modeled as an MDP process, and the state of the workload agent is defined as... That is, the first The states corresponding to the training images, where... Configure each device at the DNN model layer The computational capability vector below, This represents a regression prediction model used for predicting devices. Configure in layer The complete latency of executing this layer. This represents a dynamic network bandwidth vector composed of the current network bandwidth of each adjacent device. This indicates the sequence from the first training image to the... The cumulative system inference latency for the training images; for the current training image... The workload agent outputs a workload allocation strategy based on the training images. Workload distribution strategy Each element in the table represents the number of devices that should be allocated. The number of rows in the training images; the reward function for the workload agent is defined as follows: ,in, This refers to the overall reasoning latency of the system.

6. The method according to claim 5, characterized in that, The overall system inference latency is: ， in, The first DNN model represents the... Layers are assigned to devices Inference delay; The first DNN model represents the... Layers are assigned to devices The computational delay of the time-overlapping feature map; The first DNN model represents the... Layers are assigned to devices Transmission delay of time-overlapping feature maps.

7. The method according to claim 6, characterized in that, ， ， ， Indicates the first Total number of rows in the layer feature map; Indicates the first Layer feature maps are assigned to the device The number of rows in the feature map; Indicates device In the The number of overlapping feature map rows that need to be calculated in the layer feature map; Indicates the first Layer feature maps need to be in the device and equipment The number of data lines transmitted between them; This indicates the workload allocation strategy that assigns tasks to devices. The number of rows in the input image; The width of the input image; Main control device M and equipment Network bandwidth between.

8. The method according to claim 3, characterized in that, In step T7, the workload agent determines the timing difference error based on the time difference error. Update its own network weight This can be expressed as a formula: ， in, For learning rate, and These represent the network weights of the working agents before and after the update, respectively. It is the gradient operator, representing the gradient with respect to the gradient. Differentiate, The value network representing the workload agent is in Next state The value function, i.e. .

9. The method according to claim 3, characterized in that, In step T8, the timing difference error is... The data is then passed back to the model agent to update its policy parameters. This can be expressed as a formula: ， in, and This represents the policy parameters of the model agent before and after the update. For learning rate, It is the gradient operator, representing the gradient with respect to the gradient. Differentiate, This indicates that the model agent, under the policy network, in the model agent's... Layer state Action division by selection model The probability of.

10. A resource-aware deep neural network-based distributed inference system, characterized in that, include: Memory, configured to store computer programs; A processor is configured to execute the computer program to implement the resource-aware deep neural network-based distributed inference method as described in any one of claims 1 to 9.

Images