Feature map block distributed inference task scheduling method, system, and program products that support adaptive aggregation and boundary exchange.

Abstract

This invention discloses a feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange. The method includes: acquiring runtime information, including participating device information, network status, and DNN architecture information; obtaining the optimal aggregation point and boundary exchange strategy through a pre-trained inference model based on the runtime information; and implementing the adaptive aggregation and boundary exchange task scheduling method by executing the obtained optimal aggregation point and boundary exchange strategy. This invention discloses a feature map block-based distributed inference task scheduling method, system, and program product that supports adaptive aggregation and boundary exchange. It can dynamically select the optimal aggregation point and boundary exchange strategy based on device heterogeneity, bandwidth status, and task load during operation, enabling a dynamic balance between inference efficiency and accuracy in a multi-device collaborative environment, and achieving a balance between inference accuracy and latency optimization.

Description

Technical Field

[0001] This invention relates to the fields of distributed edge computing and artificial intelligence inference technology, specifically to a feature map block distributed inference task scheduling method, system, and program product that supports adaptive aggregation and boundary exchange. Background Technology

[0002] Distributed deep neural network inference refers to the process of collaboratively executing deep neural network (DNN) model inference tasks across multiple hardware devices. By sharing computing resources and intermediate feature data among edge nodes, it achieves efficient execution and low-latency response of complex models. Feature map block-based distributed neural network inference can effectively alleviate the performance bottleneck caused by the limited computing power of a single device, enabling intelligent perception and efficient decision-making in resource-constrained environments.

[0003] In edge video analytics scenarios, multiple cameras or terminal devices can process different feature regions of input frames in parallel, achieving overall inference through the aggregation of intermediate results. However, unreasonable feature aggregation points or frequent boundary data exchanges can lead to excessive communication overhead, increased latency, or decreased accuracy, making it difficult to balance real-time performance and accuracy. This results in increased device workload, redundant data processing operations during hardware operation, and communication latency caused by excessive device load. Therefore, distributed inference systems require trade-offs and scheduling between computational load, communication bandwidth, and inference accuracy.

[0004] Feature map block-based adaptive distributed inference aims to reduce the computational burden on a single device and improve overall inference efficiency by dividing the feature map and processing it in parallel across multiple devices to perform the computationally intensive feature extraction part. However, this process still faces the following technical challenges:

[0005] 1. There is significant heterogeneity among edge devices. The computing power, memory and execution speed of different devices vary greatly. If the tasks are not divided evenly, some devices will become performance bottlenecks, thus affecting the overall inference efficiency.

[0006] 2. Edge network bandwidth is greatly affected by environmental interference and load changes, and the transmission rate fluctuates frequently, making communication latency difficult to predict and further increasing the complexity of multi-device synchronization and scheduling.

[0007] 3. Block feature maps have cross-boundary dependencies in convolution calculations. If there is a lack of effective boundary feature exchange and aggregation mechanisms, it will lead to significant accuracy loss.

[0008] Considering the above factors, achieving efficient, stable, and accurate distributed inference in resource-constrained environments becomes extremely challenging. Existing feature map-based distributed neural network inference methods typically employ fixed or heuristic aggregation point settings. Fixed aggregation points merge parallel results at a predetermined layer of the model; heuristic aggregation points can be adjusted to a limited extent based on the status of some devices or task load, but their adaptability and adjustment range are relatively limited, making it difficult to cope with dynamic changes under conditions of large-scale heterogeneous devices and complex edge networks. Furthermore, to maintain feature map continuity and inference accuracy, existing technologies require exchanging boundary information of each feature block during each inference process, whenever encountering a model layer with a sliding window mechanism, to maintain full accuracy. This frequent boundary exchange generates high communication overhead and latency in low-bandwidth or multi-node environments, thus affecting the overall inference efficiency of the system. Summary of the Invention

[0009] This invention overcomes the shortcomings of existing technologies and provides a feature map block distributed inference task scheduling method, system, and program product that supports adaptive aggregation and boundary exchange. It introduces a hierarchical multi-armed learning (MAB) mechanism into the distributed inference framework, which can dynamically select the optimal aggregation point and boundary exchange strategy based on device heterogeneity, bandwidth status, and task load during operation. This is used to achieve a dynamic balance between inference efficiency and accuracy in a multi-device collaborative environment, and to achieve a balance between inference accuracy and latency optimization.

[0010] To achieve the above objectives, the technical solution adopted by this invention is: a feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange, comprising:

[0011] Obtain operational information, including participating device information, network status, and DNN architecture information;

[0012] Based on the operational information, the optimal aggregation point and boundary exchange strategy are obtained through reasoning using a pre-trained reasoning model.

[0013] An adaptive aggregation and boundary exchange task scheduling method is implemented by executing the obtained optimal aggregation point and boundary exchange strategy.

[0014] The reasoning method of the reasoning model includes:

[0015] Based on the acquired runtime information, perform initialization configuration to obtain the initial strategy;

[0016] Based on the initial policy, a hierarchical MAB mechanism is used to train the policy and obtain the training policy. The policy training includes action initialization operation, action selection operation, block ratio update operation, action execution operation and reward update operation.

[0017] During inference execution, the training strategy is adaptively adjusted through network-aware operations (network-aware submodule), and the action selection operation, block ratio update operation, action execution operation and reward update operation in the initial strategy training are reused to adjust the training strategy and obtain the optimal aggregation point and boundary exchange strategy.

[0018] Among them, MAB mechanism is a hierarchical multi-armed gambling machine learning mechanism; DNN is an execution deep neural network model.

[0019] In a preferred embodiment of the present invention, the participating device information includes: the number of participating devices N, the device computing power, and memory information;

[0020] The network status includes: the lowest data transmission rate B under the current network environment. min This is used to initialize the action space for subsequent training.

[0021] DNN architecture information includes: layer information for the feature extraction part of the DNN to be inferred. Further, the layer information includes the number of layers, the type of each layer, and the parameters of each layer.

[0022] In a preferred embodiment of the present invention, the initialization configuration includes the following method:

[0023] The DNN is divided into several operational blocks based on the layer information using the operational block partitioning method, and a block list is obtained, where each element is a layer index list.

[0024] The operation block partitioning method includes the following steps: each time a convolutional layer is encountered, the corresponding operation block is activated, and subsequent non-convolutional layers are grouped into the operation block; the operation block is used as a unit for feature boundary exchange.

[0025] In a preferred embodiment of the present invention, the action initialization method includes:

[0026] Define action A, which consists of two layers. The first layer is the aggregation index (starting from 1). Aggregates before the index are inferred in parallel, while aggregates after the index and the remaining network aggregates except for the feature extraction part are executed locally. The second layer, based on the decision of the first layer, generates a list of binary variables Pv of length v, where each variable P... k Determine whether to swap feature boundaries (padding) in each parallel block k; a value of 1 indicates swapping.

[0027] A two-layer bandit structure is used to store and maintain candidate actions A, including: the first layer is the aggregation point bandit, which records the number of times each aggregation point v is selected and the cumulative reward, so as to globally evaluate the merits of different aggregation points; the second layer is the feature boundary swapping bandit, which serves as a sub-bandit. For each legal aggregation point, an independent sub-action space is maintained to record the action Pv list and reward value of each parallel block under that aggregation point for swapping feature boundaries.

[0028] During action initialization, for each aggregation point agg, a depth-first search (DFS) is performed based on the device's static computing power and the minimum data transfer rate B. min The simulation latency is generated by the FLOPs of the model blocks, and the execution latency is kept below the minimum latency threshold T for the inference task to be executed on a single device. th The actions are included in the sub-action space of the corresponding aggregation point; when the sub-action space is not empty, the aggregation point is considered a valid aggregation point and added to the valid_aggs list.

[0029] And / or, methods for action selection include:

[0030] Layered The strategy selects actions on a two-layer Bandit structure. For an inference request in round t, the exploration probability decays with each training round. The first layer maintains the set of aggregate points valid_aggs and its cumulative reward agg_rewards. The second layer maintains a sub-Bandit for each valid aggregate point v, which includes the set of padding actions actions and the corresponding cumulative reward rewards.

[0031] And / or, the method for updating the block ratio includes: the first round is allocated inversely proportionally based on the static computing power of the equipment, and a periodic adaptive update mechanism is adopted, updating once every M rounds;

[0032] And / or, the methods for performing the action include:

[0033] The controller sends the current block ratio vector and the two-layer action parameters selected by the hierarchical multi-armed gambling machine algorithm (MAB) to the master device; the master device divides the input image in the width or height dimension according to the block ratio, and generates several image sub-blocks after rounding the block length;

[0034] The master device sends the aggregation point v and the exchange padding selection Pv to each slave device. The slave devices then construct the corresponding neural network exit function based on this information and prepare to receive task blocks.

[0035] The master device distributes each image block to the corresponding slave device. Each slave device performs the assigned DNN inference task and returns the resulting feature map results to the master device in blocks.

[0036] After receiving all feature map results, the master device performs feature map aggregation and subsequent local DNN inference, finally calculates the complete inference result, and sends the total inference latency back to the controller. During the offline training phase, the controller further compares the inference output with the real labels to calculate the accuracy and sends it back to the controller.

[0037] In a preferred embodiment of the present invention, the method for updating rewards includes:

[0038] After performing action A, you will receive real-time latency and accuracy or accuracy bonus, or an instant reward. :

[0039] ;in For accuracy rewards, latency rewards , Penalty for low precision (below the threshold Acc) th (for low precision) and It is to obtain the accuracy Acc and delay T in real time. total The value after normalization to the minimum and maximum values ; , β and β are the weight parameters, respectively;

[0040] Lower layer (padding exchange layer) reward update: For the current aggregation point v t The selected padding swap action Update its average reward using the exponential moving average (EMA) method:

[0041] Updated Where α is the EMA smoothing coefficient, used to balance the influence of new and old rewards, and to record the number of times the action is sampled;

[0042] Upper-layer (aggregation point layer) reward caching and updating: Each aggregation point maintains a cache list to temporarily store the most recent instant reward values ​​for that aggregation point; when the number of cached samples reaches a threshold, the cache average is calculated as the representative reward for the aggregation point. ,in The number of samples in the cache list of aggregation point v. A cache list for aggregation point v;

[0043] Update the aggregation point layer rewards using the EMA smoothing strategy: Update Then clear the cache and prepare for the next update;

[0044] Precision reward dictionary construction: The system maintains a precision reward record table for each action pair, recording the precision-related rewards for the current action. The data is stored in the precision reward dictionary in chronological order; by calculating the historical average value of the precision reward dictionary, a global action-precision reward mapping is obtained for subsequent strategy evaluation and online adjustment.

[0045] In a preferred embodiment of the present invention, policy adjustment includes: using trained MAB states and data, reusing action selection operations, block ratio update operations, action execution operations, and reward update operations from policy training to implement policy adjustment, including the following steps:

[0046] The network-aware operation collects the network transmission rate in real time before the action selection is performed. When the bandwidth is detected to be continuously decreasing or increasing for more than the preset duration, the parameter adaptive adjustment mechanism is triggered.

[0047] Select two layers of actions based on the reused action selection operation; update the optimal partitioning ratio based on the reused block ratio update operation; execute the operation through the reused actions and obtain the actual execution latency;

[0048] The comprehensive reward is calculated based on the latency of real-time feedback and the global action-precision reward mapping table generated during the offline phase. When the adjustment mechanism is in effect, a communication penalty is added. The next round of inference requests is processed, and online adjustments are continuously performed.

[0049] Specifically, the EMA smoothing parameter α and the exploration probability parameter are used. To enable lightweight online fine-tuning, this module additionally introduces network-aware operations to dynamically monitor bandwidth fluctuations during runtime.

[0050] In a preferred embodiment of the present invention, the method of the parameter adaptive adjustment mechanism includes:

[0051] Temporarily increase the EMA smoothing coefficient α to enhance the algorithm's response speed to real-time feedback;

[0052] A communication latency penalty term is added to the reward function to balance computational latency and communication cost;

[0053] The temporarily increased EMA smoothing coefficient α and the communication delay penalty term added to the reward function are gradually restored to the default value of the EMA smoothing coefficient α through the time decay factor after no anomalies are detected for a continuous period of time, and the added communication delay penalty term is automatically removed.

[0054] In a preferred embodiment of the present invention, a feature map block-based distributed inference task scheduling system supporting adaptive aggregation and boundary exchange includes:

[0055] The interconnected system initialization configuration module, offline policy training module, and online policy fine-tuning module;

[0056] The system initialization configuration module is used to collect device performance, bandwidth conditions and model structure information to complete the execution environment initialization;

[0057] The offline policy training module is used for offline training of the initial policy based on the hierarchical MAB algorithm. It includes action initialization operation, action selection operation, block ratio update operation, action execution operation and reward update operation.

[0058] The online policy fine-tuning module uses the trained policy and adaptively adjusts it through network perception operations (network perception submodule) during actual inference execution. It also reuses action selection operations, block ratio update operations, action execution operations, and reward update operations to fine-tune the policy.

[0059] The steps to implement a feature map block-based distributed inference task scheduling method that supports adaptive aggregation and boundary exchange.

[0060] In a preferred embodiment of the present invention, a computer program product supporting feature map block distributed inference task scheduling with adaptive aggregation and boundary exchange includes a computer program / instruction that, when executed by a processor, implements the steps of a feature map block distributed inference task scheduling method supporting adaptive aggregation and boundary exchange.

[0061] This invention addresses the deficiencies in the technical background, and the beneficial technical effects of this invention are:

[0062] A feature map block distributed inference task scheduling method, system, and program product that supports adaptive aggregation and boundary exchange introduces a hierarchical multi-armed learning (MAB) mechanism into the distributed inference framework. During operation, it can dynamically select the optimal aggregation point and boundary exchange strategy based on device heterogeneity, bandwidth status, and task load. This is used to achieve a dynamic balance between inference efficiency and accuracy in a multi-device collaborative environment, and to achieve a balance between inference accuracy and latency optimization. Attached Figure Description

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0064] Figure 1 This is a flowchart illustrating the feature map block distributed inference task scheduling technology supporting adaptive aggregation and boundary exchange, according to a preferred embodiment of the present invention.

[0065] Figure 2 This is a flowchart of the system initialization configuration module according to a preferred embodiment of the present invention;

[0066] Figure 3 This is a flowchart of the offline policy training module of a preferred embodiment of the present invention;

[0067] Figure 4 This is a flowchart of the action initialization operation in a preferred embodiment of the present invention;

[0068] Figure 5 This is a flowchart of the action selection operation in a preferred embodiment of the present invention;

[0069] Figure 6 This is a flowchart of the block ratio update operation in a preferred embodiment of the present invention;

[0070] Figure 7 This is a flowchart of the action execution operation of a preferred embodiment of the present invention;

[0071] Figure 8 This is a flowchart of the reward update operation in a preferred embodiment of the present invention;

[0072] Figure 9 This is a flowchart of the online strategy fine-tuning module of a preferred embodiment of the present invention;

[0073] Figure 10 This is a graph showing the algorithm convergence and performance of offline policy training in a preferred embodiment of the present invention.

[0074] Figure 11 This is a network adaptability analysis diagram of a preferred embodiment of the present invention;

[0075] Figure 12 This is a performance parameter table of a preferred embodiment of the present invention applied to different DNN models and bandwidth environments. Detailed Implementation

[0076] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention and therefore only show the components relevant to the present invention.

[0077] It should be noted that if directional indicators (such as up, down, bottom, top, etc.) are involved in the embodiments of the present invention, these directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Unless otherwise explicitly specified and limited, the terms "set," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.

[0078] Example 1, as Figures 1-9 As shown, a feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange includes:

[0079] Obtain operational information, including participating device information, network status, and DNN architecture information. Participating device information includes: the number of participating devices N, device computing power, and memory information. Network status includes: the minimum data transmission rate B under the current network environment. min This is used for the action space of subsequent initialization training. During inference, the device that initiates the inference request is designated as the master device, and other devices are marked as slave devices. Slave devices only compute the assigned tasks and summarize the results to the master device, while the master device is also responsible for dividing the feature map into blocks and managing data distribution and connections. DNN architecture information includes: layer information of the feature extraction part of the DNN to be inferred. Layer information includes the number of layers, the type of each layer, and the parameters of each layer.

[0080] Based on the operational information, the optimal aggregation point and boundary exchange strategy are obtained through reasoning using a pre-trained inference model.

[0081] An adaptive aggregation and boundary exchange task scheduling method is implemented by executing the obtained optimal aggregation point and boundary exchange strategy.

[0082] Specifically, the reasoning methods of the reasoning model include:

[0083] Based on the acquired runtime information, perform initialization configuration to obtain the initial strategy;

[0084] Based on the initial policy, a hierarchical MAB mechanism is used to train the policy and obtain the training policy. The policy training includes action initialization operation, action selection operation, block ratio update operation, action execution operation and reward update operation.

[0085] During inference execution, the training strategy is adaptively adjusted through network-aware operations. The action selection, block ratio update, action execution, and reward update operations from the initial strategy training are reused to adjust the training strategy and obtain the optimal aggregation point and boundary exchange strategy. Here, the MAB mechanism is a hierarchical multi-armed gambling machine learning mechanism; DNN is an execution deep neural network model.

[0086] Example 2, based on Example 1: The initialization configuration includes the following methods:

[0087] The DNN is divided into several operational blocks based on the layer information using the operational block partitioning method, and a block list is obtained, where each element is a layer index list.

[0088] The computation block partitioning method includes the following steps: Upon encountering a convolutional layer, a corresponding computation block is activated, and subsequent non-convolutional layers are grouped into this computation block; the computation block serves as a unit for feature boundary exchange. Specifically, non-convolutional layers include activation layers, pooling layers, and normalization layers. Complex blocks such as Basic Blocks and Bottleneck Residual Blocks are treated as separate blocks.

[0089] Example 3, based on Example 1 or Example 2, includes the following method for action initialization during policy training:

[0090] Define action A, which consists of two layers. The first layer is the aggregation index (starting from 1). Aggregates before the index are inferred in parallel, while aggregates after the index and the remaining network aggregates except for the feature extraction part are executed locally. The second layer, based on the decision of the first layer, generates a list of binary variables Pv of length v, where each variable P... k Determine whether to swap feature boundaries (padding) in each parallel block k; a value of 1 indicates swapping.

[0091] A two-layer bandit structure is used to store and maintain candidate actions A, including: the first layer is the aggregation point bandit, which records the number of times each aggregation point v is selected and the cumulative reward, so as to globally evaluate the merits of different aggregation points; the second layer is the feature boundary swapping bandit, which serves as a sub-bandit. For each legal aggregation point, an independent sub-action space is maintained to record the action Pv list and reward value of each parallel block under that aggregation point for swapping feature boundaries.

[0092] During action initialization, for each aggregation point agg, a depth-first search (DFS) is performed based on the device's static computing power and the minimum data transfer rate B. min The simulation latency is generated by the FLOPs of the model blocks, and the execution latency is kept below the minimum latency threshold T for the inference task to be executed on a single device. th The actions are included in the sub-action space of the corresponding aggregation point; when the sub-action space is not empty, the aggregation point is considered a valid aggregation point and added to the valid_aggs list.

[0093] Specifically, the methods for action selection operations include:

[0094] Layered The strategy selects actions on a two-layer Bandit structure. For an inference request in round t, the exploration probability decays with each training round. The first layer maintains the set of aggregation points `valid_aggs` and its cumulative reward `agg_rewards`. The second layer maintains a sub-Bandit for each valid aggregation point `v`, containing the set of padding actions `actions` and the corresponding cumulative reward `rewards`. Further, the action selection operation method includes the following steps: generating a random number `u1` uniformly distributed in the interval [0,1), when... An aggregation point v is selected from valid_aggs in a uniformly random manner. t (Explore); otherwise, i.e., when At that time, select the aggregation point with the largest current average reward from valid_aggs as v. t (Utilization); Lower-level specific action selection (second layer): Read the selected aggregation point v t The corresponding set of sub-actions Generate uniform random number u2 again, when , from in a uniformly random manner Extract a padding swap action from the middle ; Select the action with the highest average reward among the sub-bandits. Output: The action selector process returns a binary action: .

[0095] Specifically, the method for updating the block ratio includes: in the first round, allocation is inversely proportional to the static computing power of the devices, and a periodic adaptive update mechanism is adopted, updating once every M rounds. Further, the specific steps of the update method include: unit block time estimation: calculating the cumulative latency T of each device in M ​​rounds. i Corresponding historical allocation ratio r i Calculate the unit block time q for each device. i : The optimal proportion of device i is derived using the Karush–Kuhn–Tucker optimality conditions (KKT). Calculation formula: Output the scaling vector r and use it as the historical scaling input in the next scaling update to form a closed-loop adaptive update mechanism.

[0096] Specifically, the methods for performing actions include:

[0097] The controller sends the current block ratio vector and the two-layer action parameters selected by the hierarchical multi-armed gambling machine algorithm (MAB) to the master device; the master device divides the input image in the width or height dimension according to the block ratio, and generates several image sub-blocks after rounding the block length;

[0098] The master device sends the aggregation point v and the exchange padding selection Pv to each slave device. The slave devices then construct the corresponding neural network exit function based on this information and prepare to receive task blocks.

[0099] The master device distributes each image block to the corresponding slave device. Each slave device performs the assigned DNN inference task and returns the resulting feature map results to the master device in blocks.

[0100] After receiving all feature map results, the master device performs feature map aggregation and subsequent local DNN inference, finally calculates the complete inference result, and sends the total inference latency back to the controller. During the offline training phase, the controller further compares the inference output with the real labels to calculate the accuracy and sends it back to the controller.

[0101] Specifically, the methods for updating rewards include:

[0102] After performing action A, you will receive real-time latency and accuracy or accuracy bonus, or an instant reward. :

[0103] ;in For accuracy rewards, latency rewards , Penalty for low precision (below the threshold Acc) th (for low precision) and It is to obtain the accuracy Acc and delay T in real time. total The value after normalization to the minimum and maximum values ; , β and β are the weight parameters, respectively.

[0104] Lower layer (padding exchange layer) reward update: For the current aggregation point v tThe selected padding swap action Update its average reward using the exponential moving average (EMA) method:

[0105] Updated Where α is the EMA smoothing coefficient, used to balance the influence of new and old rewards, and to record the number of times the action is sampled.

[0106] Upper-layer (aggregation point layer) reward caching and updating: Each aggregation point maintains a cache list to temporarily store the most recent instant reward values ​​for that aggregation point; when the number of cached samples reaches a threshold, the cache average is calculated as the representative reward for the aggregation point. ,

[0107] Update the aggregation point layer rewards using the EMA smoothing strategy: Update Then clear the cache and prepare for the next update.

[0108] Precision reward dictionary construction: The system maintains a precision reward record table for each action pair, recording the precision-related rewards for the current action. The data is stored in the precision reward dictionary in chronological order; by calculating the historical average value of the precision reward dictionary, a global action-precision reward mapping is obtained for subsequent strategy evaluation and online adjustment.

[0109] Example 4, based on any of Examples 1 to 3, includes policy adjustment: using the trained MAB states and data, reusing action selection, block ratio updates, action execution, and reward updates from policy training to implement policy adjustment, including the following steps:

[0110] The network-aware operation collects network transmission rates in real time before selecting actions. When a continuous decrease or increase in bandwidth is detected exceeding a preset duration, a parameter adaptive adjustment mechanism is triggered. Specifically, the parameter adaptive adjustment mechanism of the network-aware operation includes: temporarily increasing the EMA smoothing coefficient α to enhance the algorithm's response speed to real-time feedback; adding a communication delay penalty term to the reward function to balance computational latency and communication cost; and gradually restoring the temporarily increased EMA smoothing coefficient α and the added communication delay penalty term to the default value of the EMA smoothing coefficient α through a time decay factor after no anomalies are detected for a continuous period of time, and automatically removing the added communication delay penalty term.

[0111] Select two layers of actions according to the reused action selection operation; update the optimal partition ratio according to the reused block ratio update operation; execute the operation through the reused actions and obtain the actual execution latency.

[0112] The comprehensive reward is calculated based on the latency from real-time feedback and the global action-precision reward mapping table generated during the offline phase. A communication penalty is added when the adjustment mechanism is in effect. The next round of inference requests is processed, and online adjustments are continuously performed. Specifically, the EMA smoothing parameter α and the exploration probability parameter are used. To enable lightweight online fine-tuning, this module additionally introduces network-aware operations to dynamically monitor bandwidth fluctuations during runtime.

[0113] Example 5: A feature map block-based distributed inference task scheduling system supporting adaptive aggregation and boundary exchange, comprising: an interconnected system initialization configuration module, an offline policy training module, and an online policy fine-tuning module. The system initialization configuration module is used to collect device performance, bandwidth conditions, and model structure information to complete the execution environment initialization; that is, it is used to collect system information and divide the DNN into blocks. The offline policy training module is used to perform offline training of the initial policy based on the hierarchical MAB algorithm, including action initialization, action selection, block ratio update, action execution, and reward update operations; that is, it is used to train the policy offline using configuration information. The online policy fine-tuning module uses the trained policy and adaptively adjusts it through network-aware operations during actual inference execution, reusing action selection, block ratio update, action execution, and reward update operations to adjust the policy; implementing the steps of a feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange; that is, it is used to adjust the trained policy in conjunction with real-time network changes.

[0114] Among them, such as Figure 2 As shown, the specific implementation of the system initialization configuration module includes the following steps:

[0115] S21, collect information on participating devices, network status, and DNN architecture;

[0116] S22, Traverse all layers of the DNN feature extraction part and check whether it is a convolutional layer or a complex block;

[0117] S23, when encountering a convolutional layer, start a new block and add all layers from the previous block to the block list;

[0118] S24, When a complex block is encountered, it is added to the block list separately, and the index points to the last level index of the complex block;

[0119] S25, if other layer types are encountered, add them to the layer list of the current block;

[0120] S26, increment the index by one, and check the index range;

[0121] S27. If the index exceeds the number of levels, return the block list and end the operation.

[0122] Among them, such as Figure 3 As shown, the specific implementation of the offline policy training module includes the following steps:

[0123] S31, Action Initialization Operation, generates two layers of action space: aggregation point and padding selection.

[0124] S32, begin the t-th round of training, set the reasoning task.

[0125] S33, Action selection operation, based on The algorithm selects two layers of actions in a hierarchical manner.

[0126] S34, block ratio update operation, periodically uses KKT conditions to calculate the optimal ratio update based on latency feedback.

[0127] S35, the action execution operation, sends the selected action and the updated block ratio to the master device, and the master device distributes the image to start inference and returns performance data.

[0128] S36, Reward Update Operation, updates the policy's action rewards based on performance data.

[0129] S37, check the number of iterations.

[0130] S38: When the iteration count is reached, save the strategy state and data, and end the operation.

[0131] Among them, such as Figure 4 As shown, the specific implementation of the action initialization operation includes the following steps:

[0132] S41, iterate through all possible aggregate block indices, from 0 to K;

[0133] S42, when the aggregate block index is 0, it indicates that there is no parallelism and the execution is entirely local;

[0134] S43, when the aggregate block index is greater than 0, there are v parallel blocks, and the initial padding selection vector of length v is all zero;

[0135] S44, recursively perform a DFS search starting from the position where the vector idx=0 is selected from the padding;

[0136] S45, if the current index is the last index of the vector, then select the vector current_padding according to the current padding to simulate and calculate the latency and transmission latency;

[0137] S46, When the total latency is less than the threshold, add the action (v, current_padding) to the action space;

[0138] S47. If the current index is not the last index of the vector, then set the padding at position idx+1 to 0 and 1 respectively, and check whether the latency of the simulated current_padding is greater than the threshold. If it is greater than the threshold, then prune and return to the previous level directly; otherwise, continue to recursively execute the next level.

[0139] S48, if the partial or total latency exceeds the threshold, return to the previous layer;

[0140] S49, after the recursion ends, return to the generated action space.

[0141] Among them, such as Figure 5 As shown, the specific implementation of the action selection operation includes the following steps:

[0142] S51, in the t-th round, generate a random number u1 that is uniformly distributed in the interval [0,1);

[0143] S52, when , indicating exploration, randomly selecting an aggregation point v from the list of feasible aggregation points valid_aggs. t ;

[0144] S53, when This indicates that, by utilizing the list of feasible aggregation points valid_aggs, the aggregation point with the highest current average reward is selected as v. t ;

[0145] S54, Read the selected aggregation point v t The corresponding set of sub-actions (padding selection actions) ;

[0146] S55, generate a random number u2 that is uniformly distributed in the interval [0,1);

[0147] S56, when , indicating exploration, from The U2 machine randomly extracts a swapping padding action. ;

[0148] S57, when This indicates that the action with the highest average reward among the sub-bandits is selected. ;

[0149] S58, return to the selected binary action: .

[0150] Among them, such as Figure 6 As shown, the block ratio update operation specifically includes the following steps:

[0151] S61, in round t, first check if an update is needed;

[0152] S62, if t is not a multiple of the period M, then do not update, and use the previous ratio;

[0153] S63, when t=0, the block ratio vector needs to be initialized. The ratio allocated to each device is the normalized result of the inverse of the device's computing power.

[0154] S64, when t is a multiple of period M and is not 0, then start the update, first calculating the unit block time for each device. ;

[0155] S65, Formula derived using KKT conditions. Calculate the optimal ratio;

[0156] S66, Update the proportional allocation vector.

[0157] Among them, such as Figure 7 As shown, the specific implementation of the action execution operation includes the following steps:

[0158] S71, in round t, the controller will return the two-layer action A from the action selection operation. t and block proportion vector r t Send to the master device;

[0159] S72, the master device receives the action message and, according to the block ratio vector r t Divide the image into N blocks, and simultaneously apply two layers of action A. t The message is sent to all slave devices, along with the segmented image;

[0160] S73, the device constructs a neural network output function based on the received action message and receives image block data;

[0161] S74, each slave device performs inference on the block, selects a vector according to the padding, exchanges padding with other slave devices, and returns the feature map result in blocks to the master device;

[0162] S75: The master device receives all feature map blocks, aggregates them into a complete feature map, and infers the subsequent DNN on the complete feature map to obtain the inference result. It then sends the latency (and accuracy, in the offline stage) to the controller.

[0163] S76, the controller receives performance metrics as feedback, which are used to update rewards and policies.

[0164] Among them, such as Figure 8 As shown, the reward update operation includes the following steps:

[0165] S81, in round t, calculate the immediate reward based on the feedback latency (and accuracy, in the offline phase) (in the online phase, the accuracy reward is obtained from the action-average accuracy reward table, and there is no low accuracy penalty; the communication latency penalty will be increased during network awareness adjustment).

[0166] S82, first use the EMA smoothing strategy to update the action rewards of the lower layer (padding swap layer);

[0167] S83, save the instant reward for the aggregation point action to the cache;

[0168] S84 calculates the average reward in the cache only when the number of samples in the cache reaches the threshold, updates the action reward of the upper layer (aggregation point layer) using the average reward and EMA smoothing update strategy, and clears the cache.

[0169] S85, after the update is completed, maintain a precision reward record table for each action pair, and record the precision reward of the current action. The average historical precision reward will be used to fine-tune the strategy in the subsequent online phase.

[0170] Among them, such as Figure 9 As shown, the online strategy fine-tuning module is implemented by including the following steps:

[0171] S91, starting from receiving the inference request;

[0172] S92: Before selecting an action, monitor the current network transmission rate in real time to prepare for subsequent network-aware operations;

[0173] S93, when in network awareness adjustment, check whether no network anomalies have been detected for a continuous period of time;

[0174] S94, if the condition is met, the time decay process is started, gradually reducing the EMA smoothing coefficient α to the default value, and finally removing the communication delay penalty;

[0175] S95, if not in network-aware adjustment, the counter is incremented only when the transmission rate is below or above the threshold; otherwise, the counter is reset to zero and the counter value is checked.

[0176] S96, if the counter is greater than D, it means that the transmission rate has been lower or higher than the threshold for D consecutive rounds, thus triggering the network-aware adjustment mechanism;

[0177] S97, the adjustment mechanism includes two aspects: increasing the EMA smoothing coefficient α to enhance the algorithm's response speed to real-time feedback; and adding a communication delay penalty term to the reward function to quickly perceive the impact of communication delay.

[0178] S98. If the counter is less than or equal to D or the network still has not recovered during the adjustment, proceed directly to the subsequent decision-making process. The subsequent decision-making process includes reusing action selection, block ratio update, action execution, and reward update. After this round of reasoning is completed, return to the starting point and repeat the process.

[0179] Example 6: A computer program product supporting feature map block distributed inference task scheduling with adaptive aggregation and boundary exchange, comprising a computer program / instructions that, when executed by a processor, implement the steps of a feature map block distributed inference task scheduling method supporting adaptive aggregation and boundary exchange in any of Examples 1 to 4.

[0180] Example 7 tests a feature map block distributed inference task scheduling method that supports adaptive aggregation and boundary exchange, as described in Example 4.

[0181] like Figure 10 The convergence and performance of the offline policy training module are as follows: Under experimental conditions of four edge device clusters (two Raspberry Pi 4s and two Raspberry Pi 5s) with a bandwidth of 5-10 Mb / s, using the VGG16 model and ImageNet data, an image classification task was performed. During the offline training phase, the reward continuously increased with the number of training rounds, stabilizing at around 1.35 after 8000 rounds, demonstrating the convergence of the algorithm. The average performance (converged) of the last 1000 rounds is also shown. Compared to relying entirely on local inference, this method achieves a significant 50.5% reduction in latency while only causing a 2% decrease in accuracy. This achieves an acceptable trade-off between accuracy and latency in edge bandwidth-limited environments.

[0182] like Figure 11 The network adaptability analysis shown is as follows: the online policy fine-tuning module can adaptively adjust through network-aware operations, such as... Figure 11 As shown in (a), initially, the network bandwidth stabilizes at 5-10 Mb / s. This method uses several better strategies obtained through offline training, including aggregation point actions ( Figure 11 (as shown in (c)) and the padding action ( Figure 11 (as shown in (d)). Therefore, as Figure 11 As shown in (b), the communication latency during this phase remains at a low level. However, when a significant drop in bandwidth occurs between rounds 500 and 800, the network-aware operation rapidly adjusts the MAB strategy. Through a brief period of policy exploration and reward update, the two actions fluctuate dramatically, ultimately determining a more suitable strategy for the current network environment to control the communication latency to the lowest possible level. When the network recovers, this method makes similar adjustments to restore the communication latency to normal.

[0183] When extended to different bandwidths and DNN models, such as Figure 12 As shown, the present invention was applied to different DNN models and bandwidth environments, and the results are as follows:

[0184] For ResNet series models with complex structures, this invention chooses full local execution when bandwidth is very limited, which is consistent with reality. When the bandwidth is extended to 72-77 Mb / s, this invention achieves end-to-end latency reductions of 66.2%, 67.25%, and 60.25%, respectively, with accuracy decreases of 1.37%, 0.99%, and 1.82%. For modern CNN models, ConvNeXt-Tiny, under high bandwidth conditions, achieves a 60.3% latency reduction with only a 0.09% decrease in accuracy. For the efficient EfficientNet-B0 model, this invention also chooses full local execution under high bandwidth conditions. These data demonstrate that this invention has good scalability.

[0185] Working principle:

[0186] This invention provides a feature map block distributed inference task scheduling method, system, and program product that supports adaptive aggregation and boundary exchange. It introduces a hierarchical multi-armed learning (MAB) mechanism into the distributed inference framework, which can dynamically select the optimal aggregation point and boundary exchange strategy based on device heterogeneity, bandwidth status, and task load during operation, thereby achieving a balance between inference accuracy and latency optimization.

[0187] The above specific embodiments are specific support for the concept proposed in this invention, and should not be used to limit the scope of protection of this invention. Any equivalent changes or modifications made on the basis of this technical solution in accordance with the technical concept proposed in this invention shall still fall within the scope of protection of this invention.

Claims

1. A feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange, characterized in that, include: Obtain operational information, including participating device information, network status, and DNN architecture information; Based on the operational information, the optimal aggregation point and boundary exchange strategy are obtained through reasoning using a pre-trained reasoning model. An adaptive aggregation and boundary exchange task scheduling method is implemented by executing the obtained optimal aggregation point and boundary exchange strategy. The reasoning methods of the reasoning model include: Based on the acquired runtime information, perform initialization configuration to obtain the initial strategy; Based on the initial policy, a hierarchical MAB mechanism is used to train the policy and obtain the training policy. The policy training includes action initialization operation, action selection operation, block ratio update operation, action execution operation and reward update operation. During inference execution, the training strategy is adaptively adjusted through network-aware operations, and the action selection operation, block ratio update operation, action execution operation and reward update operation in the strategy training are reused to adjust the training strategy and obtain the optimal aggregation point and boundary exchange strategy. Policy adjustment includes: using pre-trained MAB states and data, reusing action selection, block ratio update, action execution, and reward update operations from policy training to implement policy adjustment, including the following steps: The network-aware operation collects the network transmission rate in real time before the action selection is performed. When the bandwidth is detected to be continuously decreasing or increasing for more than the preset duration, the parameter adaptive adjustment mechanism is triggered. Select two layers of actions based on the reused action selection operation; update the optimal partitioning ratio based on the reused block ratio update operation; execute the operation through the reused actions and obtain the actual execution latency; The comprehensive reward is calculated based on the latency of real-time feedback and the global action-precision reward mapping table generated during the offline phase. When the adjustment mechanism is in effect, a communication penalty is added. The next round of inference requests is processed, and online adjustments are continuously performed. Methods for adaptive parameter adjustment mechanisms include: Temporarily increase the EMA smoothing coefficient α to enhance the algorithm's response speed to real-time feedback; A communication latency penalty term is added to the reward function to balance computational latency and communication cost; The temporarily increased EMA smoothing coefficient α and the communication delay penalty term added to the reward function are gradually restored to the default value of the EMA smoothing coefficient α through the time decay factor after no anomalies are detected for a continuous period of time, and the added communication delay penalty term is automatically removed.

2. The feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange according to claim 1, characterized in that: The participating device information includes: the number of participating devices N, device computing power, and memory information; The network status includes: the lowest data transmission rate B under the current network environment. min This is used to initialize the action space for subsequent training. DNN architecture information includes: layer information of the feature extraction part of the DNN to be inferred.

3. The feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange according to claim 2, characterized in that: The initialization configuration includes the following methods: The DNN is divided into several operational blocks based on the layer information using the operational block partitioning method, and a block list is obtained, where each element is a layer index list. The computation block partitioning method includes the following steps: each time a convolutional layer is encountered, the corresponding computation block is activated, and subsequent non-convolutional layers are included in the computation block; the computation block is used as a unit for feature boundary exchange.

4. The feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange according to claim 3, characterized in that: The methods for initializing actions include: Define action A, which consists of two layers. The first layer is the aggregation block indexing. The aggregation blocks before the index are inferred in parallel, while the aggregation blocks after the index and the remaining network aggregation blocks except for the feature extraction part are executed locally. The second layer generates a list of binary variables Pv of length v based on the decisions of the first layer. Each variable P k This determines whether to swap feature boundaries in each parallel block k; a value of 1 indicates a swap. A two-layer bandit structure is used to store and maintain candidate actions A, including: the first layer is the aggregation point bandit, which records the number of times each aggregation point v is selected and the cumulative reward, so as to globally evaluate the merits of different aggregation points; the second layer is the feature boundary swapping bandit, which serves as a sub-bandit. For each legal aggregation point, an independent sub-action space is maintained to record the action Pv list and reward value of each parallel block under that aggregation point for swapping feature boundaries. During action initialization, for each aggregation point agg, a depth-first search is performed based on the device's static computing power and the minimum data transfer rate B. min The simulation latency is generated by the FLOPs of the model blocks, and the execution latency is kept below the minimum latency threshold T for the inference task to be executed on a single device. th The actions are included in the sub-action space of the corresponding aggregation point; when the sub-action space is not empty, the aggregation point is considered a valid aggregation point and added to the valid_aggs list.

5. The feature map block distributed inference task scheduling method supporting adaptive aggregation and boundary exchange according to claim 4, characterized in that: The methods for selecting actions include: Layered The strategy selects actions on a two-layer Bandit structure. For an inference request in round t, the exploration probability decays with each training round. The first layer maintains the set of aggregate points valid_aggs and its cumulative reward agg_rewards. The second layer maintains a sub-Bandit for each valid aggregate point v, which includes the set of padding actions actions and the corresponding cumulative reward rewards. And / or, the method for block ratio update operation includes: the first round is allocated inversely proportionally according to the static computing power of the equipment, and a periodic adaptive update mechanism is adopted, updating once every M rounds; And / or, the methods for performing actions include: The controller sends the current block ratio vector and the two-layer action parameters selected by the hierarchical multi-armed gambling machine algorithm to the master device; the master device divides the input image in the width or height dimension according to the block ratio, and generates several image sub-blocks after rounding the block length; The master device sends the aggregation point v and the exchange padding selection Pv to each slave device. The slave devices then construct the corresponding neural network exit function based on this information and prepare to receive task blocks. The master device distributes each image block to the corresponding slave device. Each slave device performs the assigned DNN inference task and returns the resulting feature map results to the master device in blocks. After receiving all feature map results, the master device performs feature map aggregation and subsequent local DNN inference, finally calculates the complete inference result, and sends the total inference latency back to the controller. During the offline training phase, the controller compares the inference output with the real label to calculate the accuracy and sends it back to the controller.

6. The feature map block distributed inference task scheduling method supporting adaptive aggregation and boundary exchange according to claim 5, characterized in that: The methods for updating rewards include: After performing action A, you will receive real-time latency and accuracy or accuracy bonus, or an instant reward. : ;in For accuracy rewards, latency rewards , Penalty for low precision; and It is to obtain the accuracy Acc and delay T in real time. total The value after normalization to the minimum and maximum values ; , β and β are the corresponding weight parameters, respectively; Lower-level reward update: For the current aggregation point v t The selected padding swap action Update its average reward value using an exponential moving average: Updated Where α is the EMA smoothing coefficient, used to balance the influence of new and old rewards, and to record the number of times the action is sampled; Upper-level reward caching and updating: Each aggregation point maintains a cache list to temporarily store the most recent instant reward values ​​for that aggregation point; when the number of cached samples reaches a threshold, the cache average is calculated as the representative reward for the aggregation point. ,in The number of samples in the cache list of aggregation point v. A cache list for aggregation point v; Update the aggregation point layer rewards using the EMA smoothing strategy: Update Then clear the cache and prepare for the next update; Precision reward dictionary construction: Maintain a precision reward record table for each action pair, recording the precision-related rewards for the current action, The data is stored in the precision reward dictionary in chronological order; by calculating the historical average value of the precision reward dictionary, a global action-precision reward mapping is obtained for subsequent strategy evaluation and online adjustment.

7. A feature map block-based distributed inference task scheduling system supporting adaptive aggregation and boundary exchange, comprising the steps of the feature map block-based distributed inference task scheduling method supporting adaptive aggregation and boundary exchange as described in any one of claims 1-6; characterized in that, include: The interconnected system initialization configuration module, offline policy training module, and online policy fine-tuning module; The system initialization configuration module is used to collect device performance, bandwidth conditions and model structure information to complete the execution environment initialization; The offline policy training module is used for offline training of the initial policy based on the hierarchical MAB algorithm. It includes action initialization operation, action selection operation, block ratio update operation, action execution operation and reward update operation. The online policy fine-tuning module uses a trained policy and adaptively adjusts it through network-aware operations during actual inference execution. It also reuses action selection, block ratio update, action execution, and reward update operations to fine-tune the policy.

8. A computer program product supporting feature map block distributed inference task scheduling with adaptive aggregation and boundary exchange, comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the steps of the feature map block distributed inference task scheduling method supporting adaptive aggregation and boundary exchange as described in any one of claims 1-6.

Images