Bandwidth-aware model layer transfer method and system for heterogeneous edge devices

Abstract

The application discloses a bandwidth-aware model layer migration method and system for heterogeneous edge devices, which is applied to pipeline parallel inference. In the model deployment stage, the candidate boundary layers are pre-divided according to the available storage space of adjacent edge devices, and the candidate boundary layers are redundantly stored in adjacent devices. In the inference process, the stage time delay of each pipeline stage is collected, the sliding window is adaptively adjusted according to the time delay fluctuation degree, and the smoothed stage time delay is obtained; whether the migration is triggered is judged according to the difference of the smoothed time delay of each stage, the dynamic migration threshold and the cooling condition. After triggering, the target model layer allocation scheme of multiple pipeline boundaries is jointly determined by combining the predicted calculation time delay and the predicted communication time delay after migration, and the migration is completed by switching the execution state and data routing of the candidate boundary layer. The application does not need to transmit the model weight online, can adapt to the link bandwidth change, and reduce the stage load difference and the model inference time delay.

Description

Technical Field

[0001] This invention relates to a bandwidth-aware model layer migration method and system for heterogeneous edge devices, belonging to the interdisciplinary technical fields of pipelined parallel inference, model deployment, load balancing, and computer network collaborative optimization. Background Technology

[0002] As the parameter scale of deep neural network models such as visual language models and large language models continues to increase, a single resource-constrained edge device often struggles to independently complete model loading and inference computation. To meet the model inference requirements in edge scenarios, the model to be inferred can be divided into multiple consecutive sets of model layers according to the execution order of the model layers, and deployed on multiple edge devices respectively, thereby forming a pipelined parallel inference system.

[0003] During pipelined parallel inference, each edge device performs forward computation on its assigned model layer and transmits the generated intermediate activation data to downstream edge devices. The pipeline stage latency for each edge device typically includes model layer computation latency and intermediate activation data communication latency. The overall pipeline inference efficiency is limited by the slowest pipeline stage. When the latency difference between different pipeline stages is large, faster edge devices need to wait for slower edge devices to complete processing, resulting in pipeline cavitation, reducing device resource utilization, and increasing model inference latency.

[0004] Heterogeneous edge devices typically differ in computing power, storage capacity, and memory bandwidth. Communication links between devices can also dynamically change due to factors such as network congestion, wireless signal variations, shared bandwidth contention, and external interference. Even if the model has already performed load balancing according to device computing power and link status during initial deployment, fluctuations in link bandwidth during operation may still cause changes in the transmission latency of intermediate activation data, leading to an imbalance in the original model layer partitioning.

[0005] Existing model partitioning methods typically determine the model layer allocation scheme based on static device performance and static network conditions before model deployment, and do not adjust the execution affiliation of the model layers during inference, thus making it difficult to adapt to dynamic link states. Some dynamic load balancing methods can reallocate model layers at runtime, but they mainly rely on device computation latency for migration decisions, failing to fully consider the impact of intermediate activation data communication latency and dynamic bandwidth changes on pipeline stage latency. This can easily lead to problems such as communication bottlenecks not being eliminated after migration or bottlenecks shifting from one pipeline stage to another.

[0006] Furthermore, the model layer typically contains a large number of model parameters. Transferring these parameters from one edge device to another during model inference incurs significant communication overhead and may interrupt the ongoing inference task. For visual language models or large language models with a large parameter set, the time spent on online transfer of model weights can significantly reduce the performance gains from dynamic transfer.

[0007] In dynamic link environments, directly triggering model layer migration based on a single bandwidth measurement result or the stage latency of a single inference batch is susceptible to instantaneous network jitter, device scheduling, and changes in cache status, leading to false triggers. After the model layer migration is completed, the historical monitoring window may still retain latency data from before the migration. If the migration judgment is immediately executed again, it can easily cause the model layer to switch repeatedly between adjacent devices, resulting in system oscillations.

[0008] In summary, existing technologies have at least the following shortcomings: First, static model layer partitioning is difficult to adapt to changes in runtime link bandwidth; second, existing dynamic migration decisions lack joint consideration of computational and communication latency, making it difficult to achieve global pipeline load balancing; third, online migration of model weights incurs significant communication overhead; and fourth, there is a lack of a stable triggering mechanism that can distinguish between instantaneous network jitter and persistent stage imbalances. Therefore, there is an urgent need for a method and system that can sense dynamic link states, reliably identify pipeline stage imbalances, and adjust the execution boundaries of the model layer without requiring online transmission of model weights. Summary of the Invention

[0009] Purpose of the invention: This invention proposes a bandwidth-aware model layer migration method and system for heterogeneous edge devices, aiming to overcome the problems existing in the current pipelined parallel inference technology, such as the static model layer partitioning being difficult to adapt to changes in link bandwidth, instantaneous network jitter easily causing false migration triggers, local migration easily causing bottleneck shifts, and large overhead of online model weight transmission.

[0010] This invention pre-divides candidate boundary layers based on the available storage space of adjacent edge devices during the model deployment phase, and redundantly stores the model parameters of the candidate boundary layers on adjacent edge devices, providing pre-conditions for adjusting the model computation boundary during operation. During model inference, pipeline stage latency, including computation and communication latency, is used as a sensing signal of the impact of link state changes on inference performance. An adaptive sliding window is used to smooth the stage latency, and dynamic migration thresholds and migration cooling conditions are combined to identify persistent pipeline load imbalances. When the model layer migration conditions are met, the model layer computation load, edge device processing capabilities, current communication status, and the amount of intermediate activation data at the model computation boundary are considered simultaneously to jointly determine a target model layer allocation scheme for multiple pipeline boundaries. Model layer migration is then completed by adjusting the execution state of the candidate boundary layers and the intermediate activation data transmission path.

[0011] Through the above mechanism, the present invention can dynamically adjust the model calculation boundary between adjacent pipeline stages without transmitting candidate boundary layer model parameters online, adapt to the link bandwidth changes between heterogeneous edge devices, and reduce pipeline stage latency differences and model inference latency.

[0012] Technical Solution: A bandwidth-aware model layer migration method for heterogeneous edge devices, applicable to coordinating nodes and A pipelined parallel inference system with multiple inference nodes. The coordinating node and each inference node are edge devices. The model to be inferred includes multiple model layers connected in the order of model execution, and is divided into layers according to the execution order of the model layers. In each pipeline stage, each inference node undertakes the model inference calculation for one pipeline stage and transmits the intermediate activation data output by the current pipeline stage to the downstream inference node.

[0013] The coordination node is used to aggregate the operational information of each inference node, determine the pipeline load imbalance, identify the target allocation scheme, and control the migration execution of the model layer; the coordination node is implemented by an independent edge device; for ease of description, in the following text, This indicates the inference node and its corresponding pipeline stage number. .

[0014] The method includes the following: (1) Pre-partitioning and redundant storage of candidate boundary layer sets Based on the computing power and storage capacity of the edge devices where each inference node is located, as well as the resource consumption of each model layer in the model to be inferred, the model to be inferred is initially partitioned so that each inference node obtains an initial set of model layers arranged continuously in the order of model execution.

[0015] A set of candidate boundary layers is determined at the model computation boundary between every two adjacent inference nodes, and the model parameters corresponding to the set of candidate boundary layers are pre-stored in the inference nodes on both sides of the model computation boundary, so that the candidate boundary layers can be executed by the inference node on either side of the model computation boundary according to the model layer allocation scheme.

[0016] The pre-partitioning and redundant storage of the candidate boundary layer set (1) specifically includes the following steps: (1.1) Determining available storage capacity: Obtain the first The total storage capacity, initial storage occupancy, and storage safety margin of each inference node are determined, and based on the total storage capacity, initial storage occupancy, and storage safety margin of the i-th inference node, the available storage capacity of the i-th inference node for storing candidate boundary layers is determined. The i-th... The available storage capacity of each inference node is determined by the following formula: in, ; Indicates the first Available storage capacity of each inference node; Indicates the first Total storage capacity of each inference node; Indicates the first Initial storage requirements for each inference node; This represents the reserved storage safety margin for the i-th inference node. The initial storage occupancy includes the initial allocation of model parameters for the model layer, the inference cache occupancy, and the intermediate activation data cache occupancy. Further, the storage allocation ratio is set. ,and , making the first The initial storage footprint of each inference node satisfies: (1.2) Determination of the candidate boundary layer set: For the first... The inference node and the first The model computation boundary between the two inference nodes is determined by continuously selecting model layers from both sides of the model computation boundary in a direction away from the model computation boundary, based on the available storage capacity of each of the two inference nodes and the model parameter occupancy of each model layer initially allocated to the two inference nodes. Capacity constraints are then applied to the model layers that need to be redundantly stored on the opposite inference node, thereby determining the candidate boundary layer set corresponding to the model computation boundary.

[0017] First, the degree of difference in storage usage is determined based on the model parameter usage of each model layer. The degree of difference in storage usage is determined according to the following formula: in, This indicates the maximum number of model parameters initially allocated to each of the two inference nodes. This represents the minimum number of model parameters initially allocated to each of the two inference nodes in their respective model layers. This represents the average storage footprint of the model layer initially allocated to each of the two inference nodes.

[0018] When the difference in storage usage does not exceed a preset storage usage difference threshold, and the same number of model layers are selected from both sides of the model computation boundary, the number of candidate boundary layers selected from each side of the model computation boundary is determined by the average storage usage of a single model layer, specifically according to the following formula: in, Indicates the relationship between the i-th inference node and the i-th inference node. The number of candidate boundary layers selected on each side of the model computation boundary between each inference node; This represents the average storage usage of a single model layer; This indicates taking the smaller value; This indicates rounding down. The above formula applies to the case where candidate boundary layers are selected equally from both sides of the model's computational boundary, and the total number of candidate boundary layer sets is the sum of the number of layers selected from both sides.

[0019] When the difference in storage usage exceeds the preset storage usage difference threshold, or when the number of model layers selected from both sides of the model computation boundary is different, the model parameter usage of each candidate model layer is accumulated sequentially along the direction away from the model computation boundary, according to the actual model parameter usage of each model layer. Specifically, candidate model layers from the initial model layer set of the i-th inference node require redundant storage at the (i+1)-th inference node, and their accumulated model parameter usage does not exceed the available storage capacity of the (i+1)-th inference node; candidate model layers from the initial model layer set of the (i+1)-th inference node require redundant storage at the i-th inference node, and their accumulated model parameter usage does not exceed the available storage capacity of the i-th inference node. Continuous model layers that satisfy the above-mentioned opposite-side storage capacity constraints are merged to form a candidate boundary layer set corresponding to the model computation boundary.

[0020] Furthermore, for intermediate inference nodes that participate in both upstream and downstream model computation boundaries, a unified constraint is applied to the redundant storage usage of these nodes for both model computation boundaries. This constraint ensures that the sum of the model parameter usage for all candidate boundary layers in the additional redundant storage for both upstream and downstream model computation boundaries does not exceed the available storage capacity of the inference node. When determining the candidate boundary layer set corresponding to multiple model computation boundaries, the candidate boundary layer sets for each model computation boundary can be jointly determined, or determined sequentially according to a preset order of the model computation boundaries. The remaining available storage capacity of the corresponding inference node is then updated based on the already occupied redundant storage capacity.

[0021] (1.3) Redundant storage of candidate boundary layers: The model parameters of each candidate boundary layer in the candidate boundary layer set corresponding to the model computation boundary between the i-th inference node and the i+1-th inference node are stored in both the i-th inference node and the i+1-th inference node, so that each candidate boundary layer can switch execution affiliation between the two inference nodes without retransmitting the model parameters during the switching process.

[0022] (1.4) Determining the basic model layer: The first... Model layers that were initially assigned to the candidate boundary layer set in the initial model layer set of the nth inference node are removed, resulting in the nth... The set of basic model layers for each inference node. The set of basic model layers is determined according to the following formula: in, Indicates the first A set of basic model layers for each inference node; Indicates the first The initial model layer set for each inference node; Indicates the first The inference node and the first A set of candidate boundary layers among inference nodes; Indicates the first The inference node and the first A set of candidate boundary layers among inference nodes.

[0023] No. Each inference node continuously stores its base model layer set, a candidate boundary layer set shared with upstream inference nodes, and a candidate boundary layer set shared with downstream inference nodes. The stored model layer set is represented as follows: For an inference node at the beginning of the pipeline, the set of candidate boundary layers it shares with the upstream inference node is an empty set; for an inference node at the end of the pipeline, the set of candidate boundary layers it shares with the downstream inference node is an empty set.

[0024] (2) Monitoring and smoothing of pipeline stage delays During model inference, each inference node collects the stage latency of its corresponding pipeline stage. The stage latency of the pipeline stage corresponding to the i-th inference node in the t-th inference batch includes computation latency and communication latency, and is expressed as: in, Indicates the first The inference node corresponds to the pipeline stage at the [number]th inference node. Phase latency of each inference batch; Indicates the first The computational latency required for each inference node to execute the currently enabled set of model layers; Indicates the first The communication latency required for an inference node to transmit intermediate activation data to a downstream inference node.

[0025] For inference nodes located at the end of the pipeline, their communication latency can be set to the output transmission latency of the model inference result or zero. Since changes in inter-device link bandwidth will cause changes in intermediate activation data communication latency, which are further reflected in the pipeline stage latency, this invention uses the pipeline stage latency, which includes communication latency, as a sensing signal of the impact of link state changes on model inference performance.

[0026] The monitoring and smoothing of pipeline stage delays (2) specifically includes the following steps: (2.1) Historical delay benchmark calculation: The i-th inference node maintains the historical delay sequence of the corresponding pipeline stage. The historical delay sequence records the stage delay of each completed inference batch in the order of inference batches.

[0027] When model inference starts, the sliding window length of the pipeline stage corresponding to the i-th inference node is initialized to the preset minimum window length. .

[0028] For the first inference batch, since there is no historical latency prior to the current inference batch, the average latency and latency fluctuation of the historical phases are not calculated. The current sliding window length of the first inference batch is used. Set as the preset minimum window length And the delay during the stage of obtaining the first inference batch. Then, the delay of this stage is added to the historical delay sequence.

[0029] For the t-th inference batch following the first inference batch, before obtaining the current stage delay and adding it to the historical delay sequence, the sliding window length determined by the previous inference batch is used. In addition to the number of valid historical stage delays already existing in the historical delay sequence before the current inference batch, determine the number of valid historical samples used to calculate the average historical stage delay.

[0030] The number of valid historical samples is determined according to the following formula: in, This represents the number of valid historical samples used to calculate the historical stage latency mean in the t-th inference batch corresponding to the pipeline stage of the i-th inference node. This represents the length of the sliding window for the pipeline stage corresponding to the i-th inference node, as determined in the previous inference batch. This represents the number of valid historical stage delays in the historical delay sequence prior to the t-th inference batch.

[0031] Select the batch closest to the current inference batch from the historical time delay sequence. The historical delay is calculated using the following formula: in, Indicates the current inference batch number. This represents the average latency of the pipeline stage corresponding to the i-th inference node before the t-th inference batch; Indicates relative to the current number The historical reasoning batch sequence number that traces back from the previous reasoning batch; Indicates the first The inference node corresponds to the pipeline stage at the [number]th inference node. Phase latency of each inference batch.

[0032] Therefore, when the number of existing historical stage delays is not less than the sliding window length determined in the previous inference batch, the corresponding number of recent historical stage delays are selected according to the sliding window length; when the number of existing historical stage delays is less than the sliding window length, the average historical stage delay is calculated using all existing historical stage delays, so that the system gradually establishes a delay benchmark during the historical sample accumulation phase after the model inference is started.

[0033] (2.2) Calculation of latency fluctuation: For the current inference batch after the first inference batch and with a valid historical average latency, the latency fluctuation is calculated based on the normalized deviation between the current latency and the historical latency benchmark. To avoid distortion of the normalization result when the historical latency average is zero or abnormally close to zero, a preset positive latency lower limit is introduced. The larger of the historical average latency and the preset positive latency lower limit is used as the normalization benchmark. The latency fluctuation level is determined according to the following formula: in, This indicates the latency fluctuation of the pipeline stage corresponding to the i-th inference node in the t-th inference batch; Indicates the current stage delay; This represents the average latency of historical periods preceding the current inference batch; This indicates a preset positive lower limit for delay, and Under normal operating conditions, the pipeline stage latency includes the model layer computation latency, and its historical stage latency average is usually greater than zero. The preset positive latency lower limit is used to protect against abnormally zero or near-zero values. The greater the latency fluctuation, the more significant the change in the current stage latency relative to the historical operating state.

[0034] (2.3) Sliding Window Length Adjustment: For the first inference batch, since a historical stage latency benchmark has not yet been established, no sliding window adjustment based on latency fluctuation is performed, and the current sliding window length is maintained at the preset minimum window length. For subsequent inference batches after the formation of an effective historical stage latency average, the latency fluctuation is compared with a preset fluctuation threshold, and the current sliding window length is adaptively determined based on the comparison result. When the latency fluctuation is not greater than the preset fluctuation threshold, the current sliding window length is determined as the preset minimum window length; when the latency fluctuation is greater than the preset fluctuation threshold, the current sliding window length is increased as the latency fluctuation increases, and the current sliding window length is limited to within the preset maximum window length. The current sliding window length is determined according to the following formula: in, This represents the sliding window length of the pipeline stage corresponding to the i-th inference node in the t-th inference batch; Indicates the preset minimum window length; Indicates the preset maximum window length; Indicates the preset fluctuation threshold; Indicates the window scaling factor; This indicates rounding up to the nearest integer.

[0035] When the stage delay fluctuation is within the preset range, a shorter sliding window is used to improve the system's response speed to recent link status changes; when the stage delay fluctuation exceeds the preset fluctuation threshold, the sliding window is increased to introduce more historical stage delay samples and reduce misjudgments caused by instantaneous network jitter.

[0036] (2.4) Smoothing Phase Delay Calculation: Add the current phase delay to the historical delay sequence. Determine the actual number of smoothed samples based on the current sliding window length and the number of effective phase delays obtained up to the current inference batch. The actual number of smoothed samples is and The smaller of the values. When the number of valid stage delays obtained is less than the current sliding window length, smoothing is performed using all the valid stage delays obtained so far; when the number of valid stage delays reaches or exceeds the current sliding window length, smoothing is performed using the current inference batch and the nearest stage delays corresponding to the current sliding window length. Based on the actual number of smoothed samples, the smoothed stage delay corresponding to the pipeline stage of the i-th inference node is obtained according to the following formula: in, This represents the smoothing delay of the pipeline stage corresponding to the i-th inference node in the t-th inference batch; Indicates the actual number of smoothed samples; This represents the number of effective stage delays obtained up to the t-th inference batch. For the first inference batch, the actual number of smoothed samples is 1, and the smoothed stage delay is equal to the current stage delay.

[0037] Furthermore, the same insufficient window handling method is applied to the communication latency, and the smoothed communication latency of the pipeline stage corresponding to the i-th inference node is obtained based on the actual number of smoothed samples: in, This represents the smoothed communication latency of the pipeline stage corresponding to the i-th inference node in the t-th inference batch.

[0038] (2.5) Operational Information Reporting: Each inference node, according to a preset cycle or after completing an inference batch, sends the smoothing stage delay and smoothing communication delay of the corresponding pipeline stage to the coordination node. When it is necessary to determine a candidate allocation scheme, each inference node can also send the currently enabled model layer set and the amount of intermediate activation data at the current model computation boundary to the coordination node, or the coordination node can obtain the currently enabled model layer set and the amount of intermediate activation data at the current model computation boundary based on the saved model layer allocation information and model structure metadata. The coordination node summarizes the operational information of each pipeline stage to form a monitoring data set reflecting the global operational status of the pipeline.

[0039] In the first inference batch after model inference starts, each inference node establishes an initial historical latency sequence and calculates smoothing stage latency and smoothing communication latency based on existing samples. However, it does not calculate latency fluctuation, perform sliding window adjustments based on latency fluctuation, or trigger model layer migration. Once each pipeline stage has obtained at least one valid historical stage latency and can form an average historical stage latency, latency fluctuation calculation, sliding window adjustment, and model layer migration trigger judgment are performed from subsequent inference batches onwards, and smoothing stage latency and smoothing communication latency calculation continue.

[0040] (3) Triggering judgment of model layer migration: The coordinating node determines whether there is a continuous load imbalance in the pipeline based on the smoothing stage delay of each pipeline stage, and determines whether to trigger model layer migration in combination with the dynamic migration threshold and migration cooling conditions.

[0041] The triggering judgment for model layer migration in (3) specifically includes the following steps: (3.1) Calculation of pipeline load imbalance: The coordination node obtains the smoothing stage delay of each pipeline stage, and determines the pipeline load imbalance by the difference between the maximum smoothing stage delay and the minimum smoothing stage delay among the N pipeline stages.

[0042] The degree of load imbalance in the production line is determined according to the following formula: in, This represents the pipeline load imbalance degree corresponding to the t-th inference batch. The greater the pipeline load imbalance degree, the greater the time delay difference between the slowest and fastest pipeline stages, and the more severe the pipeline cavitation.

[0043] (3.2) Determination of Dynamic Migration Threshold: Calculate the average smoothing stage delay for all pipeline stages, and determine the dynamic migration threshold based on the average smoothing stage delay and a preset steady-state tolerance coefficient. The average smoothing stage delay is determined according to the following formula: The dynamic migration threshold is determined according to the following formula: in, This represents the average global smoothing stage latency corresponding to the t-th inference batch; This represents the dynamic migration threshold corresponding to the t-th inference batch; This represents the preset steady-state tolerance coefficient, and .

[0044] By employing a dynamic migration threshold related to the average latency of the global smoothing phase, the migration triggering criteria can be dynamically adjusted according to changes in the size of the model to be inferred, the size of the inference batch, the combination of devices, and the overall operating speed of the system.

[0045] (3.3) Determining the migration interval: Inference batches are numbered starting from 1. At the start of model inference, a migration batch variable is used to record the inference batch corresponding to the most recent model layer migration. Initialize to 0. For the current inference batch t, ​​the difference between the current inference batch and the migration batch variable is determined as the migration interval. After each model layer migration is completed, Updated to the inference batch number corresponding to this model layer migration.

[0046] Set migration cooling cycle The migration cooling period is not less than the maximum value among the current sliding window lengths of each inference node: This allows historical time-delay samples before model layer migration to exit the corresponding sliding window before the migration decision is executed again, reducing the possibility of repeated migration or reverse migration caused by historical data before migration.

[0047] (3.4) Model layer migration trigger: When the pipeline load imbalance is greater than the dynamic migration threshold, and the interval between the current inference batch and the inference batch corresponding to the most recent model layer migration is greater than the migration cooldown period, the model layer migration is triggered.

[0048] The model layer migration triggering condition is determined according to the following formula: in, This represents the migration trigger flag corresponding to the t-th inference batch. At that time, each inference node continues to perform pipelined parallel inference according to the execution hierarchy of the current model layer; when At that time, the coordinating node initiates the process of determining the target allocation scheme and executing the model layer migration.

[0049] (4) Determination and migration execution of the target allocation plan When the model layer migration is triggered, the coordinating node constructs multiple candidate allocation schemes based on the candidate boundary layer set between each adjacent inference node, and predicts the computation latency and communication latency of each pipeline stage under each candidate allocation scheme in order to determine the target allocation scheme.

[0050] To avoid pipeline bottlenecks shifting to other pipeline stages due to adjusting only the computational boundaries of a single model, all... The candidate boundary layers of each model's calculated boundary are jointly determined to determine their affiliation.

[0051] The determination and migration execution of the target allocation scheme in (4) specifically includes the following steps: (4.1) Candidate allocation scheme construction: For the j-th inference node and the j-th inference node... A set of candidate boundary layers among inference nodes Set allocation variables The allocation variable Represents the set of candidate boundary layers The number of candidate boundary layers executed by the j-th inference node. The allocation variable satisfies: in, Represents the set of candidate boundary layers The number of model layers included; the remaining candidate boundary layer sets The model layer consists of the first... Each inference node executes.

[0052] All The combinations of allocation variables corresponding to the computational boundaries of each model are candidate allocation schemes: Based on the candidate allocation scheme K, the basic model layer set of each inference node, and the execution affiliation of each candidate boundary layer set, reconstruct the model layer execution set of the i-th inference node under the candidate allocation scheme K. .

[0053] (4.2) Determining the prediction computation delay: Based on the model layer execution set of the i-th inference node under the candidate allocation scheme K. The computational workload of each model layer and the processing performance parameters of the model layer of the i-th inference node are used to determine the prediction computation latency of the pipeline stage corresponding to the i-th inference node.

[0054] The prediction calculation delay is determined according to the following formula: in, This represents the prediction computation delay of the pipeline stage corresponding to the i-th inference node under candidate allocation scheme K; Indicates the first l The computational workload of each model layer; This represents the processing time required for the i-th inference node to perform the unit model layer computation workload.

[0055] The computational workload of each model layer and the processing performance parameters of each inference node can be obtained through offline performance testing before model deployment, or updated based on historical computational latency during model inference.

[0056] (4.3) Predicted communication delay determination: Based on the current smoothed communication delay, the amount of intermediate activation data at the current model calculation boundary, and the amount of intermediate activation data at the new model calculation boundary corresponding to the candidate allocation scheme K, the predicted communication delay of the pipeline stage corresponding to the i-th inference node is determined.

[0057] In one implementation, the current smoothed communication delay is used to characterize the communication state of recent links, and the communication delay under the candidate allocation scheme is predicted based on the model calculation boundary changes and intermediate activation data volume changes caused by the candidate allocation scheme. The predicted communication delay is determined according to the following formula: Within a shorter timescale corresponding to the migration decision, when link state changes are relatively small, the current smooth communication latency can effectively characterize the recent link communication state during the execution of candidate allocation schemes. This represents the predicted communication latency of the pipeline stage corresponding to the i-th inference node under candidate allocation scheme K; This represents the current smooth communication latency of the pipeline stage corresponding to the i-th inference node; This represents the amount of intermediate activation data that needs to be transmitted downstream at the current model's computational boundary. This represents the amount of intermediate activation data that needs to be transmitted downstream at the computational boundary of the new model corresponding to candidate allocation scheme K.

[0058] (4.4) Determining the prediction stage delay: Add the prediction computation delay and prediction communication delay of the pipeline stage corresponding to the i-th inference node to obtain the prediction stage delay of the pipeline stage corresponding to the i-th inference node under candidate allocation scheme K: in, This represents the prediction stage delay of the pipeline stage corresponding to the i-th inference node under candidate allocation scheme K.

[0059] (4.5) Target allocation scheme selection: For any candidate allocation scheme The maximum prediction stage delay in each pipeline stage under the candidate allocation scheme is determined as the prediction pipeline cycle corresponding to the candidate allocation scheme: in, Indicates the number of stages in the production line. Represents candidate allocation schemes The corresponding predicted pipeline cycle, Represents candidate allocation schemes Next The prediction stage delay for each pipeline stage; determining the minimum prediction pipeline cycle from all candidate allocation schemes: Candidate allocation schemes whose predicted pipeline cycle is equal to the minimum predicted pipeline cycle are determined as preferred candidate allocation schemes; when there is only one preferred candidate allocation scheme, it is determined as the target allocation scheme; when there are multiple preferred candidate allocation schemes, the difference between the maximum prediction stage delay and the minimum prediction stage delay corresponding to each preferred candidate allocation scheme is determined, and the preferred candidate allocation scheme with the smallest difference is determined as the target allocation scheme. in, Indicates the target allocation scheme; (4.6) Model layer migration execution: The coordinating node allocates the scheme according to the target. The target execution device after the migration of each candidate boundary layer is determined, and a migration control command is sent to the corresponding inference node. The migration control command includes the candidate boundary layer identifier, the target execution device identifier, execution status adjustment information, and data routing adjustment information.

[0060] According to the migration control instructions, the corresponding inference node deactivates or freezes the candidate boundary layer to be migrated on the original inference node, and enables or activates the candidate boundary layer on the target inference node, so that the same candidate boundary layer is executed by one inference node in the same inference batch.

[0061] The term "disable" or "freeze" means that the original inference node skips the computation operation of the candidate boundary layer during the model inference process, while retaining the model parameters of the candidate boundary layer; the term "enable" or "activate" means adding the candidate boundary layer pre-stored in the target inference node to the current model layer execution set of the target inference node.

[0062] Based on the migrated model computation boundary, update the model operator call relationship and intermediate activation data transmission path between adjacent inference nodes, so that the upstream inference node sends the intermediate activation data generated by the last model layer it is currently executing to the inference node executing the subsequent model layer.

[0063] Since the model parameters of the candidate boundary layer have been redundantly stored in adjacent inference nodes during the model deployment phase, there is no need to transmit the model parameters of the candidate boundary layer between the original inference node and the target inference node during the migration execution process. Only the migration control instructions and the intermediate activation data transmitted normally during the model inference process are transmitted.

[0064] After the model layer migration is complete, update the current model layer execution set, model computation boundary, data routing information, and the inference batch corresponding to the most recent model layer migration for each inference node. And then enter the migration cooling cycle.

[0065] Each inference node continues to collect the stage delay after migration, thus forming a closed-loop control process of "stage delay monitoring - pipeline load imbalance judgment - target allocation scheme determination - model layer migration execution - stage delay re-monitoring".

[0066] Furthermore, in an embodiment corresponding to the foregoing limitations, the model to be inferred is a visual language model or a large language model, and the model layer includes multiple Transformer layers connected in execution order; each inference node executes consecutive Transformer layers, and intermediate activation data output from the upstream pipeline stage is transmitted between inference nodes corresponding to adjacent pipeline stages; the candidate boundary layer set consists of at least one consecutive Transformer layer located near the model computation boundary of adjacent pipeline stages. For other neural network models that can be pipelined according to the execution order of model layers, the corresponding consecutive executable layer units can be used as model layers.

[0067] The present invention also provides a bandwidth-aware model layer migration system for heterogeneous edge devices, including a coordination node, multiple inference nodes, a candidate boundary layer pre-partitioning and redundant storage module, a stage latency monitoring module, a migration triggering module, a migration decision module, and a migration execution module, wherein the coordination node and the inference node are both edge devices.

[0068] Candidate boundary layer pre-partitioning and redundant storage module: After the initial pipeline partitioning of the model to be inferred for multiple inference nodes is completed, the module determines a set of candidate boundary layers near the model computation boundary between the two inference nodes based on the available storage capacity of the two inference nodes undertaking adjacent pipeline stages for redundant storage of model layers, and pre-stores the model parameters of the candidate boundary layer set in the two inference nodes simultaneously. Stage latency monitoring module: used to collect the computation latency and communication latency of each pipeline stage during model inference, and determine the stage latency of each pipeline stage accordingly; determine the sliding window length based on the fluctuation of the current stage latency relative to the historical stage latency benchmark, smooth the stage latency of each pipeline stage based on the sliding window to obtain smooth stage latency, and smooth the communication latency based on the sliding window to obtain smooth communication latency; Migration triggering module: executed by the coordination node, used to determine the pipeline load imbalance degree and dynamic migration threshold based on the smoothing stage delay of each pipeline stage, and to trigger model layer migration when the pipeline load imbalance degree exceeds the dynamic migration threshold and the migration cooling condition related to the sliding window length is met; Migration decision module: executed by the coordination node, used to jointly combine the candidate boundary layers in the candidate boundary layer set corresponding to the computational boundary of each model to form multiple candidate allocation schemes; for each candidate allocation scheme, determine the predicted computation delay and predicted communication delay of each pipeline stage; determine the predicted stage delay of each pipeline stage based on the corresponding predicted computation delay and predicted communication delay; and select the candidate allocation scheme with the smallest maximum predicted stage delay among all pipeline stages as the target allocation scheme. Migration execution module: Used to switch the execution affiliation of the corresponding candidate boundary layer to the target inference node that pre-stores the candidate boundary layer according to the target allocation scheme, and update the transmission path of intermediate activation data.

[0069] The candidate boundary layer pre-partitioning and redundant storage module can be set in the model deployment node; the stage latency monitoring module can be set in each inference node; the migration triggering module and migration decision module can be set in the coordination node; the migration execution module is implemented collaboratively by the coordination node and each inference node.

[0070] The specific implementation process of the system is the same as the bandwidth-aware model layer migration method for heterogeneous edge devices, and will not be described again.

[0071] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the bandwidth-aware model layer migration method for heterogeneous edge devices as described above.

[0072] The present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein when the computer program / instruction is executed by one or more processors, it implements the steps of the above-described bandwidth-aware model layer migration method for heterogeneous edge devices. Attached Figure Description

[0073] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating candidate boundary layer pre-partitioning, redundant storage, and model layer migration in an embodiment of the present invention. Detailed Implementation

[0074] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0075] This embodiment provides a bandwidth-aware model layer migration method for heterogeneous edge devices, using a Transformer-based autoregressive decoding model in a visual language model or large language model as an example. The model layer consists of Transformer layers connected in execution order. The system includes a coordinating node and N inference nodes that form a pipeline sequentially, where N≥2. Both the coordinating node and each inference node are edge devices. Each of the N inference nodes undertakes a pipeline stage and executes a set of model layers arranged sequentially according to the model layer execution order.

[0076] The coordinating node is used to aggregate the operational information of each inference node, determine the pipeline load imbalance, determine the target allocation scheme, and control the migration execution of the model layer; the coordinating node is implemented by an independent edge device. The method includes the following steps: (1) Pre-partitioning and Redundant Storage of Candidate Boundary Layer Sets: After the initial pipeline partitioning of the model to be inferred for multiple inference nodes is completed, based on the available storage capacity of the two inference nodes undertaking adjacent pipeline stages for redundant storage of model layers, a candidate boundary layer set is determined near the model computation boundary between the two inference nodes, and the model parameters of the candidate boundary layer set are pre-stored in both inference nodes simultaneously; specifically including: (1.1) Determining available storage capacity: Determine the total storage capacity, initial storage occupancy, and storage safety margin of each inference node. Based on the total storage capacity, initial storage occupancy, and storage safety margin, determine the available storage capacity of each inference node for redundant storage of candidate boundary layer model parameters. Among them, the initial storage occupancy includes the model parameter occupancy of the initial allocated model layer, the inference cache occupancy, and the intermediate activation data cache occupancy. (1.2) Candidate boundary layer set determination: For two inference nodes undertaking adjacent pipeline stages, based on the available storage capacity of each of the two inference nodes and the model parameter occupancy of the model layers initially allocated to each of the two inference nodes, model layers are continuously selected from both sides of the model computation boundary in a direction away from the model computation boundary. The model parameter occupancy of the model layers that need to be redundantly stored on the opposite inference node is accumulated. Under the condition that the accumulated model parameter occupancy does not exceed the available storage capacity of the corresponding inference node, the candidate boundary layer set corresponding to the model computation boundary is determined. For inference nodes that participate in both the upstream and downstream model computation boundaries, the sum of the model parameter occupancy of all candidate boundary layers that are additionally redundantly stored for the upstream and downstream model computation boundaries does not exceed the available storage capacity of the inference node. (1.3) Redundant storage of candidate boundary layers: The model parameters of each candidate boundary layer in the candidate boundary layer set are stored simultaneously on two inference nodes on both sides of the corresponding model calculation boundary, so that each candidate boundary layer can switch execution affiliation between the two inference nodes; (1.4) Determining the basic model layer: The model layers other than the candidate boundary layer in the initial allocation of each inference node are determined as the basic model layer. The basic model layer is continuously stored and executed by the corresponding inference node, and the model parameters of the candidate boundary layer are continuously stored in the two inference nodes on both sides of the corresponding model calculation boundary.

[0077] (2) Monitoring and smoothing of pipeline stage delays: During model inference, the computational and communication delays of each pipeline stage are collected to determine the stage delay; the sliding window length is determined based on the fluctuation of the current stage delay relative to the historical stage delay baseline; the stage delay is smoothed based on the sliding window to obtain the smoothed stage delay; and the communication delay is smoothed based on the sliding window to obtain the smoothed communication delay; specifically including: (2.1) Calculation of historical latency baseline: Each inference node maintains a historical latency sequence for the corresponding pipeline stage. The historical latency sequence records the stage latency of multiple consecutive inference batches. When the model inference starts, the sliding window length of each pipeline stage is initialized to the preset minimum window length. For the first inference batch, the current sliding window length is determined to be the preset minimum window length, and the obtained current stage latency is recorded in the historical latency sequence. For the current inference batch after the first inference batch, the nearest historical stage latency is selected from the historical latency sequence before the current inference batch according to the sliding window length determined by the previous inference batch. When the number of historical stage latencies in the historical latency sequence is not less than the sliding window length determined by the previous inference batch, the nearest historical stage latency corresponding to the number of sliding window lengths determined by the previous inference batch is selected. When the number of historical stage latencies is less than the sliding window length determined by the previous inference batch, all existing historical stage latencies in the historical latency sequence are selected, and the average historical stage latency is calculated based on the selected historical stage latency. (2.2) Calculation of latency fluctuation: For the current inference batch with a valid historical stage latency average, calculate the absolute difference between the current stage latency and the historical stage latency average, and determine the latency fluctuation based on the ratio of the absolute difference to the benchmark latency value, wherein the benchmark latency value is the larger value between the historical stage latency average and the preset positive latency lower limit. (2.3) Adjustment of sliding window length: When the time delay fluctuation is not greater than the preset fluctuation threshold, the current sliding window length is determined as the preset minimum window length; when the time delay fluctuation is greater than the preset fluctuation threshold, the current sliding window length is increased according to the time delay fluctuation, and the current sliding window length is limited to within the preset maximum window length. (2.4) Smoothing delay calculation: Based on the current sliding window length, the stage delay of the current inference batch and the previous multiple inference batches is averaged to obtain the smooth stage delay of the corresponding pipeline stage; based on the current sliding window length, the communication delay of the current inference batch and the previous multiple inference batches is averaged to obtain the smooth communication delay of the corresponding pipeline stage. (2.5) Operation information reporting: Each inference node sends the smoothing stage delay and smoothing communication delay of the corresponding pipeline stage to the coordinating node.

[0078] (3) Triggering judgment for model layer migration: The coordinating node determines the pipeline load imbalance degree and dynamic migration threshold based on the smoothing stage delay of each pipeline stage, and triggers model layer migration when the pipeline load imbalance degree exceeds the dynamic migration threshold and the migration cooling condition related to the sliding window length is met; specifically including: (3.1) Calculation of pipeline load imbalance: The coordination node obtains the smoothing stage delay of each pipeline stage, and determines the pipeline load imbalance by the difference between the maximum smoothing stage delay and the minimum smoothing stage delay in each pipeline stage. (3.2) Determination of dynamic migration threshold: Calculate the average value of the smoothing stage delay of each pipeline stage, and determine the dynamic migration threshold by multiplying the average value by the preset steady-state tolerance coefficient; (3.3) Determining the migration interval: When the model inference starts, the inference batch corresponding to the most recent model layer migration is initialized to 0; the migration interval is determined based on the current inference batch and the inference batch corresponding to the most recent model layer migration; after the model layer migration is completed, the inference batch corresponding to the most recent model layer migration is updated to the inference batch corresponding to the current model layer migration. (3.4) Model layer migration trigger: The migration cooling period is determined based on the current sliding window length used in each pipeline stage. The migration cooling period is not less than the maximum value of the current sliding window length used in each pipeline stage. When the pipeline load imbalance is greater than the dynamic migration threshold and the migration interval is greater than the migration cooling period, the model layer migration is triggered.

[0079] (4) Determination and migration execution of the target allocation scheme: The execution assignment of candidate boundary layers in the candidate boundary layer set corresponding to the computational boundary of each model is jointly combined to form multiple candidate allocation schemes; for each candidate allocation scheme, the prediction computation delay and prediction communication delay of each pipeline stage are determined, and the prediction stage delay of each pipeline stage is determined according to the corresponding prediction computation delay and prediction communication delay, and the candidate allocation scheme with the smallest maximum prediction stage delay in each pipeline stage is selected as the target allocation scheme; when there are multiple candidate allocation schemes with the same maximum prediction stage delay and all of them are the minimum, the candidate allocation scheme with the smallest difference between the maximum prediction stage delay and the minimum prediction stage delay is selected as the target allocation scheme; according to the target allocation scheme, the execution assignment of the corresponding candidate boundary layer is switched to the target inference node with the pre-stored candidate boundary layer, and the transmission path of the intermediate activation data is updated. Specifically, this includes: (4.1) Candidate Allocation Scheme Construction: For each set of candidate boundary layers corresponding to the model computation boundary, the number of candidate boundary layers to be executed by the upstream inference node is determined according to the execution order of the model layers, and the remaining candidate boundary layers are determined to be executed by the downstream inference node; the execution assignments of all candidate boundary layers corresponding to the model computation boundaries are jointly combined to form multiple candidate allocation schemes, and any candidate allocation scheme is denoted as . ; (4.2) Determination of prediction computation delay: For each candidate allocation scheme, the prediction computation delay of each pipeline stage is determined based on the execution set of the model layer corresponding to each inference node, the computation workload of each model layer, and the processing time required for the corresponding inference node to execute the unit model layer computation workload. (4.3) Determining the predicted communication delay: Based on the current smoothed communication delay, the amount of intermediate activation data at the current model calculation boundary, and the amount of intermediate activation data at the new model calculation boundary corresponding to the candidate allocation scheme K, determine the predicted communication delay for the corresponding pipeline stage; (4.4) Determination of prediction stage delay: The first stage delay will be determined by the prediction stage delay. The predicted computation delay and predicted communication delay of each pipeline stage are added together to obtain the candidate allocation scheme. Next Prediction stage delay for each pipeline stage; (4.5) Target allocation scheme selection: For each candidate allocation scheme, first determine its maximum prediction stage delay and use it as the prediction pipeline cycle corresponding to the candidate allocation scheme; select the candidate allocation scheme with the smallest prediction pipeline cycle from all candidate allocation schemes; when only one candidate allocation scheme has the minimum prediction pipeline cycle, select that candidate allocation scheme as the target allocation scheme; when multiple candidate allocation schemes have the same minimum prediction pipeline cycle, compare the difference between the maximum prediction stage delay and the minimum prediction stage delay of the multiple candidate allocation schemes, and select the candidate allocation scheme with the smallest difference as the target allocation scheme. (4.6) Model layer migration execution: The coordinating node determines the candidate boundary layer to be switched, the original inference node currently executing the candidate boundary layer, and the target inference node after migration according to the target allocation scheme; the original inference node stops the execution of the corresponding candidate boundary layer, the target inference node enables the pre-stored corresponding candidate boundary layer, and updates the intermediate activation data transmission path according to the boundary calculation of the migrated model; wherein, the model parameters of the candidate boundary layer are continuously stored in the original inference node and the target inference node, and the model parameters of the candidate boundary layer are not transmitted between the original inference node and the target inference node during the migration execution process.

[0080] like Figure 1 As shown, the method in this embodiment includes pre-partitioning and redundant storage of candidate boundary layer sets, monitoring and smoothing of pipeline stage delays, triggering judgment of model layer migration, and determination and execution of target allocation schemes, specifically including the following steps.

[0081] (1) Pre-partitioning and redundant storage of candidate boundary layer sets Before model inference begins, the model is initially pipelined based on the computing power, storage capacity, and model parameter usage of each model layer of each device, and a set of candidate boundary layers is determined near the model computation boundary of adjacent devices.

[0082] (1.1) Available storage capacity can be used to determine: Obtain the total storage capacity of device i, the storage capacity occupied by the initial model deployment, and the reserved storage safety margin, and determine the available storage capacity of device i for the redundant storage candidate boundary layer: in, This indicates the available storage capacity of device i. This represents the total storage capacity of device i. This indicates the storage usage after the initial model deployment. This indicates the reserved storage safety margin for device i.

[0083] The initial storage usage includes the usage of model parameters in the initial allocation model layer, the usage of inference cache, and the usage of intermediate activation data cache; in the autoregressive decoding model of this embodiment, the inference cache includes key-value cache.

[0084] (1.2) Determination of the candidate boundary layer set: For devices i and i+1 that undertake adjacent pipeline stages, consecutive model layers are selected from both sides of the model calculation boundary according to the execution order of the model layers, based on the available storage capacity of each device and the model parameter occupancy of each model layer near the corresponding model calculation boundary. Candidate model layers from the initial model layer set of either device need to be redundantly stored on the opposite device.

[0085] When the model parameter usage of each model layer is approximately the same and is selected equally from both sides of the model calculation boundary, the number of model layers selected from each side is determined by the smaller value of the ratio of the available storage capacity of the two devices to the average storage usage of a single model layer; when the model parameter usage of each model layer is different or the number of selections on both sides is different, the model parameter usage that needs to be redundantly stored on the opposite side device is accumulated layer by layer, so that the cumulative usage of candidate model layers from device i does not exceed the available storage capacity of device i+1, and the cumulative usage of candidate model layers from device i+1 does not exceed the available storage capacity of device i.

[0086] For intermediate devices such as Device 2 that participate in both upstream and downstream model computation boundaries, the model parameter occupancy of all candidate boundary layers for which they have additional redundant storage for both model computation boundaries is calculated by merging the calculations, ensuring that the sum of the model parameter occupancy does not exceed the available storage capacity of the intermediate device. The candidate boundary layer sets for each model computation boundary can be determined jointly or sequentially according to a preset order. When using the sequential determination method, after determining the candidate boundary layer set for the previous model computation boundary, the remaining available storage capacity of the intermediate device is updated based on the redundant storage capacity it has already occupied, and the candidate boundary layer set for the next model computation boundary is determined based on the updated remaining available storage capacity.

[0087] (1.3) Redundant storage of candidate boundary layers: The model parameters of each candidate boundary layer in the candidate boundary layer set determined by the model calculation boundary between device i and device i+1 are stored in both device i and device i+1.

[0088] Candidate boundary layers are redundantly stored during the model deployment phase. During online inference, only the activation state of the candidate boundary layers is adjusted; the model parameters of the candidate boundary layers are no longer transmitted.

[0089] (1.4) Determination of the basic model layer: The model layers initially allocated to device i, excluding the candidate boundary layer, are determined as the basic model layer set of device i. The set of model layers continuously stored by device i is: in, This represents the set of candidate boundary layers between device i and its upstream devices. This represents the set of candidate boundary layers between device i and downstream devices. For device 1, For device N, .

[0090] like Figure 2 As shown, this embodiment uses device 1, device 2, and device 3, as well as model layers 1 to 11, as examples for illustration.

[0091] Device 1 stores model layers 1 to 5, where model layers 1 to 3 are the basic model layers, and model layers 4 and 5 are candidate boundary layers.

[0092] Device 2 stores model layers 4 to 8, where model layer 6 is the basic model layer, and model layers 4, 5, 7 and 8 are candidate boundary layers.

[0093] Device 3 stores model layers 7 to 11, where model layers 9 to 11 are the basic model layers, and model layers 7 and 8 are candidate boundary layers.

[0094] The candidate boundary layer set between device 1 and device 2 is as follows: The candidate boundary layer set between device 2 and device 3 is as follows: In the initial deployment state, device 1 activates model layers 1 to 4, device 2 activates model layers 5 to 8, and device 3 activates model layers 9 to 11, that is: Among them, model layer 5 in device 1, model layer 4 in device 2, and model layers 7 and 8 in device 3 are inactive, but the corresponding model parameters are continuously stored in the corresponding devices.

[0095] Figure 2 The diagram shows the basic model layer, the currently active candidate boundary layer, and the currently inactive candidate boundary layer to distinguish the type and execution state of the model layer.

[0096] (2) Monitoring and smoothing of pipeline stage delays During the model inference process, each device collects the computational latency and communication latency of the corresponding pipeline stage.

[0097] The stage delay of device i in the t-th inference batch is: in, This indicates the computational latency required for device i to execute the currently active model layer. Indicates device i to device The communication latency required to transmit intermediate activation data. Decreased inter-device link bandwidth or network congestion increases communication latency, which is reflected in the stage latency. Therefore, this embodiment uses stage latency, which includes communication latency, as the basis for sensing device operating status and changes in link bandwidth.

[0098] (2.1) Calculation of historical delay benchmark: Device i maintains a historical stage delay sequence for the corresponding pipeline stage. This historical stage delay sequence records the stage delay of each completed inference batch according to the inference batch order. When model inference starts, the sliding window length for the corresponding pipeline stage of device i is initialized to a preset minimum window length. .

[0099] For the first inference batch, the historical average latency and latency fluctuation are not calculated, and the current sliding window length is used. Determined as ; Obtain the stage delay of the first inference batch Then, the delay of this stage is added to the historical stage delay sequence.

[0100] For the t-th inference batch after the first inference batch, before adding the current stage delay to the historical stage delay sequence, the sliding window length determined by the previous inference batch is used. The number of valid historical samples is determined by considering the number of valid historical stage delays prior to the current inference batch. : in, This represents the number of valid historical stage delays prior to the t-th inference batch. Device i selects the historical stage delay sequence that is closest to the current inference batch. The historical latency is calculated for each stage, and the historical latency baseline is determined accordingly. in, This represents the average historical stage latency of device i up to the t-th inference batch. When the number of historical samples is less than a complete sliding window, the latency of all existing historical stages is used; when the number of historical samples reaches the length of the sliding window, the latency of the most recent historical stage corresponding to the length of the sliding window is used.

[0101] For example, when When the value is 4, the first inference batch only records the stage delay; the second, third, and fourth inference batches use the delays of the first, second, and third historical stages to establish historical delay benchmarks, respectively; from the fifth inference batch onwards, the delays of the four most recent historical stages are used, and there is no need to manually fill in the historical data.

[0102] (2.2) Calculation of time delay fluctuation: For the current inference batch following the first inference batch and having a valid historical latency benchmark, the latency fluctuation is calculated based on the normalized deviation between the current latency and the historical latency benchmark. To avoid distortion of the normalization result when the historical latency mean is zero or abnormally close to zero, a preset positive latency lower limit is introduced. The larger of the historical latency average and the preset positive latency lower limit is used as the normalization benchmark: in, This indicates the latency fluctuation of the pipeline stage corresponding to device i in the t-th inference batch; Indicates the current stage delay; This represents the average latency of historical periods preceding the current inference batch; This indicates a preset positive lower limit for delay, and Under normal operating conditions, the stage latency includes the model layer calculation latency, and its historical stage latency average is usually greater than zero; the preset positive latency lower limit is used to protect against abnormal zero values ​​or values ​​close to zero. The greater the latency fluctuation, the more significant the change in the device's operating state or the communication link between devices relative to the historical state.

[0103] (2.3) Adjusting the length of the sliding window: For the first inference batch, since a historical latency benchmark has not yet been established, no sliding window adjustment based on latency fluctuation is performed; the current sliding window length is maintained at the preset minimum window length. For subsequent inference batches after a valid historical latency benchmark has been established, the sliding window length is adjusted according to the latency fluctuation using the following formula: in, and These represent the minimum window length and the maximum window length, respectively. Indicates the fluctuation threshold. This indicates the window scaling factor.

[0104] When the time delay fluctuation is small, a shorter sliding window is used to improve the speed of state change perception; when the time delay fluctuation is large, the sliding window is increased to reduce the possibility of false triggering of migration caused by instantaneous time delay jitter.

[0105] (2.4) Calculation of delay during the smoothing phase: Add the current stage latency to the historical stage latency sequence. Determine the actual number of smoothed samples based on the smaller of the current sliding window length and the number of effective stage latencies obtained up to the current inference batch. When the number of effective stage delays is less than the current sliding window length, all existing effective stage delays are used; when the number of effective stage delays reaches or exceeds the current sliding window length, the nearest stage delay corresponding to the current sliding window length is used. The smoothing stage delay is calculated based on the actual number of smoothed samples. The smoothed communication latency is calculated using the same window shortage handling method and the actual number of smoothed samples: (2.5) Reporting of operational information: Each device reports the smoothing stage latency and smoothing communication latency to the coordination node; before constructing the candidate allocation scheme, it also reports the current active model layer set and the amount of intermediate active data at the current model calculation boundary to the coordination node.

[0106] In the first inference batch of this embodiment, each device completes the acquisition of stage latency and communication latency, initialization of historical sequences, smoothing latency calculation based on existing samples, and reporting of operational information. However, it does not calculate latency fluctuation, perform sliding window adjustment based on latency fluctuation, or perform model layer migration trigger judgment. From subsequent inference batches onwards, when each pipeline stage can form a valid historical latency benchmark, the normal latency fluctuation calculation, sliding window adjustment, smoothing latency calculation, and model layer migration trigger judgment process begins.

[0107] (3) Triggering judgment for model layer migration The coordinating node determines whether there is a persistent load imbalance in the current pipeline based on the smoothing phase delay reported by each device.

[0108] (3.1) Calculation of the degree of load imbalance in the production line: The difference between the maximum and minimum smoothing stage delays in each pipeline stage is defined as the degree of pipeline load imbalance. (3.2) Determination of dynamic migration threshold: Calculate the average smoothing stage delay across all pipeline stages: The dynamic migration threshold is determined based on the average delay during the smoothing phase. in, γ represents the preset steady-state tolerance coefficient. N represents the number of pipeline stages; the average global smoothing stage delay is the average smoothing stage delay of the N pipeline stages under the current inference batch, and the dynamic migration threshold is the product of this average value and the steady-state tolerance coefficient, where 0 < γ < 1.

[0109] (3.3) Determining the migration interval: Inference batches are numbered starting from 1. At the start of model inference, a migration batch variable is used to record the inference batch corresponding to the most recent model layer migration. Initialize to 0; in the t-th inference batch, combine the current inference batch t with... The difference is determined as the migration interval. After each model layer migration is completed, the... Update to the inference batch corresponding to this model layer migration.

[0110] Set migration cooling cycle The migration cooling cycle is not less than the maximum value among the current sliding window lengths of each device, so that the historical latency data before migration can exit the sliding window before the migration judgment is made again.

[0111] (3.4) Model layer migration triggering: When the pipeline load imbalance exceeds the dynamic migration threshold and the migration interval exceeds the migration cooling cycle, model layer migration is triggered. when At that time, each device maintains the current model layer allocation scheme; when At that time, the coordinating node initiates the process of determining the target allocation scheme and executing the model layer migration.

[0112] (4) Determination and migration execution of the target allocation plan The coordinating node jointly determines the execution ownership of the candidate boundary layer based on the set of candidate boundary layers among all adjacent devices.

[0113] (4.1) Construction of candidate allocation schemes: For device j and device Candidate boundary layer set between Set allocation variables , which represents the number of model layers in the candidate boundary layer set that are activated and executed by device j. The remaining candidate boundary layers in the candidate boundary layer set are activated and executed by device j+1.

[0114] The assignment variable satisfies: All allocation variables corresponding to the computational boundaries of the model constitute candidate allocation schemes: Based on the candidate allocation scheme K, reconstruct the model layer execution set corresponding to device i. The refactored model layer execution set maintains the continuous execution order of the models, and there is no duplicate or omitted execution of model layers.

[0115] (4.2) Determination of prediction calculation delay: Based on the model layer execution set of device i under candidate allocation scheme K, the computational workload of each model layer, and the time required for device i to perform a unit computational workload, determine the predicted computational latency of the i-th pipeline stage: in, This represents the computational workload of model layer l. This represents the time required for device i to perform a unit of computational work. The summation range of this expression is the set of model layer executions of device i under candidate allocation scheme K, and the left side of the equation represents the predicted computational latency of the i-th pipeline stage under candidate allocation scheme K.

[0116] (4.3) Determining the predicted communication delay: Based on the current smoothed communication latency and the amount of intermediate activation data before and after adjusting the model calculation boundary, the predicted communication latency is determined as follows: in, This represents the amount of intermediate activated data at the current model's computational boundary. This represents the amount of intermediate activation data at the computational boundary of the new model corresponding to candidate allocation scheme K. The left side of the equation represents the prediction communication latency of the i-th pipeline stage under candidate allocation scheme K; the smoothed communication latency on the right side of the equation represents the smoothed communication latency of the i-th pipeline stage in the current inference batch.

[0117] (4.4) Determination of prediction stage delay: The prediction calculation delay and the prediction communication delay are added together to obtain the prediction stage delay: (4.5) Selection of target allocation scheme: For any candidate allocation scheme The maximum prediction stage delay in each pipeline stage under the candidate allocation scheme is determined as the prediction pipeline cycle corresponding to the candidate allocation scheme: in, Indicates the number of stages in the production line. Represents candidate allocation schemes The corresponding predicted pipeline cycle, Represents candidate allocation schemes Next The prediction stage delay for each pipeline stage; determining the minimum prediction pipeline cycle from all candidate allocation schemes: Candidate allocation schemes whose predicted pipeline cycle is equal to the minimum predicted pipeline cycle are determined as preferred candidate allocation schemes; when there is only one preferred candidate allocation scheme, it is determined as the target allocation scheme; when there are multiple preferred candidate allocation schemes, the difference between the maximum prediction stage delay and the minimum prediction stage delay corresponding to each preferred candidate allocation scheme is determined, and the preferred candidate allocation scheme with the smallest difference is determined as the target allocation scheme. in, Indicates the target allocation scheme; (4.6) Model layer migration execution: The coordinating node determines the candidate boundary layer to which the execution ownership needs to be switched based on the target allocation scheme, and sends activation instructions, freeze instructions, and intermediate activation data routing update information to the corresponding devices.

[0118] The original execution device freezes the corresponding candidate boundary layer, preventing it from participating in subsequent model inference calculations, but continues to store the corresponding model parameters; the target execution device activates the same pre-stored candidate boundary layer, adding it to the target execution device's model layer execution set.

[0119] like Figure 2 As shown, when it is necessary to reduce the load of the corresponding pipeline stage of device 2, the execution ownership of model layer 5 is adjusted from device 2 to device 1.

[0120] Before the migration, the activation model layer sets for Device 1, Device 2, and Device 3 were as follows: During migration, device 1 activates the pre-stored candidate boundary layer 5, while device 2 freezes candidate boundary layer 5. After migration, the sets of activated model layers for each device are as follows: Before the migration, the model computation boundary between device 1 and device 2 was located between model layer 4 and model layer 5. Device 1 sent the intermediate activation data output from model layer 4 to device 2.

[0121] After migration, the model calculation boundary moves to between model layer 5 and model layer 6. Device 1 sends the intermediate activation data output from model layer 5 to device 2, and device 2 continues execution from model layer 6.

[0122] Model layer 5 is already stored on both device 1 and device 2 during the model deployment phase. Therefore, the above process only changes the activation state of model layer 5 and the intermediate activation data transmission path, and does not transmit the model parameters of model layer 5 between device 1 and device 2.

[0123] After the migration is complete, the coordinating node updates the current active model layer set, model computation boundaries, intermediate active data routing information, and the inference batch corresponding to the most recent migration for each device, and then enters the migration cooldown period. Afterward, each device continues to perform phase latency monitoring to adapt to subsequent changes in device computation status and inter-device link bandwidth.

[0124] This embodiment uses device 1, device 2, device 3, and model layers 1 to 11 as examples for illustration. In other embodiments, the number of devices, model layers, and candidate boundary layers can be adjusted according to the structure of the model to be inferred and the resource conditions of the edge devices, but the determination of the candidate boundary layer set still satisfies the following: the cumulative model parameter occupancy of the candidate model layers that need to be redundantly stored on the opposite device does not exceed the available storage capacity of the opposite device.

[0125] Obviously, those skilled in the art should understand that the steps of the bandwidth-aware model layer migration method for heterogeneous edge devices or the modules of the bandwidth-aware model layer migration system for heterogeneous edge devices described in the above embodiments of the present invention can be implemented using general-purpose computing devices. These modules can be centralized on a single computing device or distributed across a network of multiple computing devices working collaboratively. Optionally, they can be implemented using computer-executable program code, which can be stored in a storage device and executed by the computing device. Furthermore, without changing the necessary sequential relationship and technical logic between the steps, some steps can be executed in parallel or their execution timing can be adjusted. Alternatively, each step can be implemented as an independent integrated circuit module, or multiple modules or steps can be combined. Thus, the embodiments of the present invention are not limited to any specific hardware architecture or specific software implementation method.

Claims

1. A bandwidth-aware model layer migration method for heterogeneous edge devices, characterized in that, A pipelined parallel inference system comprising a coordinating node and multiple inference nodes, wherein the coordinating node and inference nodes are both edge devices; the method includes the following steps: (1) Pre-partitioning and redundant storage of candidate boundary layer set: After the initial pipeline partitioning of the model to be inferred is completed for multiple inference nodes, the candidate boundary layer set is determined near the model computation boundary between the two inference nodes based on the available storage capacity of the two inference nodes undertaking adjacent pipeline stages for redundant storage of model layers, and the model parameters of the candidate boundary layer set are simultaneously pre-stored in the two inference nodes. (2) Monitoring and smoothing of pipeline stage delay: During the model inference process, the computation delay and communication delay of each pipeline stage are collected to determine the stage delay; the sliding window length is determined according to the fluctuation of the current stage delay relative to the historical stage delay benchmark; the stage delay is smoothed based on the sliding window to obtain the smoothed stage delay; and the communication delay is smoothed based on the sliding window to obtain the smoothed communication delay. (3) Triggering judgment of model layer migration: The coordinating node determines the pipeline load imbalance degree and dynamic migration threshold based on the smoothing stage delay of each pipeline stage, and triggers model layer migration when the pipeline load imbalance degree exceeds the dynamic migration threshold and the migration cooling condition related to the sliding window length is met. (4) Determination and migration execution of target allocation scheme: The execution assignment of candidate boundary layers in the candidate boundary layer set corresponding to the computational boundary of each model is jointly combined to form multiple candidate allocation schemes; for each candidate allocation scheme, the prediction computation delay and prediction communication delay of each pipeline stage are determined, the prediction stage delay of each pipeline stage is determined according to the corresponding prediction computation delay and prediction communication delay, and the candidate allocation scheme with the smallest maximum prediction stage delay in each pipeline stage is selected as the target allocation scheme; When multiple candidate allocation schemes have the same maximum prediction stage delay and are all minimum, the candidate allocation scheme with the smallest difference between the maximum prediction stage delay and the minimum prediction stage delay is selected as the target allocation scheme. According to the target allocation scheme, the execution affiliation of the corresponding candidate boundary layer is switched to the target inference node that pre-stores the candidate boundary layer, and the transmission path of the intermediate activation data is updated.

2. The bandwidth-aware model layer migration method for heterogeneous edge devices according to claim 1, characterized in that, The pre-partitioning and redundant storage of the candidate boundary layer set (1) includes the following steps: (1.1) Determining available storage capacity: Determine the total storage capacity, initial storage occupancy, and storage safety margin of each inference node. Based on the total storage capacity, initial storage occupancy, and storage safety margin, determine the available storage capacity of each inference node for redundant storage of candidate boundary layer model parameters; wherein, the initial storage occupancy includes the model parameter occupancy of the initial allocated model layer, the inference cache occupancy, and the intermediate activation data cache occupancy. (1.2) Determination of candidate boundary layer set: For two inference nodes undertaking adjacent pipeline stages, model layers are continuously selected from both sides of the model computation boundary along the direction away from the model computation boundary, based on the available storage capacity of each of the two inference nodes and the model parameter occupancy of the model layers initially allocated to each of the two inference nodes; the model parameter occupancy of the model layers that need to be redundantly stored on the opposite side inference node is accumulated, and the candidate boundary layer set corresponding to the model computation boundary is determined under the condition that the accumulated model parameter occupancy does not exceed the available storage capacity of the corresponding opposite side inference node; for inference nodes that participate in both the upstream and downstream model computation boundaries, the sum of the model parameter occupancy of all candidate boundary layers that are additionally redundantly stored for the upstream and downstream model computation boundaries does not exceed the available storage capacity of the inference node; (1.3) Redundant storage of candidate boundary layers: The model parameters of each candidate boundary layer in the candidate boundary layer set are stored simultaneously on two inference nodes on both sides of the corresponding model calculation boundary, so that each candidate boundary layer can switch execution affiliation between the two inference nodes; (1.4) Determining the basic model layer: The model layers other than the candidate boundary layer in the initial allocation of each inference node are determined as the basic model layer. The basic model layer is continuously stored and executed by the corresponding inference node, and the model parameters of the candidate boundary layer are continuously stored in the two inference nodes on both sides of the corresponding model calculation boundary.

3. The bandwidth-aware model layer migration method for heterogeneous edge devices according to claim 1, characterized in that, The monitoring and smoothing of pipeline stage delays (2) includes the following steps: (2.1) Calculation of historical latency benchmark: Each inference node maintains a historical latency sequence for the corresponding pipeline stage, and the historical latency sequence records the stage latency of multiple consecutive inference batches; when the model inference starts, the sliding window length of each pipeline stage is initialized to the preset minimum window length; For the first inference batch, the current sliding window length is determined as the preset minimum window length, and the obtained current stage delay is recorded in the historical delay sequence. For the current inference batch after the first inference batch, the nearest historical stage delay is selected from the historical delay sequence before the current inference batch, based on the sliding window length determined by the previous inference batch. When the number of historical stage delays in the historical delay sequence is not less than the sliding window length determined by the previous inference batch, the nearest historical stage delay corresponding to the sliding window length determined by the previous inference batch is selected. When the number of historical stage delays is less than the sliding window length determined by the previous inference batch, all existing historical stage delays in the historical delay sequence are selected, and the average historical stage delay is calculated based on the selected historical stage delays. (2.2) Calculation of latency fluctuation: For the current inference batch with a valid historical average latency, calculate the absolute difference between the current latency and the historical average latency, and determine the latency fluctuation based on the ratio of the absolute difference to the baseline latency value, wherein the baseline latency value is the larger of the historical average latency and a preset positive latency lower limit, and the latency fluctuation is determined according to the following formula: in, Indicates the stage number of the production line. Indicates the current inference batch number; This represents the latency variation of the i-th pipeline stage in the t-th inference batch. Indicates the current stage delay. This represents the average historical stage latency calculated based on the effective historical stage latency selected before the current inference batch. This indicates a preset positive lower limit for delay, and ; (2.3) Sliding window length adjustment: When the latency fluctuation is not greater than a preset fluctuation threshold, the current sliding window length is determined as the preset minimum window length; when the latency fluctuation is greater than the preset fluctuation threshold, the current sliding window length is increased according to the latency fluctuation, and the current sliding window length is limited to within a preset maximum window length. The current sliding window length is determined according to the following formula: in, This represents the sliding window length of the i-th pipeline stage in the t-th inference batch; Indicates the preset minimum window length; Indicates the preset maximum window length; Indicates the preset fluctuation threshold; Indicates the window scaling factor; (2.4) Smoothing delay calculation: Based on the current sliding window length, the stage delay of the current inference batch and the previous multiple inference batches is averaged to obtain the smooth stage delay of the corresponding pipeline stage; based on the current sliding window length, the communication delay of the current inference batch and the previous multiple inference batches is averaged to obtain the smooth communication delay of the corresponding pipeline stage. (2.5) Operation information reporting: Each inference node sends the smoothing stage delay and smoothing communication delay of the corresponding pipeline stage to the coordination node.

4. The bandwidth-aware model layer migration method for heterogeneous edge devices according to claim 1, characterized in that, The triggering judgment for model layer migration in (3) includes the following steps: (3.1) Calculation of pipeline load imbalance: The coordination node obtains the smoothing stage delay of each pipeline stage, and determines the pipeline load imbalance degree by the difference between the maximum smoothing stage delay and the minimum smoothing stage delay in each pipeline stage. (3.2) Determination of dynamic migration threshold: Calculate the average smoothing stage delay of each pipeline stage, and determine the dynamic migration threshold by multiplying the average value by the preset steady-state tolerance coefficient; the dynamic migration threshold is determined according to the following formula: in, Indicates the stage number of the production line. Indicates the current inference batch number; Indicates the first The dynamic migration threshold corresponding to each inference batch; This represents the preset steady-state tolerance coefficient, and N represents the number of stages in the production line; Indicates the first The first stage of the production line is in the Smoothing phase latency for each inference batch; (3.3) Determining the migration interval: When the model inference starts, the inference batch corresponding to the most recent model layer migration is initialized to 0; the migration interval is determined based on the current inference batch and the inference batch corresponding to the most recent model layer migration; after the model layer migration is completed, the inference batch corresponding to the most recent model layer migration is updated to the inference batch corresponding to the current model layer migration. (3.4) Model Layer Migration Trigger: The migration cooling period is determined based on the sliding window length currently used in each pipeline stage, and the migration cooling period is not less than the maximum value among the sliding window lengths currently used in each pipeline stage; the migration cooling period is determined according to the following formula: in Indicates the migration cooling cycle. Indicates the first The first stage of the production line is in the The sliding window length used in each inference batch; when the pipeline load imbalance is greater than the dynamic migration threshold and the migration interval is greater than the migration cooling cycle, model layer migration is triggered.

5. The bandwidth-aware model layer migration method for heterogeneous edge devices according to claim 1, characterized in that, The determination and migration execution of the target allocation scheme in (4) includes the following steps: (4.1) Candidate allocation scheme construction: For each set of candidate boundary layers corresponding to the model computation boundary, determine the number of candidate boundary layers to be executed by the upstream inference node according to the execution order of the model layer, and determine the remaining candidate boundary layers to be executed by the downstream inference node; The candidate boundary layers corresponding to the computational boundaries of all models are jointly assigned to form multiple candidate allocation schemes. Any candidate allocation scheme is denoted as... ; (4.2) Determination of Predicted Calculation Latency: For each candidate allocation scheme, the predicted calculation latency of each pipeline stage is determined based on the execution set of the model layer corresponding to each inference node, the computational workload of each model layer, and the processing time required for the corresponding inference node to execute the computational workload of a unit model layer. The calculation formula is as follows: in, This indicates the inference node and its corresponding pipeline stage number. Indicates the model layer number; Represents candidate allocation schemes Next Predictive calculation delay for each stage of the pipeline; Represents candidate allocation schemes Next The execution set of the model layer corresponding to each inference node; Indicates the first The computational workload of each model layer; Indicates the first The processing time required for each inference node to perform unit model layer computation workload; (4.3) Predicted Communication Delay Determination: Based on the current smoothed communication delay, the amount of intermediate activation data at the current model calculation boundary, and the amount of intermediate activation data at the new model calculation boundary corresponding to candidate allocation scheme K, the predicted communication delay for the corresponding pipeline stage is determined; the predicted communication delay is determined according to the following formula: in, Represents candidate allocation schemes Next Predicted communication latency for each pipeline stage; Indicates the current inference batch number; Indicates the first The current smooth communication latency of each pipeline stage; This represents the amount of intermediate activation data that needs to be transmitted downstream from the current model's computational boundary. Represents candidate allocation schemes The corresponding new model calculates the amount of intermediate activation data that needs to be transmitted downstream at the boundary; for pipeline stages located at the end of the pipeline, the prediction communication latency is set to zero. (4.4) Determination of prediction stage delay: The first stage delay will be determined by the prediction stage delay. The predicted computation delay and predicted communication delay of each pipeline stage are added together to obtain the candidate allocation scheme. Next Prediction stage delay for each pipeline stage; (4.5) Target allocation scheme selection: For any candidate allocation scheme The maximum prediction stage delay in each pipeline stage under the candidate allocation scheme is determined as the prediction pipeline cycle corresponding to the candidate allocation scheme: in, Indicates the number of stages in the production line. Represents candidate allocation schemes The corresponding predicted pipeline cycle, Represents candidate allocation schemes Next The prediction stage delay for each pipeline stage; determining the minimum prediction pipeline cycle from all candidate allocation schemes: Candidate allocation schemes whose predicted pipeline cycle is equal to the minimum predicted pipeline cycle are determined as preferred candidate allocation schemes; when there is only one preferred candidate allocation scheme, it is determined as the target allocation scheme; when there are multiple preferred candidate allocation schemes, the difference between the maximum prediction stage delay and the minimum prediction stage delay corresponding to each preferred candidate allocation scheme is determined, and the preferred candidate allocation scheme with the smallest difference is determined as the target allocation scheme. in, Indicates the target allocation scheme; (4.6) Model layer migration execution: The coordinating node determines the candidate boundary layer to be switched, the original inference node currently executing the candidate boundary layer, and the target inference node after migration according to the target allocation scheme; the original inference node stops the execution of the corresponding candidate boundary layer, the target inference node enables the pre-stored corresponding candidate boundary layer, and updates the intermediate active data transmission path according to the boundary calculation of the migrated model; wherein, the model parameters of the candidate boundary layer are continuously stored in the original inference node and the target inference node, and the model parameters of the candidate boundary layer are not transmitted between the original inference node and the target inference node during the migration execution.

6. The bandwidth-aware model layer migration method for heterogeneous edge devices according to claim 1, characterized in that, The model to be inferred is a visual language model or a large language model. The model layer includes multiple Transformer layers connected in execution order. Each inference node executes consecutive Transformer layers. Intermediate activation data output from the upstream pipeline stage is transmitted between inference nodes corresponding to adjacent pipeline stages. The candidate boundary layer set consists of at least one consecutive Transformer layer located near the model computation boundary of adjacent pipeline stages.

7. A bandwidth-aware model layer migration system for heterogeneous edge devices, characterized in that, It includes a coordinating node, multiple inference nodes, and the following modules, wherein the coordinating node and inference nodes are both edge devices: Candidate boundary layer pre-partitioning and redundant storage module: After the initial pipeline partitioning of the model to be inferred for multiple inference nodes is completed, the module determines a set of candidate boundary layers near the model computation boundary between the two inference nodes based on the available storage capacity of the two inference nodes undertaking adjacent pipeline stages for redundant storage of model layers, and pre-stores the model parameters of the candidate boundary layer set in the two inference nodes simultaneously. Stage latency monitoring module: used to collect the computation latency and communication latency of each pipeline stage during model inference, and determine the stage latency of each pipeline stage accordingly; determine the sliding window length based on the fluctuation of the current stage latency relative to the historical stage latency benchmark, smooth the stage latency of each pipeline stage based on the sliding window to obtain smooth stage latency, and smooth the communication latency based on the sliding window to obtain smooth communication latency; Migration triggering module: executed by the coordination node, used to determine the pipeline load imbalance degree and dynamic migration threshold based on the smoothing stage delay of each pipeline stage, and trigger model layer migration when the pipeline load imbalance degree exceeds the dynamic migration threshold and the migration cooling condition related to the sliding window length is met; Migration decision module: executed by the coordination node, used to jointly combine the candidate boundary layers in the candidate boundary layer set corresponding to the computational boundary of each model to form multiple candidate allocation schemes; for each candidate allocation scheme, determine the predicted computation delay and predicted communication delay of each pipeline stage; determine the predicted stage delay of each pipeline stage based on the corresponding predicted computation delay and predicted communication delay; and select the candidate allocation scheme with the smallest maximum predicted stage delay among all pipeline stages as the target allocation scheme. Migration execution module: Used to switch the execution affiliation of the corresponding candidate boundary layer to the target inference node that pre-stores the candidate boundary layer according to the target allocation scheme, and update the transmission path of intermediate activation data.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the bandwidth-aware model layer migration method for heterogeneous edge devices as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by one or more processors, they implement the steps of the bandwidth-aware model layer migration method for heterogeneous edge devices as described in any one of claims 1-6.

Images