A model parallel inference scheduling method

Abstract

This invention discloses a model-parallel inference scheduling method in the field of natural language processing, applied to a management platform that communicates with multiple edge servers. The method includes: acquiring deep neural network inference tasks based on IoT devices, wherein the inference task includes model type, data batch, privacy requirements, and maximum tolerable completion time; determining model segmentation points on the IoT devices based on the privacy requirements; performing model layer computations before the segmentation point locally; and sending intermediate data and remaining computation tasks corresponding to the model layers after the segmentation point to the management platform through an access point. This invention, by obtaining task load prediction data for multiple future time slots, enables a better balance in the allocation of computing resources during the inference task scheduling process, which is beneficial for improving the completion rate of inference tasks.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically a model-parallel inference scheduling method. Background Technology

[0002] Parallel inference technology for language models plays a crucial role in modern Natural Language Processing (NLP). With the convergence of IoT and AI technologies, a large number of deep neural network inference tasks have emerged in smart IoT scenarios. These inference tasks are typically generated by resource-constrained IoT devices. Due to data privacy protection and computational power limitations, IoT devices tend to perform a portion of the model's computation locally, while offloading the remaining computational tasks to edge servers closer to the data source. This forms a distributed inference model. In the dynamic and uncertain IoT environment, the generation process of inference tasks is dynamic, and therefore the available computing resources of the edge servers fluctuate in real time due to changes in task load.

[0003] Existing inference task scheduling methods, such as greedy algorithms or static optimization algorithms, typically only calculate based on the system state of the current time slot, without considering the impact of time dimensions such as the long execution cycle of inference tasks and cross-time slot resource coupling. This leads to problems such as unbalanced allocation of computing resources and low completion rate of inference tasks under high load scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a model parallel inference scheduling method. By generating periodic Bernoulli processes based on IoT devices, probability distributions of model types, probability distributions of privacy requirements, and random distributions of channel transmission rates, task load prediction data for multiple future time slots can be obtained. This enables the scheduling process of inference tasks to better balance the allocation of computing resources, which is beneficial to improving the completion rate of inference tasks.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A model parallel inference scheduling method is applied to a management platform, which communicates with multiple edge servers, including: The deep neural network inference task is obtained based on IoT devices. The inference task includes model type, data batch, privacy requirements and maximum tolerable completion time. The model segmentation point of the IoT device is determined based on the privacy requirements. The model layer calculation before the segmentation point is executed locally. The intermediate data and remaining calculation tasks corresponding to the model layer after the segmentation point are sent to the management platform through the access point. Obtain the real-time load status and computing power of each edge server; Based on the periodic Bernoulli process generated by IoT devices, the probability distribution of model types, the probability distribution of privacy requirements, and the random distribution of channel transmission rates, task load prediction data for multiple future time slots is obtained. Based on the real-time load status, computing power, and task load prediction data, a rolling time-domain framework is used to enumerate all edge server allocation data corresponding to the inference task in the current time slot to obtain the target edge server and the estimated completion time. Based on the target edge server within a finite time domain of the prediction window, a greedy algorithm is used to accumulate the single-time slot completion progress reward of all tasks being processed in each time slot based on the task load prediction. The allocation data with the largest accumulated completion progress reward is selected as the target allocation data for the current time slot. The inference task and the corresponding intermediate data are distributed to the target edge server, and calculations are performed based on the model layer after the segmentation point of the target edge server. Obtain the inference results returned by the target edge server, and based on the actual completion time of the inference task and the estimated completion time, obtain the latest task completion rate and the reputation value of the management platform.

[0006] As a further aspect of the present invention: the real-time load status includes the completion time of all pending tasks on each server within the current and future time slots.

[0007] As a further aspect of the present invention: the task load prediction data includes expected data on the number of tasks arriving and expected data on the task computation load.

[0008] As a further aspect of the present invention: acquiring deep neural network inference tasks based on IoT devices, including: The time for acquiring deep neural network inference tasks is divided into multiple discrete time slots; at the beginning of each integer time slot, all inference tasks arriving in the previous time slot are acquired to obtain the deep neural network inference task.

[0009] As a further aspect of the present invention: determining the model segmentation point of an IoT device based on the aforementioned privacy requirements includes: Based on different segmentation points, the structural similarity between the reconstructed data obtained from the intermediate data and the original data for each deep neural network model is calculated. The structural similarity is used as a quantification value for privacy risk. Select the largest split point that satisfies the privacy requirements, such that the corresponding privacy risk quantification value does not exceed the threshold specified by the user.

[0010] As a further aspect of the present invention: the computing power is a nonlinear negative correlation function between the computing speed of multiple edge servers and the number of access devices; , This represents the minimum computing power that an edge server can provide for a single inference task. This represents the maximum computing power that an edge server can provide for a single inference task. The number of devices accessing the edge server in time slot τ. This is a function representing the computational speed of the edge server. This represents the total computing power of the server.

[0011] As a further aspect of the present invention: based on the periodic Bernoulli process generated by IoT devices, the probability distribution of model types, the probability distribution of privacy requirements, and the random distribution of channel transmission rates, task load prediction data for multiple future time slots is obtained, including: Based on the task generation cycle and generation probability of IoT devices, calculate the probability of generating inference tasks in any time slot; Based on the probability distribution of model type and the probability distribution of privacy requirements, the probability distribution of model segmentation points is obtained; Based on the model segmentation points, the computation time, transmission time, and probability distribution are obtained, and the time slot offset probability of the inference task reaching the management platform is calculated. Based on the time slot offset probability, the expected number of tasks and the expected computing load arriving at the management platform in the target time slot are calculated.

[0012] As a further aspect of the present invention: the method for calculating the progress reward for a single time slot is as follows: R k [τ] , where Δw k [τ] w is the maximum task load that an edge server can process in parallel for each inference task in time slot T. k [τ,iem] It represents the progress of the inference task within the current time slot.

[0013] As a further aspect of the present invention: the length of the prediction window is a length range, which includes an upper limit value and a lower limit value, and the upper limit value is greater than the lower limit value; When the system load is below the preset threshold, the value of the prediction window is the lower limit of the length range; when the system load is above the preset threshold, the value of the prediction window is the upper limit of the length range.

[0014] As a further aspect of the present invention: based on the actual completion time of the inference task and the estimated completion time, the latest task completion rate and the reputation value of the management platform are obtained, including: Construct an indicator function based on the actual completion time and estimated completion time of the inference task: When the indicator function is equal to 1, the inference task is completed and the upper limit of the reputation value is obtained; otherwise, the inference task is not completed within the estimated completion time and the lower limit of the reputation value is obtained. The upper limit of the reputation value is greater than the lower limit of the reputation value.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention obtains task load prediction data for multiple time slots in the future by generating periodic Bernoulli processes based on IoT devices, probability distributions of model types, probability distributions of privacy requirements, and random distributions of channel transmission rates. This enables the scheduling process of inference tasks to better balance the allocation of computing resources, which is beneficial to improving the completion rate of inference tasks. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the method steps of the present invention; Figure 2 This is a graph showing the relationship between the edge server computing speed and the number of access tasks according to the present invention. Figure 3 This diagram illustrates the impact of the number of inference tasks accessed by this invention on task completion time. Figure 3 (a) is the original state diagram of the task completion time. Figure 3 (b) A diagram showing the impact of adding a new task on task completion time. Figure 3 (c) is a diagram showing the impact of adding two new tasks on task completion time. Figure 3 (d) is a diagram showing the impact of the three new tasks on the task completion time; Figure 4 Reconstruct images and similarity comparison charts for different model layers; Figure 5 This is a comparison chart of inference metrics under different privacy split points for different datasets and models in this invention. Figure 5 (a) A graph showing inference-related metrics for the ResNet18-Cifar model at different privacy leakage rates. Figure 5 (b) A graph showing inference-related metrics for the model at different privacy leakage rates in MobileNetV2-MNIST. Figure 5 (c) A graph showing inference-related metrics for the TextRCNN-THUCNews model at different privacy leakage rates. Figure 5 (d) is a graph showing the inference-related metrics for the M5-UrbanSound8K model at different privacy leakage rates; Figure 6 This is a comparison chart of the actual number of tasks generated and the statistical expectation of this invention. Figure 6 (a) is a graph showing the expected number of task arrivals within 400 time slots. Figure 6 (b) is a graph showing the actual number of mission arrivals within 400 time slots; Figure 7 This is a comparison chart of experimental metrics for different algorithms under the set1 case of this invention. Figure 7 (a) is a graph showing the task completion rate of the algorithm under different numbers of devices. Figure 7 (b) is a graph showing the average task reward of the algorithm for different numbers of devices. Figure 7 (c) is the total reward graph of the algorithm under different numbers of devices. Figure 7 (d) shows the optimization target graph of the algorithm under different numbers of devices; Figure 8 This is a comparison chart of experimental metrics for different algorithms under the set2 case of this invention. Figure 8 (a) is a graph showing the task completion rate of the algorithm under different numbers of servers. Figure 8 (b) is a graph showing the average task reward of the algorithm for different numbers of servers. Figure 8 (c) is the total reward graph of the algorithm under different numbers of servers. Figure 8 (d) shows the optimization target graph of the algorithm under different numbers of servers; Figure 9 This is a comparison chart of experimental metrics for different algorithms under the set3 case of this invention. Figure 9 (a) is a graph showing the task completion rate of the algorithm under different task intervals. Figure 9 (b) is a graph showing the average task reward of the algorithm at different task intervals. Figure 9 (c) is the total reward graph of the algorithm under different task intervals. Figure 9 (d) shows the optimization target graph of the algorithm under different task intervals; Figure 10 This is a comparison chart of experimental metrics for different algorithms under the set4 case of this invention. Figure 10 (a) is a graph showing the task completion rate of the algorithm under different batch loads. Figure 10 (b) is a graph showing the average task reward of the algorithm under different batch loads. Figure 10 (c) is a graph showing the total reward of the algorithm under different batch loads. Figure 10 (d) shows the optimization target graph of the algorithm under different batch loads; Figure 11 This is a graph showing the influence of the time window, a key parameter of this invention, on the experiment. Figure 11 (a) is a comparison chart of task completion rates within a time window under different batch loads. Figure 11 (b) is a comparison chart of average task rewards within a time window under different batch loads. Figure 11 (c) is a comparison chart of total rewards for different batch loads within a time window. Figure 11 (d) is a comparison chart of time window optimization targets under different batch loads. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example: Please see Figures 1-11 This embodiment provides a model parallel inference scheduling method applied to a management platform, which communicates with multiple edge servers, and includes the following steps: S1: Obtain deep neural network inference tasks based on IoT devices. The inference task includes model type, data batch, privacy requirements, and maximum tolerable completion time. Determine the model segmentation point of the IoT device based on the privacy requirements. Perform model layer calculations before the segmentation point locally. Send the intermediate data and remaining calculation tasks corresponding to the model layer after the segmentation point to the management platform through the access point.

[0019] S2: Obtain the real-time load status and computing power of each edge server.

[0020] S3: Based on the periodic Bernoulli process generated by IoT devices, the probability distribution of model types, the probability distribution of privacy requirements, and the random distribution of channel transmission rates, task load prediction data for multiple future time slots is obtained.

[0021] S4: Based on real-time load status, computing power, and task load prediction data, a rolling time-domain framework is used to enumerate all edge server allocation data corresponding to the inference task in the current time slot to obtain the target edge server and the estimated completion time. Based on the target edge server within a finite time domain of the prediction window, a greedy algorithm is used to accumulate the single-time slot completion progress reward of all tasks being processed in each time slot based on the task load prediction. The allocation data with the largest accumulated completion progress reward is selected as the target allocation data for the current time slot.

[0022] S5: Distribute the inference task and the corresponding intermediate data to the target edge server, and perform calculations based on the model layer after the segmentation point of the target edge server.

[0023] S6: Obtain the inference results returned by the target edge server, and based on the actual completion time and estimated completion time of the inference task, obtain the latest task completion rate and the reputation value of the management platform.

[0024] In this embodiment, the IoT device is a group of resource-constrained devices, denoted as: These devices periodically generate data for deep neural network (DNN) inference tasks, but due to privacy concerns and resource constraints, some computations of the model need to be performed locally: a set of distributed edge servers (ESs), denoted as server S: These servers provide computing services to complete the remaining computations of the DNN model; the unified management platform for the servers is denoted as c.

[0025] As the central coordinator of the system, rather than a direct computing provider, the management platform is responsible for global resource scheduling and service quality assurance. It maintains reliable wired connections with all edge servers, monitoring their load status and computing power in real time. For ease of analysis, the entire timeframe is divided into several time slots. The duration of each time slot is Every so often, the device will randomly generate DNN inference task data requests. Due to task privacy requirements, the device will complete part of the inference task data locally and will need to request inference assistance from the server managed by the management platform, along with corresponding compensation and a maximum tolerance time for the remaining inference. The management platform will predict the completion time based on the current computing resources of the management server, make a corresponding commitment to the user, and determine the server to complete the task. The device communicates with the edge server assigned by the management platform through an access point (AP) near its location to transmit the remaining inference task data to the server. If the task is not completed within the promised completion time, the inference task data is considered a failure, and the management platform's reputation will be damaged.

[0026] In this embodiment, the real-time load status includes the completion time of all pending tasks on each server in the current and future time slots, and the task load prediction data includes the expected number of tasks arriving and the expected task computation load.

[0027] like Figure 2 As shown, in this embodiment, the basic modeling includes: considering a dynamic and uncertain smart Internet of Things (IoT) scenario, which contains a set of resource-constrained IoT devices (IoTDs), denoted as... These devices periodically generate data for deep neural network (DNN) inference tasks, but due to privacy concerns and resource constraints, some computations of the model need to be performed locally; a set of distributed edge servers (ESs), denoted as servers. These servers provide computing services to complete the remaining computations of the DNN model; the unified management platform for the servers, denoted as c, acts as the central coordinator of the system rather than a direct computing provider. The management platform is responsible for global resource scheduling and service quality assurance. It has a reliable wired connection with all edge servers and monitors the load status and computing power of each server in real time.

[0028] For ease of analysis, the entire time range was divided into several time slots. : The duration of each time slot is Every so often, the device will randomly generate DNN inference task data requests. Due to task privacy requirements, the device will complete part of the inference task data locally and will need to request inference assistance from the server managed by the management platform, along with corresponding compensation and a maximum tolerance time for the remaining inference. The management platform will predict the completion time based on the current computing resources of the management server, make a corresponding commitment to the user, and determine the server to complete the task. The device communicates with the edge server assigned by the management platform through an access point (AP) near its location to transmit the remaining inference task data to the server. If the task is not completed within the promised completion time, the inference task data is considered a failure, and the management platform's reputation will be damaged.

[0029] 2.1 Edge Server and Management Platform Modeling In the modeling process, there are multiple edge servers. : and unified management server management platform Use a triplet Represented as an edge server, where This represents the maximum number of devices that can connect to the server simultaneously. This represents the server's computational speed function (the basic unit is units of...). (This represents the number of floating-point operations per unit time), and its value changes with the number of devices connected to the server. Express the server's computational speed as a function, where the computational speed is related to the number of devices connected to the server, and the function satisfies the following constraints: ; and These represent the minimum and maximum computing power that the server can provide for a single task, respectively. Indicates in time slot The number of devices connected to the server, and the function changes non-linearly.

[0030] Indicated in the current time slot The changes in the completion time of tasks currently being handled by the server after a new task is added. Specifically, this is reflected in... After a new task (connecting one or more devices) is entered, several time slot servers calculate the time changes of each task on the future time slot servers based on the current device occupancy time.

[0031] The management platform's reputation score within the current time slot is recorded as follows: It can indirectly reflect credit rating and evaluation. The likelihood of fulfilling tasks issued by devices is considered. Specifically, the management platform acts as a scheduling center, receiving data requests for assisted inference tasks from devices and dispatching servers to assist in inference. During the execution of distributed inference task data, if a task cannot be completed on time according to the set completion time, the management platform's accumulated reputation value will decrease. For ease of analysis, the management platform uniformly distributes tasks at integer time points. If the time point at which the device transmits data to the management platform is a decimal, it is uniformly rounded up to the integer time point for the next allocation.

[0032] 2.2 IoT Device Modeling (a) DNN inference requirements for IoT devices. To capture the dynamic and uncertain service demands of the real world, it is assumed that at intervals of time (an integer number of...) (The number of cycles is related to the device itself). An IoT device generates inference task data with a certain probability.

[0033] Therefore, use To represent time slots At the beginning Information (e.g., current location, whether there is inference task data, etc.), and includes: (2.2) in, This represents the device's local computation rate per unit time, which is constant throughout the entire time domain. It is also assumed that every certain period of time... The device may generate inference task data, denoted as an indicator. ,in, Data representing reasoning tasks It has already been generated, and This indicates that the device has no inference request. To describe the dynamic time-varying nature of inference and the randomness of demand, different devices can generate inference task data at different intervals, and satisfy the following: The data follows a Bernoulli distribution, and the individuality and diversity of the devices impart different generation probabilities. Furthermore, the data models generated by DNN inference tasks from devices in different time slots may differ, as represented by... Different model categories (e.g., differences in the number of layers or the type of data processed). When hour, Defined as a tuple: ; in, Randomness is one type of model classification. Indicates the latest tolerable completion time of the reasoning. This indicates the data batch corresponding to the task. This represents the privacy requirements for generating inference task data in this time slot. Due to the limitations of local inference resources on IoT devices, many studies consider splitting the DNN model at model split points, offloading part of the model to a resource-rich server to assist in inference, thereby alleviating the constraint of insufficient local inference resources. Therefore, we consider splitting the DNN model and using indicators. Represents time slot τ inference task data Partition points of model l, partition points Previous layers in the device Executed locally, but from Then, it is unloaded to the corresponding server for execution at the last level. Specifically, if... This indicates that all tasks are processed on the server side, and This indicates that all tasks are processed locally, and the location of the model partition point depends on the device's inference privacy requirements. .

[0034] (b) Privacy problem of model partitioning: When the model is partitioned and the intermediate data is transmitted to the edge server, privacy problems will arise in the process. That is, potential attackers can reconstruct the original input data based on the content of the intermediate data, which will endanger the data. Therefore, users tend to process the first few layers, including the input, locally and then unload the remaining parts to the server for processing, so as to ensure that the intermediate data is not highly similar to the original data after reconstruction. In previous work [2], some image inference models were partitioned and tested. The privacy leakage was evaluated by reconstructing the image from the intermediate data and using structural similarity (SSIM) to calculate the similarity between the reconstructed image and the original image. Furthermore, it can be seen from [2] that in the process of partitioning the model layers for distributed inference, executing more layers locally can effectively avoid the occurrence of privacy problems, because the more local layers there are, the lower the similarity between the intermediate input data of the edge layer and the original data. SSIM quantifies the similarity of images by calculating the brightness, contrast and structural information between two images.

[0035] The formula for calculating SSIM is expressed as follows: (2.5) in , It is the mean of images i and j. and It is their variance. It is covariance. , It is a small constant for stable division. The theoretical range of SSIM is [-1, 1], where 1 represents perfect similarity, and the lower the value, the lower the similarity. This paper describes the image similarity method in detail. For audio and text data, cosine similarity is used to calculate the similarity. The main focus of this paper is to reconstruct the input data from the intermediate data and judge the privacy risks by comparing the similarity between the reconstructed data and the original data. Therefore, it is not necessary to consider some more complex privacy-related factors, such as word error rate (WER).

[0036] These similarity values ​​are normalized using... This indicates the privacy risks caused by changes in model split points. The similarity between the reconstructed image and the original data is calculated by obtaining intermediate features generated at the split points. The higher the calculated similarity, the higher the possibility of data privacy loss.

[0037] (c) Model Partitioning: Therefore, we consider partitioning the DNN model. Each ES pre-deploys the model required for the task, which can handle the partitioned model partitions. Previous modeling emphasized that each inference task data generated in a time slot has its own privacy tolerance. The model partitioning must meet the device's inference privacy requirements. Therefore, based on privacy risks, the model partitioning points can be obtained as follows: ,satisfy .and This represents the batch of task data to be processed in the entire inference task. As the number of batches increases, the overall computational workload of the task is linearly related to the number of batches. Finally, the server that assists in processing the task is given a certain reward based on the computational workload.

[0038] Then, for the partitioned model, define , This refers to the unit batch computation amount corresponding to the local and unloaded portions before and after the partition point. The local computation amount should numerically equal to the sum of the computation amounts of the layers from the first layer to the partition point. Similarly, the computation amount of the unloaded portion is the sum of the computation amounts from the partition point to the last layer. The magnitude of both depends on the model type and the corresponding partition point z. This is relevant. Also, `im[τ]` represents the intermediate data corresponding to the unit batch at the partition point location. The size of this intermediate data is also related to the model type and the location of the model partition points. Therefore, for tasks with different batch sizes, the corresponding computational load should be the product of the batch size and the computational load per unit batch.

[0039] (d) Modeling of time-varying channel characteristics of mobile IoT nodes. Each IoT device has its own trajectory throughout the entire time period. The device can communicate with access points and edge servers near its location, and the transmission rate is expressed as... , which represents the time required to transmit a unit of data. Since IoT devices are mobile and their distance from the access point is dynamically time-varying, the transmission rate is referred to in paper [7]. The model is constructed using random numbers, uniformly distributed within a certain range. This model effectively captures the channel randomness caused by device mobility, while preserving a reasonable range of variation in physical channel gain through boundary parameters, providing a probabilistic performance guarantee for subsequent resource scheduling optimization.

[0040] 2.3 Uncertainties of Interest 2.4 Introduction to the data scheduling process for the inference task of interest: Taking IoTD at the beginning of time slot τ as an example. Take the generated inference task data as an example.

[0041] Task generation: every certain time slot The beginning, assuming this is time slot τ, IoTD There may be inference task data. (if This task requires edge computing services. The first part of the model is executed locally, while the second part (including intermediate data used for inference) can be uploaded to an available server in the corresponding time slot after the first part is completed. To facilitate this allocation, an indicator is used: Representing the time slot τ inference task The partitioning points of model l vary depending on the privacy requirements of different devices in different time slots. Note that the tasks generated in time slot τ and the corresponding calculated values ​​are distinguished from other time slots using the symbol τ. However, in practice, due to the dynamic nature of resources, there is also a practical timeline that corresponds to time slot τ (natural clock). These two subsequent key time nodes are defined as follows: , respectively, represent the time (time point) when the local reasoning is completed. The time (point in time) when intermediate data arrives at the server. .

[0042] Task arrival and distribution: After completing the first part of the calculation locally, the device distributes the task in the time slot. The system requests inference assistance for the remaining tasks from nearby access points to the management platform, and provides corresponding rewards based on the computational load of the tasks. At a specific time point... The management platform receives the corresponding task and allocates it to the rounded-up time slot at that time. The system makes unified scheduling decisions for tasks arriving in the previous time slot, which means selecting the corresponding server from the servers it manages to assist in completing the inference task.

[0043] Use 0-1 decision variables This indicates the server that begins assisting in the inference process during the rounding-up time slot. , The value needs to be calculated through model partitioning decisions and depends on the execution time of the previous partition. When the partitioning decision is fixed, The value is also related to the transmission rate.

[0044] Task completion time assessment: Since multiple devices randomly generate model inference task data at different time slot intervals, due to privacy requirements, the data for the first half of the inference task is completed locally and transmitted to the management platform through the access point. The management platform makes server-assisted decisions in the rounded-up integer time slots when receiving the task. When allocating time slots, the management platform will adjust the estimated completion time (time period) of the corresponding inference part based on the changes in the current server's computing resources. Conduct an evaluation (represented as equipment) The inference task data generated in time slot τ is on the server. The estimated completion time should be less than the equipment's maximum tolerance time (time period). (The difference between local inference and transmission time). Furthermore, the further the estimated time is from the maximum tolerable time for edge-part inference, the greater the reward.

[0045] Server inference completion: Due to the reliable wired connection between the management platform and the server, the latency between them is negligible. Therefore, the server receives the allocation decision from the management platform and begins inference for the corresponding task within the same integer time slot as the management platform. Furthermore, in addition to completing its original task, the server also needs to accept various types of randomly arriving tasks. Moreover, the server's computing resources will vary depending on the connected devices, so the server's computing speed will change throughout the entire process. It should fluctuate up and down. When the server completes the task... Completion time (time period) Estimated completion time Regarding the completion of the corresponding task, the current server is capable of completing the task before the estimated completion time, supporting the increase in the overall task completion rate.

[0046] Given that edge nodes in real-world scenarios are typically powered by stable sources such as the power grid, their energy supply can be considered continuous and sufficient. Therefore, the focus of system optimization is on task completion time, i.e., task completion rate. From the perspective of the management platform and servers, a shorter estimated task completion time means higher rewards. However, as more tasks are taken on in the future, the server's computational efficiency will decrease, leading to task failures. These failures will also reduce the overall task completion rate (also known as the management platform's credibility) across all servers managed by the platform. In summary, the optimization objectives of this paper are: to maximize the successful completion rate of multi-inference task data and to improve the overall rewards for the management platform and servers.

[0047] 3.1 Inference Time Modeling Taking the example in 2.4 as an example, at the beginning of time slot τ, IoTD The generated inference task data, due to device privacy requirements The model can be obtained The system is divided into local and peripheral reasoning parts, with the dividing point as the reference.

[0048] The local inference time (time period) is obtained by the ratio of the computational load of local inference to the computational speed of the device: , So in Time slot (time point) equipment Complete the local inference task data, and transfer the intermediate data. The data is transmitted to management platform c at a rate of [value missing]. [ [ / bit], which varies randomly within a certain range. The data transmission time (time period) sent by the device is: So, the equipment Data generated in the assisted inference task during time slot τ is first uploaded through its connected access point (AP). Upon receiving the task, the AP immediately forwards its basic information (including task type, model partition point, data size, etc.) to the management platform to trigger scheduling decisions, while the complete intermediate feature data remains temporarily stored on the AP side. During the time slot (time point): The AP has completed receiving the data uploaded by the device and is ready to forward the task to the server for execution. This article ignores the data and control signaling transmission latency between the AP, management platform, and server; therefore, the rounding-up time point of the management platform in this time slot is used. Tasks arriving in the previous time slot are uniformly allocated, and selection instructions are sent to the AP and the corresponding server. The AP then forwards the intermediate feature data of the corresponding task to the selected edge server according to the instructions from the management platform. Since the management platform and each server are interconnected via a reliable wired link, the real-time resource status of the servers can be obtained promptly. Therefore, the management platform can predict the task completion time based on the server's current computing speed; the computing time is equal to the ratio of the computing load of the task on the server to the available computing speed. However, in the previous modeling, it was clarified that the computing speed exhibited by the server will vary due to the different number of connected devices. Therefore, the actual task completion time is determined at the exact hour of the time slot (time point). It can be represented as: (3.3) The specific value is equal to the actual computing speed of the server in each future time slot t after accepting the task from time slot τ. Multiply by the time slot length ∆τ, when the cumulative sum first exceeds the computational cost of the edge portion. The time slot (denoted as ≥1) is the rounded-up value of the time slot containing the completion time. Of course, this completion time can be accurate to a decimal time slot, which means first calculating the distance between the computational amount in the last time slot and the previous time slot:

[0049] Therefore, the actual task completion time (time period) should be the distance plus the sum of the preceding time slots, multiplied by the duration of a single time slot: ; The aforementioned task completion time was calculated based on specific numerical values ​​after the computational rate changes of each time slot server were clarified in the future. However, during the time slot τ generation and allocation decision-making, the management platform... This is used to determine the impact of adding new tasks on the current server tasks, thereby enabling server-assisted reasoning decisions. However, in reality, the number of devices connecting to the server in the future cannot be fully predicted when assigning tasks. Due to the random generation of tasks, the number of devices connecting to each server in future time slots may change. For example, devices leaving the server after completing existing tasks or new tasks joining will cause the server's computing resources to fluctuate with the number of tasks joining. When making actual server-assisted reasoning decisions, in order to protect the interests of the server while meeting the reasoning needs of the devices, it is necessary not only to determine whether the tasks in the first few time slots can be completed on time, but also to evaluate the tasks that may join the server in the next few time slots. Therefore, a prediction time (time period) is introduced by a management platform. (Similar to giving a user a promised completion time), the predicted time value is not entirely consistent with... The displayed task times are consistent, and the estimated completion time is always less than the maximum tolerance time (time period) for the edge portion. The maximum tolerance time for the edge portion is equal to the difference between the maximum tolerance time of the overall task and the local computation time and transmission time, i.e.: .

[0050] 3.2 Edge reasoning reward modeling; To encourage the management platform and servers to complete and accept tasks within their capabilities, the device rewards the management platform and provides additional rewards based on the difference between the management platform's estimated completion time (time period) and the device's maximum tolerable completion time (time period). The size of the base reward is related to the amount of inference task data generated by the device, the model type, and the split point. The size of the additional reward is related to the difference between the tolerable time and the estimated completion time; the smaller the estimated completion time, the further away from the tolerable time, and the greater the reward. Considering that the split point and computational load are known when the device generates inference task data due to privacy requirements, the difference is reflected in the additional reward. Therefore, the additional reward is modeled as follows:

[0051] because This is an estimated time promised by the management platform to the device, and it is not based on actual calculations. It is a variable that needs optimization.

[0052] 3.3 Credibility / Task Completion Rate; Regarding the rewards, the lower the estimated time reported by the management platform, the greater the reward. However, the corresponding server must also complete the task within the promised time; otherwise, it will face a decrease in credibility. This, in turn, limits the management platform's definition of the estimated time. (The last sentence appears to be incomplete and possibly refers to a separate issue: "Server...") Actual completed equipment The time definition of inference task data is: Using indicator functions Data indicating whether the device's inference task has been completed. A positive result indicates the task is complete; otherwise, it indicates the task was not completed before its estimated deadline. In each time slot, after the generation and allocation of inference task data, a status check is performed to determine whether the inference task data is complete. The completion rate of all inference task data generated in time slot T can be expressed as:

[0053] 3.4 Problem Definition; Therefore, the optimization goal is to maximize system rewards and task completion rate over long time slots:

[0054]

[0055] Constraints satisfied:

[0056]

[0057]

[0058] The optimization objective (3.8a) is to maximize the weighted sum of task rewards and completion rates; constraint (3.8b) states that the existence of a task is always accompanied by its completion time and estimated time, and the existence of completion time and estimated time indicates that the corresponding task has been generated; constraint (3.8c) ensures that each task is completed with the assistance of one and only one server; constraint (3.8d) restricts the allocation strategy to variables that can only be 0-1 variables; constraint (3.8e) ensures that the estimated time is within the range of the maximum tolerable completion time. Based on the previous formula derivation, the optimization objective can be further expressed as:

[0059] The actual completion time is as follows:

[0060] ; In this embodiment, the task scheduling algorithm for edge networks includes the following: In edge inference scheduling scenarios, the execution process of a task typically spans multiple time slots, and its completion time is closely related to the available computing resources of the server in future time slots. Therefore, the scheduling decision for the current time slot not only affects the current resource allocation result, but also has a continuous impact on the task completion feasibility of future time slots by changing the server's queue state.

[0061] Specifically, time coupling is mainly reflected in the following two aspects: On the one hand, the inference task data currently allocated to the server will continue to occupy computing resources for several time slots in the future, thereby reducing the effective computing power available for newly arriving tasks in subsequent time slots; on the other hand, the new tasks arriving in future time slots are random, and their number and computing load are uncertain, making it difficult to accurately determine whether the server has sufficient computing resources within the task deadline range in the current time slot.

[0062] The aforementioned dual uncertainty leads to a high degree of coupling in the scheduling decision-making process over time: whether to accept and allocate the current task depends on the outcome of future workloads; and the feasibility of future tasks, in turn, is affected by the current decision. Therefore, without introducing forward information, the scheduling problem cannot be decomposed into an independent single-slot decision problem.

[0063] To address the spatiotemporal dynamics and uncertainties of inference task data in edge networks, a two-stage scheduling framework combining prediction and optimization is proposed. This framework first decouples the problem by predicting future resource demands through spatiotemporal feature perception, and then makes task allocation decisions oriented towards quality of service assurance based on the prediction results.

[0064] By decoupling the original problem, the problem is broken down into two parts: one is predicting the future T through probabilistic analysis and other methods. [wnd] One part is the task expectation and load arriving at the management platform in one time slot, and the other part is the recursive optimization based on the predicted expectation to determine the task allocation decision for the current time slot arriving at the management platform.

[0065] 4.1 Spatiotemporal awareness-based multi-timeslot resource demand forecasting; In edge inference networks, task arrival and resource requirements exhibit significant temporal correlation and spatial heterogeneity. On the one hand, the generation of device-side inference task data is influenced by application cycles, user behavior, and device mobility, exhibiting obvious temporal randomness. On the other hand, differences in model type, privacy requirements, batch size, and transmission conditions among different tasks lead to significant variations in the computational resources and duration of time slots they occupy at the edge. These characteristics collectively result in high uncertainty in the load of edge servers over multiple future time slots, posing challenges to the stability of scheduling decisions and service quality. To address this, this section establishes a spatiotemporally aware multi-time slot resource demand prediction model from a device perspective. By jointly modeling task generation behavior, task attributes, and the computation-transmission process, it predicts the number of tasks observed by the management platform and their corresponding expected computational load over multiple future time slots, providing deterministic input parameters for subsequent scheduling optimization.

[0066] 4.1.1 Optimization of the prediction time-domain window n; In the edge inference scheduling problem, the task allocation decision in the current time slot not only affects the completion time of the task itself, but also has a continuous impact on the task's feasibility in multiple future time slots by changing the server's queue state and available computing resources. Therefore, scheduling decisions have an explicit forward coupling characteristic in the time dimension.

[0067] In a multi-task concurrent execution scenario, the available computing power of a server evolves over time, and its queue state is affected by two main factors: 1. Tasks allocated in the current time slot continue to occupy computing resources in future time slots; 2. Additional computing load introduced by newly arriving tasks in future time slots may slow down tasks in scenario 1; 3. Resources occupied by completed tasks are released, which will accelerate task progress again. To ensure that current scheduling decisions do not lead to future server overload, thereby causing existing or future tasks to violate deadline constraints, the prediction window needs to cover the time range in which the server queue state changes significantly.

[0068] Meanwhile, the selection of the prediction window length *n* directly determines the effectiveness of the prediction results for scheduling decisions and the computational complexity. Therefore, if the prediction window is set too short, it will fail to cover the entire lifecycle of task execution, leading to a systematic underestimation of the impact of scheduling decisions on future load; conversely, an excessively long window will significantly increase the computational overhead of prediction and optimization, and introduce unnecessary prediction errors. Therefore, the prediction window *T*... [wnd] The selection of the prediction window should follow the principle of covering the critical time coupling range, that is, the prediction time domain should be able to fully reflect the main impact of the current decision on the future state of the system. In other words, the prediction window should cover the number of continuous time slots corresponding to the maximum tolerable completion time of the task in the current time slot.

[0069] 4.1.2 Future Multidimensional Task Demand Prediction Based on Stochastic Processes; Taking time slot τ as an example, the management platform needs to predict the time from τ to... The potential task demands within a time slot, including the number of tasks, computational load, and batch size, allow for prediction of the server's occupied time slot length distribution. By deriving the specific probability distribution, the expected number of tasks and computational load for future time slots can be obtained.

[0070] 4.2 Time-decoupling optimization based on predicted workload; In the preceding sections, we derived the expected number of inference task data arriving at the management platform across multiple time slots, as well as the corresponding expected total computational load, starting from the device side. This yielded a predicted workload sequence for multiple future time slots. Building on this foundation, this section further investigates how to utilize this predictive information to reconstruct and solve the task scheduling problem, which originally exhibited explicit temporal coupling characteristics.

[0071] It is important to emphasize that so-called time decoupling does not mean ignoring the dependencies of tasks in the time dimension, but rather transforming the originally implicit and uncertain cross-time slot coupling into explicit and observable load parameters through predictive means, thereby enabling scheduling decisions to be deterministically optimized within a finite time domain.

[0072] 4.2.1 Modeling the time decoupling optimization problem based on predicted workload; In the previous section, by introducing multi-slot predictive workload, the temporal coupling originally caused by the arrival of future random tasks was explicitly transformed into observable load parameters. Building on this, this section formally models the scheduling decision for the current time slot, constructing a time decoupling optimization problem based on predicted workload.

[0073] In the current time slot τ, the management platform is aware of the following information: Current system status, including the computing load already carried by each edge server.

[0074] Prediction Time Domain Window Expected number of tasks to arrive: , Prediction Time Domain Window Expected average computational load of a single task within ; The above predictions are all given by the spatiotemporal awareness prediction model and are used as known parameters in the current optimization problem.

[0075] In the previous derivation, the optimization objective was to optimize the system reward and task completion rate over long time slots (within the server's promised completion time). Whether the task can be completed (the system reward is related to the server's promised completion time, while the task completion rate is a decision variable). Can the server meet the promised completion time under the specified restrictions? Complete the task on time.

[0076] In a decoupled problem, the optimization objective becomes maximizing the reward in the current time slot and the completion rate of system tasks up to date.

[0077] , ; The above predictions are all given by the previously derived spatiotemporal perception prediction model and are used as known parameters in the current optimization problem.

[0078]

[0079]

[0080] Among the first optimization objective, It is a known value. The first variable is the decision variable, which assumes that the reward from the device is only obtained upon completion of the task. The second variable, completion rate, is explicitly defined as the expected completion rate of the tasks carried by the server itself and the tasks that need to be allocated in the current time slot under the current decision. This part can be determined through the decision variable. and the computing load already carried by each edge server The task expectation for the arrival of future time slots is judged.

[0081] The ultimate optimization goal of this paper is to maximize the on-time completion rate of newly arriving tasks and their combined rewards across multiple time slots. However, since on-time completion of tasks can only be observed in future time slots, it is impractical to directly use the completion rate as immediate feedback for online decision-making.

[0082] To address this, a recursive decision-making framework with a rolling time domain was constructed.

[0083] In each time slot, the system starts from the current state, performs forward extrapolation on candidate initial decisions within a limited prediction window based on the expected arrival of future tasks, and uses the instantaneous reward of a single time slot to characterize the merits of the decisions made during the extrapolation process.

[0084] During the simulation, the future arrival of tasks is characterized by their statistical expectations. Considering the high computational complexity of jointly optimizing the entire decision sequence within the prediction window, this paper adopts a greedy strategy based on maximizing immediate rewards in each prediction time slot to select the locally optimal decision to update the simulated system state, thereby approximating the overall utility of different initial decisions within the prediction window.

[0085] In the actual execution phase, the system only executes the first step of the decision sequence with the highest predictive utility, and then repeats the state observation, prediction and decision-making in the next time slot, repeating the above process in a rolling time domain manner, thereby achieving recursive optimization in a dynamic environment.

[0086] For time domain window The choice of size has been defined in Section 4.1.1. In the section specifically assessing task completion rate, the server's load status is first formally described. In this paper, This is used to characterize the overall completion time structure of tasks carried by the server after a new task is accepted in the current time slot τ, i.e., the impact of changes in task load on the completion sequence. This change process has been visually illustrated with example diagrams in the preceding text; mathematically, this load state can be represented as a set of tasks with attributes such as remaining computational load, arrival time, and completion time constraints. First, using... Indicates server S nThe set of task indices to be processed at the beginning of time slot τ. Because the system operates in a dynamic environment, within each time slot, completed tasks may leave the server, and new tasks may be scheduled and arrive at the server. Therefore, the server's task set... Evolves over time. (And) Correspondingly, using and Let S represent the new tasks arriving at the server at the start of time slot τ that need to be processed, and the tasks that are still incomplete and occupying server resources. Then, server S... n The set of task indices that need to be processed at the beginning of time slot τ can be represented as: (4.1) At the same time, define Let be the set of tasks that complete and leave within time slot τ, then the following relationship can be obtained: (4.2) In other words, the total number of tasks the server needed to process in the previous time slot, minus the tasks completed within the previous time slot, becomes the number of tasks that are still incomplete and consuming server resources at the start of this time slot. From this, we can derive the complete update formula: (4.3) This update relationship indicates that the set of tasks that the server needs to process at the beginning of time slot τ+1 consists of three parts: one is the tasks starting from the previous time slot τ, and the other is the tasks that the server S needs to process. n Set of task indexes to be processed Secondly, tasks that were completed and completed in the previous time slot. The first two were removed because they constituted tasks that were still incomplete and consuming server resources at the start of time slot τ. Third, new tasks are scheduled by the equipment side in time slot τ and actually arrive in time slot τ+1. .therefore, Always indicates server S n The set of task indexes that need to be processed at the beginning of time slot τ, and Equation (4.3) strictly describes the rolling evolution of this set between adjacent time slots.

[0087] The updated formula above clarifies that The definition and related updates allow us to determine the number of tasks hosted on the server corresponding to time slot τ. To obtain.

[0088] Therefore, for edge server S n In time slot τ, the actual task load state it carries can be formally defined as A.

[0089] Because the management platform performs unified scheduling only in integer time slots, all tasks scheduled on the device side, after undergoing transmission and time slot alignment, only enter the server execution queue at the beginning of the next time slot. This indicates that a new arrival enters server S at the beginning of time slot τ. n A set of tasks.

[0090] A: Please note that in the previous mathematical representation, the relevant attributes were identified by time slots from the perspective of the device. However, the following part will be deduced from the perspective of the server and management platform, and the time slot identification will be different from before.

[0091] (4.4) in, Each element in the database corresponds to server S. n This refers to a task state being processed in time slot τ. For any given task state element, it contains the following properties: This indicates the total remaining computing load corresponding to the task. This indicates the time slot (point in time) at which the task arrives at the server. Due to the reliable wired connection between the management platform and the server, this arrival time slot should be an integer. This indicates the estimated completion time (time period) for the current task. This is the maximum tolerable completion time (time period) for this part on the server side.

[0092] To eliminate the timescale difference between the task generation time slot on the device side and the execution time slot on the server side, the following set of arriving tasks is introduced from the server's perspective. Definition To be the actual arrival and being received by server S at the start of server time slot τ. n The set of tasks accepted, representing for: (4.5) From the equipment's perspective This represents the specific task allocation decision variable. The time slot identifier in the upper right corner is used to distinguish tasks generated in other time slots. However, the same symbol cannot be used to represent decision variables in the server section, so it will be used subsequently. This represents the allocation decision for all tasks in time slot τ from the server's perspective. Furthermore, to achieve the transformation from the decision variables to the server-side task set k, a mapping function is used to implement time slot alignment mapping with the server. (4.6) This indicates that the device task is on server S. nUpon arrival at time slot τ, the corresponding internal server task instance k is generated. Therefore, the set of new task indices that arrive at time slot τ and enter the server execution queue, uniformly allocated by the management platform, can be represented as: .

[0093] The above definition decouples the scheduling decision timeline on the device side from the execution timeline on the server side. During scheduling and queue evolution on the server side, there is no longer an explicit dependence on device-side variables. Instead, it is only by reaching the set To characterize the new tasks that need to be processed in time slot τ. Therefore, This forms the interface layer between device scheduling and server execution.

[0094] Furthermore, since the duration of each time slot is considered sufficiently small, the server's computation rate remains constant within a single time slot. The relevant definitions of computation rate have been described previously for edge server S. n In time slot τ, before a newly arrived task has been added to the server, the computation rate per unit time slot is related to the number of tasks currently being handled by the server, expressed as:

[0095] Here, we clarify the form of the formula for the computation rate of the current time slot server in processing a single task after a new task is added:

[0096] And it satisfies the following constraints:

[0097]

[0098] and These represent the minimum and maximum computing power that the server can provide for a single task, respectively, where β represents the attenuation of the computing power allocated to the server as more tasks are connected. This indicates the number of tasks accessing the server in time slot T, and ensures that the number of tasks accessing the server does not exceed the server's maximum access limit. .

[0099] So, what is the maximum task load that the server can process in parallel for each task in time slot T?

[0100] The remaining load after the end of this time slot is then updated as follows:

[0101] Therefore, when This indicates that the current task has been completed within this time slot and will no longer occupy server resources in the next time slot. The task will then be used as... Update and delete from the server's task queue.

[0102] Define the following 0-1 indicator variable for each concurrent task k on the server side: The indicator function is set to 1, which means that task k has been completed within its estimated tolerable completion time; otherwise, it is set to 0.

[0103] It is important to note that within time slot τ, the system's completion rate is only used to characterize whether new tasks generated and accepted in that time slot can be completed within their maximum tolerable completion time. For historical tasks accepted in previous time slots, their feasibility of completion has already been assessed through scheduling and latency prediction when they access the server, and therefore, the completion rate reward is not repeatedly included in subsequent time slots. Therefore, the ultimate goal is to make allocation decisions for newly arriving tasks in each time slot, maximizing the weighted sum of the completion rate and reward for newly arriving tasks. This is also the physical meaning of the overall optimization objective. In actual task execution, whether a task can be completed within its maximum tolerable completion time can only be accurately determined several time slots later; therefore, the completion rate feedback has a significant time delay. In this case, it is not possible to directly use the completion rate as an immediate feedback signal at the start of task allocation. Therefore, when transforming the original multi-time slot optimization problem into a single-time slot recursive decision problem, it is necessary to construct a surrogate reward function that can be calculated instantly in the current time slot and reflects the potential feasibility of timely task completion. The single-slot reward does not directly reward whether the task is completed. Instead, it describes the feasibility margin of newly arrived tasks being completed on time under the current allocation decision, thereby gradually guiding the system to evolve to a strategy with a higher completion rate through recursive optimization.

[0104] Defined in the server's task set The single-timeslot feasibility reward for each task in the future time slot is: , in, It was obtained from the previous calculation of the maximum task load formula, the denominator The remaining workload of the current task in this time slot corresponds to the previous workload. The definition of τ is used to characterize the progress of a task within the current time slot. A higher task progress indicates a greater likelihood of completion under the current allocation decision. Under this reward, rolling time slot optimization aims to maximize the completion rate of existing and future tasks on the server during continuous time slot rolling. However, at the edge of the time slot window, the final decision is still based on the completion rate of the task arriving at time slot τ.

[0105] Therefore, the optimization objective under a single time slot should be:

[0106] in, It is the set of feasible server allocation decisions for all newly arrived tasks in the current time slot τ. It is based on the server allocation decision-making based on the expected arrival of future tasks, and the optimization part ensures that the task progress index of all tasks in the current time slot is maximized.

[0107] It is important to emphasize that throughout the process, the task completion rate is more important than the reward obtained by the system. The single-slot reward mentioned above is not directly equivalent to the task completion rate, but rather serves as a proxy optimization metric for the completion rate. By recursively maximizing this feasibility margin reward in each slot, the system state will gradually evolve to a load structure that is more friendly to newly arriving tasks. Finally, after the task execution window ends, the actual completion rate is used as the final metric to uniformly evaluate the strategy performance, and the completion time is evaluated according to the high completion rate, that is, the completion time at this point is used as a reference value for the estimated completion time. The overall algorithm flow is shown in Table 1 below: Table 1 Algorithm Flow .

[0108] In this embodiment, experiments are conducted to evaluate the performance of the proposed algorithm.

[0109] 5.1 Experiments related to real datasets In the system modeling section of Chapter 3, this paper points out that in multimodal distributed inference scenarios, the model partitioning position not only determines the distribution of computational load but also directly affects the degree of privacy exposure and inference performance. Specifically, the earlier the partitioning point, the fewer features are retained locally, resulting in higher privacy exposure; while the later the partitioning point, the more local computation is performed, resulting in stronger privacy protection.

[0110] To verify the impact of multimodal model segmentation on inference performance under different privacy requirements, this paper constructs a unified segmentation point evaluation framework based on three typical tasks: vision, text, and audio. It systematically analyzes the differences between distributed inference and full model inference in terms of accuracy, privacy leakage, and performance metrics.

[0111] 5.1.1 Dataset and Model Selection In the preceding modeling section, we discussed the differences in model splitting points caused by multimodal and privacy considerations. To illustrate the differences in inference-related metrics between distributed inference and the full model across different models and datasets, we evaluated models based on multiple dataset types: I) ResNet18 based on CIFAR-10, II) MobileNetV2 based on MNIST, III) TextCNN based on UrbanSound8K, and IV) M5 based on THUCNews. All models were trained and converged on a single machine, with the accuracy of the full model serving as the baseline for subsequent comparisons.

[0112] Table 2. Datasets and models used in the experiment .

[0113] To simulate different privacy levels, this paper selects multiple candidate split points according to network depth: 1) ResNet18: 8 split points; 2) MobileNetV2: 8 split points; 3) TextCNN: 5 split points; 4) M5: 5 split points. To comprehensively evaluate the impact of splitting, the following metrics are selected: 1) Top-1 Accuracy; 2) Top-5 Accuracy; 3) F1-Score; 4) Privacy leakage rate. The privacy-performance relationship is analyzed through joint analysis of multiple metrics.

[0114] like Figure 11 As shown, the experimental results indicate that the Top-1 Accuracy of segmented inference is completely consistent with that of the complete model; there is no significant difference between the F1-Score and the Top-5 Accuracy; and the single-sample prediction results are consistent at the numerical level (within the floating-point error range).

[0115] The above experiments demonstrate that model segmentation does not alter the model's functional expressiveness; piecewise forward is mathematically equivalent to complete forward; and inference errors do not originate from the model structure splitting itself. Therefore, in subsequent distributed inference systems, if accuracy fluctuations occur, their sources are more likely to include: network transmission delays or packet loss, intermediate feature compression or quantization, approximate calculations due to edge device computing power limitations, and differences in floating-point precision across heterogeneous hardware.

[0116] This section's experiments verify the equivalence of model splitting only in a single-machine environment. In subsequent hardware experiments, this paper will further deploy a distributed inference system on real-world edge devices, comprehensively considering communication overhead, actual computing power constraints, and multi-node collaboration mechanisms, thereby systematically analyzing the performance changes of parallel inference in a real-world environment. The experiments in this section provide basic comparative results for subsequent hardware experiments.

[0117] To verify the effectiveness of the proposed algorithm framework in multi-time-slot coupled resource scheduling scenarios, this section constructs a discrete-time-slot simulation environment and generates task arrival sequences based on statistical modeling methods. Multiple sets of comparative algorithms are used to evaluate the performance of different decision-making mechanisms in terms of task completion rate, system reward, and optimization target value. The system adopts a discrete-time model, the task generation process of the devices follows a random distribution, and server resources are constrained by computing power and resource contention effects.

[0118] The relevant parameters for the task arrival are as follows: Table 3 Simulation Parameter Settings .

[0119] The above parameter settings are designed to simulate an edge computing environment with dynamic reachability and resource contention. To comprehensively evaluate the performance of the proposed method, the following comparative algorithm is designed in this section: 1. Random algorithm: Randomly selects a resource allocation scheme from the feasible solution space; 2. Greedy Algorithm: Maximizes immediate reward in the current time slot; 3. Expected-based Allocation: Allocates resources based on the statistically expected number of tasks; 4. One-step prediction algorithm: Makes decisions based on the results of a single prediction. 5. Alternating Optimization: This algorithm iteratively solves the problem by alternating between resource allocation and task selection. 6. The algorithm proposed in this paper: Based on a finite prediction window, multi-slot rolling optimization is performed, and only the first step decision is executed in each time slot, and the system state is updated in a rolling manner.

[0120] 5.2.2 Comparison between actual task generation quantity and statistical expectation To verify the effectiveness of the statistical modeling method, this section compares the difference between the number of tasks generated by the actual simulation and the theoretical expected value.

[0121] like Figure 6 As shown, the comparison between the actual number of tasks arriving and the statistical expectation over 400 time slots is presented. It can be observed that: the actual arrival curve fluctuates around the statistical expectation curve; the fluctuation amplitude is related to the task generation probability and batch size distribution; and the variance of the actual arrival reflects the uncertainty brought about by the random generation mechanism.

[0122] Further calculate the variance of the number of tasks arrived. 2The results show that the deviation between the actual value and the theoretical expectation is within an acceptable range. This result demonstrates that statistical modeling methods can effectively characterize the system's task arrival characteristics, providing a theoretical basis for subsequent optimization based on expected values.

[0123] 5.2.3 Evaluation of Task Completion Rate, System Rewards, and Optimization Goals To evaluate the performance of different algorithms under high load conditions, this section designs an experimental scenario with progressively increasing task density: • Set1: Keep all other parameters unchanged, only change the number of devices; • Set2: Keep all other parameters the same, only change the number of servers; • Set3: Keep all other parameters unchanged, only change the task generation interval; • Set4: Keep all other parameters unchanged, only change the task batch.

[0124] Table 4. Parameters for different experimental settings .

[0125] In each scenario, the following metrics were calculated: 1) Task completion rate 1) Cumulative system rewards; 2) Multi-slot optimization objective function value; 3) Average task latency.

[0126] like Figures 7-10 As shown in the experimental results, under different load scenarios, it can be observed that when the system load is low, the performance differences between various algorithms are relatively limited. This is because when resources are relatively abundant, the server resource competition effect has not yet been significantly manifested. Even if a greedy strategy or static optimization method is used, it can meet most task requirements, so the difference between the task completion rate and the optimization target value is small.

[0127] However, as the number of devices increases, the number of servers decreases, the task generation interval shortens, or the batch load increases, the system gradually enters a high-load operating state. At this stage, the performance differences between different algorithms begin to widen significantly. Stochastic algorithms, lacking any structured decision-making mechanism, exhibit significant volatility in resource allocation, leading to a rapid decline in task completion rate and a significant deterioration in total reward. While greedy algorithms can achieve high immediate gains within a single time slot, their decisions are based solely on the current state, without considering resource consumption and task arrival trends in future time slots. Therefore, under high-load conditions, they are prone to causing resource congestion in subsequent time slots, manifested as increased average task latency and a decrease in the optimization objective value.

[0128] In contrast, the alternating optimization algorithm optimizes resource allocation through block-based iteration within a single time slot, exhibiting more stable performance compared to the greedy method. However, since its decision-making is still limited to a static time slot framework and it does not introduce cross-time slot coupling constraints or future load prediction mechanisms, its performance improvement potential is limited when the load continues to grow.

[0129] The proposed algorithm demonstrates superior task completion rate and system reward level across all load-enhancing scenarios. Its performance advantage is particularly pronounced in high-load regions. For example, in the Set4 experiment with batch load enhancement, as the task batch limit gradually increases from 10 to 50, the algorithm's completion rate decreases significantly less than other algorithms, while the optimization objective value remains relatively stable. This indicates that the multi-slot rolling optimization mechanism can reserve future resources in advance during the current decision-making stage, thereby mitigating the resource competition effect caused by time coupling.

[0130] From a theoretical perspective, the algorithm's advantages primarily stem from its prediction of future states and its rolling execution mechanism. Within each time slot, the algorithm estimates future task arrivals and resource consumption based on a finite prediction window and executes only the first step of the optimization sequence. This closed-loop "prediction-execution-update" mechanism enables the system to dynamically correct deviations, thereby avoiding long-term resource allocation imbalances. Therefore, in dynamic environments, the algorithm can effectively reduce global performance degradation caused by local optimal decisions.

[0131] In this embodiment, the impact analysis of key parameters is as follows: like Figure 11 As shown, the size of the scrolling time slot window was adaptively adjusted during the specific execution of the algorithm. Now, we will conduct relevant experiments on the impact of this key parameter of the algorithm.

[0132] To further analyze the impact of the key parameter in the proposed algorithm framework—the prediction window length T—on system performance, this section compares the performance of fixed window lengths (T=0, 5, 10, 20) and adaptive window strategies under different batch load scenarios. To more intuitively demonstrate the comparison of experimental results for different window lengths, the experimental metrics corresponding to the adaptive window are used as the baseline, and the mean and standard deviation of the experimental metrics for different windows are calculated.

[0133] Experimental results show that when T=0, the algorithm degenerates into a single-slot greedy decision-making mechanism, with performance significantly worse than the predictive approach, especially in high-load regions where both task completion rate and optimization objective value decrease significantly. As the prediction window length increases, the algorithm can coordinate resource allocation over a longer timescale, thus improving both system reward and completion rate. However, when the window length further increases, the performance improvement plateaus, and even exhibits slight fluctuations in some scenarios. This is because an excessively long prediction window introduces a large accumulation of prediction errors and increases the complexity of the optimization problem, making current decisions overly sensitive to uncertainties in the future.

[0134] The adaptive window mechanism achieved the most stable performance across various load scenarios. Its core idea is to use a shorter prediction window under low load conditions to reduce computational complexity, while automatically expanding the prediction range under high load or resource constraints, thereby enhancing forward-looking decision-making capabilities. Experimental results show that the adaptive window maintains superior or optimal performance in three metrics: task completion rate, average reward, and optimization target value, and exhibits stronger robustness under different load variation trends.

[0135] From a system control perspective, the prediction window length determines the sensitivity of decisions to future states. A window that is too short leads to policy short-sightedness, making it difficult to mitigate cross-time-slot resource contention; a window that is too long may amplify the impact of prediction errors. Adaptive mechanisms achieve a balance between prediction accuracy and decision-making foresight by dynamically adjusting the prediction depth based on the load, thus achieving a balance between performance and stability.

[0136] Taking time slot τ as an example, the management platform needs to predict the task demands that may arrive from time slot τ to τ+n. Task demands include the number of tasks, computational load, batch size, etc., which allows the platform to predict the distribution of time slot lengths occupied by the server.

[0137] 1.1 Basic Random Variables Equipment Collection Random variables have Indicates whether the device generates inference task data. For device u m Its task generation process is periodic n m The Bernoulli process. This generates the task cycle n. m and task generation probability p m : , ,at the same time This indicates the data type of the inference task and the batch inference requirements. Privacy requirements Channel transmission , which represents the time required to transmit a unit of data.

[0138] 1.2 Partition point distribution Task types generated by the device Randomly select from the set and and (A.1) Define the privacy degree sequence for the task type ψ l where p , where p ψl,i > p ψl,j When i < j, and the privacy requirement ρ m [τ] ∈ [ρ min , ρ max .

[0139] The case where the model partition point of device u m on the time slot τ model l is determined to be k is (A.2) For the task type ψ l , the probability that the partition point is exactly at k is (A.3) 1.3 Computation and transmission characteristics The local computation volume, edge computation volume, and intermediate data volume are respectively: and .

[0140] They represent the computation volume corresponding to the model processing of the corresponding partition, and numerically should be equal to the sum of the computation volumes of each layer constituting the partition.

[0141] The task batch size satisfies:

[0142] The corresponding batch size probability is (A.4) The local computation time is expressed as (A.5) For a given task type , the local computation time is related to the privacy requirement and the batch size, and its conditional distribution is , (A.6) where Ⅱ[·] is the indicator function.

[0143] Therefore, the overall distribution of the local computation time is: (A.7) Transmission rate meets Due to the uncertain mobility of the equipment, the transmission rate is set to within a certain range for ease of analysis. Internal random variation, then The probability density function is: (A.8) The data transmission time from the device is: (A.9) Wherein, batch size D m [τ] It is a discrete random variable. For model partitioning points, This is a function for intermediate data volume.

[0144] make Then the distribution of C is

[0145] Where Ⅱ[·] is the indicator function, ,

[0146]

[0147] Given C = C ψl ,k,d Under the condition that = c, the transmission time random variable The probability density function is: (A.11) Therefore, the probability density function of the transmission time is:

[0148] Expected number of tasks reaching the management platform: The duration of each time slot is known to be So, device u m The number of time slots required for the remaining part of the inference task data generated in time slot τ to reach the management platform is: (A.13) Define the global constant δmax as the maximum time slot offset that a task in the system may require, then: (A.14) Since the local computational load and intermediate data volume are fixed once the model split points are determined, the maximum time slot offset that the computation task may need can be calculated based on the known model type and split points.

[0149] The number of tasks arriving at the management platform in time slot t is:

[0150] The expected number of tasks arriving at the management platform in time slot t is: (A.16) Due to α m [τ] With δ m [τ] They are mutually independent and have: (A.17) therefore, (A.18) Know device u m With period n m probability p m Generate inference task data, therefore we can obtain: (A.19) Meanwhile, the probability that the time slot offset of the task arriving at the management platform is δ is: (A.20) The distribution of local computation time and transmission time has already been derived in the previous text. Given... , ,as well as At that time, the local calculation time is a fixed value: (A.21) because: We can obtain: (A.22) Therefore, the total execution time The conditional probability density function is:

[0151] Since the local computation time is a fixed value once the privacy requirements and the corresponding task type are determined, the conditional probability density function of the total execution time should also be: .

[0152] In the unconditional case, the probability density function of the total execution time is: (A.24) Time slot offset probability function:

[0153]

[0154] Ultimately, the expected number of tasks arriving at the management platform in time slot t can be obtained.

[0155] .

[0156] In the previous section, we calculated the local computation and transmission times by inversely deducing the task attributes, thus obtaining the expected number of tasks arriving at the management platform in time slot t when the time slot offset exactly matches these two values. However, predicting solely based on the expected number of tasks is insufficient; we also need to predict the number of time slots expected to be occupied by the server based on the task load. Since the random variables task batches and model partitioning layers are discrete variables, we cannot simply use probability distributions to calculate the expected number of tasks. For example, the time slot offset probability formula already includes explicit definitions for discrete factors such as model type, partitioning point, and batch.

[0157] (A.27).

[0158] Therefore, the expected computational load can be calculated using these explicit definitions. The computational load corresponding to a task generated in time slot t-δ and arriving in time slot t is defined as follows:

[0159] Therefore, the expected average load observed in time slot t should be:

[0160]

[0161] (A.28) The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A model parallel inference scheduling method, characterized in that, It is applied to a management platform that communicates with multiple edge servers, including: The deep neural network inference task is obtained based on IoT devices. The inference task includes model type, data batch, privacy requirements and maximum tolerable completion time. The model segmentation point of the IoT device is determined based on the privacy requirements. The model layer calculation before the segmentation point is executed locally. The intermediate data and remaining calculation tasks corresponding to the model layer after the segmentation point are sent to the management platform through the access point. Obtain the real-time load status and computing power of each edge server; Based on the periodic Bernoulli process generated by IoT devices, the probability distribution of model types, the probability distribution of privacy requirements, and the random distribution of channel transmission rates, task load prediction data for multiple future time slots is obtained. Based on the real-time load status, computing power, and task load prediction data, a rolling time-domain framework is used to enumerate all edge server allocation data corresponding to the inference task in the current time slot to obtain the target edge server and the estimated completion time. Based on the target edge server within a finite time domain of the prediction window, a greedy algorithm is used to accumulate the single-time slot completion progress reward of all tasks being processed in each time slot based on the task load prediction. The allocation data with the largest accumulated completion progress reward is selected as the target allocation data for the current time slot. The inference task and the corresponding intermediate data are distributed to the target edge server, and calculations are performed based on the model layer after the segmentation point of the target edge server. Obtain the inference results returned by the target edge server, and based on the actual completion time of the inference task and the estimated completion time, obtain the latest task completion rate and the reputation value of the management platform.

2. The model parallel inference scheduling method according to claim 1, characterized in that, The real-time load status includes the completion time of all pending tasks on each server within the current and future time slots.

3. The model parallel inference scheduling method according to claim 2, characterized in that, The task load prediction data includes expected data on the number of tasks arriving and expected data on the task computation load.

4. The model parallel inference scheduling method according to claim 3, characterized in that, Tasks for acquiring deep neural network inference based on IoT devices include: The time for acquiring deep neural network inference tasks is divided into multiple discrete time slots; At the beginning of each integer time slot, obtain all inference tasks that arrived in the previous time slot to obtain the deep neural network inference task.

5. The model parallel inference scheduling method according to claim 4, characterized in that, Determining the model segmentation point for IoT devices based on the aforementioned privacy requirements includes: Based on different segmentation points, the structural similarity between the reconstructed data obtained from the intermediate data and the original data for each deep neural network model is calculated. The structural similarity is used as a quantification value for privacy risk. Select the largest split point that satisfies the privacy requirements, such that the corresponding privacy risk quantification value does not exceed the threshold specified by the user.

6. The model parallel inference scheduling method according to claim 5, characterized in that, The computing power is a non-linear negative correlation function between the computing speed of multiple edge servers and the number of connected devices: , This represents the minimum computing power that an edge server can provide for a single inference task. This represents the maximum computing power that an edge server can provide for a single inference task. The number of devices accessing the edge server in time slot τ. This is a function representing the computational speed of the edge server. This represents the total computing power of the server.

7. The model parallel inference scheduling method according to claim 6, characterized in that, Based on the periodic Bernoulli process generated by IoT devices, the probability distribution of model types, the probability distribution of privacy requirements, and the random distribution of channel transmission rates, task load prediction data for multiple future time slots is obtained, including: Based on the task generation cycle and generation probability of IoT devices, calculate the probability of generating inference tasks in any time slot; Based on the probability distribution of model type and the probability distribution of privacy requirements, the probability distribution of model segmentation points is obtained; Based on the model segmentation points, the computation time, transmission time, and probability distribution are obtained, and the time slot offset probability of the inference task reaching the management platform is calculated. Based on the time slot offset probability, the expected number of tasks and the expected computing load arriving at the management platform in the target time slot are calculated.

8. The model parallel inference scheduling method according to claim 7, characterized in that, The calculation method for the progress bonus for a single time slot is as follows: R k [τ] , where Δw k [τ] w is the maximum task load that an edge server can process in parallel for each inference task in time slot T. k [τ,iem] It represents the progress of the inference task within the current time slot.

9. The model parallel inference scheduling method according to claim 8, characterized in that, The length of the prediction window is a length range, which includes an upper limit and a lower limit, with the upper limit being greater than the lower limit. When the system load is below the preset threshold, the value of the prediction window is the lower limit of the length range; when the system load is above the preset threshold, the value of the prediction window is the upper limit of the length range.

10. The model parallel inference scheduling method according to claim 9, characterized in that, Based on the actual completion time and the estimated completion time of the inference task, the latest task completion rate and the reputation value of the management platform are obtained, including: Construct an indicator function based on the actual completion time and estimated completion time of the inference task: When the indicator function is equal to 1, the inference task is completed and the upper limit of the reputation value is obtained; otherwise, the inference task is not completed within the estimated completion time and the lower limit of the reputation value is obtained. The upper limit of the reputation value is greater than the lower limit of the reputation value.

Images