A method for joint batch processing and resource allocation of edge large model inference based on double time scale

Abstract

This invention discloses a joint batch processing and resource allocation method for edge large-scale model inference based on dual time scales, comprising: dividing time into macro time slots and micro time slots; within the macro time slots, acquiring the queue backlog, queue length, request overdue degree, and current computing resource ratio during the pre-filling stage, and updating the computing resource allocation ratio for the pre-filling and decoding stages accordingly; within the micro time slots, acquiring device requests, communication status, request characteristics, and the updated resource ratio, and determining batch partitioning using adaptive batch processing, while allocating uplink bandwidth to each device; executing transmission and batch loading according to the batch partitioning and bandwidth allocation results, and performing joint inference under the updated resource ratio. This invention, through dual-time-scale hierarchical optimization, dynamically matches computing resources and load, reduces batch filling overhead, and collaboratively allocates communication and computing resources, thereby improving resource utilization efficiency, reducing end-to-end latency, and increasing long-term throughput and request latency satisfaction.

Description

Technical Field

[0001] This invention belongs to the field of edge intelligence and distributed resource optimization technology, and in particular relates to a method for joint batch processing and resource allocation of edge large model inference based on dual time scales. Background Technology

[0002] Large Language Models (LLMs), a significant milestone in the rapid development of artificial intelligence, are now widely used in various fields, such as communication network management and optimization, code generation, and data management. LLMs' excellent generation capabilities stem from their ability to fit massive amounts of data to large-scale parameter networks. However, LLM inference also requires substantial computational resources for forward propagation. Therefore, large model inference requests are often provided via cloud services. However, industrial applications are highly sensitive to latency requirements, and deploying LLMs at the edge of wireless networks promises to provide low-latency LLM inference services for industrial production. Compared to cloud services, edge servers have limited computing and memory resources. Furthermore, the dynamic wireless communication environment and limited communication resources introduce uncertainty into edge LLMs, ultimately reducing the inference performance of edge inference systems.

[0003] In wireless edge LLM inference scenarios, requests from multiple industrial devices are sent to the edge server via wireless channels for inference. Throughout the inference pipeline, end-to-end latency primarily consists of transmission latency and inference latency, with limited communication resources being a significant factor influencing LLM inference end-to-end latency. Simultaneously, the wireless channel status of industrial devices changes dynamically, but their diverse applications face varying inference latency requirements. To maximize inference satisfaction for all devices in the system, communication resources must be dynamically allocated based on the real-time wireless spectrum efficiency of each device to adapt to the changing communication environment and meet differentiated inference latency needs. When the requested input data is sent to the edge server, the model inference phase begins. LLM inference essentially involves autoregressively predicting the next token based on existing tokens. For a Decoder-only architecture, the inference process consists of a Prefill phase and a Decode phase. The Prefill phase calculates the Key and Value values ​​(called KV Cache) based on the user's input token and caches them in memory, while simultaneously generating the first output token. Since the user's input Prompt is known, multiple Prompts can be batch-processed at once to fully utilize the parallel computing capabilities of the GPU. During the batch loading phase, the input tokens vary in length. When multiple input tokens of different lengths are loaded into a batch, shorter tokens will be padded to the maximum length of the input tokens in that batch. If the token lengths within the same batch differ significantly, it will result in considerable waste of computational and memory resources. Therefore, the batching strategy should consider the distribution of requested input lengths to reduce padding overhead. Furthermore, in real-time inference scenarios, due to varying user transmission latency, while batch inference can reduce prefill latency, loading batches may introduce additional waiting latency. The requested transmission time depends on the device's wireless communication environment and bandwidth resource allocation strategy; this coupling makes joint bandwidth allocation and batching strategies crucial.

[0004] After the batch completes the Prefill phase, the Decode phase is responsible for autoregressively generating output tokens based on the KV cache generated by the Prefill. The computational load of the Decode phase increases quadratically with the output length (detailed in the system model in Part 3). Meanwhile, in a dynamic inference environment, the input and output lengths of inference requests generated by different users change dynamically and randomly, causing the relative computational load between the Prefill and Decode phases to constantly fluctuate. A pre-defined fixed-ratio computational resource allocation strategy can lead to a mismatch between resource allocation and computational demands, ultimately reducing the inference performance of the edge server. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a joint batch processing and resource allocation method for edge large model inference based on dual time scales, thereby resolving the issues present in the prior art.

[0006] To achieve the above objectives, this invention provides a method for joint batch processing and resource allocation of edge large-scale model inference based on dual time scales, comprising: The system time is divided into macro time slots and micro time slots; At the beginning of each macro slot, the higher-level state of the system is obtained; the higher-level state includes the queue backlog, queue length, request overdue degree, and the current computational resource allocation ratio between the pre-filling stage and the decoding stage; based on the higher-level state, the updated computational resource allocation ratio between the pre-filling stage and the decoding stage within this macro slot is determined. At the beginning of each micro-timeslot, the low-level state of the system is acquired; the low-level state includes the device request generation state, communication environment state, request characteristics and updated computing resource allocation ratio; based on the low-level state, an adaptive batch processing strategy is adopted to determine the request batch division within the current micro-timeslot, and uplink communication bandwidth is allocated to each device. Based on the request batch division and bandwidth allocation results, the transmission and batch loading of the request are performed, and joint inference of the pre-filling stage and the decoding stage is performed under the updated computing resource allocation ratio.

[0007] Optionally, the request overdue level is obtained by calculating the difference between the elapsed time of the request in the queue and the expected delay; the current computing resource allocation ratio between the pre-filling stage and the decoding stage includes the current computing resource allocation ratio to the pre-filling stage and the current computing resource allocation ratio to the decoding stage, and the sum of the two is 1; the updated computing resource allocation ratio between the pre-filling stage and the decoding stage also satisfies the condition that the sum of the two is 1.

[0008] Optionally, the request characteristics in the lower-level state include the input token length, output token length, and expected inference delay for each request; the communication environment state includes the spectral efficiency vector of all devices in the current micro-time slot; and the device request generation state is the request generation indication vector of all devices in the current micro-time slot.

[0009] Optionally, the sum of the uplink communication bandwidth allocated to each device shall not exceed the total system bandwidth, and the bandwidth allocated to each device shall be non-negative; in the request batch division, each request belongs to only one batch, and the number of batches in the current micro-time slot shall not exceed the total number of requests in that micro-time slot.

[0010] Optionally, the process of determining the updated computational resource allocation ratio between the pre-filling phase and the decoding phase based on the higher-level state includes: The high-level states are concatenated into a state vector, which is then input into a deep Q-network. The deep Q-network outputs the action value corresponding to the allocation ratio of each candidate resource. A candidate resource allocation ratio is randomly selected with a preset exploration probability to perform exploration, and a candidate resource allocation ratio with the highest action value is selected with a preset utilization probability to perform utilization. The selected candidate resource allocation ratio is used as the computational resource ratio of the updated pre-filling stage, and the computational resource ratio of the decoding stage is automatically determined to be 1 minus this ratio. After the current macro slot ends, the total throughput of all requests that completed inference within the macro slot is obtained as a reward. The parameters of the deep Q network are updated using the reward through the loss function, and the target Q network is updated in a soft update manner.

[0011] Optionally, the process of determining the request batch division within the current micro-slot using an adaptive batching strategy based on the lower-level state includes: The features of all requests in the lower-level state and the device spectral efficiency vector are concatenated into a global feature vector; the global feature vector is input into the encoder of the pointer network, and the encoder outputs encoded features; the hidden state of the pointer network decoder is initialized, and the hidden state is obtained by linear transformation of the global feature vector; at each decision step, the decoder calculates attention weights based on the current hidden state and the encoded features, and obtains the log probability of each unassigned request as the next action and the log probability of creating a new batch action; an action is sampled from the log probability distribution. If the sampled action is a request index, the request is added to the current batch and marked as assigned. If the sampled action is a new batch action, the construction of the current batch is completed and the next empty batch begins; the above decision steps are repeated until all requests are assigned to a batch.

[0012] Optionally, during the decoding process of the pointer network, an action mask mechanism is used to maintain the index set of allocated requests; when calculating the log probability, the log probability value corresponding to the allocated request is set to negative infinity, so that the sampling probability of the allocated request is zero; when a new batch action is sampled, the request set of the current batch is cleared, the batch cache is reset, and the hidden state of the next decision step is updated to the initial value.

[0013] Optionally, the process of allocating uplink communication bandwidth to each device includes: The global feature vector of the low-level state is input into a multilayer perceptron network, which outputs the unnormalized log probability of each device. This unnormalized log probability is used as the initial score for bandwidth allocation. The set of devices that have generated requests within the current micro-time slot is obtained, and the initial score corresponding to the devices that have not generated requests is set to negative infinity to obtain a masked score vector. The masked score vector is input into a Softmax function for normalization to obtain the bandwidth allocation probability distribution of each device. The actual bandwidth allocated to each device is determined according to the probability distribution, such that the sum of the bandwidths of all devices equals the total system bandwidth, and the bandwidth of each device is proportional to its probability value.

[0014] Optionally, when performing joint inference in the pre-filling and decoding phases, the computational load of the pre-filling phase is determined based on the maximum input token length and model parameter size within the batch, and the computational load of the decoding phase is determined based on the square of the output token length and model parameter size. The end-to-end latency of each request is obtained by summing the transmission latency, batch waiting latency, pre-filling latency, and decoding latency. The transmission latency is determined based on the input token length of the request and the bandwidth allocated to the corresponding device. The batch waiting latency is equal to the difference between the latest transmission completion time of all requests in the batch and the start time of pre-filling of the batch. The pre-filling latency is determined based on the computational load of the pre-filling phase and the computational resources allocated to the pre-filling phase. The decoding latency is determined based on the computational load of the decoding phase and the computational resources allocated to the decoding phase. The end-to-end latency of each request does not exceed the expected inference latency of that request.

[0015] Optionally, each macro slot contains an integer number of micro slots; at the end of each macro slot, a high-level reward is generated based on the total throughput of all requests that complete inference within that macro slot, and the high-level reward is used to update the parameters of the deep Q network; at the end of each micro slot, a low-level reward is generated based on the batch readiness latency and request timeout level within that micro slot, and the low-level reward is used to update the parameters of the pointer network and the multilayer perceptron network, the update adopts a near-end policy optimization algorithm, and the policy update magnitude is limited by pruning probability ratios.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: This invention employs a dual-timescale framework that divides system time into macro-time slots and micro-time slots. Within the macro-time slots, the computational resource allocation ratio is dynamically adjusted based on the computational load backlog and queue length during the pre-filling and decoding phases. Within the micro-time slots, an adaptive batching strategy and uplink bandwidth allocation are jointly used to allocate requests with similar input lengths to the same batch, reducing filling overhead. Simultaneously, transmission and batch processing waiting latency are collaboratively optimized based on channel state and request characteristics. Furthermore, hierarchical reinforcement learning is used to achieve joint training of high-level resource reallocation and low-level batch processing and bandwidth allocation. This significantly improves computational resource utilization efficiency, reduces end-to-end inference latency, and enhances long-term system throughput and request latency satisfaction in edge large-model inference scenarios. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is the architecture of an industrial wireless edge large-model inference system according to an embodiment of the present invention; Figure 2 This is a schematic diagram showing the inference performance test results of Qwen2-1.5B-Instruct on Jetson Orin NX according to an embodiment of the present invention; Figure 3 This invention presents a joint batch processing and multidimensional resource allocation algorithm based on hierarchical reinforcement learning. Figure 4 This is a flowchart illustrating the training phase implementation of an embodiment of the present invention. Figure 5 This is a flowchart illustrating the inference phase of an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] Example 1 This embodiment provides a method for joint batch processing and resource allocation of edge large model inference based on dual time scales, including: The JBABS framework considered in this embodiment is as follows: Figure 1As shown. The system considered in this embodiment includes a base station (BS) that covers multiple Industrial Autonomous Equipments (IAEs), denoted as... An edge server (ES) is deployed alongside the base station (BS) and directly connected via a fiber optic link. The edge server deploys a large model (LM) inference service. System time is divided into discrete time slots. At the start of each time slot, IAEs within the system randomly generate LM inference requests.

[0021] 1.1 Request Model; Each time slot t have N t A set of inference requests is generated, and the set of inference requests is represented as... ,but Using variables To indicate the first t IAE within each time slot v Has a reasoning request been generated? If so, then ,otherwise The inference request is represented as ,in lt v The length of the input token requested. ot v The length of the output token required by this request. The expected inference latency is given. Since the output length can be predicted with high accuracy, this embodiment assumes that the output length level of the inference is known.

[0022] 1.2 Communication model; Assume the channels in the system are non-frequency selective and decay with distance. The total spectral bandwidth of BS is expressed as... B Using measurement techniques such as CSI-RS, the controller can estimate channel gain and uplink transmission power, and thus use this information to allocate bandwidth to IAEs. (Time slots...) t The spectral efficiency is expressed as: (1) in, For IAE v The transmission power, For power gain, The power is additive white Gaussian noise. The time slot... t Assigned to the v The channel bandwidth of each IAE is expressed as: This embodiment employs a quasi-static channel model, meaning the channel state of the IAE remains constant within each time slot, but changes between different time slots. Therefore, the IAE... v In the time slot t The transmission rate is: (2) Therefore, request Transmission delay for: (3) 1.3 Batch processing model; Once a request arrives at Elasticsearch (ES), it will enter the Prefill phase in batches. This is because loading multiple requests into a single batch for processing during the Prefill phase of LLM inference is often faster than processing multiple requests individually in parallel. In the execution mode where multiple requests are inferred individually (batch size 1), the GPU will initiate multiple kernel launches, ultimately leading to a decrease in inference efficiency. t A batch of time slots is represented as Define batch decision variables as ,when Time indicates time slot t Requests within Assigned to the first k One batch.

[0023] The inference batch size is represented as b Using functions η ( b The expression represents the relationship between device inference speed and batch size. This embodiment is based on empirical experiments conducted using a Jetson OrinNX, TensorRT-LLM inference framework, and Qwen2-1.5B-Instruct model, equipped with an 8-core 2GHz CPU, a 1024-core Ampere architecture GPU, and 16GB of memory. Figure 2 As shown, experimental results indicate that increasing the batch size can significantly increase the throughput of LLM.

[0024] Therefore, the batch readiness time is the slowest transmission time of the requests within the batch. (This refers to the time slots.) t The inference batch representation at time is Therefore, for each request The batch to which it belongs is indicated as For any batch The readiness time for its Prefill phase is: (4) Meanwhile, during the batch loading process, shorter input tokens will be padded to the maximum length of tokens in the batch. Therefore, the amount of KV cache memory occupied by each request in the batch during the Prefill phase depends on the length of the longest input token in the batch.

[0025] 1.4 Computational Model; Once data allocated to the same batch arrives and memory resources are sufficient, inference is performed in batches. Decoder-only LLM models have achieved excellent performance on a wide range of tasks, such as GPT-4 and Deepseekv3. Therefore, this embodiment uses a large model based on a Decoder-only architecture as an example to model the request inference process. The inference process of a Decoder-only LLM can be divided into two stages: Prefill and Decode. In the Prefill stage, the model accepts all input tokens and computes the key and value vectors of all input tokens, as well as the first output token. Since all input key and value vectors can be reused throughout the inference process, they are stored as a KV cache. The Decode stage generates output tokens recursively. In each iteration, the model combines the input prompt and the generated output to predict the next output token as needed. This relies on both the KV cache and the computed key and value of the newly generated token from the previous iteration. The Decode process continues to iterate until the maximum generation length is reached or an end-of-sequence (EOS) token is generated. The computational latency of the two stages is described below: Pre-filling stage: In the pre-filling stage, the key and value vectors of each input prompt are obtained through linear transformation. The Q, K, and V vectors are obtained from the embedding input linear transformation layer of the input prompt. Assuming the large model has... M There are 10 Transformer layers, and each Transformer has 100 Transformer layers. I The first point of attention, the first m The first Transformer layer i The parameters of the three linear transformation matrices of the self-attention head are: This embodiment assumes The hidden layer dimension of the Transformer is represented as... d 1. The input token first obtains its embedding vector through a lookup table; this process requires no computation. Then it passes through a linear layer. , , Calculate the QKV of the output token from the previous iteration, i.e. , , Because requests within the same batch will be padded to the longest input length in that batch, the batch... B The computational load for all requests within the Prefill phase is: , (5) in Assume the total computing power of ES is C (in FLOPS), assuming t The proportion of resources allocated to the Prefill phase for each time slot is Then the prefill latency for all inference requests within the batch is: , (6) in For batch B The time to start Prefill.

[0026] Decoding Phase: The decoding phase generates subsequent tokens (excluding the first character) based on existing output tokens in an autoregressive manner. The computation of the self-attention mechanism can be represented as: (7) The calculation for the multi-head attention mechanism is as follows: (8) in, It is a linear transformation matrix used for multi-head attention fusion. Therefore, each Transformer layer generates the first... The computational load required for each token is: (9) in, This represents the intermediate layer dimension of the feedforward network. During each iteration of the decoding phase, its input length gradually increases from 1 to... According to the summation formula The computational load of the Decode process can be obtained as follows: (10) Clearly, the overhead of the decoding process increases with the output length. The speed of growth increases, and the rate of increase accelerates as the output length grows. Following the design principles of advanced continuous batching, once the prefill phase of a request is complete, it can directly enter the demodulation phase without waiting. Therefore, the request... The time taken for the Decode phase is: (11) Therefore, request The processing latency includes wireless transmission latency, waiting latency, and inference latency. It can be represented as: (12) 1.5 Memory Model; During LLM inference, memory usage primarily comes from two parts: weights and the key-value cache. In the Decode-only Transformer architecture, weights include... and Before the first inference service begins, the model weights need to be loaded into GPU memory and will reside thereafter. Therefore, the memory overhead of a large model's weights remains constant throughout the inference process. The memory usage of the weights can be represented as... .

[0027] Another portion of memory usage comes from the KV cache usage during the Prefill and Decode phases. In the Prefill phase (along with the attention matrix), Prompts within the same batch are first padded to the maximum batch length, and then the K,V mapping value for each token is calculated in parallel through a linear transformation. Therefore, for a request... The memory usage during its Prefill phase is as follows: (13) During the Decode phase, each time the next token is predicted, the key-value (KV) value of the previous token needs to be calculated and concatenated with the KV cache of previously existing tokens. Therefore, the request... The memory usage of the KV cache during the Decode phase is: (14) in, The number of bits used to encode a single floating-point number is determined by the quantization precision. This applies when the quantization precision is INT8, FP16, or FP32. They are 1 Byte, 2 Bytes, and 4 Bytes respectively.

[0028] Therefore, request The total GPU memory consumed during the Prefill and Decode stages is: (15) 1.6 Problem Formulation; Since there is a trade-off between overall system throughput and end-to-end latency at the request level, this embodiment constructs the problem as maximizing the long-term weighted system throughput and request latency satisfaction. Throughput represents the total number of request tokens processed per unit time. The overall optimization objective can be expressed as: (16) in, This indicates the maximum GPU memory capacity of ES. This represents the set of requests being processed in the t-th time slot (in either the Prefill or Decode phase). Constraint 1 states that the sum of bandwidth across all devices within a time slot cannot exceed the maximum available spectrum bandwidth. Constraint 2 states that the number of batches in each time slot cannot exceed the number of requests generated. Constraint 3 states that each request can belong to only one batch. Constraint 4 states that the resources allocated to Prefill and Decode cannot exceed the maximum available resources and cannot be zero. Constraint 5 states that the end-to-end latency of all requests should be less than the expected latency. Constraint 6 states that the KV cache occupied by requests being processed by Elasticsearch cannot exceed the maximum GPU memory capacity.

[0029] The proposed LLM edge inference joint optimization problem is essentially a combinatorial optimization problem, with decision variables including request-to-batch allocation, wireless bandwidth allocation, and computational resource allocation during the Prefill / Decode phase. Even when neglecting communication dynamics and considering only batch processing and resource allocation for a given arrival request, the problem can still be modeled as a coupled-constrained variable-size bin packing problem: the resource consumption (computation and memory) of each batch depends on the maximum input length and number of requests it contains, and different batches share limited GPU computing power and memory capacity. More specifically, if each possible batch is considered as an "item" and each possible batch corresponds to a "box," since the Variable-Sized Bin Packing (VSBP) problem has been proven to be NP-hard, and this problem introduces additional time-delay coupling constraints and nonlinear resource functions, the proposed optimization problem is NP-hard.

[0030] Since the optimization objective is long-term and communication and computing resources are limited, current decisions affect subsequent system states. Therefore, this problem is modeled as a Markov Decision Process (MDP). Markov Decision Processes can capture the temporal relationships between decisions, i.e., how current decisions affect subsequent system states and rewards. Furthermore, the randomness of request arrivals naturally aligns with the MDP framework. An MDP includes a state space. Action space Transition probability Reward function Discount Factor γ Five basic elements. Given an initial state. s 0, the agent is in any state According to strategy Select Action Post-environmental state transition to At the same time, the intelligent agent receives rewards from environmental feedback. .in θ Here are the parameters of the policy network. The objective is to maximize the long-term cumulative reward, expressed as: , (17) Among the discount factors This determines the impact of future rewards on current rewards. γ A larger value indicates a greater present value of future rewards at the current step, thus guiding the agent to make longer-term decisions. In our batch and resource allocation decision problem, both system throughput and request latency are related to two processes. Bandwidth allocation and batching decisions mainly affect the transmission process between requests entering the prefill stage and the batch loading process (referred to as Stage 1). Computational resource allocation decisions affect the computation process between the prefill and decode stages (referred to as Stage 2). Therefore, the optimization objective can be further decoupled based on the characteristics of the two stages, thus enabling hierarchical optimization. In Stage 1, the optimization objective is designed to minimize the batch prefill readiness latency, i.e., the latency from request generation to batch loading completion. In Stage 2, the optimization objective is designed to minimize the total computational latency of the prefill and decode stages. The design of the state space, action space, state transition probabilities, and reward functions at the higher and lower levels will be introduced below.

[0031] 2.1 MDP Design; (1) High-level MDP design; state space Define macro slots as At the beginning of each macro time slot, the agent acquires the state of the system. and make the corresponding actions. a ( t The status of each large time slot includes the queue backlog during the prefill phase, queue length, PD resource allocation ratio, and request timeout level. The action can be represented as: (18) in, The computational load backlog is respectively for the prefill and decode stages. It represents the timeout level of requests within the queue, calculated by the difference between the time already consumed and the expected time for requests within the queue.

[0032] Action space In each time slot The high-level strategy needs to output the resource allocation ratio for the prefill phase, and from this, obtain the ratio for the decode phase, thereby adjusting the resource allocation ratio for the PD phase. The action of each time slot can be represented as: (19) reward function In each time slot The reward is given based on the system's throughput in the previous macro slot, and the reward function is defined as follows: (20) in, μ 1 represents the weighting coefficient. Indicates the first The request for completion of each macro slot execution.

[0033] (2) Low-level MDP design; state space At the beginning of each micro-time slot, the agent determines the system state. s ( t Make corresponding batch processing and communication resource allocation decisions. The status of each micro-timeslot includes the request generation status of all devices, communication environment status, request status, and PD resource allocation ratio. Each timeslot The state can be represented as: (twenty one) in, , This means concatenating all elements to obtain a concatenated vector. , representing the spectral efficiency vector of all devices. and ,in, .

[0034] Action space In each time slot t High-level strategies require output bandwidth allocation decisions. Batch processing decision and , can be represented as: , (twenty two) in and .

[0035] reward function In each time slot t The reward is determined based on the bandwidth allocation and batch processing decisions given in the current time slot. The reward function, which includes the batch readiness latency and request timeout level within the current time slot, can be expressed as: (twenty three) in, μ 2 and μ 3 indicates the reward weight.

[0036] like Figure 3 As shown, this scheme constructs a high-level computing resource reallocation network (running in each large time slot) and a low-level joint spectrum allocation and batch processing network (running in each small time slot), which operate at different time scales. During the training phase, according to... Figure 4 The algorithm shown undergoes iterative training. Once the high- and low-level scheduling strategies converge (achieving the desired effect), then... Figure 5 The flowchart shown illustrates how the converged policy network outputs scheduling actions at different time scales.

[0037] 2.2 High-level adaptive computing resource reallocation algorithm based on DQN; Deep Q-Network (DQN) is a reinforcement learning method based on Q-learning and deep neural networks. It combines the advantages of online learning with the function approximation ability of deep neural networks.

[0038] Q-learning is a value-iterative reinforcement learning method. The agent's goal is to find an optimal state-action value to obtain the optimal policy π, thereby maximizing long-term cumulative rewards. This is achieved by taking action a in state s. The optimal state-action value can be discovered using the state-action value Q-function, defined as follows: Where γ is the discount factor (0 ≤ γ ≤ 1). Is the agent in a state? Take action below The reward obtained at that time. Q-learning uses a Q-table to store Q-values ​​and uses an online update rule with a learning rate α (0 ≤ α ≤ 1), as follows: DQN Model Training: DQN uses a deep neural network to approximate the optimal state-action value Q-function, i.e.: Here, θ represents the set of parameters of the neural network. The training process of the DQN network includes the following steps: 1. Initialize the experience replay pool size and DQN network parameters; 2. Observe the environmental status at the beginning of each time slot; 3. Input the state features into the Q-network, which operates according to the following... Decision-making based on rules: ,in A random number between 0 and 1 This is the threshold for random decision-making; 4. Based on the action Adjust resource allocation ratios; 5. Generate the interaction trajectory tuple Store in the experience replay pool; 6. Randomly sample a batch of empirical tuples from the memory buffer, input these data into the DQN network, and calculate the value of the loss function, defined as follows: ; 7. Update the parameters of the DQN network using the Adam optimizer method until the value of the loss function converges.

[0039] 8. After a fixed number of training steps, replace the parameters of the target Q-network with the parameters of the Q-network: ;in This is a soft update coefficient; 9. Continue until the desired effect is achieved.

[0040] 2.3 Low-level joint spectrum allocation and batch processing network; In each hourly slot, the low-level policy needs to formulate spectrum allocation and batch allocation strategies. The low-level state vector for each hourly slot is represented as... The Actor network employs a multi-branch structure. The underlying state vectors first pass through a global feature extractor, namely a multi-layer perceptron (MLP). Next, bandwidth allocation decisions are obtained through an MLP and a softmax function. Since the network's output dimension is fixed, but the number of requests generated in each time slot may be less than the maximum number of users, the logit of the MLP output needs to be processed before entering the softmax function. L Applying a mask can be represented as (twenty four) Next, the output after masking is subjected to a softmax operation, which can be represented as follows: (25) For the batch processing strategy, global features are used to generate batch actions via Ptr-Net. Pointer networks are sequence-to-sequence architecture networks that use a pointer mechanism similar to attention to solve combinatorial optimization problems where the output is a subset or permutation of elements in the input sequence. Ptr-Net can flexibly handle variable-length inputs, making it suitable for the random request arrival scenario studied in this embodiment. The Ptr-Net network includes an encoder and a decoder, both composed of GRU units. During the output batch decision process, a hidden state is maintained. This state is dynamically updated with the number of decision steps (maximum being the length of requests within that time slot). The initial hidden state is obtained as follows: (26) in, and These are learnable parameters. At each decision step, the hidden state is updated via GRU units: (27) This allows the strategy to be aware of previously executed actions. The action space contains... M+ One option, of which This indicates that the corresponding first i Each request is assigned to the current batch, action M This indicates the start of a new batch. To this end, we introduce a set of learnable action embedding vectors. And stack them into a matrix The logits of action selection are calculated using a dot product attention mechanism, i.e. To ensure that a request index that has already been selected is not chosen during the decision-making process, logits are processed using a mask each time the next action is selected: (28) in, This is a mask for legitimate actions. Indicates the preceding k -1 step: The set of actions already selected, used to generate... In each hourly interval, the batch processing network progressively moves from... in an autoregressive manner. Mid-sampling action ,like If so, create a new empty batch; otherwise, request... Add it to the current batch and mark it as assigned. This process continues until all are included. MAll requests have been allocated. This policy network can flexibly generate batch decisions consistent with the input length, while the dynamic masking mechanism strictly satisfies the unique batch constraint of the requests.

[0041] The low-level policy network is trained using the PPO algorithm. PPO employs General Advantage Estimation (GAE) to balance bias and variance during the update process. The TD error can be expressed as... (29) The formula for calculating the dominance function is as follows: (30) in γ As a discount factor, These are the smoothing parameters for GAE. The training loss of the PPO algorithm includes value loss and policy loss. The policy loss is the ratio of the cutoff probabilities of the new and old policies. (31) in, Value loss can be expressed as (32) in, Therefore, the total loss function can be expressed as (33) in, and These are the weights for value loss and strategy loss, respectively.

[0042] The adaptive batch processing strategy is as follows: The hierarchical LLM inference service optimization scheme proposed in this application can optimize batch processing, bandwidth allocation and computing resource allocation strategies based on real-time network bandwidth and random request arrival information, ultimately improving the throughput of LLM inference service.

[0043] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for joint batch processing and resource allocation based on double time scale edge large model inference, characterized in that, Includes the following steps: The system time is divided into macro time slots and micro time slots; At the beginning of each macro slot, the higher-level state of the system is obtained; the higher-level state includes the queue backlog, queue length, request overdue degree, and the current computational resource allocation ratio between the pre-filling stage and the decoding stage; based on the higher-level state, the updated computational resource allocation ratio between the pre-filling stage and the decoding stage within this macro slot is determined. At the beginning of each micro-timeslot, the low-level state of the system is acquired; the low-level state includes the device request generation state, communication environment state, request characteristics and updated computing resource allocation ratio; based on the low-level state, an adaptive batch processing strategy is adopted to determine the request batch division within the current micro-timeslot, and uplink communication bandwidth is allocated to each device. Based on the request batch division and bandwidth allocation results, the transmission and batch loading of the request are performed, and joint inference of the pre-filling stage and the decoding stage is performed under the updated computing resource allocation ratio.

2. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, The request overdue level is obtained by calculating the difference between the elapsed time of the request in the queue and the expected latency; the current computing resource allocation ratio between the pre-filling stage and the decoding stage includes the current computing resource allocation ratio to the pre-filling stage and the current computing resource allocation ratio to the decoding stage, and the sum of the two is 1; the updated computing resource allocation ratio between the pre-filling stage and the decoding stage also satisfies the condition that the sum of the two is 1.

3. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, The request characteristics in the lower-level state include the input token length, output token length, and expected inference latency for each request; the communication environment state includes the spectral efficiency vector of all devices in the current micro-time slot; and the device request generation state is the request generation indication vector of all devices in the current micro-time slot.

4. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, The sum of the uplink communication bandwidth allocated to each device does not exceed the total system bandwidth, and the bandwidth allocated to each device is non-negative; in the request batch division, each request belongs to only one batch, and the number of batches in the current micro-time slot does not exceed the total number of requests in that micro-time slot.

5. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, The process of determining the updated computational resource allocation ratio between the pre-filling phase and the decoding phase based on the high-level state includes: The high-level states are concatenated into a state vector, which is then input into a deep Q-network. The deep Q-network outputs the action value corresponding to the allocation ratio of each candidate resource. A candidate resource allocation ratio is randomly selected with a preset exploration probability to perform exploration, and a candidate resource allocation ratio with the highest action value is selected with a preset utilization probability to perform utilization. The selected candidate resource allocation ratio is used as the computational resource ratio of the updated pre-filling stage, and the computational resource ratio of the decoding stage is automatically determined to be 1 minus this ratio. After the current macro slot ends, the total throughput of all requests that completed inference within the macro slot is obtained as a reward. The parameters of the deep Q network are updated using the reward through the loss function, and the target Q network is updated in a soft update manner.

6. The method for joint batch processing and resource allocation of edge large model inference based on dual time scales according to claim 1, characterized in that, The process of determining the request batch division within the current micro-slot based on the aforementioned low-level state using an adaptive batching strategy includes: The features of all requests in the lower-level state and the device spectral efficiency vector are concatenated into a global feature vector; the global feature vector is input into the encoder of the pointer network, and the encoder outputs encoded features; the hidden state of the pointer network decoder is initialized, and the hidden state is obtained by linear transformation of the global feature vector; at each decision step, the decoder calculates attention weights based on the current hidden state and the encoded features, and obtains the log probability of each unassigned request as the next action and the log probability of creating a new batch action; an action is sampled from the log probability distribution. If the sampled action is a request index, the request is added to the current batch and marked as assigned. If the sampled action is a new batch action, the construction of the current batch is completed and the next empty batch begins; the above decision steps are repeated until all requests are assigned to a batch.

7. The method for joint batch processing and resource allocation of edge large model inference based on dual time scales as described in claim 6, characterized in that, The decoding process of the pointer network adopts an action mask mechanism to maintain the index set of allocated requests; when calculating the log probability, the log probability value corresponding to the allocated request is set to negative infinity, so that the sampling probability of the allocated request is zero; when a new batch action is sampled, the request set of the current batch is cleared, the batch cache is reset, and the hidden state of the next decision step is updated to the initial value.

8. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, The process of allocating uplink communication bandwidth to each device includes: The global feature vector of the low-level state is input into a multilayer perceptron network, which outputs the unnormalized log probability of each device. This unnormalized log probability is used as the initial score for bandwidth allocation. The set of devices that have generated requests within the current micro-time slot is obtained, and the initial score corresponding to the devices that have not generated requests is set to negative infinity to obtain a masked score vector. The masked score vector is input into a Softmax function for normalization to obtain the bandwidth allocation probability distribution of each device. The actual bandwidth allocated to each device is determined according to the probability distribution, such that the sum of the bandwidths of all devices equals the total system bandwidth, and the bandwidth of each device is proportional to its probability value.

9. The method for joint batch processing and resource allocation of edge large-scale inference based on dual time scales as described in claim 1, characterized in that, When performing joint inference in the pre-filling and decoding phases, the computational load of the pre-filling phase is determined based on the maximum input token length and model parameter size within the batch, while the computational load of the decoding phase is determined based on the square of the output token length and model parameter size. The end-to-end latency of each request is obtained by summing the transmission latency, batch waiting latency, pre-filling latency, and decoding latency. The transmission latency is determined based on the input token length of the request and the bandwidth allocated to the corresponding device. The batch waiting latency is equal to the difference between the latest transmission completion time of all requests in the batch and the start time of pre-filling of the batch. The pre-filling latency is determined based on the computational load of the pre-filling phase and the computational resources allocated to the pre-filling phase. The decoding latency is determined based on the computational load of the decoding phase and the computational resources allocated to the decoding phase. The end-to-end latency of each request does not exceed the expected inference latency of that request.

10. The method for joint batch processing and resource allocation of edge large model inference based on dual time scales according to claim 1, characterized in that, Each macro slot contains an integer number of micro slots. At the end of each macro slot, a high-level reward is generated based on the total throughput of all requests that completed inference within that macro slot. This high-level reward is used to update the parameters of the deep Q-network. At the end of each micro slot, a low-level reward is generated based on the batch readiness latency and request timeout level within that micro slot. This low-level reward is used to update the parameters of the pointer network and the multilayer perceptron network. The update employs a near-end policy optimization algorithm, which limits the policy update magnitude by pruning the probability ratio.

Images