Inter-cloud large model inference task scheduling method based on reinforcement learning partitioning

Abstract

The application discloses an inter-cloud large model inference task scheduling method based on reinforcement learning blocking, and aims to solve the problem that the existing scheduling methods TBT and TTFT are difficult to effectively balance. The technical scheme is to first construct an inter-cloud large model inference task scheduling system based on reinforcement learning adaptive pre-filling blocking, which is composed of a task queue management module, a real-time state acquisition module, a dynamic adaptive pre-filling blocking module and a scheduling module; the scheduling system is trained, so that the trained reinforcement learning intelligent agent can dynamically divide the pre-filling blocks according to the task state and resource state of the system in real time; when the trained scheduling system schedules the inter-cloud task, the size of the pre-filling blocks is adaptively adjusted according to the task state and resource state of the system, and the tasks that have completed pre-filling are decoded at the same time, the calculation resources are efficiently utilized, the TTFT-TBT index is optimized, and the total running time is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of inter-cloud task scheduling, and in particular to an inter-cloud large model inference task scheduling method based on reinforcement learning for pre-filling block. BACKGROUND

[0002] With the development of large language model (LLM) technology and the rise of inter-cloud task scheduling, large language models have been practically applied in natural language processing, text generation, question answering, and chip design, and have shown strong capabilities. Inter-cloud tasks refer to computing tasks or workloads executed in infrastructure or platform environments that span more than one cloud computing service provider (public cloud, private cloud, or hybrid cloud). Inter-cloud task scheduling refers to the task scheduling center on the server side allocating computing resources to tasks submitted by different cloud clients. The tasks run on the allocated computing resources and return the results to the user. With more and more services migrating to inter-cloud platforms (i.e., systems composed of different cloud service providers), the number of tasks submitted by clients and resource requirements are increasing, and with the development of artificial intelligence, the number of AI tasks in inter-cloud large model inference tasks is growing more obviously. Inter-cloud large model inference tasks refer to large model inference request tasks from multiple cloud computing service providers. A large number of large model inference request tasks from multiple clients put a lot of pressure on large language models. Therefore, it is extremely urgent for cloud providers and users to study the scheduling method of inter-cloud large model inference tasks.

[0003] Inter-cloud large model inference tasks use large models for inference on the server side. Large language models perform inference on inter-cloud large model inference tasks, which mainly include the following steps:

[0004] First, the large language model pre-fills (i.e., pre-processes the task and constructs reusable intermediate states) the inter-cloud large model inference task to obtain the completed pre-filled inter-cloud large model inference task. The large model converts the task into a Token (i.e., token) sequence that the model can process (Token is the smallest semantic unit for processing text by a large language model. For example, the question of the inference task is "What's the weather today?", then "today", "weather", "how", and "?" are four Tokens), and through the forward propagation of the Transformer network in the large language model, the hidden state and key-value cache of the Token sequence (including the context information of the task, such as attention weights and feature vectors) are calculated at once to obtain the intermediate state. The obtained intermediate state is reused in subsequent decoding, avoiding repeated calculation for the same input, and significantly reducing the computational overhead of generating each new Token.

[0005] Secondly, the large language model decodes the pre-filled inter-cloud large model inference task (i.e. generates and outputs Token sequences based on the intermediate state generated in the pre-filling stage step by step). Based on the intermediate state generated in the pre-filling stage, the Token sequence is output through the word-by-word generation mechanism, which only processes one output Token of each task at a time. The completion of decoding indicates that the large model has completed the inference request for the task and obtained the inference text or answer required by the user.

[0006] The indicators for measuring the inference effect of the large language model on the inter-cloud large model inference task include the time to first token (TTFT), the time between tokens (TBT), and the total running time (the time taken by the large language model to complete the inference of N tasks within a fixed time period Time1). TTFT measures the delay in generating the first output Token from the time the task arrives at the large model, which is mainly determined by the pre-filling time. This indicator reflects the initial response capability of the large language model. TBT measures the generation interval of consecutive output Tokens of the task, which is mainly determined by the decoding time and affects the overall fluency. Due to the significant increase in inter-cloud large model inference tasks and the difficulty in balancing TTFT and TBT, if the inter-cloud large model inference tasks are not scheduled, the TTFT and TBT indicators will be too large, increasing the initial response capability of the large model and the overall fluency, which seriously affects the user experience.

[0007] In order to reduce TTFT and TBT, there are many task scheduling algorithms in the cloud environment. Traditional large model inference task scheduling algorithms include Orca and vLLM based on pre-filling priority and FasterTransformer based on decoding priority. These three scheduling methods for large model inference tasks cannot balance TTFT and TBT. The main reason is that the pre-filling of these three methods requires exclusive use of GPU computing resources and parallel processing units, resulting in mutual exclusive execution of pre-filling and decoding stages, making it difficult to optimize the balance.

[0008] Orca and vLLM based on pre-filling priority include the following steps:

[0009] Firstly, the large model receives N concurrent tasks (N is a positive integer) within a fixed time period Time1, and performs pre-filling on these N tasks in parallel.

[0010] Second, decode the N tasks after pre-filling. If the large model receives new N' tasks within the next fixed time period Time2 while decoding, it will pause the decoding of the previous N tasks and perform pre-filling on the new N' tasks. After the pre-filling of the new N' tasks is completed, the decoding of the previous N tasks will continue. Although this reduces TTFT, prioritizing pre-filling can lead to an increase in TBT because it interferes with ongoing decoding. The length of Time2 is equal to the length of Time1.

[0011] FasterTransformer based on decoding priority will prioritize decoding during task scheduling. The method is:

[0012] First, pre-fill N tasks received within a fixed time period Time1.

[0013] Second, decode the N tasks after pre-filling. If the large model receives new N' tasks within the next fixed time period Time2 while decoding, it will pause the decoding of the previous N tasks and perform pre-filling on the new N' tasks. After the pre-filling of the new N' tasks is completed, the decoding of the previous N tasks will continue. Although this reduces TTFT, prioritizing pre-filling can lead to an increase in TBT because it interferes with ongoing decoding. The length of Time2 is equal to the length of Time1.

[0014] FasterTransformer can maintain stable TBT, but the idle GPU computing resources caused by the early completion of some decoding tasks during decoding can lead to an increase in TTFT due to the accumulation of pre-filling tasks. Because FasterTransformer does not switch to pre-filling new tasks when a large number of tasks are decoded early.

[0015] To address the shortcomings of the above pre-filling priority and decoding priority methods, a scheduling method called Sarathi-Serve has been proposed. This method divides tasks into multiple pre-filling blocks, i.e., smaller fixed Token subsequences, based on Token quantity during pre-filling of inter-cloud large model inference tasks. For example, if a task has 1280 Tokens and the block size (S) is 128 Tokens, the task is divided into K = 1280 / 128 = 10 subsequences. The steps of the Sarathi-Serve method are:

[0016] First, when the large language model performs pre-filling on N tasks within a fixed time period Time1, it pre-fills the Token sequences of the N tasks into K1, K2, K n …, K NA smaller fixed Token sub-sequence, pre-fill (i.e. calculate the hidden state of the task and cache the key value) is performed in parallel for N sub-sequences, some shorter Token sequences in N tasks will be completed first, and longer ones will be completed later.

[0017] In the second step, the large model decodes the tasks with shorter Token sequences that have completed all sub-sequence pre-filling, while the Token sub-sequences of tasks with long Token sequences that have not completed pre-filling continue to be pre-filled in parallel. Because these smaller fixed Token sub-sequences have relatively low occupancy of GPU computing resources and do not require exclusive use of the GPU, pre-filling of long Token sequence tasks and decoding of short Token sequence tasks can be performed in parallel, reducing inter-stage resource competition. This method effectively reduces TBT.

[0018] For example, N = 2, the Token sequence length of task A is 1280, the Token sequence length of task B is 2560, the block size S is 128 Token, the number of Token sub-sequences of the first task K1 = 10, and the number of Token sub-sequences of the second task K2 = 20. When Sarathi-Serve is executed, the first 10 Token sub-sequences of A and B will perform pre-filling in parallel, and the pre-filling of A will be completed first. Then, the decoding of A and the pre-filling of the last 10 sub-sequences of B are performed in parallel.

[0019] However, Sarathi-Serve mainly relies on artificial experience to set a static pre-filling block size, and cannot adaptively and dynamically adjust the pre-filling block size according to the resource state and task queue load state during task scheduling, resulting in an imbalance between TTFT-TBT, and there is still room for improvement in the total running time. And this fixed small size block will cause arithmetic intensity nonlinear decay and computing unit "hunger" phenomenon, causing waste of computing resources fragmentation.

[0020] Therefore, how to overcome the shortcomings of the existing scheduling method, how to combine the real-time and high-concurrency real-time state of the inter-cloud task scheduling scenario, and optimize the task scheduling method to further improve the balance between TTFT and TBT indicators and reduce the total running time are technical problems that technicians in the field are extremely concerned about. SUMMARY

[0021] The technical problem to be solved by the present application is that the inter-cloud task scheduling method TBT and TTFT based on the best block pre-filling method currently available cannot effectively balance the total running time, which still needs to be reduced. A reinforcement learning-based block inter-cloud large model inference task scheduling method is provided, which dynamically adjusts the pre-filling block size according to the resource and task state, realizes task scheduling, balances the TTFT and TBT indicators, and reduces the total running time.

[0022] The present application comprises the following steps:

[0023] The first step is to construct a reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system.

[0024] The reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system is composed of a task queue management module, a real-time state acquisition module, a dynamic adaptive pre-filled sub-block module, and a scheduling module.

[0025] The task queue management module is connected to the real-time state acquisition module, the scheduling module, and each client in the inter-cloud environment. The task queue management module receives inter-cloud tasks from each client in the inter-cloud environment, assigns a unique task ID to each inter-cloud task, initializes the task state to a waiting state, and places the task in the waiting queue (the waiting queue length is N, N≥500). At the same time, W tasks (W depends on the computing performance of the GPU, determined by the Tensor Core) in the waiting queue are submitted to the scheduling module and the real-time state acquisition module.

[0026] The real-time state acquisition module is connected to the task queue management module and the dynamic adaptive pre-filled sub-block module. It monitors task characteristics and resource utilization state information during inter-cloud task scheduling and passes them to the dynamic adaptive pre-filled sub-block module. The task characteristics include the number of running tasks, the number of waiting tasks, the number of completed tasks, the state encoding vector of each task (the encoding of a waiting task is 0, the encoding of a running task is 1, and the encoding of a completed task is 2), the TTFT value vector of each task, the TBT average value vector, the TBT variance vector, and the TBT standard deviation vector. The resource utilization state includes GPU computing utilization and GPU memory utilization.

[0027] The dynamic adaptive pre-filling sub-block module is connected with the real-time state acquisition module and the scheduling module, and has a reinforcement learning agent. The reinforcement learning agent receives task characteristics and resource utilization state information from the real-time state acquisition module, gives an optimal pre-filling sub-block size according to the task characteristics and the resource utilization state information, and sends the optimal pre-filling sub-block size to the scheduling module. The reinforcement learning agent is a deep reinforcement learning model composed of a policy neural network Actor and a value neural network Critic (the structure of the policy neural network Actor and the value neural network Critic can refer to the literature “Proximal policy optimization algorithms”, published on the arXiv preprint paper website, the website is https: / / arxiv.org / abs / 1707.06347, and the publication time is June 2017). The Actor is connected with the real-time state acquisition module, the scheduling module and the Critic. The Actor receives task characteristics and resource utilization state from the real-time state acquisition module, sets a Softmax output discrete action set in the Actor, selects an optimal sub-block size as a pre-filling sub-block size from the discrete action set according to the task characteristics and the resource utilization state, and obtains an optimal pre-filling sub-block size. The Actor transmits the optimal pre-filling sub-block size to the scheduling module, and simultaneously transmits a state-action pair composed of the optimal pre-filling sub-block size, the task characteristics and the resource utilization state to the Critic. The Critic is connected with the Actor, receives the state-action pair from the Actor, calculates the relative advantage of the state-action pair through an advantage function, obtains a long-term value estimate of the state-action pair, and updates the parameters of the Actor through the long-term value estimate of the state-action pair (the feedback update principle can refer to the literature

[0028] “Proximal policy optimization algorithms”, published on the arXiv preprint paper website, the website is https: / / arxiv.org / abs / 1707.06347, and the publication time is June 2017), and guides the Actor to update in the direction of higher return.

[0029] The scheduling module is a large language model (the large language model can select Qwen-7B or Yi-6B) connected with the task queue management module and the dynamic adaptive pre-padding sub-block module, and contains a running queue and a completed queue. The scheduling module receives W inter-cloud large model inference tasks from the waiting queue of the task queue management module, stores the W tasks in the running queue, receives the optimal pre-padding sub-block size from the dynamic adaptive pre-padding sub-block module, divides the Token sequences of the W tasks in the running queue that need to be pre-padded into multiple smaller Token subsequences according to the optimal pre-padding sub-block size, and performs pre-padding in batches. If there are tasks that have completed pre-padding among the W tasks, the large language model decodes those tasks that have completed pre-padding, while pre-padding the Token subsequences that have not completed pre-padding in parallel. During the pre-padding and decoding process, if a new optimal pre-padding sub-block size suitable for the current task characteristics and resource state is received from the dynamic adaptive pre-padding sub-block module, the scheduling module divides the Token sequences of the tasks according to the new optimal pre-padding sub-block size, and performs pre-padding in batches.

[0030] Secondly, the cloud inter-large model inference task scheduling system based on adaptive pre-padding sub-blocks of reinforcement learning is trained, mainly the reinforcement learning agent in the dynamic adaptive pre-padding sub-block module is trained, and the trained cloud inter-large model inference task scheduling system based on adaptive pre-padding sub-blocks of reinforcement learning is obtained, the method is:

[0031] 2.1 Initialize the training round number episode = 1.

[0032] 2.2 The task queue management module receives N inter-cloud tasks from the client in a fixed time period Time1 (usually 10-20s) under the inter-cloud environment, assigns a unique task ID to each inter-cloud task, initializes the task state to the waiting state, and places the task in the waiting queue.

[0033] 2.3 The task queue management module determines the computing power of the Tensor core of the GPU, if the computing power of the Tensor core can pre-pad W tasks at the same time, W tasks in the waiting queue are submitted to the running queue of the scheduling module, W ≤ N.

[0034] 2.4 Initialize the time step t = 1;

[0035] 2.5 The real-time state acquisition module monitors the task queue management module, the scheduling module and the GPU resource usage by using the state acquisition method, acquires the task characteristics and resource state information, and integrates the task characteristics and resource state information of the tth time step into the task characteristics and resource state information set s t , of the tth time step. tis sent to the dynamic adaptive pre-filling sub-module, the method is:

[0036] 2.5.1 Obtain the task characteristics of the t-th time step, the method is:

[0037] 2.5.1.1 Check the number of tasks in the waiting queue in the task queue management module to obtain the number of tasks waiting in the t-th time step At the same time, check the number of tasks in the running queue that enter the scheduling module to obtain the number of tasks running in the t-th time step At the same time, check the number of tasks in the completed task queue in the scheduling module to obtain the number of tasks completed in the t-th time step

[0038] 2.5.1.2 Construct the task state coding vector Encode the tasks in the waiting state as 0, the tasks in the running state as 1, and the completed tasks as 2, so includes 0, 1, 2, and the vector length is N.

[0039] 2.5.1.3 Record the time when the W tasks in the t-th time step start pre-filling execution, let ts1, …, ts w , …, ts W , 1≤w≤W, record the time when the W tasks in the t-th time step pre-filling execution is completed te1, …, te w , …, te W , calculate the TTFT value vector of the W tasks in the t-th time step The length of is W, and the w-th element of is the TTFT value TTFT tw of the w-th task in the t-th time step tw . w . w .

[0040] 2.5.1.4 Record the time interval of the W tasks in the t-th time step generating Token by word, assuming that the w-th task generates N w Token in the t-th time step, and its time interval sequence is denoted as δw1, …, δw i , …, δw Nw , where δw i represents the time interval required for the w-th task to generate the i-th Token, 1≤i≤N w .

[0041] 2.5.1.5 Calculate the TBT average value vector of the W tasks in the t-th time step length W, whose w-th element is the average value of the TBT of the w-th task in the t-th time step TBTavg tw ,

[0042]

[0043] 2.5.1.6 Calculate the TBT variance vector of W tasks in the t-th time step length W, whose w-th element is the TBT variance of the w-th task in the t-th time step TBTVar tw ,

[0044]

[0045] 2.5.1.7 Calculate the TBT standard deviation vector of W tasks in the t-th time step length W, whose w-th element is the TBT standard deviation of the w-th task in the t-th time step TBTStd tw ,

[0046]

[0047] 2.5.2 Use the read instruction of the operating system (such as the instruction "nvidia-smi

[0048] --query-gpu = utilization.gpu, memory.used, memory.total --format = csv-l1" of Linux) to obtain the usage of GPU resources in the t-th time step of the computer running the cloud inter large model inference task scheduling system based on reinforcement learning adaptive pre-padding sub-blocking, including the GPU computing utilization rate in the t-th time step and the GPU memory usage in the t-th time step

[0049] 2.5.3 The real-time state acquisition module integrates the task characteristics and resource state information in the t-th time step into the task characteristics and resource state information set s t ,

[0050] in the t-th time step and sends s t to the dynamic adaptive pre-padding sub-blocking module.

[0051] 2.6 The reinforcement learning agent of the dynamic adaptive pre-padding sub-blocking module receives s t from the real-time state acquisition module, uses s t for training, and the method is:

[0052] 2.6.1 The reinforcement learning agent receives s from the real-time state acquisition module t According to Actor t Give the pre-filled block size a at time step t. t , prefill block size a t Send to the scheduling module.

[0053] 2.6.2 The scheduling module sorts the token sequence of the W tasks in the current running queue according to a t The tokens are divided into multiple smaller token subsequences. If W1 out of W tasks have completed pre-filling, then while pre-filling the W-W1 tasks that have not completed pre-filling, decoding is performed on the W1 tasks that have completed pre-filling. After execution, the scheduling module calculates the reward r at time step t using a hierarchical normalized bi-objective reward function. t Meanwhile, the task queue management module monitors the GPU's Tensor Cores. If a Tensor Core is idle (when W1 tasks have completed decoding, a Tensor Core will be idle), the task queue management module adds tasks to the scheduling module to bring the total to W. The method is:

[0054] 2.6.2.1 From the TTFT value vector at time step t Read the TTFT value of task with task ID i from the run queue. ti Calculate the time increment Δτ for task ID i. i The method is:

[0055] If t>1, from the TTFT value vector at time step (t-1) Read the TTFT value of task with task ID i in the run queue at time step t-1. (t-1)i ,Δτ i =TTFT ti -TTFT (t-1)i Go to 2.6.2.2; If t = 1, let Δτ i =TTFT ti , see section 2.6.2.2.

[0056] 2.6.2.2 From the TBT value vector at time step t Read the average TBT value (TBTavg) of task ID i obtained at time step t. ti Calculate the mean TBT increment Δβ for task ID i in the run queue. avg，i The method is:

[0057] If t>1, the TBT value vector at time step (t-1) Read the average TBT (Task Bit) value (TBTavg) of task ID i in the run queue at time step t-1. (t-1)i Δβ avg,i =TBTavg ti -TBTavg (t-1)i Go to 2.6.2.3; If t = 1, let Δβ avg,i =TBTavg ti , see section 2.6.2.3.

[0058] 2.6.2.3 For Δτ i and Δβ avg,i Normalize Δτ i and Δβ avg,i Mapping to [-1,1] respectively, we obtain Δτ i normalized index Δτ norm and Δβ avg,i Normalized index Δβ avg,norm ,in

[0059] τ max ,τ min Represent The maximum and minimum values ​​of TTFT, β avg,max ,β avg,min Represent The maximum and minimum values ​​of the mean TBT in the middle.

[0060] 2.6.2.4 TBT variance vector of W tasks at time step t Obtain the TBT variance TBTVar of task ID i at time step t. ti The standard deviation vector of TBT for W tasks at time step t. Obtain the standard deviation of TBT for task ID i at time step t: TBTSTd ti , for TBTVar ti ,TBTStd ti Normalize them separately to obtain TBTVar ti normalized index β var，norm and TBTSTd ti normalized index β std，norm ,in β var,max represent The maximum value in, β std,max represent The maximum value in, β var,norm and β std,norm It can suppress delayed fluctuations.

[0061] 2.6.2.5 Constructing the comprehensive reward function r at time step t t :

[0062] r t =vΔτ norm +ηΔβ avg,nom +tβ var,norm +δβ std,norm . Formula (1)

[0063] Where v, η, ι, and δ are reward weight parameters, with values ​​of 0.5, 0.5, 0.5, and 0.5, respectively.

[0064] r t It will optimize based on changes in task characteristics and resource status information. Because the value is decremented by 1 during normalization and mapped to the [-1,0] interval, r t It is a negative value. When TTFT and TBT increase, Δτ norm Δβ var，norm ,β var，norm ,β std,norm It will become smaller, therefore r t The decrease in TTFT and TBT indicates a significant negative impact, which incentivizes the reinforcement learning agent to reduce the increase in TTFT and TBT, thereby achieving the optimization goal of reducing both TTFT and TBT simultaneously.

[0065] 2.6.3 The scheduling module, based on a t After pre-filling, the task characteristics and resource status information will change. The real-time status acquisition module uses the status acquisition method described in step 2.5 to monitor the task queue management module, scheduling module, and GPU resource usage, and acquire task characteristics and resource status information. The task characteristics and resource status information at time step t+1 (because the time has changed, it is time step t+1) are integrated into a set s of task characteristics and resource status information at time step t+1. t+1 , will s t+1 The task is sent to the dynamic adaptive pre-filling block module. Simultaneously, the task queue management module monitors the GPU's Tensor cores; if any Tensor cores are idle, the task queue management module adds tasks to the scheduling module, bringing the total to W.

[0066] 2.6.4 Dynamically adaptive pre-filled block module for constructing training set.

[0067] The dynamic adaptive pre-filling block module uses a quadruple (s) consisting of the state-action-reward at time step t-1 and the state at time step (t+1). t ,a t ,r t ,s t+1 Store in the playback buffer The size is generally set to store 10,000-100,000 quadruples. If the reasoning of the N inter-cloud tasks is completed, go to 2.6.5; if the reasoning of the N inter-cloud tasks is not completed, let t = t + 1, and go to 2.6.1.

[0068] 2.6.5 adopts to train the reinforcement learning agent for MaxKepoch rounds, and updates the parameters of the reinforcement learning agent, the method being:

[0069] 2.6.5.1 initializes the update number Kepoch = 1.

[0070] 2.6.5.2 inputs the K quadruples randomly extracted from to the policy neural network Actor and the value neural network Critic, K being the batch size, and the recommended value being 3-15.

[0071] 2.6.5.3 updates the policy network parameters θ of the Actor, the method being:

[0072] 2.6.5.3.1 the Critic adopts the generalized advantage estimation (GAE) technique (the principle can be referred to in the document “High-dimensional continuous control using generalized advantage estimation.”, translated as “High-dimensional continuous control using generalized advantage estimation”, published on the arXiv preprint website, the website being https: / / arxiv.org / abs / 1506.02438 (2015), the publication date being June 2015), and updates θ to a higher action direction through the advantage function A π (s t , a t ).

[0073] The advantage function is:

[0074]

[0075] Where λ is a weighting coefficient, γ is a discount factor, λ ∈ [0, 1], and γ ∈ [0, 1];

[0076] δ t+k = r t + γV π (s t+1 ) - V π (s t ), r t is the reward function value at the tth time step, and V π (st ) is a state-value estimation function that evaluates the value estimate of a state-action pair, denotes the expectation over its internal items under a certain probability distribution, T represents the last time instant at which a quadruple is drawn from , k is a positive integer from 0 to T-t.

[0077] 2.6.5.3.2 Update the policy network parameters θ by the loss function shown in equation (3):

[0078]

[0079] where ratio is the action probability ratio of the new and old policy networks, where θ' is the new policy network parameter, θ is the old policy network parameter, and ratio is limited in the interval [1-∈, 1+∈] by the double clipping operation clip function, where ∈ is a hyperparameter that controls the size of the interval, ∈ ∈ [0, 1].

[0080] 2.6.5.3.3 Update the policy network parameters θ by taking the gradient of J Actor (θ) with respect to θ. Update the policy network parameters θ to where α is a preset learning rate, generally taking a value of 0.0001-0.0005.

[0081] 2.6.5.4 Update the value network parameters φ of Critic.

[0082] 2.6.5.4.1 Update the value network parameters φ by the loss function shown in equation (4):

[0083]

[0084] where, is the multi-step discounted return calculated by GAE, V φ (s t ) is a state-value estimation function that evaluates the value estimate of a state-action pair, denotes the expectation over its internal items under a certain probability distribution, T represents the last time instant at which a quadruple is drawn from , k is a positive integer from 0 to T-t,

[0085]

[0086] 2.6.5.4.2 Update the value network parameters φ by taking the gradient of J Critic (φ) with respect to φ. Update the value network parameters φ to

[0087] 2.6.5.5 Set the update times Kepoch = Kepoch + 1.

[0088] 2.6.5.6 If Kepoch > MaxKepoch, the maximum iteration number MaxKepoch is a positive integer, and the recommended value range is 10-20, it indicates that the reinforcement learning agent completes a round of parameter update, and go to 2.7; if Kepoch ≤ MaxKepoch, go to 2.6.6.2 to continue the training in this round.

[0089] 2.7 If episode > the maximum number of rounds Max_episode (Max_episode is a positive integer, and the recommended value range is 150-200, and the deep learning model can achieve good convergence), end the training, obtain the trained reinforcement learning agent, and obtain the trained self-adaptive pre-filled block-based inter-cloud large model inference task scheduling system based on reinforcement learning, and go to the third step; if episode ≤ the maximum number of rounds Max_episode, go to 2.2 to continue the next round of training.

[0090] Third step, the trained self-adaptive pre-filled block-based inter-cloud large model inference task scheduling system based on reinforcement learning schedules the inter-cloud tasks.

[0091] 3.1 The task queue management module receives N inter-cloud tasks from the client in a fixed time period Time in the inter-cloud environment, allocates a unique task ID to each inter-cloud task, initializes the task state to a waiting state, and places the task in the waiting queue.

[0092] 3.2 The task queue management module submits W tasks in the waiting queue to the running queue of the scheduling module according to the computing power of the Tensor core of the GPU, and W ≤ N.

[0093] 3.3 Initialize the time step tp = 1.

[0094] 3.4 The real-time state acquisition module monitors the task queue management module, the scheduling module, and the GPU resource usage using the state acquisition method described in step 2.5, acquires the task features and resource state information, and integrates the task features and resource state information in the tpth time step into the task feature and resource state information set s tp tp s

[0095] is the number of tasks running in the tpth time step; is the number of tasks in the waiting queue in the task queue management module in the tpth time step;​ is the number of tasks completed by inference at the tpth time step; is the GPU computing utilization at the tpth time step, is the GPU memory usage at the tpth time step. is the task state vector at the tpth time step; is the TTFT value vector of the W tasks at the tpth time step; is the TBT average value vector of the W tasks at the tpth time step; is the TBT variance vector of the W tasks at the tpth time step; is the TBT standard deviation vector of the W tasks at the tpth time step.

[0096] 3.5 The reinforcement learning agent of the dynamic adaptive pre-padding sub-blocking module receives s tp from the real-time state acquisition module, and gives the optimal pre-padding sub-blocking size by using s tp . The trained reinforcement learning agent receives s tp from the real-time state acquisition module, and the Actor gives the optimal pre-padding sub-blocking size a tp at the tpth time step according to s tp . The a tp is sent to the scheduling module.

[0097] 3.6 The scheduling module divides the Token sequence of the W tasks in the current running queue into multiple smaller Token subsequences according to a tp . If W1 tasks among the W tasks have completed pre-padding, the decoding of the W1 tasks is performed while the pre-padding of the W-W1 tasks is performed. After the pre-padding is performed according to a tp , the task feature and resource state information will change. It is judged whether all the N tasks in the running queue are completed (i.e., whether the running queue and the waiting queue are empty). If both the running queue and the waiting queue are empty, go to 3.7. If the waiting queue is not empty (at this time, the running queue is certainly not empty), the task queue management module monitors the Tensor core of the GPU. If there is an idle Tensor core, the task queue management module supplements tasks to W to the scheduling module, and let tp = tp + 1, and return to 3.4. If the waiting queue is empty while the running queue is not empty, it means that the tasks have been completed or are still in the running queue. At this time, it is not necessary to supplement tasks to the scheduling module, let tp = tp + 1, and return to 3.4.

[0098] 3.7 The large language model completes the scheduling and inference of the N tasks, and ends.

[0099] The following beneficial effects can be achieved by adopting the present application:

[0100] 1. The first step of the present application provides a reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system; the second step trains the reinforcement learning agent, so that the trained reinforcement learning agent can dynamically divide the pre-filled sub-block according to the task state and resource state of the system; the third step uses the trained reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system, which can realize adaptive adjustment of the size of the pre-filled sub-block according to the task state and resource state of the system, so as to optimize the TTFT-TBT index and reduce the total running time.

[0101] 2. The present application realizes the division of the pre-filled sub-block into a block without exclusive GPU resources, so that while the scheduling module in step 3.6 performs pre-filling on some tasks, it can also decode those tasks that have completed pre-filling, realizing efficient utilization of computing resources.

[0102] 3. The present application dynamically adjusts the size of the pre-filled sub-block during inference of the large model, which can automatically increase the size of the pre-filled block when the computing resources are idle to improve the computing density, avoid the loss caused by the fixed small size (such as block size of 64 blocks, 128 blocks or below) in the scheduling method of the background technology, and avoid the arithmetic intensity nonlinear decay and the "starvation" phenomenon of the computing unit caused by the small size block, resulting in fragmentation of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0103] Figure 1 is the overall flowchart of the present application;

[0104] Figure 2 is the logic structure diagram of the reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system constructed in the first step of the present application;

[0105] Figure 3 is the TTFT, TBT index comparison chart when the present application and the mainstream large model inference task scheduling method are scheduled;

[0106] Figure 4 is the total running time comparison chart when the present application and the mainstream large model inference task scheduling method are scheduled. DETAILED DESCRIPTION

[0107] The specific embodiments of the present application will be described below with reference to the accompanying drawings.

[0108] As shown in Figure 1 , the present application comprises the following steps:

[0109] First, construct a reinforcement learning-based adaptive pre-filled sub-block inter-cloud large model inference task scheduling system.

[0110] The adaptive pre-filling sub-blocking inter-cloud large model inference task scheduling system based on reinforcement learning is composed of a task queue management module, a real-time state acquisition module, a dynamic adaptive pre-filling sub-blocking module, and a scheduling module. The logical structure diagram of the adaptive pre-filling sub-blocking inter-cloud large model inference task scheduling system based on reinforcement learning is shown in FIG. 1. Figure 2

[0111] The task queue management module is connected with the real-time state acquisition module, the scheduling module, and each client in the inter-cloud environment. The task queue management module receives inter-cloud tasks from each client in the inter-cloud environment, allocates a unique task ID for each inter-cloud task, initializes the task state as a waiting state, and places the task in the waiting queue (the length of the waiting queue is N, N≥500). At the same time, the task queue management module submits W tasks (W depends on the computing performance of the GPU, which is determined by the Tensor Core) in the waiting queue to the scheduling module and the real-time state acquisition module.

[0112] The real-time state acquisition module is connected with the task queue management module and the dynamic adaptive pre-filling sub-blocking module, and monitors the task characteristics and resource utilization state information during the inter-cloud task scheduling process. The task characteristics include the number of running tasks, the number of waiting tasks, the number of completed tasks, the state code vector of each task (the code of a waiting task is 0, the code of a running task is 1, and the code of a completed task is 2), the TTFT value vector of each task, the TBT average value vector, the TBT variance vector, and the TBT standard deviation vector. The resource utilization state includes the GPU computing utilization rate and the GPU memory utilization rate.

[0113] ​The dynamic adaptive pre-filling sub-block module is connected with the real-time state acquisition module and the scheduling module, and has a reinforcement learning agent. The reinforcement learning agent receives task characteristics and resource utilization state information from the real-time state acquisition module, gives an optimal pre-filling sub-block size according to the task characteristics and the resource utilization state information, and sends the optimal pre-filling sub-block size to the scheduling module. The reinforcement learning agent is a deep reinforcement learning model composed of a policy neural network Actor and a value neural network Critic (the structure of the policy neural network Actor and the value neural network Critic is referred to in the reference Proximal policy optimization algorithms, published on the arXiv preprint website with the address https: / / arxiv.org / abs / 1707.06347 and the publication time of June 2017). The Actor is connected with the real-time state acquisition module, the scheduling module and the Critic. The Actor receives task characteristics and resource utilization state from the real-time state acquisition module, sets a Softmax output discrete action set in the Actor, selects an optimal sub-block size as a pre-filling sub-block size from the discrete action set according to the task characteristics and the resource utilization state, and obtains the optimal pre-filling sub-block size. The Actor transmits the optimal pre-filling sub-block size to the scheduling module, and simultaneously transmits a state-action pair composed of the optimal pre-filling sub-block size, the task characteristics and the resource utilization state to the Critic. The Critic is connected with the Actor, receives the state-action pair from the Actor, calculates the relative advantage of the state-action pair through an advantage function, obtains a long-term value estimate of the state-action pair, and updates the parameters of the Actor through the long-term value estimate of the state-action pair (the feedback update principle can be referred to in the reference Proximal policy optimization algorithms, published on the arXiv preprint website with the address https: / / arxiv.org / abs / 1707.06347 and the publication time of June 2017), and guides the Actor to update in the direction of a higher return.

[0114] Proximal policy optimization algorithms, published on the arXiv preprint website with the address https: / / arxiv.org / abs / 1707.06347 and the publication time of June 2017).

[0115] The scheduling module is a large language model (the large language model can select Qwen-7B or Yi-6B) connected with the task queue management module and the dynamic adaptive pre-padding sub-block module, and contains a running queue and a completed queue. The scheduling module receives W inter-cloud large model inference tasks from the waiting queue of the task queue management module, stores the W tasks in the running queue, receives the optimal pre-padding sub-block size from the dynamic adaptive pre-padding sub-block module, divides the Token sequences of the W tasks in the running queue that need to be pre-padded into multiple smaller Token subsequences according to the optimal pre-padding sub-block size, and performs pre-padding in batches. If there are tasks that have completed pre-padding among the W tasks, the large language model decodes those tasks that have completed pre-padding, while pre-padding the Token subsequences that have not completed pre-padding in parallel. During the pre-padding and decoding process, if a new optimal pre-padding sub-block size suitable for the current task characteristics and resource state is received from the dynamic adaptive pre-padding sub-block module, the scheduling module divides the Token sequences of the tasks according to the new optimal pre-padding sub-block size, and performs pre-padding in batches.

[0116] Secondly, the cloud inter-large model inference task scheduling system based on adaptive pre-padding sub-blocks of reinforcement learning is trained, mainly the reinforcement learning agent in the dynamic adaptive pre-padding sub-block module is trained, and the trained cloud inter-large model inference task scheduling system based on adaptive pre-padding sub-blocks of reinforcement learning is obtained, the method is:

[0117] 2.1 Initialize the training round number episode = 1.

[0118] 2.2 The task queue management module receives N inter-cloud tasks from the client in a fixed time period Time1 (usually 10-20s) under the inter-cloud environment, assigns a unique task ID to each inter-cloud task, initializes the task state to the waiting state, and places the task in the waiting queue.

[0119] 2.3 The task queue management module determines the computing power of the Tensor core of the GPU, if the computing power of the Tensor core can pre-pad W tasks at the same time, W tasks in the waiting queue are submitted to the running queue of the scheduling module, W ≤ N.

[0120] 2.4 Initialize the time step t = 1;

[0121] 2.5 The real-time state acquisition module monitors the task queue management module, the scheduling module and the GPU resource usage by using the state acquisition method, acquires the task characteristics and resource state information, and integrates the task characteristics and resource state information of the tth time step into the task characteristics and resource state information set s t , of the tth time step. tto the dynamic adaptive pre-filling sub-module.

[0122] 2.5.1 Obtain the task feature of the t-th time step by:

[0123] 2.5.1.1 Check the number of tasks in the waiting queue in the task queue management module to obtain the number of tasks currently waiting Check the number of tasks currently running in the running queue entering the scheduling module at the same time to obtain the number of tasks currently running Check the number of tasks in the completed task queue in the scheduling module at the same time to obtain the number of tasks currently completed inference

[0124] 2.5.1.2 Construct the task state encoding vector Encode the tasks in the waiting state as 0, the tasks in the running state as 1, and the completed tasks as 2, so includes 0, 1, 2, and the vector length is N.

[0125] 2.5.1.3 Record the time when the W tasks in the t-th time step start pre-filling execution, let ts1, …, ts w , …, ts W , 1≤w≤W, record the time when the W tasks in the t-th time step pre-filling execution is completed te1, …, te w , …, te W , calculate the TTFT value vector of the W tasks in the t-th time step The length of the vector is W, and the w-th element of the vector is the TTFT value of the w-th task in the t-th time step TTFT tw , TTFT tw = te w -ts w .

[0126] 2.5.1.4 Record the time interval of the W tasks in the t-th time step generating Token by word, assuming that the w-th task generates N w Token in the t-th time step, and its time interval sequence is denoted as δw1, …, δw i , …, δw Nw , where δw i represents the time interval required for the w-th task to generate the i-th Token, 1≤i≤N w .

[0127] 2.5.1.5 Calculate the TBT average value vector of the W tasks in the t-th time step length W, whose wth element is the average value of TBT of the wth task in the tth time step TBTavg tw ,

[0128]

[0129] 2.5.1.6 Calculate the TBT variance vector of W tasks in the tth time step length W, whose wth element is the TBT variance of the wth task in the tth time step TBTVar tw ,

[0130]

[0131] 2.5.1.7 Calculate the TBT standard deviation vector of W tasks in the tth time step length W, whose wth element is the TBT standard deviation of the wth task in the tth time step TBTStd tw ,

[0132]

[0133] 2.5.2 Obtain the usage of GPU resources in the tth time step of the computer running the adaptive pre-padding block-based inter-cloud large model inference task scheduling system using the read instruction of the operating system, including the GPU computing utilization rate in the tth time step and the GPU memory usage in the tth time step

[0134] 2.5.3 The real-time state acquisition module integrates the task characteristics and resource state information in the tth time step into the task characteristics and resource state information set s in the tth time step t ,

[0135] and sends s t to the dynamic adaptive pre-padding block module.

[0136] 2.6 The reinforcement learning agent of the dynamic adaptive pre-padding block module receives s t from the real-time state acquisition module, and uses s t for training, the method being:

[0137] 2.6.1 The reinforcement learning agent receives s t from the real-time state acquisition module, and the Actor gives the pre-padding block size a t in the tth time step according to s t , and sends the pre-padding block size a t to the scheduling module.

[0138] 2.6.2 The scheduling module reads the TTFT value of the task with task ID i in the running queue from the TTFT value vector at the t-th time step t into W1 smaller Token sub-sequences, if W1 tasks among the W tasks have finished pre-filling, the scheduling module performs decoding on the W1 tasks while pre-filling on the W-W1 tasks that have not finished pre-filling. After the execution, the scheduling module calculates the reward r t Meanwhile, the task queue management module monitors the Tensorcore of the GPU, if the Tensorcore is idle (at this time, W1 tasks have finished decoding, and the Tensorcore is idle), the task queue management module supplements tasks to W to the scheduling module. The method is:

[0139] 2.6.2.1 Read the TTFT value of the task with task ID i in the running queue from the TTFT value vector at the t-th time step TTFT ti , calculate the timing increment Δτ i of the task with task ID i, the method is:

[0140] If t > 1, read the TTFT value of the task with task ID i in the running queue at the t-1-th time step from the TTFT value vector at the t-1-th time step TTFT (t-1)i , Δτ i = TTFT ti -TTFT (t-1)i , go to 2.6.2.2; if t = 1, let Δτ i = TTFT ti , go to 2.6.2.2.

[0141] 2.6.2.2 Read the TBT average value of the task with task ID i obtained at the t-th time step from the TBT value vector at the t-th time step TBTavg ti , calculate the TBT average increment Δβ avg,i of the task with task ID i in the running queue, the method is:

[0142] If t > 1, read the TBT average value of the task with task ID i in the running queue at the t-1-th time step from the TBT value vector at the t-1-th time step TBTavg (t-1)i , Δβ avg,i = TBTavg ti -TBTavg (t-1)i , go to 2.6.2.3; if t = 1, let Δβ avg，i = TBTavg ti, turn to 2.6.2.3.

[0143] 2.6.2.3. Normalization of Δτ i and Δβ avg，i , Δτ i and Δβ avg，i are mapped to [-1, 1] respectively, obtaining the normalized indicator Δτ i of Δτ norm and the normalized indicator Δβ avg，i of Δβ avg，norm , where

[0144] τ max , τ min represent the maximum and minimum of TTFT in respectively, β avg，max , β avg，min represent the maximum and minimum of TBT mean in respectively.

[0145] 2.6.2.4. TBT variance vector of W tasks in the t-th time step TBT variance TBTVar ti of task ID i in the t-th time step is obtained from the TBT standard deviation vector of W tasks in the t-th time step TBT standard deviation TBTStd ti of task ID i in the t-th time step is obtained from the TBT standard deviation vector of W tasks in the t-th time step , TBTStd ti , TBTStd ti are normalized respectively, obtaining the normalized indicator β ti of TBTVar var，norm and the normalized indicator β ti of TBTStd std，norm , where β var，max represents the maximum in , S std，max represents the maximum in , β var,norm and β std,norm can suppress delay fluctuations.

[0146] 2.6.2.5. Constructing the comprehensive reward function r t of the t-th time step:

[0147] r t = vΔτ norm + ηΔβ avg,norm + tβ var,norm + δβ std,norm . (Formula (1))

[0148] Wherein v, η, i, δ are reward weight parameters, respectively 0.5, 0.5, 0.5, 0.5.

[0149] r t Will be optimized according to the change of task characteristics and resource state information, because the value is reduced 1 operation mapping to [-1, 0] interval when normalizing, so r t Is negative. When TTFT and TBT become larger, Δτ norm , Δβ avg，norm , β var，norm , β std，norm Will become smaller, so r t Will become smaller, which represents a greater negative impact, so as to stimulate the reinforcement learning agent to reduce TTFT and TBT increase, so as to achieve the optimization goal of making TTFT and TBT indicators decrease at the same time.

[0150] 2.6.3 The scheduling module executes pre-filling according to a t , which will cause the change of task characteristics and resource state information, the real-time state acquisition module uses the state acquisition method described in step 2.5 to monitor the task queue management module, the scheduling module and the GPU resource usage, acquires the task characteristics and resource state information, integrates the task characteristics and resource state information of the t+1 time step (because the time has changed, it is the t+1 time step) into the task characteristics and resource state information set s t+1 of the t+1 time step, and sends s t+1 to the dynamic adaptive pre-filling block module. At the same time, the task queue management module monitors the Tensor core of the GPU, and if the Tensor core is idle, the task queue management module supplements the task to W to the scheduling module.

[0151] 2.6.4 The dynamic adaptive pre-filling block module constructs a training set.

[0152] The dynamic adaptive pre-filling block module stores the four-tuple composed of the state-action-reward-state of the t time step (s t , a t , r t , s t+1 ) into the replay buffer The size is generally set to store 10000-100000 four-tuples. If the inference of N inter-cloud tasks is completed, go to 2.6.5; if the inference of N inter-cloud tasks is not completed, let t=t+1, go to 2.6.1.

[0153] 2.6.5 Use The four tuples in the above formula are used to train the reinforcement learning agent for MaxKepoch rounds, and the parameters of the reinforcement learning agent are updated by the following method:

[0154] 2.6.5.1 Initialize the update number Kepoch=1.

[0155] 2.6.5.2 Input the K tuples randomly sampled from to the policy neural network Actor and the value neural network Critic, where K is the batch size, and the recommended value is 3-15.

[0156] 2.6.5.3 Update the policy network parameters θ of the Actor, by the following method:

[0157] 2.6.5.3.1 The Critic uses the generalized advantage estimation (GAE) technique (the principle can be referred to the literature “High-dimensional continuous control using generalized advantage estimation.”, translated as “High-dimensional continuous control using generalized advantage estimation”, published on the arXiv preprint website, URL: https: / / arxiv.org / abs / 1506.02438 (2015), published in June 2015), and the advantage function A π (s t , a t ) is used to guide the update of θ to the direction of actions with higher advantages.

[0158] The advantage function is:

[0159]

[0160] Where λ is the weighting coefficient, γ is the discount factor, λ∈[0,1], γ∈[0,1];

[0161] δ t+k =r t +γV π (s t+1 )-V π (s t ), r t is the reward function value at the t-th time step, V π (s t ) is the state value estimation function, which evaluates the value estimation of the state-action pair, represents the expectation of its internal items on a certain probability distribution, T represents the last time when the four tuples are sampled from , and k is a positive integer from 0 to T-t.

[0162] 2.6.5.3.2 Update the policy network parameter θ by the loss function shown in equation (3):

[0163]

[0164] where ratio is the action probability ratio of the new and old policy networks, where θ' is the new policy network parameter, θ is the old policy network parameter, and ratio is limited in the interval [1-∈, 1+∈] by the double clipping operation clip function, where ∈ is a hyperparameter controlling the size of the interval, ∈ ∈ [0, 1].

[0165] 2.6.5.3.3 Update the critic's value network parameter φ by taking the partial derivative of J Actor (φ) with respect to φ to obtain the gradient Update the policy network parameter φ to where α is a preset learning rate, generally taking a value of 0.0001-0.0005.

[0166] 2.6.5.4 Update the critic's value network parameter φ.

[0167] 2.6.5.4.1 Update the value network parameter φ by the loss function shown in equation (4):

[0168]

[0169] where, is the multi-step discounted return calculated by GAE, V φ (s t ) is the state value estimation function, which evaluates the value estimate of the state-action pair, represents the expectation of its internal items over a certain probability distribution, T represents the last time of extracting the four-tuple from , and k is a positive integer from 0 to T-t.

[0170]

[0171] 2.6.5.4.2 Update the policy network parameter φ by taking the partial derivative of J Critic (φ) with respect to φ to obtain the gradient Update the policy network parameter φ to

[0172] 2.6.5.5 Let the update number Kepoch = Kepoch + 1.

[0173] 2.6.5.6 If Kepoch > MaxKepoch, the maximum number of iterations MaxKepoch is a positive integer, and the recommended value range is

[0174] 10~20, indicating that the reinforcement learning agent completes a round of parameter update, go to 2.7; if Kepoch≤MaxKepoch, go to 2.6.6.2 to continue the training of this round.

[0175] 2.7 If episode>Max_episode (Max_episode is a positive integer, and the recommended value range is 150~200, and the deep learning model can achieve good convergence), end the training, obtain the trained reinforcement learning agent, and obtain the trained self-adaptive pre-filled block-based inter-cloud large model inference task scheduling system based on reinforcement learning, and go to the third step; if episode≤Max_episode, go to 2.2 to continue the next round of training.

[0176] Third step, the trained self-adaptive pre-filled block-based inter-cloud large model inference task scheduling system based on reinforcement learning schedules the inter-cloud tasks.

[0177] 3.1 The task queue management module receives N inter-cloud tasks from the client in a fixed time period Time in the inter-cloud environment, allocates a unique task ID to each inter-cloud task, initializes the task state to a waiting state, and places the task in the waiting queue.

[0178] 3.2 The task queue management module submits W tasks in the waiting queue to the running queue of the scheduling module according to the computing capability of the Tensor core of the GPU, and W≤N.

[0179] 3.3 Initialize the time step tp=1.

[0180] 3.4 The real-time state acquisition module monitors the task queue management module, the scheduling module, and the GPU resource usage by using the state acquisition method described in step 2.5, acquires the task features and resource state information, and integrates the task features and resource state information of the tpth time step into the task feature and resource state information set stp. tp tp Send stpto the dynamic self-adaptive pre-filled block module.

[0181] is the number of tasks running in the tpth time step; is the number of tasks in the waiting queue in the task queue management module in the tpth time step; is the number of tasks that have completed inference in the tpth time step; is the GPU computing utilization rate in the tpth time step, is the GPU memory usage rate in the tpth time step. ​It is the task state vector at time step tp; It is the TTFT value vector of the W tasks at the tp-th time step; It is the average TBT vector of W tasks at the tp-th time step; It is the TBT variance vector of the W tasks at the tp-th time step; It is the standard deviation vector of the TBT of the W tasks at the tp-th time step.

[0182] 3.5 The reinforcement learning agent of the dynamic adaptive pre-filled block module receives s from the real-time state acquisition module tp , using s tp The optimal pre-padded block size is determined by: the trained reinforcement learning agent receiving s from the real-time state acquisition module. tp According to Actor tp Give the optimal pre-filled block size a at time step tp. tp , will a tp Send to the scheduling module.

[0183] 3.6 The scheduling module will sort the token sequence of the W tasks in the current running queue according to a tp The token is divided into multiple smaller token subsequences. If W1 out of W tasks have completed pre-filling, then while pre-filling the W-W1 tasks that have not completed pre-filling, decoding is performed on the W1 tasks that have completed pre-filling. The scheduling module determines the timing based on a. tp After pre-filling, the task characteristics and resource status information will change. The system checks whether all N tasks in the run queue are completed (i.e., whether the run queue and waiting queue are empty). If both the run queue and waiting queue are empty, proceed to step 3.7. If the waiting queue is not empty (the run queue is definitely not empty at this point), the task queue management module monitors the GPU's Tensor cores. If any Tensor cores are idle, the task queue management module adds tasks to the scheduling module up to W, setting tp = tp + 1, and returns to step 3.4. If the waiting queue is empty but the run queue is not empty, it means the tasks have either finished inference or are still in the run queue. In this case, there is no need to add tasks to the scheduling module; set tp = tp + 1, and return to step 3.4.

[0184] 3.7 The large language model completes the scheduling and inference of N tasks, and then ends.

[0185] The task scheduling methods for large model inference requests, in contrast to this invention, are the four methods described in the background art:

[0186] The first kind, FasterTransformer adopts a blocking scheduling strategy, which prioritizes decoding when batch processing requests, exclusively occupies computing resources to ensure low latency, but sacrifices throughput.

[0187] The second kind, Orca adopts a pre-filling stage priority strategy, introduces iterative level batch processing, where requests can dynamically enter or exit batch processing at the granularity of a single iteration.

[0188] The third kind, vLLM proposes a PagedAttention mechanism to dynamically manage KVCache through virtual memory paging technology, realizing the coupling optimization of iterative level batch processing and pre-filling scheduling.

[0189] The fourth kind, Sarathi-Serve proposes a block pre-filling technology, which subdivides the pre-filling calculation of a single request into multiple small GPU occupancy calculation blocks and executes them in batches, allowing the pre-filling and decoding stages to be executed in parallel.

[0190] Under the same hardware environment, five kinds of task scheduling methods are experimented, the hardware environment of the experiment is Linux system-ubuntu, and the Qwen-7B model is deployed based on NVIDIA RTX 4090 graphics card to carry out online inference request. The cloud environment is composed of multiple cloud service providers, W is 65, Time is 10s. K takes 10; ∈=0.2; α is set to 0.0003; MaxKepoch is 10, Max_episode takes 150. The read instruction of the operating system adopted in step 2.5.2 is the Linux instruction "nvidia-smi--query-gpu=utilization.gpu,memory.used,memory.total--format=csv-l1". The size is set to store 100000 four-tuples.

[0191] Figure 3 The TTFT and TBT index comparison chart of the scheduling module of the present application and the above-mentioned large model inference task scheduling method is used to schedule the Qwen-7B model in N=500 concurrent cloud inference tasks. As shown in Figure 3As shown in the figure, the horizontal coordinate is the five task scheduling methods, the left vertical coordinate is the TTFT (corresponding to the column chart), and the right horizontal coordinate is the TBT average (corresponding to the line chart). The TTFT of the present application is reduced to 28 seconds, while the vLLM and Sarathi-Serve are 47 seconds and 57 seconds respectively. Compared with vLLM and Sarathi-Serve, the TTFT of the present application is reduced by more than 50%. The TBT average (the average value of the TBT index of the task, which is convenient for comparison) of the present application is reduced to 0.0529 seconds, while the TBT average of vLLM and Sarathi-Serve is 0.0801 seconds and 0.1495 seconds respectively, and the optimization amplitude is more than 60%. The TBT average of the present application is compared with Orca and FasterTransformer. Although the latter two maintain an advantage in the TBT index (Orca and FasterTransformer are 0.0225 and 0.0220 respectively, which is reduced by 50% compared with the present application), the TTFT cost is increased by 16 times (Orca and FasterTransformer are 460 seconds and 465 seconds respectively), so overall, the present application is superior to Orca and FasterTransformer.

[0192] Figure 4 The total running time comparison chart of the present application and the above-mentioned inter-cloud large model reasoning task scheduling method using model Qwen-7B to schedule 500 concurrent inter-cloud reasoning tasks is shown in Figure 6. Figure 4 As shown in the figure, the horizontal coordinate is the five task scheduling methods, and the vertical coordinate is the total running time. The total running time of Orca and FasterTransformer to complete 500 request tasks is 550 and 555 seconds respectively, the running time of vLLM is 71 seconds, the running time of Sarathi is 83 seconds, and the total running time of the present application is reduced to 43 seconds. The total running time of the present application is significantly optimized compared with all the comparison methods.

[0193] If the scheduling module uses model Yi-6B, the experimental results are roughly the same as those of the Qwen-7B model under the same hardware environment and the same parameter settings.

[0194] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. A method for scheduling inter-cloud large model inference tasks based on reinforcement learning blocking, characterized in that Comprising the following steps: The first step is to build an adaptive pre-filling sub-block inter-cloud large model inference task scheduling system based on reinforcement learning; The adaptive pre-filling sub-block inter-cloud large model inference task scheduling system based on reinforcement learning is composed of a task queue management module, a real-time state acquisition module, a dynamic adaptive pre-filling sub-block module, and a scheduling module; The task queue management module initializes the task state to a waiting state and places the task in the waiting queue, while submitting W tasks in the waiting queue to the scheduling module and the real-time state acquisition module; The real-time state acquisition module monitors task characteristics and resource utilization state information during inter-cloud task scheduling and transmits the task characteristics and resource utilization state information to the dynamic adaptive pre-filling sub-block module; The dynamic adaptive pre-filling sub-block module contains a reinforcement learning agent that receives task characteristics and resource utilization state information from the real-time state acquisition module and provides the optimal pre-filling sub-block size based on the task characteristics and resource utilization state information, and sends the optimal pre-filling sub-block size to the scheduling module; The reinforcement learning agent is a deep reinforcement learning model composed of a policy neural network Actor and a value neural network Critic; The scheduling module is a large language model that contains a running queue and a completed queue; the scheduling module divides the W inter-cloud large model inference tasks received from the task queue management module into optimal pre-filling sub-blocks and performs pre-filling in batches; if there are tasks that have completed pre-filling among the W tasks, the large language model decodes the tasks that have completed pre-filling and simultaneously pre-fills the Token sub-sequences that have not completed pre-filling in parallel; during the pre-filling and decoding process, the scheduling module divides the Token sequence of the task according to the optimal pre-filling sub-block size received from the dynamic adaptive pre-filling sub-block module that is suitable for the current task characteristics and resource state, and performs pre-filling in batches; The second step is to train the reinforcement learning agent in the dynamic adaptive pre-filling sub-block module of the adaptive pre-filling sub-block inter-cloud large model inference task scheduling system based on reinforcement learning to obtain the trained adaptive pre-filling sub-block inter-cloud large model inference task scheduling system based on reinforcement learning, the method is: 2.1 Initialize the number of training rounds episode = 1; 2.2 The task queue management module receives N inter-cloud tasks from the client under the inter-cloud environment within a fixed time period Time1, assigns a unique task ID to each inter-cloud task, initializes the task state to a waiting state, and places the task in the waiting queue, where N is a positive integer; 2.3 The task queue management module determines the computing power of the GPU's Tensor core, and if the computing power of the Tensor core can simultaneously pre-fill W tasks, then W tasks in the waiting queue are submitted to the running queue of the scheduling module, where 1 ≤ W ≤ N; 2.4 Initialize the time step t = 1; 2.5 The real-time state acquisition module uses the state acquisition method to monitor the task queue management module, the scheduling module, and the GPU resource usage, obtains the task characteristics and resource state information, and integrates the task characteristics and resource state information of the tth time step into a task characteristic and resource state information set s t s t is sent to the dynamic adaptive pre-padding subblocking module by the following method: 2.5.1 Obtain the task features at the t-th time step, get the number of tasks waiting at the t-th time step The number of running tasks at the t-th time step The number of tasks that have completed reasoning at the t-th time step The task state encoding vector at the t-th time step Including 0, 1, 2; The TTFT value vector of W tasks in the t-th time step The w-th element of is the TTFT value of the w-th task in the t-th time step TTFT tw ; The TBT average value vector of W tasks in the t-th time step The w-th element of is the average value of the TBT of the w-th task in the t-th time step TBTavg tw ; The TBT variance vector of W tasks in the t-th time step The w-th element of is the TBT variance of the w-th task in the t-th time step TBTVar tw ; The TBT standard deviation vector of W tasks in the t-th time step The w-th element of is the TBT standard deviation of the w-th task in the t-th time step TBTStd tw ; 2.5.2 Obtain the usage of the GPU resources at the t-th time step in the computer running the cloud-based large model inference task scheduling system based on reinforcement learning adaptive pre-filling sub-blocking, including the GPU computing utilization rate at the t-th time step and the GPU memory usage at the t-th time step 2.5.3 The real-time state collection module integrates the task features and resource state information of the t-th time step into the task feature and resource state information set s of the t-th time step t , and s t to a dynamic adaptive pre-padding subchunk module; 2.6 The reinforcement learning agent of the dynamic adaptive pre-padding chunking module receives s from the real-time state harvesting module t is trained using s t by training 2.6.1 The reinforcement learning agent receives s from the real-time state acquisition module t , Actor gives the pre-padding chunk size a t for the t-th time step based on s t , sends the pre-padding chunk size a t to the scheduling module; 2.6.2 The scheduling module will divide the Token sequence of the W tasks in the current running queue according to a t into multiple smaller Token subsequences, if W1 tasks among the W tasks have completed pre-filling, the decoding of the W1 tasks that have completed pre-filling is performed while the pre-filling of the W-W1 tasks that have not completed pre-filling is performed; After execution, the scheduling module calculates the reward r of the tth time step with the hierarchical normalized double-objective reward function t Meanwhile, the task queue management module monitors the Tensorcore of the GPU, and if the Tensorcore is idle, the task queue management module supplements the scheduling module with W tasks. 2.6.3 The real-time state acquisition module uses the state acquisition method described in step 2.5 to monitor the task queue management module, the scheduling module, and the GPU resource usage, to obtain task characteristics and resource state information, and to integrate the task characteristics and resource state information of the t+1th time step into the task characteristic and resource state information set s of the t+1th time step t+1 , and send s t+1 to the dynamic adaptive pre-padding sub-blocking module; at the same time, the task queue management module monitors the Tensor core of the GPU, and if there is an idle Tensor core, the task queue management module supplements tasks to W to the scheduling module. 2.6.4 Dynamic adaptive prefilling and segmenting module builds training set: the dynamic adaptive prefilling and segmenting module stores the quadruple of (s t ,a t ,r t ,s t+1 ) into the replay buffer If the reasoning of the N intercloud tasks is completed, go to 2.6.5; if the reasoning of the N intercloud tasks is not completed, let t = t + 1, and go to 2.6.

1. 2.6.5 training the reinforcement learning agent for Max Kepoch epochs using the quadruple in updating the parameters of the reinforcement learning agent, Max Kepoch being a positive integer. 2.7 If episode>Max_episode, Max_episode is a positive integer, end the training, get the trained reinforcement learning intelligent agent, and get the trained adaptive pre-filled sub-blocked inter-cloud large model inference task scheduling system based on reinforcement learning, and turn to the third step; If episode≤Max_episode, turn to 2.2 to continue the next round of training; Third step, the trained adaptive pre-filled sub-blocked inter-cloud large model inference task scheduling system based on reinforcement learning schedules the inter-cloud tasks; 3.1 The task queue management module receives N inter-cloud tasks from the clients in the cloud environment within a fixed time period Time, assigns a unique task ID to each inter-cloud task, initializes the task state as a waiting state, and places the task in the waiting queue; 3.2 The task queue management module submits W tasks in the waiting queue to the running queue of the scheduling module according to the computing capability of the Tensor core of the GPU, W≤N; 3.3 Initialize the time step tp=1; 3.4 The real-time state acquisition module uses the state acquisition method described in step 2.5 to monitor the task queue management module, the scheduling module, and the GPU resource usage, to obtain task characteristics and resource state information, and to integrate the task characteristics and resource state information of the tpth time step into a task characteristic and resource state information set of the tpth time step s tp s tp is sent to the dynamic adaptive pre-padding subblocking module; is the number of tasks in the running in the tpth time step; is the number of tasks in the waiting queue in the task queue management module in the tpth time step; is the number of tasks in the running in the tpth time step; is the number of tasks in the running in the tpth time step; is the GPU computing utilization rate in the tpth time step, is the GPU memory usage rate in the tpth time step; is the task state vector in the tpth time step; is the TTFT value vector of the W tasks in the tpth time step; is the TBT average value vector of the W tasks in the tpth time step; is the TBT variance vector of the W tasks in the tpth time step; is the TBT standard deviation vector of the W tasks in the tpth time step; 3.5 The reinforcement learning agent of the dynamic adaptive pre-padding chunking module receives s from the real-time state collection module tp , utilizes s tp to give the optimal pre-padding chunk size, method is: the trained reinforcement learning agent receives s from the real-time state collection module tp , the Actor gives the optimal pre-padding chunk size a tp at the tp-th time step according to s tp , sends a tp to the scheduling module; 3.6 The scheduling module divides the Token sequence of the W tasks in the current running queue according to a tp into multiple smaller Token subsequences, and if W1 tasks among the W tasks have completed pre-filling, the scheduling module performs decoding on the W1 tasks that have completed pre-filling while pre-filling the W-W1 tasks that have not completed pre-filling; the scheduling module determines whether the a tp After the pre-filling is performed, the task characteristics and resource state information change, and it is determined whether all the N tasks in the running queue have completed, i.e., whether the running queue and the waiting queue are empty. If both the running queue and the waiting queue are empty, go to 3.

7. If the waiting queue is not empty, the task queue management module monitors the Tensor core of the GPU. If there is an idle Tensor core, the task queue management module supplements tasks to the scheduling module to W, and let tp = tp + 1, and return to step 3.

4. If the waiting queue is empty and the running queue is not empty, let tp = tp + 1, and return to step 3.

4. 3.7 The large language model completes the scheduling and inference of N tasks, and ends.

2. The reinforcement learning block-based inter-cloud large model inference task scheduling method according to claim 1, wherein The task queue management module is connected with the real-time state acquisition module, the scheduling module, and each client in the cloud environment; the task queue management module receives inter-cloud tasks from each client in the cloud environment, assigns a unique task ID to each inter-cloud task, and the length of the waiting queue is N, N≥500; The real-time state acquisition module is connected with the task queue management module and the dynamic adaptive pre-filled sub-block module; the task characteristics include the number of running tasks, the number of waiting tasks, the number of completed tasks, the state code vector of each task, the TTFT value vector of each task, the TBT average value vector, the TBT variance vector, and the TBT standard deviation vector; in the state code vector of the task, the task code of the waiting state is 0, the task code of the running state is 1, and the task code of the completed task is 2; the resource utilization state includes the GPU computing utilization rate and the GPU memory utilization rate; The dynamic adaptive pre-padding sub-block module is connected with the real-time state acquisition module and the scheduling module, the Actor in the reinforcement learning intelligent agent is connected with the real-time state acquisition module, the scheduling module and the Critic, the Actor receives the task characteristics and the resource utilization state from the real-time state acquisition module, sets the Softmax output discrete action set in the Actor, selects the optimal sub-block size as the pre-padding sub-block size according to the task characteristics and the resource utilization state in the discrete action set, and obtains the optimal pre-padding sub-block size; the Actor transmits the optimal pre-padding sub-block size to the scheduling module, and simultaneously forms a state-action pair by combining the optimal pre-padding sub-block size with the task characteristics and the resource utilization state, and transmits the state-action pair to the Critic; the Critic is connected with the Actor, receives the state-action pair from the Actor, calculates the relative advantage of the state-action pair through the advantage function, obtains the long-term value estimation of the state-action pair, and updates the parameters of the Actor through the long-term value estimation of the state-action pair to guide the Actor to update in the direction of higher return; the large language model used by the scheduling module is Qwen-7B or Yi-6B, and is connected with the task queue management module and the dynamic adaptive pre-padding sub-block module; the scheduling module receives W inter-cloud large model inference tasks from the waiting queue of the task queue management module, stores the W tasks in the running queue, receives the optimal pre-padding sub-block size from the dynamic adaptive pre-padding sub-block module, divides the Token sequences of the W tasks in the running queue that need to be pre-filled into multiple smaller Token subsequences according to the optimal pre-padding sub-block size, and performs pre-padding in batches; if there are tasks that have completed pre-padding in the W tasks, the large language model decodes those tasks that have completed pre-padding, and simultaneously pre-fills the Token subsequences that have not completed pre-padding in parallel; in the process of pre-padding and decoding, if a new optimal pre-padding sub-block size suitable for the current task characteristics and resource state is received from the dynamic adaptive pre-padding sub-block module, the scheduling module divides the Token sequences of the tasks according to the new optimal pre-padding sub-block size, and performs pre-padding in batches.

3. The method of claim 1, wherein the method further comprises: 2.2 The fixed time period Time1 is 10-20s.

4. The method of claim 1, wherein the method further comprises: 2.5.1 The method for obtaining the task characteristics of the tth time step is: 2.5.1.1 Check the number of tasks in the wait queue in the task queue management module to obtain the number of tasks in the wait at the tth time step At the same time, check the number of tasks currently entering the running queue in the scheduling module to obtain the number of tasks in the running at the tth time step At the same time, check the number of tasks in the completed task queue in the scheduling module to obtain the number of tasks that have completed reasoning at the tth time step 2.5.1.2 Constructing the Task State Encoding Vector Tasks in the waiting state are encoded as 0, tasks in the running state are encoded as 1, and completed tasks are encoded as 2, The vector length is N; 2.5.1.3 Record the start time of pre-filling for W tasks in the t-th time step, denoted as ts1, ..., ts w , ..., ts W Let 1 ≤ w ≤ W, and record the completion times te1, ..., te1 of the pre-filled tasks in the t-th time step. w , ...,te W Calculate the TTFT value vector of W tasks in the t-th time step. The length is W, and its w-th element is TTFT. tw =te w -ts w ; 2.5.1.4 Record the time interval of W tasks generating tokens word by word in the t-th time step, assuming that the w-th task generates N w tokens in the t-th time step, and its time interval sequence is denoted as δw1,,…,δw i ,…,δw Nw , where δw i represents the time interval required for the w-th task to generate the i-th token, 1≤i≤N w ; 2.5.1.

5. Calculate the TBT average vector of the W tasks in the t-th time step of length W, whose w-th element 2.5.1.6 Compute the TBT variance vector of the W tasks in the t-th time step of length W, whose w-th element 2.5.1.7 Compute the TBT standard deviation vector of the W tasks in the t-th time step of length W, whose w-th element 5. The method of claim 1, wherein 2.5.2 The reading instruction of the operating system is the Linux instruction "nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv-l 1".

6. The method of claim 1, wherein 2.6.2 The scheduling module described in step 1.6.2 will split the Token sequence of the W tasks in the current run queue into a t subsequences, and if W1 tasks of the W tasks have completed pre-filling, the W-W1 tasks that have not completed pre-filling are pre-filled while the W1 tasks that have completed pre-filling are decoded. After execution, the scheduling module calculates the reward r of the tth time step using a hierarchical normalized double-objective reward function t Meanwhile, the task queue management module monitors the Tensorcore of the GPU. If the Tensorcore is idle, the task queue management module supplements the task to the scheduling module to the Wth method: 2.6.2.1 Reading the TTFT value of a task with ID i from the run queue at time step t TTFT(i) = TTFT(i) + Δτ(i) ti , the timing increment Δτ(i) for the task with ID i is calculated i The method is: If t > 1, the TTFT value of the task with task ID i in the run queue at the (t-1)th time step is read from the TTFT value vector at the (t-1)th time step TTFT(i, t-1) (t-1)i , Δτ i = TTFT ti - TTFT (t-1)i , go to 2.6.2.2; if t = 1, let Δτ i = TTFT ti , go to 2.6.2.2; 2.6.2.2 From the t-th time step TBT value vector Read the TBT average value TBTavg of task ID i obtained at the t-th time step ti Calculate the TBT average value increment Δβ of task ID i in the running queue avg，i The method is: If t > 1, read the TBT average TBTavgfor task with task ID i in the run queue at the (t-1)th time step from the TBT value vector at the (t-1)th time step (t-1)i avg，i ti (t-1)i , go to 2.6.2.3; if t = 1, let Δβ avg，i ti , go to 2.6.2.3;​​​​​ 2.6.2.3 Normalization of Δτ i and Δβ avg，i is performed, mapping Δτ i and Δβ avg,i to [-1, 1] respectively, obtaining normalized indicators Δτ i and Δβ norm of Δτ avg，i and Δβ avg，norm respectively, where τ max , τ min Represent The maximum and minimum values ​​of TTFT, β avg，max ,β avg，min Represent The maximum and minimum values ​​of the mean TBT in the middle; 2.6.2.4 TBT variance vector of W tasks in the t-th time step TBTVar ti , TBT standard deviation vector of W tasks in the t-th time step TBTStd ti , TBTVar ti , TBTStd ti , respectively, to obtain the normalization index β ti of TBTVar var，norm and the normalization index β ti of TBTStd std，norm , wherein β var，max represents the maximum value in β std，max represents the maximum value in β var，norm and β std，norm can suppress delay fluctuations; 2.6.2.5 Constructing the composite reward function r at the tth time step t : r t = v Δτ norm + η Δβ avg，norm + ib var，norm + δβ std，norm ; Equation (1) Wherein, ν, η, ι, δ are reward weight parameters, respectively 0.5, 0.5, 0.5, 0.

5.

7. The method of claim 1, wherein 2.6.4 as described in step The size is set to store 10,000 to 100,000 quads.

8. The method of claim 1, wherein 2.6.5 Step described in The method for updating the parameters of the reinforcement learning agent is: 2.6.5.1 Initialize the number of updates Kepoch=1; 2.6.5.2 will be input from K quadruplets are randomly extracted and input into the policy neural network Actor and the value neural network Critic, K is the batch size, K is a positive integer; 2.6.5.3 Update the policy network parameters θ of the Actor, the method is: 2.6.5.3.1 Critic adopts generalized advantage estimation (GAE) technique to estimate the advantage function A π (s t , a t ) guides the update of θ towards the direction of actions with higher advantage, and the advantage function A πθ (s t , a t ) is defined as: where λ is a trade-off coefficient, γ is a discount factor, λ ∈ [0, 1], λ ∈ [0, 1]; δ t+k = r t + γV π (s t+1 ) - V π (s t ), r t is the reward function value at the t-th time step, V π (s t ) is a state value estimation function, which evaluates the value estimate of a state-action pair, denotes the expectation over its internal terms under a certain probability distribution, T represents the last time instant at which a quadruple is drawn from , k is a positive integer from 0 to T-t; 2.6.5.3.2 Update the policy network parameters θ through the loss function shown in formula (3): wherein ratio is the action probability ratio of the new and old policy networks, where θ' is the new policy network parameter, θ is the old policy network parameter, ratio is limited to the interval [1 - e, 1 + e] by the double clipping operation clip function, where e is a hyperparameter that controls the size of the interval; 2.6.5.3.3 Update θ by setting J Actor (θ) Take the partial derivative of θ to get the gradient Update the policy network parameters θ to where α is a preset learning rate; 2.6.5.4 Update the value network parameter φ of Critic; 2.6.5.4.1 Update the value network parameter φ by the loss function shown in formula (4): wherein V for the multi-step discounted return computed by the GAE, φ (s t ) is a state-value estimation function that evaluates the value estimate for a state-action pair, denotes the expectation over its internal terms under a particular probability distribution, delta t+k = r t+k + gamma V φ′ (s t+k+1 ) - V φ′ (s t+k ); 2.6.5.4.2 By setting J Critic (φ) the gradient of φ with respect to φ Update the policy network parameters φ to 2.6.5.5 Let the update number Kepoch = Kepoch + 1; 2.6.5.6 If Kepoch > Max Kepoch, the maximum iteration number Max Kepoch is a positive integer, indicating that the reinforcement learning agent completes a round of parameter update, and a round of training is completed; if Kepoch ≤ Max Kepoch, go to 2.6.6.2 to continue the current round of training.

9. The method of claim 8, wherein the method further comprises: 2.6.5.2 The value of K in step 2 is 3-15; 2.6.5.3.2 In step 2, ∈ ∈ [0, 1]; 2.6.5.3.3 In step 2, the value of α is 0.0001-0.0005; 2.6.5.6 In step 2, the value of Max Kepoch is 10-20.

10. The method of claim 1, wherein 2.7 The value of Max_episode in step 2 is 150-200.

Images