Method and system for reasoning unloading of large model agent in edge computing scene and medium

Abstract

The invention discloses a reasoning unloading method and system for a large model agent in an edge computing scene and a medium. The method comprises the steps that a to-be-processed target large model task is input into a pre-constructed Transform-PPO model to obtain a corresponding target unloading scheme used for decomposing the target large model task into a plurality of target sub-tasks and determining corresponding unloading decisions, and the model building process comprises the step of obtaining a reasoning chain of a directed acyclic graph from the large model task through a Transform network; generating an unloading scheme comprising an unloading strategy of each subtask of the inference chain based on a PPO algorithm, wherein the unloading strategy is an unloading decision of the subtask and a probability of the unloading decision; and punishing and / or rewarding the unloading strategy for increasing and / or reducing the total delay in the last round of training to obtain an unloading scheme in the current round of training. According to the method, a better reasoning unloading scheme of the large model task can be obtained, the processing efficiency of the large model task is improved, and the equipment energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of large models and edge computing technology, and in particular to a method, system and medium for offloading reasoning of a large model intelligent agent in an edge computing scenario. Background Art

[0002] In recent years, the rapid increase of IoT devices has generated a large amount of data, which needs to be processed close to the data source to reduce transmission delay and improve response speed. 5G communication technology can quickly transmit this data to edge computing nodes, and the low latency characteristics of the 5G network itself also enable edge computing to respond more quickly. [1] In this context, the field of mobile computing has also undergone a major transformation, gradually shifting from centralized mobile cloud computing to mobile edge computing. Mobile edge computing integrates the technologies and theories of mobile computing and wireless communications. Its main function is to delegate tasks such as mobile computing, network control, and storage to network edge nodes (for example, base stations and routers). [2] , in order to run computationally intensive and latency-sensitive inference chain programs on mobile devices with limited computing resources. Mobile edge computing will significantly reduce computing latency and energy consumption of mobile devices [3] .

[0003] Inference refers to the process of running real-time data through a trained AI model to make predictions or solve tasks. Inference offloading is a specific type of task offloading. A large model refers to a machine learning model with a large number of parameters (usually more than one billion) and a complex computing structure. These models are usually built with deep neural networks and can process massive amounts of data to complete various complex tasks, such as natural language processing, computer vision, and speech recognition. A large model agent can be understood as a system that can independently plan and make decisions and use a variety of tools to complete complex tasks. In this system, the large model plays the role of a "core scheduler." This scheduler is responsible for interpreting the user's natural language input, planning a series of executable subtasks, and then calling the appropriate model to gradually complete these subtasks. The process of calling the model to complete all subtasks is called inference of the large model agent.

[0004] Humans have long been pursuing artificial intelligence (AI) that is equal to or exceeds human levels. Large language models are seen as a potential spark for general artificial intelligence (AGI), providing hope for building general large model agents. [4]. Many researchers have used large models as the basis for building large model agents and have made significant progress. The English word for agent is Agent, and the AI ​​industry has proposed various definitions of agents. Agent is a general problem solver. From the perspective of software engineering, an agent is a computer program based on a large language model that has planning and thinking capabilities, memory capabilities, and the ability to use tool functions, and can autonomously complete given tasks. However, deploying large model agents locally will cause problems with insufficient device computing power and energy consumption. How to offload multiple subtasks decided by LLM to the edge to run, thereby reducing the total latency and total energy consumption of LLMAgent to complete user requests, has become a problem to be solved. [5] .

[0005] Task offloading is a core technology in edge computing (inference offloading is a specific type of task offloading), which allows computing-intensive tasks to be transferred from resource-constrained user servers and local devices, such as smartphones and sensors, to more powerful edge servers or edge devices. The purpose of this is to reduce device energy consumption, improve task execution efficiency, and reduce latency, thereby improving the overall user experience. [6] The decision-making process of task offloading needs to comprehensively consider multiple factors such as the computing requirements of the task, the amount of data, the network status, the load and availability of the edge device, etc., aiming to optimize multiple goals such as task completion time, energy consumption and cost. In terms of technical implementation, task offloading often uses a directed acyclic graph (DAG) to represent the dependencies between tasks. [7] , and decides whether to offload through static rules or dynamic algorithms. These algorithms may include heuristic algorithms, machine learning, or reinforcement learning, which make decisions dynamically based on the current environment and system status. Although task offloading brings significant performance advantages, it also faces challenges such as network fluctuations, data security and privacy protection, and the heterogeneity of edge computing environments.

[0006] Existing large-model intelligent agent deployment solutions such as HuggingGPT are deployed on the cloud. When receiving user requests, HuggingGPT calls ChatGPT's API for task planning. ChatGPT selects the appropriate AI model to perform each subtask based on the metadata of various models in HuggingFace, and summarizes the response based on the execution results. By leveraging ChatGPT's powerful language capabilities and the rich AI models in HuggingFace, HuggingGPT can handle a variety of complex AI tasks across different modes and fields, and achieve impressive results in language, vision, speech and other challenging tasks. [8] .

[0007] However, existing large-model intelligent agent reasoning solutions rely heavily on cloud API services, which have problems such as long response time, unstable service and privacy issues. First, requests and responses to cloud models require multiple round trips in the network, which will cause considerable delays and hinder the fast model reasoning capabilities required for real-time reasoning chains. For example, robot control and instant response to voice conversations are crucial. Secondly, cloud computing will have a slower response speed and even service unavailability during peak hours. Finally, cloud-based large models have raised serious privacy issues, especially when dealing with sensitive data from hospitals and banks, as well as personal data such as text input and photos on mobile phones. At the current stage, large models are not only getting bigger, but also getting smaller. This allows large models to run on mobile devices and edge devices with limited resources.

[0008] Existing reasoning offloading methods such as deep reinforcement learning (DRL) have become a popular direction for task offloading in edge computing in recent years due to its flexible and adaptive environment characteristics. [9] DRL learns effective strategies (i.e., mapping from environmental states to actions) by interacting with the environment in order to maximize rewards. With the powerful feature extraction capabilities of DNN, DRL can effectively solve complex decision-making problems in large, high-dimensional state and action spaces. DRL is used to handle task offloading problems in MEC.

[0009] However, existing inference offloading methods have high computational complexity. In edge computing systems, time delay is a key indicator, and genetic algorithms have high computational complexity, may require a long search time, and have the risk of falling into a local optimal solution. Such long-running algorithms may cause system failure because time delay is critical to system stability and response speed.

[0010] [1]X.Kong,Y.Wu,H.Wang,F.Xia,Edge computing for internet ofeverything:Asurvey,IEEE Internet of Things Journal 9(23)(2022)23472–23485.

[0011] [2] W.Shi, J.Cao, Q.Zhang, Y.Li, L.Xu, Edge computing: Vision and challenges, IEEE Internet of Things Journal 3(5)(2016)637–646.

[0012] [3]Y.Siriwardhana,P.Porambage,M.Liyanage,M.Ylianttila,A survey onmobile augmented reality with 5g mobile edge computing:Architec-tures,applications,and technical aspects,IEEE Communications Surveys&Tutorials 23(2)(2021)1160–1192.

[0013] [4]L.Wang,C.Ma,X.Feng,Z.Zhang,H.Yang,J.Zhang,Z.Chen,J.Tang,X.Chen,Y.Lin,W.X.Zhao,Z.Wei,J.Wen,A survey onlarge language model based autonomousagents,Frontiers of Computer Science 18(6)(2024)186345.

[0014] [5]Y.Mao,C.You,J.Zhang,K.Huang,K.B.Letaief,A survey on mobile edgecomputing:The communication perspective,IEEE Communications Surveys&Tutorials19(4)(2017)2322–2358.

[0015] [6]J.Chen,X.Ran,Deep learning with edge computing:Areview,Proceedingsof the IEEE 107(8)(2019)1655–1674.

[0016] [7]W.Liang,J.Xiao,Y.Chen,C.Yang,K.Xie,K.-C.Li,B.Di Martino,Tmhd:Twin-bridge scheduling of multi-heterogeneous dependent tasks for edge computing,Future Generation Computer Systems 158(2024)60–72.

[0017] [8]Y.Shen, K.Song,

[0018] [9] J.Wang, J.Hu, G.Min, W.Zhan, AYZomaya, N.Georgalas, Dependent taskoffloading for edge computing based on deep reinforcement learning, IEEETransactions on Computers 71(10)(2022)2449–2461.

[0019] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0020] The purpose of the present invention is to provide a method, system and medium for reasoning offloading of large model intelligent agents in edge computing scenarios, which can obtain a better reasoning offloading solution for large model tasks, thereby improving the processing efficiency of large model tasks and reducing equipment energy consumption.

[0021] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0022] A method for offloading reasoning of a large model agent in an edge computing scenario, the method comprising the following steps:

[0023] Input the target large model task to be processed into the pre-built Transformer-PPO model to obtain a target offloading scheme for completing the target large model task, wherein the target offloading scheme includes multiple target subtasks, the execution order of each target subtask and an offloading decision, wherein the offloading decision includes offloading and not offloading, and the target large model task is decomposed into multiple target subtasks;

[0024] The Transformer-PPO model is established in advance through the following steps:

[0025] Design a basic model based on Transformer network and PPO algorithm;

[0026] Taking the large model task as a learning sample, the Transformer network is used to convert the large model task into an inference chain G(V,E), where G(V,E) is a directed acyclic graph, V is the set of vertices of the directed acyclic graph, which represents the set of subtasks decomposed from the large model task, and E is the set of directed edges in the directed acyclic graph. i ,v j )∈E represents subtask v i With subtask v j The dependency relationship between i ∈V,v j ∈V, i and j are both positive integers;

[0027] Generate an offloading scheme of the inference chain, wherein the offloading scheme includes an offloading strategy of each of the subtasks, and the offloading strategy is an offloading decision of the subtask and a probability of the offloading decision;

[0028] The basic model is subjected to multiple rounds of optimization training in the following manner until a trained model is obtained and used as the Transformer-PPO model: based on the PPO algorithm, the unloading scheme in this round of training is obtained by penalizing the unloading strategy that increases the total delay in the previous round of training and / or rewarding the unloading strategy that reduces the total delay in the previous round of training, wherein the total delay is the total time taken to complete the task vector according to the unloading scheme.

[0029] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the unloading strategy of the subtask in the unloading solution is expressed as follows:

[0030] π(a i |O i-1 ,G(V,E))=softmax(f Transformer (s i ))

[0031] Among them, π(a i |O i-1 , G(V,E)) represents the task vector O based on the inference chain G(V,E). i-1 Under the uninstallation scheme, subtask v i The uninstall decision is a i The probability of softmax() represents the probability distribution generated by the softmax function, f Transformer (s i ) is the output of the Transformer network.

[0032] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the basic model is subjected to multiple rounds of optimization training based on the PPO algorithm to obtain a better unloading solution, and further includes:

[0033] For each time step t, calculate the strategy ratio r of the unloading scheme t (θ), which is calculated as:

[0034]

[0035] Among them, π θ (a t |s t ) indicates that after the strategy is updated, in state s t Select action a under the influence of t The probability of Indicates that the state is s before the policy update t Select action a under the influence of t The probability of

[0036] A preset clipping loss function is used to limit the update amplitude of the offloading scheme, and the clipping loss function is expressed as follows:

[0037]

[0038] Among them, L CLIP (θ) represents the clipping loss function, E t represents the action value function, clip(r t (θ), 1-∈, 1+∈) is used to limit the amplitude of strategy update, ∈ is a hyperparameter, is the advantage function, which is expressed as: γ represents the discount factor, λ is the preset parameter, r t (θ) represents the strategy ratio, V(s t+k+1 ) indicates state s t+k+1 The value of V(s t+k ) indicates state s t+k The value of , θ represents the neural network parameters.

[0039] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, generating an unloading solution for the reasoning chain includes the following steps:

[0040] Construct a state space S according to the inference chain G(V, E), wherein the state space S includes the computing power requirements corresponding to each of the subtasks, the size of the submodel used to process each of the subtasks, the amount of data corresponding to completing each of the subtasks, the topological structure of the inference chain G(V, E), the wireless transmission rate, the local device status, and the edge device status;

[0041] The state space S is used as input and the offloading strategy calculation formula based on the softmax function is used to determine the offloading strategy of the subtask.

[0042] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the basic model is subjected to multiple rounds of optimization training based on the PPO algorithm to obtain a better unloading solution, and the following steps are also included:

[0043] A preset reward function, the calculation formula of the reward function is: in, Represents task v i The impact of the offloading decision on the total delay, Indicates that task v is included in this round i The total delay of the offloading decision under the first offloading scheme, represents the total delay under the first unloading scheme of the previous round;

[0044] In the kth round of training, for the first subtask whose unloading decision has changed, the reward function value corresponding to the first subtask is calculated using the preset reward function;

[0045] If the reward function value is positive, a positive reward is given to the unloading decision of the first subtask in the kth round of training in the k+1th round of training to increase the probability that the first subtask maintains the unloading decision in the kth round of training;

[0046] If the reward function value is negative, a negative reward is given to the offloading decision of the first subtask in the kth round of training in the k+1th round of training to reduce the probability that the first subtask maintains the offloading decision in the kth round of training.

[0047] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the total delay is determined by the following method:

[0048] Determine each sub-termination task in the reasoning chain, wherein the sub-termination task is the end point of the directed edge;

[0049] Determine the completion time of each of the sub-termination tasks, where the completion time of the sub-termination task is the time taken from the start of execution of the first sub-task in the inference chain to the completion of execution of the sub-termination task;

[0050] The maximum value of the completion time of the plurality of sub-terminated tasks is determined as the total delay.

[0051] Further, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes calculating the total delay using the following formula:

[0052]

[0053] Among them, AL On represents the total delay, is the total delay of the inference chain under the offloading scheme, and v e It indicates that the sub-termination tasks in the inference chain and the end points of the directed edges correspond one to one with the sub-termination tasks, Represents the sub-termination task v e The local execution completion time of the sub-task v e The completion time of the part / all executed on the local device, Represents the sub-termination task v e The edge device execution completion time, indicating the sub-termination task v e The time to complete part / all of the execution on the edge device, represents the completion time of each sub-terminated task in the inference chain, It indicates that the completion time of each sub-terminated task in the inference chain is maximized.

[0054] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the completion time of the subtask is calculated in the following manner:

[0055] If the subtask v i The uninstall decision is uninstall, then its completion time is the subtask v i Local execution completion time on the local device

[0056] If the subtask v i If the uninstall decision is not to uninstall, its completion time is based on the upload completion time Edge device execution completion time and download completion time Determine, wherein the upload completion time To convert the subtask v i The completion time of offloading from the local device to the edge device, and the edge device performing the completion time The completion time of the subtask executed by the edge device, the download completion time To convert the subtask v i The completion time of the transmission from the edge device to the local device.

[0057] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the local execution completion time is calculated by the following method:

[0058]

[0059] Among them, pre(v i ) represents the task v i All predecessor tasks of , max means finding the maximum value, Represents task v i Take the maximum value of the upload completion time and download completion time of all predecessor tasks. represents the preparation time of the local device, which represents the idle time before the task starts to execute. Indicates local execution delay;

[0060] The upload completion time is calculated by

[0061]

[0062] in, Indicates the preparation time of the upload link. Indicates upload delay;

[0063] The edge device execution completion time is calculated by the following method

[0064]

[0065] in, represents the preparation time of the edge device, Indicates edge device execution latency;

[0066] The download completion time is calculated by

[0067]

[0068] in, Indicates the preparation time of the download link. Indicates a download delay.

[0069] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the preparation time of the local device is calculated by the following formula: in, Represents subtask v i-1 The corresponding local device preparation time, Represents subtask v i-1 The corresponding local execution completion time;

[0070] The preparation time of the upload link is calculated by the following formula: in, Represents subtask v i-1The corresponding upload link preparation time, Represents subtask v i-1 The corresponding upload completion time;

[0071] The preparation time of the edge device is calculated by the following formula in, Represents subtask v i-1 The corresponding edge device preparation time, Represents subtask v i-1 The corresponding edge device execution completion time;

[0072] The preparation time of the download link is calculated by the following formula: in, Represents subtask v i-1 The corresponding download link preparation time, Represents subtask v i-1 The corresponding download completion time.

[0073] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the upload delay The calculation formula is: in, It is task v i The data size, R up Upload rate;

[0074] The execution delay The calculation formula is: Among them, C i It is subtask v i The amount of computation required, cs ct It is the computing power of the edge server (edge ​​device host);

[0075] The download delay The calculation formula is: in, It is subtask v i The return data size, R do is the download rate;

[0076] The local delay The calculation formula is: Among them, cs ud Indicates the computing power of the user's server.

[0077] Furthermore, based on any one of the technical solutions or a combination of multiple technical solutions mentioned above, in the process of establishing the Transformer-PPO model, the state space, environmental information, and unloading decision are used as inputs, the value judgment network is used to obtain the unloading strategy, and the basic model is trained with the minimum total delay as the optimization goal.

[0078] According to another aspect of the present invention, the present invention provides a reasoning offloading system for a large model agent in an edge computing scenario, including a local device and a plurality of edge devices communicatively connected to the local device;

[0079] The local device stores the Transformer-PPO model obtained by the reasoning offloading method of the large model agent in the edge computing scenario described in any one of the above technical solutions or a combination of multiple technical solutions.

[0080] According to another aspect of the present invention, the present invention provides a computer-readable storage medium for storing program instructions, wherein the program instructions are configured to call and execute the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0081] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0082] a. Based on the Transformer network, the present invention constructs the large model task as a directed acyclic graph reasoning chain, reasonably considers the dependencies between subtasks, and can better offload the reasoning task of the large model agent from a global perspective, avoiding the existing offloading scheme of building each subtask in isolation, which makes the reasoning offloading process of the large model task more difficult due to the unclear dependencies between each subtask;

[0083] b. Based on the PPO algorithm, the present invention uses the state space of factors including device computing power, device video memory, model size, model calculation amount, etc., and combines with the reinforcement learning mechanism to use the pre-built strategy proportion function to determine the unloading decision, and optimizes each round of training according to the reward function. It can adapt to environmental changes, avoid local optimal solutions, and achieve optimal configuration of global resources, thereby improving the real-time and accuracy of task scheduling and the overall efficiency of the system, ensuring the high performance and reliability of the edge computing system in complex dynamic environments;

[0084] c. The present invention uses a clipping loss function to limit the amplitude of the unloading strategy update. By sampling trajectories multiple times and updating the policy network parameters, the optimal unloading decision is finally output, thereby learning more complex unloading strategies. Existing heuristic algorithms and rule-based algorithms cannot generalize well to unseen situations when processing continuous or high-dimensional state spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 A schematic diagram of a reasoning offloading process for a large model task provided for an exemplary embodiment of the present invention;

[0087] Figure 2 A schematic diagram of the reasoning process of a specific reasoning chain provided for an exemplary embodiment of the present invention;

[0088] Figure 3 A schematic diagram of an optimization principle of an inference offloading solution provided for an exemplary embodiment of the present invention;

[0089] Figure 4 A flowchart of the training process of the Transformer-PPO model provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0091] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0092] The technical terms and their meanings involved in the present invention include the following:

[0093] 1. Reinforcement learning: Reinforcement learning is a machine learning algorithm that trains the model to learn to make the best decision in a specific environment. After the model training is completed, it is the algorithm that achieves the optimization goal;

[0094] 2. Heuristic algorithm: Heuristic algorithm is a method of solving problems based on experience and intuition. Its function is to simplify the solution process of complex problems and improve the speed of solving them, but it is not guaranteed to be the optimal solution. It is suitable for problems with too high a cost to solve.

[0095] 3. PPO algorithm (Proximal Policy Optimization): It is a reinforcement learning algorithm used to train intelligent agents to make decisions in the environment. It improves training stability by limiting the value of each policy update to avoid excessive value updates;

[0096] 4. Image captioning is one of the main goals of computer vision, which aims to automatically generate natural descriptions of images. It not only requires identifying salient objects in images and understanding their interactions, but also requires expressing them using natural language, which makes it very challenging;

[0097] 5. Among the many tasks of natural language processing (NLP), semantic similarity algorithm is a core technology, which is mainly used to measure the degree of similarity between two texts in meaning.

[0098] Existing task offloading technology considers the number of CPU cycles and memory size required for the task to offload the task, which is not applicable to reasoning offloading in large model agent reasoning scenarios. Similar studies include reasoning offloading of large models and reasoning offloading of deep neural network models. These studies focus more on reasoning offloading of a single model, offloading the entire model as a task to the edge device, or sharding the model and offloading the sharded model as a subtask to the edge device. However, due to the complex dependencies between subtasks, model reasoning offloading is more difficult.

[0099] Existing task offloading is based on heuristic algorithms such as genetic algorithms, which may fall into local optimal solutions rather than global optimal solutions. In edge computing environments, if the genetic algorithm selects a local optimal solution, it may lead to unreasonable resource allocation, thus affecting system performance. If an edge node is assigned too many computing tasks, it may cause the node to be overloaded while other nodes are idle, which will reduce the overall efficiency of the system.

[0100] In view of these shortcomings, the present invention mainly focuses on the problem of offloading reasoning of large-model intelligent agents deployed in edge environments, as well as the problem of task scheduling optimization of the combination of multiple terminal devices and edge nodes. Offloading the reasoning tasks of large-model intelligent agents to edge devices close to the data source, such as edge devices, edge gateways, and mobile phones, is a promising solution to the above problems. The optimization goals are: 1) Reduce latency: By optimizing task allocation and edge node selection, the time for data processing and transmission is reduced, thereby improving the response speed of the system; 2) Save energy consumption: Reasonably allocate tasks, reduce unnecessary calculations and transmissions, reduce the energy consumption of terminal devices and edge nodes, and extend the life of the equipment; 3) Protect user privacy, data will not be uploaded to the cloud, avoid enterprises from illegally using user data and data theft during transmission; 4) Improve resource utilization: By dynamically adjusting task allocation, ensure that resources are used most effectively and avoid waste of resources.

[0101] The purpose of this invention is to design a task scheduling algorithm / model for edge computing systems based on reinforcement learning for large model agent reasoning offloading. The algorithm / model aims to fully combine the factors such as device computing power, device video memory, model size, model calculation amount, etc. that need to be considered in large model reasoning and combine with reinforcement learning mechanism, adapt to environmental changes, avoid local optimal solutions, and achieve optimal configuration of global resources, thereby improving the real-time performance, accuracy and overall system efficiency of task scheduling, and ensuring the high performance and reliability of edge computing systems in complex dynamic environments.

[0102] In one embodiment of the present invention, a method for offloading reasoning of a large model agent in an edge computing scenario (LATO for short) is provided. Figure 4 , the method comprises the following steps:

[0103] Input the target large model task to be processed into the pre-built Transformer-PPO model to obtain a target offloading scheme for completing the target large model task, wherein the target offloading scheme includes multiple target subtasks, the execution order of each target subtask and an offloading decision, wherein the offloading decision includes offloading and not offloading, and the target large model task is decomposed into multiple target subtasks; wherein offloading means offloading the target subtask to an edge device for execution, and not offloading means not offloading the target subtask to an edge device and executing it on a user device;

[0104] The Transformer-PPO model is established in advance through the following steps:

[0105] Design a basic model based on Transformer network and PPO algorithm;

[0106] Taking the large model task as the learning sample, the Transformer network is used to obtain the reasoning chain G(V,E), where G(V,E) is a directed acyclic graph, V is the set of vertices of the directed acyclic graph, which represents the set of subtasks decomposed from the large model task, and E is the set of directed edges in the directed acyclic graph. i ,v j )∈E represents subtask v i With subtask v j The dependency relationship between i ∈V,v j ∈V, i and j are both positive integers;

[0107] Generate an offloading scheme of the inference chain, wherein the offloading scheme includes an offloading strategy of each of the subtasks, and the offloading strategy is an offloading decision of the subtask and a probability of the offloading decision;

[0108] The basic model is subjected to multiple rounds of optimization training in the following manner until a trained model is obtained and used as the Transformer-PPO model: based on the PPO algorithm, the unloading scheme in this round of training is obtained by penalizing the unloading strategy that increases the total delay in the previous round of training and / or rewarding the unloading strategy that reduces the total delay in the previous round of training, wherein the total delay is the total time taken to complete the task vector according to the unloading scheme.

[0109] Different from the existing uninstallation method, in this application, Figure 2 As shown, the large model agent performs task planning based on the Transformer network for the user's input, i.e., the target large model task to be processed, and decomposes the target large model task into an inference chain consisting of multiple target subtasks. Each target subtask in the inference chain is executed by a corresponding target submodel. Then, the inference chain, the information of each target submodel, and the device information are input into the algorithm (the Transformer-PPO model) provided in this application. The algorithm gives the offloading decision of each target subtask, and offloads each target subtask to the local device or edge device for inference according to the offloading decision. Finally, the inference / execution results of each target subtask are summarized on the large model agent and returned to the user.

[0110] Figure 3 A specific reasoning chain example is provided. Figures A, B, and C are respectively input into the same image captioning model. The output of the model reasoning built based on the Transformer network is the natural description text of each image. Then the texts are input into the semantic similarity search model in pairs. The output of the model reasoning is the similarity value of the text. Finally, the similarity is sent to the big model for reasoning and answering based on the user's input.

[0111] In the present invention, the reasoning chain is modeled as a directed acyclic graph (DAG) G = (V, E), where V is a set of vertices of the directed acyclic graph, which represents a set of subtasks decomposed from the large model task, and E is a set of directed edges in the directed acyclic graph. i ,v j )∈E represents the subtask v i With subtask v j The dependency relationship between i ∈V,v j ∈V, i, j are both positive integers; each vertex v i ∈V represents a model, each directed edge e(v i ,v j )∈E represents the model v i With model v j Calculate the dependencies between each subtask v i The required computing power is expressed in c i To express.

[0112] Offloading inference tasksv i To edge device j The process includes the following steps: First, the local device sends the task to the edge device through a wireless channel. Second, the edge device runs the model for inference. Finally, the edge device sends the inference result back to the local device.

[0113] The total delay is affected by all sub-model requirements (model size and model computation) and edge device system status (computational power and memory available on the server) corresponding to the inference chain. The goal of the optimized offloading scheme is to minimize the end-to-end delay of the entire inference chain DAG. In practice, the offloading decision is dynamic, depending on the state of the current model and the execution of the previous model.

[0114] When formulating an offloading plan, the priority of the task should be considered. The goal of offloading is to calculate an offloading plan O n =(o1,o2,...,o n ), to minimize the delay of the inference chain (DAG) G = (V, E), where n = |V|, represents the number of directed edges in the directed acyclic graph. Here, o i Represents task v i The uninstall decision, o i Use 0 to indicate no offloading, i.e. local reasoning, and 1 to indicate offloading, i.e. remote reasoning. The optimization goal is to minimize the total latency of the reasoning chain.

[0115] Based on the above analysis, in one embodiment of the present invention, the total delay is calculated using the following formula:

[0116]

[0117] Among them, AL On represents the total delay, is the total delay of the inference chain under the offloading scheme, and v e It indicates that the sub-termination tasks in the inference chain and the end points of the directed edges correspond one to one with the sub-termination tasks, Indicates the sub-termination task v e The local execution completion time of the sub-task v e The completion time of the part / all executed on the local device, Indicates the sub-termination task v e The edge device execution completion time, indicating the sub-termination task v e The time to complete part / all of the execution on the edge device, represents the completion time of each sub-terminated task in the inference chain, It indicates that the completion time of each sub-terminated task in the inference chain is maximized.

[0118] It should be noted that for a subtask, part of it may be completed locally, i.e., executed by a local device, and part of it may be completed remotely, i.e., executed by an edge device, and the actual delay of the application needs to wait until both parts of the subtask are completed. In addition, the starting point of the completion time in this application is the start time of the first task in the inference chain.

[0119] The local execution completion time is determined according to the device status, the data size of the subtask and the size of the submodel Upload completion time Edge device execution completion time and download completion time Among them, the device status includes the computing power of the device, the idle time of the device, the upload rate of the upload link, and the download speed of the download link. The devices include local devices, edge devices, receiving and sending devices on the upload link, and receiving and sending devices on the download link.

[0120] For a subtask, the completion time of the subtask is calculated as follows:

[0121] If the subtask v i The uninstall decision is uninstall, then its completion time is the subtask v i Local execution completion time on the local device

[0122] If the subtask v iIf the uninstall decision is not to uninstall, its completion time is based on the upload completion time Edge device execution completion time and download completion time Determine, wherein the upload completion time To convert the subtask v i The completion time of offloading from the local device to the edge device, and the edge device performing the completion time The completion time of the subtask executed by the edge device, the download completion time To convert the subtask v i The completion time of the transmission from the edge device to the local device.

[0123] For the subtask that needs to be offloaded, the offloading process involves three stages: data uploading, reasoning on the edge device, and result downloading. Correspondingly, each stage has a completion time, i.e., delay. The time required for the data uploading stage is the upload delay, the time required for reasoning on the edge device is the execution delay, and the time required for result downloading is the result delay.

[0124] The upload delay The calculation formula is:

[0125]

[0126] in, It is task v i The data size, R up Upload rate.

[0127] The execution delay The calculation formula is:

[0128]

[0129] Among them, c i It is subtask v i The amount of computation required, cs ct It is the computing power of the edge device (edge ​​device host).

[0130] The download delay The calculation formula is:

[0131]

[0132] in, It is subtask v i The return data size, R do is the download rate.

[0133] For subtasks that do not need to be offloaded, that is, subtasks that perform local reasoning, their completion time is the local delay. The calculation formula is:

[0134]

[0135] Among them, cs ud Indicates the computing power of the local device, c i It is subtask v i The amount of computation required.

[0136] The inference offloading method of the large model intelligent agent in the edge computing scenario provided by the present invention not only takes into account the upload delay, execution delay and download delay of each subtask in the process of calculating the total delay, but also takes into account the preparation time RT before the task execution in the task processing process. The preparation time represents the idle time of the device before the task starts to execute, specifically, including the local processor preparation time, that is, the local device preparation time Upload link preparation time Edge device preparation time and download link preparation time

[0137] The local processor preparation time The calculation formula is as follows:

[0138]

[0139] The upload link preparation time The calculation formula is as follows:

[0140]

[0141] The edge device preparation time The calculation formula is as follows:

[0142]

[0143] Download link preparation time The calculation formula is as follows:

[0144]

[0145] in, Represents subtask v i-1 The corresponding local device preparation time, Represents subtask v i-1 The corresponding local execution completion time; Represents subtask v i-1 The corresponding upload link preparation time, Represents subtask v i-1 The corresponding upload completion time; Represents subtask v i-1 The corresponding edge device preparation time, Represents subtask v i-1 The corresponding edge device execution completion time; Represents subtask v i-1 The corresponding download link preparation time, Represents subtask v i-1 The corresponding download completion time.

[0146] According to the above upload delay, execution delay, download delay, local processor preparation time, upload link preparation time, edge device preparation time and download link preparation time, for each subtask v i , determine the corresponding completion time of execution on each resource (local device, edge device, upload link and download link).

[0147] Local execution completion time The calculation formula is as follows:

[0148]

[0149] Among them, pre(v i ) represents task v i All predecessor tasks of , max means finding the maximum value, Represents task v i Take the maximum value of the upload completion time and download completion time of all predecessor tasks;

[0150] Upload completion time The calculation formula is as follows:

[0151]

[0152] Edge device execution completion time The calculation formula is as follows:

[0153]

[0154] Download completion time The calculation formula is as follows:

[0155]

[0156] In one embodiment of the present invention, the inventive concept of optimizing the unloading scheme based on Transformer-PPO is as follows: reinforcement learning (RL) interacts with the environment to learn a strategy to take corresponding actions to maximize the cumulative reward. The RL problem is represented by Markov Decision Process (MDP), which consists of a tuple (S, A, P, R, γ). Our goal is to find an unloading strategy π(s) to maximize the expected cumulative reward over time. The policy network π(A|S, θ) takes the state S as input and outputs the probability distribution of action A, where θ represents the neural network parameters. Action A includes 0 and 1, 0 means no unloading, and 1 means unloading. The policy network parameters are optimized using the PPO algorithm. During training, the network weights are updated using a batch of sampled trajectories using the PPO algorithm.

[0157] The main elements of MDP include the following:

[0158] State space S: includes the computing power requirements of the model, model size, data size, DAG topology, wireless transmission rate, and the state of edge devices (computing power, video memory capacity); the state space is expressed as: S = {s i |s i =(G(V,E),O i )}, where G(V,E) represents the encoded DAG, i Indicates a partial unloading plan;

[0159] Action space A: contains binary choices. If the task v i If the task v is executed locally on the local device, then A = 0. i Offload to edge device for execution, then A=1);

[0160] Reward function R: It is defined as minimizing the delay of the inference chain after the offloading decision. The reward function is calculated as: in, Represents task v i The impact of the offloading decision on the total delay, Indicates that task v is included in this round i The total delay of the offloading decision under the first offloading scheme, represents the total delay under the first unloading scheme in the previous round.

[0161] Based on the reward function, during the optimization training of the basic model, the model is optimized in the following ways:

[0162] In the kth round of training, for the first subtask whose unloading decision has changed, the reward function value corresponding to the first subtask is calculated using the preset reward function;

[0163] If the reward function value is positive, a positive reward is given to the unloading decision of the first subtask in the kth round of training in the k+1th round of training to increase the probability that the first subtask maintains the unloading decision in the kth round of training;

[0164] If the reward function value is negative, a negative reward is given to the offloading decision of the first subtask in the kth round of training in the k+1th round of training to reduce the probability that the first subtask maintains the offloading decision in the kth round of training.

[0165] We use Transformer to form our policy network. Transformer processes the task embedding of the DAG sequence and refines the representation through consecutive Transformer layers. Based on the output, the executor makes an offloading decision for each task. Meanwhile, the critic evaluates the value function of each task.

[0166] Specifically, the training process of the basic model also includes:

[0167] According to the inference chain G(V,E), a state space S is constructed, and the state space S includes the computing power requirements corresponding to each of the subtasks, the size of the submodel used to process each of the subtasks, the amount of data corresponding to each of the subtasks, the topological structure of the inference chain G(V,E), the wireless transmission rate, the local device status, and the status of the edge device. With the state space S as input, the offloading strategy calculation formula constructed based on the softmax function is used to determine the offloading strategy of the subtask.

[0168] In the basic model, the Transformer network is used to implement the following:

[0169] Input embedding vector: The embedding vector includes a task feature vector, a parent task vector and a child task vector, wherein the task feature vector includes the size and computational amount of the task, the parent task vector is the task vector executed before the current task feature vector, and the task vector executed after the current task feature vector; for the embedding vector sequence [v1,v2,...,v n ], Transformer processes input through a multi-layer attention mechanism and outputs sequence embedding; sequence embedding represents the high-dimensional feature representation generated by the task embedding vector (task dependencies (parent task and subtask information) and task features) after passing through the multi-layer attention mechanism of the Transformer model. This representation encodes the dependencies, execution priorities, and resource states between tasks into a high-dimensional space, providing input for subsequent decisions (such as the generation of task offloading strategies).

[0170] Action prediction: Based on the output of the Transformer, the decision in the action space is generated by the policy network through the softmax function to generate a probability distribution, thereby choosing whether to offload or execute locally.

[0171] The unloading strategy of the subtask in the unloading scheme is expressed as follows:

[0172] π(a i |O i-1 ,G(V,E))=softmax(f Transformer (s i ))

[0173] Among them, π(a i |O i-1 , G(V,E)) represents the task vector O based on the inference chain G(V,E). i-1 Under the uninstallation scheme, subtask v i The uninstall decision is a i The probability of softmax() represents the probability distribution generated by the softmax function, f Transformer (s i ) is the output of the Transformer network.

[0174] After performing multiple rounds of optimization training on the basic model to obtain a better unloading solution, it also includes:

[0175] For each time step t, calculate the strategy ratio r of the unloading scheme t (θ), which is calculated as:

[0176]

[0177] Among them, π θ (a t |s t ) indicates that after the policy is updated, in environment s t Select action a under the influence of t The probability of Indicates that before the policy update, in environment s t Select action a under the influence of t probability;

[0178] A preset clipping loss function is used to limit the update amplitude of the offloading scheme, and the clipping loss function is expressed as follows:

[0179]

[0180] Among them, L CLIP (θ) represents the optimization objective function, E t represents the action value function, clip(rt (θ), 1-∈, 1+∈) is used to limit the amplitude of strategy update, ∈ is a hyperparameter, is the advantage function, which is expressed as: γ represents the discount factor, λ is the preset parameter, r t (θ) represents the strategy ratio, V(s t+k+1 ) indicates state s t+k+1 The value of V(s t+k ) indicates state s t+k The value of , θ represents the neural network parameters.

[0181] In the above, in the process of establishing the Transformer-PPO model, the state space, environmental information, and unloading decision are used as inputs, the value judgment network is used to obtain the unloading strategy, and the basic model is trained with the minimum total delay as the optimization goal.

[0182] like Figure 3 As shown, the reasoning offloading method (LATO) of the large model agent in the edge computing scenario provided by the present invention adopts Transformer to process task dependencies and sequence data to generate candidate offloading decisions, while PPO optimizes these decisions to ensure long-term reward maximization. First, the Transformer receives observations of the environment and produces two results: policy logits and value functions. The policy logits are passed through the softmax function to obtain the appropriate probability distribution of available actions. Next, the executor network adopts the output of the Transformer and produces a final policy, providing a probability distribution for available actions. Finally, the critic network adopts the output of the Transformer and generates an estimate of the current state. The advantage function captures the difference between the actual return and the estimated return and the estimated value of the current state.

[0183] The reasoning offloading method (LATO) for large model agents in edge computing scenarios adopts the PPO algorithm as a policy optimization method. For a given learning task T, the PPO algorithm creates a trajectory using a sample policy π(A|S, θ) and updates the target policy π(θ) in multiple epochs. The probability ratio r t (θ) is used to update the policy of the actor, and the PPO algorithm uses a clipping proxy objective to avoid large policy updates. The combined objective includes policy loss, value function loss, and entropy bonus to encourage exploration in the policy space.

[0184] In this way, LATO can effectively optimize inference offloading decisions in edge computing environments, reduce inference chain latency, and improve the computing efficiency of local devices and resource utilization of edge devices.

[0185] For example, for the trained Transformer-PPO model, input a target large model task to be processed, obtain a target reasoning chain based on the Transformer network and input it into the PPO algorithm, and the PPO algorithm directly outputs an optimal offloading solution. However, during the training process, the PPO algorithm will evaluate and optimize the offloading solution output during each round of training. For example, if the reasoning chain contains dependency information, model information, and device information of 5 tasks, the PPO algorithm will output an offloading solution consisting of 5 offloading decisions [0, 0, 0, 0, 0] based on this information. After evaluation, it is found that the fourth decision is not good, and the next round of training will output [0, 0, 0, 1, 0]. Finally, after the model converges, it can output an optimal solution [1, 1, 0, 1, 0].

[0186] The existing task offloading is partly based on heuristic algorithms such as genetic algorithms, which may fall into local optimal solutions rather than global optimal solutions. Specifically, in an edge computing environment, if the genetic algorithm selects a local optimal solution, it may lead to unreasonable resource allocation, thus affecting system performance. If an edge device (edge ​​node) is assigned too many computing tasks, it may cause the node to be overloaded while other nodes are idle, which will reduce the overall efficiency of the system. The present invention introduces the Transformer network into the PPO algorithm through the Transformer-PPO model. The Transformer processes sequence data and generates decision candidates, while PPO optimizes these decisions to maximize long-term rewards. The trained reasoning offloading decision algorithm, namely the Transformer-PPO model, can generate large-model task offloading solutions in real time and dynamically according to the dependency relationship between the models of the reasoning chain and the model information and the device information. Therefore, the reasoning offloading method of large model intelligent agents in edge computing scenarios provided by the present invention fully combines factors such as device computing power, device video memory, model size, model calculation amount, etc. that need to be considered in large model reasoning, and combines with reinforcement learning mechanism to adapt to environmental changes, avoid local optimal solutions, and achieve optimal configuration of global resources, thereby improving the real-time and accuracy of task scheduling and the overall efficiency of the system, ensuring the high performance and reliability of the edge computing system in complex dynamic environments.

[0187] In one embodiment of the present invention, there is provided an inference offloading system for a large model agent in an edge computing scenario, comprising a local device and a plurality of edge devices communicatively connected to the local device; the Transformer-PPO model obtained by the inference offloading method for a large model agent in an edge computing scenario as described in any one of the above items is stored on the local device. The target large model task to be processed is input into the local device, and the Transformer-PPO model generates a target offloading scheme for the target large model task. The inference offloading system decomposes the target large model task according to the target offloading scheme and offloads it to the edge device for execution, or executes it on the local device without offloading it.

[0188] The system uses the planning ability of the large model to split the user's task into multiple sub-tasks that can be completed by different models. Each sub-task on the reasoning chain of the large model agent is offloaded to the user device or edge device, thereby improving the response speed of the large model agent, reducing the energy consumption of the user device, and avoiding the privacy issues caused by uploading user data to the cloud.

[0189] In one embodiment of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to be called to execute the steps of the method described in any one of the above embodiments.

[0190] It should be noted that the above-mentioned reasoning offloading system for large model intelligent agents in edge computing scenarios, computer-readable storage medium embodiments and reasoning offloading method embodiments for large model intelligent agents in edge computing scenarios belong to the same inventive concept, and all contents of the reasoning offloading method embodiments for large model intelligent agents in edge computing scenarios are incorporated into the computer-readable storage medium and computer-readable storage medium embodiments by reference.

[0191] It should be noted that, in this article, relational terms such as are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0192] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for offloading reasoning of a large model agent in an edge computing scenario, characterized in that: The method comprises the following steps: Input the target large model task to be processed into the pre-built Transformer-PPO model to obtain a target offloading scheme for completing the target large model task, wherein the target offloading scheme includes multiple target subtasks, the execution order of each target subtask and an offloading decision, wherein the offloading decision includes offloading and not offloading, and the target large model task is decomposed into multiple target subtasks; The Transformer-PPO model is established in advance through the following steps: Design a basic model based on Transformer network and PPO algorithm; Taking the large model task as a learning sample, the Transformer network is used to convert the large model task into an inference chain G(V,E), where G(V,E) is a directed acyclic graph, V is the set of vertices of the directed acyclic graph, which represents the set of subtasks decomposed from the large model task, and E is the set of directed edges in the directed acyclic graph. i ,v j )∈E represents subtask v i With subtask v j The dependency relationship between i ∈V,v j ∈V, i and j are both positive integers; Generate an offloading scheme of the inference chain, wherein the offloading scheme includes an offloading strategy of each of the subtasks, and the offloading strategy is an offloading decision of the subtask and a probability of the offloading decision; The basic model is subjected to multiple rounds of optimization training in the following manner until a trained model is obtained and used as the Transformer-PPO model: based on the PPO algorithm, the unloading scheme in this round of training is obtained by penalizing the unloading strategy that increases the total delay in the previous round of training and / or rewarding the unloading strategy that reduces the total delay in the previous round of training, wherein the total delay is the total time taken to complete the task vector according to the unloading scheme.

2. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 1 is characterized in that: The unloading strategy of the subtask in the unloading scheme is expressed as follows: π(a i |O i-1 ,G(V,E))=softmax(f Transformer (s i )) Among them, π(a i |O i-1 , G(V,E)) represents the task vector O based on the inference chain G(V,E). i-1 Under the uninstallation scheme, subtask v i The uninstall decision is a i The probability of softmax() represents the probability distribution generated by the softmax function, f Transformer (s i ) is the output of the Transformer network.

3. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 2 is characterized in that: Based on the PPO algorithm, multiple rounds of optimization training are performed on the basic model to obtain a better unloading solution, and the method further includes: For each time step t, calculate the strategy ratio r of the unloading scheme t (θ), which is calculated as: Among them, π θ (a t |s t ) indicates that after the policy is updated, in environment s t Select action a under the influence of t The probability of Indicates that before the policy update, in environment s t Select action a under the influence of t probability; A preset clipping loss function is used to limit the update amplitude of the offloading scheme, and the clipping loss function is expressed as follows: Among them, L CLIP (θ) represents the clipping loss function, E t represents the action value function, clip(r t (θ), 1-∈, 1+∈) is used to limit the amplitude of strategy update, ∈ is a hyperparameter, is the advantage function, which is expressed as: γ represents the discount factor, λ is the preset parameter, r t (θ) represents the strategy ratio, V(s t+k+1 ) indicates state s t+k+1 The value of V(s t+k ) indicates state s t+k The value of , θ represents the neural network parameters.

4. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 2 is characterized in that: Generating an offloading scheme of the inference chain includes the following steps: Construct a state space S according to the inference chain G(V, E), wherein the state space S includes the computing power requirements corresponding to each of the subtasks, the size of the submodel used to process each of the subtasks, the amount of data corresponding to completing each of the subtasks, the topological structure of the inference chain G(V, E), the wireless transmission rate, the local device status, and the edge device status; The state space S is used as input and the offloading strategy calculation formula based on the softmax function is used to determine the offloading strategy of the subtask.

5. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 1 is characterized in that: Based on the PPO algorithm, multiple rounds of optimization training are performed on the basic model to obtain a better unloading solution, and the following steps are also included: A preset reward function, the calculation formula of the reward function is: in, Represents task v i The impact of the offloading decision on the total delay, Indicates that task v is included in this round i The total delay of the offloading decision under the first offloading scheme, represents the total delay under the first unloading scheme of the previous round; In the kth round of training, for the first subtask whose unloading decision has changed, the reward function value corresponding to the first subtask is calculated using the preset reward function; If the reward function value is positive, a positive reward is given to the unloading decision of the first subtask in the kth round of training in the k+1th round of training to increase the probability that the first subtask maintains the unloading decision in the kth round of training; If the reward function value is negative, a negative reward is given to the offloading decision of the first subtask in the kth round of training in the k+1th round of training to reduce the probability that the first subtask maintains the offloading decision in the kth round of training.

6. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 1 is characterized in that: The total delay is determined by: Determine each sub-termination task in the reasoning chain, wherein the sub-termination task is the end point of the directed edge; Determine the completion time of each of the sub-termination tasks, where the completion time of the sub-termination task is the time taken from the start of execution of the first sub-task in the inference chain to the completion of execution of the sub-termination task; The maximum value of the completion time of the plurality of sub-terminated tasks is determined as the total delay.

7. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 6 is characterized in that: It also includes calculating the total delay using the following formula: Among them, AL On represents the total delay, is the total delay of the inference chain under the offloading scheme, and v e It indicates that the sub-termination tasks in the inference chain and the end points of the directed edges correspond one to one with the sub-termination tasks, Indicates the sub-termination task v e The local execution completion time of the sub-task v e The completion time of the part / all executed on the local device, Indicates the sub-termination task v e The edge device execution completion time, indicating the sub-termination task v e The time to complete part / all of the execution on the edge device, represents the completion time of each sub-terminated task in the inference chain, It indicates that the completion time of each sub-terminated task in the inference chain is maximized.

8. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 6 is characterized in that: The completion time of the subtask is calculated as follows: If the subtask v i The uninstall decision is uninstall, then its completion time is the subtask v i Local execution completion time on the local device If the subtask v i If the uninstall decision is not to uninstall, its completion time is based on the upload completion time Edge device execution completion time and download completion time Determine, wherein the upload completion time To convert the subtask v i The completion time of offloading from the local device to the edge device, and the edge device performing the completion time The completion time of the subtask executed by the edge device, the download completion time To convert the subtask v i The completion time of the transmission from the edge device to the local device.

9. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 8, characterized in that: The local execution completion time is calculated by Among them, pre(v i ) indicates that the task v i All predecessor tasks of , max means finding the maximum value, Represents task v i Take the maximum value of the upload completion time and download completion time of all predecessor tasks. represents the preparation time of the local device, which represents the idle time before the task starts to execute. Indicates local execution delay; The upload completion time is calculated by in, Indicates the preparation time of the upload link. Indicates upload delay; The edge device execution completion time is calculated by the following method in, represents the preparation time of the edge device, Indicates edge device execution latency; The download completion time is calculated by in, Indicates the preparation time of the download link. Indicates a download delay.

10. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 9 is characterized in that: The preparation time of the local device is calculated by the following formula in, Represents subtask v i-1 The corresponding local device preparation time, Represents subtask v i-1 The corresponding local execution completion time; The preparation time of the upload link is calculated by the following formula: in, Represents subtask v i-1 The corresponding upload link preparation time, Represents subtask v i-1 The corresponding upload completion time; The preparation time of the edge device is calculated by the following formula in, Represents subtask v i-1 The corresponding edge device preparation time, Represents subtask v i-1 The corresponding edge device execution completion time; The preparation time of the download link is calculated by the following formula: in, Represents subtask v i-1 The corresponding download link preparation time, Represents subtask v i-1 The corresponding download completion time.

11. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 8, characterized in that: The upload delay The calculation formula is: in, It is task v i The data size, R up Upload rate; The execution delay The calculation formula is: Among them, C i It is subtask v i The amount of computation required, cs ct It is the computing power of the edge server (edge ​​device host); The download delay The calculation formula is: in, It is subtask v i The return data size, R do is the download rate; The local delay The calculation formula is: Among them, cs ud Indicates the computing power of the user's server.

12. The reasoning offloading method of a large model agent in an edge computing scenario according to claim 1, characterized in that: In the process of establishing the Transformer-PPO model, the state space, environmental information, and unloading decision are used as inputs, the value judgment network is used to obtain the unloading strategy, and the basic model is trained with the minimum total delay as the optimization goal.

13. A reasoning offloading system for large model agents in edge computing scenarios, characterized in that: A local device and a plurality of edge devices communicatively connected to the local device; The Transformer-PPO model obtained by using the reasoning offloading method of the large model agent in the edge computing scenario as described in any one of claims 1 to 12 is stored on the local device.

14. A computer-readable storage medium for storing program instructions, characterized in that: The program instructions are configured to be called to execute the steps of the method according to any one of claims 1 to 12.

Images