A task offloading terminal decision, a server decision method and a task offloading system

Abstract

The application discloses a task offloading terminal decision-making, server decision-making method and task offloading system, relates to the Internet of Vehicles technical field, and utilizes a multilayer perceptron to extract task features in real time and map the task features into priority weights; subsequently, a top-layer mobile edge computing server serves as a leader and utilizes a deep deterministic policy gradient algorithm to jointly optimize a computing resource unit cost and a computing power distribution weight; a bottom-layer terminal serves as a follower and utilizes a multi-agent approximate policy optimization algorithm to make a distributed task offloading decision; and through alternating training of the top-layer and bottom-layer algorithms, optimal decision-making of a terminal task offloading scheme is realized.

Description

Technical Field

[0001] This application relates to the field of vehicle networking, and in particular to a task unloading terminal decision-making and server decision-making method and a task unloading system. Background Technology

[0002] With the explosive growth in the number of IoT devices, Mobile Edge Computing (MEC) effectively alleviates network transmission pressure and reduces computing latency for terminal devices by deploying computing resources at the network edge. In current MEC research, task offloading decisions and resource cost assessments are often treated as two separate processes, or static cost assessment models are used to simplify system design. Researchers mostly utilize traditional optimization methods or single-agent reinforcement learning to find the optimal offloading scheme. These methods often assume a stable system environment and that all tasks have the same processing priority. Moreover, these methods primarily focus on user-side performance optimization, neglecting the revenue needs of resource providers (such as base station operators) in dynamically fluctuating environments.

[0003] In practical applications, the above methods suffer from the following problems: First, existing research often treats cost assessment and offloading as independent processes, lacking an effective interactive feedback mechanism. This makes it impossible to guide users to allocate computing resources rationally through price adjustments when task volume surges. Second, existing static cost assessment mechanisms cannot reflect the load fluctuations and cost changes of edge servers in real time, leading to low resource utilization or reduced operator revenue. Third, existing offloading decision algorithms do not adequately consider the heterogeneous priority of tasks and cannot distinguish between critical and urgent tasks and ordinary tasks. This can easily lead to timeouts for high-priority tasks when system resources are strained. Fourth, due to the conflict of interest between operators and users, existing offloading algorithms struggle to simulate the complex conflicts and game relationships between them. As a result, in a multi-user dynamic competitive environment, the algorithms often fail to converge to the overall optimal equilibrium point of the system and are prone to getting trapped in local optima, making it difficult for the system to reach a stable equilibrium state in complex dynamic scenarios. Summary of the Invention

[0004] The purpose of this application is to provide a terminal task unloading decision-making method and server decision-making method and a task unloading system, which can achieve the optimal decision-making of terminal task unloading scheme through alternating training of upper and lower layer algorithms.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for making a task offloading terminal decision, including: The task data of the current time slot terminal is input into the multilayer perceptron network to calculate the priority weight of the task, and the priority weight is sent to the mobile edge computing server. The task data includes: task data size, task computing density, remaining time before task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server. Receive the action vector returned by the mobile edge computing server; the action vector includes: computing energy consumption, computing resource unit cost, and time occupation unit cost; Based on the action vector, channel gain, and task margin, the task offloading decision of the terminal is calculated using a multi-agent approximation strategy optimization model; the task offloading decision is the ratio of task execution on the terminal local and on the mobile edge computing server. On the mobile edge computing server, the priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of the terminals within the coverage area, and the overall action vector of the terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the mobile edge computing server in the current time slot. When the number of alternating calculations between the mobile edge computing server and the terminal reaches a preset number, a task unloading decision is executed on the terminal.

[0006] In one embodiment of this application, the task data of the current time slot terminal is input into a multilayer perceptron network to calculate the priority weight of the task, specifically including: The task data is normalized to obtain normalized data; The normalized data is subjected to a nonlinear transformation using the ReLU activation function to obtain nonlinear data; The nonlinear data is combined in a high dimension using the weight matrix of a multi-layer fully connected structure to obtain the first high-dimensional data. The first high-dimensional data is mapped to priority weights using the Sigmoid activation function.

[0007] In one embodiment of this application, the task data of the current time slot terminal is input into a multilayer perceptron network to calculate the task priority weight. The calculation formula is as follows: ; ; in, For the terminal i In the time slot t Task priority weights For the priority evaluation function of a multilayer perceptron network, For the terminali Multidimensional task feature vectors For parameters related to the priority evaluation function, For the terminal i In the time slot t Task data size, For the terminal i In the time slot t Task computation density, For the terminal i The maximum tolerable delay for the task. - t This represents the remaining time before the task deadline. For the terminal i In the time slot t The task importance coefficient, This is the terminal type identifier. Mobile edge computing server j In the time slot t The current load, Mobile edge computing server j In the time slot t The queue status.

[0008] In one embodiment of this application, the priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of terminals within the coverage area, and the overall action vector of terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the mobile edge computing server in the current time slot, specifically including: The deep deterministic policy gradient model includes a first Actor network and a first Critic network; The multi-layer fully connected neural network in the first Actor network performs high-dimensional feature extraction on the priority weights, the current computing power load ratio of the mobile edge computing server, the overall priority weights of terminals within the coverage area, and the overall action vectors of terminals within the coverage area of ​​the previous time slot, to obtain the first high-dimensional data. The Sigmoid activation function in the first Actor network linearly maps the first high-dimensional data to a preset interval to obtain an initial action vector; The first Critic network minimizes the error between the initial action vector and the target action vector calculated by the target network to obtain the gradient adjustment direction of the initial action vector; The first Actor network adjusts the parameters of the initial action vector multiple times according to the gradient adjustment direction provided by the first Critic network, and the maximum value of the initial action vector is used as the action vector output.

[0009] In one embodiment of this application, the action vectors of all terminals within the coverage area of ​​the current time slot mobile edge computing server are calculated using the following formula: ; in: ; ; Represents the action vector function. Representation function Related to C, F, and α, adjusting the values ​​of C and F can make... To achieve the maximum value, C is the cost evaluation strategy variable, F is the resource allocation variable, α is the proportion matrix of tasks offloaded from all terminals to server j, C1, C2, and C3 are constraints, and M is the number of mobile edge computing servers. N is the number of terminals. , , To terminate in time slot t i Mobile edge computing servers j cost, Mobile edge computing server j The computational resource cost per unit in time slot t, Mobile edge computing server j The time unit cost of occupying time slot t Mobile edge computing server j Computational energy consumption, It is the energy consumption coefficient of the base station. The preset maximum cost budget, The preset maximum allowable energy consumption, , These are the preset minimum and maximum unit cost of computing resources, respectively. , These are the preset minimum and maximum costs per unit of time occupied. For time slots t terminal i Task offloaded to mobile edge computing server j The proportion, For the terminal i In the time slot t Task data size, For the terminal i The density is calculated in time slot t. For the terminal i Allocate computing power in time slot t Let j be the physical capacity of the mobile edge computing server.

[0010] In one embodiment of this application, based on the action vector, channel gain, and task margin, a multi-agent approximation strategy optimization model is used to calculate the terminal's task offloading decision, specifically including: The multi-agent approximation strategy optimization model includes a second Actor network and a second Critic network. The action vector, channel gain, and task margin are input into the second Actor network to obtain the initial task offloading decision. The second Critic network minimizes the error between the initial task unloading decision and the target task unloading decision calculated by the target network, thereby obtaining the gradient adjustment direction of the initial task unloading decision; The second Actor network adjusts the parameters of the initial task unloading decision multiple times based on the gradient adjustment direction provided by the second Critic network to obtain the terminal's task unloading decision.

[0011] In one embodiment of this application, the task offloading decision of the terminal is calculated using a multi-agent approximation strategy optimization model based on the action vector, channel gain, and task margin. The calculation formula is as follows: ; in, The task offloading decision function for the multi-agent approximation policy optimization model. Representation function Related to C, F, and α, adjusting the value of α can make... To achieve the maximum value, C is the cost evaluation strategy variable, F is the resource allocation variable, α is the offloading matrix, C4, C5, and C6 are constraints, and M is the number of mobile edge computing servers. N is the number of terminals. , , , These are latency weight, energy consumption weight, and cost weight, respectively. For the terminal i In the time slot t The time delay, For the terminal i In the time slot t energy consumption For the terminal i In the time slot t The cost and expense, , , These are the preset maximum allowable latency, maximum allowable energy consumption, and maximum cost budget, respectively. For the terminal i In the time slot t Task offloaded to mobile edge computing server jThe proportion, For the terminal i Maximum tolerable delay for the task For the terminal i The maximum battery capacity.

[0012] In one embodiment of this application, the terminal i In the time slot t latency and terminal i In the time slot t energy consumption The calculation formula is: ; ; ; ; ; ; ; ; in, For the terminal The delay in executing the task locally, For the terminal Tasks are offloaded to mobile edge computing servers Total delay, including transmission delay and computational delay , For the terminal The energy consumption of tasks executed locally. For the terminal Task offloaded to mobile edge computing server energy consumption For terminal transmission power, For the terminal i Local computing power The effective capacitance coefficient of the terminal CPU. In time slot t terminal i The proportion of tasks executed locally. In time slot t terminal i Tasks are offloaded to mobile edge computing servers j The proportion, For the terminal i In the time slot t Task data size, For the terminal iThe task calculates density in time slot t; For the terminal With mobile edge computing servers Instantaneous uplink transmission rate between Sub-channel bandwidth, Indicates channel gain. This represents the power of additive white Gaussian noise.

[0013] Secondly, this application provides a task offloading server decision-making method, including: The priority weight of terminal transmission, the current computing power load ratio of the mobile edge computing server, the overall priority weight of terminals within the coverage area, and the overall action vector of terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the current time slot. At the terminal, the task data of the current time slot terminal is input into the multilayer perceptron network to calculate the priority weight of the task, and the priority weight is sent to the mobile edge computing server. The task data includes: task data size, task computing density, remaining time before task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server. The terminal also receives the action vector returned by the mobile edge computing server, and calculates the terminal's task offloading decision based on the action vector, channel gain, and task margin using a multi-agent approximation strategy optimization model; the action vector includes: computing energy consumption, computing resource unit cost, and time unit cost; the task offloading decision is the ratio of task execution on the terminal's local machine and on the mobile edge computing server. On the mobile edge computing server, the priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of the terminals within the coverage area, and the overall action vector of the terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the mobile edge computing server in the current time slot. When the number of alternating calculations between the mobile edge computing server and the terminal reaches a preset number, a task unloading decision is executed on the terminal.

[0014] Thirdly, this application provides a task unloading decision system, including: Terminal and mobile edge computing server side; The terminal is configured to input task data into a multilayer perceptron network, calculate the priority weight of the task, and send the priority weight to the mobile edge computing server; receive the action vector returned by the mobile edge computing server, and input the action vector, channel gain, and task margin into a multi-agent approximate policy optimization model to calculate the task offloading decision for the current time slot of the terminal; and execute the task offloading decision on the terminal when the number of alternating calculations on the mobile edge computing server and the terminal reaches a preset number. The task data includes: task data size, task computation density, remaining time before the task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server; the action vector includes: computational energy consumption, unit cost of computational resources, and unit cost of occupied time. The mobile edge computing server is used to input the priority weight transmitted by the vehicle terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of the terminals within the coverage area, and the overall action vector of the terminals within the coverage area of ​​the previous time slot into the deep deterministic policy gradient model to calculate the action vector of the terminals within the coverage area of ​​the current time slot.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a task offloading terminal decision-making method, a server decision-making method, and a task offloading system. The task offloading terminal decision-making includes: an efficient resource management framework based on reinforcement learning of a hierarchical Stackelberg game model. The system is modeled as a master-slave game model, with the upper-layer mobile edge computing server acting as the leader, using a deep deterministic policy gradient algorithm to jointly optimize the unit cost of computing resources and the weight of computing power allocation; and multiple lower-layer terminals acting as followers, using a multi-agent approximation policy optimization algorithm to optimize the task offloading ratio and local execution strategy based on received price feedback, combined with their own task latency requirements and priority labels. Through alternating optimization of actions at the upper and lower layers, the system achieves optimal decision-making for terminal task offloading schemes while satisfying the service quality requirements of services with different priorities. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an in-vehicle edge computing network system according to one embodiment of this application.

[0018] Figure 2 This is a schematic diagram of a multilayer perceptron network provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of a deep deterministic policy gradient algorithm provided in an embodiment of this application.

[0020] Figure 4 The flowchart of a vehicle distributed task offloading decision algorithm based on multi-agent approximation strategy optimization is provided as an embodiment of this application.

[0021] Figure 5 The flowchart illustrates an algorithm implementation based on a hierarchical alternation mechanism, as provided in one embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] This application is primarily applied to resource scheduling between base stations or roadside units and mobile terminal devices in mobile edge computing scenarios. It aims to address the problems of uneven resource allocation, excessive user overhead, and unbalanced system revenue caused by insufficient cost evaluation mechanisms and inconsistent offloading decisions when edge servers handle heterogeneous tasks with different priorities. In the proposed scenario, this invention proposes an efficient resource management framework based on a layered Stackelberg game model and reinforcement learning. Specifically, the system is first modeled as a master-slave game model. The upper layer (leader) is the operator, responsible for perceiving the network-wide task load and priority distribution, and dynamically adjusting the unit cost of computing resources using an improved layered reinforcement learning algorithm. The lower layer (followers) consists of multiple terminal users who, based on received price feedback and their own task latency requirements and priority labels, optimize the task offloading ratio and local execution strategy. Through alternating optimization of actions at both the upper and lower layers, the system achieves a game equilibrium between operator profits and user costs while satisfying the service quality requirements of services with different priorities.

[0026] 1. Research Scenario and Problem Modeling like Figure 1 As shown, this invention is based on a multi-base station vehicular edge computing (VEC) network, which includes a group of mobile edge computing servers denoted as... And a group of heterogeneous vehicles, denoted as To accurately reflect actual traffic composition, vehicles are divided into two categories based on functional requirements: Autonomous Vehicles (AVs) and Traditional Vehicles (TVs). Each vehicle... In each time slot Generate a computationally intensive task that can be executed locally or offloaded to an available MEC server. (Vehicle) The generated tasks consist of multidimensional tuples ,in Indicates the amount of data. This indicates computational density (CPU cycles required per bit). For maximum tolerable delay, This indicates the inherent importance of the task. The unloading decision is... Defined, this value represents the value that is unloaded to the server. The proportion of tasks, while the remaining part Then it will be processed locally.

[0027] (1) Communication and Computation Model By employing the widely used Orthogonal Frequency Division Multiple Access (OFDMA) technology, the vehicle's... With mobile edge computing servers Instantaneous uplink transmission rate between: (1) in, Sub-channel bandwidth, For vehicle transmission power, Indicates channel gain. This represents the power of additive white Gaussian noise. Therefore, when the vehicle... Offload the task components to the server. At that time, transmission delay and corresponding energy consumption They are respectively: (2) (3) Both local computation and edge offloading increase the overall execution overhead, reflected in latency and energy consumption. For vehicles... The local execution portion of the task has a completion delay. and energy consumption Given by the following formula: (4) (5) in, Indicates local computing power. This represents the effective capacitance coefficient of the vehicle's CPU.

[0028] Unload to MEC server Task latency includes transmission latency and computation latency: (6) in, Indicates server Assigned to vehicles The computing resources. Because local execution and edge transmission can be performed in parallel, vehicles... The total task processing latency is defined as: (7) vehicle Total energy consumption includes local computing energy consumption and transmission energy consumption: (8) MEC Server The calculated energy consumption is: (9) in, It is the energy consumption coefficient of the base station.

[0029] (2) Priority Model To quantify the relative urgency and importance of heterogeneous tasks, we perform a quantification of each task. In the time slot Define a priority index The index is evaluated by a priority evaluation function. This function determines the multidimensional task feature vector. Mapped to scalar values: (10) in, These represent the parameters of the evaluation model. Feature vectors. The state space of the task terminal server is described and defined as follows: (11) This vector integrates data size, computational density, remaining time before deadline, importance coefficient, vehicle type identifier, and the current load and queue status of the mobile edge computing server. By employing a lightweight multilayer perceptron, this mechanism can adaptively map complex environmental states to standardized priority values, laying the foundation for subsequent priority-aware cost evaluation strategies.

[0030] (3) Optimization Objective To comprehensively reflect system performance, we define two independent optimization subproblems. and , representing the optimization goals of the vehicle layer and the MEC server layer, respectively.

[0031] 1) Total vehicle cost Starting from the vehicle level, the core objective is to minimize the cumulative overhead of all participating vehicles. For each individual vehicle... Its processing cost The multi-objective function consists of standardized latency, energy consumption, and associated costs. Based on the cost evaluation strategy and resource allocation scheme determined by the MEC server, the optimization problem at the vehicle layer can be expressed as: (12) in, , , These are latency weight, energy consumption weight, and cost weight. They reflect the different service quality preferences of vehicle users and can be adjusted according to specific scenarios and tasks. , , These are the system's preset maximum allowable latency, maximum allowable energy consumption, and maximum cost budget, respectively. These are normalized denominator benchmarks, ensuring that the values ​​of each item are within the range of [0,1]. This represents the maximum battery capacity of vehicle i.

[0032] This invention employs a deep reinforcement learning-driven dynamic cost evaluation mechanism. The cost evaluation policy variable C and the resource allocation variable F are not calculated using traditional static formulas, but are adaptively generated by an agent based on a deep reinforcement learning algorithm according to the real-time network state. The cost evaluation policy variable C belongs to a set, referred to as C (hereinafter referred to as C). t comp，j C t time，j This refers to the unit cost of computational resources per computational cycle and the unit cost of time occupied per unit of time in each time slot; the resource allocation variable F is also a set, referring to... This refers to the computing power that each time slot edge server provides to each vehicle.

[0033] Due to resource allocation It is to calculate the computational delay. and The prerequisite for moderate time-consuming operations is that there is an inherent coupling between vehicle unloading response and server resource allocation. Constraints (C1)-(C3) ensure the unloading matrix... Simultaneously meet the deadlines and energy capacity limits of each task. Total cost Representative vehicle Expenses related to the server. The cost The calculation method is as follows: (13) in, , It is a cost evaluation strategy variable dynamically generated by the upper-layer base station (leader) through the DDPG algorithm. The unit cost of computing resources per computing cycle represents the cost that a vehicle needs to pay to rent the computing power of a base station. It represents the time cost per unit of time occupied, used to adjust the duration of edge server usage and prevent long-running tasks from maliciously consuming resources.

[0034] 2) Total server revenue and resource allocation As the leader, the MEC server layer determines the cost evaluation strategy. and computing resource allocation scheme This achieves a balance between profitability and resource sustainability. Specifically, the server needs to allocate a certain number of CPU cycles to each offload task. The optimization problem of the MEC server layer can be described as follows: (This is to optimize the revenue generated from service time and corresponding costs.) (14) C6 ensures that the total computing resources allocated to all vehicles do not exceed the server's physical capacity. This layered approach reveals a complex two-way coupling relationship: the server layer readjusts the incentive mechanism of the vehicle layer through joint cost evaluation and resource quotas, while... The collective behavior of the vehicles ultimately determined The actual benefits realized in the process.

[0035] 2. Hierarchical Stackelberg Reinforcement Learning (HSRL) Algorithm To address the established game theory model and non-convex optimization problem, this invention designs an HSRL algorithm framework, which optimizes the long-term dynamic cost evaluation and task unloading decision by decomposing them into three related algorithmic sub-processes.

[0036] Step 1: Task Prioritization Based on Lightweight Multilayer Perceptron (MLP) To provide differentiated services for heterogeneous tasks in resource allocation, the system first calculates task priorities based on task attributes using a lightweight MLP network. The detailed process is as follows: Figure 2 As shown.

[0037] The procedure for this step is as follows: in each time slot The system uses the task's data size, computational density, remaining time before the deadline, importance coefficient, vehicle type identifier, and the current load and queue status of the mobile edge computing server as a 7-dimensional input vector. During the mapping process, the input features first undergo normalization preprocessing to eliminate dimensional differences, and then enter a neural network containing two hidden layers. The hidden layers use the ReLU activation function. A nonlinear transformation is performed, and heterogeneous features are combined in high dimension through the weight matrix of a multi-layer fully connected structure, thereby automatically identifying the cross-influence between features (such as the superposition effect of high computational density and tight deadlines). Finally, the output layer uses the Sigmoid activation function to map the processed features into a continuous priority weight index. This metric directly affects the latency penalty term of the vehicle utility function in subsequent steps, ensuring that high-priority tasks receive better scheduling logic in resource contention, thereby achieving differentiated service awareness for heterogeneous tasks. The MLP hidden layer calculates the priority weight of the task through nonlinear mapping. .

[0038] Step 2: Joint cost assessment and computing power allocation for base stations based on DDPG (upper-layer leader) As the leader in a Stackelberg game, the Base Station (BS) faces the problem of maximizing payoffs in a continuous action space. This invention introduces the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the decision-making process at the base station level. In the cost evaluation and computing power allocation decisions of the upper-level leader, the DDPG algorithm achieves policy evolution through a dual feedback loop of "decision" and "evaluation".

[0039] like Figure 3 As shown (the environment map in the upper left corner of the image is...) Figure 1 (A schematic diagram of an in-vehicle edge computing network system in the embodiment) The working method of this step is as follows: The Actor policy network acts as the decision-maker and is responsible for observing the current system state on the base station side. (This includes the current computing power load ratio of the base station, the overall priority distribution of vehicles within the coverage area, and the revenue feedback from the previous time slot). In the specific mapping implementation, the state vector undergoes high-dimensional feature extraction via a multi-layer fully connected neural network. The output layer uses the Sigmoid activation function to linearly map the neuron activation values ​​to a preset physical value range, thereby directly generating a value containing the unit cost of computing resources. Cost per unit of time occupied and the allocated computing power The continuous action vectors are generated. Meanwhile, the Critic value network, acting as the evaluator, receives the current state and the actions output by the Actor as input, and is responsible for predicting the cumulative utility gain that the decision can obtain in the long-term game. During the update process, the Critic improves the evaluation accuracy by minimizing the Bellman error between the target value generated by the target network and the current estimate, while the Actor adjusts its parameters along the policy gradient direction provided by the Critic to maximize the expected Q value. This collaborative mechanism enables the base station to dynamically adjust prices according to real-time load, balancing service revenue and its own operating energy costs while meeting vehicle unloading needs, ultimately achieving the optimal payoff equilibrium under the Stackelberg game.

[0040] Through DDPG's Actor-Critic mechanism, base stations can learn how to adjust prices based on real-time load, achieving a balance between revenue and resource utilization.

[0041] Step 3: Vehicle Distributed Task Offloading Decision Based on Multi-Agent Proximal Policy Optimization (MAPPO) (Lower-Level Follower) After receiving the cost assessment and resource allocation plan from the base station, the vehicle uses the MAPPO algorithm to make follower decisions in a distributed environment.

[0042] like Figure 4 As shown (the environment map at the top of the image is...) Figure 1 (A schematic diagram of an in-vehicle edge computing network system in the embodiment) The working method of this step is as follows: each vehicle acts as an intelligent agent, based on local perception state (Including channel gain, local task margin, and received price), utilize the Actor network to output the task offloading ratio. The vehicle aims to minimize overall overhead, and its reward function comprehensively considers total latency, total energy consumption, and the cost to the base station (see Equation 13). The MAPPO algorithm solves the non-stationary environment problem in multi-vehicle competition scenarios by sharing local states among vehicles and optimizing the advantage function, making the unloading decision more collaborative.

[0043] Step 4: Algorithm Implementation Based on Layered Alternation Mechanism This invention achieves game equilibrium through alternating training of the above algorithm, the specific process of which is as follows: Figure 5 As shown.

[0044] The system first fixes the cost evaluation strategy of the upper-layer DDPG, and then each vehicle agent in the lower layer learns the optimal response strategy at a specific cost level through multiple rounds of MAPPO iteration. Subsequently, the upper-layer base station collects the unloading response feedback from the vehicles, calculates the actual utility, and updates the cost evaluation strategy using the DDPG gradient. Through this iterative process, the upper-layer cost evaluation strategy and the lower-layer unloading actions gradually converge to a stable state, that is, reach the Stackelberg equilibrium point.

[0045] This invention addresses the limitations of current mobile edge computing (MEC) systems, such as conflicts of interest and insufficient priority awareness in dynamic cost assessment and task decision-making, and proposes a systematic solution. This invention simultaneously considers maximizing operator profits and minimizing user computational overhead, introducing task priority constraints to construct a multi-objective optimization problem, and utilizing a hierarchical reinforcement learning framework to decompose it into two related sub-tasks: cost assessment and offloading, for collaborative optimization.

[0046] Specifically, the proposed HSRL algorithm framework optimizes the operator's resource cost assessment strategy and the user's task offloading scheme through interactive training of upper and lower layer models. The upper-layer model uses hierarchical reinforcement learning to perceive the global network state and outputs dynamic resource costs to adjust system load and ensure operator revenue. Based on this, the lower-layer model optimizes the ratio of local execution to edge offloading for each terminal device, combining the currently received price feedback and task priority labels. In terms of method design, the upper-layer algorithm reduces the search difficulty of complex decision space through a hierarchical architecture, while the lower-layer algorithm achieves fine-grained resource matching for heterogeneous task requirements. Through alternating iterative training, the two can converge to the Stackelberg equilibrium point more quickly than traditional single-layer reinforcement learning algorithms, effectively shortening the decision response time.

[0047] Ultimately, while ensuring that high-priority tasks are processed first and user service quality requirements are met, this invention improves the utilization rate of operator resources and effectively reduces the processing latency and economic cost of terminal devices in complex and dynamic network environments, providing efficient and scalable algorithmic support for the market-oriented operation of edge computing.

[0048] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging data between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection.

[0049] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0050] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0051] It should be noted that the user data (including but not limited to user device data, user personal data, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all data and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0052] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0053] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0054] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for making a decision on a task unloading terminal, characterized in that, The terminal decision-making method includes: The task data of the current time slot terminal is input into the multilayer perceptron network to calculate the priority weight of the task, and the priority weight is sent to the mobile edge computing server. The task data includes: task data size, task computing density, remaining time before task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server. Receive the action vector returned by the mobile edge computing server; the action vector includes: computing energy consumption, computing resource unit cost, and time occupation unit cost; Based on the action vector, channel gain, and task margin, the task offloading decision of the terminal is calculated using a multi-agent approximation strategy optimization model; the task offloading decision is the ratio of task execution on the terminal local and on the mobile edge computing server. On the mobile edge computing server, the priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of the terminals within the coverage area, and the overall action vector of the terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the mobile edge computing server in the current time slot. When the number of alternating calculations between the mobile edge computing server and the terminal reaches a preset number, a task unloading decision is executed on the terminal.

2. The task unloading terminal decision method according to claim 1, characterized in that, The task data of the current time slot terminal is input into the multilayer perceptron network to calculate the task priority weights, specifically including: The task data is normalized to obtain normalized data; The normalized data is subjected to a nonlinear transformation using the ReLU activation function to obtain nonlinear data; The nonlinear data is combined in a high dimension using the weight matrix of a multi-layer fully connected structure to obtain the first high-dimensional data. The first high-dimensional data is mapped to priority weights using the Sigmoid activation function.

3. The task unloading terminal decision method according to claim 1, characterized in that, The task data of the current time slot terminal is input into the multilayer perceptron network to calculate the task priority weight. The calculation formula is as follows: ； ； in, For the terminal i In the time slot t Task priority weights For the priority evaluation function of a multilayer perceptron network, For the terminal i Multidimensional task feature vectors For parameters related to the priority evaluation function, For the terminal i In the time slot t Task data size, For the terminal i In the time slot t Task computation density, For the terminal i The maximum tolerable delay for the task. - t This represents the remaining time before the task deadline. For the terminal i In the time slot t The task importance coefficient, This is the terminal type identifier. Mobile edge computing server j In the time slot t The current load, Mobile edge computing server j In the time slot t The queue status.

4. The task unloading terminal decision method according to claim 1, characterized in that, The priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of terminals within the coverage area, and the overall action vector of terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the mobile edge computing server in the current time slot, specifically including: The deep deterministic policy gradient model includes a first Actor network and a first Critic network; The multi-layer fully connected neural network in the first Actor network performs high-dimensional feature extraction on the priority weights, the current computing power load ratio of the mobile edge computing server, the overall priority weights of terminals within the coverage area, and the overall action vectors of terminals within the coverage area of ​​the previous time slot, to obtain the first high-dimensional data. The Sigmoid activation function in the first Actor network linearly maps the first high-dimensional data to a preset interval to obtain an initial action vector; The first Critic network minimizes the error between the initial action vector and the target action vector calculated by the target network to obtain the gradient adjustment direction of the initial action vector; The first Actor network adjusts the parameters of the initial action vector multiple times according to the gradient adjustment direction provided by the first Critic network, and the maximum value of the initial action vector is used as the action vector output.

5. The task unloading terminal decision method according to claim 1, characterized in that, The action vectors of all terminals within the coverage area of ​​the current time-slot mobile edge computing server are calculated using the following formula: ； in: ; ; Represents the action vector function. Representation function Related to C, F, and α, adjusting the values ​​of C and F can make... To achieve the maximum value, C is the cost evaluation strategy variable, F is the resource allocation variable, α is the proportion matrix of tasks offloaded from all terminals to server j, C1, C2, and C3 are constraints, and M is the number of mobile edge computing servers. N is the number of terminals. , As an adjustment coefficient for cost and energy consumption, To terminate in time slot t i Mobile edge computing servers j The cost, Mobile edge computing server j The computational resource cost per unit in time slot t, Mobile edge computing server j The time unit cost of time slot t Mobile edge computing server j Computational energy consumption, It is the energy consumption coefficient of the base station. The preset maximum cost budget, The preset maximum allowable energy consumption, , These are the preset minimum and maximum unit cost of computing resources, respectively. , These are the preset minimum and maximum costs per unit of time occupied. For time slots t terminal i Task offloaded to mobile edge computing server j The proportion, For the terminal i In the time slot t Task data size, For the terminal i The density is calculated in time slot t. For the terminal i Allocate computing power in time slot t Let j be the physical capacity of the mobile edge computing server.

6. The task unloading terminal decision method according to claim 1, characterized in that, Based on the action vector, channel gain, and task margin, the terminal's task offloading decision is calculated using a multi-agent approximation strategy optimization model, specifically including: The multi-agent approximation strategy optimization model includes a second Actor network and a second Critic network. The action vector, channel gain, and task margin are input into the second Actor network to obtain the initial task offloading decision. The second Critic network minimizes the error between the initial task unloading decision and the target task unloading decision calculated by the target network, thereby obtaining the gradient adjustment direction of the initial task unloading decision; The second Actor network adjusts the parameters of the initial task unloading decision multiple times based on the gradient adjustment direction provided by the second Critic network to obtain the terminal's task unloading decision.

7. The task unloading terminal decision method according to claim 1, characterized in that, Based on the action vector, channel gain, and task margin, the task offloading decision of the terminal is calculated using a multi-agent approximation strategy optimization model. The calculation formula is as follows: ； in, The task offloading decision function for the multi-agent approximation policy optimization model. Representation function Related to C, F, and α, adjusting the value of α can make... To achieve the maximum value, C is the cost evaluation strategy variable, F is the resource allocation variable, α is the offloading matrix, C4, C5, and C6 are constraints, and M is the number of mobile edge computing servers. N is the number of terminals. , , , These are latency weight, energy consumption weight, and cost weight, respectively. For the terminal i In the time slot t The time delay, For the terminal i In the time slot t energy consumption For the terminal i In the time slot t The cost and expense, , , These are the preset maximum allowable latency, maximum allowable energy consumption, and maximum cost budget, respectively. For the terminal i In the time slot t Task offloaded to mobile edge computing server j The proportion, For the terminal i Maximum tolerable delay for the task For the terminal i The maximum battery capacity.

8. The task unloading terminal decision method according to claim 7, characterized in that, The terminal i In the time slot t latency and terminal i In the time slot t energy consumption The calculation formula is: ； ； ； ； ； ； ； ； in, For the terminal The delay in executing the task locally, For the terminal Tasks are offloaded to mobile edge computing servers Total delay, including transmission delay and computational delay , For the terminal The energy consumption of tasks executed locally. For the terminal Task offloaded to mobile edge computing server energy consumption For terminal transmission power, For the terminal i Local computing power The effective capacitance coefficient of the terminal CPU. In time slot t terminal i The proportion of tasks executed locally. In time slot t terminal i Tasks are offloaded to mobile edge computing servers j The proportion, For the terminal i In the time slot t Task data size, For the terminal i The task calculates density in time slot t; For the terminal With mobile edge computing servers Instantaneous uplink transmission rate between Sub-channel bandwidth, Indicates channel gain. This represents the power of additive white Gaussian noise.

9. A method for making a decision on a task offloading server, characterized in that, The server decision-making method includes: The priority weight of terminal transmission, the current computing power load ratio of the mobile edge computing server, the overall priority weight of terminals within the coverage area, and the overall action vector of terminals within the coverage area of ​​the previous time slot are input into the deep deterministic policy gradient model to calculate the action vector of all terminals within the coverage area of ​​the current time slot. At the terminal, the task data of the current time slot terminal is input into the multilayer perceptron network to calculate the priority weight of the task, and the priority weight is sent to the mobile edge computing server. The task data includes: task data size, task computing density, remaining time before task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server. The terminal also receives the action vector returned by the mobile edge computing server, and calculates the terminal's task offloading decision based on the action vector, channel gain, and task margin using a multi-agent approximation strategy optimization model; the action vector includes: computing energy consumption, computing resource unit cost, and time unit cost; the task offloading decision is the ratio of task execution on the terminal's local machine and on the mobile edge computing server. When the number of alternating calculations between the mobile edge computing server and the terminal reaches a preset number, a task unloading decision is executed on the terminal.

10. A task unloading decision system, characterized in that, The decision-making system includes: Terminal and mobile edge computing server side; The terminal is configured to input task data into a multilayer perceptron network, calculate the priority weight of the task, and send the priority weight to the mobile edge computing server; receive the action vector returned by the mobile edge computing server, and input the action vector, channel gain, and task margin into a multi-agent approximate policy optimization model to calculate the task offloading decision for the current time slot of the terminal; and execute the task offloading decision on the terminal when the number of alternating calculations on the mobile edge computing server and the terminal reaches a preset number. The task data includes: task data size, task computation density, remaining time before the task deadline, task importance coefficient, terminal type identifier, current load of the mobile edge computing server, and queue status of the mobile edge computing server; the action vector includes: computational energy consumption, unit cost of computational resources, and unit cost of occupied time. The mobile edge computing server is used to input the priority weight transmitted by the terminal, the current computing power load ratio of the mobile edge computing server, the overall priority weight of the terminals within the coverage area, and the overall action vector of the terminals within the coverage area of ​​the previous time slot into the deep deterministic policy gradient model to calculate the action vector of the terminals within the coverage area of ​​the current time slot.

Images