Unloading strategy and pricing strategy joint optimization method and device, electronic equipment, chip, storage medium and computer program product

Abstract

The embodiment of the application provides a kind of unloading strategy and pricing strategy joint optimization method, unloading strategy and pricing strategy joint optimization device, electronic equipment, chip, storage medium and computer program product, the unloading strategy and pricing strategy joint optimization method is applied to edge server, and the edge server and one or more vehicles constitute Stakelberg game model;The edge server is leader;The vehicle is follower;The method comprises: with first utility function as optimization goal, training first deep reinforcement learning model in the edge server, until reaching first preset condition, obtain trained first deep reinforcement learning model;During the training of the first deep reinforcement learning model, according to preset interval, first pricing strategy is sent to each vehicle of the one or more vehicles;The first pricing strategy is the pricing strategy generated in the training process of the first deep reinforcement learning model.

Description

Technical Field

[0001] This application relates to the field of edge computing technology, specifically to a method for joint optimization of offloading and pricing strategies, a device for joint optimization of offloading and pricing strategies, an electronic device, a chip, a storage medium, and a computer program product. Background Technology

[0002] Current research on computational offloading typically assumes that edge server resources can be shared unconditionally, thus only considering the interests of vehicle users during the optimization process. In reality, edge resources, which are fiercely contested by vehicles, are limited. If the computational offloading strategy is optimized solely from the perspective of vehicle users, it will not only compromise the completeness of the computational offloading strategy parameters but also hinder the continuous interaction between vehicles and edge servers, thereby affecting the stability of the vehicle edge computing system. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, chip, storage medium, and computer program product for jointly optimizing unloading and pricing strategies.

[0004] The joint optimization method for offloading and pricing strategies provided in this application is applied to an edge server, where the edge server and one or more vehicles constitute a Stackelberg game model; the edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the offloading strategy; the method includes:

[0005] Using a first utility function as the optimization objective, a first deep reinforcement learning model in the edge server is trained until a first preset condition is met, resulting in a trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles;

[0006] The final pricing strategy is determined using the trained first deep reinforcement learning model, and the final pricing strategy is sent to each of the one or more vehicles; wherein,

[0007] During the training of the first deep reinforcement learning model, a first pricing strategy is sent to each of the one or more vehicles at preset intervals; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model.

[0008] The joint optimization method for unloading and pricing strategies provided in this application is applied to vehicles. An edge server and one or more vehicles constitute a Stackelberg game model, where the edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the unloading strategy; the method includes:

[0009] The system receives a first pricing strategy sent by the edge server; the first pricing strategy is a pricing strategy generated during the training of a first deep reinforcement learning model in the edge server.

[0010] Using a second utility function as the optimization objective, the system interacts with the environment to obtain multiple second empirical data points. Based on these multiple second empirical data points, the second deep reinforcement learning model in the vehicle is iteratively updated until a second preset condition is met, resulting in a trained second deep reinforcement learning model. The second utility function characterizes the vehicle's satisfaction with task unloading. The environment includes the first pricing strategy.

[0011] The final pricing strategy sent by the edge server is received, and the final offloading strategy is determined through the trained second deep reinforcement learning model.

[0012] The unloading and pricing strategy joint optimization apparatus provided in this application embodiment is applied to an edge server, wherein the edge server and one or more vehicles constitute a Stackelberg game model; the edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the unloading strategy; the apparatus includes:

[0013] First training unit: used to train the first deep reinforcement learning model in the edge server with the first utility function as the optimization objective until the first preset condition is met, and obtain the trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles;

[0014] First determining unit: used to determine the final pricing strategy through the trained first deep reinforcement learning model, and send the final pricing strategy to each of the one or more vehicles;

[0015] The first training unit is further configured to send a first pricing strategy to each of the one or more vehicles at preset intervals during the training of the first deep reinforcement learning model; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model.

[0016] The unloading and pricing strategy joint optimization device provided in this application embodiment is applied to vehicles. An edge server and one or more vehicles constitute a Stackelberg game model. The edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the unloading strategy. The device includes:

[0017] The second training unit is used to receive a first pricing strategy sent by the edge server; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model in the edge server; it interacts with the environment using a second utility function as the optimization objective to obtain multiple second empirical data; it iteratively updates the second deep reinforcement learning model in the vehicle based on the multiple second empirical data until a second preset condition is met, thus obtaining the trained second deep reinforcement learning model; the second utility function characterizes the vehicle's satisfaction with task unloading; the environment includes the first pricing strategy.

[0018] The second determining unit is used to receive the final pricing strategy sent by the edge server and determine the final offloading strategy through the trained second deep reinforcement learning model.

[0019] The electronic device provided in this application includes a processor and a memory. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute any of the unloading strategy and pricing strategy joint optimization methods provided in this application.

[0020] The chip provided in this application includes a processor for calling and running a computer program from a memory, causing a device with the chip installed to execute any of the unloading strategy and pricing strategy joint optimization methods provided in this application.

[0021] The storage medium provided in this application embodiment is used to store a computer program, which causes a computer to execute any of the unloading strategy and pricing strategy joint optimization methods provided in this application embodiment.

[0022] The computer program product provided in this application includes a computer program that, when executed by a processor, implements any of the unloading strategy and pricing strategy joint optimization methods provided in this application.

[0023] The joint optimization method for offloading and pricing strategies provided in this application involves an edge server and one or more vehicles forming a Stackelberg game model. The edge server acts as the leader, determining the pricing strategy, while the vehicles act as followers, determining the offloading strategy. The edge server uses its payoff as a first utility function to ensure its own payoff and improve the stability of the vehicle edge computing system. Based on the Stackelberg game model, the first utility function is used as the optimization objective of a first deep reinforcement learning model. The first pricing strategy generated during training is sent to the vehicles, thereby achieving joint optimization of the edge server's offloading and pricing strategies. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 1 ;

[0026] Figure 2 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 2 ;

[0027] Figure 3 This is a schematic diagram of a vehicle edge computing scenario provided in an embodiment of this application;

[0028] Figure 4 A schematic diagram of the joint scheduling structure of the two-layer DRL provided in this application embodiment;

[0029] Figure 5 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 3 ;

[0030] Figure 6 A schematic diagram of the structure of the unloading strategy and pricing strategy joint optimization device 600 provided in the embodiments of this application;

[0031] Figure 7 A schematic diagram of the structure of the unloading strategy and pricing strategy joint optimization device 700 provided in the embodiments of this application;

[0032] Figure 8 A schematic structural diagram of an electronic device provided in the embodiments of this application;

[0033] Figure 9 This is a schematic structural diagram of the chip provided in the embodiments of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] It should be noted that, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, in the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0037] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0038] The Internet of Vehicles (IoV) is a crucial component of Intelligent Transportation Systems (ITS), playing a vital role in smart cities. It is an essential means of ensuring travel safety and a necessary step towards achieving a safe China. Currently, in-vehicle applications such as driving safety, information sharing, and in-vehicle entertainment are emerging in large numbers. These tasks often involve low latency or energy constraints, and relying solely on resource-constrained vehicles makes it difficult to guarantee Quality of Service (QoS), hindering task execution. Mobile Edge Computing (MEC), as a promising paradigm, pushes resources from cloud computing closer to the terminal, using 5G communication to provide low-latency, highly reliable edge services.

[0039] MEC-enabled IoV is essentially a distributed heterogeneous computing system. Proper task offloading and scheduling are crucial to fully leveraging the performance advantages of this computing environment. Therefore, computation offloading technology is key to improving user experience with MEC and is of significant research importance. The offloading decision problem is an optimization problem, requiring solutions to questions such as whether to offload tasks, which tasks to offload, when to offload tasks, and where to offload tasks. Latency and energy consumption are two important metrics in the computation offloading process. Based on these different metrics, current research can be divided into three categories: optimization focused on latency, optimization focused on energy consumption, and optimization that comprehensively considers both latency and energy consumption.

[0040] However, current research on computation offloading typically addresses the issue from the perspective of vehicle terminals, assuming that edge server resources can be shared unconditionally, with the sole aim of protecting vehicle interests. In reality, edge resources, fiercely contested by vehicle terminals, are limited, and service providers seek to profit through edge resource trading. If the interests of service providers are not considered, edge servers will have no incentive to assist end-users with tasks; maximizing profits through trading edge service resources is the essence of their service provision. Optimizing computation offloading strategies solely from the vehicle user's perspective is detrimental to both the completeness of computation offloading strategy parameters and the continuous interaction between vehicles and edge servers, thus affecting the stability of the vehicle edge computing system. Therefore, designing effective incentive mechanisms to encourage edge servers to participate in edge services is imperative. Resource pricing is an effective incentive, ensuring that service providers consider important economic factors. Researching resource pricing-based incentive mechanisms during computation offloading is not only a prerequisite and fundamental guarantee for the stable operation of the Vehicular Edge Computing Network (VECN), but also improves the completeness of parameters affecting computation offloading strategies, which is necessary and crucial for VECN resource management.

[0041] Current research on resource pricing typically employs auction theory to derive pricing strategies, requiring a third party as the auctioneer. However, for VECNs (Vehicle Energy Management Systems), identifying an entity to assume this role is challenging. Furthermore, in academia, describing the interactions during the pricing process using game theory models has attracted considerable attention, with many scholars modeling the interaction process as Stackelberg games. However, in model construction, there is often a focus on task processing latency or only on vehicle energy consumption, with few studies considering the trade-off between latency and energy consumption. Additionally, the maximum tolerable latency varies across different tasks, and this factor should also be fully considered during the modeling process.

[0042] Current research often constructs vehicle utility functions from the perspectives of latency or energy consumption. However, in vehicle edge computing scenarios, the tasks generated by vehicles are heterogeneous. Security tasks have low latency tolerance, while entertainment tasks have relatively high latency tolerance. Therefore, when constructing vehicle utility functions, the latency characteristics of the tasks should also be considered to ensure the realism of the scenario.

[0043] In game theory models, obtaining the Nash equilibrium is a crucial issue. Optimal strategies can be derived using the first derivative of the utility function, a method suitable for scenarios with complete information, where vehicle users are willing to share personal information. However, this violates the principle of protecting user privacy. Utilizing intelligent optimization methods to obtain the equilibrium point is also effective, but it is not suitable for the highly dynamic characteristics of connected vehicles. Reinforcement learning is an important approach for adaptively obtaining optimal strategies; however, the two-layer deep reinforcement learning (DRL) structure poses challenges to the convergence of traditional reinforcement learning. Furthermore, how to avoid DRL getting trapped in local optima during the process of obtaining the Nash equilibrium is also an issue that has not been thoroughly investigated.

[0044] Reinforcement learning is an effective method for solving computation offloading optimization problems, such as deep Q-networks and deep deterministic policy gradients (DDPG). This application utilizes reinforcement learning algorithms to solve the joint optimization problem of computation offloading and resource pricing, constructing a two-layer DRL model to obtain pricing and offloading policies respectively. Achieving Nash equilibrium and avoiding getting trapped in local optima is a challenge for game models in large-scale vehicle-to-everything (V2X) scenarios. Furthermore, convergence is also a concern for two-layer DRL architectures. To better adapt DRL to vehicle edge computing scenarios, this application improves upon traditional DRL by proposing a novel reinforcement learning architecture. Specifically, for the two-layer DRL, an asynchronous parallel structure is adopted in the upper layer to improve the agent's exploration ability and convergence, while a distributed architecture is adopted in the lower layer to ensure the privacy of vehicle users.

[0045] Figure 1 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 1 ,like Figure 1 As shown in the embodiment of this application, a joint optimization method for offloading and pricing strategies is provided and applied to an edge server. The edge server and one or more vehicles constitute a Stackelberg game model. The edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the offloading strategy. The method includes the following steps:

[0046] Step 101: Using the first utility function as the optimization objective, train the first deep reinforcement learning model in the edge server until the first preset condition is met, and obtain the trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles.

[0047] In this embodiment of the application, the first utility function characterizes the revenue of the edge server in performing the unloading task of the one or more vehicles. The revenue is the vehicle payment minus the energy consumption of the edge server in performing the vehicle unloading task, which can be abstracted into the following mathematical formula:

[0048]

[0049] Where, p i For the pricing strategy targeting vehicle i, e is the cost of the edge server executing a unit bit task, and x is the cost of the edge server executing a unit bit task. i For the unloading task of the i-th vehicle, where N is the total number of vehicles, set the price cap for edge server computing resources to p. max In the interaction between the edge server and the vehicle, the edge server's goal is to maximize efficiency by setting a reasonable pricing strategy. Therefore, the optimization problem on the edge server side can be expressed as:

[0050]

[0051] In this embodiment, the edge server acts as an intelligent agent, interacts with the environment, and obtains a series of trajectories, i.e., multiple first experience data, which are used to train the first deep reinforcement learning model.

[0052] In this embodiment of the application, each first experience data includes: a first state, a first action, a first reward, and a second state; the first state is the initial first state space of the first experience data; the first action is the pricing strategy issued by the edge server; the first reward can be obtained through a first utility function; the second state is the first state space after applying the first action based on the first state.

[0053] In this embodiment of the application, the first state space includes: environmental information, historical pricing strategies, and historical unloading strategies of each vehicle in one or more vehicles.

[0054] Based on this, in an optional embodiment of this application, the step of training a first deep reinforcement learning model in the edge server using the utility function of the edge server as the optimization objective until a first preset condition is met, to obtain the trained first deep reinforcement learning model, includes:

[0055] Using the utility function of the edge server as the optimization objective, the system interacts with the environment to obtain multiple first empirical data points; based on these first empirical data points, a first deep reinforcement learning model is trained until a first preset condition is met, resulting in the trained first deep reinforcement learning model; wherein...

[0056] Each of the plurality of first experience data includes: a first state, a first action, a first reward, and a second state; wherein,

[0057] The first state is the initial first state space of the first experience data; the first action is a pricing strategy; the first reward is obtained through the first utility function; the second state is the first state space after the first action is applied; the first state space includes: environmental information, historical pricing strategies, and historical unloading strategies of each vehicle in one or more vehicles.

[0058] In this embodiment, the historical pricing strategy can adopt the pricing strategy of the first L time slots of the edge server; the historical unloading strategy of the vehicle can adopt the unloading strategy of the first L time slots, where L is a preset value.

[0059] In this embodiment of the application, the environmental information includes one or more of the following: the vehicle's location, the vehicle's speed, the vehicle's computing power, the location of the edge server, the computing power of the edge server, etc.

[0060] In this embodiment of the application, in order to enhance the exploration ability of the intelligent agent and fully explore the action space, the first deep reinforcement learning model adopts the Actor-Critic algorithm framework, designs an asynchronous parallel deep deterministic policy gradient algorithm, utilizes the multi-threaded structure of the edge server, and each thread interacts with the environment copy as a worker and asynchronously iteratively updates the public network.

[0061] Based on this, in an optional embodiment of this application, the first deep reinforcement learning model adopts the Actor-Critic algorithm framework; the step of using the utility function of the edge server as the optimization objective, interacting with the environment to obtain multiple first empirical data; and training the first deep reinforcement learning model based on the multiple first empirical data until a first preset condition is met to obtain the trained first deep reinforcement learning model includes:

[0062] Multiple thread networks are deployed using the multi-CPU threads of the edge server; the multiple thread networks include a public network and multiple worker networks.

[0063] The multiple worker networks interact with multiple environment replicas to obtain multiple first experience data; wherein, the multiple worker networks correspond one-to-one with the multiple environment replicas, and the environment replicas are replicas of the environment;

[0064] Based on the multiple first empirical data, the public network is asynchronously iteratively updated through the multiple worker networks until the first preset condition is met, resulting in an updated public network; the updated public network is the first deep reinforcement learning model after training.

[0065] In this embodiment of the application, in order to facilitate the updating of the public network by the thread network, the thread network and the public network have the same network structure, that is, the network structure includes: an online actor network, a target actor network, an online critic network and a target critic network.

[0066] In this embodiment, a worker network with a single thread is used as an example for detailed description. The actor network parameters of the public network are: and The network parameters for critic are: and The actor network parameters of the thread network are and During the execution phase, thread i interacts with a copy of its environment, determining the action to be taken (a) by inputting the state space into an online actor network. t After thread i performs an action on the environment, it receives an immediate reward r. t And the next state s t+1 The network model is updated using n-step rewards; that is, thread i acquires its trajectory (s) through interaction with the environment. t ,a t ,r t ,s t+1 ,...s t+n-1 ,a t+n-1 ,r t+n-1 ,s t+n The interaction trajectories are stored in a shared experience pool. During the training phase, thread i randomly samples a batch of trajectories from the experience pool and inputs them into the critic network, updating both the critic and actor networks. Subsequently, the shared network is updated using the update gradients of the actor and critic networks. After the shared network is updated, thread i downloads the shared network parameters and updates its own network. Each thread updates the shared network as soon as it obtains the update gradient, thus ensuring that updates between threads are asynchronous.

[0067] In this embodiment of the application, the first preset condition includes any one of the following conditions: the first deep reinforcement learning model converges, or a preset number of training rounds is reached.

[0068] Step 102: Determine the final pricing strategy using the trained first deep reinforcement learning model, and send the final pricing strategy to each of the one or more vehicles; wherein, during the training of the first deep reinforcement learning model, the first pricing strategy is sent to each of the one or more vehicles at preset intervals; the first pricing strategy is the pricing strategy generated during the training of the first deep reinforcement learning model.

[0069] In this embodiment, the trained first deep reinforcement learning model interacts with the current environment to determine the final pricing strategy, and sends the final pricing strategy to each vehicle for the vehicle to make an unloading strategy decision.

[0070] In this embodiment, the pricing strategy can be a uniform pricing strategy or a discriminatory pricing strategy; this embodiment does not limit the specific strategy. It is understood that a uniform pricing strategy means all vehicles are priced the same, while a discriminatory pricing strategy means different vehicles may be priced differently.

[0071] In this embodiment of the application, during the training process of the first deep reinforcement learning model, the pricing strategy generated during the training process is sent to each vehicle at a preset interval, so that the vehicle can complete the training of the deep reinforcement learning model on the vehicle side.

[0072] In this embodiment of the application, the preset interval can be a preset number of rounds. After each preset number of rounds, the public network interacts with the environment to generate a first pricing strategy and sends the pricing strategy to the vehicle.

[0073] In this embodiment, the edge server and one or more vehicles constitute a Stackelberg game model; the edge server is the leader and the vehicles are the followers. The interaction between the edge server and the vehicles can be divided into two phases. In the first phase, the edge server determines the price, and in the subsequent second phase, the vehicles determine the unloading strategy.

[0074] Figure 2 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 2 ,like Figure 2 As shown in the embodiment of this application, a joint optimization method for offloading and pricing strategies is provided, applied to vehicles. An edge server and one or more vehicles constitute a Stackelberg game model, where the edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the offloading strategy; the method includes the following steps:

[0075] Step 201: Receive the first pricing strategy sent by the edge server; the first pricing strategy is the pricing strategy generated during the training of the first deep reinforcement learning model in the edge server.

[0076] In this embodiment of the application, the edge server and one or more vehicles constitute a Stackelberg game model; the edge server is the leader, the vehicles are the followers, and the vehicles receive a first pricing strategy as part of the environment.

[0077] Step 202: Using the second utility function as the optimization objective, interact with the environment to obtain multiple second empirical data; iteratively update the second deep reinforcement learning model in the vehicle based on the multiple second empirical data until the second preset condition is met, to obtain the trained second deep reinforcement learning model; the second utility function represents the vehicle's satisfaction with task unloading; the environment includes the first pricing strategy.

[0078] In this embodiment, the second utility function is the vehicle's utility function, which reflects the user satisfaction resulting from task unloading. During task unloading, latency and energy consumption are important indicators affecting user experience. Therefore, the first information for determining the second utility function in this embodiment includes: the latency saved by the vehicle after adopting the unloading strategy and the energy saved by the vehicle after adopting the unloading strategy. Furthermore, in this embodiment, to characterize the vehicle's sensitivity to latency, the vehicle's satisfaction with acquiring computing resources is also included as an influencing factor of the second utility function. To meet latency constraints and achieve a high level of satisfaction with acquiring computing resources, the vehicle needs to unload more tasks. In other words, for vehicles with high latency sensitivity, they are encouraged to acquire more computing resources. The first information in this embodiment also includes: the vehicle's satisfaction with acquiring computing resources.

[0079] Based on this, in an optional embodiment of this application, the second utility function is determined by first information; the first information includes: the vehicle's satisfaction with acquiring computing resources, the latency saved by the vehicle after adopting the unloading strategy, and the energy saved by the vehicle after adopting the unloading strategy.

[0080] refer to Figure 3 , Figure 3 This is a schematic diagram of a vehicle edge computing scenario provided in an embodiment of this application, such as... Figure 3 As shown, on a one-way road, there are M edge servers on one side of the road, and N vehicles traveling at a fixed speed v. i The set of driving, vehicles, and edge servers is represented as A period of time can be divided into equal time slots, defined as τ = {1, 2, ..., t, ..., T}. Assuming each vehicle generates only one computationally intensive, latency-sensitive task in each time slot, the task generated by vehicle i in time slot t is... Where D i c represents the size of the input data. i C represents the number of CPU cycles required to complete a task of one bit. i,t This indicates the number of CPU cycles required to complete the task. This represents the maximum tolerable latency for the task. A task can be arbitrarily divided into two parts, with one part executed locally and the other offloaded to an edge server for processing, leveraging the edge server's resources to reduce execution latency. Specifically, define task T... i The unloading ratio is ρ i According to the unloading strategy, the vehicle offloads this part of the task to the corresponding edge server for execution. In this embodiment, the unloading strategy can be characterized by the unloading ratio. Furthermore, considering that both the vehicle and the edge server are heterogeneous, their computing capabilities are respectively... It's important to note that if the vehicle leaves the edge server's coverage area after the unloading task is completed, the results will be transmitted between the edge servers. Compared to the size of the input data, the amount of task results is very small, and the edge servers have high transmission power, so the result transmission latency will be negligible.

[0081] In this embodiment, a communication model between vehicles and edge servers is established. When a task vehicle unloads its task to a fixed edge node, it needs to communicate with the Road Side Unit (RSU). In a multi-node collaborative computational unloading scenario, assuming n vehicles unload their tasks to the MECm connected to the RSU, the upload data rates between task vehicle i and fixed edge server m are as follows:

[0082]

[0083] Among them, w i It is the upload power of mission vehicle i, B V2I It is the communication bandwidth of the RSU. σ is the channel gain of mission vehicle i and fixed edge server m. 2 It is noise power.

[0084] The calculation model for task unloading is as follows: when part of task x... i During unloading, the execution latency and energy consumption of local task processing can be expressed as follows: and Here, κ represents energy conversion efficiency, which is related to the vehicle's processor architecture. When part of the task x... i During unloading, the unloading task can be divided into two phases: the task transmission phase and the task execution phase. Therefore, the task processing latency includes transmission latency and execution latency, with transmission latency denoted as... Execution delay is represented as The total latency of task unloading processing is expressed as The energy consumed by the vehicle is transmission energy consumption, expressed as Among them, w iLet be the vehicle's transmission power. The total processing latency is the maximum of the task's local processing latency and the total task offloading latency, which can be expressed as: The total energy consumption for task processing is the sum of the energy consumed by local task processing and the energy consumed by task offloading and transmission, which can be expressed as:

[0085] In this embodiment of the application, the second utility function can be designed as follows:

[0086]

[0087] in, It refers to the level of satisfaction with the vehicle's access to computing resources. Re represents the proportion of the task unloaded by vehicle i to the total unloaded task by all vehicles, and Re is the fixed reward for the vehicle to obtain computing resources. i It is a variable reward factor; This represents the maximum tolerable delay of vehicle i in time slot t; the smaller the maximum tolerable delay of vehicle i, the lower the re delay. i The larger the latency, the more tasks the vehicle needs to offload in order to meet latency constraints and achieve high satisfaction. In other words, re i By giving greater rewards to vehicles that are highly sensitive to latency, they are encouraged to access more computing resources;

[0088] This refers to the time saved by the vehicle adopting an unloading strategy, i.e., partial task x. i The latency of local processing is reduced by the amount of tasks x. i Delay in unloading process; This refers to the energy saved by the vehicle after adopting an unloading strategy, i.e., partial task x. i Energy consumption processed locally minus part of the task x i Transmission energy consumption during unloading processing; p i x i b1 represents the expenditure on acquiring computing resources for the vehicle; b1, b2, b3, and b4 are weighting factors.

[0089] Based on this, in an optional embodiment of this application, the satisfaction level of the vehicle in acquiring computing resources is determined by second information; the second information includes: the proportion of the task unloaded by the vehicle to the total task unloaded by all vehicles, a first reward factor, and a second reward factor; the first reward factor is a fixed reward for the vehicle acquiring computing resources, and the second reward factor is a variable reward factor; wherein...

[0090] The smaller the maximum tolerable delay of the vehicle's task, the larger the value of the second reward factor.

[0091] In this embodiment of the application, during the calculation of unloading, the vehicle's objective is to maximize utility; therefore, the optimization problem on the vehicle side can be expressed as:

[0092]

[0093] Constraint C1 ensures that the task completion delay is within the tolerable delay range. If the task delay exceeds the tolerable delay, the task is considered to have failed, and a penalty ξ_p is imposed. Constraint C2 ensures that the task unloading amount does not exceed the task's input data amount; therefore, the unloading ratio ρ is... i The valid space is [0,1]. Constraint C3 states that the energy consumed in performing the task cannot exceed the vehicle's energy. Constraint C4 restricts the vehicle's upload power from exceeding its maximum transmission power.

[0094] In this embodiment, the vehicle acts as an intelligent agent, interacting with the environment to obtain a series of trajectories, i.e., multiple second empirical data, which are used to train a second deep reinforcement learning model.

[0095] In this embodiment of the application, each second experience data includes: a third state, a second action, a second reward, and a fourth state; the third state is the initial second state space of the second experience data; the second action is the vehicle unloading strategy; the second reward can be obtained through a second utility function; the fourth state is the second state space after applying the second action based on the third state, and the second state space includes: vehicle-related information, historical unloading strategies of each vehicle in one or more vehicles, historical pricing strategies, the current first pricing strategy, and the location of the edge server.

[0096] Based on this, in an optional embodiment of this application, each piece of second experience data in the plurality of second experience data includes: a third state, a second action, a second reward, and a fourth state; the third state is the initial second state space of the second experience data; the second action is an unloading strategy; the second reward is obtained through the second utility function; the fourth state is the second state space after the second action is applied; the second state space includes: vehicle-related information, historical unloading strategies of each vehicle in one or more vehicles, historical pricing strategies, the current first pricing strategy, and the location of the edge server.

[0097] In this embodiment of the application, each time the vehicle receives the first pricing strategy, it interacts with the environment once and generates a second empirical data.

[0098] In this embodiment, to enhance the agent's exploration capabilities and fully explore the action space, the second deep reinforcement learning model employs the Actor-Critic algorithm framework with a distributed architecture. Each vehicle is treated as an agent, and a Markov game is defined for the N agents using a state space, action space, and reward function. The state space includes: vehicle-related information, historical unloading strategy information for all vehicles, pricing information for edge servers, and the location of the edge servers. The unloading ratio ρ of the vehicle task is used as the basis for the evaluation. i This represents the vehicle's unloading strategy. Specifically, the state obtained from the environment includes: the edge server's pricing for vehicle i in the current time slot, the edge server's historical pricing strategy for all vehicles over the previous L time slots, and the vehicle task T. i Relevant information includes the historical unloading strategies of other vehicles in the previous L time slots, the vehicle's location, speed, and the location of the edge server. The task unloading ratio of each vehicle is used as the agent's action; the vehicle's action space is continuous, and the valid action space is [0,1]. A distributed structure is adopted during the vehicle's unloading strategy formulation process. Each vehicle receives a reward based on its chosen unloading strategy, and the reward function is defined as the vehicle utility function.

[0099] In this embodiment, each agent has two actor networks and two critic networks. The actor networks include an online network μ and a target network μ′, with corresponding parameters θ. μ and θ μ′ The two networks have the same structure, a design that reduces neural network oscillations and achieves stable convergence. During action generation, an ε-greedy strategy is adopted to better balance exploration and exploitation. After the online actor network outputs an action, the agent interacts with the environment to obtain the action and immediate reward for the next time slot, and stores the trajectory in an experience cache for use in subsequent training phases. This method breaks the dependencies between sampled data. During centralized offline training, in addition to its own information, the agent can also obtain relevant information from other agents, such as their state and actions. It should be noted that the information from other agents does not involve the strategies used during online task execution, indicating that privacy and security issues are not involved. The i-th trajectory of the agent can be represented as (s... i ,a i ,r i ,s i+1 ),in Represents the state of all agents. Represents the actions of all agents, and similarly, This represents the state of all agents in the next time slot. When updating the network, s i and a iAs input to the Q-network, it outputs the current agent's state-action value function through a deep neural network (DNN).

[0100] In this embodiment of the application, the second preset condition includes any one of the following conditions: the second deep reinforcement learning model converges, or a preset number of training rounds is reached.

[0101] Step 203: Receive the final pricing strategy sent by the edge server, and determine the final offloading strategy through the trained second deep reinforcement learning model.

[0102] In this embodiment of the application, after the second deep reinforcement learning model has been trained, the final unloading strategy is determined according to the final pricing strategy determined by the edge server.

[0103] refer to Figure 4 , Figure 4 This is a schematic diagram of the joint scheduling structure of the two-layer DRL provided in the embodiments of this application, as shown below. Figure 4 As shown, the upper-layer DRL is the network on the edge server side, employing an asynchronous parallel structure. Multiple networks are deployed using the multi-CPU threads of the edge server, including a public network and multiple worker networks represented by parallel threads. These thread networks operate independently, interacting with their respective environment replicas. It's important to note that the environment replicas are copies of the real environment, and the initial states of all environment replicas are identical. Multiple threads asynchronously update the public network. Every preset number of rounds after an update, the public network sends the first pricing strategy to the vehicles as part of their state space. To facilitate updates from the thread networks to the public network, the thread networks and the public network share the same network structure: an online actor network and a destination network. The system consists of an actor network, an online critic network, and a target critic network. The lower layer is the vehicle-side network, employing a distributed architecture where each vehicle acts independently as an agent. A Markov game is defined for the N agents using a state space, action space, and reward function. The interaction between the vehicles and the edge server is described as a Stackelberg game process, where the vehicle is defined as a follower and the edge server as the leader. The interaction between the edge server and the vehicle can be divided into two phases: in the first phase, the edge server determines its pricing strategy, and in the subsequent second phase, the vehicle determines its unloading strategy. Backward induction can be used to prove that the Nash equilibrium exists and is unique for the edge server under both uniform and discriminatory pricing models.

[0104] refer to Figure 5 , Figure 5 A schematic diagram of the implementation process of the joint optimization method for uninstallation strategy and pricing strategy provided in the embodiments of this application. Figure 3 ,like Figure 5As shown, in this embodiment, the joint optimization method of unloading strategy and pricing strategy includes the following steps:

[0105] Step 501: The vehicle terminal generates a computationally intensive, latency-sensitive task and sends a resource request to the edge server.

[0106] The vehicle generates computationally intensive and latency-sensitive tasks, sends resource requests to the edge server, and requests to obtain resources on the edge server to complete the task computation.

[0107] Step 502: Scenario Construction. Establish a vehicle edge computing scenario where vehicles and edge servers interact. This scenario includes scenario construction, communication model, computing model, and Stackelberg game model.

[0108] The vehicle edge computing scenario that establishes interaction between the vehicle and the edge server includes: scenario construction, communication model, computing model, and Stackelberg game model. For related content, please refer to the relevant descriptions in the foregoing embodiments for understanding, and will not be repeated here.

[0109] Step 503: The edge server determines the optimized resource pricing strategy based on the pricing model, namely, uniform pricing or discriminatory pricing model.

[0110] The pricing strategy for edge servers can be either uniform pricing or discriminatory pricing. The pricing strategy is determined by a first deep reinforcement learning model after training.

[0111] Step 504: The edge server broadcasts the determined price to the vehicle users.

[0112] The edge server sends the determined pricing strategy to vehicle users via broadcast.

[0113] Step 505: The vehicle terminal optimizes the unloading strategy based on resource prices to determine the unloading volume.

[0114] The vehicle determines its unloading strategy based on the pricing strategy sent by the edge server, using a trained second reinforcement learning model.

[0115] Step 506: The vehicle terminal, according to the unloading strategy, unloads the corresponding volume of tasks to the edge server for execution.

[0116] According to its own unloading strategy, the vehicle unloads the corresponding amount of tasks and the edge server completes the task calculation, realizing edge computing of vehicle tasks.

[0117] The joint optimization method for offloading and pricing strategies provided in this application constructs a two-layer reinforcement learning architecture. The utility function of the edge server and the utility function of the vehicle are used as the optimization objectives of the upper and lower layers of reinforcement learning, respectively, to obtain the pricing and offloading strategies. Asynchronous parallel and distributed architectures are designed in the two deep reinforcement learning models, which not only enhances the agent's exploration capabilities and ensures the convergence of the two-layer network, but also protects the privacy of vehicle users. During the construction of the vehicle utility function, a variable reward factor is defined, and its impact on the task tolerance latency on vehicle satisfaction is formalized.

[0118] This application proposes a computational offloading scheme based on Stackelberg game theory, aiming to maintain continuous interaction between the edge layer and the terminal layer and ensure the stability of the VECN by designing incentives based on resource pricing. The interaction between vehicles and edge servers is described as a Stackelberg game process, where vehicle terminals are defined as followers and edge servers as leaders. To maximize the utility of all entities, utility functions are defined for vehicles and edge servers respectively. The vehicle utility function comprehensively considers latency, energy consumption, task urgency, and resource acquisition expenditure. Using backward induction, it is proved that the Nash equilibrium of the edge server exists and is unique under both uniform pricing and discriminatory pricing models. A dynamically iterative game scheme and a multi-agent DRL algorithm are proposed to obtain optimization strategies, thereby ensuring the stability of the VECN.

[0119] This application embodiment also provides a joint optimization device 600 for unloading strategy and pricing strategy, see reference. Figure 6 , Figure 6 This is a schematic diagram of the structure of the joint optimization device for offloading and pricing strategies provided in an embodiment of this application. The joint optimization device for offloading and pricing strategies in this embodiment is applied to an edge server, which forms a Stackelberg game model with one or more vehicles. The edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the offloading strategy; the joint optimization device 600 for offloading and pricing strategies includes:

[0120] First training unit: used to train the first deep reinforcement learning model in the edge server with the first utility function as the optimization objective until the first preset condition is met, and obtain the trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles;

[0121] First determining unit: used to determine the final pricing strategy through the trained first deep reinforcement learning model, and send the final pricing strategy to each of the one or more vehicles;

[0122] The first training unit is further configured to send a first pricing strategy to each of the one or more vehicles at preset intervals during the training of the first deep reinforcement learning model; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model.

[0123] In this embodiment, the first training unit is configured to: interact with the environment using the utility function of the edge server as the optimization objective to obtain multiple first experience data; train a first deep reinforcement learning model based on the multiple first experience data until a first preset condition is met, thereby obtaining the trained first deep reinforcement learning model; wherein each of the multiple first experience data includes: a first state, a first action, a first reward, and a second state; wherein the first state is the initial first state space of the first experience data; the first action is a pricing strategy; the first reward is obtained through the first utility function; the second state is the first state space after applying the first action; the first state space includes: environmental information, historical pricing strategies, and historical unloading strategies of each vehicle in one or more vehicles.

[0124] In this embodiment, the first deep reinforcement learning model adopts the Actor-Critic algorithm framework; the first training unit is used to deploy multiple thread networks through the multi-CPU threads of the edge server; the multiple thread networks include a public network and multiple worker networks; the multiple worker networks interact with multiple environment replicas respectively to obtain multiple first experience data; wherein, the multiple worker networks correspond one-to-one with the multiple environment replicas, and the environment replicas are replicas of the environment; based on the multiple first experience data, the public network is asynchronously iteratively updated through the multiple worker networks until a first preset condition is reached to obtain an updated public network; the updated public network is the trained first deep reinforcement learning model.

[0125] Those skilled in the art should understand that Figure 6 The implementation functions of each unit in the unloading strategy and pricing strategy joint optimization device 600 shown can be understood with reference to the relevant description of the aforementioned method. Figure 6 The functions of each unit in the unloading strategy and pricing strategy joint optimization device 600 shown can be implemented by a program running on a processor or by specific logic circuits.

[0126] This application embodiment also provides a joint optimization device 700 for unloading strategy and pricing strategy, see reference. Figure 7 , Figure 7This is a schematic diagram of the structure of the joint optimization device for offloading and pricing strategies provided in an embodiment of this application. In this embodiment, the joint optimization device for offloading and pricing strategies is applied to vehicles. An edge server and one or more vehicles constitute a Stackelberg game model. The edge server is the leader, used to determine the pricing strategy; the vehicles are the followers, used to determine the offloading strategy. The joint optimization device 700 for offloading and pricing strategies includes:

[0127] The second training unit is used to receive a first pricing strategy sent by the edge server; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model in the edge server; it interacts with the environment using a second utility function as the optimization objective to obtain multiple second empirical data; it iteratively updates the second deep reinforcement learning model in the vehicle based on the multiple second empirical data until a second preset condition is met, thus obtaining the trained second deep reinforcement learning model; the second utility function characterizes the vehicle's satisfaction with task unloading; the environment includes the first pricing strategy.

[0128] The second determining unit is used to receive the final pricing strategy sent by the edge server and determine the final offloading strategy through the trained second deep reinforcement learning model.

[0129] In this embodiment of the application, each piece of second experience data in the plurality of second experience data includes: a third state, a second action, a second reward, and a fourth state; the third state is the initial second state space of the second experience data; the second action is an unloading strategy; the second reward is obtained through the second utility function; the fourth state is the second state space after the second action is applied; the second state space includes: vehicle-related information, historical unloading strategies of each of one or more vehicles, historical pricing strategies, the current first pricing strategy, and the location of the edge server.

[0130] In this embodiment of the application, the second utility function is determined by first information; the first information includes: the vehicle's satisfaction with acquiring computing resources, the latency saved by the vehicle after adopting the unloading strategy, and the energy saved by the vehicle after adopting the unloading strategy.

[0131] In this embodiment of the application, the satisfaction level of the vehicle in obtaining computing resources is determined by a second piece of information; the second piece of information includes: the proportion of the task unloaded by the vehicle to the total task unloaded by all vehicles, a first reward factor, and a second reward factor; the first reward factor is a fixed reward for the vehicle to obtain computing resources, and the second reward factor is a variable reward factor; wherein, the smaller the maximum tolerable latency of the vehicle's task, the larger the value of the second reward factor.

[0132] Those skilled in the art should understand that Figure 7 The implementation functions of each unit in the unloading strategy and pricing strategy joint optimization device 700 shown can be understood with reference to the relevant description of the aforementioned method. Figure 7 The functions of each unit in the unloading strategy and pricing strategy joint optimization device 700 shown can be implemented by a program running on a processor or by specific logic circuits.

[0133] Figure 8 This is a schematic structural diagram of an electronic device 800 provided in an embodiment of this application. Figure 8 The illustrated electronic device 800 includes a processor 810, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0134] Optionally, such as Figure 8 As shown, the electronic device 800 may further include a memory 820. The processor 810 can retrieve and run computer programs from the memory 820 to implement the methods described in the embodiments of this application.

[0135] The memory 820 can be a separate device independent of the processor 810, or it can be integrated into the processor 810.

[0136] Optionally, such as Figure 8 As shown, the electronic device 800 may also include a transceiver 830, which the processor 810 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0137] The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include an antenna, and the number of antennas may be one or more.

[0138] The electronic device 800 may specifically be the unloading strategy and pricing strategy joint optimization device 600 / unloading strategy and pricing strategy joint optimization device 700 in the embodiments of this application. The electronic device 800 can implement the corresponding processes implemented by the unloading strategy and pricing strategy joint optimization device 600 / unloading strategy and pricing strategy joint optimization device 700 in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0139] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by the processor 810 of a communication device 800 to perform the steps described in any of the foregoing methods.

[0140] Figure 9 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 9The chip 900 shown includes a processor 910, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0141] Optionally, such as Figure 9 As shown, chip 900 may further include memory 920. Processor 910 can retrieve and run computer programs from memory 920 to implement the methods described in this embodiment.

[0142] The memory 920 can be a separate device independent of the processor 910, or it can be integrated into the processor 910.

[0143] Optionally, the chip 900 may also include an input interface 930. The processor 910 can control the input interface 930 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0144] Optionally, the chip 900 may also include an output interface 940. The processor 910 can control the output interface 940 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0145] The chip can be applied to the electronic device 800 in the embodiments of this application, and the chip can implement the corresponding processes implemented by the electronic device 800 in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0146] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0147] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0148] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0149] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] This application also provides a storage medium for storing a computer program. This storage medium can be applied to the electronic device 800 in this application embodiment, and the computer program causes the computer to execute the corresponding processes implemented by the electronic device 800 in the various methods of this application embodiment; for brevity, further details are omitted here.

[0151] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0156] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or electronic device 800, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for jointly optimizing unloading and pricing strategies, characterized in that, Applied to edge servers, where the edge servers and one or more vehicles constitute a Stackelberg game model; The edge server acts as the leader, used to determine the pricing strategy; The vehicle is a follower used to determine the unloading strategy; the method includes: Using a first utility function as the optimization objective, a first deep reinforcement learning model in the edge server is trained until a first preset condition is met, resulting in a trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles; The final pricing strategy is determined using the trained first deep reinforcement learning model, and the final pricing strategy is sent to each of the one or more vehicles; wherein, During the training of the first deep reinforcement learning model, a first pricing strategy is sent to each of the one or more vehicles at preset intervals; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model.

2. The method according to claim 1, characterized in that, The step of training a first deep reinforcement learning model in the edge server using the utility function of the edge server as the optimization objective until a first preset condition is met, to obtain the trained first deep reinforcement learning model, includes: Using the utility function of the edge server as the optimization objective, the system interacts with the environment to obtain multiple first empirical data points; based on these first empirical data points, a first deep reinforcement learning model is trained until a first preset condition is met, resulting in the trained first deep reinforcement learning model; wherein... Each of the plurality of first experience data includes: a first state, a first action, a first reward, and a second state; wherein, The first state is the initial first state space of the first experience data; the first action is a pricing strategy; the first reward is obtained through the first utility function; the second state is the first state space after the first action is applied; the first state space includes: environmental information, historical pricing strategies, and historical unloading strategies of each vehicle in one or more vehicles.

3. The method according to claim 2, characterized in that, The first deep reinforcement learning model adopts the actor-critic algorithm framework; the utility function of the edge server is used as the optimization objective, and it interacts with the environment to obtain multiple first empirical data. A first deep reinforcement learning model is trained based on the aforementioned multiple first empirical data until a first preset condition is met, resulting in the trained first deep reinforcement learning model, including: Multiple thread networks are deployed using the multi-CPU threads of the edge server; the multiple thread networks include a public network and multiple worker networks. The multiple worker networks interact with multiple environment replicas to obtain multiple first experience data; wherein, the multiple worker networks correspond one-to-one with the multiple environment replicas, and the environment replicas are replicas of the environment; Based on the multiple first empirical data, the public network is asynchronously iteratively updated through the multiple worker networks until the first preset condition is met, resulting in an updated public network; the updated public network is the first deep reinforcement learning model after training.

4. A method for jointly optimizing unloading and pricing strategies, characterized in that, Applied to vehicles, an edge server and one or more vehicles form a Stackelberg game model, with the edge server acting as the leader to determine pricing strategies. The vehicle is a follower used to determine the unloading strategy; the method includes: The system receives a first pricing strategy sent by the edge server; the first pricing strategy is a pricing strategy generated during the training of a first deep reinforcement learning model in the edge server. Using a second utility function as the optimization objective, the system interacts with the environment to obtain multiple second empirical data points. Based on these multiple second empirical data points, the second deep reinforcement learning model in the vehicle is iteratively updated until a second preset condition is met, resulting in a trained second deep reinforcement learning model. The second utility function characterizes the vehicle's satisfaction with task unloading. The environment includes the first pricing strategy. The final pricing strategy sent by the edge server is received, and the final offloading strategy is determined through the trained second deep reinforcement learning model.

5. The method according to claim 4, characterized in that, Each piece of second experience data in the plurality of second experience data includes: a third state, a second action, a second reward, and a fourth state; the third state is the initial second state space of the second experience data; the second action is an unloading strategy; the second reward is obtained through the second utility function; the fourth state is the second state space after the second action is applied; the second state space includes: vehicle-related information, historical unloading strategies of each vehicle in one or more vehicles, historical pricing strategies, the current first pricing strategy, and the location of the edge server.

6. The method according to claim 4 or 5, characterized in that, The second utility function is determined by the first information; the first information includes: the vehicle's satisfaction with acquiring computing resources, the latency saved by the vehicle after adopting the offloading strategy, and the energy saved by the vehicle after adopting the offloading strategy.

7. The method according to claim 6, characterized in that, The satisfaction level of the vehicle in acquiring computing resources is determined by a second piece of information; the second piece of information includes: the proportion of the task unloaded by the vehicle to the total task unloaded by all vehicles, a first reward factor, and a second reward factor; the first reward factor is a fixed reward for the vehicle acquiring computing resources, and the second reward factor is a variable reward factor; wherein... The smaller the maximum tolerable delay of the vehicle's task, the larger the value of the second reward factor.

8. A device for jointly optimizing unloading and pricing strategies, characterized in that, Applied to edge servers, where the edge servers and one or more vehicles constitute a Stackelberg game model; The edge server acts as the leader, used to determine the pricing strategy; The vehicle is a follower used to determine the unloading strategy; the device includes: First training unit: used to train the first deep reinforcement learning model in the edge server with the first utility function as the optimization objective until the first preset condition is met, and obtain the trained first deep reinforcement learning model; the first utility function represents the benefit of the edge server in performing the unloading task of the one or more vehicles; First determining unit: used to determine the final pricing strategy through the trained first deep reinforcement learning model, and send the final pricing strategy to each of the one or more vehicles; The first training unit is further configured to send a first pricing strategy to each of the one or more vehicles at preset intervals during the training of the first deep reinforcement learning model; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model.

9. A device for jointly optimizing unloading and pricing strategies, characterized in that, Applied to vehicles, an edge server and one or more vehicles form a Stackelberg game model, with the edge server acting as the leader to determine pricing strategies. The vehicle is a follower used to determine the unloading strategy; the device includes: The second training unit is used to receive a first pricing strategy sent by the edge server; the first pricing strategy is a pricing strategy generated during the training of the first deep reinforcement learning model in the edge server; it interacts with the environment using a second utility function as the optimization objective to obtain multiple second empirical data; it iteratively updates the second deep reinforcement learning model in the vehicle based on the multiple second empirical data until a second preset condition is met, thus obtaining the trained second deep reinforcement learning model; the second utility function characterizes the vehicle's satisfaction with task unloading; the environment includes the first pricing strategy. The second determining unit is used to receive the final pricing strategy sent by the edge server and determine the final offloading strategy through the trained second deep reinforcement learning model.

10. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 1 to 3, or the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 4 to 7.

11. A chip, characterized in that, include: A processor is configured to retrieve and run a computer program from memory, causing a device equipped with the chip to perform the joint optimization method of the unloading strategy and pricing strategy as described in any one of claims 1 to 3, or the joint optimization method of the unloading strategy and pricing strategy as described in any one of claims 4 to 7.

12. A storage medium, characterized in that, Used to store a computer program that causes a computer to execute the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 1 to 3, or the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 4 to 7.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 1 to 3, or the joint optimization method of unloading strategy and pricing strategy as described in any one of claims 4 to 7.

Images