Vehicle-mounted semantic communication task unloading method based on multi-agent game

Abstract

The invention provides a vehicle-mounted semantic communication task unloading method based on a multi-agent game, and the method comprises the steps: constructing a multi-leader-single follower Stackelber game model for semantic communication task resource transaction between a vehicle and edge calculation, designing an optimal resource pricing strategy and a distribution excitation strategy under multiple constraint conditions, and carrying out the optimal resource pricing strategy and the distribution excitation strategy; the utility functions of the vehicle and the roadside unit are respectively optimized, so that the benefit balance of the two parties and the efficient utilization of computing resources are realized; under the condition that the optimal resource allocation strategy is determined, a multi-agent deep reinforcement learning mode is adopted, a resource pricing strategy is adaptively learned, semantic communication resource transaction is optimized, and efficient and reliable interaction is realized. According to the method, efficient and self-adaptive semantic task transaction can be realized in a vehicle-mounted semantic communication environment with incomplete information, and economic benefit maximization of independent vehicle users and edge nodes can be guaranteed by determining resource pricing and allocation strategies under game equilibrium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile edge computing, and in particular to a vehicle-mounted semantic communication task offloading method based on multi-agent game. BACKGROUND

[0002] With the rapid development of vehicle-mounted edge computing technology, modern vehicle networking systems are facing major challenges in communication and computing optimization. In emerging application scenarios such as autonomous driving and vehicle-road cooperation, the system has put forward more stringent requirements for communication bandwidth, delay sensitivity and computing real-time performance. Traditional data transmission methods have been difficult to meet these needs, while semantic communication, as a new task-oriented information transmission paradigm, can achieve dynamic data compression and intelligent transmission based on context understanding and semantic importance by extracting and transmitting semantic features rather than raw bit streams, thereby significantly improving communication efficiency and reducing network load. However, in high-density vehicle scenarios, vehicle-mounted terminals are limited by size, power consumption and cost, and their local computing resources often cannot meet the needs of real-time semantic analysis and processing.

[0003] Therefore, when performing semantic tasks, vehicles often rely on adjacent edge computing nodes to provide computing power support to achieve timely processing of high-computing-load tasks and deep extraction of semantic information. This demand has promoted the deep integration of vehicle networking and edge computing, significantly improving the efficiency and accuracy of task processing. However, it has also brought a series of challenges in resource trading and management, such as computing power pricing, service quality guarantee, and incentive mechanism design. In addition, in the process of semantic task offloading in vehicle networking edge environment, tasks are highly heterogeneous, involving factors such as vehicle local computing power, geographical location differences, and service demand diversity. At the same time, due to the high requirements of vehicle-mounted edge environment for safety and privacy and the characteristics of resource-limited, the system faces uncertain competition and multiple constraints under incomplete information.

[0004] To address the above problems and needs, a reasonable incentive mechanism and pricing strategy need to be designed to effectively trade edge computing resources in the context of incomplete information in vehicle-mounted semantic communication offloading. In the process of semantic task offloading, the adaptive optimization of computing resource pricing and allocation is ensured to maximize the utility of independent individuals and achieve the purpose of reducing computing energy consumption and improving resource utilization. SUMMARY

[0005] The present application provides a vehicle-mounted semantic communication task offloading method based on multi-agent game, to solve the problems of unreasonable computing resource pricing and allocation, difficulty in maximizing the utility of independent individuals, high computing energy consumption, and low resource utilization in the context of incomplete information in vehicle-mounted semantic communication offloading in the prior art.

[0006] The application provides a vehicle-mounted semantic communication task offloading method based on multi-agent game, comprising: A vehicle-mounted semantic task transaction scenario based on Stackelberg game is constructed; in the scenario, a vehicle and a roadside unit participate in semantic task offloading together, the vehicle processes state data perceived and collected, determines semantic tasks, and offloads the semantic tasks to an edge node; the edge server is deployed on the roadside unit and serves as an edge node to provide semantic services for the vehicle; the vehicle sets a resource pricing strategy for the roadside unit as a follower, and the roadside unit makes a computing resource allocation decision according to the given price of each vehicle; the semantic task of the vehicle is represented by a tuple , wherein refers to the number of semantic tasks to be offloaded to the roadside unit, is the average computing resource requirement of a single task, including semantic encoding resource requirement and semantic decoding resource requirement , represents a time delay constraint parameter of semantic task offloading; Based on the scenario, a first-stage game model of vehicle pricing optimization problem and a second-stage game model of roadside unit resource allocation optimization problem are established respectively; the backward induction method is used to solve the optimal resource allocation strategy, to realize roadside unit utility maximization while motivating the roadside unit to participate in the transaction; based on the optimal resource allocation strategy, the multi-agent reinforcement learning is used to obtain the optimal resource pricing strategy of the vehicle, and finally the optimal offloading scheme of the vehicle at the roadside unit is determined.

[0007] Further, in the first-stage game model, the utility function of the vehicle is constructed, and the utility function of the vehicle is constructed based on service experience income and computing resource purchase cost, the service experience income reflects the positive incentive of the vehicle due to time delay reduction during task offloading; the first-stage game model is used to solve the vehicle pricing optimization problem under the constraints of time delay, pricing and competition of the vehicle , to maximize the utility function of the vehicle , and to determine the optimal resource pricing strategy.

[0008] The optimization problem of the first-stage game model is represented as follows: st In the formula, For vehicles From roadside unit The service experience benefit function obtained from that location, i.e. ,in Revenue per unit of user experience , and These are weighting coefficients, and all are greater than 0. For vehicles Computational resources allocated for semantic encoding For vehicles Maximum computing capacity, For vehicles To the roadside unit Transmission delay, roadside unit For vehicles Allocated computing resources roadside unit For vehicles The maximum allocated computing resources. Additionally, for vehicles... roadside units Pricing of unit computing resources, For vehicles Maximum unit resource pricing, For vehicles Total latency caused by the unloading task This is the preset delay threshold.

[0009] Furthermore, in the second-stage game model, roadside units were constructed. utility function The roadside unit utility function Defined as the difference between the total revenue from computing resources obtained from vehicle payments and the energy consumption cost incurred in completing the semantic task, wherein the energy consumption cost is calculated based on a quadratic function; the second-stage game model is used to determine the resource pricing strategy based on the first-stage game model under the constraints of offloading latency and computing power of the edge server. roadside units utility function With the goal of maximizing resource allocation, we solve the problem of optimizing resource allocation for roadside units and determine the optimal resource allocation strategy.

[0010] The optimization problem of the second-stage game model is expressed as follows: st In the formula, roadside unit The energy consumption coefficient, roadside unit The unit energy consumption cost. roadside unit The number of floating-point operations that a GPU can perform per cycle. Additionally, Represented as vehicles Semantic task encoding is performed locally and transmitted to the roadside unit. The total delay, i.e. ,in For vehicles The latency of performing semantic encoding locally, For semantic tasks from vehicles To the roadside unit Transmission delay.

[0011] Furthermore, determining the optimal resource allocation strategy includes: In the second-stage game model, based on roadside units The resource allocation optimization problem is solved by deriving a unique optimal resource allocation strategy, which constitutes a subgame perfect Nash equilibrium. Through backward induction, given the leader's resource pricing strategy, the roadside units are analyzed... utility function The analysis is performed, and the existence and uniqueness of the optimal solution are proved using optimization theory, thereby solving for the utility function. Maximization problem.

[0012] Consider roadside units For vehicles The allocated computing resources are distributed according to weighted normalization, i.e. and the utility function Regarding control variables Find the first and second derivatives, which can be expressed as: In the formula, it is observed that The first derivative has a unique zero, and the second derivative is negative, thus yielding the roadside unit. utility function about It is strictly concave. According to the first-order optimality condition, i.e. Obtain roadside unit Optimal response strategy , can be represented as: Furthermore, taking the vehicle as the intelligent agent, the intelligent agent includes a first-stage game model and a second-stage game model. The first-stage game model includes a policy network. Based on the optimal resource allocation strategy, multi-agent reinforcement learning is used to obtain the optimal resource pricing strategy for the vehicle, including: S1. Parameter Initialization: Set the number of training rounds. and training rounds Initialize batch size and policy network parameters ; S2, Environment Reset: Reset the vehicle. initial environmental state And clear the experience replay buffer; S3, Condition Observation: Through vehicles Get the current environment status Update historical observations in historical space Generate observations ; S4. Strategy Decision: Observations... Input Policy Network Output vehicles Resource pricing strategy ; S5, Roadside Unit Response: Based on the resource pricing strategy for each vehicle and the roadside unit... Optimal resource allocation strategy Determine the allocation requirements for edge computing resources; S6, Status Update: Environment status changes from... Migrate to Calculate instant rewards ,like Update the instant reward to the optimal utility threshold. ; S7, Experience Storage: Transferring Samples Store in the experience replay buffer; S8. Network update condition judgment: Determine whether the training round is an integer multiple of the batch size. If yes, proceed to step S9; otherwise, jump to S10. S9. Network Update: Take samples of batch size evenly from the experience replay buffer, jointly optimize the policy loss function and value loss function, and update the policy network parameters to... ; S10, Round Update and Termination Judgment: Determine whether the number of training rounds in a single session is less than the first preset threshold. If so, return to step S3; otherwise, proceed to step S11. S11, round update and training termination: determining whether the number of training rounds is less than a second preset threshold value, if yes, returning to step S2, otherwise outputting the optimal resource pricing strategy of the vehicle .

[0013] Further, in step S2, in the current training round , the state space is defined as the union of the current resource pricing strategy of the vehicle , the resource pricing strategies of other vehicles, and the resource allocation strategy of the roadside unit . When the round , the state space is denoted as: Further, in step S3, a partially observable space model is constructed, which defines the observation space of the vehicle in the current game round as the joint space of the historical resource pricing strategy of the vehicle , the historical resource pricing strategies of other vehicles, and the historical resource allocation strategy of the roadside unit in the past game rounds. The corresponding observation value can be represented as: Further, in step S4, after receiving the observation value , the vehicle determines the resource pricing strategy of the vehicle under the preset pricing constraints to maximize the utility function of the vehicle , wherein the pricing constraints include that the resource pricing strategy of the vehicle is within a preset cost threshold range, , represents the lower bound cost of the resource pricing strategy, represents the upper bound price of the resource pricing strategy.

[0014] Further, in step S6, after state transition, the vehicle obtains a reward according to the current environmental state and the resource pricing strategy ; sets explicit upper and lower utility boundaries to divide the utility function of the vehicle into three intervals, and calculates the reward according to the utility function : when the utility function of the vehicle exceeds the optimal utility threshold When the vehicle is in use, a reward of 1 is given; when the vehicle is in use... utility function Within the preset utility threshold range Internal time, endowed with based on Normalized linear reward; when vehicle utility function Below the minimum utility threshold At that time, the reward is set to 0. Vehicle The reward function can be expressed as: In the formula, Indicates vehicle In the present The utility value of each round. It is a vehicle In the The highest utility value obtained before the round, i.e., the optimal utility threshold. For vehicles In the The maximum utility value among the lowest utility values ​​obtained before the round is the minimum utility threshold.

[0015] Furthermore, in step S9, the policy loss function is jointly optimized based on the policy-value function framework and the proximal policy optimization (PPO) algorithm. and value loss function .

[0016] Specifically, the following steps are taken: First, to accurately estimate the value loss function, the mean squared error (MSE) is used to optimize the value loss function. This makes the predicted value approximate the target. The value loss function can be expressed as: In the formula, For the current round Next strategy The value function, To start from training rounds The initial cumulative discounted target reward. To adapt to dynamic processes in complex environments within reinforcement learning, a strategy is utilized. The expected value is used to characterize the overall behavior of the vehicle agent in a multi-stage game. The value function can be expressed as: Secondly, regarding the policy loss function , a clipping mechanism is used to constrain the magnitude of policy update. This mechanism prevents the collapse of policy performance caused by a single update that is too large by limiting the deviation range of the policy update rate, ensuring the stability of training. The policy loss function can be expressed as: wherein, is the importance ratio, i.e. . is the Generalized Advantage Estimation (GAE) function, is the clipping function. Considering the generalized advantage estimation combined with multi-step temporal difference error, a balance is achieved between bias and variance, so that the advantage value of the action can be estimated more stably. The generalized advantage estimation function can be expressed as: wherein, is the multi-step temporal difference error. Considering the case where a single update is too large and causes the policy performance to collapse, the clipping function According to the positive and negative of the advantage value , different clipping processes are performed on the policy update rate , and the clipping function can be expressed as: wherein, the coefficient is used to limit the upper bound of the policy advantage when it is positive, to control excessive greed, is used to limit the lower bound when the policy disadvantage is negative to prevent rapid degradation. Finally, the network parameters are updated by optimizing the difference between the value loss function and the policy loss function, and introducing a regularization term. The updated policy network parameters can be expressed as: wherein, is the coefficient of the value loss function, is the number of batch experience samples, is the regularization term, used to prevent the policy from degrading too early, is the regularization term weight coefficient.

[0017] The application provides a vehicle-mounted semantic communication task offloading method based on multi-agent game. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the application or prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 FIG. 1 is a flowchart of a vehicle-mounted semantic communication task offloading method based on multi-agent game provided by an embodiment of the application.

[0020] Figure 2 FIG. 2 is a scene diagram of vehicle-mounted semantic task transaction based on Stackelberg game provided by an embodiment of the application.

[0021] Figure 3 FIG. 3 is a flowchart of obtaining an optimal resource pricing strategy of a vehicle based on multi-agent reinforcement learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the application will be clearly and completely described below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0023] The terms "first", "second" and the like in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, and the objects distinguished by "first", "second" are generally of a kind, and do not limit the number of objects, for example, the first object can be one or more.

[0024] With the rapid development of intelligent networked vehicles and vehicle-mounted edge computing technology, the demand for efficient communication and low-delay computing collaboration of the vehicle networking system is increasingly urgent. As a task-oriented information transmission paradigm, semantic communication can dynamically optimize data compression and transmission based on semantic importance, significantly improving communication efficiency. However, in high-density vehicle scenarios, the computing resources of vehicle-mounted terminals are limited, making it difficult to meet the demand for real-time semantic processing, thus relying on edge nodes to provide computing power support, thereby causing resource trading problems between vehicles and edge nodes. In addition, due to the heterogeneity of vehicle tasks and the incomplete information of the edge environment, how to design a flexible and adaptive resource allocation mechanism has become a key challenge.

[0025] To this end, the present invention provides a vehicle-mounted semantic communication task offloading method based on multi-agent game, which can realize intelligent resource collaborative optimization between vehicles and edge nodes. Under the premise of incomplete information, the computing resource pricing and resource allocation are dynamically optimized, thereby significantly improving the execution efficiency of semantic tasks in a vehicle networking environment, reducing system computing overhead and energy consumption, and improving economic efficiency.

[0026] Embodiment 1 Figure 1 The flowchart of the vehicle-mounted semantic communication task offloading method based on multi-agent game provided by the embodiment of the present invention is shown in FIG. 1. Figure 1 As shown in FIG. 1, a vehicle-mounted semantic communication task offloading method based on multi-agent game is provided, and the method flow steps are only as one possible implementation of the present invention.

[0027] A vehicle-mounted semantic task transaction scenario based on Stackelberg game is constructed. Based on the scenario, a first-stage game model of the vehicle pricing optimization problem and a second-stage game model of the roadside unit resource allocation optimization problem are established respectively. The backward induction method is used to solve the optimal resource allocation strategy, which can motivate roadside units to participate in transactions while maximizing the roadside unit utility. Based on the optimal resource allocation strategy, the multi-agent reinforcement learning is used to obtain the optimal resource pricing strategy of the vehicle, and finally the optimal offloading scheme of the vehicle at the roadside unit is determined.

[0028] Figure 2 The scenario diagram of the vehicle-mounted semantic task transaction based on Stackelberg game provided by the embodiment of the present invention is shown in FIG. 2. Figure 2 As shown in FIG. 2, in the scenario, a vehicle and a roadside unit cooperatively participate in semantic task offloading, the vehicle processes the perceived and collected state data, determines a semantic task, and offloads the semantic task to an edge node; the edge server is deployed on the roadside unit as an edge node to provide semantic services for the vehicle; the vehicle sets a resource pricing policy for the roadside unit as a follower, and the roadside unit makes a computing resource allocation decision according to the given price of each vehicle; the semantic task of the vehicle is represented by a tuple , wherein denotes the number of semantic tasks to be offloaded to the roadside unit, is the average computing resource requirement of a single task, including semantic encoding resource requirement and semantic decoding resource requirement , denotes a time delay constraint parameter of semantic task offloading.

[0029] Further, in the whole process of vehicle semantic communication task offloading based on Stackelberg game, the overall system time delay is mainly composed of three parts, including the first execution time delay of local encoding of the data collected by the vehicle, the transmission time delay of the semantic task in the communication link, and the second execution time delay of decoding of the received semantic task on the edge side. Specifically, first, the vehicle end encodes the collected data to extract semantics, and the first execution time delay can be represented as: In the formula, denotes the first execution time delay, is the number of floating point operations executable by the GPU of the vehicle per cycle, is the computing resource of the vehicle for executing semantic encoding.

[0030] Subsequently, the task encoded by semantics is transmitted to the orthogonal frequency division multiple access wireless channel. Considering that the semantic rate is defined based on the semantic unit and the amount of semantic information carried by it, the semantic rate can be represented as: In the formula, is the communication bandwidth between the vehicle and the roadside unit , is the accuracy of task execution, which is related to the average number of semantic symbols and the signal-to-noise ratio . In addition, is the average semantic information, is the average semantic symbol of each image sequence, the number of image sequences for each image.

[0031] Further, the transmission delay of the semantic task can be expressed as: Finally, after receiving the semantic task from the vehicle , the roadside unit will decode it. The second execution delay of this decoding process can be expressed as: wherein, is the number of floating-point operations that the GPU of the roadside unit can execute per cycle, is the computation resource of the roadside unit is the computation resource allocated to the vehicle for semantic decoding, i.e., the resource allocation strategy, is the total computation resource of the roadside unit is the total computation resource allocated to the vehicle set , is the pricing of the roadside unit for the unit computation resource of the vehicle , i.e., the resource pricing strategy. In summary, the total delay of the task of the vehicle in the entire offloading process can be expressed as: Further, the first-stage game model is an optimal pricing model for the vehicle, which comprehensively considers market demand elasticity, offloading delay, and competitive environment, etc., to maximize the utility of the vehicle. To balance the quality of service and cost, a utility function is constructed, which includes the service experience benefit and the cost of purchasing computation resource. To promote the active participation of the roadside unit , the service experience benefit reflects the positive incentive brought by the vehicle in the process of offloading tasks due to the reduction of delay, etc. Under the constraints of total offloading delay, pricing, and competition, the vehicle maximizes its own utility function by controlling the resource pricing strategy.

[0032] wherein, the optimization problem of the first-stage game model is expressed as follows: s.t. wherein, is the service experience benefit function obtained by the vehicle from the roadside unit , i.e. , wherein a unit experience benefit, , and is a weight coefficient and is greater than 0, is a vehicle assigned computing resources for semantic encoding, is a vehicle maximum computing capacity, is a roadside unit assigned maximum computing resources. In addition, is a vehicle set allowed maximum transmission delay, is a vehicle maximum unit resource pricing, is a preset delay threshold.

[0033] Further, the second-stage game model is a roadside unit optimal resource allocation model, which aims to ensure that the roadside unit can maximize its utility in the vehicle-mounted semantic communication and edge computing fusion environment on the basis of the first-stage vehicle determined resource pricing strategy. The utility function of the roadside unit is defined as the difference between the total income of computing resources obtained from vehicle payments and the energy cost generated by completing semantic tasks. To quantify the dynamic characteristics of energy consumption with resource allocation changes, the model introduces a quadratic energy consumption function to model energy consumption. To improve resource utilization, under the premise of meeting the system offloading delay and computing capacity upper bound constraints, the utility function is maximized by controlling the total edge computing resource allocation strategy assigned to all vehicles.

[0034] wherein the optimization problem of the second-stage game model is represented as follows: s.t. In the formula, is the energy consumption coefficient of the roadside unit , is the unit energy cost of the roadside unit . is the number of floating-point operations per cycle that the GPU of the roadside unit can perform. In addition, denotes the total delay of the vehicle executing semantic task encoding locally and transmitting to the roadside unit , that is, .

[0035] ​Further, in the second stage of Stackelberg game, the roadside unit as a follower, the resource allocation problem can derive a unique optimal response strategy, which constitutes a subgame perfect Nash equilibrium. Through backward induction, under the premise of knowing the given resource pricing strategy of the leader, the utility function of the roadside unit is analyzed, and the existence and uniqueness of the optimal solution are proved by combining the quasi-concavity or monotonicity theory, so as to solve the utility maximization problem. Considering the roadside unit The computing resources allocated to the vehicle are normalized by weight, that is , and the utility function is The first-order derivative and the second-order derivative are obtained by taking the first-order derivative and the second-order derivative, and the first-order derivative and the second-order derivative can be expressed as: In the formula, it is observed that the first-order derivative of has a unique zero point, and the second-order derivative is negative, so that the utility function of the roadside unit is With respect to , the function is strictly concave. According to the first-order optimality condition, that is , the optimal response strategy of the roadside unit can be expressed as: Figure 3 The flowchart of the optimal resource pricing strategy of the vehicle obtained based on multi-agent reinforcement learning provided by the embodiment of the application is shown in FIG. 1. As shown in FIG. 1, in the embodiment, the vehicle is taken as an agent, and the agent includes a first-stage game model and a second-stage game model. The first-stage game model includes a policy network. Based on the optimal resource allocation strategy, the optimal resource pricing strategy of the vehicle is obtained by using multi-agent reinforcement learning, which includes the following steps. Figure 3 S1, parameter initialization: setting the number of training rounds and the training round , initializing the batch size and the policy network parameter ; S2, environment reset: resetting the initial environment state of the vehicle , and emptying the experience replay buffer; S3, state observation: obtaining the current environment state of the vehicle , updating the historical observation value of the history space , and generating the observation value ; ​​S4. Strategy Decision: Observations... Input Policy Network Output vehicles Resource pricing strategy ; S5, Roadside Unit Response: Based on the resource pricing strategy for each vehicle and the roadside unit... Optimal resource allocation strategy Determine the allocation requirements for edge computing resources; S6, Status Update: Environment status changes from... Migrate to Calculate instant rewards ,like Update the instant reward to the optimal utility threshold. ; S7, Experience Storage: Transferring Samples Store in the experience replay buffer; S8. Network update condition judgment: Determine whether the training round is an integer multiple of the batch size. If yes, proceed to step S9; otherwise, jump to S10. S9. Network Update: Take samples of batch size evenly from the experience replay buffer, jointly optimize the policy loss function and value loss function, and update the policy network parameters to... ; S10, Round Update and Termination Judgment: Determine whether the number of training rounds in a single session is less than the first preset threshold. If so, return to step S3; otherwise, proceed to step S11. S11, Round Update and Training Termination: Determine if the number of training rounds is less than the second preset threshold. If yes, return to step S2; otherwise, output the vehicle. The optimal resource pricing strategy.

[0036] Furthermore, in step S2, in the current training round In this context, the state space is defined as the union of the current vehicle pricing strategy with other vehicle pricing strategies and the roadside unit computational resource allocation strategy. When the round... At that time, the state space is denoted as: Furthermore, in step S3, since vehicle resource pricing in the Stackelberg game is a multi-agent decision optimization problem under conditions of incomplete information, a partially observable space model is constructed for this optimization problem. In this model, the vehicle... The intelligent agent can only make decisions based on local environmental observations. Specifically, the first... Vehicles in each round of the game The observation space is defined as: the past In each round, the vehicles historical resource pricing strategy of other vehicles, and roadside units joint space of historical resource allocation strategies of other vehicles, and corresponding observation values may be represented as: Further, in step S4, when receiving observation values , the vehicle agent determines a resource pricing strategy to maximize its own utility. Given the lower bound cost and upper bound price of the resource pricing strategy, the strategy space of the vehicle may be represented as and the strategy of the vehicle agent may be represented as , where is the neural network parameter.

[0037] Further, in step S6, after state transition, the vehicle will obtain a reward according to the current environment state and the resource pricing strategy ; by setting explicit upper and lower utility boundaries, the utility function of the vehicle is divided into three intervals, and the reward is calculated according to the utility function : when the utility function of the vehicle exceeds the optimal utility threshold , a reward of 1 is given; when the utility function of the vehicle is within the preset utility threshold range , a normalized linear reward based on is given; when the utility function of the vehicle is lower than the lowest utility threshold , the reward is set to 0. The reward function of the vehicle may be represented as: wherein represents the utility value of the vehicle in the current round, is the highest utility value obtained by the vehicle before the round, i.e., the optimal utility threshold, is the highest utility value obtained by the vehicle before the The maximum utility value among the lowest utility values ​​obtained before the round is the minimum utility threshold.

[0038] Furthermore, in step S9, the network parameters are updated iteratively using the policy-value function framework and the proximal policy optimization algorithm. This considers that the policy loss function and the value loss function share neural network parameters. The total loss function is composed of the policy loss function. and value loss function Together they constitute the following: First, to accurately estimate the value loss function, the mean squared error is used to optimize the value loss function. This makes the predicted value approximate the target value. The value loss function can be expressed as: In the formula, For the current round Next strategy The value function, To start from training rounds The initial cumulative discounted target reward. To adapt to dynamic processes in complex environments within reinforcement learning, a strategy is utilized. The expected value of the vehicle agent is used to characterize its overall behavior in a multi-stage game. The value function can be expressed as: Secondly, regarding the policy loss function A shearing mechanism is used to constrain the magnitude of policy updates. This mechanism prevents policy performance from collapsing due to excessively large single updates by limiting the deviation range of the policy update ratio, thus ensuring training stability. The policy loss function can be expressed as: In the formula, Importance ratio, i.e. . The generalized advantage estimation (GAE) function is used. Let be the shear function. Considering the generalized dominance estimation combined with multi-step time-series difference error, a balance is achieved between bias and variance, thus allowing for a more stable estimation of the action's dominance value. The generalized dominance estimation function can be expressed as: In the formula, This is for multi-step time-series difference errors. Considering the possibility of a single update being too large, causing policy performance to collapse, the shearing function... Based on the dominance value The sign of the value affects the strategy update ratio. Different shearing processes can be performed, and the shearing function can be expressed as: where the coefficient is used to limit the upper bound of the advantage of the policy when it is positive, to control excessive greediness, is used to limit the lower bound of the disadvantage of the policy when it is negative, to prevent rapid degradation. Finally, the updated policy network parameters can be represented as: where is the coefficient of the value loss function, is the number of batch experience samples, is the regularization term, used to prevent the policy from degrading too early, is the regularization term weight coefficient.

[0039] The embodiment of the application provides a vehicle-mounted semantic communication task offloading method based on multi-agent game, which can realize efficient semantic perception and offloading cooperation of the vehicle-mounted terminal on the task, and through the multi-agent reinforcement learning game algorithm, the resource pricing strategy and the resource allocation strategy are jointly optimized under incomplete information, so that the purposes of reducing system communication load, improving task processing efficiency and enhancing resource utilization rate can be achieved. The specific beneficial effects of the embodiment of the application are embodied in: (1) The vehicle end uses semantic communication technology to efficiently compress task data, and then transmits the processed task data to the edge server end through a wireless channel for semantic decoding and task execution, so that the communication transmission overhead can be significantly reduced; (2) In the first stage of the game, a service experience based on multi-dimensional delay is designed as an incentive mechanism for the vehicle revenue item, and the computing resource fee paid by the vehicle to the roadside unit is designed as a cost item, which together constructs the utility function of the vehicle, improves the service quality on the premise of reasonable resource overhead; (3) In the second stage of the game, a weight-normalized computing resource allocation mechanism is designed based on the roadside unit utility function composed of the resource fee paid by the vehicle and the energy consumption cost, and the total computing resource allocated to the vehicle is taken as a control variable, so that the effective use of computing resources is realized on the premise of balancing system energy consumption and revenue; (4) In view of the incomplete information problem caused by privacy security, the Stackelberg game between the vehicle and the roadside unit is modeled as a partially observable Markov decision process, and a multi-agent reinforcement learning solution scheme for the vehicle end is designed, and the optimal strategy is obtained by using the action-value function framework and the proximal policy optimization method.

[0040] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments. Based on such an understanding, the above technical solutions can be embodied in the form of a software product in essence or in the form of a part of the prior art. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0041] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for vehicle semantic communication task offloading based on multi-agent game, characterized in that, Comprise: A scenario of vehicle semantic task transaction based on Stackelberg game is constructed; in the scenario, a vehicle and a roadside unit participate in semantic task offloading together, the vehicle processes the state data perceived and collected, determines a semantic task, and offloads the semantic task to an edge node; the edge server is deployed on the roadside unit and serves as the edge node to provide semantic services for the vehicle; the vehicle sets a resource pricing strategy for the roadside unit as a leader, and the roadside unit makes a calculation resource allocation decision according to the given price of each vehicle as a follower; the semantic task of the vehicle is represented by a tuple , wherein denotes the number of semantic tasks to be offloaded to the roadside unit, is the average calculation resource requirement of a single task, includes semantic encoding resource requirement and semantic decoding resource requirement , denotes a time delay constraint parameter of semantic task offloading; Based on the scene, a first-stage game model of vehicle pricing optimization problem and a second-stage game model of roadside unit resource allocation optimization problem are established respectively; the backward induction method is used to solve the optimal resource allocation strategy, which maximizes the roadside unit utility while encouraging the roadside unit to participate in the transaction; Based on the optimal resource allocation strategy, the multi-agent reinforcement learning is used to obtain the optimal resource pricing strategy of the vehicle, and finally the optimal unloading scheme of the vehicle at the roadside unit is determined.

2. The multi-agent game based vehicular semantic communication task offloading method according to claim 1, characterized in that, In the first-stage game model, vehicles were constructed. utility function The vehicle utility function It is constructed based on service experience benefits and computing resource purchase costs, wherein the service experience benefits reflect the vehicle's... The positive incentive resulting from reduced latency during the unloading task; the first-stage game model is used in the vehicle Under the constraints of latency, pricing, and competition, to maximize vehicle performance utility function With the objective of solving the vehicle pricing optimization problem, we determine the optimal resource pricing strategy.

3. The method of claim 1, wherein, In the second-stage game model, roadside units were constructed. utility function The roadside unit utility function Defined as the difference between the total revenue from computing resources obtained from vehicle payments and the energy consumption cost incurred in completing the semantic task, wherein the energy consumption cost is calculated based on a quadratic function; The second-stage game model is used to determine an optimal resource pricing strategy based on the first-stage game model under the constraints of the offloading delay and computing power of the edge server with a roadside unit of the roadside unit The optimal resource allocation strategy is determined by solving the roadside unit resource allocation optimization problem with the objective of maximizing the utility function 4. The method of claim 1, wherein, The optimal resource allocation strategy comprises: In the game process of the second stage game model, based on the resource allocation optimization problem of roadside unit , the unique optimal resource allocation strategy is derived to constitute a sub-game perfect Nash equilibrium; through backward induction, the utility function of the roadside unit is analyzed under the premise that the leader gives a resource pricing strategy, and the existence and uniqueness of the optimal solution are proved by combining the optimization theory, thereby solving the utility function maximization problem; considering that the pricing weight of the roadside unit based on the vehicle is the vehicle allocation calculation resource, the utility function of the roadside unit is derived to obtain the first and second order derivatives of the utility function with respect to the control variable , it is observed that the first order derivative has a unique zero point and the second order derivative is negative, and it is obtained that the utility function of the roadside unit is strictly concave with respect to , and according to the first order optimality condition, the optimal resource allocation strategy of the roadside unit is obtained.

5. The method of claim 1, wherein, Taking the vehicle as an agent, the agent comprises a first-stage game model and a second-stage game model, the first-stage game model comprises a policy network, and the multi-agent reinforcement learning is used to obtain the optimal resource pricing strategy of the vehicle based on the optimal resource allocation strategy. S1, parameter initialization: set the number of training rounds and training rounds , initialize the batch size and policy network parameters ; S2, Environment Reset: Reset the initial environment state of the vehicle and clear the experience replay buffer. and clear the experience replay buffer. S3, state observation: by vehicle acquire current environment state , update historical observation value of historical space , generate observation value ; S4, policy decision: determine the resource pricing policy for the vehicle based on the observation input policy network , output resource pricing policy for the vehicle ; S5, Roadside unit response: determining edge computing resource allocation requirements according to resource pricing strategies of each vehicle and optimal resource allocation strategies of roadside units of roadside units​ S6, state update: the environment state from migrates to , the immediate reward is computed , and the immediate reward is updated to the optimal utility threshold ; S7, Experience store: store the transition sample into the experience replay buffer; S8, network update condition judgment: judge whether the training round is an integer multiple of the batch size, if yes, execute step S9, otherwise jump to S10; S9, network update: uniformly take samples of batch size from the experience replay buffer, jointly optimize the policy loss function and the value loss function, and update the policy network parameters to ; S10, round update and termination judgment: judge whether the single training round is less than the first preset threshold, if yes, return to step S3, otherwise execute step S11; S11, round update and training termination: it is judged whether the training round number is less than a second preset threshold, if yes, returning to step S2, otherwise outputting the optimal resource pricing strategy of the vehicle .

6. The method of claim 5, wherein the multi-agent game based vehicular semantic communication task offloading is characterized by, In step S2, in the current training round, the state space of the vehicles is defined as the union of the resource pricing policy of the vehicles and the resource allocation policy of the roadside units .

7. The method of claim 5, wherein the multi-agent game based vehicular semantic communication task offloading is characterized by, In step S3, a partial observable space model is constructed, which defines the observable space of the vehicles in the current game round as the joint space of the historical resource pricing strategies of the vehicles and the historical resource allocation strategies of the roadside units in the past multiple historical game rounds. ​​​ 8. The method of claim 5, wherein the multi-agent game based vehicular semantic communication task offloading is characterized by, In step S4, upon receiving the observation value , the vehicle determines a resource pricing policy of the vehicle under preset pricing constraints, so as to maximize a utility function of the vehicle , wherein the pricing constraints comprise that the resource pricing policy of the vehicle is within a preset cost threshold range, , a lower bound cost of the resource pricing policy, and an upper bound price of the resource pricing policy.

9. The method of claim 5, wherein the multi-agent game based vehicular semantic communication task offloading is characterized by, In step S6, after the state transition, the vehicle Based on the current environmental status and corresponding resource pricing strategies Receive rewards ; Set clear upper and lower utility boundaries, and classify vehicles utility function Divide into three intervals, according to the utility function Determine the reward When the vehicle utility function Exceeding the optimal utility threshold When the vehicle is in use, a reward of 1 is given; when the vehicle is in use... utility function Within the preset utility threshold range Internal time, endowed with based on Normalized linear reward; when vehicle utility function Below the minimum utility threshold At that time, the reward is set to 0.

10. The method of claim 5, wherein the multi-agent game based vehicular semantic communication task offloading is characterized by, In step S9, the following steps are included: based on a policy-value function framework and a proximal policy optimization algorithm, jointly optimize a policy loss function and a value loss function and a value loss function , are current parameters of the policy network Considering the combination of multi-step time difference error in generalized advantage estimation, a clipping mechanism is introduced to constrain the different deviation ranges of the policy update amplitude according to the positive and negative of the advantage value; the difference between the optimized policy loss function value and the optimized value loss function value is calculated, a regularization term is introduced, and the policy network parameters are updated.

Images