Internet of vehicles resource scheduling method and system for public transportation

Abstract

The invention discloses an Internet of Vehicles resource scheduling method and system for public transport, and the method comprises the steps: taking an RSU as a leader, taking a bus as a follower, and constructing a Stackelberg game model; and based on the Stackelberg game model, fusing MALPPO multi-agent deep reinforcement learning, and outputting an optimal resource allocation strategy. The system comprises a training game model building module and a multi-agent reinforcement learning module. According to the invention, bandwidth allocation can be optimized, so that a communication system can better adapt to continuously changing communication environments and requirements. The method can be widely applied to the field of communication resource optimization.

Description

Technical Field

[0001] The present invention relates to the field of communication resource optimization, and in particular to a vehicle network resource scheduling method and system for public transportation. Background Art

[0002] With the vigorous development of a new round of technological revolution and industrial transformation, automobiles are deeply integrated with technologies such as artificial intelligence, communication networks, and large models, and intelligence and networking have become important directions for industrial development. The rapid development of the Internet of Things and wireless communication technologies is also constantly promoting the transformation of traditional vehicles to the Internet of Vehicles. In order to improve the riding experience of buses, a certain industry plans to pilot the transformation of card swiping devices into public Wi-Fi on some buses. Passengers can swipe cards through mobile phone apps and access free on-board Wi-Fi. To this end, it is necessary to realize highly personalized information services for buses. By accurately analyzing passengers' travel habits, preferences, and real-time locations, differentiated, highly popular, and highly interesting content can be pushed on different buses to realize personalized recommendations for information services, ultimately improving passenger satisfaction and industrial benefits. The specific implementation is as follows: Combined with federated learning, wireless Wi-Fi is installed on buses. Buses use wireless Wi-Fi hotspot coverage and management systems to obtain passengers' information browsing data. Roadside units (RSUs) act as intermediate processors to receive passenger data and organize and transmit it to the central server. The central server processes the data and sends the results down, so that different buses can push content that is different, popular, and interesting, and personalized information recommendations can be made to enhance passengers' experience of using the bus network.

[0003] As a dynamic mobile environment, the existing recommendation system fails to adapt to the dynamic mobile environment with high real-time requirements and dynamically adjust the recommended content. In a mobile environment, the delay and instability of data transmission will seriously affect the response speed and accuracy of the service. Different buses have different computing and storage capabilities. Buses with old equipment or excessive passenger flow will cause excessive computing data, which will cause serious device heterogeneity. If the Internet of Vehicles system wants to provide passengers with a good service experience, it needs to allocate a large amount of communication and computing resources. Such computing-intensive tasks are generally beyond the capacity of buses. The design of the resource scheduling scheme of the traditional system does not fully consider the heterogeneity of devices, resulting in unstable system operation or response delays, which seriously affects the user experience. Summary of the invention

[0004] In view of this, in order to solve the technical problem that the existing personalized recommendation method does not consider the situation of device heterogeneity, has the defect of insufficient optimization of communication resource allocation, and thus causes the recommendation system to run slowly, the present invention proposes a vehicle network resource scheduling method for public transportation, the method comprising the following steps:

[0005] With RSU as the leader and bus as the follower, a Stackelberg game model is constructed;

[0006] Based on the Stackelberg game model, MALPPO multi-agent deep reinforcement learning is integrated to output the optimal resource allocation strategy.

[0007] In some embodiments, the Stackelberg game model has the following two stages:

[0008] In the first stage, RSU selects a broadband pricing strategy;

[0009] In the second stage, the bus observes the strategy selected by the RSU and responds accordingly based on its own needs.

[0010] In some embodiments, in the second stage, the utility function of the bus is:

[0011]

[0012] in, represents the utility function of the ith bus to the jth RSU, represents the satisfaction level of the i-th bus with the j-th RSU, represents a quantitative measure of QoE for passengers, represents the loan pricing of the j-th RSU for the i-th bus, represents the bandwidth demand of the i-th bus for the j-th RSU, represents the signal-to-noise ratio transmitted from the jth RSU to the i-th bus, represents the spectral efficiency of the jth RSU transmitting data to the ith bus, D i represents the task volume of the personalized federated recommendation system model in the i-th bus, represents the data processing quality before the jth RSU transmits to the i-th bus, It represents the minimum required lower limit of the model service quality supported by the RSU after the i-th bus offloads the personalized federated recommendation system model to the RSU.

[0013] In some embodiments, in the first stage, the utility function of the RSU is:

[0014]

[0015] Among them, V j represents the utility function of the jth RSU, c j represents the bandwidth cost of the jth RUS, and I represents the set of buses.

[0016] In some embodiments, in the Stackelberg game model, there are the following limitations:

[0017] Phase 2:

[0018] In the second stage, the bus determines the optimal bandwidth purchasing strategy that maximizes its utility by considering the bandwidth price strategy vector published by the RUS;

[0019] Phase 1:

[0020] The non-cooperative game between RSUs has a Nash equilibrium;

[0021] The multi-leader multi-follower Stackelberg game between RSU and bus has Stackelberg equilibrium;

[0022] The Stackelberg equilibrium between the RSU and the bus is unique.

[0023] In some embodiments, the step of outputting the optimal resource allocation strategy based on the Stackelberg game model and integrating MALPPO multi-agent deep reinforcement learning specifically includes:

[0024] Define state space, action space, and reward function;

[0025] At each time step, the RSU decides on an action based on the current state, and the environment gives a corresponding reward;

[0026] Put the state, action and reward into the cache pool, and update the actor network and critic network in combination with the updated strategy function, value function and loss function;

[0027] Extract the information related to RSU from the buffer queue, generate a new state, and enter the next time step;

[0028] Iterate until the simulation results fit and are close to the theoretical game equilibrium solution, and output the optimal resource allocation strategy.

[0029] The present invention also proposes a vehicle network resource scheduling system for public transportation, comprising:

[0030] The game model building module is used to build a Stackelberg game model with RSU as the leader and the bus as the follower;

[0031] The multi-agent reinforcement learning module is based on the Stackelberg game model and integrates MALPPO multi-agent deep reinforcement learning to output the optimal resource allocation strategy.

[0032] Based on the above scheme, the present invention provides a vehicle network resource scheduling method and system for public transportation. Firstly, in view of the problem of actual bus resource utilization, a Stackelberg game model is established. The bus makes corresponding adjustments based on the strategy, optimizes bandwidth allocation, obtains appropriate resources, and enables the communication system to better adapt to the ever-changing communication environment and needs, improves the system's ability to cope with complex real-life bus conditions, and enhances passengers' riding experience; secondly, the Stackelberg game model is integrated with MALPPO multi-agent deep reinforcement learning. The Stackelberg game model provides modeling of the mutual influence between buses and in the multi-agent system, uses deep learning technology to train agents to learn the optimal resource allocation strategy, and MALPPO multi-agent deep reinforcement learning realizes collaboration and competition between RSUs, thereby better optimizing resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of the steps of a vehicle network resource scheduling method for public transportation according to the present invention;

[0034] Figure 2 It is a scene schematic diagram of the present invention;

[0035] Figure 3 is a schematic diagram of the MALPPO embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the framework of the MADRL embodiment of the present invention. DETAILED DESCRIPTION

[0037] The purpose of this invention is to propose a resource scheduling optimization solution based on the Stackelberg game framework, to provide an optimal strategy for the two-way selection between RSU and buses, to optimize the allocation of traffic resources, and to solve the problem of slow system operation or even crash caused by heterogeneous equipment. Figure 2 .

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0040] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0041] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0042] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0043] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0044] Reference Figure 1 , which is a flow chart of an optional example of a vehicle network resource scheduling method for public transportation proposed in the present invention. The method can be applied to a computer device. The resource scheduling method proposed in this embodiment may include but is not limited to the following steps:

[0045] Step S1, taking RSU as the leader and bus as the follower, constructing Stackelberg game model;

[0046] Step S2: Based on the Stackelberg game model, the MALPPO multi-agent deep reinforcement learning is integrated to output the optimal resource allocation strategy.

[0047] In some feasible embodiments, the step S1 specifically includes:

[0048] On the bus, a corresponding personalized federated recommendation system model will be generated according to the composition of the passenger group to provide high-quality content recommendation services. This embodiment proposes a MLMF SG (Multi-Leader Multi-Follower Stackelberg Game) framework, in which RSU plays the role of leader, and the set of RSU is recorded as The bus (hereinafter referred to as BUS) plays the role of follower and is recorded as One iteration of the entire game will be decomposed into two stages: the first stage: RSU strategy selection, each RSU in the REU set J selects a broadband pricing strategy, and the second stage: strategy response, while each BUS in the bus set I observes the strategy selected by the RSU and responds accordingly according to its own needs. Because there are multiple personalized federated recommendation system models on a single BUS, the BUS can offload them to different RSUs and obtain resources from multiple sources. Thus, a single RSU j Use sufficient bandwidth and computing resources to help train the personalized federated recommendation system model. i RSU j The bandwidth requirement is recorded as RSU j About BUS i The bandwidth pricing is denoted as Both RSU and BUS optimize their utility by adjusting strategies. This embodiment uses reverse induction to derive the utility functions of RSU and BUS. Therefore, the game is decomposed into the following two sub-games.

[0049] BUS i Bandwidth requirements in the second phase: In the second phase, the bandwidth from RSU is determined j Purchased bandwidth requirements This purchase decision is made to facilitate the migration of the corresponding personalized federated recommendation system model. The bandwidth demand is guided by the bandwidth price policy released by RSU in phase 1. Assume that RSU j The transmission power sent to BUS is m j .set up For RSU j The signal-to-noise ratio (SNR) of the transmission is expressed as:

[0050]

[0051] in and σ j represent the channel gain and additive white Gaussian noise respectively, γ represents the path loss exponent; h is a constant, representing the uplink channel fading coefficient.

[0052] Defining RSUs j To the BUS iThe spectral efficiency of transmitted data is Expressed as

[0053]

[0054] Therefore, RSU j About BUS i The transmission rate is

[0055]

[0056] AoI (Age of Information) has been widely used to quantify the freshness of data arriving at the destination. It is defined as the time since the latest received update was generated at the source, which is an effective and accurate indicator for improving time performance. Similarly, in order to quantify the freshness of the migration task of the personalized federated recommendation system model, a new indicator AoMM (Age of Model Migration) is proposed based on the concept of AoI, which is defined as the time between the last successfully received task block and the generation of the first task block in the model migration. BUS i The task volume of the personalized federated recommendation system model is defined as |D i ∣, for RSU j and BUS i , the corresponding AoMM is

[0057]

[0058] The data processing quality of RSU is also a key selection evaluation criterion for BUS. This is a well-known and widely used quality metric. Combined with the specific practical scenario of this embodiment, this embodiment redefines the RSU service quality metric (Quality of RSU Service, QoRS) to quantify the data processing quality before RSU transmission, which consists of multiple components:

[0059]

[0060] Where G represents RSU j The GPU computing performance, t represents the training delay, l represents the queuing delay, and O represents the GPU occupancy rate.

[0061] use Satisfaction factor indicates BUS i Select RSU j The satisfaction of personalized federated recommendation system model migration is used to model the effect of data set transmission rate on BUS i The impact of satisfaction:

[0062]

[0063] use Indicates BUS i RSU j Timeliness satisfaction factor for data set transmission Indicates BUS i Sensitivity to data transmission rate. Inspired by Gustav Fechner's extension of the relationship between human perception and relative stimulus changes, namely Weber-Fechner's law, as well as Stevens' Power law and utility theory in economics, this embodiment uses the ln(·) function to simulate the bus's perception of service quality, which is essentially the passenger's perception of the service quality of the entire system. The logarithmic function can well capture the nonlinear characteristics of human perception, that is, at low stimulus levels, perception changes are more sensitive, while at high stimulus levels, perception changes become less sensitive, avoiding other resource overhead and cost increases caused by blindly pursuing further improvements in service quality when service quality is high. Therefore, BUS i Migrate the personalized federated recommendation system model to RSU j After RSU j For BUS i The level of satisfaction with the service experience provided is expressed as:

[0064]

[0065] RSU j The quality of data processing has a minimum BUS does not consider sending model migration request links to RSUs whose data processing quality is below this threshold.

[0066] at the same time It can also be mapped as a quantitative measure of QoE (quality of experience) for passengers, with higher values ​​indicating enhanced QoE. i RSU j Request bandwidth resources, then BUS i RSU j The utility function is:

[0067]

[0068] In the formula Indicates BUS i The meta-immersion coefficient, make sure The following abbreviations are used replace Right now

[0069]

[0070] Since multiple RSUs providing similar resource services are considered, BUS i You can choose to request bandwidth resources from any RSU, BUS i The utility function is expressed as:

[0071]

[0072] in Indicates BUS i For each RSU j The bandwidth requirement, Indicates each RSU j The policy decision process is driven by the goal of maximizing the respective utility. i , each BUS i Determine its optimal bandwidth demand strategy vector b i , optimize the bandwidth price strategy vector p i The optimization problem of BUS bandwidth purchase is stated as follows:

[0073]

[0074] Where R i It's a bus i The payment budget of BUS determines its bandwidth purchase strategy, which depends on the pricing strategy adopted by RSU. On the other hand, the accumulated bandwidth purchased by BUS will affect the sales price strategy of RSU. The interaction of this two-layer game lays the foundation for the derivation of the utility function of RSU in this embodiment.

[0075] Pricing strategy for RSU in the first phase: In the first phase, each RSU j Formulate its BUS i The bandwidth pricing strategy is determined based on the analysis of the expected bandwidth purchase demand of BUS. In order to facilitate model migration, RSU needs to purchase bandwidth from the system market. Subsequently, bandwidth resources are allocated to BUS for the migration of the personalized federated recommendation system model, and the related bandwidth cost is recorded as c j , which consists of RSU j Therefore, RSU j The utility function of contains two parts: income and cost, which can be expressed as:

[0076]

[0077] RSUs are priced according to non-cooperative competition. j According to the strategy of all BUS (i.e. b j) decides to maximize its utility function V j , thus obtaining the optimal strategy In addition, this embodiment also needs to consider the constraints of the RSU utility function, that is, the price constraint, which is expressed as in Represents RSU j Announced broadband pricing strategy, p -j Represents RSU j The maximum bandwidth price is determined by market rules. Therefore, the bandwidth price optimization problem of the RSU as the leader is expressed as:

[0078]

[0079] In the subsequent analysis, considering the utility functions of RSU and BUS, the reverse deduction method is applied to analyze the equilibrium of the multi-leader and multi-follower Stackelberg game.

[0080] In some feasible embodiments, the Stackelberg game model further includes the following definitions and limitations:

[0081] Definition 1: (Stackelberg equilibrium) Let and Respectively represent BUS i To RSU j Optimal bandwidth requirements and RSU j To BUS i The optimal bandwidth price is . Therefore It depends on the strict fulfillment of the following two conditions:

[0082] 1) There is a sub-game between RSUs, and they strictly compete for bandwidth prices in a non-cooperative manner. The Nash equilibrium of the sub-game can be obtained if the following conditions are met:

[0083]

[0084] 2) In Under the optimal pricing strategy, BUS i The response of This response will become the utility function The only demand strategy that maximizes For BUS i The best response.

[0085]

[0086] The optimal strategy of BUS in the second stage: BUS considers the bandwidth price strategy vector p released by RSU in the second stage. jTo determine the optimal bandwidth purchasing strategy that maximizes its utility.

[0087] Proposition 1: For any given bandwidth selling price RSUs j , BUS i The optimization problem is strictly convex, so the optimal strategy for BUS to obtain bandwidth requirements can be expressed as:

[0088]

[0089] The optimal strategy for bandwidth demand when the second constraint in equation (11) is relaxed can be expressed as The optimal strategy for bandwidth demand when the constraints are tight can be expressed as Specifically:

[0090]

[0091] prove:

[0092] 1) When the second constraint in equation (11) is relaxed, the utility function U is obtained i For bandwidth demand strategy The first and second order derivatives of are:

[0093]

[0094] Utility function U i The negative second-order derivative indicates that the utility function U i About BUS i The bandwidth requirement strategy used With strict concavity, the utility function U i The first-order derivative of has a unique zero, so BUS i The optimization problem is a convex optimization problem; in this embodiment, U i The first-order derivative of is zero, and this embodiment can obtain the optimal response strategy under constraint relaxation

[0095]

[0096] Consider whether the first constraint in Eq.(a.11) is satisfied. When BUS i No RSUs will be paid j Purchasing bandwidth resources Therefore, if the second constraint in Eq.(a.11) is relaxed, then BUS i RSU j The best response strategy is

[0097]

[0098] 2) When the second constraint in Eq. (a.11) is tight, this embodiment will obtain Use Lagrangian function to find BUS i The optimal strategy is:

[0099]

[0100] where λ i is the Lagrange multiplier, obtained from the Karush Kuhn-Tucker (KKT) condition in Eq. (a.24):

[0101]

[0102] By solving equation (25), we can get BUS i RSU j The optimal strategy is expressed as:

[0103]

[0104] Substituting equation (28) into equation (27), we get the following expression:

[0105]

[0106] Substituting equation (30) into equation (28), we can obtain i The optimal customization demand strategy

[0107]

[0108] Will and Replace with A and B

[0109]

[0110] Consider whether the first constraint in equation (11) is satisfied, that is, if BUS i No RSUs will be paid j Purchasing bandwidth resources Therefore, if the second constraint in (11) is tight, then BUS i RSU j The best response strategy is:

[0111]

[0112] The above results show that RSU j The higher the bandwidth price you set, the higher the BUS iThe less bandwidth purchased. Based on the above analysis, BUS can adjust its optimal bandwidth purchase strategy according to the bandwidth sales price provided by RSU.

[0113] Phase 1: RSU's optimal equilibrium strategy in the first phase: By predicting the optimal bandwidth purchase strategy of the BUS, the RSU plays the role of a leader in the first phase, with the goal of maximizing the value of its utility function. The non-cooperative game is based on the premise that each individual is rational and each game participant pursues the optimal decision. Therefore, in the context of the RSU announcing the decision first, each RSU participating in the personalized federated learning model migration acts in a self-interested and rational manner, formulating strategies with the main goal of maximizing its utility. Therefore, this embodiment describes the competitive dynamics between RSUs as a non-cooperative game, and this embodiment hopes that this game has a Nash equilibrium and an equilibrium solution.

[0114] Lemma 1: The existence of a Nash equilibrium for a non-cooperative game in a subgame is guaranteed when the following three conditions are met: the set of players is finite; both strategy sets are closed, bounded, and exhibit convexity; and the utility function exhibits continuity and quasi-concavity over the range of the strategy space.

[0115] Theorem 1: There is a Nash equilibrium in the non-cooperative game between RSUs.

[0116] prove:

[0117] 1) When the second constraint of equation (11) is tight:

[0118] The BUS given in equation (27) i The optimal strategy Substitute RSU j The utility function V j have to:

[0119]

[0120] By calculating V j right The first and second order derivatives of , we get the following expressions:

[0121]

[0122] Then, V j The first-order derivative of is set to zero, and we can get RSU j About BUS i The optimal strategy is expressed as

[0123]

[0124] in Representatives in addition to RSU jAll RSU policies except j The policies of other RSUs are taken into account in determining its policy.

[0125] When the second constraint condition of equation (11) is relaxed:

[0126] In this embodiment, BUS i Substituting the optimal strategy (as specified in Eq.(a.18)) into RSU j In the utility function:

[0127]

[0128] By calculating V j right The first and second order derivatives of , we get the following expressions:

[0129]

[0130] Similarly, the utility function V j To meet the strict concave property, V j about The first-order derivative of is set to zero, that is, RSU j About BUS i The optimal strategy is expressed as

[0131]

[0132] It is obvious that no matter what the constraints are, RSU j The utility function V j The second-order derivative of is negative, indicating its concavity. In addition, the strategy set of RSU satisfies the following basic properties: closed, bounded, and convex. The strategy set of RSU is closed, which means that the set contains all possible strategies, and for any set of strategies, their linear combination also belongs to the inside of the set. This means that in the strategy set of RSU, no possible strategies are missed, and no boundary points of any strategy will fall outside the strategy set; the strategy set of RSU is bounded, which means that the strategies of the set have upper and lower limits and will not extend infinitely. This means that within the range of strategies that RSU can choose, each strategy has a clear limit and will not grow or decrease infinitely; the strategy set of RSU is convex, which means that for any two strategies in the set, their linear combination is also in the set. In other words, the points on the line between any two strategies in the strategy set of RSU are also optional strategies. This ensures that there will be no singular points or discontinuities in the strategy selection of RSU, and an intermediate strategy can be found between any two strategies.

[0133] In addition, considering the finiteness of the RSU set and the continuity of its utility function, according to Lemma 1, it can be proved that the Nash equilibrium between RSUs exists and is unique.

[0134] Theorem 2: The multi-leader and multi-follower Stackelberg game between RSU and BUS has a Stackelberg equilibrium.

[0135] Proof: Theorem 1 proves that the existence of Nash equilibrium between RSUs has been formally established, and the previous article has proved that after the RSU announces its pricing strategy, the BUS has the ability to select the optimal broadband demand strategy, that is, it can make a unique response to maximize its utility for all RSUs' given pricing strategies. Therefore, it can be inferred that the Stackelberg game between RSU and BUS reaches the Stackelberg equilibrium.

[0136] Lemma 2: If the following three properties are satisfied, then the Stackelberg equilibrium is guaranteed to be unique. Positive: Monotonicity: If p'>p, then Scalability: For any χ>1, then in, Representative RSU j The optimal strategy.

[0137] Theorem 3: The Stackelberg equilibrium between RSU and BUS is unique.

[0138] Proof: From Eq. is satisfied. Moreover, yes The increasing function of As p increases, it increases. Therefore, when p'>p, that is, When Finally, since χ>1, then χ 2 >χ. Therefore, for This embodiment has

[0139]

[0140] Based on the above analysis, it has been proved that in this case, RSU jThe optimal strategy function has a unique optimal point, that is, the overall Stackelberg game proves that there is a unique Stackelberg equilibrium solution. In the case of Stackelberg game with Stackelberg equilibrium, each RSU independently determines its optimal strategy, and no RSU can improve its payoff by unilaterally changing its selected strategy. Once all RSUs have announced their pricing strategies, the BUS proceeds to determine their optimal bandwidth requirements to maximize their utility.

[0141] In summary, when The optimal strategy function of RSUj is:

[0142]

[0143] In some feasible embodiments, the step S2 specifically includes:

[0144] In order to solve the challenges brought by incomplete information, this embodiment uses the Multi-Agent LSTM-based Proximal Policy Optimization (MALPPO) algorithm to achieve the game equilibrium of the Stackelberg game model by exploring the optimal solution of buses and behaviors in the Stackelberg game model. The MALPPO algorithm enables individuals as intelligent agents to quickly converge to near-optimal decision-making capabilities. Compared with the traditional DRL method that focuses on predicting fixed strategies and single-step models, the method of this embodiment can effectively decompose the problem using the Markov property.

[0145] Multi-Agent LSTM-based Proximal Policy Optimization (MALPPO), MAPPO assumes that information is shared between agents and adopts a training method called centralized training and decentralized execution (CTDE). During the training process, it maintains a global evaluation network to evaluate the state values ​​of all agents. This training method is not suitable for our privacy protection scenario. Therefore, the method of this embodiment adopts a completely decentralized training method, in which each agent has a decentralized actor and a decentralized crtitc; in order to speed up convergence and reduce the risk of falling into local optimality during training, this embodiment adopts a long short-term memory (LSTM) network as the basic architecture of actors and critics. Both the actor and critic networks are composed of two multi-layer perceptrons (MLPs). The local observation variables are used as the input of the MLP to extract feature vectors, which are then input into the LSTM network with two hidden layers. The memory units in the LSTM network dynamically manage the retention and forgetting of information. Finally, the output is obtained through the MLP of two operations or state values ​​to update the internal network architecture of the agent. The specific architecture is as follows Figure 3 shown.

[0146] Training DRL agents requires an MDP environment, which is formulated by abstracting the dynamic relationship between buses and RSUs into a Stackelberg game. Therefore, this strategic interaction can be transformed into a multi-agent MDP. In this framework, each RSU acts as an independent agent and guides decision making under the MDP model. This environment has I buses to jointly support the complexity of the learning environment. Specifically, let represents an MDP. and γ j Represents the state space, action space, state transition probability, reward function and discount factor for each RSU j.

[0147] At each time step t, The interaction between RSU j and the environment can obtain its current state, which is defined as s j (t). Throughout the training process, the DRL agent representing RSU j is used to interact with the environment. At each time step, when RSU j is based on the current state s j (t) to decide the action p j (t), the environment will immediately give a reward r j(t). The MADRL system contains a finite buffer queue, denoted as D, which is used to store historical operation data. The capacity of the finite buffer queue is defined as L. The information related to the RSU is extracted from the buffer queue to generate a new state so as to enter the next time step t+1.

[0148] (1) State space: This embodiment uses to represent the state space of RSU j. The state of RSU j includes its current state and the history of all past strategies of RSU j and BUS to RSU j. The price vector of RSU j at time step t and the bandwidth demand vector of all buses to RSU j are denoted as p j (t) and b j (t). The state of RSU j is determined by the historical records of the interactions between RSU j and the front L wheels of all buses. Therefore, at time step t, the state of RSU j can be expressed as s j (t) = {p j (tL),b j (tL),…,p j (t-1),b j (t-1)}.

[0149] (2) Action space: This embodiment uses To represent the action space of RSU j, at each time step t, RSUj decides its own action p j (t), this decision process depends on the observed state s j (t) The information contained in

[0150] (3) Reward function: This embodiment uses To represent the reward function of RSU j, that is, after the state changes, the reward obtained by RSU j depends on the current state s j (t) and the corresponding action p j (t). The reward function of RSU j is defined as the utility function of RSU j constructed in the Stackelberg game in this embodiment. At time step t, the reward function of RSU j is expressed as r j (t) = Vj ( p j (t),p -j (t),b j (t)). The average reward of all RSUs is

[0151] like Figure 4As shown in the figure, in the actor-critic network setting, there are two indispensable components, namely the actor network and the critic network. The PPO algorithm is a policy-based reinforcement learning algorithm that improves the policy by optimizing an alternative objective function. It can avoid training instability caused by excessive updates by limiting the step size of policy updates, while maintaining the simplicity and computational efficiency of the algorithm. In addition, the PPO algorithm can maintain a good balance between exploration and execution through its design. Many features make it perform well in applications in a variety of tasks and environments. In this embodiment, the A2C network in the MADRL framework is represented as In the PPO algorithm, for each RSU j, on the one hand, the actor network is a parameter θ j The decision function This function is used to generate the action of RSU j, that is, the pricing strategy p j , and promotes interaction with the environment. On the other hand, the critic network, a parameter ω j The value function It is used to evaluate the performance of the actor network (i.e., the agent) and guide the actor's actions in subsequent stages. Specifically, the main goal of the critic network is to minimize the time difference error, which is expressed as:

[0152]

[0153] in and Corresponding to the current state s j (t) and the subsequent state s j (t+)1 associated value function. Therefore, by minimizing the expected value of the square of the time difference error, the loss function of the critic network is obtained, which can be expressed as

[0154]

[0155] In addition, the objective function of the actor network is

[0156]

[0157] (b.3)

[0158] in is the strategy parameter used to sample p j ,and represents the strategy used for importance sampling. η(f j ) is an interval partitioning function with a given parameter:

[0159]

[0160] Among them, κ is an adjustable hyperparameter, Represents the estimated value of the advantage function, calculated as follows

[0161]

[0162] The MADRL framework uses classical neural networks, especially DNNs, to parameterize the A2C network. Considering the network structure, the actor and critic networks usually consist of a fully connected layer consisting of an input layer, multiple hidden layers, and an output layer containing a large number of parameters (such as neurons and weights). This example considers an actor network with H layers, and represents the weight in the hth fully connected layer as θ j (h), where h∈{1,2,…,H}. By setting the bias of the DNN to 0 and inputting the state s(t) at time step t into the first layer, the output of the first layer is calculated as

[0163] o j(1) =o j(1) (θ j(h) s j (t))#(49)

[0164] where σ j(1) represents the nonlinear response of the first layer, which is usually set to the ReLU function. The output data of each layer is propagated as input to the subsequent layers. Therefore, the output of the hth layer is represented as

[0165] o j(h) =σ j(h) (θ j(h) o j(h-1) )#(50)

[0166] Finally, the action at time step t output by the actor network is the price policy p j (t), as shown in the following formula:

[0167] p j (t) = σ j(H) (θ j(H) o j(H-1) )#(51)

[0168] Parameter update of A2C network

[0169] The loss function of the actor network is

[0170] p j (t) = σ j(H) (θ (H) o j(H-1 ))#(52)

[0171] When D records are accumulated in the playback buffer, the actor and critic networks are updated. Specifically, RSUj uses the gradient ascent method to update the parameters of the actor network, which can be expressed as

[0172]

[0173] in represents the learning rate used to train the actor network parameters, θ j(h)′ is the update parameter of the actor network. Moreover, the critic network parameters are updated by gradient descent, which can be expressed as the following formula

[0174]

[0175] in represents the learning rate used in the critic network training process, ω j(h)′ Represents the update parameters of the critic network.

[0176] Based on the above solution, the present invention solves the personalized recommendation problem in the bus system by proposing a new communication resource optimization solution. The solution focuses on the problem of heterogeneous equipment in the bus system, and how to effectively allocate and optimize computing and communication resources in a dynamically changing environment to ensure the efficiency of data transmission and the quality of personalized services.

[0177] 1. In view of the strong heterogeneity of vehicle equipment and the inconsistent model aggregation capabilities, sufficient computing resources and broadband resources are selected to help buses train local models. In the actual bus movement process, the coverage range and the required bandwidth of the channel directly affect the selection of vehicle pairs. For this reason, this embodiment establishes a Multi-Leader Multi-Follower Stackelberg game between buses and to help each bus make the best choice and offload the task to the appropriate one. When the vehicle is moving at high speed, it is impossible to maintain a connection with the same bus continuously. Therefore, it is necessary to reselect the model training and migrate the partially trained model from the previous one. After the new round of Stackelberg game is completed, the old and new establish a migration connection, and the old model issued and the newly obtained model complete the model migration between them.

[0178] 2. A fast-converging MALPPO algorithm is designed to verify the uniqueness and equilibrium solution of the mutual selection between buses and. This embodiment uses the MALPPO algorithm in MADRL to solve the equilibrium solution of the game model. The multi-agent game can effectively simulate the game in the multi-head resource market, and realize that the agent can perceive the changes in the environment in a timely manner, quickly and adaptively formulate communication resource optimization strategies, and has strong robustness in dealing with random states.

[0179] A vehicle network resource scheduling system for public transportation, comprising:

[0180] The training game model building module is used to build a Stackelberg game model with RSU as the leader and the bus as the follower;

[0181] The multi-agent reinforcement learning module is based on the Stackelberg game model and integrates MALPPO multi-agent deep reinforcement learning to output the optimal resource allocation strategy.

[0182] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0183] A vehicle networking resource scheduling device for public transportation:

[0184] at least one processor;

[0185] at least one memory for storing at least one program;

[0186] When the at least one program is executed by the at least one processor, the at least one processor implements the vehicle network resource scheduling method for public transportation as described above.

[0187] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0188] A storage medium stores processor-executable instructions, which are used to implement a vehicle network resource scheduling method for public transportation when executed by the processor.

[0189] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0190] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A vehicle network resource scheduling method for public transportation, characterized in that: The following steps are involved: With RSU as the leader and bus as the follower, a Stackelberg game model is constructed; Based on the Stackelberg game model, MALPPO multi-agent deep reinforcement learning is integrated to output the optimal resource allocation strategy.

2. A vehicle network resource scheduling method for public transportation according to claim 1, characterized in that: The Stackelberg game model has the following two stages: In the first stage, RSU selects a broadband pricing strategy; In the second stage, the bus observes the strategy selected by the RSU and responds accordingly based on its own needs.

3. A vehicle network resource scheduling method for public transportation according to claim 2, characterized in that: In the second stage, the utility function of the bus is: in, represents the utility function of the i-th bus to the j-th RSU, represents the satisfaction level of the i-th bus with the j-th RSU, represents a quantitative measure of QoE for passengers, represents the loan pricing of the j-th RSU for the i-th bus, represents the bandwidth demand of the i-th bus for the j-th RSU, represents the signal-to-noise ratio transmitted from the jth RSU to the i-th bus, represents the spectral efficiency of the jth RSU transmitting data to the ith bus, D i represents the task volume of the personalized federated recommendation system model in the i-th bus, represents the data processing quality before the jth RSU transmits to the i-th bus, It represents the minimum required lower limit of the model service quality supported by the RSU after the i-th bus offloads the personalized federated recommendation system model to the RSU.

4. A vehicle network resource scheduling method for public transportation according to claim 3, characterized in that: In the first stage, the utility function of RSU is: Among them, V j represents the utility function of the jth RSU, c j represents the bandwidth cost of the jth RUS, and I represents the set of buses.

5. A vehicle network resource scheduling method for public transportation according to claim 4, characterized in that: In the Stackelberg game model, there are the following limitations: Phase 2: In the second stage, the bus determines the optimal bandwidth purchasing strategy that maximizes its utility by considering the bandwidth price strategy vector published by the RUS; Phase 1: The non-cooperative game between RSUs has a Nash equilibrium; The multi-leader multi-follower Stackelberg game between RSU and bus has Stackelberg equilibrium; The Stackelberg equilibrium between the RSU and the bus is unique.

6. A vehicle network resource scheduling method for public transportation according to claim 5, characterized in that: The step of outputting the optimal resource allocation strategy based on the Stackelberg game model and integrating MALPPO multi-agent deep reinforcement learning specifically includes: Define state space, action space, and reward function; At each time step, the RSU decides on an action based on the current state, and the environment gives a corresponding reward; Put the state, action and reward into the cache pool, and update the actor network and critic network in combination with the updated strategy function, value function and loss function; Extract the information related to RSU from the buffer queue, generate a new state, and enter the next time step; Iterate until the simulation results fit and are close to the theoretical game equilibrium solution, and output the optimal resource allocation strategy.

7. A vehicle network resource scheduling system for public transportation, characterized in that: include: The game model building module is used to build a Stackelberg game model with RSU as the leader and the bus as the follower; The multi-agent reinforcement learning module is based on the Stackelberg game model and integrates MALPPO multi-agent deep reinforcement learning to output the optimal resource allocation strategy.

Images