Wireless resource allocation method and related devices based on multi-agent reinforcement learning and dynamic graph learning

Abstract

This invention belongs to the field of wireless resource allocation technology and discloses a wireless resource allocation method and related apparatus based on multi-agent reinforcement learning and dynamic graph learning. The wireless resource allocation method includes: at each wireless resource allocation time step, constructing a heterogeneous temporal graph using channel states and communication relationships in the vehicular network environment; performing dynamic graph representation learning on the heterogeneous temporal graph using a heterogeneous temporal graph neural network, obtaining the embedding features of each node through intra-relationship aggregation, inter-relationship aggregation, and cross-time aggregation; and making decisions based on the embedding features of each node using a multi-agent deep deterministic policy gradient reinforcement learning model to obtain a wireless resource allocation scheme. This invention can extract heterogeneous and temporal information beneficial to current decision-making from historical channel state information to support resource allocation decisions, thereby improving the reliability, stability, and overall service quality of vehicular network communication.

Description

Technical Field

[0001] This invention belongs to the field of wireless resource allocation technology, and specifically relates to a wireless resource allocation method and related apparatus based on multi-agent reinforcement learning and dynamic graph learning. Background Technology

[0002] Next-generation wireless communication networks face the enormous challenge of surging mobile data demand. The dense deployment of smart devices and the scarcity of wireless resources restrict the improvement of wireless network communication performance. Therefore, designing wireless resource allocation algorithms to achieve better dynamic allocation of communication network resources is particularly important for further improving the communication performance of wireless communication networks.

[0003] The wireless resource allocation problem can be modeled as a mathematical optimization problem, but such optimization problems typically exhibit characteristics such as non-convexity, multi-objective nature, and mixed-integer nonlinear programming, making them difficult to solve effectively using traditional convex optimization methods. Given this, previous research has often employed other mathematical optimization methods to obtain suboptimal solutions to the wireless resource allocation problem. Deep Reinforcement Learning (DRL), compared to traditional mathematical optimization methods, has lower complexity and better dynamic adaptability, enabling it to find approximate optimal solutions to the wireless resource allocation problem in dynamic network environments. It is now widely used in the field of wireless communication network resource allocation.

[0004] Despite the numerous advantages of deep reinforcement learning in wireless resource allocation tasks, two key issues remain to be addressed in its application in Vehicle-to-Everything (V2X) scenarios. First, in real-world V2X communication scenarios, accurate Channel State Information (CSI) is often unavailable, forcing reliance on historical CSI for resource allocation decisions. This can lead to a disconnect between resource allocation decisions and real-time channel conditions, resulting in degraded communication link quality. Furthermore, due to the high-speed movement of vehicles, the acquired CSI becomes outdated rapidly, making accurate real-time CSI virtually impossible to obtain in high-mobility scenarios. Second, future V2X systems involve platooning, which involves many different types of nodes and links, resulting in heterogeneous communication networks and further increasing the difficulty of wireless resource allocation. This network heterogeneity makes it difficult to adapt resource allocation to the differentiated needs of various nodes and links, leading to low resource utilization and potentially reducing the reliability, stability, and overall service quality of V2X communication. Summary of the Invention

[0005] The purpose of this invention is to provide a wireless resource allocation method and related apparatus based on multi-agent reinforcement learning and dynamic graph learning to solve one or more of the aforementioned technical problems. The technical solution disclosed in this invention can extract heterogeneous and temporal information beneficial to current decision-making from historical channel state information to support resource allocation decisions, thereby improving the reliability, stability, and overall service quality of vehicle-to-everything (V2X) communication.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning, comprising the following steps: At each wireless resource allocation time step, a heterogeneous time sequence graph is constructed using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In the heterogeneous time sequence graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. A heterogeneous temporal graph neural network is used to perform dynamic graph representation learning on the heterogeneous temporal graph. By processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-temporal aggregation, the embedding features of each node are obtained. Based on the embedding features of each node, a multi-agent deep deterministic policy gradient reinforcement learning model is used to make decisions and obtain a wireless resource allocation scheme. This model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents make distributed decisions based on their respective observation spaces to allocate channels and transmit power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives. A reward function is constructed, and the model is centrally trained and its parameters are updated until convergence to obtain the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

[0007] A further improvement to the technical solution of this invention lies in the step of constructing a heterogeneous timing graph using the channel state and communication relationships in the vehicle-to-everything (V2X) environment at each wireless resource allocation time step. The specific definition of a heterogeneous time series diagram is: ; ; In the formula, yes Time-series heterogeneous graph; yes The set of nodes in a time-varying heterogeneous graph; yes The set of edges in a time-varying heterogeneous graph; It is a heterogeneous timing diagram; It is the size of the time window; It is a set of edges spanning time, used to describe Time-based heterogeneous graphs and Relationships between heterogeneous graphs at different times.

[0008] A further improvement to the technical solution of this invention lies in that each node feature includes the channel gain of its associated link, specifically... The characteristics of the team leader node are represented as follows: ; In the formula, for t Time, number n The feature vector of the fleet leader node; k Channel number, The set of all channels; for t Time, Number k On each channel, numbered as n From the fleet leader node to the base station node I Channel gain of L2I link; m Number the team members, A collection of all team members; for t Time, Number k On each channel, numbered as n The channel gain of the L2M link from the fleet leader node to the fleet member node m within the same fleet; The characteristics of the team member nodes are represented as follows: ; In the formula, for t Time, number m The feature vectors of the team member nodes; A collection of all team leader nodes; The characteristics of a base station node are represented as follows: ; In the formula, for tTime base station node I eigenvectors.

[0009] A further improvement to the technical solution of this invention lies in the step of using a heterogeneous temporal graph neural network to perform dynamic graph representation learning on the heterogeneous temporal graph, and obtaining the embedding features of each node through processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-temporal aggregation, including: A linear transformation is performed on the original features of each type of node to unify the node feature dimensions and obtain the transformed features. Based on the transformed features, the attention scores between the target node and its neighboring nodes with the same relationship are calculated according to the multi-head attention mechanism. The embedding feature vector of the target node is obtained by weighting and aggregating the features of neighboring nodes based on the attention score.

[0010] A further improvement to the technical solution of this invention lies in the node. The specific process of calculating the embedded feature vector is as follows: ; In the formula, for t Time, Number l In a layered network, nodes v Embedded feature vectors; and Based on nodes v The type determines the learnable weights and transformation matrix. For nodes v Type; for t Time, Number l In a layered network, nodes v New features of the current layer obtained through aggregation within and between relations; For time t, the first In a layered network, nodes v The embedded feature vector.

[0011] A further improvement to the technical solution of the present invention is that the reward function is expressed as: ; In the formula, for t The global reward value at any given moment; , , These are all hyperparameters, representing the weight coefficients of the first, second, and third terms, respectively. A collection of all team leader nodes. n Assign a number to the team leader node; for t Time, numberedn The transmission rate of the L2I link from the fleet leader node to the base station node; For the number n The set of all team member nodes within the team leader node. m The node number for each member of the team; for t Time, numbered n The team leader node is numbered within the same team. m The reward value of the L2M link for each member node of the fleet; To limit the transmission delay of collaborative sensing information, To coordinate the remaining transmission time of the sensing information, .

[0012] A further improvement to the technical solution of the present invention lies in that, during the calculation of the reward function, t Time, numbered n The team leader node is numbered within the same team. m Reward value of L2M link for fleet member nodes The calculation expression is: ; In the formula, represent t Time, numbered n The team leader node is numbered within the same team. m The remaining transmission capacity of the L2M links of the fleet member nodes; This is a fixed reward value after the transmission is completed; for t Time, numbered n The team leader node to the team with the number m The transmission rate of the L2M link of the fleet member nodes.

[0013] A second aspect of the present invention provides a wireless resource allocation system based on multi-agent reinforcement learning and dynamic graph learning, comprising: A heterogeneous timing graph construction module is used to construct a heterogeneous timing graph at each wireless resource allocation time step using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In this heterogeneous timing graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. The node embedding feature acquisition module is used to perform dynamic graph representation learning on the heterogeneous temporal graph using a heterogeneous temporal graph neural network, and obtain the embedding features of each node through processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-time aggregation. The decision-making module is used to make decisions based on the embedded features of each node using a multi-agent deep deterministic policy gradient reinforcement learning model to obtain a wireless resource allocation scheme. The multi-agent deep deterministic policy gradient reinforcement learning model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents make distributed decisions based on their respective observation spaces to allocate channels and transmit power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives. A reward function is constructed, and the multi-agent deep deterministic policy gradient reinforcement learning model is centrally trained and its parameters are updated until convergence to obtain the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

[0014] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning as described in any one of the first aspects of the present invention.

[0015] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning as described in any one of the first aspects of the present invention.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a vehicle-to-everything (V2X) resource allocation algorithm based on heterogeneous temporal graph neural network (HTGNN) and multi-agent reinforcement learning. This algorithm can capture heterogeneous information in the communication network and make predictions using historical channel states even in the absence of real-time channel states, thus effectively modeling the "heterogeneous" and "time-varying" characteristics of V2X. Specifically, the technical solution disclosed in this invention first uses a heterogeneous temporal graph neural network (HTGNN) to learn dynamic graph representations of the V2X communication network. The learned node embeddings provide observations for each agent in the subsequent multi-agent reinforcement learning. Then, multi-agent reinforcement learning is performed using multi-agent deep deterministic policy gradient (MADDPG), employing two types of agents to allocate channels and power for different types of links. To further explain, addressing the issue of "decision-making disconnect due to lack of real-time channel state information," this invention employs a heterogeneous temporal graph neural network (HTMN) to construct a heterogeneous temporal graph by integrating historical CSI with the vehicle network topology and temporal relationships. Through intra-relationship, inter-relationship, and cross-time aggregation, it mines the spatiotemporal patterns in historical data, generating accurate node embeddings to replace real-time CSI as the basis for decision-making, thus avoiding link quality degradation caused by outdated CSI. Furthermore, addressing the issue of "poor resource allocation adaptability due to vehicle network heterogeneity," this invention designs two types of heterogeneous agents. Based on the node embedding features output by HTGNN, they are adapted to L2I (high-speed requirement) and L2M (high-reliability requirement) links, respectively. Through centralized training and distributed execution of MADDPG, channel and power allocation strategies are customized for different types of links, improving resource utilization. This solves the problem of insufficient communication reliability and stability caused by network heterogeneity, ultimately achieving an overall improvement in the quality of vehicle network services. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning, as described in an embodiment of the present invention.

[0019] Figure 2This is a schematic diagram of a vehicle-to-everything (V2X) wireless resource allocation framework based on multi-agent reinforcement learning and dynamic graph learning in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of a wireless resource allocation system based on multi-agent reinforcement learning and dynamic graph learning in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0023] Example 1 Please see Figure 1 The present invention provides a wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning, comprising the following steps: Step 1: At each wireless resource allocation time step, a heterogeneous time sequence graph is constructed using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In the heterogeneous time sequence graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. Step 2: Use a heterogeneous temporal graph neural network to perform dynamic graph representation learning on the heterogeneous temporal graph. By processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-temporal aggregation, the embedding features of each node are obtained. Step 3: Based on the embedding features of each node, a multi-agent deep deterministic policy gradient reinforcement learning model is used to make decisions and obtain a wireless resource allocation scheme. The multi-agent deep deterministic policy gradient reinforcement learning model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents are used to make distributed decisions based on their respective observation spaces, allocating channels and transmission power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives, constructs a reward function, and performs centralized training and parameter updates on the model until convergence, thus obtaining the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

[0024] In the technical solution disclosed in this invention embodiment, at each resource allocation time step, the resource allocation framework constructs a heterogeneous time-series graph using the channel state and communication relationships in the vehicle-to-everything (V2X) environment, and encodes the environmental information using a dynamic graph neural network. Each agent in the resource allocation framework receives the encoded information from the dynamic graph neural network and uses it as the agent's state to make decisions, thereby allocating transmission channels and transmit power to each link in the V2X. The technical solution disclosed in this invention embodiment can extract heterogeneous and time-series information beneficial to current decisions from historical channel state information to support resource allocation decisions, thereby improving the reliability, stability, and overall service quality of V2X communication. Example 2 Please see Figure 2 In this embodiment of the invention, a dynamic vehicle-to-everything (V2X) environment with a fleet of vehicles is considered. It is assumed that there is a base station in the center of the area, and the area contains... There are 10 teams, and each team has a team leader (PL). Each team contains There are 100 Platoon Members (PMs), and the total number of PMs across all teams is 100. Let the PL set be... , No. The PM collection of each team is All team PMs gathered as Each PL (Physical Provider) sends a Cooperative Awareness Message (CAM) via multicast to all PMs within the fleet to facilitate safety information communication between vehicles; this type of link is denoted as L2M (Level 2 Message). Simultaneously, the PL also transmits entertainment information to the base station; this type of link is denoted as L2I (Level 2 Information). High reliability is required for safety information communication between the PL and PMs, while high data rate is required for entertainment information transmitted from the PL to the base station.

[0025] Assuming the scenario contains There are 10 available channels, denoted as the channel set. All L2M links within the same fleet use the same channel and transmit power; two types of binary variables are defined. and These are used to identify whether a link is using a channel. : (1) (2) In the formula, , These are used to identify the first L2I link, the first Does the L2M link use a channel? binary variables.

[0026] In vehicle-to-everything (V2X) communication, each L2I link needs to consider interference from L2M links using the same channel and other L2I links; among them, the first The L2I link in the 1st The signal-to-interference-plus-noise ratio (SINR) on each channel can be expressed as: (3) In the formula, For the first The L2I link in the 1st Signal-to-interference-to-noise ratio on each channel; Representing the team The transmit power of PL on the corresponding L2I link, Representing the team The transmit power of PL on the corresponding L2I link, Representing the team The PL transmits the CAM's transmit power to the PM within the same vehicle team. Representing the team The PL and the base station in the channel Channel gain on Focus on the interference channel gain of the PL (Power Line Controller) of different vehicle groups at the base station, which represents the vehicle group. The PL and the base station in the channel The interference channel gain on This represents noise power.

[0027] Each L2M link needs to consider interference from other convoy L2M links using L2I links on the same channel; similarly, the links numbered as follows... PL and the team's number Links between PMs in the channel SINR can be expressed as: (4) In the formula, Representative number is The PL transmits the CAM's transmit power to the PM within the same vehicle team. Focus on the interference channel gain between different vehicle convoys, represented by the number [missing information]. The PM and the different vehicle number are PLs in the channel Interference channel gain.

[0028] According to the L2I link in the channel The channel capacity can be calculated from the SINR. (5) Similarly, based on the L2M link in the channel The channel capacity can be calculated from the SINR. (6) in, This represents the transmission bandwidth.

[0029] In the above scenario, it is necessary to allocate channels and transmit power for each L2M and L2I link, while maximizing the transmission rate of L2I and the transmission success rate of L2M. Therefore, the optimization objective of this invention can be expressed as: (7) In the formula, Representing the team The size of the CAM data packets that the PL needs to transmit. Represents the length of the time step. This represents the total number of time steps required to reach the CAM transmission delay constraint.

[0030] In the context of vehicle-to-everything (V2X) networks, the rapid movement of vehicles leads to rapid changes in channel states. These rapid changes are closely related to the topology and temporal relationships of the V2X network. In practical V2X scenarios, obtaining accurate real-time channel states is difficult; therefore, it is necessary to utilize historical V2X channel states to predict the current channel state to aid resource allocation decisions. The technical solution disclosed in this invention constructs a heterogeneous temporal graph from historical channel states and learns these states using dynamic graph learning to capture the heterogeneous topology and temporal information of the V2X channel, providing rich state information for current resource allocation. In the heterogeneous graph, this invention models PL, PM, and base station as three different types of nodes. The transmission link from PL to PM within the same vehicle fleet, the interference link from PL to PM in a different vehicle fleet, and the link from PL to base station are modeled as three different types of edges. Each node in the heterogeneous graph contains the channel gain of its associated links. For base station nodes, its... The time node characteristics include all L2I links at The channel gain on each channel has the following dimension: Then the characteristics of the base station node can be represented as follows: (8) For each PL node ,That The time node characteristics include the corresponding L2I transmission link and its connection with the PMs of all fleets. The channel gain on each channel has the following dimension: PL node The characteristics can be represented as: (9) For each PM node ,That The time node feature includes its PL with all teams in The channel gain on each channel has the following dimension: PM node The characteristics can be represented as: (10) This invention represents a heterogeneous graph at multiple historical moments as a heterogeneous time series graph. A heterogeneous time series graph can be defined as: (11) (12) in, It is the size of the time window. yes heterogeneous graph at time, yes The set of points in a time-varying heterogeneous graph. yes The edge set of a time-varying heterogeneous graph. It is a set of edges spanning time, which describes heterogeneous graphs at time and Relationships in heterogeneous time-series graphs. In the heterogeneous time-series graph constructed in this invention, node features change frequently over time.

[0031] Heterogeneous time series graphs can be used for graph representation learning using HTGNN; interpretably, HTGNN is a discrete-time dynamic graph learning method that can perform dynamic graph learning for heterogeneous graphs at multiple time points. It can simultaneously model the heterogeneous graph structure and temporal dynamic changes in complex channel scenarios, and better utilize historical channel states to fit the state representation at the current time point.

[0032] HTGNN includes three aggregation methods: intra-relational aggregation, inter-relational aggregation, and cross-time aggregation.

[0033] Before performing relational aggregation, a linear transformation layer is needed to ensure that all types of nodes have the same dimension. The transformation can be represented as: (13) in, yes Time Node The original characteristics, yes After linear transformation, the node Features Representative node Node type, It is a learnable transformation matrix.

[0034] For a target node, its different neighbor nodes, even in the same type of relationship, will contribute differently during the learning process. Therefore, an attention coefficient is assigned to each neighbor. Time Node In relation The set of neighboring nodes on is ,for , Nodes and The attention score between nodes can be represented as: (14) (15) in, It is the LeakyReLU function. This represents the network layer number where the operation takes place. and These are the learnable transformation matrix and the attention vector, respectively. represent Time Node In the Layer embedding, This represents the vector concatenation operation. After obtaining the attention scores of all neighboring nodes, the target node's attention score can be calculated through the aggregation operation. The updated embedding. Time Node In relation The first Layer embedding can be represented as: (16) in, This represents the first in the multi-head attention mechanism. One point of attention, This represents the total number of attention heads.

[0035] Through intra-relation aggregation, the target node collects embeddings from multiple relations. Based on this, inter-relation aggregation aims to fuse the target node's embeddings across different relations, which also requires learning multiple attention scores to measure the weight of each relation. The calculation of inter-relation attention scores can be expressed as: (17) in, represent Always being in relationships The set of connected nodes , and These are the learnable transformation matrix, attention vector, and bias vector, respectively.

[0036] After obtaining the attention scores for all relations, the updated embedding of the target node can be calculated through aggregation operations. Time Node The Layer embedding can be represented as: (18) in, Representatives and nodes The set of all connected relationships.

[0037] Before performing cross-time aggregation, it is necessary to encode the location of the node features at each time step. This operation can be represented as: (19) in, Represents a moment, The dimension of the embedding vector. An index representing one of the dimensions. It is a frequency encoding function that represents a time-dependent sine wave, when... When it is an odd number ,when When it is even .

[0038] To measure the cross-time importance of nodes, the Transformer attention mechanism is used. The computation of the query vector and key vector can be represented as: (20) (twenty one) in, and Representing two different moments, and It is a learnable matrix.

[0039] After obtaining the query vector and key vector, the node At the present moment With another moment The attention score across time can be expressed as: (twenty two) Based on the attention scores across time, the embeddings of nodes across time can be aggregated to obtain... The cross-time embedding representation of layer nodes, and the aggregation computation, can be represented as: (twenty three) in, It is a fixed time window size.

[0040] After obtaining the cross-time node embedding representation, it is then fused with the node embedding representation of the previous layer to obtain the final node embedding representation of that layer. The computation process of the final embedded representation can be represented as follows: (twenty four) in, and These are the learnable weights and the transformation matrix, respectively.

[0041] In the vehicle-to-everything (V2X) scenarios involved in the invention, at any time Only know the past The channel state at each moment, i.e., only the previous moment's channel state can be obtained. The heterogeneous graph at different time points maintains the same topology, but due to the rapid movement of vehicles causing frequent changes in channel state, node characteristics change frequently. Using HTGNN to learn from the heterogeneous graph at historical time points can capture the changes in channel state over time while learning the heterogeneous topology information of the graph, serving as a basis for subsequent resource allocation.

[0042] In this embodiment of the invention, based on the graph node embedding output by the above-mentioned HTGNN, the MADDPG multi-agent reinforcement learning method is used to realize the resource allocation task to perform channel allocation and power allocation of L2I links and L2M links in the Internet of Vehicles, so as to maximize the rate of L2I links and maximize the reliability of L2M links.

[0043] This invention designs two types of intelligent agents to jointly allocate resources in the vehicle-to-everything (V2X) network. Each L2I link corresponds to a first-type intelligent agent, responsible for resource allocation within that link; this agent is called the L2I agent. Each vehicle fleet corresponds to a second-type intelligent agent, responsible for resource allocation across all L2M links within that fleet; this agent is called the L2M agent. The two types of agents have different observation spaces and use different policy networks. Because L2I and L2M links have different communication optimization requirements within the network, this approach can adapt to network heterogeneity and learn more adaptive resource allocation strategies for different types of links.

[0044] At time step , No. The observation space of an L2I agent can be represented as: (25) in, and Representing the current time step base station node and PL node respectively. The node embeddings obtained after learning by HTGNN The base station was in a state of emergency. Interference experienced on each channel.

[0045] At time step , No. The observation space of an L2M agent can be represented as: (26) in, Represents the current time of the marching team The node embeddings obtained by learning the PL nodes through HTGNN. Represents the current time of the marching team The set of node embeddings obtained by learning all PM nodes within the HTGNN. Representing the previous time of the infantry team The set of disturbances experienced by all PM nodes within the system. Represents the current time of the marching team The set of CAM transmissions for all L2M links within the network. Represents the current time of the marching team The set of remaining CAM transmission times for all L2M links within the network.

[0046] The action space of each L2I agent is defined as That is, the first Each L2I link is allocated channels and power. The action space of each L2M agent is defined as... That is, a car team All L2M links within the same region are uniformly allocated the same channel and power. Because this invention uses MADDPG, the agent can select power within a continuous interval, demonstrating the advantage of MADDPG's ability to use a continuous action space, resulting in more accurate convergence than traditional DQN which uses a discrete action space.

[0047] The goal of reinforcement learning is to allocate resources between L2I and L2M links in an environment to maximize the transmission rate of L2I links and the transmission success rate of L2M links. For fleet management systems... L2M links in Its reward can be defined as: (27) in, This represents the remaining transmission capacity of the L2M transmission link. If the transmission on the L2M link has not yet ended, the reward value is set to the transmission rate of the link.

[0048] If the transmission on the L2M link is complete, a fixed reward value is given. The total reward function can be expressed as: (28) in, , , All of these are hyperparameters. Represents the transmission delay limit of CAM. Represents the remaining transmission time of the CAM. , The first term represents the time already transmitted. The third term serves as a penalty; the closer the transmitted time gets to the latency limit, the greater the penalty, urging the algorithm to complete the CAM transmission as quickly as possible.

[0049] In MADDPG, each agent employs an Actor-Critic network architecture. The Actor network outputs actions, and the Critic network outputs Q-values. Both the Actor and Critic networks have a target network with identical structure. MADDPG is characterized by Centralized Training and Decentralized Execution (CTDE). At each time step of the execution process, each agent selects an action based on its own policy network, receives feedback from the environment, and stores an experience record in the experience replay pool. ,in Represents joint observations by all intelligent agents. Represents the joint action of all intelligent agents. Representing intelligent agents The rewards received Representing intelligent agents The next observation. During the training phase, a batch is randomly sampled from the experience replay pool, and the network parameters are updated. The loss function of a Critic network for an agent can be expressed as: (29) (30) in, and Representing intelligent agents The Critic and the Target Critic and Representing intelligent agents The parameters of the Critic network and the parameters of the target network. This represents the discount factor. And it's used to update the agent. The gradient of the Actor network can be expressed as: (31) in, Representing intelligent agents Actor network parameters, Representative strategy.

[0050] For the target network of Actor and Critic, a soft update method is used, and its parameter update method can be expressed as: (32) (33) in, It is a hyperparameter that can be used to adjust the magnitude of the target network parameter updates.

[0051] The technical solutions disclosed in the embodiments of the present invention have the following substantial improvements and significant advancements compared to traditional methods: (1) Capturing the spatiotemporal characteristics of wireless networks based on dynamic graph learning; Explained, traditional vehicle network resource allocation schemes do not fully model the heterogeneous and temporal characteristics of vehicle networks; This invention fully captures the spatiotemporal characteristics of vehicle networks by introducing dynamic graph learning methods, providing richer environmental information for resource allocation decisions.

[0052] (2) Resource allocation decision based on heterogeneous multi-agent reinforcement learning; Explained, traditional multi-agent reinforcement learning schemes usually only model a single type of multi-agent, but in the vehicular network based on formation, there are multiple conflicting optimization objectives in the network, which are difficult to optimize at the same time; This invention introduces a heterogeneous agent scheme, using different types of agents to optimize different objectives, which effectively improves the resource allocation performance of this multi-objective optimization problem.

[0053] (3) A joint learning framework combining dynamic graph learning and multi-agent reinforcement learning; Explained, the technical solution of this embodiment of the invention designs a joint learning framework combining dynamic graph learning and multi-agent reinforcement learning. Specifically, dynamic graph learning is responsible for capturing the spatiotemporal features of the environment to update node embeddings, while the multi-agent uses the updated node embeddings as the agent's state for distributed resource allocation decisions. The two learn together during the training process. This invention's framework combines the advantages of dynamic graph learning in spatiotemporal feature learning with the advantages of multi-agent reinforcement learning in distributed dynamic decision-making.

[0054] Example 3 The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0055] Please see Figure 3 In this embodiment of the invention, a wireless resource allocation system based on multi-agent reinforcement learning and dynamic graph learning is provided, comprising: A heterogeneous timing graph construction module is used to construct a heterogeneous timing graph at each wireless resource allocation time step using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In this heterogeneous timing graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. The node embedding feature acquisition module is used to perform dynamic graph representation learning on the heterogeneous temporal graph using a heterogeneous temporal graph neural network, and obtain the embedding features of each node through processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-time aggregation. The decision-making module is used to make decisions based on the embedded features of each node using a multi-agent deep deterministic policy gradient reinforcement learning model to obtain a wireless resource allocation scheme. The multi-agent deep deterministic policy gradient reinforcement learning model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents make distributed decisions based on their respective observation spaces to allocate channels and transmit power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives. A reward function is constructed, and the multi-agent deep deterministic policy gradient reinforcement learning model is centrally trained and its parameters are updated until convergence to obtain the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

[0056] Example 4 In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to execute operations of a wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning.

[0057] Example 5 In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning in the above embodiments.

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning, characterized in that, Includes the following steps: At each wireless resource allocation time step, a heterogeneous time sequence graph is constructed using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In the heterogeneous time sequence graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. A heterogeneous temporal graph neural network is used to perform dynamic graph representation learning on the heterogeneous temporal graph. By processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-temporal aggregation, the embedding features of each node are obtained. Based on the embedding features of each node, a multi-agent deep deterministic policy gradient reinforcement learning model is used to make decisions and obtain a wireless resource allocation scheme. This model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents make distributed decisions based on their respective observation spaces to allocate channels and transmit power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives. A reward function is constructed, and the model is centrally trained and its parameters are updated until convergence to obtain the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

2. The method of claim 1, wherein, In each wireless resource allocation time step, during the step of constructing a heterogeneous timing graph using channel states and communication relationships in the vehicular network environment, The specific definition of a heterogeneous time series diagram is: ； ； wherein, is a time-heterogeneous graph; is a set of nodes of the time-heterogeneous graph; is a set of edges of the time-heterogeneous graph; is a heterogeneous temporal graph; is a time window size; is a set of edges across time, used to describe a relationship between the time-heterogeneous graph and a relationship between the time-heterogeneous graph and 3. The method of claim 1, wherein, Each node's characteristics include the channel gain of its associated link, specifically... The characteristics of the team leader node are represented as follows: ； In the formula, is t the feature vector of the fleet leader node numbered n at the moment; i k is the channel number, is the set of all channels; is t is the time instant, the k channel number, the channel gain of the L2I link from the platoon leader node with index n to the base node I ; m is the platoon member number, is the set of all platoon members; is t is the time instant, the k channel number, the channel gain of the L2M link from the platoon leader node with index n to the platoon member node m within the same platoon; The characteristics of the team member nodes are represented as follows: ； wherein is t the time instant, the feature vector of the fleet member node numbered m is the set of all fleet leader nodes​ The characteristics of a base station node are represented as follows: ； In the formula, for t Time base station node I eigenvectors.

4. The wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning according to claim 1, characterized in that, The steps for learning dynamic graph representations of the heterogeneous time series graph using a heterogeneous temporal graph neural network, and obtaining the embedding features of each node through intra-relational aggregation, inter-relational aggregation, and cross-temporal aggregation, include: A linear transformation is performed on the original features of each type of node to unify the node feature dimensions and obtain the transformed features. Based on the transformed features, the attention scores between the target node and its neighboring nodes with the same relationship are calculated according to the multi-head attention mechanism. The embedding feature vector of the target node is obtained by weighting and aggregating the features of neighboring nodes based on the attention score.

5. A wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning according to claim 4, characterized in that, node The specific process of calculating the embedded feature vector is as follows: ； In the formula, for t Time, Number l In a layered network, nodes v Embedded feature vectors; and Based on nodes v The type determines the learnable weights and transformation matrix. For nodes v Type; for t Time, Number l In a layered network, nodes v New features of the current layer obtained through aggregation within and between relations; For time t, the first In a layered network, nodes v The embedded feature vector.

6. The wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning according to claim 1, characterized in that, The reward function is expressed as follows: ； In the formula, for t The global reward value at any given moment; , , These are all hyperparameters, representing the weight coefficients of the first, second, and third terms, respectively. A collection of all team leader nodes. n Assign a number to the team leader node; for t Time, numbered n The transmission rate of the L2I link from the fleet leader node to the base station node; For the number n The set of all team member nodes within the team leader node. m The node number for each member of the team; for t Time, numbered n The team leader node is numbered within the same team. m The reward value of the L2M link for each member node of the fleet; To limit the transmission delay of collaborative sensing information, To coordinate the remaining transmission time of the sensing information, .

7. A wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning according to claim 6, characterized in that, During the calculation of the reward function, t Time, numbered n The team leader node is numbered within the same team. m Reward value of L2M link for fleet member nodes The calculation expression is: ； In the formula, represent t Time, numbered n The team leader node is numbered within the same team. m The remaining transmission capacity of the L2M links of the fleet member nodes; This is a fixed reward value after the transmission is completed; for t Time, numbered n The team leader node is numbered within the same team. m The transmission rate of the L2M link of the fleet member nodes.

8. A wireless resource allocation system based on multi-agent reinforcement learning and dynamic graph learning, characterized in that, include: A heterogeneous timing graph construction module is used to construct a heterogeneous timing graph at each wireless resource allocation time step using the channel state and communication relationships in the vehicle-to-everything (V2X) environment. In this heterogeneous timing graph, the fleet leader node, fleet member nodes, and base station nodes are three different types of nodes, and the transmission link from the fleet leader to a member within the same fleet, the interference link from the fleet leader to a member of another fleet, and the link from the fleet leader to the base station are three different types of edges. Each node feature includes the channel gain of its associated link. The node embedding feature acquisition module is used to perform dynamic graph representation learning on the heterogeneous temporal graph using a heterogeneous temporal graph neural network, and obtain the embedding features of each node through processing methods such as intra-relational aggregation, inter-relational aggregation, and cross-time aggregation. The decision-making module is used to make decisions based on the embedded features of each node using a multi-agent deep deterministic policy gradient reinforcement learning model to obtain a wireless resource allocation scheme. The multi-agent deep deterministic policy gradient reinforcement learning model includes two types of agents: an L2I agent corresponding to each convoy leader's link to the base station and an L2M agent corresponding to each convoy. The L2I and L2M agents make distributed decisions based on their respective observation spaces to allocate channels and transmit power to the corresponding links. The multi-agent deep deterministic policy gradient reinforcement learning model uses maximizing the L2I link transmission rate and maximizing the L2M link transmission success rate as optimization objectives. A reward function is constructed, and the multi-agent deep deterministic policy gradient reinforcement learning model is centrally trained and its parameters are updated until convergence to obtain the wireless resource allocation scheme. Among them, the link type for transmitting entertainment information from the fleet leader to the base station is denoted as L2I link; the link type for sending collaborative perception information from the fleet leader to all fleet members in the fleet via multicast is denoted as L2M link.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the wireless resource allocation method based on multi-agent reinforcement learning and dynamic graph learning as described in any one of claims 1 to 7.

Images