An information age optimization method and system based on graph reinforcement learning for internet of vehicles

Abstract

The application discloses a kind of vehicle networking information age optimization method and system based on graph reinforcement learning, first constructs the batch modeling of vehicle state and limited buffer queue structure, and the perception data is modeled as multiple data packet batches and is managed in queue;Graph neural network is used to model V2V link topology, and large-scale channel embedding representation reflecting topological structure is extracted;A multi-agent reinforcement learning system based on centralized training and distributed execution framework is constructed, each agent makes decisions according to the local state containing graph embedding features, outputs a hybrid action space, and receives the inverse number of end AoI as a reward;Introduce the graph embedding supervision mechanism based on advantage function, align the topological features with the long-term optimization goal;Through multiple rounds of training to update network parameters, finally realize the autonomous optimization control of each agent to packet loss and power.The application can effectively coordinate packet queue management and wireless resource allocation, and realize efficient and low-overhead AoI minimization in complex dynamic topology environment.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of wireless communication, intelligent transportation, and artificial intelligence. Specifically, it relates to a method and system for optimizing the information age of vehicle networks based on graph reinforcement learning, which is used to jointly optimize queue management and power control in order to minimize the information age index (AoI). Background Technology

[0002] With the rapid development of vehicle-to-everything (V2X) and autonomous driving technologies, efficient transmission of vehicle-to-vehicle status information has become a key support for achieving collaborative perception and decision-making. Vehicle-to-vehicle (V2V) communication enables low-latency information sharing between vehicles, which is of great significance for improving functions such as collision avoidance, lane cooperation, and platoon control. However, traditional communication performance indicators (such as throughput, latency, and packet loss rate) are insufficient to accurately characterize the "freshness" of vehicle perception information, making it difficult to meet the time-sensitive decision-making requirements.

[0003] Therefore, Age of Information (AoI) has been proposed to measure the time delay difference between the received information and the actual state, and it is a novel performance indicator widely used in current information update systems. Existing AoI optimization methods mostly rely on traditional model-driven mechanisms (such as linear programming and queuing analysis), which are significantly limited in real-world, highly dynamic vehicle-to-everything (V2X) environments. Furthermore, perception data typically consists of high-dimensional information (such as radar, camera, and lidar data), often requiring splitting into multiple packets for transmission, significantly increasing the complexity of queue management.

[0004] In recent years, reinforcement learning (RL) methods have been used for dynamic communication resource scheduling, but their expressive power in complex V2V graph structures is limited, making it difficult to capture topology-aware behavioral patterns. Graph neural networks (GNNs) provide a natural way to model V2V topologies, but how to deeply integrate them with reinforcement learning remains a challenge, especially in practical deployments where high communication overhead and large information latency are problems. Therefore, there is an urgent need for an efficient, deployable resource management method that adapts to complex topologies and information update requirements. Summary of the Invention

[0005] Technical problems: This invention aims to solve the following technical problems: (1) Batch modeling and granular transmission of high-dimensional state information: In the case that the actual perception state needs to be composed of multiple data packets, how to design a queue management and batch scheduling mechanism under limited buffer so that the data AoI can be effectively controlled. (2) Topology perception resource control under interference coupling environment: How to effectively model the V2V communication topology and embed it into the reinforcement learning strategy in the scenario of vehicle dynamic distribution and link interference change, so as to improve the structural perception capability and decision efficiency of the collaborative strategy. (3) Distributed learning and execution in multi-agent system: Under the premise that only the local state can be observed, how to realize the collaborative optimization decision of each agent on packet loss and transmission power, while ensuring that the training stage has centrality and the deployment stage has distributed execution capability. (4) Strengthening the correlation between graph neural network features and decision performance: How to introduce an effective supervision mechanism so that the graph embedding features can be highly correlated with the decision performance of multi-agent, thereby improving the learning efficiency and final task performance.

[0006] Technical Solution: The present invention provides a method for optimizing the age of vehicle network information based on graph reinforcement learning, comprising the following steps:

[0007] S1. Construct a batch modeling and finite buffer queue structure for vehicle states. Model the perceived state data of each vehicle as a batch consisting of multiple interdependent data packets, and manage the queues of the two most recently arrived state batches. Based on this, model the problem of jointly optimizing the active discarding decision of data packets and the transmit power control strategy to minimize the average information age (AoI) of the system as a constrained optimization problem.

[0008] S2. Utilize graph neural networks to model the V2V link topology, and extract a large-scale channel embedding representation that reflects the topology by aggregating the interference relationships between nodes; the graph embedding representation will be used as part of the agent's state for subsequent reinforcement learning decisions;

[0009] S3. Construct a multi-agent reinforcement learning system based on a centralized training and distributed execution CTDE framework: Define an agent for each V2V link, which makes decisions based on its local state. The local state integrates the graph embedding features generated in step S2, small-scale instantaneous channel information, the number and age of the remaining data packets in the batch in the transmitter queue, and the receiver AoI value. Each agent's action is a hybrid action, including discrete actions for state batch discard control and continuous actions for transmit power control. The reward function for each agent is the negative of its receiver AoI value.

[0010] S4. Introduce a graph embedding supervision mechanism based on the dominance function: During the centralized training process in step S3, the dominance function value calculated by the centralized evaluation network is used to generate a soft supervision signal for each agent, which is used to guide the graph neural network training in step S2, so that the generated graph embedding features are aligned with the long-term AoI optimization target and the task relevance of its feature representation is improved.

[0011] S5. Through multiple rounds of training iterations, update the parameters of the policy network and evaluation network in step S3, as well as the parameters of the graph neural network in steps S2 and S4. Finally, in actual deployment, each agent can autonomously optimize and control packet loss and power based on its local state according to the trained local policy network, thus realizing distributed AoI optimization.

[0012] in,

[0013] Step S1 specifically includes:

[0014] S201, Batch Modeling and Queue Structure: The perception state generated by each vehicle at each time step is represented as a data batch, and each batch contains... Each vehicle's transmitter buffer is designed with a dual-queue structure, capable of storing a maximum of two state batches simultaneously. Let represent the queue length of the m-th transmitter buffer in time slot n, where This indicates the number of unsent data packets remaining in the earlier batches. This indicates the number of unsent data packets remaining in the later-arriving batches. Simultaneously, the corresponding age is defined. in This represents the age of the i-th batch of data packets in the m-th transmitter buffer. (Definition) Let AoI be the receiver of the m-th vehicle.

[0015] S202. Queue Management: Based on different queue states and whether new batches have arrived, the following queue management strategy is adopted:

[0016] (1) When the buffer is empty, i.e. If a new batch arrives, it will be stored directly in the buffer, and its age will be initialized to zero. In this case, no discarding or replacement is required, and the transmitter's queue length and age are represented as follows:

[0017]

[0018] Where ∞ indicates that there are no data packets in the current time slot, ρ m [n]∈{0,1} indicates whether a new batch arrives at the m-th transmitter in time slot n, and u represents the number of data packets contained in each batch.

[0019] (2) A batch of data packets already exists in the buffer, i.e. If a new batch of data packets arrives at the start of a time slot, it is stored in the buffer as the next batch, with an initial age of zero. No replacement occurs; both batches of data packets are retained. The transmitter queue length and age are updated as follows:

[0020]

[0021] (3) Two batches of data packets already exist in the buffer, namely and If a new batch arrives at the start of a time slot, the later-arriving batch (i.e., the second batch) in the buffer will be replaced by the new batch. The age of the new batch is initialized to zero. At this point, the system must decide whether to continue transmitting the earlier batch or discard it based on the packet loss rate. The transmitter's queue length and age will be updated accordingly.

[0022]

[0023] S203. Resource Allocation: After each time slot ends, the buffer is updated according to the transmission strategy. A first-come-first-served (FCFS) principle is adopted, prioritizing transmission from batches that arrived earlier. The data packets. If there is remaining transmission capacity in the current time slot, the remaining capacity will continue to transmit batches. Data packets in the middle.

[0024] (1) Only some of the earlier arriving batches can be transmitted: if Then only a portion of the earlier batch will be transmitted, where y m [n] represents the number of data packets that the m-th transmitter can transmit in time slot n. In this case, the transmission capacity has been completely consumed by earlier batches, while later batches remain unchanged. The transmitter-side queue length, data packet age, and receiver-side AoI are updated as follows:

[0025]

[0026] (2) Transmit the entire first batch and transmit only a portion of the second batch: If the transmission capacity in time slot n is sufficient to fully transmit the first batch and partially transmit the second batch, i.e. All of the first batch All data packets will be transmitted, and remaining capacity will be allocated to the second batch. The transmitter's queue length, data packet age, and receiver's AoI will be updated.

[0027]

[0028] (3) Transmit all data in both batches: If the available transmission capacity in time slot n is sufficient to transmit all remaining data packets in both batches, i.e. The buffer is then completely emptied at the end of the time slot. The transmitter queue length, packet age, and receiver AoI are updated.

[0029] q m [n+1] = (0,0),

[0030]

[0031] S204. Problem Modeling: The optimal decision to send or drop data packets within a given time slot is inherently influenced by the transmit power level. Therefore, it is necessary to jointly optimize the transmit power p[n] and the packet drop factor γ[n] to minimize the long-term average AoI. The corresponding optimization problem can be expressed as:

[0032]

[0033] st 0≤p m [n]≤P max

[0034]

[0035] Wherein, the transmit power vector p[n] = [p1[n],…,p M [n] T The packet discard factor vector γ[n] = [γ1[n],…,γ M [n] T P max For maximum transmit power, link set Time slot set M represents the number of links, and N represents the total number of time slots.

[0036] Step S2 specifically includes:

[0037] S301. Construct a graph structure model: Model each link in the V2V communication network as a node in a graph, denoted as . This represents the m-th V2V link; the interference relationships between links are used to construct the edge e in the graph. jm ∈ε, where e jm This indicates the interference of link j on link m.

[0038] S302, Node Feature Encoding: The feature vector of each node is:

[0039]

[0040] Among them, PL m This represents the large-scale channel fading of link m. and These represent the two-dimensional position coordinates of the transmitter. and These represent the two-dimensional position coordinates of the receiver.

[0041] S303, Graph Neural Network Information Aggregation and Embedding Generation: The GraphSAGE structure is used as the backbone network of the graph neural network. Unlike traditional GraphSAGE implementations with a fixed sample size, the number of sampled neighbors for each node is dynamically determined based on the size of the current graph. Specifically, for a graph containing M nodes, the number of sampled neighbors D for each node is...

[0042]

[0043] in, This represents the floor operation. This allows GNNs to adaptively balance representation richness and computational efficiency across different network sizes and densities. The neighborhood aggregation and feature transformation performed at each layer are represented as follows:

[0044]

[0045] in, Let σ(·) be the hidden representation of node m in layer k, and let σ(·) be the activation function. This represents the sampling neighborhood at layer k. The symbol || denotes a concatenation operation along the feature dimension, enabling the model to jointly encode its own information and neighborhood information before transformation. After two rounds of aggregation, the final output embedding of node m is represented as...

[0046]

[0047] Where := represents an assignment operation, and this scalar value is embedded in f. m It is then used as part of the agent's local state input for action decisions.

[0048] Step S3 specifically includes:

[0049] S401. Agent State Space Modeling: Each agent constructs a state representation based on the local observable information of its V2V link, representing the state of agent m in time slot n. Defined as

[0050]

[0051] in, This is the feature embedding generated by the GNN for link m. This embedding originates from large-scale channel fading and is shared by all agents at the beginning of each decision cycle. Here, a decision cycle consists of N consecutive time slots, during which the large-scale channel fading is assumed to be constant. This represents the instantaneous small-scale fading coefficient between the transmitter and receiver of the m-th V2V link, which is updated in each time slot.

[0052] S402, Hybrid Action Space Definition: Each agent jointly outputs a hybrid action in each time slot, a m [n]=(γ m [n],p m [n]), where discrete action γ m [n] is used to control whether to actively discard the current queue batch; continuous actions p m [n] is used to allocate the transmit power in the current time slot, and its value ranges from [0, P]. max The action space of agent m Defined as

[0053]

[0054] Therefore, the space for joint operations is This hybrid structure enables agents to combine queueing conditions, packet age, and channel quality to optimize transmission behavior in complex vehicle environments.

[0055] S403, Reward Function Design: The goal of the reinforcement learning framework is to minimize the long-term average AoI of all V2V links. To ensure that the learning process of each agent aligns with this system-level objective, the reward for the m-th agent in time slot n is defined as the negative of its current receiver AoI:

[0056]

[0057] Step S4 specifically includes:

[0058] S501. In each round of training, based on the state-action trajectory of each agent, the advantage function value A is calculated using a centralized evaluation network. m [n]:

[0059] A m [n] = r m [n]+αV(s m [n+1])-V(s m [n]),

[0060] Where α∈(0,1] is the discount factor, and V(s) is the state value function.

[0061] S502. Normalize all advantage values ​​of each agent within the current decision-making cycle and calculate their time average as the supervision target of the graph neural network.

[0062]

[0063] S503, embedding the topological features f extracted by the graph neural network. m With normalized mean dominance Alignment, constructing a graph embedding supervised loss function:

[0064]

[0065] S504. To enhance the generalization ability and training stability of the embedding, the following two regularization terms are introduced:

[0066] (1) L2 regularization of graph embedding parameters:

[0067]

[0068] Where, θ gnn,i Let λ represent the i-th weight parameter of the graph neural network. r It is the regularization coefficient that controls the intensity of the penalty.

[0069] (2) Inter-node embedding difference smoothing term:

[0070]

[0071] Where, λ s It is a hyperparameter of the balance smoothness constraint.

[0072] S505, the joint training objective function of the final graph neural network is defined as:

[0073] L GNN =L align +L reg +L smooth .

[0074] By minimizing the aforementioned loss function, supervised alignment between graph embedding features and reinforcement learning advantage functions is achieved, thereby enhancing the GNN representation's ability to support downstream policy optimization and its task relevance.

[0075] Step S5 specifically includes:

[0076] S601. Initialization Phase: Initialize the policy network parameters θ for all agents. m Centralized evaluation network φ and graph neural network parameters θ GNN It also initializes an experience replay buffer to collect trajectory samples from multiple decision cycles.

[0077] S602, Network Structure Design:

[0078] (1) The policy network includes:

[0079] State coding layer: The input is the agent's local state s m[n].

[0080] Branching structure: Discrete branch output discard action γ m The softmax probability of [n]; the continuous branch output transmit power action p m The mean and variance of the Gaussian distribution [n] are used to sample the power value.

[0081] (2) The evaluation value network is a centralized structure, and the input is the concatenation of the states of all agents, S[n] = [s1[n]; ...; s M [n], outputting the state value function V for each agent. m [n] is used to calculate the dominance function.

[0082] S603, Sampling and Training Phase:

[0083] All agents in the environment act according to the current policy. Perform actions and collect trajectory data {s m [n],a m [n],r m [n],s m [n+1]}, calculate the dominance function A m [n], update the network loss of the actor policy:

[0084]

[0085] Where, ρ n It is the ratio of the current action probability to the previous action probability. The hyperparameter ∈ controls the pruning range to ensure the stability of training.

[0086] Updated assessment of network loss:

[0087]

[0088] in, To learn the target value in one step using time difference, V(s) m [n]) represents the current evaluation network prediction value.

[0089] S504. Jointly train the graph neural network at the end of each decision cycle: calculate the average advantage of each agent. As a soft tag, the guide graph is embedded in f m Learning; minimizing the supervised loss L in graph networks GNN .

[0090] S505, Deployment Phase: In the actual system, each agent independently generates hybrid actions using its local policy network. m [n]=(γ m [n],p m[n]) can complete the discarding and power control without global information and communication, thus achieving distributed AoI optimization.

[0091] This invention also provides a vehicle-to-everything (V2X) information age optimization system based on graph reinforcement learning, used to implement the method described above, the system comprising:

[0092] The batch modeling and queue management module is used to implement batch modeling of vehicle status and limited buffer queue management.

[0093] The graph neural network topology modeling module is used to implement V2V link topology modeling and feature embedding extraction.

[0094] The multi-agent reinforcement learning decision-making module is used to realize agent state space modeling, mixed action decision-making, and reward calculation;

[0095] The graph embedding supervised training module is used to implement a graph embedding supervision mechanism based on the dominance function.

[0096] The parameter optimization and distributed deployment module is used to optimize network parameters and enable autonomous optimization control of each agent.

[0097] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0098] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0099] This invention, by introducing a graph reinforcement learning mechanism, achieves unified optimization of information update timeliness and communication resource scheduling in the vehicle-to-everything (V2X) environment, demonstrating significant technical advantages and application value. It employs batch modeling and a dual-queue buffer structure to accurately depict the dynamic transmission process of perceived states under limited buffering. Under conditions of constrained communication resources, it significantly reduces the system's average information age (AoI) by jointly controlling the discarding and transmission power of data batches through a hybrid action strategy. Simultaneously, by leveraging graph neural networks for deep modeling of link topology, it effectively mines interference relationships between links and generates topological embeddings, enhancing the multi-agent policy's ability to perceive spatial interference coupling. In the multi-agent reinforcement learning architecture, the algorithm adopts a centralized training and distributed execution mechanism, allowing the policy optimization process to fully utilize global information, while the deployment phase is independently executed based on local observations, balancing system convergence stability with the low communication overhead requirements of practical applications. Furthermore, this invention innovatively introduces a graph embedding collaborative training method using the advantage function as the supervision signal, closely aligning graph representation with policy performance and further enhancing the practicality of topological features in decision optimization. In summary, this invention not only enhances the system's generalization and adaptive capabilities in dynamic topologies and partially observable environments, but also provides an efficient and reliable solution for time-sensitive vehicle-to-everything (V2X) communication and collaborative control. Attached Figure Description

[0100] Figure 1 This is the system architecture of an embodiment of the present invention;

[0101] Figure 2 This illustrates the trend of average AoI during training in this embodiment of the invention.

[0102] Figure 3 The average AoI variation trend of each algorithm under different data packet lengths in the embodiments of the present invention;

[0103] Figure 4 This is the average AoI performance of various algorithms under different data packet arrival probabilities in the embodiments of the present invention;

[0104] Figure 5 This represents the average AoI performance of various algorithms under different numbers of V2V links in the embodiments of the present invention. Detailed Implementation

[0105] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0106] Example 1

[0107] The embodiments of this invention can be deployed in intelligent connected V2V communication scenarios to improve the freshness of status information updates and transmission efficiency, such as... Figure 1 As shown in the figure, this invention provides a method for optimizing the age of vehicle network information based on graph reinforcement learning. The following description, in conjunction with the system composition and method flow, illustrates this method:

[0108] Within each time period, there are M pairs of V2V communication links in the environment. Vehicles generate high-dimensional state data through the perception system. The data is encoded into batches consisting of multiple data packets and input into the limited buffer structure of the transmitter. Each vehicle transmitter uses a dual-queue structure to manage the two most recently arrived state batches. The system decides whether to discard one of the batches and at what power to transmit in the current time slot based on a hybrid action strategy.

[0109] The system employs a graph-based multi-agent reinforcement learning structure. First, the communication topology is modeled as a graph G = (V, E), where each node represents a V2V link, and edges represent interference relationships between links. The GraphSAGE network is then used to extract the topology embedding f from static information such as node location and path loss. m This embedding is then input into the policy network as part of the local state of each agent. The local state of each agent consists of the following information: topological embedding features f m Small-scale channel gain Number of remaining data packets in two status batches in the queue and age and and the AoI value of the receiving end. Each agent outputs a hybrid action through a policy network: a discrete action controls whether to discard old batches, and a continuous action controls the current transmit power. During training, a centralized training and distributed execution framework is employed: all agents share a centralized evaluation network for estimating the state-value function and calculating the advantage function; each agent independently maintains its policy network and outputs the policy. Policy training uses the PPO algorithm, aiming to maximize the advantage function and enhance the expression of low-AoI policies.

[0110] To enhance the relevance of graph embeddings to the task, the system designs a graph embedding supervision mechanism based on the dominance function. In each training round, the average dominance value is calculated for each agent's trajectory. As a soft label for graph embedding, the consistency between graph features and policy performance is enhanced by minimizing the mean squared error loss. Furthermore, an L2 regularization term and a time smoothing term are introduced to jointly constitute the training loss of the graph neural network. After training convergence, each agent deploys its local policy network to the vehicle terminal. Without centralized control, it independently generates hybrid actions based on local states, achieving distributed, autonomous queue management and power control, thereby ensuring optimal information timeliness without increasing communication overhead.

[0111] To enable those skilled in the art to better understand the present invention, the convergence of the training process in this embodiment and its comparison with the baseline method are given below for further explanation.

[0112] This embodiment uses a V2V communication network as the simulation environment. The system simulation environment is constructed in a rectangular road area (750m×1298m) with multiple V2V links deployed. Vehicles are randomly distributed across multiple lanes at a speed of 36km / h, and inter-link interference under shared spectrum bandwidth is considered to fully reproduce the interference structure and topological dynamics of dense vehicle-to-everything (V2X) communication in urban traffic. Vehicles sample state information in batches, with each batch consisting of multiple interdependent data packets, reflecting the high-dimensional characteristics of actual environmental perception data (such as LiDAR point clouds, image frames, etc.). The system updates small-scale fading in each time slot (1ms in length, corresponding to the channel coherence time at a carrier frequency of 2GHz and a vehicle speed of 36km / h), and updates large-scale fading (path loss, shadowing effect, etc.) every N time slots for topology modeling of the graph neural network. The main simulation parameters include: bandwidth 1MHz, maximum transmit power 10dBm, noise power -104dBm, data packet length 3600 bits, batch arrival probability 0.8, the presence of a V2V communication link in the environment, vehicle antenna height 1.5 meters, vehicle antenna gain 3dBi, vehicle receiver noise figure 9dB, and communication duration N = 100 milliseconds.

[0113] To verify the performance of this invention, the following four representative strategies were selected as baselines for comparison: (1) Random strategy: In each time slot, the agent randomly decides whether to discard data packets and the transmission power; (2) AoI threshold strategy: Using the AoI of the receiver as the criterion, when the AoI is greater than 3, the current batch is discarded and transmitted at 70% of the maximum power; otherwise, the batch is retained and transmitted at 30% power (“Online learning of goal-oriented status updating with unknown delay statistics” in IEEE Journal on Selected Areas in Communications, vol.42, no.11, pp.3293-3305, Nov.2024); (3) ITLinQ algorithm: A classic interference-aware resource allocation algorithm that does not allow packet loss and prioritizes the retention of all data for transmission (“ITLinQ: A new approach for spectrum sharing in device-to-device communication systems,” in IEEE Journal on Selected Areas in Communications, vol.42, no.11, pp.3293-3305, Nov.2024). Communications, vol.32, no.6, pp.1139-1151, Jun.2014); (4) WMMSE algorithm: based on the power control strategy of minimizing weighted mean square error, it also does not allow packet loss and tends to maximize system and rate (“An iteratively weighted MMSE approach to distributed sum-utilitymaximization for a MIMO interfering broadcast channel,” in IEEE Transactions on Signal Processing, vol.59, no.9, pp.4331–4340, Sep.2011).

[0114] Figure 2The figure illustrates the trend of average AoI during training, comparing the performance of the proposed algorithm with various baseline methods over 600 training rounds. Initially, the AoI value is high, indicating that the system has not yet learned an effective strategy. However, within the first 100 rounds, the AoI of the proposed method rapidly decreases and subsequently stabilizes at approximately 2 milliseconds, demonstrating good convergence and the ability to minimize timeliness. Overall, the proposed algorithm consistently outperforms other methods during training, achieving lower and more stable AoI control. Notably, while ITLinQ and WMMSE are effective in rate optimization, their AoI performance is poor due to neglecting queuing status and timeliness objectives, indicating that traditional throughput-oriented strategies are insufficient to meet real-time data update requirements. In contrast, although the random strategy lacks task awareness, its random packet loss and power fluctuations help maintain information freshness to some extent in cache-constrained scenarios, demonstrating a certain degree of adaptability. In summary, this figure verifies the rapid AoI optimization capability and practical deployment stability of the proposed method during training.

[0115] Figure 3 The average AoI (Aspect-Oriented Identification) of various algorithms is shown as the packet length increases from 1200 bits to 3600 bits. Overall, the AoI of all methods increases with the increase in packet length, reflecting the transmission delay caused by larger data. However, the increase varies significantly among different methods. The method proposed in this invention maintains the lowest AoI under all packet length conditions and has the smallest increase, demonstrating superior scheduling ability and robustness, especially maintaining a stable low value when the packet length is 3600 bits. In contrast, the random strategy performs reasonably well under constrained buffer conditions, thanks to the certain match between its random packet loss behavior and the system structure, which can passively maintain information age to some extent. Although ITLinQ and WMMSE perform resource allocation based on channel state information, they are less effective in AoI control due to the lack of task relevance and queuing awareness. The AoI threshold strategy frequently triggers packet loss as the packet length increases, leading to update interruptions and a significant increase in AoI. In summary, Figure 3 The results show that the algorithm of the present invention has stronger adaptability and timeliness guarantee when facing increased data load, which is significantly better than the existing comparative methods and has high engineering promotion value.

[0116] Figure 4This paper compares the control capabilities of various algorithms on average AoI under different packet arrival probabilities. The overall trend shows that as the arrival probability increases, the system gains more opportunities for state updates, and the AoI of all methods generally decreases. The method of this invention maintains the lowest AoI under all arrival probabilities, demonstrating excellent adaptability and robustness. Especially in high arrival rate scenarios, it can still fully leverage the advantages of queue management and power control strategies to maintain system timeliness. Even for random strategies, frequent data arrivals alleviate policy uncertainty to some extent, allowing some valid data to be transmitted in a timely manner; however, compared with this invention, its AoI is still significantly higher. ITLinQ, WMMSE, and AoI threshold strategies lack flexible adjustment mechanisms, making it difficult to fully respond to changes in update frequency, and their overall stability and optimization capabilities are limited. In summary, Figure 4 The invention's method demonstrates strong generalization performance and timeliness control capabilities in dynamic data arrival environments, making it suitable for frequent state perception scenarios in actual vehicle-to-everything (V2X) networks and possessing significant practical application value.

[0117] Figure 5 The average AoI performance of various algorithms is shown under different numbers of V2V links (system scale). As the number of links increases from 4 to 8, the AoI of all methods increases, reflecting the update latency caused by increased resource contention under multi-link concurrency. However, the method proposed in this invention maintains the lowest AoI across all scales, demonstrating excellent scalability and scheduling robustness. This method, by jointly modeling link interference relationships and buffer states, adaptively optimizes scheduling and power control, prioritizing critical links under resource-constrained conditions, thereby effectively controlling the overall AoI. In contrast, traditional heuristic strategies such as ITLinQ and WMMSE lack the ability to model queuing states and multi-agent collaboration, making it difficult to cope with the dynamic and time-sensitive challenges brought by system expansion, resulting in a significant increase in AoI. In summary, Figure 5 This further verifies the stability and scheduling efficiency of the method of the present invention in a large-scale vehicle network environment, and is applicable to real-time status update tasks in dense communication scenarios.

[0118] Example 2

[0119] The vehicle network information age optimization system based on graph reinforcement learning proposed in this invention mainly includes the following components, which are described in text form below:

[0120] I. System Architecture Overview

[0121] Deployed in a vehicle-to-vehicle (V2V) communication environment, this system aims to minimize the information age (AoI) of each vehicle receiver through intelligent scheduling and resource control. The system employs a multi-agent architecture of "centralized training and distributed execution," integrating graph neural networks (GNNs) for topology modeling and reinforcement learning (RL) for decision optimization, demonstrating good scalability and practicality.

[0122] II. Batch Modeling and Queue Management Module

[0123] This module is responsible for batch processing and management of vehicle perception status data. Each vehicle periodically generates high-dimensional status information (such as sensor data), which is then split into multiple data packets to form a batch. The sending end adopts a dual-queue buffer structure, retaining only the two most recently arrived batches. The system dynamically executes queue management strategies such as storage, initialization, and replacement based on the buffer status (whether it is empty, whether a batch already exists, or whether a new batch has arrived), and follows the First-Come, First-Served (FCFS) principle for data transmission.

[0124] III. Graph Neural Network Topology Modeling Module

[0125] This module models the entire V2V communication network as a graph structure, where each communication link is considered a node, and the edges between nodes represent interference relationships between links. Each node's features include its large-scale channel fading information and the two-dimensional location coordinates of the transmitter and receiver. A GraphSAGE network is used to aggregate neighborhood information, generating a low-dimensional embedding representation that reflects the network topology. This embedding is then used as part of each agent's state for subsequent reinforcement learning decisions.

[0126] IV. Multi-agent reinforcement learning decision-making module

[0127] Each V2V link corresponds to an agent whose observed state includes: topological embeddings generated by a graph neural network, instantaneous small-scale channel information, the number and age of remaining packets in the two batches in the transmitter queue, and the AoI value at the receiver. The agent outputs a hybrid action: a discrete action to decide whether to discard the current queued batch, and a continuous action to control the transmit power. The reward function is designed as the negative of the receiver's AoI to encourage reducing the information age.

[0128] V. Graph Embedded Supervised Training Module

[0129] To enhance the relevance between graph embeddings and task objectives, the system introduces a dominance function-based supervision mechanism during the training phase. A centralized evaluation network calculates the dominance value for each agent, and its time average is used as a soft label. By minimizing the difference between the embedding value and the dominance value, the system guides the graph neural network to generate a more task-relevant topological representation. Simultaneously, regularization and smoothing terms are introduced to improve training stability and generalization ability.

[0130] VI. Parameter Optimization and Distributed Deployment Module

[0131] The system employs the Proximal Policy Optimization (PPO) algorithm for centralized training, updating the policy network and evaluating network parameters. Graph neural networks and reinforcement learning networks are trained alternately for joint optimization. After training, each agent deploys its local policy network, making independent decisions based solely on local observations during actual operation, achieving distributed, low-communication-overhead autonomous control.

[0132] VII. System Workflow

[0133] 1. Initialize all network parameters and experience replay buffer;

[0134] 2. Within each decision cycle, the agent executes actions according to the current policy and collects the state-action-reward trajectory;

[0135] 3. During the intensive training phase, the PPO algorithm is used to update the policy and value network;

[0136] 4. After each round of training, train the neural network using the dominance value supervised graph;

[0137] 5. After training converges, each agent runs independently, achieving distributed AoI optimization.

[0138] VIII. Summary

[0139] This system minimizes information age in dynamic vehicle-to-everything (V2V) environments by fusing graph neural networks and multi-agent reinforcement learning. It features structure perception, collaborative decision-making, efficient training, and easy deployment, making it suitable for high-density, highly dynamic V2V communication scenarios and demonstrating significant practical value and promising prospects for wider adoption.

[0140] The above description, in conjunction with the accompanying drawings, is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the claims. It should be understood that any equivalent changes made without departing from the spirit of the present invention are within the scope of protection covered by the claims.

Claims

1. A method for optimizing the age of vehicle network information based on graph reinforcement learning, characterized in that, The method includes the following steps: S1. Construct a batch modeling and finite buffer queue structure for vehicle states. Model the perceived state data of each vehicle as a batch consisting of multiple interdependent data packets, and manage the queues of the two most recently arrived state batches. Based on this, model the problem of jointly optimizing the active discarding decision of data packets and the transmit power control strategy to minimize the average information age (AoI) of the system as a constrained optimization problem. S2. Utilize graph neural networks to model the V2V link topology, and extract a large-scale channel embedding representation that reflects the topology by aggregating the interference relationships between nodes; the graph embedding representation will be used as part of the agent's state for subsequent reinforcement learning decisions; S3. Construct a multi-agent reinforcement learning system based on a centralized training and distributed execution CTDE framework: Define an agent for each V2V link, which makes decisions based on its local state. The local state integrates the graph embedding features generated in step S2, small-scale instantaneous channel information, the number and age of the remaining data packets in the batch in the transmitter queue, and the receiver AoI value. Each agent's action is a hybrid action, including discrete actions for state batch discard control and continuous actions for transmit power control. The reward function for each agent is the negative of its receiver AoI value. S4. Introduce a graph embedding supervision mechanism based on the dominance function: During the centralized training process in step S3, the dominance function value calculated by the centralized evaluation network is used to generate a soft supervision signal for each agent, which is used to guide the graph neural network training in step S2, so that the generated graph embedding features are aligned with the long-term AoI optimization target and the task relevance of its feature representation is improved. S5. Through multiple rounds of training iterations, update the parameters of the policy network and evaluation network in step S3, as well as the parameters of the graph neural network in steps S2 and S4. Finally, in actual deployment, each agent can autonomously optimize and control packet loss and power based on its local state according to the trained local policy network, thus realizing distributed AoI optimization.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: The perception state generated by each vehicle at each time step is represented as a batch containing multiple data packets; Each vehicle's transmitter buffer is designed with a dual-queue structure to store the two most recently arrived status batches; Define the queue length and batch age, and execute queue management strategies, including storage, initialization, and replacement, based on the buffer status and the arrival of new batches. After each time slot ends, the buffer status is updated according to the transmission capacity and the First-Come, First-Served (FCFS) principle. The joint optimization problem is modeled as finding the optimal transmit power vector and packet drop factor vector to minimize the long-term average AoI, expressed as: Wherein, the transmit power vector p[n] = [p1[n],…,p M [n] T The packet discard factor vector γ[n] = [γ1[n],…,γ M [n] T P max For maximum transmit power, link set Time slot set Let M be the AoI value of the m-th vehicle receiver in time slot n, where M is the number of links and N is the total number of time slots.

3. The method according to claim 2, characterized in that, The queue management strategy includes: make Let represent the queue length of the m-th transmitter buffer in time slot n, where This indicates the number of unsent data packets remaining in the earlier batches. This indicates the number of unsent data packets remaining in the later-arriving batches; it also defines the corresponding age. in This represents the age of the data packets in the i-th batch of data packets in time slot n within the m-th transmitter buffer; when The transmitter's queue length and age are expressed as: Where ∞ indicates that there are no data packets in the current time slot, ρ m [n]∈{0,1} indicates whether a new batch arrives at the m-th transmitter in time slot n, and u represents the number of data packets contained in each batch; when and The transmitter queue length and age are updated as follows: when and The transmitter queue length and age will be updated accordingly:

4. The method according to claim 3, characterized in that, Updating the buffer state according to the first-come, first-served (FCFS) principle includes: if Then only a portion of the earlier batch will be transmitted, where y m [n] represents the number of data packets transmitted by the m-th transmitter in time slot n, where the transmitter's queue length, data packet age, and receiver's AoI are updated as follows: when The transmitter queue length, packet age, and receiver AoI are updated as follows: when The transmitter queue length, packet age, and receiver AoI are updated to q. m [n+1] = (0,0) 5. The method according to claim 1, characterized in that, Step S2 specifically includes: Each link in the V2V communication network is modeled as a graph node, and the edges of the graph are constructed based on the interference relationships between the links. Each node's feature vector includes its large-scale channel fading and its two-dimensional location coordinates for the transmitter and receiver; The GraphSAGE network is used as the backbone network, and the number of sampled neighbors for each node is dynamically determined based on the current graph size. For a graph containing M links, the number of sampled neighbors for each node is... for in, This indicates a round-down operation; A low-dimensional topology-aware embedding for each node is generated through neighborhood aggregation and feature transformation. This scalar value embedding is used as the local state input for the agent.

6. The method according to claim 4, characterized in that, In step S3 The state of agent m in time slot n Defined as in, It is the feature embedding generated for link m by the graph neural network in step S2. It is the instantaneous small-scale fading coefficient of the m-th link. and These represent the remaining data packets for the two batches, respectively. These are the ages of the first batch of data packets in the m-th transmitter buffer, the ages of the second batch of data packets in the m-th transmitter buffer, and the AoI value of the m-th vehicle receiver, respectively. The hybrid action of agent m in time slot n is defined as: a m [n]=(γ m [n],p m [n]), where discrete action γ m [n] is used to control whether to actively discard the current queue batch; continuous actions p m [n] is used to allocate the transmit power in the current time slot, and its value ranges from [0, P]. max Action space of agent m Defined as Joint Operations Space The reward for agent m in time slot n is defined as:

7. The method according to claim 6, characterized in that, Step S4 specifically includes: In each training round, based on the state-action trajectory of each agent, the centralized evaluation network in step S3 is used to calculate the dominance function value A of agent m in time slot n. m [n]: A m [n]=r m [n]+αV( s m[n+1])-V(s m [n]), Where α∈(0,1] is the discount factor, and V(s) is the state value function output by the centralized evaluation network; For each agent, normalize all advantage values ​​within the current decision cycle and calculate their time mean, which serves as the supervision target for the graph neural network. Constructing a graph embedding supervised loss function The topological embedding features f extracted by the graph neural network m With normalized mean dominance Alignment; To enhance the generalization ability and training stability of the embeddings, an L2 regularization term for the graph embedding parameters is introduced into the loss function. and inter-node embedding difference smoothing terms Where, θ gnn,i Let λ represent the i-th weight parameter of the graph neural network. r It is the regularization coefficient that controls the intensity of the penalty, λ. s It is a hyperparameter of the balance smoothness constraint; By minimizing the joint training objective function L GNN =L align +L reg +L smooth This enables supervised alignment between graph embedding features and reinforcement learning advantage functions.

8. The method according to claim 1, characterized in that, Step S5 specifically includes: Initialize the policy network parameters, centralized evaluation network parameters, and graph neural network parameters for all agents; The policy network adopts a branch structure. Its input is the local state of the agent. The discrete branch outputs the softmax probability of the discard action, and the continuous branch outputs the mean and variance of the Gaussian distribution of the transmit power action. The evaluation network has a centralized structure, with the input being the concatenation of the states of all agents and the output being the state value function corresponding to each agent. During the training phase, all agents perform actions according to the current policy and collect trajectory data. The Proximal Policy Optimization (PPO) algorithm is used to update the policy network and value network. At the end of each decision cycle, a graph neural network is jointly trained according to the mechanism described in step S4. During the deployment phase, each agent independently generates hybrid actions using its local policy network to achieve distributed autonomous control.

9. A vehicle-to-everything (V2X) information age optimization system based on graph reinforcement learning, used to implement the method according to any one of claims 1-8, characterized in that, The system includes: The batch modeling and queue management module is used to implement batch modeling of vehicle status and limited buffer queue management. The graph neural network topology modeling module is used to implement V2V link topology modeling and feature embedding extraction. The multi-agent reinforcement learning decision-making module is used to realize agent state space modeling, mixed action decision-making, and reward calculation; The graph embedding supervised training module is used to implement a graph embedding supervision mechanism based on the dominance function. The parameter optimization and distributed deployment module is used to optimize network parameters and enable autonomous optimization control of each agent.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images