Heterogeneous Internet of Vehicles resource scheduling method based on graph embedding hierarchical reinforcement learning

Abstract

The invention belongs to the technical field of Internet of Vehicles (IoV) and distributed artificial intelligence, and discloses a heterogeneous Internet of Vehicles resource scheduling method based on graph embedding hierarchical reinforcement learning. The thought of learning value and scheduling overhead decomposition is migrated to a deep reinforcement learning algorithm, and a hierarchical quality-driven deep reinforcement learning (HQ-DRL) framework is provided; according to the invention, a hierarchical cooperative Actor-Critic (GE-HSAC) algorithm based on graph embedding is designed based on a framework, and the problem of short-view decision making caused by the fact that learning quality and communication cost cannot be considered at the same time in a traditional scheduling strategy is effectively solved; in addition, the defect that the convergence rate of the global model is slowed down due to the fact that communication resources are wasted for transmitting the low-value model is avoided; through collaborative optimization of the high-layer intelligent agent and the low-layer intelligent agent, the convergence speed of a distributed learning task can be remarkably accelerated in a dynamic and heterogeneous car networking environment finally, the utilization rate of communication resources is improved, and the performance advantage higher than that of a traditional single-target optimization method is shown.

Description

Technical Field

[0001] This invention belongs to the fields of Internet of Vehicles (IoV) and distributed artificial intelligence technology, and in particular relates to a heterogeneous IoV resource scheduling method based on hierarchical quality-driven deep reinforcement learning. Background Technology

[0002] Driven by the vision of 6G communication, IoV (Internet of Vehicles) is moving towards full-scenario, autonomous driving collaboration, and immersive experiences. Its essence lies in treating vehicle nodes as collaborative individuals, building a dynamic digital twin of the city based on collaborative perception and real-time interaction, and achieving beyond-line-of-sight perception and collaborative decision-making.

[0003] However, a key challenge in turning this ambitious vision into reality lies in leveraging massive amounts of edge vehicle data. This requires us to efficiently train powerful collaborative AI models while protecting data privacy. For this reason, Federated Learning (FL), with its inherent privacy protection and distributed nature, stands out and is widely recognized as a key enabling technology for achieving this goal.

[0004] While the application of federated learning to vehicle-to-everything (FL-in-IoV) holds great promise, its effectiveness is severely limited by the resource scheduling efficiency of the underlying wireless network. This scheduling problem is the core bottleneck of current research, essentially involving matching the optimal nodes and communication paths for model interaction within the rapidly changing vehicle network. Traditional scheduling strategies (such as maximum signal-to-noise ratio and minimum latency greedy algorithms) focus only on the physical performance of the communication link, neglecting the updating value of the model data being transmitted. However, numerous studies have shown that such strategies, which only "perceive communication" without "perceive learning," are clearly short-sighted.

[0005] We must recognize that the true value of a single communication within a federated learning framework lies in its ability to deliver the maximum effective information gain to the global model. A high-speed link that transmits outdated or redundant model updates will not only fail to promote learning but will also hinder the overall convergence process by wasting valuable communication resources.

[0006] To solve these complex decision-making problems, the scheduling field has introduced DRL (Dual Relationship Management) agents, which are highly intelligent decision-makers. DRL agents learn more complex scheduling solutions than classical heuristics by interacting with the environment. However, numerous studies have shown that such strategies, which only "perceive communication" without "perceive learning," are significantly short-sighted.

[0007] We must recognize that the true value of a single communication within a federated learning framework lies in its ability to deliver the maximum effective information gain to the global model. A high-speed link that transmits outdated or redundant model updates will not only fail to promote learning but will also hinder the overall convergence process by wasting valuable communication resources.

[0008] To solve the aforementioned complex decision-making problems, the scheduling field has introduced DRL (Reinforcement Reinforcement Learning), a powerful intelligent decision-maker. DRL agents learn more complex scheduling solutions than those based on classical heuristics by interacting with the environment. However, applying DRL to federated learning scheduling in vehicular networks still faces significant challenges, primarily due to the limitations of existing flattened decision-making mechanisms. First, there is an inherent conflict in the objective functions. Federated learning aims for long-term model convergence, while resource scheduling aims for immediate communication overhead. In flattened reinforcement learning, forcibly coupling these two disparate objectives with different time scales into the same reward function easily leads to the agent neglecting one aspect for the other, making it difficult to learn the globally optimal strategy. Second, there is the curse of dimensionality in the state-action space. In heterogeneous vehicular networks, simultaneous decisions on peer node selection and communication path allocation are required. If a flattened network is used, the action space explodes in a Cartesian product fashion (number of nodes × number of paths), making training extremely difficult to converge. Therefore, an intelligent scheduling scheme that decouples "high-level learning value mining" from "low-level communication cost control" is urgently needed. Summary of the Invention

[0009] This invention aims to solve the resource scheduling problem of distributed consensus learning in heterogeneous vehicle networks, namely, to jointly optimize the selection of peer nodes (a decision based on long-term learning value) and the allocation of communication paths (a decision based on instant communication cost) in order to maximize the global convergence speed.

[0010] This invention introduces the idea of ​​"decoupling learning value from scheduling cost" into deep reinforcement learning algorithms, proposing a Hierarchical Quality-Driven Deep Reinforcement Learning (HQ-DRL) framework. This framework decomposes the complex joint scheduling problem into two sub-problems: the upper-level sub-problem is a learning-oriented peer node selection problem based on multidimensional model quality; the lower-level sub-problem is an efficiency-oriented communication path selection problem based on communication link cost.

[0011] To solve the aforementioned hierarchical decision-making problem, this invention further designs a graph-embedded hierarchical synthetic synergistic Actor-Critic (GE-HSAC) algorithm. This algorithm addresses the structural heterogeneity of the decision-making space in heterogeneous vehicular networks by constructing a heterogeneous policy network comprising a high-level graph sensing loop and a low-level link-sensitive loop. Specifically, the high-level layer employs a graph attention network (GAT) to jointly model the "self-quality-interaction relationship" of peer nodes. It utilizes an attention mechanism to dynamically allocate weights based on the semantic complementarity and topological connection probability between nodes, effectively preventing feature smoothing and achieving accurate identification and screening of high-value interaction targets. The low-level layer employs a lightweight multilayer perceptron (MLP) to independently decide the optimal communication path based on real-time link status.

[0012] More importantly, to address the inconsistency between long-term quality objectives and immediate cost objectives in the hierarchical structure, the GE-HSAC algorithm introduces a cooperative Bellman update mechanism. This mechanism constructs a cooperative TD objective value that includes a communication penalty term, forcing the higher-level value network to internalize the actual communication costs incurred by lower-level executions when evaluating the long-term returns of node selection strategies. This backward value injection mechanism effectively establishes a closed-loop feedback between the two-layer agents, preventing the higher-level strategy from getting trapped in local optima by ignoring execution costs.

[0013] The innovation of this invention lies in the deep integration of heterogeneous network architecture and collaborative training mechanism through the GE-HSAC algorithm under the hierarchical quality-driven decision framework. This enables the agent to adaptively find the global optimal balance between maximizing model learning benefits and minimizing communication overhead in the dynamic and complex vehicle network environment, which significantly improves the convergence speed of distributed learning tasks.

[0014] The adopted technical solution is: a heterogeneous vehicle network resource scheduling method based on hierarchical quality-driven deep reinforcement learning, including the following steps:

[0015] Step 1: Construct and quantify a 3D heterogeneous vehicle networking system model

[0016] Step 1.1: Define system scenarios and components

[0017] To accurately describe the physical and network environment in which this method is applied, we first need to establish a three-dimensional heterogeneous vehicle-to-everything (V2X) system model consisting of roadside units (RSUs), ground mobile vehicles, and low Earth orbit (LEO) satellites; we represent the set of roadside units as... , This represents the total number of RSUs; the system employs a decentralized distributed consensus learning paradigm, where each RSU... Independent training of local models It also maintains in real time a lightweight set of class prototypes that can characterize the distribution of local data features. It broadcasts prototype state and synchronizes model parameters with selected peer RSU nodes via the Gossip protocol.

[0018] Step 1.2: Quantify the terrestrial communication path

[0019] To facilitate intelligent decision-making regarding subsequent agent communication paths, more precise modeling of these paths is needed. The quantification of their performance lays the foundation for subsequent cost-effectiveness decisions. Furthermore, a hierarchical communication model is constructed, and two communication paths are quantified. The first type is the direct ground connection path. To evaluate the quality of this path, its signal-to-noise ratio (SNR) is calculated, which comprehensively reflects the relationship between signal strength and background noise. and The signal-to-noise ratio of the ground channel between them is:

[0020]

[0021] in For transmission power, For antenna gain, The distance between the two, This is the path loss index. For channel bandwidth, Let be the noise power spectral density; based on the signal-to-noise ratio, the theoretical data rate can be obtained according to Shannon's theorem, which directly determines the transmission speed of the model parameters:

[0022]

[0023] Based on this, a communication latency model is constructed as the basis for measuring path cost in subsequent decision-making processes; the size is... The communication latency required for the model parameters to be transmitted via the ground path is:

[0024]

[0025] in For fixed propagation and processing delays;

[0026] Step 1.3: Quantify the satellite communication path

[0027] The second type is the satellite relay path; it provides a wide-area coverage alternative when terrestrial connections are interrupted; its communication latency is composed of the transmission time of uplink and downlink, as well as the latency caused by satellite relay and long-distance propagation.

[0028]

[0029] in The total propagation and processing latency; its data rate and The calculation also follows Shannon's theorem, and the path loss adopts the free space path loss (FSPL) model. calculate;

[0030] Step 2: Establish an optimization problem aimed at accelerating model convergence.

[0031] Step 2.1: Determine the optimization objective

[0032] Based on the above system model, the resource scheduling problem is constructed as a sequential decision problem. Therefore, the time delay of the RSU behavior at the t-th decision time is... Including its communication delay and local computer latency The communication latency is determined by the path model in step one, while the computation latency is related to operations such as RSU local processing model aggregation; therefore, the global optimization objective of this method aims to learn an optimal scheduling strategy. To minimize the expected total time required to achieve global model convergence. :

[0033]

[0034] in, It is the total number of decision steps required to achieve convergence. This represents the scheduling strategy.

[0035] Step 2.2: Determine the constraints

[0036] To ensure decision-making and physical feasibility, this optimization problem needs to be solved under the following conditions: Let the decision variables... Indicates at time RSU Choose As peer nodes; let the decision variables... Indicates whether to select a path To communicate;

[0037]

[0038] This constraint ensures that the same RSU can only select the same node peer at any given time;

[0039]

[0040] This constraint ensures that once a node pair is selected, there must be one and only one communication path activated.

[0041]

[0042] This constraint specifies the ultimate goal of the learning task, namely that the loss of the global model must eventually not exceed a preset convergence threshold; the problem is an NP-hard mixed-integer nonlinear programming (MINLP) problem.

[0043] Step 3: Decouple the optimization problem using a hierarchical reinforcement learning framework.

[0044] To address the aforementioned scheduling challenges involving multiple objectives and scales, this invention proposes and defines a Hierarchical Quality-Driven Deep Reinforcement Learning (HQ-DRL) framework. This framework is a two-layer closed-loop decision system comprising the following core components and mechanisms.

[0045] (1) Framework definition:

[0046] The HQ-DRL framework orthogonally decomposes the resource scheduling task in heterogeneous vehicular networks across time and function dimensions, constructing two collaborative agents. First, a high-level quality-driven agent operates on a slower time scale (corresponding to the model aggregation cycle). Its core objective is to maximize long-term learning quality (i.e., model convergence speed and accuracy), aiming to search for the optimal peer-to-peer interaction nodes in the macroscopic topology. Second, a low-level cost-sensitive agent operates on a faster time scale (corresponding to the link transmission cycle). Its goal is to minimize instantaneous communication costs, responsible for planning the optimal communication path under constraints specified by the higher-level agent.

[0047] (2) Interaction mechanism:

[0048] The HQ-DRL framework achieves collaboration between high and low layers through a unique two-way interaction mechanism. First, it uses forward action constraints and high-level decision-making (target node). This directly constitutes part of the lower-level state space, limiting the search scope of the lower-level agent and allowing it to focus on link optimization between specific node pairs instead of blindly searching the entire network. Second, backward value injection: the actual communication cost generated after the lower layer executes a specific communication action (…). This serves as a penalty signal fed back to the higher-level reward function. This compels the higher-level agent to not only identify nodes with high data quality, but also to consider the reachability cost of those nodes, thus achieving a dynamic trade-off between quality and cost.

[0049] Based on the above framework definition, we break down the original optimization problem into the following two sub-problems:

[0050] Step 3.1: Define the high-level peer node selection subproblem

[0051] The higher-level subproblem focuses on solving the strategy of "who to communicate with," with the goal of selecting peer nodes that maximize long-term learning benefits (i.e., maximize model convergence speed). ;

[0052] Step 3.2: Define the low-level communication path selection subproblem

[0053] The lower-level subproblems focus on solving the strategy of "how to communicate", and their goal is to select the communication path with the lowest cost to complete the model interaction, given that the higher-level decision has been made.

[0054] Step 4: Formalize the high-level peer node selection problem into a Markov decision process.

[0055] Step 4.1: Determine the state space

[0056] High-level state space Modeled as a local RSU Directed weighted graph centered on Unlike traditional methods that focus solely on node attributes, this invention argues that the selection of peer nodes depends not only on their own state but also on their relative relationship with the central node (semantic complementarity and connection feasibility).

[0057] picture It contains two types of feature information:

[0058] (1) Node Features: Describes candidate nodes The absolute quality, including model freshness Aggregated data volume and model noise variance .

[0059] (2) Edge Features: Describes candidate nodes With the central node The relative relationship between them mainly includes two aspects. First, the semantic divergence of the model. This feature is based on the local class prototype representation defined in step 1.1. prototype of peer node class The Euclidean distance is used to derive the difference between the two models in the feature space. The greater the difference, the higher the potential information gain. This is a typical pairwise relationship feature, which cannot be expressed solely through node features. Second, link connectivity probability. While higher levels do not decide on specific paths, they need to be aware of the macroscopic topology. This feature reflects the prior probability that available communication links exist between nodes, including satellite or ground connections, serving as a pre-screening indicator for nodes that are too far apart to interact.

[0060] Step 4.2: Determine the motion space

[0061] Action space of high-level intelligent agents It is a discrete space, meaning that a target interaction node is selected from all reachable peer nodes. :

[0062]

[0063] Step 4.3: Determine the reward function

[0064] To align the learning objectives of the agents with the global optimization objective and to establish a collaborative mechanism between high- and low-level agents, the high-level reward function... Designed as a composite function balancing learning gain and communication cost, it not only includes the expected decrease in the global model loss function resulting from a single model interaction, but also introduces the actual communication cost arising from low-level decisions as a penalty term.

[0065]

[0066] The first term represents the expected decrease in the model loss function. Each RSU synchronizes its local loss function value through low-cost channel broadcasting or the Gossip protocol, thereby obtaining an effective estimate of the global loss. It is the low-level intelligent agent in Normalized communication cost fed back after selecting the optimal path at any time; It is the coupling coefficient that balances learning gain and communication overhead.

[0067] Step 5: Formalize the low-level communication path selection problem into a Markov decision process.

[0068] Step 5.1: Determine the state space

[0069] Low-level state space Focus on the target nodes selected by senior management The relevant real-time communication link information aims to provide the most direct and relevant cost information for lower-level decision-making; it includes: ground path channel information. and all available satellite relay path channel information. ,in To reduce the economic cost of using satellite link normalization; in addition, the state space is zero-padded or truncated to construct a fixed-dimensional feature vector;

[0070] Step 5.2: Determine the motion space

[0071] Low-level motion space It is a discrete space, that is, from the connection and Choose one of the available communication paths:

[0072]

[0073] Step 5.3: Determine the reward function

[0074] Low-level reward function The design goal is to guide the agent to learn a purely cost-optimal strategy, that is, to complete the communication task assigned by the higher layer in the fastest or most economical way; it is designed to be the negative of the generalized communication cost generated by the selected path:

[0075]

[0076] in and These are the end-to-end communication latency and normalized cost of the selected path, respectively. and It is a weight hyperparameter;

[0077] Step 6: Construct and train the hierarchical policy network using the graph embedding-based Actor-Critic (GE-HSAC) algorithm.

[0078] To address the two major challenges in resource scheduling within heterogeneous vehicular networks (V2N) and the inconsistency between long-term quality objectives and immediate cost objectives, this invention proposes a graph-embedded hierarchical synthetic synergistic Actor-Critic (GE-HSAC) algorithm based on the HQ-DRL framework. This method, by designing a heterogeneous compatible network architecture and introducing a cooperative Bellman update mechanism, achieves hierarchical cooperation and joint optimization between high- and low-level agents, thereby effectively improving resource scheduling performance in heterogeneous V2N environments.

[0079] Step 6.1: Construct a heterogeneous hierarchical strategy network architecture

[0080] This invention designs a heterogeneous network architecture that includes high-level graph-aware loops and low-level link-sensitive loops:

[0081] (1) High-level graph embedding policy network:

[0082] For high-level state space The topological characteristics of graphs are used to construct a policy network based on the graph attention mechanism (GAT). .

[0083] This network includes a feature extraction layer and a graph attention aggregation layer. For graphs... any node in (i.e., potential peer nodes), the network first takes its original feature vector (containing model semantic divergence) Model freshness Mapped to a high-dimensional embedding vector Subsequently, a multi-head attention aggregation operation is performed to calculate the central node. with neighboring nodes Attention coefficient between :

[0084]

[0085] in, The weight matrix is ​​a learnable matrix. Let || be the attention vector, and || denote the concatenation operation. By weighted aggregation of neighbor features, a node embedding representation that integrates the network topological context is generated. Finally, the output layer outputs the probability distribution of selecting each pair of peer nodes through the Softmax function, realizing the mapping from non-Euclidean space to discrete action space.

[0086] (2) Low-level link-sensitive policy network:

[0087] For the lower-level state space Based on the vectorization characteristics and real-time requirements, a lightweight fully connected network (MLP) is constructed as the policy network. .

[0088] The network receives actions from higher levels. A specific subset of link states (including ground SNR, satellite link cost, etc.) is identified, and link features are extracted through multi-layer nonlinear transformation to directly output the selection probability of each available communication path (direct connection / relay).

[0089] Step 6.2: Perform the Co-Bellman Update Training Process

[0090] The core of the GE-HSAC algorithm lies in overcoming the problem of inconsistent objectives between upper and lower layers in hierarchical reinforcement learning. This invention introduces a penalized coupled cooperative TD objective, forcing the higher-level value network to internalize the communication costs of lower layers when evaluating long-term learning value.

[0091] (1) Experience playback and sampling

[0092] During the training phase, joint experience tuples across time steps will be used. Stored in the experience replay pool In the middle. Among them, Includes high- and low-level reward decomposition. During training, from... The data is updated by randomly sampling small batches of data.

[0093] (2) High-level collaborative Critic network update

[0094] High-level Critic Network The aim is to evaluate the long-term value of the "node selection" action. To achieve collaboration, this invention reconstructs the Bellman update objective and defines a collaborative TD objective value. for:

[0095]

[0096] in, This represents the expected communication cost of executing this high-level decision under the current low-level strategy. For the target network, parameters are updated by minimizing the cooperative mean square Bellman error (S-MSBE). :

[0097]

[0098] This mechanism ensures that high-level critics not only focus on model convergence speed ( It also perceives the execution cost of lower layers through penalty terms, thereby avoiding the selection of nodes with high learning benefits but excessive communication costs.

[0099] (3) Low-level goal-oriented Actor update

[0100] The update objective of the low-level Actor network is to update the target node specified in the high-level network. Under constraints, find the path with the lowest cost. Update using a variant of the deterministic policy gradient (DPG) manifold:

[0101]

[0102] in For low-level Critic networks, this study specifically evaluates the communication cost efficiency of path selection.

[0103] (4) Target network soft update

[0104] To ensure the stability of the training process, a soft update mechanism is used to synchronize the target network parameters:

[0105] ,

[0106] in This is the smoothing coefficient.

[0107] Through collaborative decision-making and continuous learning between high- and low-level intelligent agents, this method can ultimately achieve efficient and rapid convergence of distributed learning tasks in a dynamic and heterogeneous vehicle network environment.

[0108] Compared with the prior art, the beneficial effects of the present invention are:

[0109] 1. This invention proposes a resource scheduling architecture based on hierarchical quality-driven deep reinforcement learning (HQ-DRL), effectively solving the decision-making problem caused by multi-objective coupling. This invention innovatively decomposes the complex heterogeneous vehicle network resource scheduling task orthogonally in the time and functional dimensions, decoupling it into high-level "learning quality-driven" peer node selection and low-level "communication cost-driven" path planning. This hierarchical design not only significantly reduces the dimensionality of the state-action space, avoiding the curse of dimensionality faced by traditional flat reinforcement learning, but also overcomes the short-sighted drawback of traditional methods that struggle to balance "long-term learning efficiency" and "instant communication cost" in dynamic environments.

[0110] 2. This invention achieves globally optimal scheduling in heterogeneous networks through graph embedding and a collaborative feedback mechanism. Relying on a hierarchical collaborative Actor-Critic algorithm (GE-HSAC) based on graph embedding, the higher layers utilize a graph attention network (GAT) to accurately perceive network topology and model semantic divergence, identifying high-value collaborative nodes; the lower layers, through a collaborative Bellman update mechanism, inject actual communication costs as a penalty signal into the higher-layer value evaluation. This bidirectional closed-loop mechanism ensures that the system can obtain the maximum model learning gain with the lowest communication cost, significantly accelerating the global convergence speed of distributed consensus learning in heterogeneous vehicular networks and improving the utilization rate of communication resources. Attached Figure Description

[0111] Figure 1 Hierarchical Quality-Driven Deep Reinforcement Learning (HQ-DRL) framework diagram;

[0112] Figure 2 Training flowchart for hierarchical collaborative Actor-Critic (GE-HSAC) based on graph embedding. Detailed Implementation

[0113] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0114] refer to Figure 1 - Figure 2 The method for resource scheduling in heterogeneous vehicle networks based on graph embedding hierarchical reinforcement learning includes the following steps:

[0115] Step 1: Constructing and quantifying a 3D heterogeneous vehicle networking system model and optimization problem

[0116] The main purpose of this step is to construct an accurate and model-based environment model, and to formally define the core technical problems that need to be solved based on the model, so as to lay a theoretical and physical foundation for subsequent intelligent decision-making problems and optimization algorithm design.

[0117] First, a three-dimensional heterogeneous vehicle-to-everything (V2X) physical model was constructed, incorporating Roadside Units (RSUs), ground-based mobile vehicles, and low-Earth orbit (LEO) satellites. The core decision nodes in the system are defined as Roadside Units (RSUs), whose set can be mathematically represented as follows: ,in This represents the total number of RSUs in the system. The entire system operates within a decentralized, distributed consensus learning framework, specifically implemented using the Gossip protocol. Within this framework, each RSU... Each maintains and trains a local model independently. and through a smartly selected peer RSU node Synchronize the model parameters to jointly promote the gradual convergence of a high-performance global model.

[0118] Then, to provide an accurate cost-benefit assessment before making scheduling decisions, the available communication link options for agents in the network were modeled and quantified. Communication paths were mainly divided into two categories. The first category is direct ground-based communication paths. When two RSUs... When an available terrestrial link exists, its communication quality will be quantitatively evaluated. The evaluation process first calculates the signal-to-noise ratio (SNR) between the two links, using the following formula:

[0119]

[0120] In the formula, This represents the transmitter's transmission power. and For the gain of the transmitter and receiver antennas, This represents the distance between two RSUs. It is the propagation path loss index. For channel fading, Represents channel bandwidth. Let be the noise power spectral density. Based on the signal-to-noise ratio mentioned above, and combined with the traditional Shannon formula, the transmission rate on this link can be calculated:

[0121]

[0122] Furthermore, a data size of [size missing] is transmitted via a ground path. The communication latency required for the model parameters can be precisely established as:

[0123]

[0124] in This delay takes into account signal propagation delay and node processing delay. When there is a direct ground connection or poor ground link quality, the second type of communication path is a satellite relay communication path. The total communication delay includes RSU-satellite uplink transmission delay, satellite-RSU downlink transmission delay, satellite relay, and long-distance propagation delay. Its calculation formula is as follows:

[0125]

[0126] in and The calculation is similar to that of ground path rate calculation, except that path loss is calculated using the FSPL model for spatial path loss. calculate, It represents the total propagation delay and processing delay of the satellite link.

[0127] Based on the aforementioned physical and communication models, the technical challenges that this method aims to address are ultimately formalized into a global optimization model. The optimization objective of this model is to learn an optimal scheduling strategy. The goal is to minimize the expected total time required for the global model to reach a preset convergence threshold, so that each RSU agent in the system selects an optimal peer node and an optimal path at each decision time. This optimization objective can be modeled as the following optimization problem:

[0128]

[0129] in, It is the total number of decision steps required for the model to converge. It is in the The total delay generated at each decision moment consists of communication delay. and local computing latency It consists of two parts. Solving this optimization problem must be done under several constraints of physical feasibility and logical integrity. First, the single peer-to-peer node selection constraint (C1): This constraint ensures that each RSU can only choose one peer node to interact with at any given time, thus avoiding resource conflicts and logical confusion. Second, the single communication path selection constraint (C2): It guarantees that once a pair of interacting nodes is determined, there must be exactly one communication path between these nodes activated for data transmission. Third, the learning task convergence constraint (C3) is expressed as follows: This constraint clarifies the ultimate goal of the entire learning task: the loss function value of the global model must ultimately not exceed a pre-set convergence threshold.

[0130] Step 2: Decouple the optimization problem using a hierarchical reinforcement learning framework.

[0131] To address the inherent complexity of the aforementioned optimization problem, this invention proposes a hierarchical quality-driven deep reinforcement learning (HQ-DRL) framework. The basic idea of ​​this approach is to decouple a highly coupled joint optimization problem into hierarchical sub-problems that are separate in decision-making and objective-oriented aspects but closely related in execution. Specifically:

[0132] (1) High-level sub-problem (peer node selection): focuses on solving the strategic problem of "who to communicate with". Its decision-making goal is to select the node that is most likely to bring the greatest long-term learning benefits to the convergence of the global model based on a multi-dimensional comprehensive evaluation of the model quality of each potential peer node. This is a decision-making process driven by "learning quality".

[0133] (2) Lower-level sub-problems (communication path selection): Focuses on solving the execution problem of "how to communicate". Its decision-making objective is to select the path with the lowest communication cost to efficiently complete the model interaction under the guidance of higher-level decision (i.e., selecting the communication object) and based on real-time link status information. This is a decision-making process driven by "communication cost".

[0134] Step 3: Formalize the hierarchical subproblems into Markov Decision Processes (MDPs).

[0135] The above high- and low-level subproblems are formalized into independent MDPs so that reinforcement learning algorithms can be applied.

[0136] Sub-step 1: MDP construction for the high-level peer node selection sub-problem

[0137] First, the high-level state space ( It is constructed as a local RSU Directed weighted graph centered on The graph structure contains two types of key feature information: (1) Node features: describing candidate nodes The absolute quality, including model freshness (Measuring the timeliness of node model information), aggregated data volume (Representing the data weights behind the model for this node) and model noise variance (Reflecting the stability of model parameters) Three dimensions. (2) Edge features: describing candidate nodes With the central node The relative relationships between them include model semantic divergence. (Based on the Euclidean distance or KL divergence calculated from the prototypes of both classes, representing the difference between the two models in the feature space; the greater the difference, the higher the potential information gain) and link connectivity probability. (Reflects the prior probability that there are available physical links between nodes, serving as a pre-screening indicator for topology awareness).

[0138] Secondly, the space for action at higher levels ( () is a discrete space, representing the selection of a unique peer node from all RSU sets except itself as the target peer node for this interaction. Decision set:

[0139] .

[0140] Finally, the high-level reward function ( The design of this function, intended to enable collaboration between high- and low-level agents and prevent higher-level agents from prioritizing learning gains while ignoring communication costs, is defined as a composite function that balances learning gains and communication costs.

[0141]

[0142] The first term represents the expected decrease in the global model loss function resulting from a single model interaction; the second term... It is the actual execution communication cost fed back by the low-level intelligent agent; It is the coupling coefficient that balances learning gain and communication overhead.

[0143] Sub-step 2: MDP construction for the low-level communication path selection sub-problem

[0144] First, the lower-level state space ( Focusing on the target nodes already selected by senior management Directly relevant real-time link cost information. This state consists of two parts: one part is the channel information of the ground path, represented as... The other part is the channel and cost information for all visible satellite relay paths, represented as... .

[0145] Secondly, the lower-level action space ( ) represents from the connection and The decision set for selecting one of all available communication paths to perform the communication task: .

[0146] at last, Reward function ( The design is to assign a negative value to the generalized communication cost generated by the selected path, guiding the agent to learn a cost-optimal strategy:

[0147]

[0148] in and These represent the total end-to-end communication latency and normalized cost of the selected path, respectively. and The hyperparameters used to adjust the weights.

[0149] Step 4: Train the policy network using the hierarchical collaborative Actor-Critic algorithm based on graph embedding (GE-HSAC).

[0150] To address the multi-objective coupling problem in heterogeneous vehicle network resource scheduling, this invention employs the GE-HSAC algorithm to perform end-to-end optimization of the hierarchical strategy.

[0151] Sub-step 1: Constructing a heterogeneous layered strategy network architecture

[0152] First is the High-Level Graph Embedding Policy Network (GAT). This addresses the high-level state... Given the graph structure characteristics, a graph attention network is used as a policy function approximator. The network includes a feature extraction layer and a multi-head graph attention layer. For the graph... Any neighboring node in The network first uses an attention mechanism to calculate its relationship with the central node. Attention coefficient between :

[0153]

[0154] The neighbor features are weighted and aggregated using this coefficient to generate a node embedding representation that integrates network topology and semantic context. Finally, the probability distribution of each peer node is selected by outputting Softmax.

[0155] Secondly, there's the low-level link-sensitive policy network (MLP). (Targeting...) state Leveraging the vectorization characteristics, a lightweight multilayer perceptron (MLP) is used as the policy function approximator. It receives a specific subset of link states determined by higher-level actions and directly outputs the selection probability of each available communication path.

[0156] Sub-step 2: Perform a Synergistic Bellman Update

[0157] The training process follows the (GE-HSAC) paradigm and introduces a penalty coupling mechanism to force the high-level Critic network to internalize the communication costs of the low-level layers. Specifically, it includes the following steps:

[0158] (1) Experience storage. At each time step, joint experience tuples will be stored. Store in the experience replay pool .in Includes rewards for both high-level and low-level decomposition.

[0159] (2) High-level collaborative Critic update. To address goal inconsistency, a high-level Critic network... The update no longer uses the standard TD target, but instead adopts the collaborative TD target value. :

[0160]

[0161] because The penalty items for low-level feedback are already included. By minimizing the Synergistic Mean Square Bellman Error (Synergistic MSBE) This allows high-level Critics to accurately assess the true utility of "high-value but high-cost" actions, thereby avoiding the selection of nodes with high learning benefits but excessive communication costs.

[0162] (3) Low-level goal-oriented update. The low-level network adopts the standard deterministic policy gradient method, with the single objective of minimizing path cost in parameter updates. Update.

[0163] (4) Target network soft update. A soft update mechanism is adopted. Synchronize the target network parameters to ensure training stability.

[0164] Through the iterative optimization process described above, it is ultimately possible to ensure that the system can quickly and effectively complete the distributed learning task and converge in a dynamic and heterogeneous vehicle network environment.

[0165] Step 5: Simulation Verification Platform Setup and Performance Evaluation

[0166] To rigorously and scientifically verify the authenticity and effectiveness of the proposed HQ-DRL method, a detailed and complete simulation platform needs to be designed and simulation experiments conducted. For the construction of the simulation platform, it is recommended to use mainstream industry simulation toolchains, including: SUMO (Simulation of Urban Mobility) for simulating urban traffic flow and the large / small movement trajectories of vehicles; NS-3 (Network Simulator 3) for simulating communication protocols in heterogeneous network environments and fading and interference models in channel environments; and mainstream deep learning frameworks in the Python environment, such as PyTorch or TensorFlow, for implementing the proposed GE-HSAC algorithm and complex network models. A complete simulation scenario needs to be defined for specific scenario parameters. This includes specifying a real-world city traffic map and setting vehicle density and speed distribution, including Gaussian distribution and RSU geographic distribution; setting network parameters according to relevant communication standards, including the IEEE 802.11p / bd standard for terrestrial communication and the DVB-S2 / RCS2 standard for satellite communication; specifying a specific federated learning task, namely training an image classification or object detection model using a set of images collected in a distributed manner from vehicles (CIFAR-10 or a more challenging autonomous driving dataset), while setting the model size. and the convergence threshold of a good learning model .

[0167] To quantitatively evaluate the performance of this method, a set of performance evaluation metrics needs to be defined.

[0168] (1) Global model convergence speed: The convergence speed is represented by a curve of the loss function value or accuracy of the global model on the independent test set as a function of simulation time (communication rounds). The smaller / higher the value, the faster the method.

[0169] (2) Communication resource utilization: Calculate the total communication overhead (total amount of data transmitted or total cumulative delay) required for convergence.

[0170] (3) Average decision delay: the average of the total delay for each decision step. Indicates real-time performance.

[0171] To highlight the innovation and superiority of this method, a set of representative benchmark comparison algorithms is needed:

[0172] (1) Random scheduling: The agent randomly selects peer nodes and paths at every moment as the lower bound of performance.

[0173] (2) Communication-optimal greedy scheduling: This represents a "communication-aware" strategy. The agent always selects the node and path with the minimum communication delay or the highest signal-to-noise ratio, ignoring the learning value of the model.

[0174] (3) Learning-optimal greedy scheduling: This represents the "learning perception" strategy. The agent always chooses the model semantic divergence. The largest node interacts with the other node, ignoring the communication cost.

[0175] (4) Flattened DRL: This algorithm uses a single non-hierarchical deep reinforcement learning agent, including DDPG or PPO), whose action space is a huge Cartesian product of (peer nodes × communication paths).

[0176] The comparison can intuitively verify the superiority of the hierarchical decoupling architecture of this invention in reducing decision-making complexity and improving learning efficiency.

[0177] In summary, through the complete process of the above specific implementation scheme, the present invention can ultimately be fully implemented, deployed, and verified, thereby achieving its goal of providing an intelligent, efficient, and adaptively advanced resource scheduling system that balances long-term learning quality and real-time communication costs in complex heterogeneous vehicle networking environments.

[0178] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0179] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous vehicle network resource scheduling method based on graph embedding hierarchical reinforcement learning, characterized in that, The method includes the following steps: Step 1: Construct and quantify a 3D heterogeneous vehicle networking system model Step 1.1: Define system scenarios and components First, a three-dimensional heterogeneous vehicle-to-everything (V2X) system model needs to be established, consisting of roadside units (RSUs), ground mobile vehicles, and low-orbit LEO satellites. Step 1.2: Quantify the terrestrial communication path A hierarchical communication model was constructed, and two communication paths were quantified; the first type is the direct ground connection path. Step 1.3: Quantify the satellite communication path The second category is satellite relay paths; Step 2: Establish an optimization problem aimed at accelerating model convergence. Step 2.1: Define the optimization objective; Step 2.2: Determine the constraints; Step 3: Decouple the optimization problem using a hierarchical reinforcement learning framework. Step 3.1: Define the subproblem of selecting high-level peer nodes; Step 3.2: Define the low-level communication path selection subproblem; The lower-level subproblems focus on solving the strategy of "how to communicate", and their goal is to select the communication path with the lowest cost to complete the model interaction, given that the higher-level decision has been made. Step 4: Formalize the high-level peer node selection problem into a Markov decision process. Step 4.1: Determine the state space; Step 4.2: Determine the motion space; Step 4.3: Determine the reward function; Step 5: Formalize the low-level communication path selection problem into a Markov decision process. Step 5.1: Determine the state space; Step 5.2: Determine the motion space; Step 5.3: Determine the reward function; Step 6: Use the graph embedding-based hierarchical collaborative Actor-Critic algorithm, namely GE-HSAC, to train the hierarchical policy network. Step 6.1: Construct a heterogeneous layered strategy network architecture; High-level strategy network It is constructed using a graph attention network (GAT); while the low-level policy network... It is constructed using a multilayer perceptron (MLP). Step 6.2: Perform the co-Bellman update training process.

2. The heterogeneous vehicle network resource scheduling method based on graph embedding hierarchical reinforcement learning according to claim 1, characterized in that, In step 6, the high-level policy network... The construction and training of [the system] are characterized by the following features: A graph attention network (GAT) is used as a function approximator for the high-level policy network. The graph attention network is defined in step 4 and consists of a graph-structured state space composed of peer node model quality features, model semantic divergence calculated based on class prototypes, and communication environment features. For input; This graph attention network learns node representations through a multi-head attention mechanism. In each network layer, iterative attention aggregation and update steps are performed. First, the attention coefficients between the center node and each neighbor node are calculated, which represent the importance of the neighbor node features to the center node. Then, the feature vectors of all neighbor nodes are weighted and aggregated according to the attention coefficients, and the representation of the center node is updated by combining its own features. The network learns a context-aware embedding representation that integrates network topology, node model quality, and link communication state; Finally, the output layer of the graph attention network generates a normalized probability distribution through a softmax function. This distribution corresponds to the probability value of selecting each peer node as a communication target. The GE-HSAC algorithm is then used, based on the high-level reward function defined in step 4.3, which integrates the decrease in the global model loss function with the communication cost penalty term from lower-level feedback. The parameters of the graph attention network Continuous optimization and updates are performed; the high-level reward function guides the high-level policy network to balance model quality and communication overhead when selecting peer nodes.

3. The heterogeneous vehicle network resource scheduling method based on hierarchical quality-driven reinforcement learning according to claim 1, characterized in that, In step 6, the low-level policy network is... The construction and training of [the system] are characterized by the following features: A multilayer perceptron (MLP) is used as the function approximator for the lower-level policy network. The multilayer sensor receives a fixed-dimensional vector as input, which is defined in step 5 and contains real-time cost indicators, including latency and cost, of all available paths connecting the target node, including ground paths and satellite paths. The input vector undergoes a nonlinear transformation through multiple fully connected hidden layers of the multilayer perceptron to extract the complex relationship between the costs of different communication paths. Finally, the output layer of the multilayer perceptron assigns a selection probability to each available communication path through a Softmax function to achieve fast path decision-making; A hierarchical collaborative Actor-Critic algorithm based on graph embedding, namely GE-HSAC, is employed, based on the low-level reward function related to communication cost defined in step 5.

3. The parameters of the multilayer perceptron We will continue to optimize and update it.

4. The heterogeneous vehicle network resource scheduling method based on hierarchical quality-driven reinforcement learning according to claim 2, characterized in that, The steps of updating the graph attention network using the graph embedding-based Actor-Critic (GE-HSAC) algorithm include: a) Action selection and environment interaction: At each time step of training, the high-level agent selects actions based on the current graph attention network strategy. The output probability distribution is sampled, and then a higher-level action is selected. And perform the action; b) Experience storage: After the action is executed, the actual communication cost is received from the underlying agent, and a high-level reward that incorporates a communication penalty is calculated. Observe the state of the next high level and the experience tuple Store in the experience replay pool ; c) Cooperative Critic Network Update: Randomly select from the experience replay pool A small batch of experience points is extracted to update the high-level Critic network. A penalty coupling mechanism is introduced here to calculate the cooperative TD target value. The calculation method is as follows: ； Minimize the cooperative mean square Bellman error using gradient descent, i.e., the Synergistic MSBE loss function. To complete the parameter update: ； The collaborative update mechanism forces the high-level Critic network to internalize the communication execution costs of the lower layers when evaluating long-term value; wherein and These are the target Critic network and the target Actor network, respectively, where parameter updates are slower. d) Actor network update: using policy gradient To update the parameters of the high-level Actor network, i.e., the graph attention network. : ； And update the network parameters along the gradient of this policy; e) Target network iterative update: To ensure training stability, the parameters of both target networks are softly updated after each iteration. ； in It is a soft update coefficient.

5. A heterogeneous vehicle network resource scheduling method based on hierarchical quality-driven deep reinforcement learning according to claim 3, characterized in that, The steps of updating the multilayer perceptron using the graph embedding-based Actor-Critic (GE-HSAC) algorithm include: a) Experience storage: At each time step of the underlying decision-making process, experience tuples are stored. Store in the experience replay pool ; b) Critic network update: from the experience replay pool Randomly sample small batches of data and minimize the loss function. Update a low-level Critic network : ； Where TD target value ; c) Actor network update: using policy gradient To update the parameters of the lower-level Actor network, i.e., the multilayer perceptron. : ； And along this policy gradient, from low to high layers, the parameters of the low-level Actor network are updated using gradient series. : ； in The learning rate for the lower-level Actor network; d) Target network update: After each iteration, the parameters of the underlying target network are softly updated.

6. A resource scheduling system in a heterogeneous vehicle network, based on the heterogeneous vehicle network resource scheduling method according to any one of claims 1-5, characterized in that, The system includes: a) Model building module, used to build a heterogeneous vehicle network system model including roadside units (RSU), ground vehicles and low-orbit satellites (LEO), and to quantify the performance of available ground direct connection paths and satellite relay paths between the roadside units, and to establish communication latency and data rate models for each path. b) Problem formalization module, which is used to formalize the resource scheduling problem into an optimization problem with the objective of minimizing the expected total time for the distributed learning task to achieve global model convergence, and to set constraints on peer node selection and communication path selection; c) A hierarchical decision-making module for performing hierarchical reinforcement learning decisions, the hierarchical decision-making module comprising: i. A high-level decision unit is configured to construct a graph-structured state space centered on a local roadside unit at each decision time. The state space contains node features and edge features of potential peer nodes, and utilizes a graph attention network as a high-level policy network to make a high-level decision to select a target peer node based on the output of the state space. ii. A low-level decision unit, configured to, upon receiving the high-level decision, construct a real-time link state space containing all available communication paths connecting the target peer node, and utilize a multilayer perceptron as a low-level policy network to output a low-level decision to select a specific communication path; d) A network training module is used to calculate the low-level independent reward and the high-level coupled reward that incorporates the feedback from the execution of high-level and low-level decisions in the environment; and to update the parameters of the high-level policy network and the low-level policy network respectively using the GE-HSAC algorithm based on cooperative Bellman update.

7. The system as described in claim 6, characterized in that, The model quality information acquired by the high-level decision-making unit includes at least one of the following: a) Model freshness of peer nodes; b) The amount of data aggregated by the model of peer nodes; c) Model semantic divergence calculated based on the distance between the local and peer node class prototypes; d) Noise variance of model parameters for peer nodes; e) Probability of link connectivity between local roadside units and peer nodes.

8. The system as described in claim 6, characterized in that, The high-level policy network is an attention network, or GAT, and its configuration is as follows: It includes a feature embedding layer and a multi-head graph attention layer. First, the original features of potential peer nodes are mapped into high-dimensional embedding vectors. Then, the attention mechanism is used to calculate the attention coefficients between the center node and each neighbor node. The coefficients represent the importance of the neighbor nodes to the center node. Based on the coefficients, the features of neighboring nodes are weighted and aggregated to generate a node embedding representation that incorporates the heterogeneous network topology context, and the probability distribution for selecting each peer node is generated through the output layer.

9. The system as described in claim 6, characterized in that, The multilayer perceptron (MLP) in the low-level decision-making unit is configured as follows: it receives a fixed-dimensional vector as input, which is composed of real-time channel state information and cost indicators of all available communication paths connecting target peer nodes, including ground and satellite paths; it performs nonlinear transformation through a fully connected hidden layer to extract cost features of different paths; and it assigns selection probabilities to each available communication path through the output layer.

Images