Internet of vehicles task unloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning

Abstract

The invention relates to the technical field of computers, in particular to an Internet of Vehicles task unloading optimization method based on a semantic heterogeneous graph and hierarchical reinforcement learning, which comprises the following steps: establishing a vehicle state estimation model, calculating the future position and speed state of each vehicle, and further calculating the remaining connection time between the vehicles; the method comprises the following steps: introducing remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantic heterogeneous graph structure, comprehensively capturing a composite semantic relationship in an Internet of Vehicles task unloading environment, and generating a global state representation vector; establishing a hierarchical reinforcement learning strategy, taking the global state representation vector as an input basis of the hierarchical reinforcement learning strategy, and outputting an optimal unloading path and a target node index; according to the method, through fusion of Kalman filtering link prediction, heterogeneous graph state modeling and a layered Actor-Critic reinforcement learning architecture, the key problems of insufficient link unstable state expression, action space explosion and the like in Internet of Vehicles task unloading are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning for Internet of Vehicles. BACKGROUND

[0002] With the continuous development of intelligent transportation and Internet of Vehicles technology, task offloading has become an important means to relieve the computing power pressure of on-board devices and improve service performance in the intelligent vehicle environment. In typical Internet of Vehicles application scenarios, a large number of perception and decision tasks generated by vehicles often have high computational complexity and strict real-time requirements. If completely relying on local processing, it will be difficult to meet the quality of service guarantee. Therefore, the industry has proposed a strategy of dynamically offloading tasks to adjacent vehicles or edge servers deployed on the roadside for processing.

[0003] To achieve efficient task offloading and distribution, there are currently three representative technical paths: offloading methods based on centralized or heuristic strategies: this kind of method mostly uses greedy algorithm, shortest path selection or load balancing scheme, makes task-node matching decision according to preset rules, usually relies on global state information and formulates fixed strategy under static topology. The advantage is that it is simple to implement and efficient to run, but due to the lack of adaptive ability, it is difficult to cope with the frequent changes of connection state and node load in Internet of Vehicles environment, leading to false distribution in dynamic network, lack of flexibility and generalization ability;

[0004] Dynamic offloading methods based on deep reinforcement learning: in recent years, reinforcement learning has been widely used in Internet of Vehicles task scheduling. It learns offloading strategies autonomously according to the current system state by constructing a policy network, and realizes online intelligent decision-making. Although this method improves the adaptability of the strategy, most studies directly use vectorized state input, ignoring the structural multiple association information between tasks and computing nodes, resulting in insufficient state expression ability. In addition, due to the large number of offloading targets faced by each task, the dimension of action space grows exponentially, and the policy network training faces the problems of slow convergence and poor generalization, especially in multi-task scenarios.

[0005] Structured state modeling method by introducing graph neural network: to improve the state modeling ability, some studies use graph neural network (GNN) to encode the system structure, regarding vehicle nodes, task nodes and edge servers as entity nodes in the graph, and extract structural semantic representation through graph embedding method. Although this kind of method enhances the state expression ability to some extent, most of them are still based on isomorphic static graph construction, which can only capture the basic connection information between nodes, and is difficult to cover the dependency relationship between tasks, the dynamic changes of communication link and the semantic differences between heterogeneous node types. This modeling method has obvious limitations in expressing complex offloading environment, which restricts the accuracy and generalization ability of reinforcement learning strategy.

[0006] In addition, in actual operation, the high-speed mobility of the Internet of Vehicles causes obvious uncertainty in the connection time of the communication link. The existing method relies on indicators such as instantaneous bandwidth or RSSI to make decisions, ignoring the forward-looking prediction of the connection stability or the remaining communication time. This will cause part of the offloaded tasks to fail or timeout during transmission due to link interruption, seriously affecting the stability and service timeliness of the system. SUMMARY

[0007] In view of this, the present application proposes a task offloading optimization method for the Internet of Vehicles based on semantic heterogeneous graph and hierarchical reinforcement learning, aiming to solve at least one of the problems in the above background technology.

[0008] The present application provides a task offloading optimization method for the Internet of Vehicles based on semantic heterogeneous graph and hierarchical reinforcement learning, comprising the following steps:

[0009] S1, a vehicle state estimation model is established to calculate the future position and speed state of each vehicle, and further calculate the remaining connection time between vehicles;

[0010] S2, introduce the remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capture the complex semantic relationship in the task offloading environment of the Internet of Vehicles, and generate a global state representation vector;

[0011] S3, a hierarchical reinforcement learning strategy is established, and the global state representation vector is used as the input basis of the hierarchical reinforcement learning strategy, and the optimal offloading path and target node index are output.

[0012] In some embodiments, when the step S1 of establishing a vehicle state estimation model, calculating the future position and speed state of each vehicle, and further calculating the remaining connection time between vehicles, comprises:

[0013] The vehicle state estimation model is established based on the Kalman filtering method, and each vehicle v j A state vector s j (t) is established j (t), y j (t), v x,j (t), v y,j (t) Τ , where x j , y j represent the two-dimensional position coordinates of the vehicle, and v x,j , v y,j represent the speed components in each direction;

[0014] The vehicle updates the state estimation, including the estimated mean and covariance matrix of position and velocity, according to the previous state and observation input at each time step;

[0015] According to the Kalman prediction update process, the optimal state estimation value of all vehicles is obtained at the current time t For calculating the future relative trajectory and position change between any pair of vehicles (v u ,v v );

[0016] Based on the predicted position and velocity, the remaining sustainable communication time between any pair of vehicles is calculated This value is used as the dynamic edge weight of the node-node edge.

[0017] In some embodiments, when the remaining sustainable communication time between any pair of vehicles is calculated based on the predicted position and velocity , it includes:

[0018] Any pair of communication nodes is (i,j);

[0019] Relative speed

[0020] Relative distance

[0021] The remaining connection time is , where R max represents the maximum communication coverage radius.

[0022] In some embodiments, when the remaining connection time between vehicles in step S2 is introduced as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, fully capture the complex semantic relationship in the task offloading environment of Internet of Vehicles, and generate a global state representation vector, it includes:

[0023] The task set is denoted as Each task τ i has a computing intensity c i , a data volume d i , and a maximum tolerable delay δ i , and its state feature representation is a three-dimensional vector

[0024] The computing nodes include a set of vehicle nodes V={v1,v2…,v V} and a set of RSU nodes Each node characterizes its current system state, including available computing power μ j , task queue length q j and communication bandwidth b j , which is uniformly represented as a node feature vector

[0025] In some embodiments, the step S2 of introducing the inter-vehicle remaining connection time as a dynamic edge weight to build the multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationships in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, further comprises:

[0026] Building a unified multi-semantics heterogeneous graph structure for representing the dependency relationship between tasks, the reachable relationship between tasks and computing nodes, and the communication connection state between nodes;

[0027] wherein the node set includes three types of heterogeneous entities: task nodes, vehicle nodes, and RSU nodes, corresponding to task execution units, mobile computing units, and fixed edge resource units, respectively;

[0028] The edge set ε includes:

[0029] Task-task edges ε ττ for modeling the execution order or data dependency relationship between tasks;

[0030] Task-node edges ε τn for representing the offloading reachability between tasks and computing nodes;

[0031] Node-node edges ε nn for representing the communication connectivity between computing nodes.

[0032] In some embodiments, the step S2 of introducing the inter-vehicle remaining connection time as a dynamic edge weight to build the multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationships in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, further comprises:

[0033] Designing independent feature input channels for task nodes, vehicle nodes, and RSU nodes, respectively, and mapping their original feature vectors to the same dimension representation space through type-aware linear transformation, including:

[0034] Through the projection function f t (·), f v (·), f r (·), the task node vehicle node and RSU node are converted into initial embedding vectors

[0035] For each hop propagation layer, the node embedding of the l-th layer is updated by the weighted information from all types of neighbors, in the form of: wherein, denotes the edge type set, denotes the neighbor set of node i under edge type e, is the normalized coefficient calculated based on the attention mechanism, W r is the trainable transformation matrix corresponding to the edge type, and σ(·) is a nonlinear activation function;

[0036] The embedding results of all task nodes are combined into a task embedding matrix The embeddings of all optional target nodes are combined into a target node embedding matrix and a global state representation vector is generated through a pooling function

[0037] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and target node index, includes:

[0038] A high-level strategy is used to determine the offloading mode of each task τ i , and the output result is one of three discrete actions: local execution (0), offloading to a neighboring vehicle node (1), or uploading to an edge server RSU (2);

[0039] The task node embedding vector and the global state embedding are taken as joint inputs and processed through a set of feedforward neural networks, finally generating a three-dimensional vector representing the scores of each offloading mode, and then normalized to a probability distribution through a softmax function.

[0040] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and target node index, further includes:

[0041] After the high-level strategy determines that the task needs to be offloaded, a dynamic threshold network is introduced to adaptively calculate a filtering threshold θ∈[0,1] for each task; the threshold network takes the embedding representation of the task and the global state vector of the system as inputs, and processes them through a set of trainable feedforward neural network structures, finally outputs a task-specific continuous threshold θ i so that it falls within the interval [0,1];

[0042] After obtaining the threshold θ i , the current task τ i and all candidate nodes n jsemantic similarity score between where f(·) can be a dot product, a bilinear mapping, or other similarity function.

[0043] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and target node index, further comprising:

[0044] After completing the high-level offloading mode determination and dynamic threshold screening, for each task τ i , receiving its embedding vector and the filtered candidate node embedding set as input, constructing an action representation space between task-node pairs;

[0045] Using a multi-layer perception network with an attention mechanism, calculate the matching score q j of each candidate target node n ij with the current task, and normalize the score to a probability distribution π(a i =n j |τ i ) through a softmax function, used to select the offloading target node from it

[0046] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and target node index, further comprising: using the time difference method to minimize the mean square error between the current estimated value and the actual return.

[0047] Compared with the prior art, the application has the beneficial effects that the application considers the influence of frequent changes of Internet of Vehicles communication connection and unstable link on task offloading success rate, and introduces a connection duration prediction mechanism based on Kalman filtering. By modeling the relative motion state between vehicles, the remaining connection duration is predicted in real time, and a link filtering factor is introduced in the offloading strategy, which effectively avoids the link interruption during relay transmission of the task, thereby significantly improving the robustness and execution success rate of the system offloading. The application considers the complex semantic relationship between task dependency, node heterogeneity and link dynamics, and proposes a semantic decomposition type heterogeneous graph state modeling method. By constructing a task-node graph and a task-task graph, the offloading cost between tasks and targets and the execution dependency logic within tasks are captured respectively. Combined with a heterogeneous graph neural network to extract embedding vectors, structural and semantic fusion is realized in the state expression stage, thereby providing more accurate environment perception for the subsequent offloading strategy, and significantly improving the accuracy and scalability of the offloading decision. The application considers the problem of exponential growth of action space dimension in the multi-task-multi-node offloading strategy, and designs a hierarchical Actor-Critic reinforcement learning architecture, which first screens the offloading candidate set at the high level, and then makes fine decisions on the specific offloading target at the low level. This mechanism effectively compresses the decision space, relieves the training instability of the strategy network under large action space, and improves the generalization performance and convergence speed of the strategy in complex offloading scenarios.

[0048] The foregoing general description and the following detailed description are merely exemplary and explanatory, rather than limiting the present disclosure.

[0049] Other features and aspects of the present disclosure will become more apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0051] Figure 1 The semantic heterogeneous graph and hierarchical reinforcement learning based Internet of Vehicles task offloading optimization method flowchart provided for the embodiments of the present application;

[0052] Figure 2 The semantic heterogeneous graph and hierarchical reinforcement learning based Internet of Vehicles task offloading optimization method flowchart provided for the embodiments of the present application;

[0053] Figure 3The flowchart illustrates the vehicle network task offloading optimization method based on semantic heterogeneous graphs and hierarchical reinforcement learning, as provided in this embodiment of the invention. Detailed Implementation

[0054] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] See Figures 1-3 As shown, a vehicle networking task offloading optimization method based on semantic heterogeneous graphs and hierarchical reinforcement learning according to an embodiment of this application includes the following steps:

[0056] S1. Establish a vehicle state estimation model to estimate the future position and speed of each vehicle, and further calculate the remaining connection time between vehicles.

[0057] S2. Introduce the remaining connection time between vehicles as dynamic edge weights to construct a multi-semantic heterogeneous graph structure, comprehensively capture the complex semantic relationships in the vehicle networking task offloading environment, and generate a global state representation vector;

[0058] S3. Establish a hierarchical reinforcement learning strategy, with the global state representation vector serving as the input basis for the hierarchical reinforcement learning strategy, and outputting the optimal unloading path and target node index.

[0059] In some specific embodiments, step S1, which involves establishing a vehicle state estimation model, calculating the future position and speed of each vehicle, and further calculating the remaining connection time between vehicles, includes:

[0060] The vehicle state estimation model is established based on the Kalman filter method and is oriented towards each vehicle v j Establish state vector s j (t)=[x j (t),y j (t),v x,j (t),v y,j (t)] Τ , where x j ,y j Represents the two-dimensional position coordinates of the vehicle, v x,j ,v y,j This represents its velocity components in each direction;

[0061] The vehicle updates the state estimation, including the estimated mean and covariance matrix of position and velocity, according to the previous state and observation input at each time step;

[0062] According to the Kalman prediction update process, the optimal state estimation value of all vehicles is obtained at the current time t For calculating the future relative trajectory and position change between any pair of vehicles (i,j) u ,v v );

[0063] Based on the predicted position and velocity, the remaining sustainable communication time between any pair of vehicles is calculated This value is used as the dynamic edge weight of the node-node edge.

[0064] In some embodiments, the remaining sustainable communication time between any pair of vehicles is calculated based on the predicted position and velocity , including:

[0065] Any pair of communication nodes is (i,j);

[0066] Relative speed

[0067] Relative distance

[0068] The remaining connection time is where R max represents the maximum communication coverage radius.

[0069] It should be understood that in the Internet of Vehicles scenario, due to the high-speed movement of nodes, the persistence of link connection cannot be modeled statically. The present application uses a Kalman filter to estimate the vehicle position-velocity state vector S i (t)=[x i (t),y i (t),v x,i (t),v y,i (t)] T in real time. At each time step, the vehicle node receives external positioning information and speed observation, and estimates its state at the next time through the state transition function and the observation update function, improving the accuracy of position prediction.

[0070] In the Internet of Vehicles offloading environment, the high dynamic mobility of vehicles will cause the link connection to have obvious non-stationarity and uncertainty. In order to realize the dynamic modeling and prediction of the reachability of the communication link, the present application introduces a vehicle state estimation model in the system state modeling stage, which is used to calculate the future position and velocity state of each vehicle, and further calculate the remaining communication time between vehicles to support the dynamic connectivity modeling of node-node edges.

[0071] This model is built based on the Kalman filter method and is designed for each vehicle v. j Establish state vector s j (t)=[x j (t),y j (t),v x,j (t),v y,j (t)] Τ , where x j ,y j Represents the two-dimensional position coordinates of the vehicle, v x,j ,v y,j This represents the velocity components in each direction. At each time step, the vehicle updates its state estimate based on the previous state and observed inputs, including the estimated mean and covariance matrices of position and velocity. Following the Kalman prediction update process, the system obtains the optimal state estimate for all vehicles at the current time t. Used to calculate any vehicle pair (v u ,v v The future relative trajectory and positional changes between them.

[0072] Based on predicted location and speed, the remaining sustainable communication time between any vehicle pair (i.e., the expected duration within the communication radius R) can be calculated. The calculation employs either a linear position change model or a velocity hold-off model, based on the current state estimation. and Derive the distance variation curves of the future locations of the two workshops over time, and calculate the maximum time window within which the distance is less than the communication radius R, as the basis for determining the distance. This value is used as the dynamic edge weight input to the heterogeneous graph state modeling module for node-to-node edges, and is used to characterize the reliability and timeliness of the link.

[0073] The vehicle state estimation module executes periodically, receiving real-time vehicle state observation data provided by the system perception layer or traffic simulation module. It supports online estimation and high-frequency refresh mechanism, enhances the adaptability of the model to link instability, and provides prediction-driven structural information support for the policy network.

[0074] Based on the state estimation results, for any pair of communication nodes (i,j), this invention calculates their relative velocity. and relative distance Further estimate of remaining connection time is Where R max This represents the maximum coverage radius of the communication. The prediction results are embedded into the graph model as edge weight information of the communication edges, which helps to avoid communication paths that are close to disconnection during strategy selection.

[0075] In some embodiments, the step S2 of introducing the inter-vehicle remaining connection time as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationships in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, comprises:

[0076] The task set is denoted as Each task τ i has a computing intensity c i , a data volume d i , and a maximum tolerable delay δ i , and its state feature representation is a three-dimensional vector

[0077] The computing nodes include a set of vehicle nodes V = {v1, v2, …, v V} and a set of RSU nodes Each node characterizes its current system state, including available computing capability μ j , task queue length q j , and communication bandwidth b j , which are uniformly represented as a node feature vector

[0078] In some embodiments, the step S2 of introducing the inter-vehicle remaining connection time as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationships in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, further comprises:

[0079] A unified multi-semantics heterogeneous graph structure is constructed to represent the dependency relationship between tasks, the reachability relationship between tasks and computing nodes, and the communication connection state between nodes;

[0080] The node set includes three types of heterogeneous entities: task nodes, vehicle nodes, and RSU nodes, corresponding to task execution units, mobile computing units, and fixed edge resource units, respectively;

[0081] The edge set ε includes:

[0082] Task-task edges ε ττ model the execution order or data dependency relationship between tasks;

[0083] Task-node edges ε τn represent the offloading reachability between tasks and computing nodes;

[0084] Node-node edges ε nn represent the communication connectivity between computing nodes.

[0085] In some specific embodiments, when introducing the remaining connection time between vehicles as dynamic edge weights to construct a multi-semantic heterogeneous graph structure in step S2, comprehensively capturing the complex semantic relationships in the vehicle-to-everything (V2X) task offloading environment, and generating a global state representation vector, the method further includes:

[0086] Independent feature input channels are designed for task nodes, vehicle nodes, and RSU nodes, and their original feature vectors are mapped to a representation space of the same dimension through type-aware linear transformation, including:

[0087] Through the projection function f t (·), f v (·),f r (·) Separately set the task nodes Vehicle node With RSU node Convert to initial embedding vector

[0088] For each hop propagation layer, the node embedding of the l-th layer... Updated using weighted information from all types of neighbors, in the following format: in, Represents a set of edge types. This represents the set of neighbors of node i under edge type e. W represents the normalized coefficients calculated based on the attention mechanism. r Let σ be the trainable transformation matrix corresponding to the edge type, and σ(·) be the nonlinear activation function.

[0089] The embedding results of all task nodes are combined into a task embedding matrix. The embeddings of all optional target nodes are combined into a target node embedding matrix. And a global state representation vector is generated using a pooling function.

[0090] It should be understood that, in this invention, the set of tasks to be processed in the Internet of Vehicles is represented as... Each task τ i It possesses specific properties, including the calculated intensity c. i (Unit: Mcycles), Data volume d i (Unit: MB) and maximum tolerable execution delay δ i (Unit: seconds). Meanwhile, the computing nodes in the system consist of a set of vehicle nodes. and edge server node set Composition, each node Includes its available computing resources μ j Current task queue length qj and communication bandwidth b j Attributes. The task will select whether to offload and offload to which target node according to the state characteristics of these nodes.

[0091] In the present application, the tasks and computing nodes in the system are first structured and modeled, providing a foundation for subsequent heterogeneous graph construction and state representation.

[0092] The task set is denoted as Each task τ i has a computing intensity c i (unit: Mcycles), a data volume d i (unit: MB), and a maximum tolerable latency δ i (unit: seconds), whose state characteristics are represented as a three-dimensional vector

[0093] The computing nodes include a set of vehicle nodes and a set of RSU nodes Each node characterizes its current system state, including available computing capacity μ j (unit: GHz), task queue length q j , and communication bandwidth b j (unit: Mbps), which are uniformly represented as a node feature vector

[0094] The feature representations of all task nodes and computing nodes are input to the subsequent graph neural network for embedding processing, which is used to construct the connection relationships of task-task edges, task-node edges, and node-node edges in the multi-semantics heterogeneous graph structure, and serves as the input basis for the state modeling module.

[0095] On the basis of task and node modeling, the present application constructs a unified multi-semantics heterogeneous graph structure for representing the dependency relationship between tasks, the reachable relationship between tasks and computing nodes, and the communication connection state between nodes.

[0096] Among them, the node set includes three types of heterogeneous entities: task nodes, vehicle nodes, and RSU nodes, corresponding to task execution units, mobile computing units, and fixed edge resource units, respectively.

[0097] The edge set ε includes three semantic types:

[0098] (1) Task-task edge ε ττ , used to model the execution order or data dependency relationship between tasks, if task τ i must be executed before τ jAfter execution, a directed edge (τ j ,τ i ) is added in the graph.

[0099] (2) Task-node edge ε τn , used to represent the offloading reachability between tasks and computing nodes, that is, if task τ i can be offloaded to node n j and the node resources meet the basic availability condition, a edge (τ i ,n j ) is added.

[0100] (3) Node-node edge ε nn , used to represent the communication connectivity between computing nodes, especially in V2V and V2I modes, if there is a reachable wireless link between node pair (n i ,n j ), an undirected edge or a directed edge is added to represent the connection relationship.

[0101] For node-node edges, the remaining connection time output by the link prediction module is introduced as a dynamic edge weight, which is used to describe the stability and timeliness of the communication link.

[0102] The entire graph structure is represented using a sparse adjacency matrix, and supports type labeling and edge weight embedding, which facilitates subsequent node semantic representation calculation through heterogeneous graph attention mechanism. The construction of the graph structure provides a unified input view for the graph neural network, and realizes the fusion and propagation of state semantic information between tasks, nodes and links.

[0103] To realize the information representation of the graph structure, the application uses a heterogeneous graph attention mechanism to construct a graph encoding network.

[0104] The network calculates embedding vectors for different types of nodes, such as task node embedding vehicle node embedding RSU node embedding Through a multi-hop attention mechanism, information is aggregated between neighbor nodes, and each node embedding vector represents its local state and context semantics. Finally, all task node embeddings are concatenated into a state matrix Z τ , all offloading target node embeddings are combined into Z n , and the overall system state is represented as after global pooling, which is input to the policy network.

[0105] After completing the construction of the multi-semantics heterogeneous graph structure, the application uses a heterogeneous graph neural network to perform embedding representation calculation on different types of nodes in the graph, so as to extract a unified state vector containing structural semantics, resource state and communication information.

[0106] Specifically, the system designs independent feature input channels for task nodes, vehicle nodes and RSU nodes respectively, and maps their original feature vectors to the same dimension representation space through type-aware linear transformation, i.e. through the projection function f t (·), f v (·), f r (·) respectively converts the task nodes vehicle nodes and RSU nodes into initial embedding vectors

[0107] Subsequently, based on the three types of edges (task-task, task-node, node-node) defined in the heterogeneous graph structure, the application designs a multi-channel heterogeneous graph attention mechanism to calculate the attention weight between node pairs in different edge types and perform normalized aggregation.

[0108] For each hop propagation layer, the node embedding of the l-th layer is updated by the weighted information from all types of neighbors, in the form of: ) where, denotes the set of edge types, denotes the neighbor set of node i under edge type e, is the normalized coefficient calculated based on the attention mechanism, W r is the trainable transformation matrix corresponding to the edge type, and σ(·) is a nonlinear activation function.

[0109] This process can be iterated for multiple layers to achieve cross-type and cross-structure information transmission.

[0110] Finally, the embedding results of all task nodes are combined into a task embedding matrix The embedding of all optional target nodes (vehicles and RSUs) is combined into a target node embedding matrix and a global state representation vector z global is generated through a pooling function (such as average pooling or attention pooling).

[0111] This embedding result serves as the input basis for subsequent high-level offloading mode determination, dynamic threshold generation and low-level target node selection, realizing the end-to-end conduction of heterogeneous graph semantics to the policy network.

[0112] In some specific embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and target node index, including:

[0113] A high-level strategy for each task to be offloaded τi determines its offloading mode, and outputs a result as one of three discrete actions: local execution (0), offloading to a neighboring vehicle node (1), or uploading to an edge server RSU (2);

[0114] receiving a task node embedding vector and a global state embedding As a joint input, it is processed by a set of feedforward neural networks, and finally generates a three-dimensional vector representing the scores of each offloading mode, which is then normalized to a probability distribution by a softmax function.

[0115] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, when outputting the optimal offloading path and target node index, further comprises:

[0116] After the high-level strategy determines that the task needs to be offloaded, a dynamic threshold network is introduced to adaptively calculate a filtering threshold θ∈[0,1] for each task; the threshold network takes the embedding representation of the task and the global state vector of the system as input, and is processed by a set of trainable feedforward neural network structures, and finally outputs a task-specific continuous threshold θ through a sigmoid activation function i so that it falls within the interval [0,1];

[0117] After obtaining the threshold θ i , the semantic similarity scores between the current task τ i and all candidate nodes n j are calculated in turn where f(·) can be a dot product, a bilinear mapping, or other similarity functions.

[0118] In some embodiments, the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, when outputting the optimal offloading path and target node index, further comprises:

[0119] After completing the high-level offloading mode determination and dynamic threshold screening, for each task τ i that is determined to need to be offloaded, its embedding vector and the filtered candidate node embedding set are received as input to construct an action representation space between task-node pairs;

[0120] A multi-layer perception network with an attention mechanism is used to calculate the matching score q j of each candidate target node n ijand normalized into a probability distribution p(a | a, s) by a softmax function i = n j | t i ), from which the offloading target node is selected

[0121] In some embodiments, the step S3 of establishing the hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputs the optimal offloading path and the target node index, further comprises: using the time difference method to minimize the mean square error between the current estimated value and the actual return.

[0122] It should be understood that in the hierarchical reinforcement learning architecture of the present application, the high-level policy module is used to determine the offloading mode of each task to be offloaded t i The output result is one of three types of discrete actions: local execution (0), offloading to a neighboring vehicle node (1), or uploading to an edge server RSU (2).

[0123] The module receives the task node embedding vector and the global state embedding as joint inputs, which are processed by a set of feedforward neural networks, and finally generate a three-dimensional vector representing the scores of each offloading mode, which are then normalized into a probability distribution by a softmax function. The offloading mode is determined by sampling or selecting the action corresponding to the maximum probability in the distribution. The high-level policy network structure can support batch parallel processing of multiple tasks, and the selection result of the offloading mode will be passed to the subsequent candidate node screening and target node selection module as a constraint condition, which is used to limit the type of target node set corresponding to each task (such as local, V2V or V2I). As part of the hierarchical Actor-Critic policy architecture, the high-level policy network is optimized together with the low-level policy network and the state-action value evaluation network during the training process, and is updated end-to-end through the policy gradient method to ensure the consistency of the offloading mode determination and the subsequent scheduling process.

[0124] To further improve the decision-making efficiency of the low-level policy, the present application introduces a specially designed dynamic threshold network after the high-level policy determines that the task needs to be offloaded, which is used to adaptively calculate a screening threshold θ∈[0,1] for each task. The threshold network takes the embedding representation of the task and the system global state vector as inputs, which are processed by a set of trainable feedforward neural network structures (such as MLP), and finally output a task-specific continuous threshold θ i through a sigmoid activation function to ensure that it falls within the interval [0,1].

[0125] The threshold θ iAfterwards, the system calculates the semantic similarity score between the current task τ i and all candidate nodes n j . where f(·) can be dot product, bilinear mapping or other similarity function. When s ij < θ, it means that the target node is less suitable for the task, and the system will automatically exclude this node n j from the subsequent low-level target node selection process.

[0126] This mechanism builds an adaptive "filter gate" between the high-level and low-level strategies, which can dynamically control the size of the action space according to the personalized semantic features of the task, effectively reducing the interference of low-value candidate nodes, and thus reducing invalid exploration and redundant computation in the strategy training process. The threshold network is trained jointly with the overall strategy network, and in the continuous iteration process, it can adaptively learn the reasonable screening criteria for different tasks in different environmental states, providing strong support for the efficient convergence of hierarchical reinforcement learning strategies.

[0127] After completing the high-level unloading mode determination and dynamic threshold screening, the target node selection is performed by the low-level strategy module. This module receives the embedding vector of each task τ i that is determined to be unloaded and the screened candidate node embedding set as input, and constructs the action representation space between the task-node pairs. Since the action space has been significantly compressed in the previous two stages, the low-level strategy network can achieve more fine-grained target node evaluation within a limited range.

[0128] Specifically, the low-level strategy uses a multi-layer perception network (MLP) with an attention mechanism to calculate the matching score q j of each candidate target node n ij with the current task, and normalize the score to a probability distribution π(a i = n j | τ i ) through the softmax function, which is used to select the unloading target node This strategy can either perform actions through probability sampling or select the maximum probability node in the reasoning phase. This network structure supports batch processing of task-node pairs, and the output dimension is adaptive to the size of the action space of the current task.

[0129] The low-layer policy network is an under-layer actor in the actor-critic architecture, and is trained together with a high-layer policy and value evaluation network (critic), and adopts a policy gradient method to optimize, so as to minimize the expected delay and resource overhead in a task execution cost function. Since the action space has been compressed by the high-layer structure and the dynamic threshold mechanism, the stability and efficiency of the low-layer policy training are significantly improved, and the task-node matching decision can be more accurately completed, and the actual vehicle networking offloading environment with concurrent tasks and heterogeneous nodes can be adapted.

[0130] To support the policy learning process, the critic network is designed to evaluate the value function Q(st,at) of the state-action pair. The critic network accepts the state embedding and the offloading decision as input, and outputs the long-term expected return of the current action in the current state, which is an important basis for the optimization direction of the policy. Through joint actor-critic structure training, stable updating of the policy network is ensured and convergence is accelerated.

[0131] In the hierarchical reinforcement learning policy architecture proposed in the application, the state-action value evaluation network (critic network) is used to evaluate the value of the state-action pair (st,at) under the current policy, and to provide quantitative feedback for the updating direction of the policy network. This module predicts the long-term expected return that can be obtained after taking a certain action under a specific state under the current policy, thereby guiding the joint optimization of high-layer offloading mode selection and low-layer target node decision.

[0132] The input of the network consists of two parts: state representation and action representation. The state representation is composed of the global state embedding vector The embedding of the current task and the target node embedding are spliced or fused to reflect the resource state, candidate space and communication reachability of the task in the current environment; the action representation is the result of the current offloading decision, including the offloading mode m i and the index or embedding vector of the selected target node .

[0133] The critic network usually adopts a feedforward neural network structure, which accepts the above state-action joint representation and outputs a scalar Q(s t ,a t ), representing the estimated value of the state-action pair. This value measures the cumulative return of the current decision in the subsequent multiple time steps, which is an important basis for the gradient update of the actor policy network. The critic network is trained using the time difference method (such as TD error), and the goal is to minimize the mean square error between the current estimated value and the actual return, that is:

[0134] Where yt = r t + γQ(s t+1 , a t+1 )

[0135] During the training process, the Critic network is iteratively optimized together with the high-level Actor and the low-level Actor, and uses an experience replay mechanism to sample task offloading trajectories to enhance training stability. By introducing this module, the system can provide targeted feedback during policy generation, promoting rapid convergence of the policy and avoiding the policy from falling into a local optimum, thereby improving overall offloading success rate and resource utilization efficiency.

[0136] The present application constructs a task-node graph and a task-task graph to represent the offloadable target relationship of the task and the dependency order between tasks, respectively, extracts multi-dimensional heterogeneous embedding features through a graph neural network, and uses the features to construct a structured state representation, thereby improving the global information perception ability of the offloading policy.

[0137] The high-low layer structure is used to decouple the policy generation process. First, the high-level policy is used to filter the offloading candidate set. Then, based on the high-level output result and the task embedding, a dynamic threshold mechanism is introduced as an intermediate filtering strategy. The filtering threshold is dynamically generated according to the semantic similarity between the task and the candidate node, and is used to eliminate nodes with weak relevance, thereby further reducing the action space range of the low-level policy. Finally, the low-level policy is used to finely select the candidate target, effectively compressing the action space, improving the policy training efficiency and stability, and adapting to large-scale task-node matching scenarios.

[0138] A prediction module is introduced to estimate the communication connection duration between the vehicle-mounted nodes in real time, and is used as a condition for judging the availability of the offloading path, thereby assisting the offloading policy to select a more stable path and reducing the risk of offloading failure caused by link disconnection.

[0139] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0140] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0141] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0142] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0143] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning for Internet of Vehicles, characterized in that, The method comprises the following steps: S1, establishing a vehicle state estimation model, calculating the future position and speed state of each vehicle, and further calculating the remaining connection time between vehicles; S2, introducing the remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationship in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector; S3, establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputting the optimal offloading path and target node index.

2. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning for Internet of Vehicles according to claim 1, characterized in that, In the step S1 of establishing a vehicle state estimation model, calculating the future position and speed state of each vehicle, and further calculating the remaining connection time between vehicles, the following steps are included: The vehicle state estimation model is established based on the Kalman filter method and is oriented towards each vehicle v j Establish state vector s j (t)=[x j (t),y j (t),v x,j (t),v y,j (t)] Τ , where x j ,y j The two-dimensional position coordinates of the vehicle, v x,j ,v y,j This represents its velocity components in each direction; The vehicle updates the state estimation according to the previous state and observation input at each time step, including the estimated mean and covariance matrix of the position and speed; According to the Kalman prediction update process, obtain the optimal state estimation value of all vehicles at the current time t for extrapolating the future relative trajectory and position change between any pair of vehicles (v u ,v v ) Based on the predicted position and velocity, the remaining sustainable communication time between any pair of vehicles is calculated This value is used as the dynamic edge weight for the node-node edges.

3. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning for Internet of Vehicles according to claim 2, characterized in that, calculating a remaining sustainable communication time between any pair of vehicles based on the predicted positions and velocities when, including: Any pair of communication nodes is (i, j); Relative speed relative distance The remaining connection time is where R max denotes the maximum communication coverage radius.

4. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 3, characterized in that, In the step S2 of introducing the remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationship in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, the following steps are included: A set of tasks is denoted as Each task τ i with a computation intensity c i , a data volume d i and a maximum tolerable latency δ i Its state characteristics are represented as a three-dimensional vector The computing nodes include a set of vehicle nodes V = {v1, v2, …, vN} and a set of RSU nodes V} and RSU nodes Each node characterizes its current system state, including available computing capacity μ j , task queue length q j and communication bandwidth b j , which are collectively represented as a node feature vector x nj = [μ j , q j , b j ].

5. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 4, characterized in that, In the step S2 of introducing the remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationship in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, the following steps are also included: Building a unified multi-semantic heterogeneous graph structure to represent dependencies between tasks, reachability between tasks and computing nodes, and communication connection status between nodes; wherein the set of nodes The heterogeneous entities include task nodes, vehicle nodes, and RSU nodes, which correspond to task execution units, mobile computing units, and fixed edge resource units, respectively. The edge set ε includes: task-task edges ε ττ for modeling execution order or data dependency relationships between tasks; Task-Node edge ε τn to represent offload reachability between a task and a compute node; node-node edge ε nn to represent the communication connectivity between computing nodes.

6. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 5, characterized in that, In the step S2 of introducing the remaining connection time between vehicles as a dynamic edge weight to construct a multi-semantics heterogeneous graph structure, comprehensively capturing the complex semantic relationship in the task offloading environment of the Internet of Vehicles, and generating a global state representation vector, the following steps are also included: Independent feature input channels are designed for task nodes, vehicle nodes and RSU nodes respectively, and the original feature vectors are mapped to the same dimension representation space through type-aware linear transformation, including: By projecting functions f t (·), f v (·), f r (·) respectively, the task nodes vehicle nodes and RSU nodes into initial embedding vectors For each propagation layer, the node embedding of the l-th layer is updated by the weighted information from all types of neighbors, in the form of: where, denotes the set of edge types, denotes the set of neighbors of node i under edge type e, is the normalization coefficient calculated based on the attention mechanism, W r is the trainable transformation matrix corresponding to the edge type, and σ(·) is a nonlinear activation function. The embedding results of all task nodes are combined into a task embedding matrix The embeddings of all optional target nodes are combined into a target node embedding matrix And generate a global state representation vector z through a pooling function global .

7. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 6, characterized in that, In the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputting the optimal offloading path and target node index, the following steps are included: High-level strategy, for each task τ to be offloaded i Decide its offloading mode, output is one of three discrete actions: local execution (0), offloading to a neighboring vehicle node (1), or uploading to an edge server RSU (2); Receiving task node embedding vector Global state embedding z global As joint input, it is processed by a set of feedforward neural networks, which finally generate a three-dimensional vector representing the score of each offloading mode, which is then normalized to a probability distribution by a softmax function.

8. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 7, characterized in that, In the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputting the optimal offloading path and target node index, the following steps are also included: After the high-level policy decision task is offloaded, a dynamic threshold network is introduced to adaptively calculate a screening threshold θ ∈ [0, 1] for each task; the threshold network takes the embedding representation of the task as input and the system global state vector z global as input, and processes them through a set of trainable feedforward neural network structures, and finally outputs a task-specific continuous threshold θ through a sigmoid activation function i so that it falls within the interval [0, 1]; Obtaining a threshold value θ i Subsequently, the semantic similarity score between the current task τ i and all candidate nodes n j is calculated in turn where f(·) can be a dot product, bilinear mapping, or other similarity function.

9. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 8, characterized in that, In the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputting the optimal offloading path and target node index, the following steps are also included: After the high-level offloading mode determination and dynamic threshold screening are completed, for each task τ i , receive its embedding vector and the screened candidate node embedding set as input, construct the action representation space between task-node pairs; A multi-layer perceptron with an attention mechanism is used to score each candidate target node n j A match score q is computed for the current task ij and normalized by a softmax function into a probability distribution π(a i = n j |τ i for selecting the offload target node from 10. The task offloading optimization method based on semantic heterogeneous graph and hierarchical reinforcement learning according to claim 9, characterized in that, In the step S3 of establishing a hierarchical reinforcement learning strategy, the global state representation vector as the input basis of the hierarchical reinforcement learning strategy, outputting the optimal offloading path and target node index, the following steps are also included: adopting the time difference method to minimize the mean square error between the current estimated value and the actual return.

Images