A vehicle-mounted edge computing task offloading method based on deep reinforcement learning

Abstract

This invention relates to a method and system for offloading tasks in vehicle-mounted edge computing based on deep reinforcement learning, comprising: aggregating multiple heterogeneous edge devices into an edge computing cluster through node aggregation; modeling the task offloading problem as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency, setting actions, constraints, and states; generating the offloading ratio of tasks between the local machine and each cluster based on the current state at each action, and constructing an immediate reward based on the energy consumption and latency after each task offloading, and iteratively updating the strategy to guide the action based on the feedback of the immediate reward; obtaining the offloading ratio of each task between the local machine and each cluster; performing fine-grained task allocation within each cluster based on a greedy strategy; and performing dynamic two-stage task offloading based on the cluster in a multi-vehicle scenario to reduce latency and energy consumption, and improve the processing efficiency and system stability of vehicle-mounted edge computing tasks.

Description

Technical Field

[0001] This invention relates to the fields of vehicle-mounted edge computing and intelligent scheduling of vehicle networks, and in particular to a method for offloading vehicle-mounted edge computing tasks based on deep reinforcement learning. Background Technology

[0002] With the development of 5G / 6G wireless communication and artificial intelligence technologies, vehicular edge computing (VEC) is widely used in the Internet of Vehicles (IoV) to support highly latency-sensitive in-vehicle intelligent services such as augmented reality and autonomous driving. By deploying heterogeneous edge devices such as roadside units (RSUs), base stations, and smart traffic lights near vehicles, vehicles can offload some computationally intensive tasks to edge devices for processing, thereby reducing local energy consumption and task processing latency.

[0003] However, edge devices in VEC exhibit heterogeneity, with significant differences in computing power, service capabilities, and resource allocation strategies among different devices. Furthermore, vehicle location, network environment, and task arrival patterns change rapidly over time, leading to the following challenges during task unloading:

[0004] (1) Latency and energy consumption are difficult to optimize simultaneously: Although offloading tasks to edge devices can reduce local computing energy consumption, it may increase transmission, queuing and processing latency. In latency-sensitive scenarios, excessive latency may lead to a decline in service quality or even traffic safety risks.

[0005] (2) The service capabilities of heterogeneous edge devices vary greatly: some RSUs and small devices can only provide limited services. The offloading strategy under the traditional homogeneous assumption cannot accurately adapt to diverse service requests, resulting in tasks being assigned to unsuitable nodes and reducing offloading efficiency.

[0006] (3) High computational overhead for global optimal unloading: Due to the heterogeneity of task types and edge devices, obtaining the optimal solution for task unloading in large-scale scenarios usually requires unacceptable algorithm execution time. Designing an algorithm that can achieve a near-optimal solution while reducing time complexity is a necessary consideration.

[0007] To address these issues, some studies have attempted to use greedy algorithms, heuristic strategies, or iterative optimization for task offloading. However, these approaches still suffer from slow convergence, difficulty adapting to dynamic changes, and the inability to simultaneously optimize energy consumption and latency. Furthermore, while some reinforcement learning-based methods have explored task offloading, they still fall short in supporting dynamic adjustment of task offloading ratios and efficient resource scheduling in heterogeneous multi-node clusters. Summary of the Invention

[0008] This invention addresses the technical problems existing in the prior art by providing a deep reinforcement learning-based method for offloading vehicular edge computing tasks. In dynamic vehicular network environments, it can balance the mobility of vehicles and the needs of latency-sensitive tasks, efficiently adapt to the computing and service capabilities of heterogeneous edge devices, and through dynamic strategies, while ensuring service quality and system energy efficiency, it can achieve joint optimization of latency and energy consumption for task offloading in heterogeneous edge device environments in vehicular network scenarios with dynamic network changes, thereby improving the stability, efficiency and scalability of the offloading process.

[0009] According to a first aspect of the present invention, a method for offloading vehicle-mounted edge computing tasks based on deep reinforcement learning is provided, comprising: step 1, aggregating multiple heterogeneous edge devices through nodes to form an edge computing cluster;

[0010] Step 2: Model the task unloading problem as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. Set the action to output the unloading ratio between the local machine and each cluster for each task. Set the constraint that the latency is less than the maximum tolerable latency. Set the state to include: the current task unloading ratio, the service set of each cluster and the attributes of the task to be unloaded.

[0011] Step 3: At each action, generate the unloading ratio of the task between the local machine and each cluster based on the current state, and construct an instant reward based on the energy consumption and latency after each task unloading. Iterate and update the strategy to guide the action based on the instant reward feedback; thus obtaining the unloading ratio of each task between the local machine and each cluster.

[0012] Step 4: Within each cluster, perform fine-grained task allocation based on a greedy strategy.

[0013] Based on the above technical solution, the present invention can also be improved as follows.

[0014] Optionally, step 1 includes:

[0015] Step 101: Count the set of services that are not yet covered by the current cluster to be aggregated, and select the service with the fewest covered services among the remaining nodes. As a service to be supplemented;

[0016] Step 102: If the current cluster to be generated is empty, randomly select one from the remaining nodes that contains... nodes Join the cluster and set the node Remove from candidate nodes;

[0017] If the current cluster to be generated is not empty, calculate the weighted cluster center coordinates of the current cluster, and select the node that is closest to the cluster center and contains the cluster center from the remaining nodes. nodes Join the cluster and set the node Remove from candidate nodes;

[0018] Step 103: Continue executing step 102. When the current cluster has full service capabilities, generate the next cluster; when the remaining number of services is insufficient to generate a new complete service cluster, select nodes sequentially from the remaining nodes. Calculate the node Distance to the cluster centers of all generated clusters, for each node Assign to the nearest cluster As a supplementary computing resource.

[0019] Optionally, the formula for calculating the weighted cluster center coordinates of the current cluster in step 102 is:

[0020]

[0021] in, Indicates the first A set of edge nodes in a cluster to be generated. Represents a node The number of cores; , for Mid-edge device nodes The horizontal and vertical coordinates;

[0022] In step 103, the computing node The formula for the distance to the cluster centers of all generated clusters is:

[0023]

[0024] in, , For nodes The coordinates.

[0025] Optionally, the state set in step 2 is represented as follows:

[0026] ;

[0027] in, This indicates the task unloading status as of the current moment, where Indicates the number of computing clusters. and Representing tasks Unload to local and cluster The uninstallation rate;

[0028] This represents a list of service sets contained in each cluster, where Represents a cluster The set of services included;

[0029] Indicates the first A list of attributes for each task;

[0030] The action is represented as:

[0031]

[0032] in, Indicates task The action, Indicates task The proportion calculated locally. Indicates task Unload to cluster The proportion.

[0033] Optionally, the formula for the instant reward constructed in step 3 is:

[0034]

[0035] in, This indicates that the vehicle has completed its mission. Total energy consumption Indicates task The completion delay This is the time delay weighting coefficient. , As a reward adjustment factor, Indicates the task's sequence number. This indicates the total number of tasks in a round.

[0036] Optionally, the objective function for iteratively updating the strategy guiding the action in step 3 is:

[0037]

[0038] in,

[0039]

[0040]

[0041] in, and They represent the old and new strategies, respectively. Represents the state value function, Indicates cumulative return. Indicates task The action, Indicates task state, This represents the expectation of the trajectory distribution at time step t. A hyperparameter representing the range to be truncated.

[0042] Optionally, step 4 includes:

[0043] Step 401: Select a set of child nodes within the cluster that have the services required to process the current task; construct a calculation function for the optimization target value based on the latency and energy consumption cost when the task is unloaded to any child node; and prioritize the candidate child nodes based on the optimization target value.

[0044] Step 402: Remove child nodes whose optimization target value is higher than the average level;

[0045] Step 403: Normalize the optimization target values ​​of the remaining child nodes to obtain weights, and determine the unloading ratio of the task among the child nodes based on the weights.

[0046] Optionally, the function for calculating the optimization target value is:

[0047]

[0048] in, Indicates task Unload to child nodes within the cluster Total time delay This represents the corresponding total energy consumption. , These are the time delay and energy consumption adjustment weighting coefficients, respectively.

[0049] Optionally, the formula for calculating the weight in step 403 is:

[0050]

[0051] in, Indicates task Unload to a single child node Optimize target value, This represents the average of the optimization objective values ​​for all remaining child nodes;

[0052] The formula for calculating the unloading ratio of tasks across child nodes is as follows:

[0053]

[0054] in, Indicates task It has been assigned to the cluster at the cluster level. The uninstallation rate This indicates that the task is in the cluster. Internal unloading to child nodes The proportion.

[0055] According to a second aspect of the present invention, a vehicle-mounted edge computing task offloading system based on deep reinforcement learning is provided, comprising: a vehicle layer, an edge computing cluster layer, and a cloud service layer;

[0056] The vehicle layer generates time-sensitive computing tasks during the driving process;

[0057] The edge computing cluster layer consists of multiple heterogeneous edge devices aggregated through nodes to form various edge computing clusters.

[0058] The cloud service layer models the task unloading problem generated by the vehicle layer as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. The action is set to output the unloading ratio between the local machine and each cluster for each task, with the constraint that latency is less than the maximum tolerable latency. The state includes: the current task unloading ratio, the service set of each cluster, and the attributes of the task to be unloaded. At each action, the unloading ratio between the local machine and each cluster is generated based on the current state, and an instant reward is constructed based on the energy consumption and latency after each task unloading. The iterative update of the action strategy is guided by the feedback of the instant reward. This yields the unloading ratio of each task between the local machine and each cluster.

[0059] After obtaining the offloading ratio of each task between the local machine and each cluster, each cluster in the edge computing cluster layer performs fine-grained task allocation based on a greedy strategy.

[0060] This invention provides a method and system for offloading tasks in vehicle-mounted edge computing based on deep reinforcement learning. The overall process follows the approach of "structure optimization—policy training—fine-grained scheduling," gradually realizing abstract cluster management of heterogeneous edge resources, reinforcement learning optimization of cross-cluster offloading strategies, and energy-efficiency-aware task scheduling among child nodes within the cluster. The HNA algorithm is used to reconstruct and aggregate the service capabilities of heterogeneous edge nodes distributed in the system before task offloading, significantly reducing the complexity of heterogeneous resource management and providing a unified cluster abstraction for subsequent offloading strategies. The core policy optimization module, based on the CBPPO algorithm, optimizes the task offloading ratio between clusters. Implemented based on the Proximal Policy Optimization (PPO) framework, it can achieve efficient convergence and dynamic adaptive adjustment of offloading strategies under varying network conditions. The GICOA algorithm is used to finely allocate offloading tasks within the cluster, ensuring maximum resource utilization and balanced node load. A simple and efficient greedy strategy is used to jointly optimize latency and energy consumption within the cluster, effectively reducing system processing bottlenecks. Attached Figure Description

[0061] Figure 1 A flowchart of a method for offloading vehicle edge computing tasks based on deep reinforcement learning provided by the present invention;

[0062] Figure 2 A flowchart illustrating an embodiment of a deep reinforcement learning-based method for offloading vehicle-mounted edge computing tasks provided by the present invention;

[0063] Figure 3 This invention provides a structural block diagram of an in-vehicle edge computing task offloading method based on deep reinforcement learning. Detailed Implementation

[0064] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0065] In vehicular edge computing (VEC) networks built around cities, vehicles, as the primary task generators, continuously produce a large number of latency-sensitive and computationally intensive tasks. These tasks typically need to be processed under strict latency constraints. However, due to the limited local computing power of vehicles, executing all tasks locally would lead to high energy consumption and processing latency, impacting the real-time computing efficiency of vehicles while in motion. Offloading some computing tasks to nearby edge devices can effectively reduce local energy consumption, but it also introduces additional communication latency. Therefore, achieving a balance between energy consumption and latency by rationally allocating the offloading ratio between local and edge devices while meeting task latency constraints has become a key problem that needs to be solved.

[0066] Figure 1 This invention provides a flowchart of a task offloading method for vehicular edge computing based on deep reinforcement learning. In a vehicular edge computing environment with clustered heterogeneous edge node collaboration based on deep reinforcement learning, it describes a two-stage dynamic task offloading approach for latency-sensitive tasks. The vehicular edge computing environment refers to a network that uses C-V2X protocol wireless communication to access heterogeneous edge computing nodes to achieve computation offloading and data sharing. Figure 1 As shown, the task unloading method includes:

[0067] Step 1: Aggregate multiple heterogeneous edge devices into an edge computing cluster through node aggregation.

[0068] Step 2: Model the task unloading problem as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. Set the action to output the unloading ratio between the local machine and each cluster for each task, set the constraint that the latency is less than the maximum tolerable latency, and set the state to include: the current task unloading ratio, the service set of each cluster, and the attributes of the task to be unloaded.

[0069] After aggregating edge nodes to form a fully capable edge computing cluster, a task offloading strategy based on deep reinforcement learning is proposed to optimize energy consumption and latency under latency constraints. This strategy utilizes model-free reinforcement learning, enabling the agent to continuously adjust the offloading strategy through ongoing interaction with the environment and guided by reward feedback. This achieves dynamic adaptive adjustment of the offloading ratio, resulting in an efficient and stable task offloading strategy between clusters. The task offloading problem is modeled as a Markov decision process. At each time step, the agent observes the current state and generates the offloading ratio of tasks between the local machine and each edge computing cluster as its action. It receives immediate reward feedback from the environment to guide the iterative update of the strategy, minimizing the weighted sum of total system energy consumption and latency, while ensuring that the task completion latency is less than the maximum tolerable latency constraint.

[0070] Step 3: At each action, generate the unloading ratio of the task between the local machine and each cluster based on the current state, and construct an instant reward based on the energy consumption and latency after each task unloading. Iterate and update the strategy based on the feedback of the instant reward to guide the action; thus obtaining the unloading ratio of each task between the local machine and each cluster.

[0071] Step 4: Perform fine-grained task allocation within each cluster based on a greedy strategy.

[0072] This invention provides a method for offloading vehicle-mounted edge computing tasks based on deep reinforcement learning. In multi-vehicle scenarios, it performs dynamic two-stage task offloading based on a cluster, reducing latency and energy consumption, and improving the processing efficiency and system stability of vehicle-mounted edge computing tasks.

[0073] Example 1

[0074] Embodiment 1 provided by this invention is an embodiment of a method for offloading vehicle-mounted edge computing tasks based on deep reinforcement learning, as provided by this invention. Figure 2 The diagram shown is a flowchart of an embodiment of a method provided by the present invention, in conjunction with... Figure 1 and Figure 2 It can be seen that embodiments of this task unloading method include:

[0075] Step 1: Aggregate multiple heterogeneous edge devices into an edge computing cluster through node aggregation.

[0076] In heterogeneous edge node scenarios, to avoid the unacceptable computational overhead of searching for the globally optimal offloading solution for each task, and the convergence difficulties caused by the differences in service deployment in reinforcement learning output dimension, a node clustering algorithm is proposed. This algorithm aggregates edge devices with limited service capabilities into edge computing clusters with complete service capabilities, providing support for simplifying the design of subsequent deep reinforcement learning offloading strategies and improving scheduling efficiency.

[0077] In specific implementation, this embodiment of the invention provides a vehicle-mounted edge computing task offloading system model suitable for heterogeneous edge device environments. A set of vehicles is defined as the task generator, represented as... There are multiple edge devices in the network that deploy different service sets. The set of edge devices is defined as... The system is defined as providing the following set of services: ,vehicle The local computing power (CPU frequency) is denoted as Edge devices Described as ,in, Indicates the type of edge device (e.g., RSU, RPU, smart traffic lights, etc.). Indicates the single-core computing frequency. Represents the number of cores. This represents a list of the number of service instances that have been deployed. For example, edge devices. The number of cores is 8, and the service list is as follows. Indicates that the service has been provided. Deploy 2 instances, service , Deploy 3 instances respectively.

[0078] In one possible embodiment, step 1 includes:

[0079] Step 101: In the early stage of aggregation, when there are enough unassigned service types, count the set of services that are not yet covered by the current cluster to be aggregated, and select the service with the fewest coverages among the remaining nodes. As a service to be supplemented.

[0080] Step 102: If the current cluster to be generated is empty, randomly select one from the remaining nodes that contains... nodes Join the cluster and set the node Remove from candidate nodes.

[0081] If the current cluster to be generated is not empty, calculate the weighted cluster center coordinates of the current cluster, and select the node that is closest to the cluster center and contains the cluster center from the remaining nodes. nodes Join the cluster and set the node Remove from candidate nodes.

[0082] The formula for calculating the weighted cluster center coordinates of the current cluster is:

[0083]

[0084] in, Indicates the first A set of edge nodes in a cluster to be generated. Represents a node The number of cores; , for Mid-edge device nodes The horizontal and vertical coordinates.

[0085] Step 103: Continue executing step 102. When the current cluster has complete service capabilities, a strong computing node cluster is generated, and the next cluster is generated. In the later stages of cluster aggregation, when the remaining number of services is insufficient to generate a new complete service cluster, the remaining nodes are selected sequentially. Calculate the node Distance to the cluster centers of all generated clusters, for each node Assign to the nearest cluster As a supplementary computing resource.

[0086] compute nodes The formula for the distance to the cluster centers of all generated clusters is:

[0087]

[0088] in, , For nodes The coordinates.

[0089] Through a node aggregation mechanism, edge devices with complementary service capabilities and heterogeneous computing resources are combined into an edge computing cluster with complete service capabilities. The aggregation process fully considers the service types, geographical locations, and core resources deployed on the devices, and employs heuristic algorithms to achieve efficient cluster mapping of heterogeneous nodes, providing a cluster-level scheduling foundation for subsequent offloading.

[0090] Step 2: Model the task unloading problem as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. Set the action to output the unloading ratio between the local machine and each cluster for each task, set the constraint that the latency is less than the maximum tolerable latency, and set the state to include: the current task unloading ratio, the service set of each cluster, and the attributes of the task to be unloaded.

[0091] In one possible embodiment, the state set in step 2 is represented as follows:

[0092] .

[0093] in, This indicates the task unloading status as of the current moment, where Indicates the number of computing clusters. and Representing tasks Unload to local and cluster The uninstallation ratio; from arrive The unloading ratio between these two values ​​is valid, while the rest are zero.

[0094] This represents a list of service sets contained in each cluster, where Represents a cluster The set of services it includes.

[0095] Indicates the first The attribute list of each task describes the information of the tasks to be uninstalled in the current state.

[0096] The agent's action is to offload tasks to the cluster network while satisfying model constraints. Specifically, in the current state... Next, the intelligent agent will... Each task is offloaded to a set of cluster nodes, determining the offloading ratio between local processing and cluster processing. Due to the heterogeneity of edge devices within different clusters, different clusters exhibit different service performance when processing different tasks. Therefore, it is necessary to determine an appropriate offloading ratio for each task to minimize task latency and energy consumption.

[0097] One direct approach is to allow the agent to directly output the unloading ratio of all tasks in a single action; however, in real-world network environments, the sheer number of tasks and edge devices leads to an excessively large action space, making model convergence difficult. To address this issue, an action structure that outputs the unloading ratio for each task is designed, and a reward shaping mechanism is introduced to provide small positive rewards in each effective step, thereby effectively accelerating the model's convergence process.

[0098] The action is represented as:

[0099]

[0100] in, Indicates task The action, Indicates task The proportion calculated locally. Indicates task Unload to cluster The proportion.

[0101] For reinforcement learning models, the reward function determines whether an effective task offloading strategy can be learned during training. As mentioned in the action section, a step-by-step offloading strategy is adopted to address the action space explosion problem. Since reinforcement learning is based on Markov chains, the agent needs to receive a reward at each action step, which contradicts the task offloading problem. Evaluating the decision quality after all tasks are offloaded makes it difficult to effectively evaluate the decision at each step.

[0102] To address this issue, a reward shaping mechanism is introduced, setting intermediate rewards during a complete unloading cycle. Specifically, after each task unloading, the current unloading state is considered an intermediate solution, and an intermediate reward is calculated using a reward function. For example, during the unloading of the [number]th [task]... After each task, for the previous The unloading quality of each task is evaluated, and appropriate positive rewards are given to promote the progress of the entire unloading process. After all tasks are unloaded, a larger reward or penalty is given based on the overall unloading quality, thereby guiding the model to learn a good global unloading strategy.

[0103] By utilizing a cluster-based near-end policy optimization algorithm, a reinforcement learning agent is deployed at the service layer. Based on vehicle location information, task attributes, and the status of each edge cluster, the offloading ratio of tasks between the local area and different clusters is dynamically generated. This ensures that the offloading strategy adapts to changes in the dynamic environment and achieves joint optimization of energy consumption and latency while meeting latency constraints.

[0104] Step 3: At each action, generate the unloading ratio of the task between the local machine and each cluster based on the current state, and construct an instant reward based on the energy consumption and latency after each task unloading. Iterate and update the strategy based on the feedback of the instant reward to guide the action; thus obtaining the unloading ratio of each task between the local machine and each cluster.

[0105] In one possible embodiment, the formula for the instant reward constructed in step 3 is:

[0106]

[0107] in, This indicates that the vehicle has completed its mission. Total energy consumption Indicates task The completion delay This is the time delay weighting coefficient. , As a reward adjustment factor, Indicates the task's sequence number. This indicates the total number of tasks in a round.

[0108] In one possible embodiment, the objective function for iteratively updating the strategy guiding the actions in step 3 is:

[0109]

[0110] in,

[0111]

[0112]

[0113] in, and They represent the old and new strategies, respectively. Represents the state value function, Indicates cumulative return. Indicates task The action, Indicates task state, This represents the expectation of the trajectory distribution at time step t. It is a hyperparameter that indicates the range to be truncated.

[0114] The Proximal Policy Optimization (PPO) algorithm is used for learning. The network structure of the new and old policies is used to improve sample utilization and ensure the stability of policy updates. During the update process, Generalized Advantage Estimation (GAE) is used to smooth the calculation of advantage values.

[0115] Through the above process, the trained strategy can dynamically adjust the task offloading scheme based on real-time observation information without relying on precise environmental modeling, making the offloading process portable and generalizable. At the same time, it can minimize the total system energy consumption while satisfying latency constraints, thereby achieving efficient task offloading optimization in dynamic vehicle edge computing environments.

[0116] In step 3, the offloading ratio of computing tasks among the edge computing clusters was determined, achieving preliminary task allocation at the global level. Building on this, the task offloading problem is further simplified to determining the specific task allocation ratio among the edge child nodes within each cluster, in order to further reduce offloading latency and optimize energy consumption allocation structure. Given that the edge node structure has already been optimized during the node aggregation phase, and the number of nodes within each cluster is limited, this sub-problem is relatively small. Based on this characteristic, a small-scale task offloading method based on a greedy strategy is proposed to achieve fine-grained task allocation within the cluster, balancing allocation efficiency and node resource load balancing.

[0117] Step 4: Perform fine-grained task allocation within each cluster based on a greedy strategy.

[0118] In one possible embodiment, step 4 includes:

[0119] Step 401: Select a set of child nodes within the cluster that have the services required to process the current task; construct a calculation function for the optimization target value based on the latency and energy consumption cost when the task is unloaded to any child node; and prioritize the candidate child nodes based on the optimization target value.

[0120] In one possible implementation, the function for calculating the target value is:

[0121]

[0122] in, Indicates task Unload to child nodes within the cluster Total time delay This represents the corresponding total energy consumption. , These are the time delay and energy consumption adjustment weighting coefficients, respectively.

[0123] Step 402: Remove child nodes whose optimization target value is higher than the average level.

[0124] To avoid assigning tasks to inefficient child nodes that are physically far away or heavily loaded, a child node filtering mechanism is designed. By removing child nodes whose optimization target value is higher than the average level, the number of child nodes that actually participate in task unloading is limited, thereby improving unloading efficiency and unloading latency.

[0125] In practice, let the set of optimization objective values ​​for the selected child nodes be . Calculate the set mean If child node of If the child node is removed, it will not participate in the unloading allocation.

[0126] Step 403: Normalize the optimization target values ​​of the remaining child nodes to obtain weights, and determine the unloading ratio of the task among the child nodes based on the weights.

[0127] In one possible embodiment, the formula for calculating the weight in step 403 is:

[0128]

[0129] in, Indicates task Unload to a single child node Optimize target value, This represents the average of the optimization objective values ​​for all remaining child nodes.

[0130] The formula for calculating the unloading ratio of tasks across child nodes is as follows:

[0131]

[0132] in, Indicates task It has been assigned to the cluster at the cluster level. The uninstallation rate This indicates that the task is in the cluster. Internal unloading to child nodes The proportion.

[0133] Within each edge cluster, a fine-grained task allocation is further performed based on a greedy strategy. The task is finely divided according to the computing power, service coverage, and physical distance of the child nodes, so as to achieve node load balancing and energy consumption minimization within the cluster, thereby improving the overall resource utilization and response efficiency of the system.

[0134] By employing the greedy unloading method described above, the unloading task achieves load balancing within the cluster, fully utilizing the computing and service resources of each node. This enables joint optimization of latency and energy consumption during the unloading process, effectively reducing queuing latency and improving system task processing efficiency. It provides efficient and stable fine-grained unloading support for latency-sensitive tasks in vehicle edge computing environments.

[0135] Example 2

[0136] Embodiment 2 provided by this invention is an embodiment of an in-vehicle edge computing task offloading system based on deep reinforcement learning. Figure 3 This is a structural diagram of an embodiment of an in-vehicle edge computing task offloading system based on deep reinforcement learning provided by the present invention, combined with... Figure 3 It is known that the implementation of this task offloading system includes: a vehicle layer, an edge computing cluster layer, and a cloud service layer.

[0137] In practice, a layered architecture is adopted, dividing the vehicular edge computing network into a cloud service layer based on CPU cores, an edge computing cluster layer formed by node aggregation, and a vehicle layer as the task generator. The vehicle layer generates latency-sensitive computing tasks during operation. The vehicle uploads these tasks to the edge computing cluster for execution via vehicle-to-edge communication, reducing local power consumption and processing latency. The edge computing cluster refers to a computing cluster with complete service capabilities, formed by the aggregation of multiple heterogeneous edge devices through nodes. The cloud service layer is responsible for fine-grained task processing when tasks arrive.

[0138] The vehicle layer generates time-sensitive computing tasks during the driving process.

[0139] The edge computing cluster layer consists of multiple heterogeneous edge devices aggregated into various edge computing clusters.

[0140] The cloud service layer models the task offloading problem generated by the vehicle layer as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. The action is set to output the offloading ratio between the local machine and each cluster for each task, with the constraint that latency is less than the maximum tolerable latency. The state includes: the current task offloading ratio, the service set of each cluster, and the attributes of the task to be offloaded. At each action, the offloading ratio between the local machine and each cluster is generated based on the current state, and an immediate reward is constructed based on the energy consumption and latency after each task offloading. The iterative update of the action strategy is guided by the feedback of the immediate reward. This yields the offloading ratio of each task between the local machine and each cluster.

[0141] After obtaining the offloading ratio of each task between the local machine and each cluster, each cluster in the edge computing cluster layer performs fine-grained task allocation based on a greedy strategy.

[0142] The cloud service layer utilizes a cluster-based near-end policy optimization algorithm. Through a reinforcement learning agent, it generates the task offloading ratio between the vehicle and various edge computing clusters based on vehicle location, task attributes, and cluster resource availability. This minimizes overall energy consumption and latency during task offloading while satisfying the maximum tolerable latency constraint, resulting in an efficient and stable inter-cluster task offloading strategy. After allocating the offloading ratio between clusters, each edge computing cluster further refines and distributes the received tasks to specific edge nodes within the cluster according to the offloading ratio. A fine-grained offloading method based on a greedy strategy is employed. Offloading priorities are calculated based on the computing power, service capabilities, and physical distance of the edge nodes within the cluster to the vehicle. Nodes with lower priorities are eliminated, and weights are determined by normalizing priority values. Based on these weights, the offloading ratio of tasks is allocated among the nodes within the cluster, achieving load balancing and efficient resource utilization within the cluster, further reducing queuing latency and transmission energy consumption.

[0143] The invention provides an embodiment of an in-vehicle edge computing task offloading system based on deep reinforcement learning. The in-vehicle edge computing environment refers to an intelligent vehicle network that forms and supports vehicle latency-sensitive services through vehicle-to-edge wireless communication. Heterogeneous edge devices include edge nodes with different computing resources and service capabilities, such as RSUs, base stations, and intelligent traffic lights. Vehicles can offload tasks to edge devices for collaborative computing through wireless communication.

[0144] In an vehicular edge computing environment, a system model is established, comprising a vehicle layer, an edge device layer, and a service layer. The vehicle acts as the task generator, while edge devices support different types of task offloading based on computing power, service deployment, and available resources. For tasks generated by the vehicle, under the constraint of maximum tolerable latency, it is necessary to comprehensively consider transmission latency, propagation latency, queuing latency, computation latency, and local and edge computing energy consumption, balancing the offloading ratio with the offloading target to achieve joint optimization of latency and energy consumption.

[0145] During the offloading process, a heuristic strategy is first used to dynamically aggregate heterogeneous edge devices. By analyzing the types of services supported by the edge devices, their computing power, location distance, and resource status, heterogeneous edge devices with complementary service capabilities and close proximity are aggregated into a fully functional edge computing cluster. This clustering approach reduces the transmission and management overhead caused by cross-node offloading, providing a scalable, load-balanced, and efficient scheduling foundation for subsequent task offloading.

[0146] Then, task offloading decisions are made at the cluster level. A deep reinforcement learning method based on proximal policy optimization is used, taking environmental information such as vehicle location, task characteristics, and edge cluster status as state inputs, and combining the latency and energy consumption performance after task offloading as reward feedback. This trains the agent to obtain the offloading ratio between the vehicle's local location and different edge clusters. During training, reward shaping and probabilistic policy selection improve policy convergence speed and stability, enabling dynamic adaptive adjustment of the offloading ratio under dynamic network conditions. This ensures that the offloading process meets task latency constraints while reducing total system energy consumption.

[0147] Finally, fine-grained task offloading and allocation are further completed within the cluster. Based on a greedy strategy, the cluster child nodes are prioritized according to static indicators such as node computing power, service matching degree, and physical distance. Tasks are preferentially allocated to edge nodes with high service capacity, sufficient resources, and close physical distance. The final offloading ratio is determined by normalized weighted allocation, so as to achieve efficient allocation and load balancing of tasks within the cluster, effectively reduce queuing latency and computing energy consumption, and improve the system's resource utilization and task processing efficiency.

[0148] It is understood that the in-vehicle edge computing task offloading system based on deep reinforcement learning provided by the present invention corresponds to the in-vehicle edge computing task offloading method based on deep reinforcement learning provided in the foregoing embodiments. The relevant technical features of the in-vehicle edge computing task offloading system based on deep reinforcement learning can be referred to the relevant technical features of the in-vehicle edge computing task offloading method based on deep reinforcement learning, and will not be repeated here.

[0149] This invention provides a method and system for offloading vehicular edge computing tasks based on deep reinforcement learning. It consists of three key algorithm modules: a heuristic node aggregation algorithm (HNA), a cluster-to-cluster offloading decision algorithm based on near-end policy optimization (CBPPO), and a greedy intra-cluster task allocation algorithm (GICOA). The overall process follows the approach of "structure optimization—policy training—fine-grained scheduling," gradually realizing abstract cluster management of heterogeneous edge resources, reinforcement learning optimization of cross-cluster offloading strategies, and energy-efficiency-aware task scheduling among sub-nodes within the cluster. The execution flow of each of the three algorithms is described below.

[0150] First, the HNA algorithm is used to reconstruct and aggregate the service capabilities of heterogeneous edge nodes distributed throughout the system before task unloading. After initializing the target service set and the candidate edge node set, the algorithm determines whether the currently aggregated services meet the full coverage requirement. If not, it selects a node from the candidate nodes that can provide the most scarce service and adds it to the current aggregation set. And remove it from the candidate set. Repeat this process until the current service set is complete, at which point... Add to cluster list For the remaining unaggregated nodes, they are assigned to the nearest cluster based on their distance from each aggregated cluster to supplement computing resources. This algorithm can significantly reduce the complexity of heterogeneous resource management and provide a unified cluster abstraction for subsequent offloading strategies.

[0151] Secondly, the CBPPO algorithm, as the core policy optimization module for optimizing the task offloading ratio among clusters in this invention, is implemented based on the Proximal Policy Optimization (PPO) framework. During each training round, the agent first determines whether the current task allocation has been completed; if not, it optimizes the policy network. and state-value function network Calculate the advantage function in the current state. If the policy update conditions are met, the agent executes the action. Get the next state The algorithm saves relevant state transition data to an experience pool for subsequent updates. The action vector output by the strategy represents the offloading ratio of the current task between the local machine and various edge computing clusters, thus guiding the actual scheduling process. This algorithm can achieve efficient convergence and dynamic adaptive adjustment of the offloading strategy under varying network conditions.

[0152] Finally, the GICOA algorithm is used to fine-tune the allocation of offloaded tasks within the cluster, ensuring maximum resource utilization and balanced node load. The algorithm first reads the input task queue and the corresponding set of serviceable child nodes, and then calculates the comprehensive computational performance index for each candidate node. Subsequently, the average statistical efficiency was calculated. The method involves removing low-quality nodes with below-average performance. The remaining nodes are then normalized and used as offload weights to proportionally distribute tasks across nodes. Once all tasks are fully scheduled, the final execution plan is output. This approach uses a simple and efficient greedy strategy to jointly optimize latency and energy consumption within the cluster, effectively reducing system processing bottlenecks.

[0153] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0159] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for offloading vehicle-mounted edge computing tasks based on deep reinforcement learning, characterized in that, The task unloading method includes: Step 1: Aggregate multiple heterogeneous edge devices into an edge computing cluster through node aggregation; Step 2: Model the task unloading problem as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. Set the action to output the unloading ratio between the local machine and each cluster for each task. Set the constraint that the latency is less than the maximum tolerable latency. Set the state to include: the current task unloading ratio, the service set of each cluster and the attributes of the task to be unloaded. Step 3: At each action, generate the unloading ratio of the task between the local machine and each cluster based on the current state, and construct an instant reward based on the energy consumption and latency after each task unloading. Iterate and update the strategy to guide the action based on the instant reward feedback; thus obtaining the unloading ratio of each task between the local machine and each cluster. Step 4: Within each cluster, perform fine-grained task allocation based on a greedy strategy; Step 1 includes: Step 101: Count the set of services that are not yet covered by the current cluster to be aggregated, and select the service with the fewest covered services among the remaining nodes. As a service to be supplemented; Step 102: If the current cluster to be generated is empty, randomly select one from the remaining nodes that contains... nodes Join the cluster and set the node Remove from candidate nodes; If the current cluster to be generated is not empty, calculate the weighted cluster center coordinates of the current cluster, and select the node that is closest to the cluster center and contains the cluster center from the remaining nodes. nodes Join the cluster and set the node Remove from candidate nodes; Step 103: Continue executing step 102. When the current cluster has full service capabilities, generate the next cluster; when the remaining number of services is insufficient to generate a new full service cluster, select nodes sequentially from the remaining nodes. Calculate the node Distance to the cluster centers of all generated clusters, for each node Assign to the nearest cluster As a supplementary computing resource; The formula for the instant reward constructed in step 3 is as follows: in, This indicates that the vehicle has completed its mission. Total energy consumption Indicates task The completion delay This is the time delay weighting coefficient. , As a reward adjustment factor, Indicates the task's sequence number. This indicates the total number of tasks in a round.

2. The task unloading method according to claim 1, characterized in that, The formula for calculating the weighted cluster center coordinates of the current cluster in step 102 is as follows: in, Indicates the first A set of edge nodes in a cluster to be generated. Represents a node The number of cores; , for Mid-edge device nodes The horizontal and vertical coordinates; In step 103, the computing node The formula for the distance to the cluster centers of all generated clusters is: in, , For nodes The coordinates.

3. The task unloading method according to claim 1, characterized in that, The state set in step 2 is represented as follows: ； in, This indicates the task unloading status as of the current moment, where Indicates the number of computing clusters. and Representing tasks Uninstall to local and cluster The uninstallation rate; This represents a list of service sets contained in each cluster, where Represents a cluster The set of services included; Indicates the first A list of attributes for each task; The action is represented as: in, Indicates task The action, Indicates task The proportion calculated locally. Indicates task Unload to cluster The proportion.

4. The task unloading method according to claim 1, characterized in that, The objective function for iteratively updating the strategy guiding the actions in step 3 is: in, in, and They represent the old and new strategies, respectively. Represents the state value function, Indicates cumulative return. Indicates task The action, Indicates task state, This represents the expectation of the trajectory distribution at time step t. A hyperparameter representing the range to be truncated.

5. The task unloading method according to claim 1, characterized in that, Step 4 includes: Step 401: Select a set of child nodes within the cluster that have the services required to process the current task; construct a calculation function for the optimization target value based on the latency and energy consumption cost when the task is unloaded to any child node; and prioritize the candidate child nodes based on the optimization target value. Step 402: Remove child nodes whose optimization target value is higher than the average level; Step 403: Normalize the optimization target values ​​of the remaining child nodes to obtain weights, and determine the unloading ratio of the task among the child nodes based on the weights.

6. The task unloading method according to claim 5, characterized in that, The function for calculating the optimization objective value is: in, Indicates task Unload to child nodes within the cluster Total time delay This represents the corresponding total energy consumption. , These are the time delay and energy consumption adjustment weighting coefficients, respectively.

7. The task unloading method according to claim 5, characterized in that, The formula for calculating the weight in step 403 is as follows: in, Indicates task Unload to a single child node Optimize target value, This represents the average of the optimization objective values ​​for all remaining child nodes; The formula for calculating the unloading ratio of tasks across child nodes is as follows: in, Indicates task It has been assigned to the cluster at the cluster level. The uninstallation rate This indicates that the task is in the cluster. Internal unloading to child nodes The proportion.

8. A vehicle-mounted edge computing task offloading system based on deep reinforcement learning, characterized in that, The task offloading system includes: a vehicle layer, an edge computing cluster layer, and a cloud service layer; The vehicle layer generates time-sensitive computing tasks during the driving process; The edge computing cluster layer consists of multiple heterogeneous edge devices aggregated through nodes to form various edge computing clusters. The cloud service layer models the task unloading problem generated by the vehicle layer as a Markov decision process with the objective of minimizing the weighted sum of total energy consumption and latency. The action is set to output the unloading ratio between the local machine and each cluster for each task, with the constraint that latency is less than the maximum tolerable latency. The state includes: the current task unloading ratio, the service set of each cluster, and the attributes of the task to be unloaded. At each action, the unloading ratio between the local machine and each cluster is generated based on the current state, and an instant reward is constructed based on the energy consumption and latency after each task unloading. The iterative update of the action strategy is guided by the feedback of the instant reward. This yields the unloading ratio of each task between the local machine and each cluster. After obtaining the offloading ratio of each task between the local machine and each cluster, each cluster in the edge computing cluster layer performs fine-grained task allocation based on a greedy strategy. The edge computing cluster layer consists of various edge computing clusters formed by the aggregation of multiple heterogeneous edge devices through nodes, including: Step 101: Count the set of services that are not yet covered by the current cluster to be aggregated, and select the service with the fewest covered services among the remaining nodes. As a service to be supplemented; Step 102: If the current cluster to be generated is empty, randomly select one from the remaining nodes that contains... nodes Join the cluster and set the node Remove from candidate nodes; If the current cluster to be generated is not empty, calculate the weighted cluster center coordinates of the current cluster, and select the node that is closest to the cluster center and contains the cluster center from the remaining nodes. nodes Join the cluster and set the node Remove from candidate nodes; Step 103: Continue executing step 102. When the current cluster has full service capabilities, generate the next cluster; when the remaining number of services is insufficient to generate a new full service cluster, select nodes sequentially from the remaining nodes. Calculate the node Distance to the cluster centers of all generated clusters, for each node Assign to the nearest cluster As a supplementary computing resource; The formula for the constructed instant reward is: in, This indicates that the vehicle has completed its mission. Total energy consumption Indicates task The completion delay This is the time delay weighting coefficient. , As a reward adjustment factor, Indicates the task's sequence number. This indicates the total number of tasks in a round.

Images