A method and system for offloading computing in Internet of Vehicles based on multi-objective reinforcement learning
Through the multi-objective reinforcement learning method, the problems of insufficient computing power and unbalanced resource allocation in the Internet of Vehicles are solved, and the delay, energy consumption and privacy security are optimized during vehicle calculation and unloading are achieved, ensuring load balance in the MEC environment. and user information security.
Patent Information
- Application Number
- CN202111156213.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-09-29
AI Technical Summary
In the vehicle network driven by 5G network, the computing power of the on-board computing unit is insufficient, and it is difficult to meet the real-time requirements of low latency. The prior art has failed to effectively consider the security and privacy of offload tasks and the load balancing of resource allocation, especially It is in a time-varying and uncertain MEC environment.
Using the Internet of Vehicles Computational Unloading Method based on Multi-Objective Reinforcement Learning, the cost model is established and multi-objectives are optimized through the RMDDQN-Learning method and the Chebischev scale method. The value of RBF neural network is dynamically adjusted to realize vehicle calculations. Dynamic tradeoffs on latency, energy consumption, load balancing and privacy security during the uninstall process.
It realizes the reduction of vehicle unloading delay and energy consumption in the Internet of Vehicles, ensures load balancing and user information security in the MEC environment, and provides more efficient computing resource allocation and delay restriction service quality assurance.
Smart Images

Figure CN113961204B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computation offloading and resource allocation in a vehicle-connected network driven by a 5G network, and specifically to a vehicle-connected network computation offloading method and system based on multi-objective reinforcement learning. Background Art
[0002] As one of the most promising technologies in the 5G era, traditional vehicle ad hoc networks are rapidly developing towards the Internet of Vehicles (IoV). In-vehicle applications and services are becoming more and more abundant (such as autonomous driving, video-assisted real-time navigation, augmented reality, etc.), which are often computationally intensive, energy-intensive and low-latency applications. However, the limited computing power of the on-board computing unit has become a bottleneck for these applications, making it difficult to meet the real-time requirements of low latency. Mobile edge computing (MEC) is becoming a new and attractive computing paradigm, which pushes cloud computing capabilities closer to vehicular terminals, that is, VT can offload computationally intensive tasks to roadside units (RSUs) connected to MEC servers through the wireless access network vehicular-to-roadside (V2R), thereby expanding the computing power, battery capacity and storage capacity of mobile devices. However, how to make offloading decisions and resource allocation in a time-varying and uncertain MEC environment is a key challenge. On the one hand, current research only optimizes based on latency and energy consumption, without considering the security and privacy of offloaded tasks. On the other hand, due to the limited computing, storage, bandwidth and other resources of MEC servers, the load balancing of RSU has a significant impact on task execution. To make matters worse, user needs may change over time. For example, some in-vehicle applications require less latency when an emergency is detected, but require longer battery life when no event is detected, which cannot effectively improve the efficiency of vehicle network computing offloading. Therefore, a decision module is needed that can dynamically weigh the importance of multiple goals to ensure higher network performance (including offloading latency, energy consumption, and load balancing) during computing offloading, while protecting the privacy and security of offloaded tasks. Summary of the invention
[0003] The purpose of the present invention is to provide a method and system for vehicle network computing offloading based on multi-objective reinforcement learning to overcome the shortcomings of the prior art.
[0004] In order to achieve the above object, the present invention adopts the following technical scheme:
[0005] A method for offloading computing in an Internet of Vehicles based on multi-objective reinforcement learning comprises the following steps:
[0006] S1, establish a cost model based on the cost of the vehicle unloading process;
[0007] S2, using the RMDDQN-Learning method to perform multi-objective optimization on the cost model according to the cost of the vehicle calculation unloading process, and obtain the Pareto optimal solution of multi-objective optimization;
[0008] S3, using Chebyshev scaling method to scalarize the rewards of multiple objectives of the vehicle calculation unloading process cost, so as to approach the Pareto optimal solution;
[0009] S4 uses RBF neural network to learn the value of multiple objectives and dynamically adjust the weights, so as to obtain the optimal solution among multiple objectives, and perform Internet of Vehicles computing offloading based on the optimal solution among multiple objectives.
[0010] Furthermore, the cost of the vehicle computation offloading process includes latency, energy consumption, privacy entropy, load balancing, and vehicle mobility.
[0011] Furthermore, suppose that at a certain time t0, there are N mobile devices MD within the service range of the RSU, represented by N = {1, 2, ..., N}. Each mobile device MD can send an unloading request to the roadside unit RSU within the service range. The unloading task of MD n is Γ n =(b n ,c n ,d n ), where b n Indicates the amount of input data for the task, c n The total number of CPU cycles required to complete the task, d n Indicates the maximum delay that MD n can tolerate.
[0012] Furthermore, the vehicle calculates the unloading process task Γ n The total execution time T and total energy consumption E are:
[0013]
[0014]
[0015] I n is the unloading variable, w is the bandwidth of MU n, f n is the computing resource allocated to MU n, Total latency of execution in local and offload modes, respectively.
[0016] Furthermore, the load balancing of the RSU equipment is expressed as
[0017]
[0018]
[0019]
[0020] Load i (t0) represents the comprehensive load of all resources of RSU i during t0, and lb represents the number of load balancing indicators.
[0021] Furthermore, the computation offloading in the Internet of Vehicles scenario is modeled as a constrained multi-objective optimization:
[0022]
[0023] Among them, I and f are offloading variables and computing resource variables respectively, and constraint C1 means to ensure that all tasks are completed within the deadline d n before; Constraint C2 means that the task is either executed locally or offloaded to the RSU for execution; Constraint C3 means that each offloaded task can obtain the computing resources allocated by the RSU to ensure the smooth execution of the task; Constraint C4 means that the computing resources allocated to each MD do not exceed the total computing resources of the RSU, and C5 is the latency and service scope constraint, that is, task Γ n The total execution time should not exceed the maximum tolerance time t p .
[0024] Furthermore, Tchebycheff is used to scale the q vector and measure the distance between the multi-objective solution x and each objective function f, that is,
[0025]
[0026] in,
[0027] For state s and action a, replace the objective function value f with Get the scalarized SQ value (SQ-value):
[0028]
[0029] Furthermore, the Gaussian function used in the weighted RBF network between multi-objective reinforcement learning objectives is
[0030]
[0031] Among them, ||xc|| represents the Euclidean distance, χ represents the width (shape parameter), and the second layer implements weighted summation. The network output is as follows
[0032]
[0033] Among them, M is the number of RBF, W is i Represents RBFφ i The weight, S i Represents RBFφi The parameter vector of .
[0034] Furthermore, during the weight update process, the data obtained by the system exploring the environment is stored, and then random sampling is used to mix past experience with current experience to update the parameters of the deep neural network.
[0035] A vehicle network computing unloading system based on multi-objective reinforcement learning includes an optimization module and an unloading module. The optimization module is used to store a cost model established according to the cost of a vehicle computing unloading process, and simultaneously adopts an RMDDQN-Learning method to perform multi-objective optimization on the cost model according to the cost of the vehicle computing unloading process to obtain a Pareto optimal solution of the multi-objective optimization. The Chebyshev scaling method is used to scalarize the rewards of multiple objectives of the cost of the vehicle computing unloading process, thereby approaching the Pareto optimal solution. The unloading module adopts an RBF neural network to learn the value dynamic adjustment weights between multiple objectives, thereby obtaining the optimal solution between multiple objectives, and performs vehicle network computing unloading output according to the optimal solution between multiple objectives.
[0036] Compared with the prior art, the present invention has the following beneficial technical effects:
[0037] The present invention discloses a method for offloading computing in an Internet of Vehicles based on multi-objective reinforcement learning. A cost model is established according to the cost of the vehicle computing offloading process. The RMDDQN-Learning method is used to perform multi-objective optimization on the cost model according to the cost of the vehicle computing offloading process, and a Pareto optimal solution for multi-objective optimization is obtained, which satisfies the simultaneous optimization of multiple optimization objectives involved in the vehicle computing offloading process. A learning method based on an RBF neural network is used to learn the decision value of each objective, so as to better dynamically adjust the weight of each objective. The purpose of multi-objective optimization is to achieve energy consumption, task delay, load balancing of RSUs, and privacy security of offloading tasks by jointly considering offloading decisions and allocation of computing resources. By optimizing multiple indicators of computing offloading, the vehicle offloading delay and energy consumption in the Internet of Vehicles are kept at a low basic level, so that resource-constrained devices can offload computing-intensive tasks to edge devices, thereby providing unique delay-limited service quality assurance for a wide range of businesses.
[0038] The present invention takes into account the mobility of terminal vehicles, and jointly optimizes latency, energy consumption, privacy entropy, and load balancing, and introduces a multi-objective reinforcement learning algorithm RMDDQN-Learning to solve the optimization problem of multiple conflicting objectives. Each DDQN agent pursues rewards on different objectives respectively, dynamically approaching the Pareto frontier. The obtained multi-objective optimal solution pursues the highest privacy entropy while minimizing the latency, energy consumption, and load balancing of computational offloading. The value changes of different objectives are adjusted dynamically. It can effectively reduce the latency and energy consumption of computational offloading in the Internet of Vehicles, and ensure the load balancing of MEC and the security of user information.
[0039] Furthermore, in the time-varying MEC environment, the privacy entropy of the offloaded tasks is guaranteed, ensuring the privacy security of the end users. In the computing cluster of multiple MEC servers, RMDDQN-Learning also optimizes the load balancing indicators, making fuller use of the edge server resources provided by the service operator. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a schematic diagram of a vehicle networking scenario enabled by MEC in an embodiment of the present invention.
[0041] Figure 2 It is a scaled schematic diagram of a multi-objective optimization problem based on the Chebyshev method in an embodiment of the present invention.
[0042] Figure 3 1 is a Markov diagram of single target and multi-target in an embodiment of the present invention.
[0043] Figure 4 It is a framework diagram of the vehicle networking computing offloading algorithm based on the RBF-based multi-objective DDQN reinforcement learning method in an embodiment of the present invention.
[0044] Figure 5 It is a comparison of energy consumption and time in the embodiment of the present invention. DETAILED DESCRIPTION
[0045] The present invention is further described in detail below in conjunction with the accompanying drawings:
[0046] like Figure 1 As shown, a method for offloading vehicle network computing based on multi-objective reinforcement learning includes the following steps:
[0047] S1, construct a cost model for the computational offloading process, including latency, energy consumption, privacy entropy, load balancing, and consider the changes in the vehicle's communication parameters during the process, that is, consider the mobility of the vehicle.
[0048] Define M = {1, 2, ..., M} to represent the set of RSU units, which is used to calculate the delay of RSU units. Suppose at a certain time t0, there are N mobile devices MD within the service range of RSU, represented by N = {1, 2, ..., N}. Each MD can send an offload request to the RSU within the service range. Define the offload task Γ of MD n n =(b n ,c n ,d n ), where b n Indicates the input data size of the task, c n The total number of CPU cycles required to complete the task, d n Indicates the maximum delay that MD n can tolerate.
[0049] Considering the binary offloading mode, each MD either n Either execute the task on the local device or offload it to the edge node RSU for execution. Let the offload variable be I n ∈{0,1}, where I n =0 means the task is executed locally, otherwise, I n =1 means offloading the task to RSU for execution. Then the total execution delays in local and offloading modes (offloading the task to RSU) are for
[0050]
[0051]
[0052] The corresponding task Γ n The total execution energy consumption is
[0053]
[0054]
[0055] In the above formula, the local execution CPU frequency and energy consumed per CPU cycle of the nth mobile device MD n are respectively τ n , the local execution time of MD n should not exceed the maximum tolerable delay d n . p n is the power of MU n, w is the bandwidth of MU n, f n is the computing resource allocated to MU n. Combining the latency and energy consumption in the local mode and the offloading mode, we can get task Γ n The total execution time T and total energy consumption E are
[0056]
[0057]
[0058] Secondly, privacy entropy, as a quantitative privacy measurement method, is widely used to measure the security of task data transmission. This paper introduces privacy entropy to ensure the security index of transmitted data, that is, the larger the privacy entropy, the safer the task transmission. Suppose each task Γ n The arrival of follows a Poisson process, so the entire task arrival process is also a Poisson process; it can be expressed as
[0059]
[0060] in Then task Γ n and n The congruence relationship between
[0061]
[0062] Therefore, task Γ n The privacy entropy is calculated as
[0063]
[0064] Finally, due to the limited computing, storage, and bandwidth resources of the MEC server, the load balancing of the RSU has a significant impact on task execution. This paper considers the load balancing of the offloading process and expresses the load balancing of all RSU devices as
[0065]
[0066]
[0067]
[0068] Load i (t0) represents the comprehensive load of all resources of RSU i during t0, and lb represents the number of load balancing indicators. To simplify the problem, the present invention considers calculating the CPU utilization, memory, and network bandwidth of RSU, that is, lb = 3. k It satisfies the conditions The weight of each resource, L k (t) is the utilization rate of each resource within each time interval δ. The commonly used top command can be used to obtain system information and then analyze it. avg(t0) is the average load of all computing devices. LB(t0) represents the load balancing value. The smaller the value, the better the load balancing result.
[0069] S2, using the RMDDQN-Learning method to perform multi-objective optimization on the cost model according to the cost of the vehicle computing offloading process: The computing offloading in the Internet of Vehicles scenario is modeled as a constrained multi-objective optimization problem (CMOP). This model comprehensively considers the latency, energy consumption and load balancing in the computing offloading process, while weighing the security and privacy of the offloading task. That is, the computing offloading in the Internet of Vehicles scenario is modeled as a constrained multi-objective optimization problem. That is, the optimization problem can be described as
[0070]
[0071] Among them, I and f are offloading variables and computing resource variables respectively. Constraint C1 means to ensure that all tasks are completed within the deadline d. n before; Constraint C2 means that the task is either executed locally or offloaded to the RSU for execution; Constraint C3 means that each offloaded task can obtain the computing resources allocated by the RSU to ensure the smooth execution of the task; Constraint C4 means that the computing resources allocated to each MD do not exceed the total computing resources of the RSU. C5 is the latency and service scope constraint, that is, task Γ n The total execution time should not exceed the maximum tolerance time t p On the other hand, to ensure the successful completion of the unloading task, task Γ n It must be completed before the mobile device (MD) moves out of the service area R provided by RSU m.
[0072] S3, the multi-objective reinforcement learning algorithm executes multiple DDQNs in parallel to obtain the Pareto optimal solution of the multi-objective optimization problem. The computation offloading problem under MEC is expressed as a control strategy decision problem based on the MDP model, that is, Among them, R1=T, R2=E, R3=-H, R4=LB.
[0073] S4, by combining the Chebyshev scaling method, the rewards of multiple objectives are scalarized to approach the Pareto optimal solution. That is, Tchebycheff is used to scale the q vector (Chebyshev metric). Usually, the utopian point z * is used as a reference, l p -metrics are used to measure distance, and each objective function is given a weighting coefficient w o . Measure the multi-objective solution x to each objective function f (i.e. f o ,o=1,...,m), that is
[0074]
[0075] in, In the case of p = ∞, this metric is also called the Chebyshev metric.
[0076]
[0077] For state s and action a, replace the objective function value f with Get the scalarized SQ value (SQ-value):
[0078]
[0079] S5 uses RBF network to dynamically adjust the weights of the values between learning objectives, thereby obtaining the optimal solution between multiple objectives, so that VT can maximize the security and privacy of the unloading task on the basis of minimizing the network performance (delay, energy consumption, load balancing) during the unloading process. RBF includes two layers of neural networks. The first layer is composed of RBF to generate the input vector x = [x1, x2, ..., x d ] is a nonlinear change. The spatial dimension d is equal to the number of RBFs. RBF is a real function of the distance between the input x and some fixed points c (called centers). The Gaussian function used in the weighted RBF network between the multi-objective reinforcement learning objectives of the present invention is
[0080]
[0081] Among them, ||xc|| represents the Euclidean distance, χ represents the width (shape parameter), and the second layer implements weighted summation. The network output is as follows
[0082]
[0083] Among them, M is the number of RBF, W is i Represents RBFφ i The weight, S i Represents RBFφ i The network learning process requires a set of learning instances, each of which contains the network input (the coordinate vector of the interpolation node) and the target output (the function value in the node). During the learning process, the training samples are used to update the instantaneous mean square error (MSE).
[0084] Right now
[0085]
[0086] n represents the number of interpolation points, e j is the approximation error of the jth node. j represents the coordinates of the jth node, u(x j) is the network output of the jth node, T j is the known function value of the short-term memory module of the jth node. Therefore, Figure 5 As shown in the figure, during the training process of the RBF-based multi-objective DDQN reinforcement learning computation offloading algorithm (RMDDQN-Learning), the q-values of different objective functions under action a are scaled by the Chebyshev scaling method to find the optimal strategy. At the same time, the weighted RBF network dynamically updates the weight of the target, thereby guiding multi-objective reinforcement learning.
[0087] To improve learning efficiency, the reinforcement learning agent on each target stores the data obtained by the system exploring the environment during the weight update process. Then, random sampling is used to mix past experience with current experience and update the parameters of the deep neural network. This not only reduces data correlation, but also makes samples reusable.
[0088] The network update process is as follows:
[0089] like Figure 4 As shown in Figure 2, we adopt a multi-objective reinforcement learning method of multiple double DQN (DDQN) to optimize multiple indicators in the computation offloading process. i There is a Q i (s; a; θ i ) function, which corresponds to each goal (including latency, energy consumption, load balancing, and privacy entropy). The DDQN algorithm solves the problem that DQN may overfit the Q value, that is, it adopts Replace the original To update the Q value. DDQN has two value functions, one for selecting actions (the strategy for the current state) and one for evaluating the value of the current state. The parameters of these two value functions are denoted as θ and θ ′ DDQN eliminates the problem of overestimation by decoupling the selection of the target Q value action and the calculation of the target Q value. The execution process of the RMDDQN-Learning algorithm is shown in Algorithm 1. Each DDQN is iteratively optimized through the following loss function
[0090]
[0091] The gradient method is used to optimize the value of MSE in RBF-based weight network learning. At the kth iteration of gradient descent, the parameter vector θ of RBFN is adjusted as follows:
[0092] θ k+1 =θ k +Δθ k+1 (twenty four)
[0093] in is the correction of the vector θ, and η is the learning rate. is the function (23) at iteration k with parameter θ k The gradient vector obtained by the value of . When the function in (23) reaches a small MSE value, the calculation process of the gradient update is completed.
[0094] like Figure 1 As shown in the figure, a method for offloading computing in IoV based on multi-objective reinforcement learning includes a mobile-aware IoV scenario. With the assistance of a mobile edge computing (MEC) server, each MD enjoys the services provided by a roadside unit (RSU) within a certain range, and can offload computing tasks to a MEC server (MES) associated with the RSU to enhance the self-generated computing power and obtain low task execution latency and energy consumption.
[0095] Figure 2 It is a schematic diagram of the scaling of the multi-objective optimization problem based on the Chebyshev method.
[0096] After establishing the energy consumption, latency, privacy entropy, and load balancing models, a multi-objective reinforcement learning method is used to perform computational offloading. In the reward scaling method for multi-objective optimization problems, a nonlinear scaling method is adopted, that is, the Chebyshev scaling method is used to scalar multiple target values. This method overcomes the problem that when some convex Pareto optimal sets are locally concave, linear functions cannot be used to discover some Pareto-dominated actions when used for action selection.
[0097] Figure 3 is a Markov diagram of single-objective and multi-objective examples in the embodiment of the present invention. The present invention expresses the computation offloading problem under MEC as a control strategy decision problem based on the MDP model, that is,<S,A,P,R,γ> Unlike traditional Markov models, multi-objective reinforcement learning returns a vector instead of a scalar, i.e. it maps state-action pairs to a reward vector Among them, R1 = T, R2 = E, R3 = -H, R4 = LB. The agent on each target optimizes its own target on different targets.
[0098] Figure 4This is a framework diagram of the vehicle network computing offloading algorithm based on the RBF-based multi-objective DDQN reinforcement learning method in the embodiment of the present invention. Multiple DDQNs can be executed in parallel to obtain the Pareto optimal solution to the multi-objective optimization problem. That is, RMDDQN-Learning can optimize the network performance (latency, energy consumption, load balancing) of computing offloading while ensuring the security and privacy of task offloading. This method also proposes a novel dynamic weight learning method based on RBF neural network, namely radial basis function neural network, to learn the value changes of each target and dynamically adjust the weights between targets.
[0099] at last, Figure 5 This is a comparison of energy consumption and time in the embodiment of the present invention. It can be seen that in the process of offloading decision optimization, energy consumption and latency are inversely correlated. The RMDDQN-Learning we proposed can strike a balance between the two and maintain the minimum energy consumption and latency among all algorithms.
[0100] Example:
[0101] The present invention establishes a complete computational offloading scheme and sets the task size to [50, 1000] KB according to the characteristics of the 4G cellular network. The number of CPU cycles required for each task is [0.2, 1] Gigacycles, and the CPU frequency of MEC is [1.5, 4.5] GHz. When the bandwidth is 10 MHz, the transmission rate of MU can be determined according to the Shannon formula. When the noise power is -172 dBm, the transmission power is 10 dBm. This experiment uses TensorFlow 1.10 and Python 3.5 on Centos 7.9 to implement the RMDDQN-Learning offloading algorithm (RBF-based multi-objective DDQN reinforcement learning computation offloading algorithm). For comparison, we compare it with other benchmark algorithms. They are NSGA-II: Improved Fast Elite Non-dominated Sorting Genetic Algorithm, which uses genetic algorithm to compromise multiple objectives in MEC network, adopts multi-objective Q-learning algorithm based on Chebyshev Scalarization, adopts multi-objective DQN algorithm based on decision value, and randomly generates offloading decisions. Figure 5 As shown in the figure, the results show that RMDDQN-Learning learns the value of each target and dynamically adapts the weights between targets through the RBF network, thereby ensuring the balance between multiple targets and achieving the best energy efficiency performance in the comparison of various benchmark algorithms.
Claims
1. A method for offloading computing in Internet of Vehicles based on multi-objective reinforcement learning. It is characterized in that The following steps are involved: S1, establish a cost model based on the cost of the vehicle computing unloading process. The cost of the vehicle computing unloading process includes latency, energy consumption, privacy entropy, load balancing and vehicle mobility; Suppose at a certain time t 0 , N mobile devices MD within the service range of RSU, denoted as N = {1,2,...,N}, each mobile device MD can send an unloading request to the roadside unit RSU within the service range, and the unloading task of MD n is Γ n =(b n ,c n ,d n ), where b n Indicates the input data size of the task, c n The total number of CPU cycles required to complete the task, d n Indicates the maximum delay that MD n can tolerate; Vehicle calculation unloading process task Γ n The total execution time T and total energy consumption E are: I n is the unloading variable, w is the bandwidth of MU n, f n is the computing resource allocated to MU n, are the total latency of execution in local and offload modes, respectively; The load balancing of RSU devices is expressed as Load i (t 0 ) indicates that RSU i is at t 0 The combined load of all resources during the period, lb represents the number of load balancing indicators; S2, using the RMDDQN-Learning method to perform multi-objective optimization on the cost model according to the cost of the vehicle calculation unloading process, and obtain the Pareto optimal solution of multi-objective optimization; S3, using Chebyshev scaling method to scalarize the rewards of multiple objectives of the vehicle calculation unloading process cost, so as to approach the Pareto optimal solution; S4, uses RBF neural network to learn the value of multiple objectives and dynamically adjust the weights, so as to obtain the optimal solution among multiple objectives, and performs Internet of Vehicles computing offloading according to the optimal solution among multiple objectives; Modeling computational offloading in the IoV scenario as a constrained multi-objective optimization: Among them, I and f are offloading variables and computing resource variables respectively, and the constraint C 1 Indicates that all tasks are guaranteed to be completed within the deadline d n Completed before; constraint C 2 Indicates that the task is either executed locally or offloaded to the RSU for execution; constraint C 3 It means that each offload task can obtain computing resources allocated by RSU to ensure the smooth execution of the task; constraint C 4 Indicates that the computing resources allocated to each MD do not exceed the total computing resources of the RSU. 5 are the delay and service range constraints, that is, task Γ n The total execution time should not exceed the maximum tolerance time t p ; Tchebycheff is used to scale the q vector and measure the distance value from the multi-objective solution x to each objective function f, that is, in, For state s and action a, replace the objective function value f with Get the scalarized SQ value (SQ-value): The Gaussian function used in the weighted RBF network between multi-objective reinforcement learning objectives is: Among them, ||xc|| represents the Euclidean distance, χ represents the width (shape parameter), and the second layer implements weighted summation; the network output is as follows Among them, M is the number of RBF, W is i Represents RBFφ i The weight, S i Represents RBFφ i The parameter vector of .
2. According to claim 1, a method for offloading computing in Internet of Vehicles based on multi-objective reinforcement learning, It is characterized in that During the weight update process, the data obtained by the system exploring the environment is stored, and then random sampling is used to mix past experience with current experience to update the parameters of the deep neural network.
3. A vehicle networking computing offloading system that implements the vehicle networking computing offloading method based on multi-objective reinforcement learning as claimed in claim 1, It is characterized in that It includes an optimization module and an unloading module. The optimization module is used to store a cost model established according to the cost of the vehicle calculating the unloading process. At the same time, the RMDDQN-Learning method is used to perform multi-objective optimization on the cost model according to the cost of the vehicle calculating the unloading process, and the Pareto optimal solution of the multi-objective optimization is obtained. The Chebyshev scaling method is used to scalarize the rewards of multiple objectives of the cost of the vehicle calculating the unloading process, so as to approach the Pareto optimal solution. The unloading module uses RBF neural network to learn the value of multiple targets and dynamically adjust the weights, so as to obtain the optimal solution among multiple targets, and calculates the unloading output of the Internet of Vehicles based on the optimal solution among multiple targets.
