A dynamic adjustment method for edge nodes in Internet of Vehicles based on reinforcement learning

Abstract

The present invention relates to the technical field of vehicle networking, and discloses a method for dynamically adjusting edge nodes of a vehicle networking based on reinforcement learning, comprising the following steps: establishing a vehicle edge computing network; analyzing the reliability of the vehicle edge computing network, and establishing a reliability model of the vehicle edge computing network; analyzing the energy consumption of the vehicle edge computing network, and establishing an energy consumption model of the vehicle edge computing network; constructing constraint conditions of the vehicle edge computing network, and obtaining a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption; using a multi-objective decision analysis method to convert the dual objectives into a single objective, obtaining a single-objective optimization function with the goal of maximizing deployment effect and modeling it as a Markov decision process, and using an asynchronous dominant actor critic method for multi-threaded solution to obtain a deployment plan for vehicle edge nodes; the method improves the reliability of the vehicle edge computing network and shortens the delay, and can achieve target optimization with high reliability and low energy consumption at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Vehicles (IoV), and in particular to a method for dynamically adjusting IoV edge nodes based on reinforcement learning. Background Art

[0002] With rapid economic development, the scale of automotive networks has expanded dramatically, leading to increasingly prominent problems such as traffic congestion, resource shortages, and traffic accidents. The emergence of Internet of Vehicles (IoV) technology aims to address these issues, and its development has contributed to the exponential growth of computationally intensive and latency-critical applications such as autonomous driving, virtual reality (VR), and augmented reality (AR). However, vehicles typically have limited computing resources, making them unable to provide the computing resources required by these applications. Therefore, mobile cloud computing (MCC) has been proposed as a promising paradigm. Mobile cloud computing offloads tasks generated by vehicles to resource-rich cloud servers. However, if all tasks in the network are offloaded to the cloud, it can lead to network congestion, especially for latency-critical applications. To address this problem, vehicular edge computing (VEC) is considered a promising new solution. VEC combines mobile edge computing (MEC) and the Internet of Vehicles (IoV). VEC offloads tasks generated by vehicles to edge servers located close to the user's location. Compared with traditional mobile cloud computing, vehicle edge computing helps to reduce the burden on the backhaul network. Edge nodes are key facilities in edge computing, and edge nodes in the Internet of Vehicles are usually roadside units (RSUs). However, not all areas can achieve full coverage of roadside units, and the deployment of roadside units in suburban areas may be insufficient, resulting in coverage gaps in some areas. Moreover, the existing edge node deployment methods cannot meet the time-varying service requirements in the network, and the resource demand in the vehicle edge computing network is different at different times. During peak times, roads are congested and resource demand is high, which often leads to resource competition, network congestion, and low system reliability. During non-peak times, resource demand is low, resulting in idling of roadside units and waste of resources. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for dynamic adjustment of edge nodes in the Internet of Vehicles based on reinforcement learning. By establishing a vehicle edge computing network, the vehicle edge computing network is modeled as a dual-objective optimization problem of maximizing reliability and minimizing energy consumption. A multi-objective decision analysis method is used to transform the dual-objective optimization problem into a single-objective optimization problem, and the asynchronous dominant actor-critic method is used to solve the single-objective optimization problem, thereby solving the problem of insufficient coverage of roadside units in suburban areas and difficulty in deploying them to meet time-varying service requirements.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] A method for dynamically adjusting edge nodes in an Internet of Vehicles (IoV) based on reinforcement learning includes the following steps:

[0006] S1. Abstract the study area into a rectangular area, introduce buses as mobile roadside units, establish a vehicle edge computing network, and obtain the tasks generated by vehicles in the vehicle edge computing network as well as the input data, task computing amount, delay constraints, and importance of the tasks;

[0007] S2. Based on the vehicle edge computing network established in step S1, analyze the reliability of the vehicle edge computing network and establish a reliability model of the vehicle edge computing network;

[0008] S3. Based on the vehicle edge computing network established in step S1, analyze the energy consumption of the vehicle edge computing network and establish an energy consumption model of the vehicle edge computing network;

[0009] S4. Based on the reliability model of the vehicle edge computing network established in step S2 and the energy consumption model of the vehicle edge computing network established in step S3, the constraints of the vehicle edge computing network are constructed to obtain a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption;

[0010] S5. Based on the dual-objective optimization function obtained in step S4 with the goals of maximizing reliability and minimizing energy consumption, a multi-objective decision analysis method is used to convert the dual objectives into a single objective, thereby obtaining a single-objective optimization function with the goal of maximizing deployment effect;

[0011] S6. Based on the single-objective optimization function obtained in step S5 with the goal of maximizing the deployment effect, the edge node deployment problem of the single-objective optimization function is modeled as a Markov decision process, and an asynchronous dominant actor-critic method is used for multi-threaded solution to obtain a deployment plan for vehicle edge nodes.

[0012] The present invention has the following beneficial effects:

[0013] 1. This paper proposes a reinforcement learning-based dynamic adjustment method for IoV edge nodes. This method models the dynamic deployment of roadside units (ROUs) in IoVs. By incorporating transmission probability, task effectiveness, task importance, and required latency as reliability assessment factors for the established vehicle edge computing network, it establishes a reliability assessment mechanism for the vehicle edge computing network, thereby improving the reliability of the vehicle edge computing network.

[0014] 2. Dynamically adjust the activation-low-power-sleep state of fixed roadside units and the activation-sleep state of mobile roadside units to meet the time-varying network resource requirements of the Internet of Vehicles and achieve the goal of high reliability and low energy consumption optimization;

[0015] 3. Leverage the asynchronous advantage actor-critic method for multi-threaded training to shorten the latency of edge node deployment in the vehicle edge network by sharing training parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for dynamic adjustment of edge nodes in an Internet of Vehicles based on reinforcement learning proposed by the present invention;

[0017] Figure 2 Schematic diagram of the edge node deployment process in the vehicle edge computing network. DETAILED DESCRIPTION

[0018] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0019] like Figure 1 As shown, a method for dynamic adjustment of edge nodes in an Internet of Vehicles based on reinforcement learning includes the following steps S1-S6:

[0020] S1. Abstract the study area into a rectangular area, introduce buses as mobile roadside units, establish a vehicle edge computing network, and obtain the tasks generated by vehicles in the vehicle edge computing network as well as the input data, task computing amount, delay constraints, and importance of the tasks.

[0021] In this embodiment, this step is the modeling process of the research scenario, the input is the research scenario, and the output is the relevant representation of the research scenario, specifically Figure 2 As shown, Figure 2The black part indicates the area where fixed roadside units (f-RSUs) exist, the white part indicates the area where fixed roadside units (f-RSUs) do not exist, the white triangle indicates the bus departure station, and the black triangle indicates the bus terminal station; the solid arrow and the dotted arrow are bus lines 1 and 2, respectively. Figure 2 In this study, a bus equipped with an edge server is used as a mobile roadside unit (MRU) to construct a vehicle edge computing network model. This improves the reliability and stability of the vehicle edge computing network, thereby obtaining the tasks generated by the vehicle in the vehicle edge computing network, as well as the input data, task computation amount, latency constraints, and importance of the tasks. Furthermore, the vehicle proposed in this invention is a user vehicle.

[0022] Specifically, step S1 includes S11-S15:

[0023] S11. The study area is treated as a rectangular area, with black representing the presence of roadside unit coverage and white representing the absence of roadside unit coverage. The roadside units in the black area are treated as fixed roadside units, and buses are used as mobile roadside units to dynamically fill the gaps in roadside unit coverage and establish a vehicle edge computing network.

[0024] S12. If the radius of the fixed roadside unit is r, then As the side length, divide the rectangular area into several The set of square areas is AR={1,2,…,ar,…,N AR}, and fix the fixed roadside unit at the center of each square area, and the set of fixed roadside units is FR = {1,2,…,fr,…,N FR}, where ar, N AR Respectively represent the arth and Nth of the set of square areas AR AR square area, fr, N FR Respectively represent the fr, Nth of the set of fixed roadside units FR FR Fixed roadside unit.

[0025] In this embodiment, each fixed roadside unit is equipped with certain computing resources, including Represents the computational resources of a fixed roadside unit fr.

[0026] S13. Denote the set of mobile roadside units as MR = {1, 2, ..., mr, ..., N MR Since the computing resources provided by the mobile roadside unit in each square area are dynamically changing, the computing resources provided by the mobile roadside unit in the square area ar are Among them, mr, N MRRespectively represent the mrth and Nth of the set MR of mobile roadside units MR Mobile roadside units, represents the computing resources provided by the mobile roadside unit mr in the square area ar at time t, Indicates whether the mobile roadside unit mr at time t is within the square area ar. Represents the computing resources that the mobile roadside unit mr can provide.

[0027] In this embodiment, since the bus's route is relatively fixed, the driving trajectory is regular, and the running time is stable, it provides a reliable and predictable node for the vehicle edge computing network. As part of public transportation, the operation and maintenance costs of buses are relatively low. Using existing buses as mobile edge nodes does not require a large amount of capital investment and deployment of special equipment, so it is highly cost-effective. In addition, during peak periods, when resource demand is high, mobile roadside units can also dynamically supplement the computing resources in the vehicle edge computing network. Among them, is a binary time-varying indicator variable, used to indicate whether the mobile roadside unit mr at time t is within the square area ar. Then the mobile roadside unit mr is in the square area ar, otherwise it is not in the square area ar.

[0028] S14. The set of vehicles is M1 = {1, 2, ..., n, ..., M}. If each vehicle can only generate one task in the same time slot, in the vehicle edge computing network model, the time is divided into multiple operation cycles K1 = {1, 2, ..., k, ..., K}, then the cycle k is discretized into a set of several time slots: Time Slot The task set generated by the vehicle is Where M represents the total number of vehicles, K represents the total number of cycles, and I represents the total number of time slots in cycle k. denote the 1st, 2nd, i and I time slots of cycle k respectively, They represent the tasks generated by vehicle 1, vehicle 2, vehicle m, and vehicle M in the i-th time slot of cycle k respectively.

[0029] In this embodiment, it is assumed that each vehicle generates only one task in the same time slot, so the time is divided into multiple operation cycles, and each cycle k is discretized into several time slots.

[0030] S15, It is represented by a quadruple Among them D m represents the mission of vehicle m The amount of input data, C m represents the mission of vehicle m The task computation amount, τ m represents the mission of vehicle m The delay constraint, w m represents the mission of vehicle m importance.

[0031] S2. Based on the vehicle edge computing network established in step S1, analyze the reliability of the vehicle edge computing network and establish a reliability model of the vehicle edge computing network.

[0032] In this embodiment, based on the establishment of the vehicle edge computing network in step S1, the reliability of the vehicle edge computing network is analyzed to establish a reliability model, in which the input is the relevant representation of the research scenario and the output is the reliability under the scenario. Since the reliability of a product is defined as its ability to perform a specific function under specific conditions. Therefore, the reliability of the vehicle edge computing network is related to the ability to complete the task within the specified delay constraints. The ability of the vehicle edge computing network to complete the task can be reflected in the success probability, effectiveness, required delay and importance of the task. Therefore, in this step, the success probability, effectiveness, required delay and importance are analyzed respectively to establish a reliability model for the vehicle edge computing network.

[0033] Specifically, step S2 includes S21-S26:

[0034] In this embodiment, the task completion process mainly includes three stages: data upload, data transmission, and result calculation. The failure probabilities of these three stages can be modeled using a Poisson distribution. Given the failure rates of the three stages as λ1, λ2, and λ3, respectively, we can derive the failure probabilities for these three stages.

[0035] S21. Based on the tasks generated by the vehicle in the vehicle edge computing network, calculate the failure probabilities of the three processes of data upload, data transmission, and result calculation during task completion, namely:

[0036]

[0037] in, In the data upload phase, the task generated by vehicle m in the i-th time slot of cycle k is represented by The failure probability of being uploaded to the access fixed roadside unit i′, In the data transmission phase, the task generated by vehicle m in the i-th time slot of cycle k is represented by The failure probability of transmitting from the access fixed roadside unit i′ to the target fixed roadside unit n is, In the result calculation phase, the task generated by vehicle m in the i-th time slot of cycle k is represented by The failure probability calculated on the target fixed roadside unit n, e represents the exponential function, λ1, λ2, and λ3 represent the failure rates of data upload, data transmission, and result calculation, respectively. represents the task generated by vehicle m in the i-th time slot of cycle k Upload delay to fixed roadside unit i′, represents the task generated by the fixed roadside unit i′ to vehicle m in the i-th time slot of period k The transmission delay to the target fixed roadside unit n, represents the task generated by vehicle m in the i-th time slot of cycle k Computational delay at the target fixed roadside unit n.

[0038] In this embodiment, failure at any stage will result in the task The failure of the task The probability of successful completion depends on three stages.

[0039] S22. Calculate the probability of successful completion of the task based on the failure probabilities of the three processes of data upload, data transmission, and result calculation during task completion, namely:

[0040]

[0041] Among them, S m,i′,n represents the task generated by vehicle m in the i-th time slot of cycle k The probability of successful completion is calculated by transmitting the data from the access fixed roadside unit i' to the target fixed roadside unit n.

[0042] In this embodiment, if the completion of a task does not meet the delay constraint, the result of the task is considered invalid. Therefore, the binary variable δ is introduced. m To indicate whether the delay constraint is satisfied.

[0043] S23, introduce a binary variable to determine whether the task meets the delay limit, that is:

[0044]

[0045] Among them, δ m represents the task generated by vehicle m in the i-th time slot of cycle k Whether the delay limit is met, if represents the judgment condition, T m,i′,n represents the task generated by connecting vehicle m to the i-th time slot of period k by accessing a fixed roadside unit i′ The delay of unloading to the target fixed roadside unit n, v m represents the task generated by vehicle m in the i-th time slot of cycle k The specified delay constraints.

[0046] In this example, if a task is effective, the sooner it is completed, the more valuable it is. Therefore, the reliability of the vehicle edge computing network is closely related to the latency required to complete the task. Standardized latency is used to evaluate the vehicle edge computing network's ability to complete tasks.

[0047] S24. Calculate the normalized delay using the following formula:

[0048]

[0049] Among them, m,i′,n represents the task generated by vehicle m in the i-th time slot of cycle k Normalized delay of transmission from access fixed roadside unit i' to target fixed roadside unit n.

[0050] In this embodiment, due to poor communication environment or insufficient computing resources in the vehicle edge computing network, some tasks cannot be completed. In order to maintain the normal operation of the vehicle edge computing network, it is necessary to ensure the completion of important tasks. Therefore, the function f(w m ) to describe the impact of importance on the reliability of vehicle edge computing networks.

[0051] S25. Obtain the impact of description importance on the reliability of the vehicle edge computing network, namely:

[0052] f(w m )=(dw m ) a′

[0053] Where f(·) represents the impact of the description importance on the reliability of the vehicle edge computing network, d represents the corrected importance arrangement, is a constant, and w m represents the task generated by vehicle m in the i-th time slot of cycle k The importance of a′ represents the impact of the important reaction task and is a constant.

[0054] In this embodiment, considering the success probability, effectiveness, required delay and importance of the task, the reliability of the vehicle edge computing network is modeled as

[0055] S26. Based on the probability of successful task completion, whether the task meets the delay constraint, the normalized delay, and the impact of description importance on the reliability of the vehicle edge computing network, a reliability model of the vehicle edge computing network is established, namely:

[0056]

[0057] Where Γ represents the reliability model of the vehicle edge computing network.

[0058] S3. Based on the vehicle edge computing network established in step S1, the energy consumption of the vehicle edge computing network is analyzed and an energy consumption model of the vehicle edge computing network is established.

[0059] In this embodiment, based on the establishment of the vehicle edge computing network in step S1, the energy consumption of the vehicle edge computing network is analyzed to establish an energy consumption model, where the input is the relevant representation of the research scenario and the output is the energy consumption in the scenario. In the vehicle edge computing network, energy consumption mainly includes equipment maintenance energy consumption, equipment switching energy consumption, data transmission energy consumption, task calculation energy consumption and result feedback energy consumption. Therefore, these energy consumptions are analyzed to establish an energy consumption model. Among them, the equipment refers to fixed roadside units and mobile roadside units. In addition, since the result feedback energy consumption of all tasks is extremely small compared to other energy consumption, this item can usually be ignored.

[0060] Specifically, step S3 includes S31-S35:

[0061] S31. Calculate the maintenance energy consumption of all fixed roadside units and all mobile roadside units within period k. The specific process is S3101-S310:

[0062] S3101, when the fixed roadside unit is in the activated state, calculate the i-th time slot of the period k in the square area ar The maintenance energy consumption of the fixed roadside unit is:

[0063]

[0064] in, represents the i-th time slot of period k in the square area ar when the fixed roadside unit is in the active state The maintenance energy consumption of the fixed roadside unit fr, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain a fixed roadside unit fr, represents the computing resources that the fixed roadside unit fr can provide. Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain fixed roadside unit fr transmission, Indicates the transmission rate of the fixed roadside unit fr.

[0065] S3102: When the fixed roadside unit is in a low power consumption state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the fixed roadside unit is:

[0066]

[0067] in, Indicates the i-th time slot of period k in the square area ar when the fixed roadside unit is in low power state Maintaining energy consumption of fixed roadside unit fr.

[0068] S3103: When the fixed roadside unit is in a dormant state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the fixed roadside unit is:

[0069]

[0070] in, Indicates the i-th time slot of period k in the square area ar when the fixed roadside unit is in sleep state Maintaining energy consumption of fixed roadside unit fr.

[0071] S3104, calculate the i-th time slot of period k in the square area ar. The total maintenance energy consumption of the fixed roadside unit is:

[0072]

[0073] in, represents the i-th time slot of period k in the square area ar The total maintenance energy consumption of the fixed roadside unit fr, Indicates that the fixed roadside unit fr is in the i-th time slot In the state, active, halfactive, and sleep respectively represent the active state, low power state, and sleep state.

[0074] S3105. Calculate the i-th time slot of cycle k. In the rectangular area, the maintenance energy consumption of all fixed roadside units is:

[0075]

[0076] in, represents the i-th time slot of period k In the rectangular area, the maintenance energy consumption of all fixed roadside units is Indicates that the fixed roadside unit fr is in the i-th time slot Is it in the square area ar?

[0077] S3106: When the mobile roadside unit is in the active state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the mobile roadside unit is:

[0078]

[0079] in, Indicates the i-th time slot of period k in the square area ar when the mobile roadside unit is in the active state The maintenance energy consumption of the mobile roadside unit mr, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain the mobile roadside unit mr, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain the transmission of mobile roadside unit mr, Represents the transmission rate of the mobile roadside unit mr.

[0080] S3107: When the mobile roadside unit is in a dormant state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the mobile roadside unit is:

[0081]

[0082] in, Indicates that when the mobile roadside unit is in sleep state, the i-th time slot of cycle k in the square area ar Maintaining energy consumption of mobile roadside unit mr.

[0083] S3108. Calculate the i-th time slot of period k in the square area ar. The total maintenance energy consumption of the mobile roadside unit is:

[0084]

[0085] in, represents the i-th time slot of period k in the square area ar The total maintenance energy consumption of the mobile roadside unit mr, Represents the mobile roadside unit mr in the i-th time slot Status in.

[0086] S3109, calculate the i-th time slot of cycle k In the rectangular area, the maintenance energy consumption of all mobile roadside units is:

[0087]

[0088] in, represents the i-th time slot of period k In the rectangular area, the maintenance energy consumption of all mobile roadside units is Represents the mobile roadside unit mr in the i-th time slot Is it in the square area ar?

[0089] S310, according to the calculated period k i-th time slot The maintenance energy consumption of all fixed roadside units in the rectangular area and the calculated time slot of cycle k The maintenance energy consumption of all mobile roadside units in the rectangular area is calculated by calculating the maintenance energy consumption of all fixed roadside units and all mobile roadside units within period k, that is:

[0090]

[0091] Among them, e hold (k) represents the maintenance energy consumption of all fixed roadside units and all mobile roadside units in period k.

[0092] S32. Calculate the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to active mode within period k, that is:

[0093]

[0094] Among them, E sq (k) represents the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to active mode within period k, represents the switching energy consumption of all fixed roadside units, Represents the switching energy consumption of all mobile roadside units.

[0095] S33. The vehicle edge system calculates the data transmission energy consumption of all tasks in the network within calculation cycle k. The specific process is S331-S333:

[0096] S331. Calculate the data transmission energy consumption of vehicle m uploading data to the fixed roadside unit, that is:

[0097]

[0098] in, represents the data transmission energy consumption when vehicle m uploads data to the fixed roadside unit fr, z m,fr represents the task generated by vehicle m in the i-th time slot of cycle k Whether it is unloaded onto a fixed roadside unit fr, P m,fr represents the task generated by vehicle m in the i-th time slot of cycle k Transmission power to the fixed roadside unit fr, R m,fr represents the transmission rate between the vehicle m generating the task and the fixed roadside unit fr, B m,frrepresents the bandwidth between vehicle m and the fixed roadside unit fr, log2(·) represents the logarithmic function, h represents the channel state, N0 represents the noise power, α represents the path loss factor, d m,fr Represents the distance between vehicle m and the fixed roadside unit fr.

[0099] In this embodiment, z m,fr is a binary indicator variable used to indicate the task generated by vehicle m in the i-th time slot of cycle k Whether it is unloaded to the fixed roadside unit fr, if z m,fr =1, it means the task is unloaded to the fixed roadside unit fr. If z m,fr =0, it means that the task has not been unloaded to the fixed roadside unit fr.

[0100] S332. Calculate the data transmission energy consumption of vehicle m when uploading data to the mobile roadside unit, that is:

[0101]

[0102] in, represents the data transmission energy consumption when vehicle m uploads data to mobile roadside unit mr, z m,mr represents the task generated by vehicle m in the i-th time slot of cycle k Whether it is unloaded onto the mobile roadside unit mr, P m,r represents the task generated by vehicle m in the i-th time slot of cycle k Transmission power to the mobile roadside unit mr, R m,mr B represents the transmission rate between the vehicle m generating the task and the mobile roadside unit mr. mmr represents the bandwidth between vehicle m and mobile roadside unit mr, d m,mr Represents the distance between vehicle m and mobile roadside unit mr.

[0103] In this embodiment, z m,mr is a binary indicator variable that indicates the task generated by vehicle m in the i-th time slot of cycle k. Whether it is unloaded onto the mobile roadside unit mr, if z m,mr =1, it means the task is unloaded to the mobile roadside unit mr. If z m,mr =0, it means that the task has not been unloaded to the mobile roadside unit mr.

[0104] S333. Calculate the data transmission energy consumption of all tasks in the vehicle edge system computing network based on the calculated data transmission energy consumption of vehicle m uploading data to the fixed roadside unit and the calculated data transmission energy consumption of vehicle m uploading data to the mobile roadside unit, that is:

[0105]

[0106] Among them, E trans (k) represents the data transmission energy consumption of all tasks in the vehicle edge system computing network within period k.

[0107] S34: Calculate the data computing energy consumption of all tasks in the vehicle edge computing network within cycle k. The specific process is S341-S343:

[0108] S341. The computing energy consumption required for computing the task on a fixed roadside unit is:

[0109]

[0110] in, represents the task generated by vehicle m in the i-th time slot of cycle k The computational energy required for calculation on a fixed roadside unit fr, represents the task generated by vehicle m in the i-th time slot of cycle k Computational power on fixed roadside units fr.

[0111] S342, the computing energy consumption required when computing the task on the mobile roadside unit, namely:

[0112]

[0113] in, represents the task generated by vehicle m in the i-th time slot of cycle k The computational energy required for calculations on the mobile roadside unit mr, represents the task generated by vehicle m in the i-th time slot of cycle k Computational power on mobile roadside units mr.

[0114] S343. Calculate the data computing energy consumption of all tasks in the vehicle edge computing network based on the computing energy consumption required when the computing task is calculated on a fixed roadside unit and the computing energy consumption required when the computing task is calculated on a mobile roadside unit, that is:

[0115]

[0116] Among them, E comp (k) represents the data computing energy consumption of all tasks in the vehicle edge computing network.

[0117] S35. Based on the calculated maintenance energy consumption of all fixed roadside units and all mobile roadside units in period k, the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to active mode in period k, the data transmission energy consumption of all tasks in the vehicle edge system computing network in period k, and the data calculation energy consumption of all tasks in the vehicle edge computing network in period k, the energy consumption model of the vehicle edge computing network is:

[0118] E all (k) = E hols (k)+E sw (k)+E trans (k)+E comp (k)

[0119] Among them, E all (k) represents the total energy consumption of the vehicle edge computing network in period k.

[0120] S4. Based on the reliability model of the vehicle edge computing network established in step S2 and the energy consumption model of the vehicle edge computing network established in step S3, the constraints of the vehicle edge computing network are constructed to obtain a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption.

[0121] Specifically, step S4 includes S41-S47:

[0122] Because in the vehicle edge computing network, each fixed roadside unit f-RSU has three states: active, low power and dormant. In the active state, the fixed roadside unit f-RSU can perform all functions such as access, transmission and calculation. In the low power state, the fixed roadside unit f-RSU cannot perform task calculations and can only perform basic access and data transmission as an access fixed roadside unit. Since computing power consumption is usually greater than transmission power consumption, retaining only the transmission function of the fixed roadside unit f-RSU can greatly reduce its power consumption and maintain a low power state. In the dormant state, the fixed roadside unit f-RSU loses the access function, transmission function and computing function at the same time. This area is equivalent to a coverage gap, so there is no energy consumption. In addition, since frequent state switching will bring about additional large amounts of energy consumption, in each time slot of cycle k In the equation, the state of the fixed roadside unit fr remains unchanged, that is,

[0123] S41. The activation state, low power consumption state, and sleep state of the fixed roadside unit are represented as follows:

[0124]

[0125] Among them, x fr (k) represents the state of the fixed roadside unit fr in cycle k.

[0126] In this embodiment, any fixed roadside unit f-RSU must exist within the study area, so the fixed roadside unit f-RSU binary location indicator variable should meet the following conditions: and

[0127] S42, will Perform range constraints, namely:

[0128]

[0129] in, Indicates whether the fixed roadside unit fr is in the square area ar.

[0130] In this embodiment, for the task to be offloaded A binary indicator variable z that indicates whether it is unloaded to a fixed roadside unit fr m,fr Satisfy the following:

[0131] S43, z m,fr Perform range constraints, namely:

[0132]

[0133] in, represents the i-th time slot of vehicle m in cycle k Generated tasks Whether it is unloaded onto a fixed roadside unit fr.

[0134] In this embodiment, each mobile roadside unit (m-RSU) has two states: active and dormant. In the active state, the m-RSU can perform all functions, including access, transmission, and computing. In the dormant state, the m-RSU loses all access, transmission, and computing functions, degenerating into an ordinary vehicle. Therefore, the two states of the m-RSU can be expressed as follows:

[0135] S44. The activation state and the dormant state of the mobile roadside unit are expressed as:

[0136]

[0137] Among them, y mr (k) represents the state of the mobile roadside unit mr in period k.

[0138] Similarly, in order to avoid the additional large amount of energy consumption caused by frequent switching of mobile roadside units m-RSU, in each time slot of cycle k In the process, the state of the mobile roadside unit mr remains unchanged, that is,

[0139] In this embodiment, for any mobile roadside unit m-RSU, in the same time slot The mobile roadside unit m-RSU binary location indicator variable must exist in the study area and should meet the following two conditions:

[0140] S45, will Perform range constraints, namely:

[0141]

[0142] In this embodiment, for any task to be offloaded A binary indicator variable z that indicates whether it is unloaded to the mobile roadside unit mr m,mr The following conditions are met:

[0143] S46, z m,mr Perform range constraints, namely:

[0144]

[0145] in, represents the i-th time slot of vehicle m in cycle k Generated tasks Whether it is unloaded onto the mobile roadside unit mr.

[0146] S47, based on as well as Obtain a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption, namely:

[0147]

[0148] Among them, max means taking the maximum value, min means taking the minimum value, Γ(k) represents the reliability model of the vehicle edge computing network in period k, Represents an arbitrary symbol, st represents making it satisfy the condition, and C1, C2, C3, C3, C4, C5, and C6 represent the 1st to 6th constraints respectively.

[0149] S5. Based on the dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption obtained in step S4, a multi-objective decision analysis method is used to transform the dual objectives into a single objective, thereby obtaining a single-objective optimization function with the goal of maximizing deployment effect.

[0150] In this embodiment, maximizing reliability and minimizing energy consumption are two conflicting objectives, making it difficult to achieve optimality simultaneously, and it's also impossible to unify the units and value ranges. Therefore, we first use the multi-objective decision analysis method (TOPSIS) to transform these multiple objectives into a single objective, obtaining a single-objective optimization function that maximizes deployment effectiveness. This facilitates the subsequent use of deep reinforcement learning methods to determine the dynamic edge node deployment strategy. The maximizing deployment solution is the solution that maximizes deployment effectiveness.

[0151] Specifically, step S5 includes S51-S55:

[0152] S51、Γ(k) and E all (k) Perform standardization processing to obtain the reliability standardization value and energy consumption standardization value of the vehicle edge computing network, namely:

[0153]

[0154] Where v(k)′ represents the normalized reliability value of the vehicle edge computing network within period k, Γ(k) represents the reliability model of the vehicle edge computing network within period k, and Γ(k) min represents the minimum reliability value of the vehicle edge computing network within period k, Γ(k) max represents the maximum reliability value of the vehicle edge computing network within period k, E all (k)′ represents the normalized energy consumption of the vehicle edge computing network in period k, E all (k) max represents the maximum energy consumption of the vehicle edge computing network within period k, E all (k) min Represents the minimum energy consumption value of the vehicle edge computing network within period k.

[0155] In this embodiment, the purpose of standardization is to solve the problem of different dimensions of each indicator, that is, to unify the indicator dimensions.

[0156] S52. Establish a normalized maximization deployment plan based on the reliability normalization value and the energy consumption normalization value of the vehicle edge computing network, namely:

[0157] Dep(k)=[Γ(k),E all (k)]=[Γ(k)′,E all (k)′]

[0158] Where Dep(k) represents the normalized maximization deployment solution within period k.

[0159] In this embodiment, based on the multiple objectives of maximizing reliability and minimizing energy consumption, the optimal solution of the normalized maximization deployment solution is (1, 0) and the worst solution is (0, 1). Therefore, Dep(k) can be used to calculate the relative distance between the normalized maximization deployment solution and the optimal solution and the worst solution in the two-dimensional objective space.

[0160] S53. Calculate the relative distance between the normalized maximization deployment solution and the optimal solution and the worst solution in the two-dimensional target space, that is:

[0161]

[0162] Among them, Dist + (k) represents the relative distance between the normalized maximization deployment solution and the optimal solution in the two-dimensional target space within period k, Dist - (k) represents the relative distance between the normalized maximized deployment solution and the worst solution in the two-dimensional target space within period k.

[0163] Since the closer the Euclidean distance to the optimal solution and the farther the Euclidean distance to the worst solution in the two-dimensional target space, the better the maximization deployment plan is, the multi-objective function can be transformed into the following single-objective function.

[0164] S54. According to the calculated relative distances between the normalized maximization deployment solution and the optimal solution and the worst solution in the two-dimensional target space, a single objective function is obtained, namely:

[0165]

[0166] Where Obj(k) represents the single objective function within period k.

[0167] In this embodiment, 0≤Obj(k)≤1, and the closer Obj(k) is to 1, the better the deployment effect. In addition, the objective function comprehensively considers reliability and energy consumption, so the objective function represents the deployment effect.

[0168] S55. Based on the single objective function, a single objective optimization function with the goal of maximizing the deployment effect is obtained, namely:

[0169]

[0170] In this embodiment, by Solving this problem yields a dynamic edge node deployment strategy. Due to the large-scale and highly dynamic nature of the Internet of Vehicles, this problem can be solved using deep reinforcement learning (DRL). This deployment problem is modeled as a Markov decision process and solved in step S6 using the asynchronous dominant actor-critic (A3C) method.

[0171] S6. Based on the single-objective optimization function obtained in step S5 with the goal of maximizing the deployment effect, the edge node deployment problem of the single-objective optimization function is modeled as a Markov decision process, and an asynchronous dominant actor-critic method is used for multi-threaded solution to obtain a deployment plan for vehicle edge nodes.

[0172] Specifically, step S6 includes:

[0173] According to the obtained single-objective optimization function with the goal of maximizing the deployment effect, the vehicle is used as an intelligent agent in reinforcement learning to construct the vehicle's state space, workspace and reward function. The asynchronous advantage actor-critic method is used for multi-threaded solution to obtain the deployment plan of the vehicle edge node, that is, the deployment state of the fixed roadside unit and the deployment state y of the mobile roadside unit. mr (k) and the final deployment effect Obj(k).

[0174] Specifically, based on the obtained single-objective optimization function with the goal of maximizing the deployment effect, the vehicle is used as the intelligent agent in reinforcement learning. The specific process of constructing the vehicle's state space, workspace, and reward function is as follows:

[0175] First, construct the state space of the vehicle, namely:

[0176]

[0177] Among them, s(k) represents the state space of the vehicle in period k, D(k) represents the input data volume of the task set in period k, C(k) represents the task computation volume of the task set in period k, τ(k) represents the delay constraint of the task set in period k, w(k) represents the importance of the task set in period k, F(k) represents the computing resources of the roadside unit, and L(k) represents the position of the vehicle set in period k. represents the position of the set of roadside units in period k.

[0178] Among them, the computing resources of the roadside unit are:

[0179] F(k)=[F fRSU (k),F mRSU (k)]

[0180] Among them, F fRSU (k) represents the remaining computing resources of the fixed roadside unit in period k, F mRSU (k) represents the remaining computing resources of the mobile roadside unit in period k.

[0181] In this embodiment, D m Expressed as Among them D m (k) represents the mission of vehicle m The amount of input data in cycle k, Indicates the mission of vehicle 1 The amount of input data in cycle k, Indicates the mission of vehicle 2 The amount of input data in cycle k, Represents the mission of vehicle M The amount of input data in cycle k.

[0182] C m Expressed as Among them C m (k) represents the mission of vehicle m The amount of task computation within cycle k, Indicates the mission of vehicle 1 The amount of task computation within cycle k, Indicates the mission of vehicle 2 The amount of task computation within cycle k, Represents the mission of vehicle M The amount of task computation in cycle k.

[0183] Will w m Expressed as where w m (k) represents the mission of vehicle m Importance within period k, Indicates the mission of vehicle 1 Importance within period k, Indicates the mission of vehicle 2 Importance within period k, Represents the mission of vehicle M Importance within period k.

[0184] τ m Expressed as where τ m represents the mission of vehicle m The delay constraint within period k, Represents the tasks of vehicle 1, vehicle 2, and vehicle M respectively Delay constraint within period k.

[0185] The position of the set of vehicles in period k is expressed as in They represent the positions of vehicle 1, vehicle 2, and vehicle M at cycle k respectively.

[0186] The position of the set of mobile roadside units in period k is expressed as in They represent the positions of mobile roadside unit 1, mobile roadside unit 2, and mobile roadside unit MR at cycle k respectively.

[0187] Secondly, based on the vehicle’s state space, the vehicle’s action space is constructed, namely:

[0188]

[0189] Among them, a(k) represents the action space of the vehicle in period k, a fr (k) represents the action of the fixed roadside unit in period k, a mr (k) represents the action of the mobile roadside unit in period k, x1(k) represents the state of the fixed roadside unit 1 in period k, x2(k) represents the state of the fixed roadside unit 2 in period k, and x FR (k) represents the state of the fixed roadside unit FR in period k, y1(k) represents the state of the mobile roadside unit 1 in period k, y2(k) represents the state of the mobile roadside unit 2 in period k, and y MR (k) represents the state of the mobile roadside unit MR in period k.

[0190] Finally, construct the vehicle's reward function, namely:

[0191] R(k)=Obj(k)

[0192] Where R(k) represents the reward function of the vehicle in cycle k.

[0193] Specifically, the asynchronous advantage actor-critic method is used for multi-threaded solution to obtain the deployment scheme of the vehicle edge nodes, that is, the deployment state of the fixed roadside unit and the deployment state of the mobile roadside unit y mr (k) and the specific process of the final deployment effect Obj(k) are as follows:

[0194] First, initialize the global variables θ, θ of the global network v and the interactive upload time k in the thread network max .

[0195] Secondly, open multiple threads and reset the parameter gradient dθ, dθ for each thread network v , and the global variables θ, θ in the global network v Synchronize to the current thread network for update, obtain the network parameters of the current thread network, and count the cycles of the thread network, that is:

[0196]

[0197] Among them, θ′, θ v ′ represents the updated network parameters of the current thread network.

[0198] By observing the state space s(k) of the vehicle edge computing network, for each cycle k, the policy function π(a(k)|s(k);θ′) is obtained. Then, the corresponding action space a(k) is selected according to the policy function, and the reward function R(k) is calculated at the same time to obtain the next state space s(k+1), where the intermediate s(k+1) represents the state space of the vehicle in cycle k+1.

[0199] Finally, the global variables of the global network are updated using the next state space s(k+1), namely:

[0200]

[0201] Among them, dθ″, dθ v ″ represents the updated parameter gradient.

[0202] Until the thread network reaches the preset interaction upload time, the optimal network parameters of the vehicle edge nodes uploaded by the thread network to the global network and the deployment plan of the vehicle edge nodes are obtained, that is, the deployment state of the fixed roadside unit and the deployment state of the mobile roadside unit y mr (k) and the final deployment effect Obj(k).

[0203] In summary, the present invention proposes a method for dynamic adjustment of edge nodes in the Internet of Vehicles based on reinforcement learning. First, the dynamic deployment problem of roadside units in the Internet of Vehicles is modeled. By taking transmission probability, task effectiveness, task importance and required delay as reliability evaluation factors of the established vehicle edge computing network, a reliability evaluation mechanism of the vehicle edge network is constructed, thereby improving the reliability of the vehicle edge computing network. Secondly, by dynamically adjusting the activation-low power consumption-sleep state of fixed roadside units and the activation-sleep state of mobile roadside units, the time-varying network resource requirements in the Internet of Vehicles are met, and the target optimization of high reliability and low energy consumption is achieved. Finally, the asynchronous advantage actor-critic method is used for multi-threaded training, and the delay of edge node deployment in the vehicle edge network is shortened by sharing training parameters.

[0204] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0205] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A method for dynamic adjustment of edge nodes in Internet of Vehicles based on reinforcement learning, characterized in that: The following steps are involved: S1. Abstract the study area into a rectangular area, introduce buses as mobile roadside units, establish a vehicle edge computing network, and obtain the tasks generated by the vehicles in the vehicle edge computing network as well as the input data, task computing amount, delay constraints, and importance of the tasks; S2. Based on the vehicle edge computing network established in step S1, analyze the reliability of the vehicle edge computing network and establish a reliability model of the vehicle edge computing network; Wherein, step S2 specifically includes: S21. According to the tasks generated by the vehicle in the vehicle edge computing network, the failure probabilities of the three processes of data upload, data transmission and result calculation in task completion are calculated, namely: in, In the data uploading phase, the task generated by vehicle m in the i-th time slot of cycle k is Uploaded to the fixed roadside unit i ′ The failure probability, represents the task generated by vehicle m in the i-th time slot of cycle k during the data transmission phase From access to fixed roadside units ′ The failure probability of transmission to the target fixed roadside unit n, In the result calculation phase, the task generated by vehicle m in the i-th time slot of cycle k is The failure probability calculated on the target fixed roadside unit n, e represents the exponential function, λ1, λ2, λ3 represent the failure rates of data upload, data transmission and result calculation, respectively. represents the task generated by vehicle m in the i-th time slot of cycle k Upload to fixed roadside unit ′ Upload delay, represents a fixed roadside unit i ′ The task generated by vehicle m in the i-th time slot of cycle k The transmission delay to the target fixed roadside unit n, represents the task generated by vehicle m in the i-th time slot of cycle k The computation delay at the target fixed roadside unit n; S22. Calculate the probability of successful completion of the task based on the failure probabilities of the three processes of data upload, data transmission and result calculation during task completion, that is: Among them, S m,i′,n represents the task generated by vehicle m in the i-th time slot of cycle k Connected to fixed roadside unit i ′ Transmitted to the target fixed roadside unit n to calculate the probability of successful completion; S23, introduce binary variables to determine whether the task meets the delay limit, that is: Among them, δ m represents the task generated by vehicle m in the i-th time slot of cycle k Whether the delay limit is met, if indicates the judgment condition, T m,i′,n Indicates that by accessing a fixed roadside unit i ′ The task generated by vehicle m in the i-th time slot of cycle k The delay of unloading to the target fixed roadside unit n, τ m represents the task generated by vehicle m in the i-th time slot of cycle k The specified latency constraints; S24. Calculate the standardized delay using the following formula: Among them, m,i′,n represents the task generated by vehicle m in the i-th time slot of cycle k Normalized delay of transmission from access fixed roadside unit i′ to target fixed roadside unit n; S25. Obtain the impact of description importance on the reliability of the vehicle edge computing network, namely: f(w m )=(d-w m ) a′ Where f(·) represents the impact of description importance on the reliability of the vehicle edge computing network, d represents the corrected importance ranking, which is a constant, and w m represents the task generated by vehicle m in the i-th time slot of cycle k The importance of ′ Indicates the effect size of the important reaction task, which is a constant; S26. Based on the probability of successful completion of the task, whether the task meets the delay constraint, the standardized delay, and the impact of the description importance on the reliability of the vehicle edge computing network, a reliability model of the vehicle edge computing network is established, namely: Among them, Γ represents the reliability model of the vehicle edge computing network, and M represents the total number of vehicles; S3. Based on the vehicle edge computing network established in step S1, analyze the energy consumption of the vehicle edge computing network and establish an energy consumption model of the vehicle edge computing network; Wherein, step S3 specifically includes: S31, calculating the maintenance energy consumption of all fixed roadside units and all mobile roadside units within period k, the specific process is: S3101, when the fixed roadside unit is in an activated state, calculate the i-th time slot of the cycle k in the square area ar The maintenance energy consumption of the fixed roadside unit is: in, represents the i-th time slot of cycle k in the square area ar when the fixed roadside unit is in the activated state The maintenance energy consumption of the fixed roadside unit fr, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain the fixed roadside unit fr, represents the computing resources that the fixed roadside unit fr can provide. Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain fixed roadside unit fr transmission, represents the transmission rate of the fixed roadside unit fr; S3102: When the fixed roadside unit is in a low power consumption state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the fixed roadside unit is: in, Indicates that when the fixed roadside unit is in a low power state, the i-th time slot of the cycle k in the square area ar The maintenance energy consumption of the fixed roadside unit fr; S3103: When the fixed roadside unit is in a dormant state, calculate the i-th time slot of the cycle k in the square area ar. The maintenance energy consumption of the fixed roadside unit is: in, Indicates that when the fixed roadside unit is in sleep state, in the i-th time slot of cycle k in the square area ar The maintenance energy consumption of the fixed roadside unit fr; S3104, calculate the i-th time slot of period k in the square area ar. The total maintenance energy consumption of the fixed roadside unit is: in, represents the i-th time slot of period k in the square area ar The total maintenance energy consumption of the fixed roadside unit fr, Indicates that the fixed roadside unit fr is in the i-th time slot In the state, active, halfactive, and sleep respectively represent the active state, low power state, and sleep state; S3105, calculate the i-th time slot of cycle k In the figure, the maintenance energy consumption of all fixed roadside units in the rectangular area is: in, represents the i-th time slot of period k The maintenance energy consumption of all fixed roadside units in the rectangular area is Indicates that the fixed roadside unit fr is in the i-th time slot Whether it is in the square area ar, AR represents the set of square areas, and FR represents the set of fixed roadside units; S3106: When the mobile roadside unit is in an activated state, calculate the i-th time slot of the period k in the square area ar. The maintenance energy consumption of the mobile roadside unit is: in, represents the i-th time slot of cycle k in the square area ar when the mobile roadside unit is in the activated state The maintenance energy consumption of the mobile roadside unit mr, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain the mobile roadside unit mr calculation, represents the computing resources that the mobile roadside unit mr can provide, Indicates the i-th time slot The proportional coefficient of energy consumption required to maintain the transmission of mobile roadside unit mr, represents the transmission rate of the mobile roadside unit mr; S3107: When the mobile roadside unit is in a dormant state, calculate the i-th time slot of the cycle k in the square area ar. The maintenance energy consumption of the mobile roadside unit is: in, Indicates that when the mobile roadside unit is in sleep state, in the i-th time slot of cycle k in the square area ar The maintenance energy consumption of the mobile roadside unit mr; S3108, calculate the i-th time slot of period k in the square area ar. The total maintenance energy consumption of the mobile roadside unit is: in, represents the i-th time slot of period k in the square area ar The total maintenance energy consumption of the mobile roadside unit mr, Represents the mobile roadside unit mr in the i-th time slot Status in S3109, calculate the i-th time slot of cycle k In the figure, the energy consumption of all mobile roadside units in the rectangular area is: in, represents the i-th time slot of period k In the figure, the maintenance energy consumption of all mobile roadside units in the rectangular area, MR represents the set of mobile roadside units, Represents the mobile roadside unit mr in the i-th time slot Whether it is in the square area ar; S310, according to the calculated i-th time slot of cycle k The maintenance energy consumption of all fixed roadside units in the rectangular area and the calculated time slot of cycle k The maintenance energy consumption of all mobile roadside units in the rectangular area is calculated by calculating the maintenance energy consumption of all fixed roadside units and all mobile roadside units within period k, that is: Among them, E hold (k) represents the maintenance energy consumption of all fixed roadside units and all mobile roadside units in period k, T k Indicates that the period k is discretized into a set of several time slots; S32, calculating the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to active mode within cycle k, that is: Among them, E sw (k) represents the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to active mode in period k, represents the switching energy consumption of all fixed roadside units, represents the switching energy consumption of all mobile roadside units; S33, the vehicle edge system calculates the data transmission energy consumption of all tasks in the network within the calculation cycle k, and the specific process is as follows: S331, calculating the data transmission energy consumption of vehicle m uploading data to the fixed roadside unit, that is: in, represents the data transmission energy consumption when vehicle m uploads data to the fixed roadside unit fr, z m,fr represents the task generated by vehicle m in the i-th time slot of cycle k Whether it is unloaded to a fixed roadside unit fr, D m represents the mission of vehicle m The amount of input data, P m,fr represents the task generated by vehicle m in the i-th time slot of cycle k The transmission power to the fixed roadside unit fr, R m,fr represents the transmission rate from the vehicle m generating the task to the fixed roadside unit fr, B m,fr represents the bandwidth between vehicle m and the fixed roadside unit fr, log2(·) represents the logarithmic function, h represents the channel state, N0 represents the noise power, α represents the path loss factor, and d m,fr represents the distance between vehicle m and the fixed roadside unit fr; S332, calculating the data transmission energy consumption of vehicle m uploading data to the mobile roadside unit, that is: in, represents the data transmission energy consumption when vehicle m uploads data to mobile roadside unit mr, z m,mr represents the task generated by vehicle m in the i-th time slot of cycle k Whether it is unloaded onto the mobile roadside unit mr, P m,mr represents the task generated by vehicle m in the i-th time slot of cycle k Transmission power to the mobile roadside unit mr, R m,mr represents the transmission rate from the vehicle m generating the task to the mobile roadside unit mr, B mmr represents the bandwidth between vehicle m and mobile roadside unit mr, d m,mr represents the distance between vehicle m and mobile roadside unit mr; S333, according to the calculated data transmission energy consumption of vehicle m uploading data to the fixed roadside unit and the calculated data transmission energy consumption of vehicle m uploading data to the mobile roadside unit, calculate the data transmission energy consumption of all tasks in the vehicle edge system computing network, that is: Among them, E trans (k) represents the data transmission energy consumption of all tasks in the vehicle edge system computing network within period k, and M1 represents the set of vehicles; S34, calculate the data computing energy consumption of all tasks in the vehicle edge computing network within cycle k, the specific process is: S341, the computing energy consumption required when computing the task on a fixed roadside unit, namely: in, represents the task generated by vehicle m in the i-th time slot of cycle k The computational energy required for computing on a fixed roadside unit fr, C m represents the mission of vehicle m The amount of computational tasks, represents the task generated by vehicle m in the i-th time slot of cycle k Computational power on fixed roadside units fr; S342, the computing energy consumption required when computing the task on the mobile roadside unit, that is: in, represents the task generated by vehicle m in the i-th time slot of cycle k The computational energy required for calculations on the mobile roadside unit mr, represents the task generated by vehicle m in the i-th time slot of cycle k Computational power on the mobile roadside unit mr; S343. Calculate the data computing energy consumption of all tasks in the vehicle edge computing network according to the computing energy consumption required when the computing task is calculated on the fixed roadside unit and the computing energy consumption required when the computing task is calculated on the mobile roadside unit, that is: Among them, E comp (k) represents the data computing energy consumption of all tasks in the vehicle edge computing network; S35. According to the calculated maintenance energy consumption of all fixed roadside units and all mobile roadside units within period k, the energy consumption of all fixed roadside units and all mobile roadside units when switching from sleep mode to activation mode within period k, the data transmission energy consumption of all tasks in the vehicle edge system computing network within period k, and the data calculation energy consumption of all tasks in the vehicle edge computing network within period k, the energy consumption model of the vehicle edge computing network is: E all (k)=E hold (k)+E sw (k)+E trans (k)+E comp (k) Among them, E all (k) represents the total energy consumption of the vehicle edge computing network in period k; S4. According to the reliability model of the vehicle edge computing network established in step S2 and the energy consumption model of the vehicle edge computing network established in step S3, the constraint conditions of the vehicle edge computing network are constructed to obtain a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption; Wherein, step S4 specifically includes: S41, the activation state, low power consumption state and sleep state of the fixed roadside unit are represented as: Among them, x fr (k) represents the state of the fixed roadside unit fr in period k; S42, will Perform range constraints, namely: in, Indicates whether the fixed roadside unit fr is in the square area ar; S43, z m,fr Perform range constraints, namely: in, represents the i-th time slot of vehicle m in cycle k The tasks generated Whether it is unloaded onto a fixed roadside unit fr; S44, the activation state and the dormant state of the mobile roadside unit are represented as: Among them, y mr (k) represents the state of the mobile roadside unit mr in period k; S45, Perform range constraints, namely: S46, z m,mr Perform range constraints, namely: in, represents the i-th time slot of vehicle m in cycle k The tasks generated Whether it is unloaded onto the mobile roadside unit mr; S47, based on as well as Obtain a dual-objective optimization function with the goals of maximizing reliability and minimizing energy consumption, namely: Among them, max means taking the maximum value, min means taking the minimum value, Γ(k) represents the reliability model of the vehicle edge computing network in period k, represents any symbol, st represents making it satisfy the condition, C1, C2, C3, C3, C4, C5, C6 represent the 1st to 6th constraints respectively; S5. According to the dual-objective optimization function with the objectives of maximizing reliability and minimizing energy consumption obtained in step S4, a multi-objective decision analysis method is used to transform the dual objectives into a single objective, thereby obtaining a single-objective optimization function with the objective of maximizing deployment effect; Wherein, step S5 specifically includes: S51. For Γ(k) and E all (k) Perform standardization processing to obtain the reliability standardization value of the vehicle edge computing network and the energy consumption standardization value of the vehicle edge computing network, namely: Where Γ(k) ′ represents the normalized reliability value of the vehicle edge computing network within period k, Γ(k) min represents the minimum reliability value of the vehicle edge computing network within period k, Γ(k) max represents the maximum reliability value of the vehicle edge computing network within period k, E all (k) ′ represents the normalized value of energy consumption of the vehicle edge computing network in period k, E all (k) max represents the maximum energy consumption of the vehicle edge computing network within period k, E all (k) min represents the minimum energy consumption value of the vehicle edge computing network within period k; S52. According to the standardized value of the reliability of the vehicle edge computing network and the standardized value of the energy consumption of the vehicle edge computing network, a normalized maximization deployment scheme is established, namely: Dep(k)=[Γ(k),E all (k)]=[Γ(k)′,E all (k)′] Where Dep(k) represents the normalized maximization deployment solution within period k; S53, calculate the relative distance between the normalized maximization deployment scheme and the optimal solution and the worst solution in the two-dimensional target space, that is: Among them, Dist + (k) represents the relative distance between the normalized maximization deployment solution and the optimal solution in the two-dimensional target space within period k, Dist _ (k) represents the relative distance between the normalized maximization deployment solution and the worst solution in the two-dimensional target space within period k; S54, according to the calculated relative distances of the normalized maximization deployment scheme with the optimal solution and the worst solution in the two-dimensional target space, a single objective function is obtained, namely: Where Obj(k) represents the single objective function within period k; S55. According to the single objective function, a single objective optimization function with the goal of maximizing the deployment effect is obtained, that is: S6. According to the single-objective optimization function obtained in step S5 with the goal of maximizing the deployment effect, the edge node deployment problem of the single-objective optimization function is modeled as a Markov decision process, and the asynchronous dominant actor-critic method is used for multi-threaded solving to obtain a deployment plan for the vehicle edge nodes.

2. The method for dynamic adjustment of edge nodes of Internet of Vehicles based on reinforcement learning according to claim 1 is characterized in that: Step S1 specifically includes: S11. The study area is regarded as a rectangular area, black is used to indicate the existence of roadside unit coverage, white is used to indicate the absence of roadside unit coverage, the roadside units in black are regarded as fixed roadside units, and buses are regarded as mobile roadside units to dynamically fill the gaps in roadside unit coverage and establish a vehicle edge computing network; S12. If the radius of the fixed roadside unit is r, then is the side length, divide the rectangular area into several The set of square areas is AR = {1,2,…,ar,…,N AR }, and fix the fixed roadside unit at the center of each square area, and the set of fixed roadside units is FR = {1,2,…,fr,…,N FR }, where ar, N AR Respectively represent the arth and Nth of the set of square areas AR AR square area, fr, N FR Respectively represent the fr, Nth and FR Fixed roadside units; S13, the set of mobile roadside units is represented as MR = {1, 2, ..., mr, ..., N MR }, since the computing resources provided by the mobile roadside unit in each square area are dynamically changing, the computing resources provided by the mobile roadside unit in the square area ar are Among them, mr, N MR Respectively represent the mrth and Nth MR Mobile roadside units, represents the computing resources provided by the mobile roadside unit mr in the square area ar at the tth time, It indicates whether the mobile roadside unit mr at the tth moment is within the square area ar. It represents the computing resources that the mobile roadside unit mr can provide; S14, the set of vehicles is M1 = {1, 2, ..., m, ..., M}. If each vehicle can only generate one task in the same time slot, in the vehicle edge computing network model, the time is divided into multiple operation cycles K1 = {1, 2, ..., k, ..., k}, then the cycle k is discretized into a set of several time slots: Time Slot The task set generated by the vehicle is Where M represents the total number of vehicles, K represents the total number of cycles, and I represents the total number of time slots in cycle k. denote the 1st, 2nd, i and I time slots of cycle k respectively, They represent the tasks generated by vehicle 1, vehicle 2, vehicle m, and vehicle M in the i-th time slot of cycle k respectively; S15, It is represented by a four-tuple Where D m represents the mission of vehicle m The amount of input data, C m represents the mission of vehicle m The task computation amount, τ m represents the mission of vehicle m The delay constraint, w m represents the mission of vehicle m The importance of.

3. The method for dynamic adjustment of edge nodes of Internet of Vehicles based on reinforcement learning according to claim 2 is characterized in that: Step S6 specifically includes: According to the obtained single-objective optimization function with the goal of maximizing the deployment effect, the vehicle is used as an intelligent agent in reinforcement learning to construct the vehicle's state space, workspace and reward function. The asynchronous advantage actor-critic method is used for multi-threaded solution to obtain the deployment plan of the vehicle edge nodes, that is, the deployment state of the fixed roadside unit and the deployment state y of the mobile roadside unit with decision variables mr (k) and the final deployment effect Obj(k).

4. The method for dynamic adjustment of edge nodes of Internet of Vehicles based on reinforcement learning according to claim 3 is characterized in that: According to the obtained single-objective optimization function with the goal of maximizing the deployment effect, the vehicle is used as an intelligent agent in reinforcement learning. The specific process of constructing the vehicle's state space, workspace, and reward function is as follows: First, construct the state space of the vehicle, namely: Among them, s(k) represents the state space of the vehicle in period k, D(k) represents the amount of input data of the task set in period k, C(k) represents the task calculation amount of the task set in period k, τ(k) represents the delay constraint of the task set in period k, w(k) represents the importance of the task set in period k, F(k) represents the computing resources of the roadside unit, L(k) represents the position of the vehicle set in period k, represents the position of the set of mobile roadside units in period k; Among them, the computing resources of the roadside unit are: F(k)=[F fRSU (k),F mRSU (k)] Among them, F fRSU (k) represents the remaining computing resources of the fixed roadside unit in period k, F mRSU (k) represents the remaining computing resources of the mobile roadside unit in period k; Secondly, based on the vehicle's state space, the vehicle's action space is constructed, namely: Among them, a(k) represents the action space of the vehicle in period k, a fr (k) represents the action of the fixed roadside unit in period k, a mr (k) represents the action of the mobile roadside unit in period k, x1(k) represents the state of the fixed roadside unit 1 in period k, x2(k) represents the state of the fixed roadside unit 2 in period k, and x FR (k) represents the state of the fixed roadside unit FR in period k, y1(k) represents the state of the mobile roadside unit 1 in period k, y2(k) represents the state of the mobile roadside unit 2 in period k, and y MR (k) represents the state of the mobile roadside unit MR in period k; Finally, construct the vehicle's reward function, namely: R(k)=Obj(k) Among them, R(k) represents the reward function of the vehicle in cycle k.

5. The method for dynamic adjustment of edge nodes of Internet of Vehicles based on reinforcement learning according to claim 4 is characterized in that: The asynchronous advantage actor-critic method is used for multi-threaded solution to obtain the deployment scheme of the vehicle edge nodes, that is, the deployment state of the fixed roadside unit and the deployment state of the mobile roadside unit y mr (k) and the specific process of the final deployment effect Obj(k) is as follows: First, initialize the global variables θ, θ of the global network v and the interactive upload time k in the thread network max ; Secondly, start multiple threads and reset the parameter gradient dθ, dθ for each thread network v , and the global variables θ, θ in the global network v Synchronize to the current thread network for update, obtain the network parameters of the current thread network, and count the cycles of the thread network, that is: Among them, θ ′ ,θ v ′ Represents the updated network parameters of the current thread network; By observing the state space s(k) of the vehicle edge computing network, for each cycle k, the strategy function π(a(k)|s(k);θ ′ ), then select the corresponding action space a(k) according to the policy function, and calculate the reward function R(k) to get the next state space s(k+1), where s(k+1) represents the state space of the vehicle in period k+1; Finally, the next state space s(k+1) is used to update the global variables of the global network, namely: Among them, dθ″, dθ v ″ represents the updated parameter gradient; Until the thread network reaches the preset interactive upload time, the optimal network parameters of the vehicle edge nodes uploaded by the thread network to the global network and the deployment scheme of the vehicle edge nodes are obtained, that is, the deployment state of the fixed roadside unit and the deployment state of the mobile roadside unit y mr (k) and the final deployment effect Obj(k).

Images