Vehicle networking service offloading migration and load balancing method, and system using the method

Abstract

The present invention proposes a method for offloading, migrating, and load balancing services in an Internet of Vehicles (IoV) network, and a system utilizing the method. The method comprises: a vehicle (VU) obtaining a task and offloading some computationally intensive tasks to an MEC server connected to the VU; obtaining the communication delay from the VU offloading the task to the MEC, the local computational delay of the computationally intensive task placed on the VU for data processing, and the computational delay of the offloaded task on the MEC server; constructing an objective function; solving the objective function, obtaining the task offloading ratio, and determining whether to migrate the current computational task, thereby controlling the execution of the VU and MEC servers. The present invention takes into account system latency optimization under conditions of time-varying channels and random task arrival, ensuring the completion rate of vehicle tasks within the task tolerance delay while optimizing the task migration and load balancing algorithms between the MEC servers providing services, thereby meeting the requirements of low latency for completing computationally intensive tasks and alleviating excessive computing resource load.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle networking, and specifically relates to a vehicle networking service offloading migration and load balancing method, and a system utilizing the method. Background Art

[0002] With the development of the Internet of Things (IoT), complex applications such as autonomous driving, VR (Virtual Reality) / AR (Augmented Reality), and image recognition are emerging one after another. This has also brought with it a large number of time-sensitive and computationally intensive tasks, requiring significant computing resources and high QoS (Quality of Service). Therefore, mobile edge computing (MEC) in vehicles has gradually matured and emerged on the technological stage.

[0003] MEC is a method for efficiently processing the explosive computing tasks of VUs (Vehicle Units). In the MEC system, each VU offloads the task to the edge computing server, so that the local vehicle only performs the necessary computing content, allowing the VU's large number of computing-intensive tasks to be processed more quickly.

[0004] High-speed mobility of VUs is one of the key challenges of MEC systems. Because users move across the coverage area of ​​MEC servers, when users leave the coverage area of ​​the current MEC server, the system needs to migrate tasks from the current MEC server to the target MEC server. Since the number of task migrations in a high-mobility environment is greater than that in a low-mobility environment, if the vehicle's tasks are unequally offloaded and migrated to specific MEC servers, performance will degrade in terms of time delay and service quality. Therefore, the selection of the target MEC server has a significant impact on the performance of the MEC system. In addition, when the computational load of the MEC server in the current area is too large, migrating the task to other MEC servers and returning it to the current VU after the computation is completed can also improve the task completion rate within the delay constraint. In order to meet the QoS requirements of the computing tasks and improve the throughput of the system, load balancing is also required between the vehicle edge computing servers.

[0005] The high mobility of vehicles across multiple MEC servers and the increase in the number of vehicles will lead to load imbalance among MEC servers, thereby degrading QoS and causing large latency.

[0006] In view of the high-speed mobility of vehicle units and the high QoS problem that needs to be solved by high-complexity tasks, there is a task migration algorithm based on Q-Learning and DQN to minimize the migration and communication costs respectively.

[0007] In the existing technology, MEC system optimization solutions based on distributed DRL rarely consider the randomness of task arrival and the time-varying nature of the channel. In addition, DRL methods such as DQN and Q-Learning have inevitable overestimation problems and perform poorly in large state-action spaces. The use of policy gradient methods will lead to higher estimation result variance, making the policy more sensitive and even causing training failure. Moreover, these methods do not adopt a delayed update strategy, resulting in fluctuations in network training results. Summary of the Invention

[0008] The present invention aims to solve the technical problems existing in the prior art and provides a method for offloading, migrating and load balancing of Internet of Vehicles services, and a system utilizing the method.

[0009] To achieve the above-mentioned object of the present invention, according to a first aspect of the present invention, the present invention provides a method for offloading, migrating and load balancing of Internet of Vehicles services, which comprises the following steps:

[0010] S1, the nth VU obtains the task Task n ={λ n ,C n ,T tolerance},λ n is the task data volume of the VU, C n Calculate the total number of CPU cycles required for this task for this VU, T tolerance is the tolerated delay of the task, n∈N, N is the total number of vehicles in the Internet of Vehicles, and N is a positive integer;

[0011] S2: Classify the task types into time-sensitive tasks and computation-intensive tasks. If the task is time-sensitive, it is placed on the VU for data processing; if the task is computation-intensive, part of the computation-intensive task is offloaded to the MEC server connected to the VU for calculation;

[0012] S3: Get the communication delay from the VU offload task to the MEC The latency of local computation for computationally intensive tasks placed on the VU for data processing And the computational delay of the offloaded task on the MEC server

[0013] S4: Determine the computational delay of offloaded tasks on the MEC server. If the QoS requirements are not met, the tasks that do not meet the QoS requirements will be migrated between MEC servers to achieve load balancing and obtain the migration delay. Migration costs And the load balancing factor LBF; if the computing delay of all offloaded tasks on the MEC server All meet QoS requirements, reducing migration delay Migration costs And calculate the load balancing factor LBF; S5, construct the objective function:

[0014]

[0015] St T <T tolerance ,x n ∈[0,1], (if the delay exceeds the tolerance, the QoS is not met)

[0016]

[0017] Among them, ω represents the number of times tasks are migrated between different MEC servers, λ l , w are custom weight parameters, w∈[0,1], x n is the unloading ratio;

[0018] S6, solves the objective function, obtains the task offloading ratio, and whether to migrate the current computing task, and uses this to control the execution of the VU and MEC servers.

[0019] Based on the characteristics of compute-intensive tasks, this paper considers the state of computing resources, introduces a load balancing factor, proposes task migration costs, and constructs a model for partial computation offloading, task migration, and load balancing. This approach optimizes system latency under conditions of time-varying channels and random task arrival. This approach not only ensures the completion rate of vehicle tasks within the task tolerance delay, but also optimizes the task migration and load balancing algorithms between the MEC servers providing the service, meeting the requirements for low latency in completing compute-intensive tasks and alleviating excessive computing resource load.

[0020] According to a preferred embodiment of the present invention, the objective function is solved using the DRL-MATD3 algorithm, and the specific steps are as follows:

[0021] S11, initialize three common networks for each base station agent i, including two critic networks and an actor network Three ordinary networks with random network parameters θ i,1 ,θ i,2 and

[0022] S12, for each agent i Initialize three target networks, including two target_critic network parameters Q′ θi,1 ,Q′ θi,2 and a target_actor network parameter Copy the three initialized common network parameters to the target network, i.e. θ′ i,1 =θ i,1 ,θ′ i,2 =θ i,2 , And initialize the playback buffer Buf;

[0023] S13, initialize a random noise ξ for detecting action, and receive the vehicle initial state s=(s l ,s2…,s N ), where s n ={Task n ,h n ,l},

[0024] Task n ={λ n ,C n ,T tolerance} represents the mission information of vehicle n, h n The current channel condition between vehicle n and the base station, l represents the computational load of the MEC server within the current BS service range;

[0025] S14, set the maximum number of training times T1, t1 is the training number sequence number, T1 is a positive integer, let t1 = 1, and perform the following operations:

[0026] For each agent i , choose a random action a j ~μ ii (s j )+ξ i ,ξ i is noise, j∈(1,N) represents the VU number,

[0027] Explore the neural network, determine the current strategy, and execute the action space a=(a1,…,a N ), where a n ={x n ;q n},x n Indicates the uninstall ratio, q n It represents the migration decision of the task offloaded to the MEC server. Solving the action space is to solve the next offloading decision and task migration decision.

[0028] After executing the next action, observe the reward r of the current strategy and the new state s′,

[0029] r=-(λ l T+wC migrate +(1-w)LBF)

[0030] The experience tuple (s, a, r, s′) obtained by executing the current exploration is stored in the playback buffer Buf, and the state s = s′ is updated;

[0031] For each agent i Perform the following operations: When the number of tuples in Buf reaches a certain value, randomly sample N samples (s j ,a j ,r j ,s ′j ), set the target value y j =r+γ*target_q_min, γ is the discount rate, which is a weight parameter set by humans and calculated based on the target value y j θ for the critic network i,n The parameters are updated; in which, when calculating the target value y j By using two critic networks to estimate the Q value, a double network is implemented, and a relatively small Q value, target_q_min, is selected as the target value y j Parameters;

[0032] When the number of updates of the critic network reaches a certain parameter value, the parameters of the actor network and the target_critic and target_actor networks are updated once;

[0033] S15, let t1 = t1 + 1 and execute step S14 again until all training rounds are completed to obtain the final trained network.

[0034] This patent utilizes the MATD3 algorithm, which, compared to DRL algorithms like DDPG and DQN, boasts dual Q-value networks and delayed updates, enabling more stable and accurate network training. In MATD3, we use two networks to estimate Q-values, selecting a relatively small value as the target for network updates. This overcomes the common DRL algorithm problem of the critic network overestimating the Q-values ​​of actions. Furthermore, by delaying updates to the actor network, we make actor network training more stable.

[0035] To achieve the above-mentioned object of the present invention, according to a second aspect of the present invention, a distributed Internet of Vehicles service offloading, migration, and load balancing system is provided, which includes M base stations arranged along a one-way lane, each base station is equipped with K antennas, and each base station is connected to an MEC server. There are N moving vehicles along the road, each vehicle carries a single antenna with limited computing resources, and K, M, and N are all positive integers and K>N;

[0036] The vehicle and the MEC server obtain the task offloading ratio and whether to migrate the current computing task according to the method of the present invention, and execute it accordingly.

[0037] The distributed Internet of Vehicles service offloading migration and load balancing system of the present invention takes into account the system delay optimization under the conditions of channel time variation and random task arrival, which not only ensures the completion rate of vehicle tasks within the task tolerance delay, but also optimizes the task migration and load balancing between the MEC servers providing services.

[0038] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0040] Figure 1 This is a structural diagram of a distributed Internet of Vehicles service offloading migration and load balancing system in a preferred embodiment of the present invention;

[0041] Figure 2 is a flow chart of an objective function solving algorithm in a preferred embodiment of the present invention;

[0042] Figure 3 This is a neural network structure diagram in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0044] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "vertical", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0045] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0046] In a vehicle network MEC system, Figure 1 As shown in the figure, multiple 5G macro base stations are deployed along a one-way lane. Each base station (BS) is equipped with K antennas and is connected to an MEC server. There are N moving vehicles along the road, each carrying a single antenna with limited computing resources. In this system, we consider K>N.

[0047] The present invention provides a method for offloading, migrating and load balancing of Internet of Vehicles services, which includes the following steps:

[0048] S1, the nth VU obtains the task Task n ={λ n ,C n ,T tolerance},λ n is the task data volume of the VU, C n Calculate the total number of CPU cycles required for this task for this VU, T tolerance is the tolerated delay of the task, n∈N, N is the total number of vehicles in the Internet of Vehicles, and N is a positive integer;

[0049] S2: Classify the task types into time-sensitive tasks and computation-intensive tasks. If the task is time-sensitive, it is placed on the VU for data processing; if the task is computation-intensive, part of the computation-intensive task is offloaded to the MEC server connected to the VU for calculation;

[0050] S3: Get the communication delay from the VU offload task to the MEC The latency of local computation for computationally intensive tasks placed on the VU for data processing And the computational delay of the offloaded task on the MEC server

[0051] S4: Determine the computational delay of offloaded tasks on the MEC server. Does not meet QoS requirements (T exceeds the tolerance delay T tolerance That is, it does not meet QoS requirements). Tasks that do not meet QoS requirements are migrated between MEC servers to achieve load balancing and obtain migration delay. Migration costs And the load balancing factor LBF; if the computing delay of all offloaded tasks on the MEC server All meet QoS requirements, reducing migration delay Migration costs And calculate the load balancing factor LBF;

[0052] Tasks (issued by onboard systems, such as collaborative systems or navigation systems) arrive at each VU in a random, independent, and identically distributed manner. Task types are categorized as time-sensitive or compute-intensive, with each VU assigned to either local execution services or offloaded services. For time-sensitive tasks (those with strict processing time requirements, requiring rapid completion and return of results, and thus receiving higher priority, such as real-time vehicle trajectory prediction), the long uplink and downlink data transmission latency and cloud data processing latency make these tasks unable to meet the time-sensitive requirements. Therefore, these tasks are processed locally in the VU. For compute-intensive tasks (those requiring computations significantly more than memory access, communication, and I / O, requiring extensive computation and consuming CPU resources, such as solving complex computational problems or high-definition video decoding), since these tasks require extensive data computation but are not time-sensitive, some can be offloaded to the MEC, alleviating and freeing up some of the VU's local computing power. Channels are constantly changing due to interference between MIMO-NOMA channels and time-varying path loss caused by task migration between VUs.

[0053] The method for offloading some computationally intensive tasks to the MEC server connected to the VU for computation is as follows:

[0054] In the current time slot, N vehicles on different lanes enter the coverage of M different base stations at different speeds along the vehicle's travel direction. After the offloading decision is made, each VU sends the task to be offloaded to the base station m within its coverage area. The base station m manages the vehicles through the uplink and detects the received signal and noise of each vehicle from the received signals of all vehicles, and further determines the signal to interference plus noise ratio (SINR) of each vehicle, where M is the total number of base stations, m is the base station number, and m∈M.

[0055] The base station sends the task calculation results and the determined SINR back to each VU in the next time slot (time period);

[0056] The base station detects the SINR at time t and sends the SINR to the vehicle at the next moment. The vehicle receives the SINR detected at time t-1 at time t. SINR(t-1) is the vehicle's local observation of the channel change at time t. SINR, as the vehicle's local observation of the channel change, reflects the channel condition. A good channel condition results in a short task transmission delay; conversely, a bad channel condition results in a long transmission delay.

[0057] At time t, the base station with the largest SINR(t-1) is selected as the base station for offloading tasks.

[0058] Vehicle mobility affects channel conditions, which in turn affect task offloading transmission latency. Tasks of varying sizes have different tolerances for task delays. Tasks that fail after the tolerance are considered failed. Therefore, the task success rate is defined as the number of tasks successfully completed within the tolerance for the task / the total number of tasks reached. The number of task failures reflects system latency. In the case of multiple base stations, collaboration is required between base stations. When the computational load on a base station becomes excessive, some tasks are transferred to other base stations, such as the base station in the area the vehicle is about to enter. This reduces overall system latency and achieves load balancing across base stations.

[0059] Get the communication delay from VU offload task to MEC The method is:

[0060] For each time slot, the channel vector of vehicle n is h n (t),

[0061] The received signal of the base station is expressed as

[0062]

[0063] Among them, p o,n (t) represents the transmission power of the vehicle; s n (t) represents unit variance; n(t) represents Gaussian white noise vector;

[0064] The transformation of the channel state between time t and t-1 is expressed as follows:

[0065]

[0066] Among them, ρ n is the normalized channel correlation coefficient between time slots t and t-1, e(t) is the error vector, which is Gaussian distributed, so that H(t) = [h1(t),…,h N (t)] represents the KxN channel matrix between BS and N vehicles, K is the number of rows in the channel matrix, N is the number of columns in the channel matrix, and H + (t) acts as a ZF detector to detect the VU signal received from y(t), and the linear decoder is expressed as:

[0067]

[0068] Among them H H (t) is the conjugate transpose of H(t),

[0069] use Indicates H +The nth row of (t) exists When i=j, δ ij =1, otherwise δ ij = 0, the received signal of vehicle n is expressed as:

[0070]

[0071] Then the signal-to-noise ratio of vehicle n at time t is:

[0072]

[0073] [A] nn represents the nth element of matrix A, is the variance;

[0074] Then the achievable data transmission rate between vehicle n and MEC server m can be calculated as:

[0075] r n (t) = Blog2(1+γ n (t))

[0076] B is the bandwidth allocated to each vehicle,

[0077] Then the communication delay from vehicle n unloading part of the task to MEC is expressed as

[0078]

[0079] r n (t) is the data transmission rate between vehicle n and MEC, x n ∈[0,1] is the unloading ratio, λ n is the task size. Since the calculation result size is much smaller than the task data size, the transmission delay of the calculation result from the MEC server to vehicle n can be ignored.

[0080] The computing model describes the latency required to execute a computing task, which consists of two parts: one is the local computing latency of the task, and the other is the computing latency on the server when the computing-intensive task is offloaded to the MEC server.

[0081] Time-sensitive tasks have strict requirements on latency, so these tasks are all placed on local computing and only have local computing latency. When compute-intensive tasks arrive, part of the task is offloaded to the BS, and the MEC server deployed and connected to the BS processes the task. For the partial tasks offloaded to the BS, we make the following considerations: since the amount of computational result data is much smaller than the amount of data uploaded to the MEC, the delay in returning the data results after task data processing can be ignored. For the compute-intensive tasks that arrive, without considering task migration, the processing latency is max (partial task upload latency + server computation latency, and the remaining task local computation latency).

[0082] Get the latency of local computation for computationally intensive tasks placed on the VU for data processing The method is:

[0083]

[0084] Among them, f n is the CPU computing power of the vehicle when performing local computing tasks, C n is the total number of CPU cycles required to compute task n, x n The ratio of tasks offloaded to MEC servers for vehicle units.

[0085] Get the computational latency of the offloaded task on the MEC server The method is:

[0086]

[0087] Among them, C n x n The number of CPU cycles required to calculate the tasks offloaded to the MEC server, f m is the computing capacity of MEC server m, Expressed as whether task n is computed by MEC server m. In this formula, the computational latency of MEC server m increases linearly with the computational requirements of the task, leading to unreliable service and significant computational latency. Therefore, to meet QoS requirements and improve system throughput, it is necessary to balance the computational load across MEC servers. MEC servers with excessive computational loads can migrate some tasks to other servers to achieve load balancing.

[0088] After the vehicle offloads the task to the MEC server next to the BS, the BS decides whether to migrate the computing task to the MEC server next to another BS, especially considering the current server load and the high-speed mobility of the vehicle (the vehicle is about to leave the current service area). Since the task offloaded from the VU to the MEC server is already a partial task, the task is no longer split during task migration, but is migrated as a whole. During the migration process, additional migration delay and migration cost are generated, as well as a trade-off between the queuing waiting time in the MEC server. An important factor affecting whether a task should be migrated is the server load. The analysis of server load will be mentioned in the load balancing model.

[0089] When the MEC server decides to migrate computing tasks from MECm to MECm', a migration delay will occur.

[0090] When the MEC server migrates the computing task from the mth MEC to the m'th MEC, the migration delay is expressed as:

[0091]

[0092] MEC servers are connected by wires. b is the bandwidth between MEC servers, λ n is the total data size, λ n x n The amount of data to be migrated;

[0093] The computation migration caused by the mobility of the vehicle will incur additional costs, such as the computation replication cost from MEC m to MEC m' and the resource release of MEC m hosting the current task. The computation migration cost is determined by the amount of computation tasks, and the migration cost is expressed as:

[0094]

[0095] Where μ is the weight factor, μ>0.

[0096] The method to obtain the load balancing factor LBF is:

[0097] The computational load L of the mth MEC server m Expressed as:

[0098]

[0099] M represents the number of MEC servers, N represents the number of tasks, which is equal to the number of vehicles, and L m It is represented as the accumulation of computing tasks on the mth MEC server. If task n is processed by server m, then

[0100] The average computational load of all MECs Expressed as:

[0101]

[0102] To determine whether the computational load is fairly distributed among MECs in the system, load balance is measured by the deviation of the computational load and the load balancing factor LBF is defined as:

[0103]

[0104] S5, based on the above model, calculates the total delay of a task for the system as T. In order to meet the task tolerance delay, the overall delay of the system is optimized through task migration and load balancing scheduling. The overall delay includes the task transmission delay when the vehicle uploads the task to the MEC server, the task processing delay on the MEC server, and the task migration delay between different MEC servers. The objective function constructed is:

[0105]

[0106] St T <T tolerance ,x n ∈[0,1],

[0107]

[0108] Among them, ω represents the number of times tasks are migrated between different MEC servers, λ l and w are custom weight parameters, w∈[0,1], x n is the unloading ratio, λ l is the balance T and The ratio between.

[0109] In this system, each vehicle can be regarded as self-interest, competing for MEC computing resources (the vehicle cannot perceive the load of the MEC server and only offloads tasks). Different MEC servers collaborate to minimize system latency and load balancing.

[0110] S6, solves the objective function, obtains the task offloading ratio, and whether to migrate the current computing task, and uses this to control the execution of the VU and MEC servers.

[0111] In this embodiment, the method for solving the objective function is:

[0112] DDPG algorithm (Deep Deterministic Policy Gradient Algorithm), or DQN algorithm (Deep Q Network Algorithm), or Q-Learning (Q Learning Algorithm), or DPP0 (Distributed Proximal Policy Optimization Algorithm) or DRL-MATD3 algorithm (Distributed Reinforcement Learning-Multi-Agent Double Delay Deep Deterministic Policy Gradient Algorithm).

[0113] It is preferred to use the DRL-MATD3 algorithm to solve the objective function, such as Figure 2 and Figure 3 As shown, the specific steps are:

[0114] S11, initialize three common networks for each base station agent i, including two critic networks and an actor network Three ordinary networks with random network parameters θ i,1 ,θ i,2 and i is the serial number of the base station, i∈M;

[0115] S12, for each agent i Initialize three target networks, including two target_critic network parameters Q′ θi,1 ,Q′ θi,2 and a target_actor network parameter Copy the three initialized common network parameters to the target network, i.e. θ′ i,1 =θ i,1 ,θ′ i,2 =θ i,2 , And initialize the playback buffer Buf;

[0116] S13, initialize a random noise ξ for detecting action, and receive the initial state s = (s l ,s2…,s N ), where s n ={Task n ,h n ,l},Task n ={λ n ,C n ,T tolerance} represents the mission information of vehicle n, h n The current channel condition between vehicle n and the base station, l represents the computational load of the MEC server within the current BS service range;

[0117] S14, set the maximum number of training times T1, t1 is the training number sequence number, T1 is a positive integer, let t1 = 1, and perform the following operations:

[0118] For each agent i , choose a random action a j ~μ i (s j )+ξ i ,ξ i is noise, j∈(1,N) represents the VU number, μ i (s j ) indicates the network Enter s j state;

[0119] Explore the neural network, determine the current strategy, and execute the action space a=(a1,…,a N ), where a n ={x n ;q n},x n represents the task unloading ratio of the nth vehicle, q n It represents the migration decision of the task of offloading the n-th vehicle to the task on the MEC server. Solving the action space is to solve the next offloading decision and task migration decision.

[0120] After executing the next action, observe the reward r of the current strategy and the new state s′,

[0121] r=-(λ l T+wC migrate +(1-w)LBF)

[0122] The experience tuple (s, a, r, s′) obtained by executing the current exploration is stored in the playback buffer Buf, and the state s = s′ is updated;

[0123] For each agent i Perform the following operations: When the number of tuples in Buf reaches a certain value, randomly sample N samples from Buf, where the empirical tuple of the jth sample is (s j ,a j ,r j ,s′ j ), set the target value y j =r+γ*target_q_min, γ is the discount rate, which is a weight parameter set by humans and calculated based on the target value y j θ for the critic network i,n The parameters are updated; in which, when calculating the target value y jWhen the Q value is estimated by using two critic networks (the Q value can be obtained by inputting the state s and action a into the critic network, and the specific calculation method can adopt the existing technology), the double network is realized, and the relatively small Q value, namely target_q_min, is selected as the target value y j Parameters;

[0124] When the number of updates of the critic network reaches a certain parameter value, the parameters of the actor network and the target_critic and target_actor networks are updated once;

[0125] S15, let t1 = t1 + 1 and execute step S14 again until all training rounds are completed to obtain the final trained network.

[0126] This paper optimizes multi-agent task migration and load balancing strategies. It collects vehicle task information, calculates channel conditions based on a communication model, and calculates the computational load of MEC servers within the base station's service range based on a load balancing model. These elements are used to construct a state space. A random state within the state space is selected and input into the action network. The resulting random action is then subjected to noise exploration by the neural network to determine the strategy for the current step.

[0127] This paper utilizes the MATD3 algorithm, which, compared to DRL algorithms like DDPG and DQN, has the advantages of a dual Q-value network and delayed updates, enabling more stable and accurate network training. In MATD3, we use two networks to estimate Q-values, selecting a relatively small value as the target for network updates. This overcomes the common DRL algorithm problem of the critic network overestimating the Q-values ​​of actions. Furthermore, by delaying updates to the actor network, we make actor network training more stable.

[0128] The present invention takes into account the system delay optimization under the conditions of channel time variation and random task arrival, which not only ensures the completion rate of vehicle tasks within the task tolerance delay, but also optimizes the task migration and load balancing algorithm between MEC servers providing services, meeting the requirements of low latency for completing computing-intensive tasks and alleviating excessive load on computing resources.

[0129] The present invention also provides a distributed Internet of Vehicles service offloading migration and load balancing system, which includes M base stations arranged along a one-way lane, each base station is equipped with K antennas, and each base station is connected to an MEC server. There are N moving vehicles along the road, each vehicle carries a single antenna with limited computing resources, K, M, and N are all positive integers and K>N;

[0130] The vehicle and the MEC server obtain the task offloading ratio and whether to migrate the current computing task according to the method of the present invention, and execute it accordingly.

[0131] Throughout this specification, reference to terms such as "preferred embodiment," "one embodiment," "some embodiments," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0132] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for offloading, migrating and load balancing of Internet of Vehicles services, characterized in that: The steps include: S1, the nth VU obtains the task Task n ={λ n ,C n ,T tolerance },λ n is the task data volume of the VU, C n Calculate the total number of CPU cycles required for this task for this VU, T tolerance is the tolerated delay of the task, n∈N, N is the total number of vehicles in the Internet of Vehicles, and N is a positive integer; S2: Classify the task types into time-sensitive tasks and computation-intensive tasks. If the task is time-sensitive, it is placed on the VU for data processing; if the task is computation-intensive, part of the computation-intensive task is offloaded to the MEC server connected to the VU for calculation; S3: Get the communication delay from the VU offload task to the MEC The latency of local computation for computationally intensive tasks placed on the VU for data processing And the computational delay of the offloaded task on the MEC server Get the communication delay from VU offload task to MEC The method is: For each time slot, the channel vector of vehicle n is h n (t), The received signal of the base station is expressed as: Among them, p o,n (t) represents the transmission power of the vehicle, s n (t) represents unit variance, n(t) represents Gaussian white noise vector; The transformation of the channel state between time t and t-1 is expressed as follows: Among them, ρ n is the normalized channel correlation coefficient between time t and t-1, e(t) is the error vector, which is Gaussian distributed, so that H(t) = [h1(t),…,h n (t)] represents the KxN channel matrix between the BS and N vehicles, K is the number of antennas, and H + (t) acts as a ZF detector to detect the VU signal received from y(t), and the linear decoder is expressed as: Among them H H (t) is the conjugate transpose of H(t), use Indicates H + The nth row of (t) exists When i=j, δ ij =1, otherwise δ ij =0, the received signal of vehicle n is expressed as: Then the signal-to-noise ratio of vehicle n at time t is: [A] nn represents the nth element of matrix A, is the variance; Then the achievable data transmission rate between vehicle n and MEC server m can be calculated as: r n (t)=B log2(1+γ n (t)) B is the bandwidth allocated to each vehicle, Then the communication delay from vehicle n unloading part of the task to MEC is expressed as r n (t) is the data transmission rate between vehicle n and MEC, x n ∈[0,1] is the unloading ratio, λ n is the size of the task; Get the latency of local computation for computationally intensive tasks placed on the VU for data processing The method is: Among them, f n is the CPU computing power of the vehicle when performing local computing tasks, C n is the total number of CPU cycles required to compute task n, x n The proportion of tasks offloaded to the MEC server for vehicle units; Get the computational latency of the offloaded task on the MEC server The method is: Among them, C n x n The number of CPU cycles required to calculate the tasks offloaded to the MEC server, f m is the computing capacity of MEC server m, Indicates whether task n is calculated by MEC server m; S4: Determine the computational delay of offloaded tasks on the MEC server. If the QoS requirements are not met, the tasks that do not meet the QoS requirements will be migrated between MEC servers to achieve load balancing and obtain the migration delay. Migration costs And the load balancing factor LBF; if the computing delay of all offloaded tasks on the MEC server All meet QoS requirements, reducing migration delay Migration costs And calculate the load balancing factor LBF; Get migration delay Migration costs The method is: When the MEC server migrates the computing task from the mth MEC to the m'th MEC, the migration delay is expressed as: MEC servers are connected by wires. b is the bandwidth between MEC servers, λ n is the total data size, λ n x n The amount of data to be migrated; The migration cost is expressed as: Where μ is the weight factor, μ>0; The method to obtain the load balancing factor LBF is: The computational load L of the mth MEC server m Expressed as: M represents the number of MEC servers, N represents the number of tasks, which is equal to the number of vehicles, and L m It is represented as the accumulation of computing tasks on the mth MEC server. If task n is processed by server m, then The average computational load of all MECs Expressed as: To determine whether the computational load is fairly distributed among MECs in the system, load balance is measured by the deviation of the computational load and the load balancing factor LBF is defined as: S5, construct the objective function: S.t T<T tolerance ,x n ∈[0,1], Where ω represents the number of times tasks are migrated between different MEC servers. λ l , w are custom weight parameters, w∈[0,1], x n is the unloading ratio; S6, solves the objective function, obtains the task offloading ratio, and whether to migrate the current computing task, and uses this to control the execution of the VU and MEC servers; The DRL-MATD3 algorithm is used to solve the objective function. The specific steps are as follows: S11, initialize three common networks for each base station agent i, including two critic networks and an actor network Three ordinary networks with random network parameters θ i,1 ,θ i,2 and i is the serial number of the base station; S12, for each agent i Initialize three target networks, including two target_critic network parameters Q ′ θi,1 ,Q ′ θi,2 and a target_actor network parameter Copy the three initialized common network parameters to the target network, i.e. θ′ i,1 =θ i,1 ,θ′ i,2 =θ i,2 , And initialize the playback buffer Buf; S13, initialize a random noise ξ for detecting action, and receive the initial state s = (s l ,s2…,s n ), where s n ={Task n ,h n ,l},Task n ={λ n ,C n ,T tolerance } represents the mission information of vehicle n, h n The current channel condition between vehicle n and the base station, l represents the computational load of the MEC server within the current BS service range; S14, set the maximum number of training times T1, t1 is the training number sequence number, T1 is a positive integer, let t1 = 1, and perform the following operations: For each agent i , choose a random action a j ~μ i (s j )+ξ i ,ξ i is noise, j∈(1,N) represents the VU number, μ i (s j ) indicates the network Enter s j state, Explore the neural network, determine the current strategy, and execute the action space a=(a1,…,a n ), where a n ={x n ;q n }, x n Indicates the uninstall ratio, q n It represents the migration decision of the task offloaded to the MEC server. Solving the action space is to solve the next offloading decision and task migration decision. After executing the next action, observe the reward r of the current strategy and the new state s ′ , r=-(λ l T+wC migrate +(1-w)LBF) The experience tuple (s,a,r,s ′ ) is stored in the playback buffer Buf and the state s=s is updated ′ ; For each agent i Perform the following operations: When the number of tuples in Buf reaches a certain value, randomly sample N samples (s j ,a j ,r j ,s ′j ), set the target value y j =r+γ*target_q_min, γ is the discount rate, which is a weight parameter set by humans and calculated based on the target value y j and Q value for the critic network θ i,n The parameters are updated, where the target value y is calculated j By using two critic networks to estimate the Q value, a double network is implemented, and a relatively small Q value, target_q_min, is selected as the target value y j Parameters; When the number of updates of the critic network reaches a certain parameter value, the parameters of the actor network and the target_critic and target_actor networks are updated once; S15, let t1 = t1 + 1 and execute step S14 again until all training rounds are completed to obtain the final trained network.

2. The method for offloading, migrating and load balancing of Internet of Vehicles services according to claim 1, characterized in that: The method for offloading some computationally intensive tasks to the MEC server connected to the VU for computation is as follows: In the current time slot, N vehicles on different lanes enter the coverage of M different base stations at different speeds along the vehicle's travel direction. After the offloading decision is made, each VU sends the task to be offloaded to the base station m within its coverage area. The base station m manages the vehicles through the uplink and detects the received signal and noise of each vehicle from the received signals of all vehicles, and further determines the signal to interference plus noise ratio (SINR) of each vehicle, where M is the total number of base stations, m is the base station number, and m∈M. The base station sends the task calculation results and the determined SINR back to each VU in the next time slot; The base station detects the SINR at time t and sends the SINR to the vehicle at the next moment. The vehicle receives the SINR detected at time t-1 at time t. Then SINR(t-1) is the vehicle's local observation of the channel change at time t. At time t, the base station with the largest SINR(t-1) is selected as the base station for offloading tasks.

3. The method for offloading, migrating and load balancing of Internet of Vehicles services according to claim 1, characterized in that: The solution to the objective function is: DDPG algorithm, or DQN algorithm, or Q-Learning algorithm, or DPP0 algorithm.

4. A distributed Internet of Vehicles service offloading migration and load balancing system, characterized in that: It includes M base stations set up along a one-way lane, each equipped with K antennas, and each base station is connected to an MEC server. There are N moving vehicles along the road, each carrying a single antenna with limited computing resources. K, M, and N are all positive integers and K>N; The vehicle and the MEC server obtain the task offloading ratio and whether to migrate the current computing task according to the method described in one of claims 1-3, and execute accordingly.

Images