Task allocation method for intelligent vehicle in end-edge-cloud dynamic offloading framework

Abstract

The application discloses an unloading task allocation method for intelligent vehicles in an end-edge-cloud dynamic unloading framework. The method first establishes an end-edge-cloud architecture composed of a cloud server, multiple roadside units with edge servers deployed and multiple intelligent vehicles, then defines the execution rules of dynamic unloading in the architecture, and on the basis thereof, defines a task processing model, a task transmission model, a residual quantity model and an utility function which comprehensively considers the task consumption time and energy consumption. An improved multi-agent reinforcement learning algorithm is used to optimize the maximum utility function index of the architecture, and the running mode of the network is set to centralized training and distributed execution. During running, each intelligent vehicle can make a decision on the unloading action at the moment according to the current environmental state of the vehicle. The method is simple in operation, high in practicability, and improves the service quality of the end-edge-cloud architecture and the resource consumption of vehicle task execution.

Description

Technical Field

[0001] This invention belongs to the field of dynamic resource task allocation and scheduling technology in edge-cloud framework, particularly a dynamic resource scheduling method based on reinforcement algorithm, and especially a method for allocating unloading tasks for intelligent vehicles in edge-cloud dynamic unloading framework. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and mobile edge computing (MEC), vehicles have become a key node in the IoT ecosystem. As mobile intelligent devices, vehicles can directly provide users with various advanced intelligent services. Through the deployment of V2X (vehicle-to-everything) communication systems, vehicles can achieve real-time intelligent transportation services and generate a large amount of task data. Currently, common ITS services include autonomous driving, traffic navigation, and video surveillance. The emergence of connected and autonomous vehicles (CAVs) has brought new opportunities and challenges to the automotive industry. One key challenge is how to effectively process the large amounts of data generated by CAVs in real time, ensuring timely and efficient task processing for vehicles. Simultaneously, when a large number of vehicles are processing tasks simultaneously, how to allocate computing resources and bandwidth among local, edge devices, and cloud servers, and optimize time and cost overhead, is a hot research direction in this field. Therefore, researchers are actively exploring solutions based on edge computing and machine learning technologies to improve the performance and efficiency of ITS systems.

[0003] In the real-world environment of connected vehicles, network conditions, user demands, and computing resources are constantly changing. How to formulate reasonable offloading and processing decisions for each task based on the environment is a key research direction in this field. Alasmari et al. (K. Alasmari, R. Green, and M. Alam, "Mobile edge offloading using markov decision processes," Edge Computing–EDGE 2018: Second International Conference, pp. 80-90, 2018.) abstracted the service offloading decision process into a Markov Decision Process (MDP). Later, Kiumarsi et al. (B. Kiumarsi, K. Vamvoudakis, H. Modares, and F. Lewis, "Optimal and Autonomous Control Using Reinforcement Learning: A Survey," IEEE Transactions on Neural Networks and Learning) Systems, vol. 29, no. 6, pp. 2042-2062, 2018. Research has found that reinforcement learning (RL) methods can effectively solve Markov decision problems. They abstracted the entire decision generation process into an interaction between the agent and the environment. Arulkumaran et al. (K. Arulkumaran, M. Deisenroth, M. Brundage, and A. Bharath, “Deep Reinforcement Learning: A Brief Survey,” IEEE Signal Processing) (Magazine, vol. 34, no. 6, pp. 26-38, 2017.) proposed combining reinforcement learning and deep learning (DL) to form deep reinforcement learning (DRL), which can ultimately obtain a decision generation model through training on a large amount of high-dimensional simulation data. While the above work proposed a good optimization scheme, it is mainly aimed at static offloading, meaning that the processing offloading decision for each task is generated immediately after the task is issued, allowing edge servers to download, process, and store relevant data in advance, thereby reducing task transmission latency and network bandwidth consumption. However, static offloading cannot cope with changes in tasks and environments. When computational tasks have real-time requirements, network transmission environments are unstable, and the size of computational tasks is uncertain, static offloading models cannot provide users with reasonable decisions. Summary of the Invention

[0004] The purpose of this invention is to address the problems existing in the prior art by providing a method for allocating unloading tasks for intelligent vehicles in an edge-cloud dynamic unloading framework.

[0005] The technical solution to achieve the objective of this invention is as follows: On one hand, a method for allocating unloading tasks for intelligent vehicles in an edge-cloud dynamic unloading framework is provided, the method comprising:

[0006] Step 1: Establish an edge-cloud architecture including a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. Define the execution rules for dynamic offloading in this edge-cloud architecture, and on this basis, define a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task consumption time and energy consumption.

[0007] Step 2: Optimize the system’s maximum utility function index using an improved multi-agent reinforcement learning algorithm, and set the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at this moment based on its own environmental state.

[0008] Furthermore, it also includes:

[0009] Step 3: After training is complete, each intelligent vehicle uses its local policy network to independently make unloading decisions.

[0010] On the other hand, an unloading task allocation system for intelligent vehicles in an edge-cloud dynamic unloading framework is provided, the system comprising:

[0011] The first module is used to establish an edge-cloud architecture including a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. It defines the execution rules for dynamic offloading in the edge-cloud architecture, and on this basis, defines a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task consumption time and energy consumption.

[0012] The second module is used to optimize the system's maximum utility function index using an improved multi-agent reinforcement learning algorithm, and sets the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at that moment based on its own environmental state.

[0013] Furthermore, the system also includes:

[0014] The third module is used to enable each intelligent vehicle to independently make unloading decisions using its local policy network after training is completed.

[0015] Compared with the prior art, the significant advantages of this invention are:

[0016] (1) It meets the real-time requirements of unloading tasks for intelligent vehicles: By using the idea of ​​repeated game theory, each task can make multiple unloading decisions. Under the circumstances that the computation task has real-time requirements, the network transmission environment is unstable, and the size of the computation task is uncertain, the utilization rate of system resources can be improved.

[0017] (2) A residual quantity model was built: In the dynamic unloading model, a residual quantity model was constructed to measure the situation of a specific device in a specific state, so as to facilitate the decision-making model to make subsequent judgments.

[0018] (3) The unloading strategy of intelligent vehicles is optimized: This invention uses a multi-agent reinforcement learning algorithm adapted to the dynamic unloading model to maximize the utility function of the system which is composed of time consumption and energy consumption, thereby optimizing the unloading strategy of intelligent vehicles.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the method for allocating unloading tasks for intelligent vehicles within the edge-cloud dynamic unloading framework.

[0021] Figure 2 This is a schematic diagram of the system architecture model.

[0022] Figure 3 This is the flowchart of the RT-MADDPG algorithm. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0025] The dynamic offloading framework proposed in this invention for edge-cloud refers to the unloading task of intelligent vehicles, which involves making offloading decisions through the decision network in multi-agent reinforcement learning RT-MADDPG, and updating the decision network by the value network. The decision network and the value network mutually restrain and compete with each other to achieve the goal of optimizing the system.

[0026] In one embodiment, combined Figure 1 This paper provides a method for allocating unloading tasks for intelligent vehicles in an edge-cloud network framework, including the following steps:

[0027] Step 1: Establish an edge-cloud architecture comprising a cloud server, multiple roadside units deployed with edge servers, and multiple intelligent vehicles, such as... Figure 2 As shown, the execution rules for dynamic unloading in this edge-cloud architecture are defined, and on this basis, the task processing model, task transmission model, remaining quantity model, and utility function that integrates task consumption time and energy consumption are defined.

[0028] Step 2: Optimize the system’s maximum utility function index using an improved multi-agent reinforcement learning algorithm, and set the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at this moment based on its own environmental state.

[0029] Step 3: After training is complete, each intelligent vehicle can independently make unloading decisions by using its local policy network each time, without interfering with each other.

[0030] Furthermore, in one embodiment, step 1 establishes an edge-cloud architecture comprising a cloud server, multiple roadside units deployed with edge servers, and multiple intelligent vehicles. It defines the execution rules for dynamic offloading within this edge-cloud architecture and, based on these rules, defines a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task time and energy consumption. Specifically, this includes:

[0031] Step 1.1, define the description information of intelligent vehicles in the edge-cloud system: V represents the set of intelligent vehicles, and intelligent vehicle k is represented by a tuple. Describe the performance of the intelligent vehicle itself, among which, P represents the local computing power of the k-th intelligent vehicle. k This represents the transmission power of the kth intelligent vehicle (k).

[0032] Step 1.2, define the description information of the roadside unit: RSU represents the set of roadside units, and the performance of the j-th roadside unit RSUj is expressed by the quadruple {f CPU-mec M, b j area j} represents, where f CPU_mecThis indicates the CPU operating frequency of a single CPU core in the edge server, where M represents the total number of CPU cores in the edge server, and b... j The area represents the bandwidth of roadside unit j. i RSU j The set of tasks performed by intelligent vehicles within the coverage area. The edge servers of the roadside unit possess strong computing power, with each edge server having a fixed number of CPU cores. During each task execution, the edge server needs to allocate a certain number of CPU cores, and each CPU core can only be occupied by one batch of task data per time slot. Therefore, the total computing power provided by the edge servers of the roadside unit to the intelligent vehicles is represented by f. mec f mec =f CPU_mec ·M;

[0033] Step 1.3, define the description information of the cloud server in the edge-cloud system: the performance of cloud server C is expressed by the quadruple {f CPU_c ,MC,f c b c} indicates that f CPU_c This represents the single-core CPU operating frequency of the cloud server's CPU cores; MC represents the number of CPU cores the cloud server can provide to the intelligent vehicle at any given time (the cloud server provides a fixed number of CPU cores to the intelligent vehicle user, which can be seen as providing a fixed amount of computing resources to the user); the computing power that each cloud server can provide to the intelligent vehicle at any given time is f. c f c =f CPU_c ·MM, where MM represents the number of CPU cores provided by the cloud server, b c Indicates the bandwidth of the cellular network;

[0034] Step 1.4, define the task model in the edge-cloud system: TASK represents a set of tasks, and the i-th task TASK... i Through the triple {D i ,γ i ,w i} indicates that D i For the total task, γ i For task type, w i γ represents the data processing density of the i-th task; i Choose 0, 1, and 2, where 0 represents a time-sensitive task, 1 represents a neutral task, and 2 represents an energy-sensitive task.

[0035] Step 1.5, define the dynamic task offloading rules in the edge-cloud system: starting from the time of task execution, with a fixed time interval t0 as the time slot, every t0 time interval, each intelligent vehicle makes a decision on the execution plan for the current task.

[0036] Intelligent vehicles based on unloading decision variable z ki Determine whether to perform an offload operation in the current time slot, z ki When the value is 0, no unloading process is performed, and the existing decision is executed; ki When z is 1, unload the process and execute the new decision; when z is 1... ki When the value is 1, it is determined by the unloading location variable x. ki Determines whether the task data in the current time slot is uploaded to the roadside unit or executed on the cloud server, x ki When x equals 0, intelligent vehicle k unloads part of the data from task i to the roadside unit for execution; ki When the value equals 1, the intelligent vehicle k will offload part of the data from task i to the cloud server for execution; that is, the two offloading methods cannot be performed simultaneously. If the data is offloaded to the roadside unit for execution, the edge server will allocate η. in Proportional bandwidth transmission offload data, allocated M in The number of CPU cores used to process offloaded data, where the allocation ratio η in The range is between 0 and 1, and M is allocated. in The number must not exceed the total number of cores in the device, and CPU cores currently in use cannot be reallocated during this process; if offloaded to a cloud server for execution, the cloud server provides f for each intelligent vehicle task requesting service. c Computing resources;

[0037] y i y represents the total number of times task i was unloaded. i ≤N, where N∈[1,+∞], and the maximum value of N is determined by the number of time slots covered by the task's local execution; let the start time of task i be t. i0 The time of the first unloading process is set to t. i1 The proportion of unloaded data to the total data volume is α. i1 The amount of data unloaded is α. i1 ·D i The time of the second unloading process is set to t. i2 The proportion of unloaded data to the total data volume is α. i2 The amount of data unloaded is α. i2 ·D i ...and so on, let t be the time when the nth unloading process occurs. in The proportion of unloaded data to the total data volume is α. in The amount of unloaded data is αi n· D i α i1 α i2 ...α in The initial value is 0, t i1t i2 ...t in The initial value of is set to ∞, and correspondingly, t in trans This represents the time of the nth transmission.

[0038] Step 1.6, define the task data transmission model and task processing model in the edge-cloud system: When the intelligent vehicle decides to unload and execute a task, if task i is unloaded and executed for the nth time, then its transmission time is expressed as:

[0039]

[0040] Where α in ·D i For t in The amount of tasks to be unloaded at any given time, where B represents the transmission bandwidth, and P represents the amount of data to be unloaded at any given time. k G represents the transmission power of intelligent vehicle k. k This represents the channel gain during transmission; when z in =1,xi n When B = 0, task data is unloaded from the intelligent vehicle to the roadside unit, B = η. in ·b j When z in =1,xi n When B = 1, the task data is unloaded from the intelligent vehicle to the cloud server, and B = b. c The corresponding transmission energy consumption E in trans Represented as:

[0041] E in trans =P k ·t in trans

[0042] Among them, P k The transmission power of intelligent vehicle k;

[0043] Furthermore, to distinguish the uninstallation location, t will be used subsequently. in mu_trans t represents the time it takes for task data to be transmitted to the roadside unit. in cu-trans This indicates the time it takes to be transmitted to the cloud server, and specifies that:

[0044]

[0045]

[0046] Task execution is divided into three execution modes: local task execution by the intelligent vehicle, task execution by the roadside unit, and task execution by the cloud server; the latency of task i during a certain decision-free period is described as follows:

[0047] t i =|data i |·w i / f

[0048] Among them, |data i | Indicates the amount of data on the device, w i Let f be the data density, and f be the computing resources; when the data is processed locally, f is the computing resources of the local vehicle. When data is processed at the roadside unit, f represents the computing resources allocated to task i by the edge server. Assuming the order of the previous decision was n, then f = M. in ·f CPU_mec M in Let f be the number of CPU cores allocated to task i during the nth unloading; when the data is executed on the cloud server, f is the computing resources allocated to task i by the cloud server, f = f c ;

[0049] The execution energy consumption corresponding to the above execution time is expressed as follows:

[0050] e i =M′·δ·(f CPU ) 3 ·t i

[0051] Where M′ is the number of device cores, δ is the calculation frequency correlation coefficient, which depends on the hardware conditions of the cloud server CPU, and f CPU t represents the number of clock cycles for a single-core CPU in this device. i The above execution time;

[0052] When the execution location is an intelligent vehicle, the intelligent vehicle is locally regarded as having a single-core CPU, so M is 1, and δ is taken as the local calculation frequency correlation coefficient δ. loc fCPU is When the execution location is a roadside unit, M is the number of CPU cores allocated in the previous decision. Assuming the order of the previous decision is n, then M = M in δ is taken as the edge calculation frequency correlation coefficient δ mec f CPU f CPU_mec When the execution location is a cloud server, M is the fixed number of cores MC allocated to the cloud server, and δ is the cloud computing frequency correlation coefficient δ. c ;

[0053] Step 1.7: Establish a residual quantity model for evaluating the real-time status of the edge-cloud system. This residual quantity model includes: remaining data volume, remaining time, and remaining energy consumption.

[0054] The remaining data volume refers to the amount of data that the current device needs to execute but has not yet executed, based on observations and calculations from the current moment, if the current decision is maintained.

[0055] Remaining time refers to the time required for the current device to process all the data if the current decision is maintained, based on the current observation.

[0056] Residual energy consumption refers to the total energy consumed by the equipment in the remaining time if the current decision is maintained;

[0057] The remaining data volume of the intelligent vehicle, roadside unit, and cloud server are described below.

[0058] The remaining local data volume is represented by |data i loc_left |(t) represents the value when t is... i0 ≤t<t i1 If local data has not yet been unloaded, then the remaining local data volume should equal the total local data volume minus the local data volume already executed, i.e.:

[0059] |data i loc_left |(t)=max(D i -(tt i0 )·f loc / w i ,0)

[0060] When the nth unloading decision has been given, and the (n+1)th unloading decision has not yet been given, i.e., t in ≤t<t i(n+1) The remaining local data volume is represented as:

[0061]

[0062] Where 1≤n≤N, (tt) i0 )·f loc / w i (tt) i0 The amount of data completed within the time period. This indicates the amount of data that has been unloaded. It's important to note that, although the latest decision may indicate that some data requiring unloading is still locally awaiting transfer, this portion of data is still considered unexecutable locally.

[0063] For decision α i1 …α in ...α iN Each needs to satisfy [t] i1 , t i2 )…[t in , t i(n+1) )...[t iN , t i(N+1)Decision constraints within a time period. When n = N, t i(n+1 It has no practical meaning and can be regarded as ∞, that is, t. i(N+1) =∞. That is, t in The amount of unloaded data at time α in It cannot exceed Remaining local data volume at any given time The time is t in Before time t, and infinitely close to t in The constraints at any given moment are as follows:

[0064]

[0065] The remaining data volume of the roadside unit is represented by |data i mu_left |(t) represents the amount of data remaining in the roadside unit before the first unloading and transmission is completed.

[0066] When the nth unloading decision is given, and the nth unloading data has not yet been completely transmitted, i.e., t in ≤t<t in +t in mu_trans The remaining data volume of the roadside unit can be represented as The amount of data remaining at time t minus the amount of data since time t in The amount of data completed from the specified time:

[0067]

[0068] Where 1≤n≤N, express The amount of data remaining at time (tt) in )·f CPU_mec ·M in / w i (tt) in The amount of data completed within the time period. Note that although the data from the nth unloading operation has not yet been fully transferred and cannot be executed, the computing resources allocated to task i by roadside unit j have already been transferred from f. CPU_mec ·M i(n-1) Updated to f given in the nth decision. CPU-mec ·M in .

[0069] When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, i.e., t in +t in mu_trans ≤t<t i(n+1) The remaining data volume of the roadside unit can be represented as:

[0070] |data i mu_left |(t)

[0071] =max((1-x in )·α in ·D i +|data i mu-left |(t in +t in mu_trans)- -(tt in -t in mu_trans )·f CPU _mec ·M in / w i ,0)

[0072] Where 1≤n≤N, (1-x in )·α in ·D i |data represents the data from the nth uninstallation. i mu_left |(t in +t in mu _trans ) - It means (t) in +t in mu_trans ) - The amount of data remaining at time tt in -t in mu_trans )·f CPU_mec ·M in / w i (tt) in -t in mu_trans The amount of data completed within the time period.

[0073] The remaining data volume on a cloud server refers to the amount of data that has been offloaded to the cloud server but has not yet been processed. This article uses |data. i cu _left |(t) represents the amount of data remaining on the cloud server. Before the first unloading and transfer is completed, the amount of data remaining on the cloud server is 0.

[0074] When the nth unloading transfer is completed, and the (n+1)th unloading transfer is not yet completed, i.e., t in +t in cu_trans ≤t<t i(n+1) +t i(n+1)cu_trans The remaining data volume on the cloud server is as follows:

[0075] |data i cu_left |(t)

[0076] =max(x in ·α in ·D i +|data i cu_left |(t in +t in cu_trans ) - -(tt in -t in cu-trans )·f c / w i ,0)

[0077] Where 1≤n≤N, x in ·α in ·D i |data represents the data from the nth uninstallation. i cu-left |(t in +t in cu-trans ) - It means (t) in +t in cu-trans ) - The amount of data remaining at time tt in -t in cu-trans )·fc / w i (tt) in -t in cu_trans The amount of data completed within the time period.

[0078] The remaining time for intelligent vehicles (i.e., local, roadside units, and cloud servers) is described below.

[0079] The estimated remaining local time refers to the time required to execute the current local remaining data if the current decision remains unchanged at time t and beyond, expressed as:

[0080] T i loc_l e ft (t)=|data i loc_left |(t)·w i / f loc

[0081] The remaining time for a roadside cell refers to the time required to process the remaining data of the roadside cell if the current decision remains unchanged at time t and beyond. Before the first unloading decision, the remaining time for the roadside cell is 0. When the nth unloading decision is given, and the nth unloading data transmission has not yet been completed, i.e., t... in ≤t<t in +t in mu_trans The remaining time for the roadside unit is represented as follows:

[0082]

[0083] Where 1≤n≤N, Indicates from t in +t in mu_trans Starting from time n, the remaining execution time required for all data except for the data unloaded in the nth iteration; (1-x) in )·(t in +t in mu_trans -t) represents the remaining transmission time for the nth unloading of data; (1-x) in )·α in ·D i ·w i / (f CPU_mec ·M in ) indicates from t in +t in mu_trans Starting from time n, the remaining execution time required for the nth data unloading.

[0084] When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, i.e., t in +t in mu_trans ≤t<t i(n+1) The remaining time for each roadside unit is represented as follows, where 1 ≤ n ≤ N:

[0085] T i mu_left (t)=|data i mu_l e ft |(t)·w i / (f CPU_mec ·M in )

[0086] The remaining time for the cloud server refers to the time required to execute the remaining data on and after time t, assuming the current decision remains unchanged. Before the first unloading decision is completed, the remaining time for the cloud server is 0. When the nth unloading decision is given, and the nth unloading data has not yet been completely transferred, i.e., time t... in≤t<t in +t in cu_trans Similarly, the remaining time for the roadside unit can be represented as follows, where 1 ≤ n ≤ N:

[0087] T i cu-left (t)=max(|data i cu-left |(t in )·w i / f c -t in cu_trans ,0)+x in ·(t in +t in cu-trans -t)+x in ·α in ·D i ·w i / f c

[0088] Where max(|data) i cu_left |(t in )·w i / f c -t in cu-trans ,0) indicates from t in +t in cu_trans Starting from time n, the remaining execution time required for all data except for the data unloaded during the nth time; x in ·(t in +t in cu_trans -t) represents the remaining transmission time for the nth unloading of data; x in ·α in ·D i ·w i / f c Indicates from t in +t in cu_trans Starting from time n, the remaining execution time required for the nth data unloading;

[0089] When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, i.e., t in +t in cu_trans ≤t<t i(n+1) Similarly, the remaining time for the roadside unit is represented as follows, where 1≤n≤N:

[0090] Ti cu_left (t)=|data i cu_left |(t)·w i / f c

[0091] The remaining energy consumption of intelligent vehicles, namely local units, roadside units, and cloud servers, is described below.

[0092] Local remaining energy consumption refers to the energy consumption of executing the remaining local data while keeping the current decision unchanged at time t and beyond. It is represented as follows:

[0093] E i loc_left (t)=δ loc ·(f loc ) 3 ·T i loc_left (t)

[0094] Roadside unit remaining energy consumption refers to the energy consumed in transmitting and executing the remaining roadside unit data while maintaining the current decision at time t and beyond. Before the first unloading decision, the remaining energy consumption of the roadside unit is 0. The remaining energy consumption of the roadside unit is divided into two parts: the remaining energy consumption for roadside unit execution and the remaining energy consumption for roadside unit transmission.

[0095] When the nth unloading decision is given, and the nth unloading data transmission has not yet been completed, i.e., t in ≤t<t in +t in mu_trans The remaining energy consumption of the roadside unit is represented as follows, where 1≤n≤N:

[0096] E i mu-left_ca (t)= mec ·(f CPU-mec ) 2 ·(|data i mu_left |(t)+(1-x in )·α in ·D i )·w i

[0097] When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, i.e., t in +t in mu_tran s≤t<t i(n+1) The remaining energy consumption of the roadside unit is represented as follows, where 1≤n≤N:

[0098] E imu_left_ca (t)= mec ·(f CPU-mec ) 2 ·|data i mu_left |(t)·w i

[0099] The remaining transmission energy consumption of the roadside unit is represented as follows:

[0100]

[0101] Based on the above formula, the remaining energy consumption of the roadside unit can be obtained as follows:

[0102] E i mu_left (t)=E i mu_left_ca (t)+E imu_left_up (t)

[0103] The remaining energy consumption of a cloud server refers to the energy consumed in transmitting and executing the remaining data on and after time t, while maintaining the current decision. The remaining energy consumption of a cloud server is divided into two parts: the remaining energy consumption of roadside unit execution and the remaining energy consumption of roadside unit transmission.

[0104] The remaining transmission power consumption of the cloud server is represented as follows:

[0105]

[0106] The remaining execution energy consumption of the cloud server is represented as follows, where 1≤n≤N:

[0107]

[0108] Based on the above formula, the remaining energy consumption of the cloud server can be obtained as follows:

[0109] E i cu_left (t)=E i cu_left_ca (t)+E i cu_left_up (t)

[0110] Step 1.8: Calculate the total time and total energy consumed by the task using the residual quantity model;

[0111] Since task i can be executed in parallel on the local machine, the roadside unit, and the cloud server, the total time of task i must first be calculated separately for the time spent on the local machine, the roadside unit, and the cloud server.

[0112] The local execution time of task i refers to the time from the start of task i's execution to the completion of the local task. Based on the definition of remaining local time, the total local execution time can be expressed as follows:

[0113]

[0114] Where T i loc_left (t i0 This indicates that when task i is unloaded 0 times (i.e., all executions are performed locally), the total local execution time of task i is t. i0 The local remaining time at a given moment; similarly, T i loc_left (t i1 ), ......, T i loc_left (t iN ) represent the cases where task i has been unloaded 1, ..., N times respectively; t in -t i0 +T i loc_left (t in This indicates that when task i is unloaded n times, the total local execution time of task i is t. in The local remaining time at each moment plus the start time t i0 to t in The time between;

[0115] The total edge latency of task i refers to the time from the local execution time of task i, including the waiting time before the roadside unit receives the unloading data from task i, to the end of the last execution of sub-data of task i by the roadside unit; according to the definition of the remaining time of the roadside unit, the total edge latency T is... i mec Represented as:

[0116]

[0117] Where T i mu-left (t i0 This indicates that when task i is unloaded 0 times (i.e., all executions are performed locally), the total local execution time of task i is t. i0 The remaining time of the roadside cell at time T. If the initial roadside cell had no task data, then at this time... i mec =0; similarly, T i mu_left (t i1 ), ......, T i mu_left (t iN) represent the cases where task i has been unloaded 1, ..., N times respectively; t in -t i0 +T i mu_left (t in This indicates that when task i is unloaded n times, the total execution time of the roadside unit of task i is t. in The remaining time of the roadside unit at time t plus the start time t i0 to t in The time between;

[0118] The total cloud execution time for task i refers to the time from the local execution time of task i, including the waiting time before the cloud server receives the unloading data of task i, to the end of the last execution of the sub-data of task i by the cloud server; according to the definition of the remaining time of the cloud server, the total cloud execution time T is... i c Represented as:

[0119]

[0120] Where T i cu_left (t i0 This indicates that when task i is unloaded 0 times (i.e., all executions are performed locally), the total local execution time of task i is t. i0 The remaining time on the cloud server at any given moment. If the initial cloud server has no task data, then at this time T... i c =0; similarly, T i cu_left (t i1 ), ......, T i cu_left (t iN ) represent the cases where task i has been unloaded 1, ..., N times respectively; t in -t i0 +T i cu_left (t in This indicates that when task i is unloaded n times, the total execution time of task i on the cloud server is t. in The remaining time on the cloud server plus the start time t at any given moment i0 to t in The time between;

[0121] Because of the parallelism of task processing, the total time T consumed by task i is... i This can be represented as the maximum of local latency, edge latency, and cloud server latency. It is expressed as follows:

[0122] T i =max(T)i loc T i mec T i c )

[0123] The total energy consumption of task i is the sum of local energy consumption, edge energy consumption, and cloud energy consumption. The total local energy consumption is related to the total local execution time, as shown below:

[0124] E i loc =δ loc ·(f loc ) 3 ·T i loc

[0125] The total edge power consumption includes both transmission power consumption and execution power consumption. The total edge transmission power consumption for task i is:

[0126]

[0127] The total energy consumption for edge execution of task i is:

[0128]

[0129] Among them, T in mec_ca This indicates that for task i, the edge server uses computing resources f. CPU_mec ·M in The execution time. If the nth uninstallation does not exist, i.e., t... in As we approach infinity, we have:

[0130] T in mec_ca =0

[0131] If the nth uninstallation is the last uninstallation, that is... Then we have:

[0132]

[0133] If the nth uninstallation is not the last uninstallation, that is... Then we have:

[0134]

[0135] in, Indicates t in ≤t<t in +t in mu_trans During this period, computing resources f CPU_mec ·M in The duration used by task i; Indicates t in +t in mu_trans ≤t<t i(n+1) During this period, computing resources f CPU_mec ·M in The duration used by task i.

[0136] The total edge power consumption of task i can be expressed as:

[0137] E i memec =E i mu_trans +E i mu_execute

[0138] Total cloud energy consumption includes total transmission energy consumption and total execution energy consumption. The total transmission energy consumption for task i is:

[0139]

[0140] The total energy consumption for cloud-based execution can be expressed as:

[0141] E i cu_execute =MC·δ c ·(f CPU_c ) 3 ·T in c_ca

[0142] Among them, T in c_ca The total execution time in the cloud is represented as:

[0143] T i c_ca =(x i1 ·α i1 ·D i +x i2 ·α i2 ·D i +...+x in ·α in ·D i )·w i / f c

[0144] The total energy consumption of Task i in the cloud can be expressed as:

[0145] E i c =E i cu_tcans +E i cu_execute

[0146] Total energy consumption E for task i i It is represented as follows:

[0147] E i =E i loc +E i m +E i c ;

[0148] Step 1.9, define the utility function of the energy and time required for the vehicle to complete the task as follows:

[0149]

[0150] in, It is the weight of the delay component of task i. It is the weight of the energy consumption component of task i. T i E represents the total time consumed in task i. i This represents the energy consumption of task i, i.e., the total energy consumed. This represents the processing time corresponding to the longest latency strategy for task i. The processing energy corresponding to the longest energy-consuming strategy for task i is represented as follows:

[0151]

[0152]

[0153] Furthermore, in one embodiment, step 2 uses an improved multi-agent reinforcement learning algorithm to optimize the system's maximization utility function problem, and sets the network training to centralized training and distributed execution. During runtime, each intelligent vehicle can make a decision on the unloading action based on its current environmental state, as follows:

[0154] Step 2.1 defines improving the vehicle's QoE as reducing latency and energy consumption, i.e., maximizing the overall system efficiency.

[0155]

[0156]

[0157]

[0158]

[0159]

[0160] The meanings of the above constraints are as follows:

[0161] C1: The execution time of a task cannot exceed its validity period;

[0162] C2: The allocated transmission bandwidth cannot exceed the existing bandwidth.

[0163] C3: The allocated computing resources strategy cannot exceed the existing amount of resources;

[0164] C4: The amount of data to be unloaded cannot exceed the total amount of data in the task;

[0165] Among them, T i valid Indicates the effective time of task i; Represents any task i that belongs to the task set TASK; This represents the sum of the bandwidth proportions allocated to all tasks by roadside unit j; This indicates the total number of core units allocated to roadside units; This indicates the proportion of data unloaded by task i relative to the total data.

[0166] Step 2.2: Build a multi-agent learning framework and define the state space, action space and reward function: one intelligent vehicle corresponds to one intelligent agent, the state of the intelligent vehicle, edge server and cloud server in each time slot of the service architecture corresponds to the reinforcement learning environment, and the utility function of the task corresponds to the reward of the multi-agent.

[0167] (1) State space

[0168] At each time slot, each intelligent vehicle makes a corresponding decision based on the state information in the environment. Let the state space of intelligent vehicle k at time slot t be defined as follows:

[0169]

[0170] Among them, f loc For the local execution rate of intelligent vehicles, f mec For the computing resources of edge servers, f c For cloud server computing resources, This represents the proportion of bandwidth already allocated to the roadside unit. T represents the proportion of computing resources that have been allocated to and are still being used by edge servers. i loc_left (t) represents the remaining local time for the intelligent vehicle to process task i at time t, E i loc_left (t) represents the local remaining total energy consumption of the intelligent vehicle processing task i at time t, where T is the energy consumption of the vehicle. i mu_left (t) represents the remaining total time for the roadside unit to process task i, E imu_left (t) represents the remaining total energy consumption of the roadside unit processing task i, T i cu_left (t) represents the remaining total time for the cloud server to process task i, E i cu-left (t) represents the remaining total energy consumption of the cloud server in processing task i. y represents the remaining time within the deadline of task i. i γ represents the number of times data has been uploaded for task i. i The task type for task i.

[0171] (2) Action space

[0172] For vehicle k at time slot t, this system model defines five decision variables for vehicle k, encompassing decisions on whether to unload, the unloading location and proportion, and resource allocation. The decision variables can be represented as:

[0173]

[0174] Among them, z i This refers to whether the intelligent vehicle unloads data at time t, z i A value of 1 indicates that data will be unloaded in this time slot, while a value of 0 indicates that data will not be unloaded in this time slot. i This refers to the location where the data is unloaded, x i A value of 0 indicates offloading to a roadside unit, while a value of 1 indicates offloading to a cloud server. i This refers to the proportion of unloaded data to the total data volume. i and α i If and only if z i Valid when M is 1. i This refers to the number of CPU cores allocated to roadside units in this decision. η i This refers to the transmission bandwidth allocated to the roadside units in this decision.

[0175] 3) Reward function

[0176] In the constructed multi-agent reinforcement learning algorithm model, each intelligent vehicle receives a reward value in each time slot, and the sum of all intelligent vehicle reward values ​​is the total reward value for that time slot. To maximize the overall utilization of the vehicles, we improve the training effect of the reinforcement network by allocating the system's wireless and computational resources. Combining the constraints (C1)-(C3), the reward of agent k in time slot t can be calculated. Represented as:

[0177]

[0178] in, The constraints on roadside unit resource allocation and bandwidth allocation are real-time constraints; ti ≤t i valid This indicates whether a task is completed within the time limit, but it's not a real-time constraint; its validity is determined after the task is completed. Therefore, although the intelligent vehicle receives a reward value in each time slot, this reward value is not determined in real-time but rather after the vehicle's current task is completed. This is a negative parameter, representing the penalty for a violation of the decision. The total reward required for reinforcement learning includes the sum of rewards obtained by all agents in the current region, and can be expressed as:

[0179]

[0180] Step 2.3, combined Figure 3 A multi-agent learning network RT-MADDPG under a dynamic model is constructed and trained centrally: for the k-th agent, i.e., intelligent vehicle k, the input of its decision network is the current observation information of intelligent vehicle k. The output is the motion space of the intelligent vehicle k. Each network makes decisions independently of the decision networks of other intelligent vehicles; after each decision, the system stores the experiential data characterized by state space, action space and reward into the experience replay buffer for use in training the network during the centralized execution phase.

[0181] The network training process is completed in a centralized training phase. Assume that K intelligent vehicles participate in this process within the coverage area of ​​roadside units. For the k-th intelligent vehicle, in time slot t, the intelligent vehicle inputs its own observations. Through policy network Get action Described as:

[0182]

[0183] in, It is about exploring noise;

[0184] Based on the execution of the action, obtain the corresponding reward value for that action. And the next state

[0185] Each intelligent vehicle performs the above operation once in each time slot, and... The data is stored in the global experience replay buffer D. When the number of samples in the global experience replay buffer D reaches a preset fixed number, the data in the global experience replay buffer D is randomly selected for training. If the global experience replay buffer D is full, the latest experience data will overwrite the data in the global experience replay buffer D in the order of entry.

[0186] First, define the parameters in the network: in the decision network, the parameter of the eval network is θ, and the parameter of the target network is θ′; in the value network, the parameter of the eval network is ω, and the parameter of the target network is ω′.

[0187] In centralized training, the state of intelligent vehicle k is... and actions By inputting the value network, one can obtain the... Estimate and evaluate the Q-value; calculate the state using the time difference method. and actions The theoretical true value y k :

[0188]

[0189] in For observations of all agents in time slot t+1; For the action decisions of all agents in time slot t+1, the state s of the next time slot is... t+1 Input the target network to get ω′ k This represents the parameter set corresponding to the target network;

[0190] The network error loss function can be obtained by comparing the Q-value output of the value network with the theoretical true value mentioned above.

[0191] L(w k ) = E n [(Q(s t a t ;w k )-y k ) 2 ]

[0192] Where Q(s) t a t ;w k The parameters of the decision network are obtained from the eval network in the value network at time slot t, and this network is deployed on the central server; the parameters of the decision network are updated through the deterministic policy gradient method, and the gradient of its loss function is calculated as follows:

[0193]

[0194] The meaning of the above formula is as follows: The loss gradient under the given condition is, right gradient and For θ n Gradient product;

[0195] By iteratively updating the decision network and value network several times, the optimal decision that maximizes utilization is finally obtained. At this point, the cumulative reward value should be at its maximum value, i.e.:

[0196]

[0197] At this point, the optimal decision network π * Able to achieve γ t-1 r t The total reward value is the maximum value;

[0198] Step 2.4: The distributed network deployed on the intelligent vehicle is updated to a trained decision network. The intelligent vehicle can then input the environmental space into the decision network to obtain a more reasonable action space. Furthermore, all relevant data from the distributed execution is stored in an experience buffer for use in the next centralized execution training.

[0199] Step 2.5, repeat steps 2.3 and 2.4 until both the decision network and the value network have been trained.

[0200] Furthermore, in one embodiment, after the training described in step 3 is completed, each intelligent vehicle only needs to use its local policy network each time to independently complete the unloading decision, and the vehicles do not interfere with each other, as detailed below:

[0201] Step 3.1: Replace the decision network on each smart car with the network updated during the last training session;

[0202] Step 3.2: The value network and experience buffer are no longer used; each vehicle makes an independent unloading decision.

[0203] In one embodiment, an unloading task allocation system for intelligent vehicles in an edge-cloud dynamic unloading framework is provided, the system comprising:

[0204] The first module is used to establish an edge-cloud architecture including a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. It defines the execution rules for dynamic offloading in the edge-cloud architecture, and on this basis, defines a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task consumption time and energy consumption.

[0205] The second module is used to optimize the system's maximum utility function index using an improved multi-agent reinforcement learning algorithm, and sets the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at this moment based on its own environmental state.

[0206] The third module is used to enable each intelligent vehicle to independently make unloading decisions using its local policy network after training is completed.

[0207] Specific limitations regarding the unloading task allocation system for intelligent vehicles within the edge-cloud dynamic unloading framework can be found in the above section on the limitations of the unloading task allocation method for intelligent vehicles within the edge-cloud dynamic unloading framework, and will not be repeated here. Each module in the aforementioned edge-cloud dynamic unloading framework's intelligent vehicle unloading task allocation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0208] The method proposed in this invention is simple to operate and highly practical. While ensuring mission security, it improves the efficiency of the edge-cloud system and reduces the energy consumption of vehicles performing missions.

[0209] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for allocating unloading tasks for intelligent vehicles in an edge-cloud dynamic unloading framework, characterized in that, The method includes: Step 1: Establish an edge-cloud architecture including a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. Define the execution rules for dynamic offloading in this edge-cloud architecture, and on this basis, define a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task consumption time and energy consumption. Step 2: Use the improved multi-agent reinforcement learning algorithm to optimize the system's maximum utility function index, and set the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at this moment based on its own environmental state. Step 1 establishes an edge-cloud architecture comprising a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. It defines the execution rules for dynamic offloading within this architecture and, based on these, defines a task processing model, a task transmission model, a surplus model, and a utility function that integrates task time and energy consumption. Specifically, this includes: Step 1.1, define the description information of intelligent vehicles in the edge-cloud system: V represents the set of intelligent vehicles, and intelligent vehicle k is represented by a tuple. , Describe the performance of the intelligent vehicle itself, among which, This represents the local computing power of the intelligent vehicle k. This represents the transmission power of the kth intelligent vehicle (k). Step 1.2, define the description information of the roadside unit: RSU represents the set of roadside units, the j-th roadside unit Performance through quadruples , , , It means that among them This indicates the CPU operating frequency of a single CPU core in the edge server, where M represents the total number of CPU cores in the edge server. This represents the bandwidth of roadside unit j. express The set of tasks performed by intelligent vehicles within the coverage area; the edge server of the roadside unit provides the total computing power to the intelligent vehicles, represented as... , Step 1.3, define the description information of the cloud server in the edge-cloud system: the performance of cloud server C is expressed by a quadruple. , express, This indicates the single-core CPU operating frequency of the cloud server's CPU kernel. This indicates the number of CPU cores that a cloud server can provide to an intelligent vehicle each time; the computing power that each cloud server can provide to an intelligent vehicle per service is... , This indicates the number of CPU cores provided by the cloud server. Indicates the bandwidth of the cellular network; Step 1.4, define the task model in the edge-cloud system: TASK represents a set of tasks, and the i-th task... Through triplet It means that among them Total workload For task type, This represents the data processing density of the i-th task; Choose 0, 1, and 2, where 0 represents a time-sensitive task, 1 represents a neutral task, and 2 represents an energy-sensitive task. Step 1.5, define the dynamic task offloading rules in the edge-cloud system: starting from the task execution time, at fixed time intervals... As a time slot, every [time interval] Each intelligent vehicle makes a decision on the execution plan for the current task once a time. Intelligent vehicles based on unloading decision variables Determine whether to perform an offload operation in the current time slot. When the value is 0, no uninstallation process is performed, and the existing decision is executed. When the value is 1, unload the process and execute the new decision; when... When it is 1, it is determined by the unloading location variable. The decision is made as to whether the task data for the current time slot is uploaded to the roadside unit or executed on the cloud server. When the value equals 0, the intelligent vehicle k will offload part of the data of task i to the roadside unit for execution; When the value equals 1, intelligent vehicle k will offload part of the data from task i to the cloud server for execution; that is, the two offloading methods cannot be performed simultaneously. If the data is offloaded to the roadside unit for execution, the edge server will allocate... Proportional bandwidth transmission offload data, allocation The number of CPU cores used to process offloaded data, where the allocation ratio is... The range is between 0 and 1, and the allocation is... The number of CPU cores cannot exceed the total number of cores in the device, and CPU cores currently in use cannot be reallocated during this process; if offloaded to a cloud server for execution, the cloud server provides services to each requesting intelligent vehicle task. Computing resources; This indicates the total number of times task i was unloaded. ,in , The maximum value is determined by the number of time slots covered by the task's local execution; the start time of task i is set to... The time of the first unloading process is set to... The percentage of unloaded data in the total data volume is: The amount of data to be unloaded is The time of the second unloading process is set to... The percentage of unloaded data in the total data volume is: The amount of data to be unloaded is And so on, the time of the nth unloading process is set to... The percentage of unloaded data in the total data volume is: The amount of data to be unloaded is , , ... The initial value is 0. , ... The initial value is set to ∞, correspondingly, This represents the time of the nth transmission. Step 1.6, define the task data transmission model and task processing model in the edge-cloud system: When the intelligent vehicle decides to unload and execute a task, if task i is unloaded and executed for the nth time, then its transmission time is expressed as: in for The amount of tasks that are constantly being unloaded. Indicates transmission bandwidth. This represents the transmission power of the intelligent vehicle k. Indicates the channel gain during transmission; when At that time, task data is unloaded from the intelligent vehicle to the roadside unit. ;when At that time, task data is unloaded from the intelligent vehicle to the cloud server. Corresponding transmission power consumption Represented as: in, The transmission power of intelligent vehicle k; Furthermore, using This indicates the time it takes for the task data to be transmitted to the roadside unit. This indicates the time it takes to be transmitted to the cloud server, and specifies that: Task execution is divided into three execution modes: local task execution by the intelligent vehicle, task execution by the roadside unit, and task execution by the cloud server; the latency of task i during a certain decision-free period is described as follows: in, This indicates the amount of data on the device. For data density, For computing resources; when data is executed locally, For the computing resources of local vehicles, When data is processed in the roadside unit, Assign computing resources to task i on the edge server, assuming the previous decision order was n, then ,in The number of CPU cores allocated to task i during the nth unloading; when data is executed on the cloud server. Allocate computing resources to task i on the cloud server. ; The execution energy consumption corresponding to the above execution time is expressed as follows: in, For the number of device cores, To calculate the frequency correlation coefficient, this parameter depends on the hardware capabilities of the cloud server's CPU. This refers to the number of clock cycles for a single-core CPU in this device. The above execution time; When the execution location is an intelligent vehicle, the intelligent vehicle is locally regarded as having a single-core CPU. =1, Take the local calculation frequency correlation coefficient , for When the execution location is a roadside unit, The number of CPU cores allocated for the previous decision, assuming the order of the previous decision is n, then... , Take the edge calculation frequency correlation coefficient , for When the execution location is a cloud server, Allocate a fixed number of cores to cloud servers , Take the frequency correlation coefficient of cloud computing Step 1.7: Establish a residual quantity model for evaluating the real-time status of the edge-cloud system. This residual quantity model includes: remaining data volume, remaining time, and remaining energy consumption. The remaining data volume refers to the amount of data that the current device needs to execute but has not yet executed, based on observations and calculations from the current moment, if the current decision is maintained. Remaining time refers to the time required for the current device to process all the data if the current decision is maintained, based on the current observation. Residual energy consumption refers to the total energy consumed by the equipment in the remaining time if the current decision is maintained; Step 1.8: Calculate the total time and total energy consumed by the task using the residual quantity model; Step 1.9, define the utility function of the energy and time required for the vehicle to complete the task as follows: in, It is the weight of the delay component of task i. It is the weight of the energy consumption component of task i. ; This represents the time spent on task i, i.e., the total time consumed. This represents the energy consumption of task i, i.e., the total energy consumed. This represents the processing time corresponding to the longest latency strategy for task i. The processing energy corresponding to the longest energy-consuming strategy for task i is represented as follows: Total time spent on task i This represents the maximum value of local time, edge time, and cloud server time, as detailed below: In the formula, This indicates the total local time. Indicates the total time spent at the edge. Indicates the total time spent in the cloud; Total energy consumption of task i , represented as: This indicates the total local energy consumption. Indicates the total time spent at the edge. This indicates the total energy consumption in the cloud.

2. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 1, characterized in that, The remaining data volume of the intelligent vehicle (i.e., local, roadside unit, and cloud server) in step 1.7 is as follows: (1) Remaining local data volume Indicates; when If local data has not yet been unloaded, then the remaining local data volume equals the total local data volume minus the locally executed data volume, i.e.: When the nth uninstallation decision has been given, and the (n+1)th uninstallation decision has not yet been given, that is... The remaining local data volume is represented as: in, , express The amount of data completed within the time period. Indicates the amount of data that has been unloaded; Regarding decision Each needs to satisfy Decision constraints within a time period; when hour, It has no practical significance; it can be regarded as ,Right now Right now, The amount of data unloaded at any given moment Cannot exceed Remaining local data volume at any given time , The moment is Before that moment, and infinitely close. The constraints at any given moment are as follows: (2) Remaining data volume of roadside units This indicates that the remaining data volume of the roadside unit is 0 before the first unloading and transmission is completed; When the nth unloading decision is given, and the nth unloading data has not yet been completely transmitted, that is... The remaining data volume of the roadside unit is represented as The amount of data remaining at time minus the amount of data at time 1 The amount of data completed from the specified time, i.e.: in, , express The amount of data remaining at each moment. express The amount of data completed within the time period; When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, that is... The remaining data volume of the roadside unit is represented as follows: in, , This represents the data from the nth uninstallation. express The amount of data remaining at a given time. express The amount of data completed within the time period; (3) The remaining data volume of the cloud server refers to the amount of data that has been unloaded to the cloud server but has not yet been executed. This indicates that the cloud server has 0 data remaining before the first unloading and transfer is complete. When the nth unloading transfer is completed, but the (n+1)th unloading transfer is not yet completed, that is... The remaining data volume on the cloud server is represented as follows: in, , This represents the data from the nth uninstallation. express The amount of data remaining at a given time. express The amount of data completed within the time period.

3. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 2, characterized in that, In step 1.7, the remaining time for the intelligent vehicle, including the local unit, roadside unit, and cloud server, is as follows: (1) Local remaining time refers to the time required to execute the current local remaining data if the current decision remains unchanged at time t and thereafter, expressed as : (2) The remaining time of the roadside unit refers to the time required to execute the remaining data of the roadside unit if the current decision remains unchanged at time t and thereafter; Before the first unloading decision, the roadside unit has 0 remaining time; when the nth unloading decision is given, and the nth unloading data transmission has not yet been completed, i.e. The remaining time for the roadside unit is expressed as : in, , Indicates from Starting from time n, the remaining execution time required for all data except for the data unloaded in the nth time; This represents the remaining transmission time for the nth data unloading operation. Indicates from Starting from time n, the remaining execution time required for the nth data unloading; When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, that is... Roadside unit remaining time It is represented as follows, where : (3) The remaining time of the cloud server refers to the time required to execute the remaining data of the server if the current decision remains unchanged at time t and thereafter; Before the first uninstallation decision is completed, the remaining time on the cloud server is 0; when the nth uninstallation decision is given, and the nth uninstallation data has not yet been completely transmitted, i.e. Similarly, the remaining time for roadside units and the remaining time for cloud servers are also relevant. It is represented as follows, where : in, Indicates from Starting from time n, the remaining execution time required for all data except for the data unloaded in the nth time; This represents the remaining transmission time for the nth data unloading operation. Indicates from Starting from time n, the remaining execution time required for the nth data unloading; When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, that is... Similarly, the remaining time for the roadside unit is represented as follows: :

4. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 3, characterized in that, In step 1.7, the remaining energy consumption of the intelligent vehicle, including the local unit, roadside unit, and cloud server, is as follows: (1) Local residual energy consumption refers to the energy consumption of executing local residual data if the current decision remains unchanged at time t and thereafter. Represented as: (2) The remaining energy consumption of the roadside unit refers to the energy consumption of transmitting and executing the remaining data of the roadside unit if the current decision remains unchanged at time t and thereafter; Before the first unloading decision, the remaining energy consumption of the roadside unit is 0; the remaining energy consumption of the roadside unit is divided into two parts: the remaining energy consumption of the roadside unit execution and the remaining energy consumption of the roadside unit transmission. When the nth unloading decision is given, and the nth unloading data transmission has not yet been completed, that is... Roadside units perform remaining energy consumption It is represented as follows, where : When the nth unloading data transmission is completed, and the (n+1)th unloading decision has not yet been given, that is... The remaining energy consumption of the roadside unit is represented as follows, where : Remaining transmission power consumption of roadside units It is expressed as follows: Based on the above formula, the remaining energy consumption of the roadside unit can be obtained. for: (3) The remaining energy consumption of the cloud server refers to the energy consumption for transmitting and executing the remaining data of the cloud server if the current decision remains unchanged at time t and thereafter; the remaining energy consumption of the cloud server is divided into two parts: the remaining energy consumption of the roadside unit execution and the remaining energy consumption of the roadside unit transmission. Remaining transmission power consumption of cloud server It is expressed as follows: Remaining execution power consumption of cloud server It is represented as follows, where : Based on the above formula, the remaining energy consumption of the cloud server can be obtained. for: 。 5. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 4, characterized in that, Step 1.8, which involves calculating the total time and energy consumed by the task using the residual quantity model, specifically includes: (1) The local time of task i refers to the time from the start of task i execution to the completion of the local task execution; according to the definition of local remaining time, the total local time is calculated. Represented as: in, Indicates task When the number of uninstalls is 0, meaning the entire process is executed locally, the task... The total local execution time is The local remaining time at any given moment; similarly, ... Representing tasks Cases where the number of uninstallations is 1, ..., N; Indicates task Uninstall count The situation, the task The total local execution time is The local remaining time plus the start time at the moment arrive The time between; The total edge latency of task i refers to the time from the local execution time of task i, including the waiting time before the roadside unit receives the unloading data from task i, to the end of the last execution of sub-data of task i by the roadside unit; according to the definition of the remaining time of the roadside unit, the total edge latency is... Represented as: in, Indicates task When the number of uninstalls is 0, meaning the entire process is executed locally, the task... The total local execution time is The remaining time of the roadside cell at time 1. If the initial roadside cell had no task data, then at this time... It is 0; similarly, ... Representing tasks Cases where the number of uninstallations is 1, ..., N; Indicates task Uninstall count The situation, the task The total execution time of the roadside unit is The remaining time of the roadside unit at any given time plus the start time. arrive The time between; The total cloud execution time for task i refers to the time from the local execution time of task i, including the waiting time before the cloud server receives the unloaded data of task i, to the end of the last execution of the sub-data of task i by the cloud server; according to the definition of the remaining time of the cloud server, the total cloud execution time is calculated as follows. Represented as: in, Indicates task When the number of uninstalls is 0, meaning the entire process is executed locally, the task... The total local execution time is The remaining time on the cloud server at any given moment. If the initial cloud server has no task data, then at this time... It is 0; similarly, ... Representing tasks Cases where the number of uninstallations is 1, ..., N; It indicates that the task Uninstall count The situation, the task The total execution time of the cloud server is The remaining time on the cloud server plus the start time. arrive The time between; (2) The total energy consumption of task i is the sum of the local total energy consumption, the edge total energy consumption, and the cloud total energy consumption; The total local energy consumption is related to the total local execution time, and is expressed as follows: : Total edge power consumption includes total transmission power consumption and total execution power consumption; Among them, the total energy consumption of edge transmission in task i for: Total energy consumption for edge execution of task i for: in, This indicates that for task i, the edge server uses computing resources. Execution time; if the nth uninstallation does not exist, i.e. Then we have: If the nth uninstallation is the last uninstallation, that is... Then we have: in, This indicates the total amount of data that needs to be processed, including... The remaining data volume of the roadside unit at time 1 and the first The decision to unload the amount of data that has not yet arrived; For the first Edge computing resources allocated in the next decision-making phase; If the nth uninstallation is not the last uninstallation, that is... Then we have: in, express During this period, computing resources The duration used by task i; express During this period, computing resources The duration used by task i; The total edge energy consumption of task i can be expressed as: Total cloud energy consumption includes total transmission energy consumption and total execution energy consumption; Among them, the total energy consumption of transmission for task i for: Total energy consumption for cloud execution Represented as: in, The total execution time in the cloud is represented as: The total energy consumption of task i in the cloud Represented as: 。 6. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 5, characterized in that, Step 2 describes using an improved multi-agent reinforcement learning algorithm to optimize the system's maximum utility function index, and setting the network to operate with centralized training and distributed execution. During operation, each intelligent vehicle makes a decision on the unloading action based on its current environmental state, specifically including: Step 2.1 defines improving the vehicle's QoE as reducing latency and energy consumption, i.e., maximizing the overall system efficiency. The meanings of the above constraints are as follows: The execution time of a task cannot exceed its validity period; The allocated transmission bandwidth cannot exceed the existing bandwidth. C3: The allocated computing resources strategy cannot exceed the existing amount of resources; C4: The amount of data to be unloaded cannot exceed the total amount of data in the task; in, Indicates task The effective time; Indicates any belonging to the task set Tasks in ; Represents roadside unit The sum of the bandwidth proportions allocated to all tasks; This indicates the total number of core units allocated to roadside units; Indicates task The percentage of uninstalled data out of the total data; Step 2.2: Build a multi-agent learning framework and define the state space, action space and reward function: one intelligent vehicle corresponds to one intelligent agent, the state of the intelligent vehicle, edge server and cloud server in each time slot of the service architecture corresponds to the reinforcement learning environment, and the utility function of the task corresponds to the reward of the multi-agent; (1) State space At each time slot, each intelligent vehicle makes a corresponding decision based on the state information in the environment; Define the state space of intelligent vehicle k at time slot t as follows: : in, For the local execution rate of intelligent vehicles, For the computing resources of edge servers, For cloud server computing resources, This represents the proportion of bandwidth already allocated to the roadside unit. This represents the proportion of computing resources that have been allocated to and are still being used by edge servers. Let t be the total remaining local time for the intelligent vehicle to process task i. Let be the local remaining total energy consumption of the intelligent vehicle processing task i at time t. The remaining total time for processing task i in the roadside unit. The remaining total energy consumption for processing task i in the roadside unit. The remaining total time for the cloud server to process task i. The remaining total energy consumption for processing task i on the cloud server. The remaining time within the deadline for task i. This represents the number of times data has been uploaded for task i. For task i, the task type; (2) Action space For intelligent vehicle k at time slot t, this system model defines five decision variables for intelligent vehicle k, including decisions on whether to unload, the location and proportion of unloading, and resource allocation; decision variables Represented as: in, This indicates whether the intelligent vehicle has unloaded data at time t. A value of 1 indicates that data is selected to be unloaded in this time slot. A value of 0 indicates that no data is unloaded in this time slot; Indicates the location where the data is unloaded. A value of 0 indicates unloading to a roadside unit. A value of 1 indicates that the data is being offloaded to a cloud server. This indicates the proportion of unloaded data to the total data volume; and If and only if It is valid when the value is 1; This indicates the number of CPU cores in the roadside units allocated in this decision. This indicates the transmission bandwidth allocated to the roadside unit in this decision; (3) Reward function Combining the constraints C1 to C3 above, the reward for intelligent vehicle k in time slot t is... Represented as: in, The constraints are for resource allocation and bandwidth allocation for roadside units, and are real-time constraints. This indicates whether the task is completed within the time limit. It is not a real-time constraint and needs to be determined after the task ends. A negative parameter serves as the penalty for illegal decisions; the total reward required for reinforcement learning. Including the total reward obtained by all agents in the current region, expressed as: Step 2.3: Construct the RT-MADDPG multi-agent learning network under the dynamic model and perform centralized training: For the k-th agent, i.e., intelligent vehicle k, the input of its decision network is the current observation information of intelligent vehicle k. The output is the motion space of the intelligent vehicle k. Each network makes decisions independently of the decision networks of other intelligent vehicles; after each decision, the system stores the experiential data characterized by state space, action space and reward into the experience replay buffer for use in training the network during the centralized execution phase. The network training process is completed in a centralized training phase. Assume that K intelligent vehicles participate in this process within the coverage area of ​​roadside units. For the k-th intelligent vehicle, in time slot t, the intelligent vehicle inputs its own observations. Through policy network , to obtain action The description is as follows: in, It is about exploring noise; Based on the execution of the action, obtain the corresponding reward value for that action. and the next state ; Each intelligent vehicle performs the above operation once in each time slot, and... The data is stored in the global experience replay buffer D. When the number of samples in the global experience replay buffer D reaches a preset fixed number, the data in the global experience replay buffer D is randomly selected for training. If the global experience replay buffer D is full, the latest experience data will overwrite the data in the global experience replay buffer D in the order of entry. First, define the parameters in the network: the parameters of the eval network in the decision network are... The parameters of the target network are The parameters of the eval network in the value network are: The parameters of the target network are ; In centralized training, the state of intelligent vehicle k is... and actions By inputting the value network, one can obtain the... Estimate and evaluate the Q-value; calculate the state using the time difference method. and actions Theoretical true value : in , represents the observations of all agents in time slot t+1; For the action decisions of all agents in time slot t+1, the state of the next time slot will be determined. Input the target network to get ; This represents the parameter set corresponding to the target network; The network error loss function can be obtained by comparing the Q-value output of the value network with the theoretical true value mentioned above. in, The values ​​are derived from the eval network in time slot t, which is deployed on a central server; the decision network parameters are updated using a deterministic policy gradient method, and the gradient of its loss function is calculated as follows: The meaning of this formula is that... The loss gradient in the case is right gradient and right Gradient product; By iteratively updating the decision network and value network several times, the optimal decision that maximizes utilization is finally obtained. At this point, the cumulative reward value should be at its maximum value, i.e.: The optimal decision network at this time Capable of achieving The total reward value is the maximum value; Step 2.4: The distributed network deployed on the intelligent vehicle is updated to the trained decision network. The intelligent vehicle then inputs the environmental space into the decision network to obtain the action space; and stores all relevant data of the distributed execution into the experience buffer for use in the next centralized execution training. Step 2.5, repeat steps 2.3 and 2.4 until both the decision network and the value network have been trained.

7. The method for allocating unloading tasks for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 1, characterized in that, The method also includes: Step 3: After training, each intelligent vehicle independently makes unloading decisions using its local policy network; specifically including: Step 3.1: Replace the decision network on each intelligent vehicle with the decision network updated during the last training process; Step 3.2: Without using a value network and experience buffer, each intelligent vehicle independently makes the unloading decision.

8. An edge-cloud dynamic unloading task allocation system for intelligent vehicles based on the method of any one of claims 1 to 7, characterized in that, The system includes: The first module is used to establish an edge-cloud architecture including a cloud server, multiple roadside units with deployed edge servers, and multiple smart vehicles. It defines the execution rules for dynamic offloading in the edge-cloud architecture, and on this basis, defines a task processing model, a task transmission model, a remaining quantity model, and a utility function that integrates task consumption time and energy consumption. The second module is used to optimize the system's maximum utility function index using an improved multi-agent reinforcement learning algorithm, and sets the network to operate in a centralized training and distributed execution mode. During operation, each intelligent vehicle makes a decision on the unloading action at that moment based on its own environmental state.

9. The unloading task allocation system for intelligent vehicles in the edge-cloud dynamic unloading framework according to claim 8, characterized in that, The system also includes: The third module is used to enable each intelligent vehicle to independently make unloading decisions using its local policy network after training is completed.

Images