Digital twin assisted vehicle task unloading and computing resource allocation optimization method

Abstract

The invention belongs to the technical field of Internet of Vehicles wireless communication, and particularly relates to a digital twinborn assisted vehicle task unloading and computing resource allocation optimization method, which comprises the following steps: acquiring information of vehicles and RSUs, constructing vehicle twinborn bodies and RSU twinborn bodies, and constructing a digital twinborn network; total time delay and total cost of the system are determined based on the digital twinborn network, the total time delay comprises local calculation time delay, vehicle twinborn auxiliary calculation time delay, GAP errors, uplink time delay and RSU calculation time delay, and the total cost comprises RSU calculation cost and vehicle twinborn auxiliary calculation cost; establishing an optimization problem model according to the total time delay and the total cost of the minimized system; and solving the optimization problem model by the digital twin network. According to the method, the average time delay of the system can be effectively reduced in a typical urban traffic scene, the resource use cost is reduced, and the collaborative advantage of the digital twinning technology and the reinforcement learning framework in edge computing resource optimization is improved.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology for connected vehicles, and in particular to a method for optimizing vehicle task offloading and computing resource allocation assisted by digital twins. Background Art

[0002] With the booming development of intelligent transportation and Internet of Vehicles, the data generated by vehicles is growing exponentially. According to statistics, in the scenario of autonomous driving, the amount of data generated by a single car can reach several GB per hour. The massive data processing demand makes it difficult for traditional cloud computing models to meet the strict requirements of vehicles for real-time computing, and vehicle edge computing technology has emerged. This technology performs computing tasks at the edge of the network close to the vehicle, effectively reducing data transmission delays and improving system response speed. It provides key support for applications such as autonomous driving decision-making and real-time road condition monitoring, and has become an important force in promoting the development of intelligent transportation.

[0003] Vehicle edge computing faces many challenges in practical applications. On the one hand, the computing resources of vehicles and RSUs (roadside unit servers) are limited and in dynamic change. When a vehicle is driving, the continuous changes in its position and speed will affect the communication quality with the RSU and the computing resources that can be obtained; the differences in hardware configurations of different vehicles and the changes in load during operation also make the computing capabilities of the vehicles themselves uneven. On the other hand, the existing task offloading and computing resource allocation strategies are insufficient, making it difficult to fully utilize computing resources, resulting in long task processing delays. Under the traditional offloading strategy, the average processing delay of complex tasks is high, which cannot meet real-time requirements, and there is a lack of effective incentive mechanisms to mobilize the enthusiasm of vehicles to participate in auxiliary computing. Summary of the invention

[0004] In view of this, the present invention proposes a digital twin-assisted vehicle task offloading and computing resource allocation optimization method, constructs a digital twin network architecture, utilizes the interaction between the vehicle and RSU twins and the physical layer to obtain multi-dimensional information, establishes task calculation, partial offloading and cost models, uses the digital twin network to obtain the optimal offloading strategy, utilizes the characteristics of the vehicle twin that has vehicle status information and task information to assist the vehicle in task calculation, establishes an optimization model, uses the A3C algorithm to solve the optimization problem model, and then obtains the optimal offloading decision and resource allocation plan, realizes the optimization of vehicle edge offloading, and provides a strong guarantee for the efficient operation of the intelligent transportation system.

[0005] In a first aspect, an embodiment of the present invention provides a vehicle task offloading and computing resource allocation optimization method assisted by a digital twin, including: Obtain vehicle and RSU information, build vehicle twins and RSU twins, and build a digital twin network; Determine the total delay and total cost of the system based on the digital twin network, wherein the total delay includes local computing delay, vehicle twin assisted computing delay, GAP error, uplink delay and RSU computing delay, and the total cost includes RSU computing cost and vehicle twin assisted computing cost; Establish an optimization problem model based on minimizing the total delay and total cost of the system; The digital twin network solves the optimization problem model.

[0006] In a possible implementation, the total delay The calculation formula is: , , in, N represents the total number of vehicles, T Indicates the total number of time slots, Indicates the task calculation delay, Indicates the uplink delay. Indicates the RSU calculation delay, Indicates the total delay of vehicle users.

[0007] In one possible implementation, the uplink delay for: , in, Indicates the task transfer rate, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The size of the tasks generated within; RSU calculation delay for: , in, Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates that RSU is allocated to the task computing resources.

[0008] In one possible implementation, the total delay of the vehicle user for: , , , , in, Indicates the local computing latency, represents the vehicle twin-assisted computing delay, Gap represents the GAP error, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates vehicle n The computing power of The vehicle twin maps the vehicle’s computational capabilities through a virtual representation, Represents the computational power error generated during the vehicle twin mapping process.

[0009] In one possible implementation, the total cost for: , , , in, Indicates the RSU calculation cost, represents the vehicle twin auxiliary computing cost, N represents the total number of vehicles, T Indicates the total number of time slots, Indicates vehicle n In time slot t Uninstall ratio within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates the unit price of RSU calculation, Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within Represents the unit price of vehicle twin assisted computing.

[0010] In a possible implementation, the optimization problem model is: , in, represents the delay weight factor, represents the price weight factor,x represents the uninstall decision, Indicates that RSU is allocated to the task of computing resources, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within represents the total delay, represents the total cost; The model satisfies the following constraints: , , , , , in, Indicates the task calculation delay, Indicates the maximum tolerable delay, Indicates vehicle n No. m The offloading strategy of each task, Indicates the maximum computing resource of the RSU.

[0011] In one possible implementation, the digital twin network solves the optimization problem model as follows: Step S41, constructing state space, action space and reward function; Step S42, initializing the global Actor network parameters and the global Critic network parameters, and initializing the local Actor network parameters and the local Critic network parameters in each parallel asynchronous thread to the global Actor network parameters and the global Critic network parameters; Step S43, execute all threads in parallel, and obtain the current state of each thread during its execution. , the local Actor network generates an action probability distribution based on the current state, and randomly samples the action from the action probability distribution ; Perform sampling actions in the environment and get timely rewards and the next state ; Experience To store; Step S44, the local critic network calculates the temporal difference error based on experience; Step S45, calculating the Actor network loss and the Critic network loss respectively based on the temporal difference error, and updating the local Actor network parameters and the local Critic network parameters respectively; Step S46, looping steps S43-S45 until the loop termination condition is met, and outputting the optimal strategy represented by the global Actor network.

[0012] In one possible implementation, the state space is: ,in, Indicates vehicle n In time slot t The tasks generated within Indicates vehicle n The twins of Indicates n Vehicle and k The distance between RSUs; The action space is: ; The reward function is: .

[0013] In one possible implementation, the timing differential error for: , in, Indicates that the agent is t The instant reward after performing the action at any time, represents the discount factor, Represents the value estimate of the local Critic network in the next state, Represents the value estimate of the local Critic network in the current state.

[0014] In one possible implementation, the calculation formula for updating the local Actor network parameters is: , , in, Represents local Actor network parameters, represents the learning rate of the Actor network, represents the gradient of the Actor network loss with respect to the network parameters, Indicates Actor network loss, Represents the output action probability distribution of the local Actor network; The calculation formula for updating the local Critic network parameters is: , , in, Represents the local Critic network parameters, represents the learning rate of the Critic network, Represents the gradient of the Critic network loss with respect to the network parameters, Represents the Critic network loss.

[0015] The present invention applies digital twin technology to mobile edge computing networks. By constructing a digital twin network architecture, a task offloading model, and a task computing model, combined with information interaction between the twin layer and the physical layer, the computing task parameters are comprehensively and accurately acquired; a partial offloading mode is adopted, an offloading and cost model is designed, and price incentives are set for vehicle twins to assist in computing; the vehicle twin and the RSU twin together constitute a digital twin network, jointly execute the A3C algorithm, and collaboratively achieve the optimal decision for task offloading, achieve the optimal offloading decision and computing resource allocation, and after obtaining the optimal decision, use the vehicle twin as a computing node to assist the vehicle in computing the tasks that are not offloaded to the RSU. The method of the present application effectively improves computing efficiency and accuracy, significantly reduces task processing latency and computing costs, can adapt to dynamically changing environments, optimizes computing resource allocation, and provides a strong guarantee for the efficient operation of vehicle edge computing. Compared with the prior art, the present invention has achieved the following significant technical effects.

[0016] (1) The present invention sets the vehicle twin and the RSU twin in the cloud. The vehicle twin and the RSU twin together constitute a digital twin network, and information is shared between the vehicle twin and the RSU twin. By constructing a virtual representation in the digital twin network, the characteristic states of the physical vehicle and RSU and the prediction of the system can be reflected in real time, thereby achieving better computing resource scheduling, achieving efficient utilization of computing resources and maximizing the interests of both parties.

[0017] (2) The present invention innovatively proposes to use the digital twin technology to build a twin to assist the physical layer entity in task calculation, and adopts "total cost" as the optimization indicator. By integrating the RSU computing cost and the vehicle twin auxiliary computing cost, the vehicle twin is encouraged to participate in the vehicle task calculation. Different from the existing technology that is limited to energy consumption or resource occupation, the present invention comprehensively covers the cost elements of the calculation process from the comprehensive cost dimension, while ensuring the task completion delay, making full use of the computing nodes in the system to complete the task calculation.

[0018] (3) The present invention combines digital twin technology with the A3C algorithm to achieve optimal offloading decisions and computing resource allocation. Digital twin technology maps the state of the physical system in real time by constructing a virtual model that is highly similar to the physical entity. The A3C algorithm, as an asynchronous and parallel deep reinforcement learning algorithm, can use the digital twin environment to evaluate and update strategies simultaneously in multiple threads based on the large amount of real-time data provided by different twins. The parallel learning characteristics of the A3C algorithm can significantly accelerate the convergence speed of the model and shorten the time for strategy optimization. The high consistency between the twins constructed by digital twin technology and the physical entity, and the experience and strategies learned by the A3C algorithm in the digital twin environment, can more accurately balance the delay and cost factors, effectively reduce the overall system loss, and can be effectively migrated to the actual physical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic flow chart of a digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in an embodiment of the present invention; Figure 2 A scenario model diagram of a digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in an embodiment of the present invention; Figure 3 The optimal offloading solution and the simulation diagram of the minimum system cost obtained by the A3C algorithm under the digital twin framework in the digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in the embodiment of the present invention; Figure 4 A comparison chart of system simulation costs of different calculation methods based on obtaining the optimal unloading solution provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0024] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0025] See also Figure 1 , which is a flow chart of a digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in an embodiment of the present invention. Figure 1 As shown, it mainly includes the following steps.

[0026] Step S01, obtain vehicle and RSU information, build vehicle twins and RSU twins, and build a digital twin network. Specifically: Apply digital twin technology to mobile edge computing networks, map vehicles and RSUs (roadside unit servers) to build vehicle twins and RSU twins, and use virtual representation to set the data update frequency so that the twins can reflect the characteristics of their corresponding physical devices in real time and predict the status of the physical devices, thereby establishing digital twin networks and digital twin-assisted vehicle edge computing models.

[0027] The vehicle twin interacts with the physical layer vehicle uninterruptedly to obtain information such as vehicle location, speed, and computing power; the RSU twin obtains information such as the remaining computing resources, channel resources, and road conditions of the physical layer RSU, and uses the digital twin network to assist the physical layer in making offloading decisions. The vehicle twin stores all information about the tasks generated by the vehicle and has computing resources. After the digital twin network assists the vehicle in offloading tasks to the RSU, the vehicle twin is used to assist in vehicle task calculations and establish a task calculation model.

[0028] It should be noted that regardless of whether the vehicle offloads tasks to the RSU twin, the vehicle twin assists the vehicle in task calculation, and the assistance ratio is jointly constrained by the reward mechanism and delay constraints.

[0029] See also Figure 2 , which is a scenario model diagram of the digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in an embodiment of the present invention. Figure 2As shown in the figure, the vehicle twin and RSU twin are set in the cloud, and the vehicle twin and RSU twin together constitute a digital twin network, and information is shared between the twins. By building a virtual representation in the digital twin network, the characteristic state of the physical layer and the prediction of the system are reflected in real time. , the vehicle twin set is , the RSU set is , the RSU twin set is .

[0030] In the time slot set Each vehicle generates a delay-sensitive task in each time slot , n Indicates the vehicle number, t Indicates the time slot number, ,in, Indicates vehicle n In time slot t The size of the task generated within (referred to as "task size"), Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within (which can be referred to as "CPU cycles required to complete the task"), Indicates the completion of the vehicle n In time slot t The maximum tolerable delay of tasks generated within (referred to as "maximum tolerable delay" for short).

[0031] The vehicle twin continuously interacts with the physical vehicle through the digital twin network to obtain information about the physical vehicle. ,in and Respectively represent vehicles n The position and velocity of Indicates vehicle n The computing power of The vehicle twin maps the vehicle’s computational capabilities through a virtual representation, Represents the computing power error generated during the vehicle twin mapping process. While the vehicle twin provides vehicle information to the digital twin network, it uses the computing power of the mapped physical layer vehicle to assist the vehicle in task calculations.

[0032] Step S02, based on the digital twin network, determine the total delay and total cost of the system, the total delay includes local computing delay, vehicle twin auxiliary computing delay, GAP error, uplink delay and RSU computing delay, the total cost includes RSU computing cost and vehicle twin auxiliary computing cost. Specifically: A task calculation model is established. In the scenario of computing task offloading, the task transmission rate of the uplink delay is calculated based on the task transmission rate. A partial offloading mode is adopted to construct the task delay formulas in RSU calculation, local calculation, and vehicle twin-assisted calculation respectively. The errors in the twin and main computing capabilities are considered at the same time to derive the task calculation delay formula. The total delay formula of the computing task is determined by integrating local, vehicle twin, and RSU computing resources.

[0033] The offloading and cost model is designed, and partial offloading is adopted. The task delay is jointly affected by local computing, the proportion of vehicle twin auxiliary computing, and the proportion of RSU partial offloading. The RSU computing unit price and the vehicle twin auxiliary computing unit price are given, and the cost formulas for RSU to complete the task and the cost formulas for vehicle twin auxiliary computing to complete the task are constructed respectively. Furthermore, in order to encourage vehicle twin auxiliary computing, the vehicle twin auxiliary computing unit price is made higher than the RSU computing unit price.

[0034] For the offloading decision, partial offloading is considered. For the offloading action, the present invention considers 0 to indicate no offloading to the RSU, and the system's task completion delay is affected by the ratio of local calculation and vehicle twin auxiliary calculation; 1 indicates offloading to the RSU, and the task delay is affected by three parts: local calculation, vehicle twin auxiliary calculation ratio, and RSU offloading ratio. n In time slot t The uninstall ratio is (referred to as "unloading ratio"), and meets ,but Indicates vehicle n In time slot t The proportion of vehicle users in the calculation; n In time slot t The proportion of vehicle twin-assisted calculations performed within (referred to as “vehicle twin auxiliary calculation ratio”) and meets ,but Indicates vehicle n In time slot t The proportion of local calculations within.

[0035] Local computing latency of some tasks in local computing for: , in, Indicates the local computing latency, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within (which can be referred to as "CPU cycles required to complete the task"), Indicates vehicle n computing power.

[0036] When the vehicle twin performs auxiliary calculation, the vehicle twin auxiliary calculation delay for: , in, Representing the vehicle twin maps the computing power of the vehicle through a virtual representation.

[0037] It should be noted that although digital twin technology can map physical layer parameters in real time, in actual situations, there is a gap in the computing power of different twins and entities. Gap , GAP error Gap The calculation formula is: , in, Represents the computational power error generated during the vehicle twin mapping process.

[0038] In summary, the total vehicle-user delay considering vehicle twin-assisted computing is for: , When offloading some tasks to the RSU, the uplink latency is considered first. , the calculation formula is: , in, Indicates the uplink delay. Indicates the task transfer rate, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The size of the task generated within.

[0039] Furthermore, the task transfer rate The calculation formula is: , in, represents bandwidth, Represents the signal-to-noise ratio.

[0040] RSU calculation delay of some tasks offloaded to RSU for: , in, Indicates the RSU calculation delay, Indicates vehicle n In time slot t Uninstall ratio within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates that RSU is allocated to the task computing resources (referred to as “RSU allocatable computing resources”).

[0041] In summary, when computing tasks are offloaded to the RSU and assisted by the vehicle twin, the task computing delay is for: , It can be concluded that the total delay of the system for: , in, N represents the total number of vehicles, T Indicates the total number of time slots.

[0042] Based on the above embodiments, this embodiment further provides a method for calculating the total cost of completing a task.

[0043] The vehicle twin and the RSU twin are placed in the cloud to form a global virtual layer, in which decision-making and offloading ratio optimization are performed, and the vehicle twin is used to assist in completing vehicle task calculations based on the optimization results.

[0044] Assume that the unit price of RSU calculation is , then the RSU calculation cost required to complete the task is for: , Digital twin technology can map the physical layer information. When a vehicle generates a task, the vehicle twin can obtain all the information of the task. The present invention considers using the vehicle twin to assist in computing to reduce the time delay of task completion. Assume that the unit price of the vehicle twin for vehicle twin assistance computing is , then the vehicle twin assisted computing cost that needs to be paid to the resource provider is for: , in, Indicates vehicle n In time slott Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within the vehicle.

[0045] Then, the total cost required to complete all vehicle tasks in the system is calculated for: , It should be noted that since the cloud-based digital twin network allows the vehicle twin to assist in calculations while making the optimal offloading decision, the present invention takes into account that digital twin technology requires huge computing resources and construction costs, such as cloud service fees, physical equipment maintenance costs, etc. Therefore, the price of vehicle twin assisted calculations is considered to be higher than the price of RSU offloading calculations to incentivize cloud-based vehicle twin assisted calculation tasks.

[0046] Step S03: Establish an optimization problem model based on minimizing the total delay and total cost of the system. Specifically: In order to minimize the total delay and total cost of the system, an optimization problem model is established by jointly optimizing the unloading decision of vehicle users, the computing resource allocation of RSUs, the ratio of twin-assisted computing, and the ratio of RSU unloading. The optimization problem model is: , in, represents the delay weight factor, represents the price weight factor, x represents the uninstall decision, Indicates that RSU is allocated to the task of computing resources, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within represents the total delay, Indicates the total cost.

[0047] The model satisfies the following constraints: , , , , , in, Indicates the task calculation delay; Indicates the maximum tolerable computation delay; Indicates vehicle n of m The offloading strategy of a task is , indicating vehicle n of m A task is not uninstalled, if , indicating vehicle n of m Uninstall all tasks; Indicates the maximum computing resource of the RSU.

[0048] It should be noted that the first constraint indicates that the task calculation delay cannot exceed the maximum tolerable calculation delay; the second constraint indicates whether the task is offloaded; the third constraint indicates that the task is partially offloaded, and the offloading ratio is [0,1]; the fourth constraint indicates the proportion of vehicle twin auxiliary task calculation, and the offloading ratio is [0,1]; the fifth constraint indicates that the total computing resources allocated by RSU to the offloaded task must not exceed the maximum computing resources of RSU.

[0049] Step S04: The digital twin network solves the optimization problem model. Specifically: In this embodiment, the A3C algorithm is used to solve the optimization problem model.

[0050] The A3C (Asynchronous Advantage Actor-Critic) algorithm is an asynchronous parallel algorithm based on the Actor-Critic framework. It avoids the problem of the DQN algorithm causing errors due to overestimation of the value function, which in turn affects the search for the optimal strategy and algorithm convergence. The A3C algorithm runs multiple threads simultaneously, and each thread has a copy of the agent.

[0051] Step S04 is specifically as follows: constructing the state space, action space and reward function, and constructing the A3C algorithm model; initializing the global Actor network and global Critic network parameters of the A3C algorithm model, as well as the local Actor network and local Critic network parameters in multiple parallel asynchronous threads; in the operation of each thread, the agent obtains the state of the environment, generates the action probability distribution through the local Actor network and samples the action, and after executing the action, interacts with the environment to obtain the reward and new state, calculates the temporal difference error using the local Critic network, updates the local Actor network and Critic network parameters, and loops until the training termination condition is met, and finally obtains the optimal unloading decision and resource allocation plan to achieve the optimization of vehicle edge computing.

[0052] In this embodiment, the process of solving the optimization problem model using the A3C algorithm is as follows: Step S41, constructing the state space, action space and reward function, and constructing the A3C algorithm model.

[0053] Construct the state space, the state set is ,in, Indicates vehicle n In time slot t The tasks generated within Indicates vehicle n The twins of Indicates n Vehicle and k The distance between RSUs. The state includes the status information of vehicle users in the entire network. The computing power of the vehicle reflects the level of local computing resources. The computing power of the vehicle twin and the computing power error determine the effectiveness of the vehicle twin assisted computing. Task-related information of vehicle users, such as task size, CPU cycles required to complete the task, and maximum tolerable task delay, are directly related to the difficulty and time requirements of task processing. Environmental information in the scene, such as the location coordinates of the nearest RSU, can be used to calculate the distance between the vehicle and the RSU; the remaining computing resources, channel resources, and road conditions of the RSU, these environmental factors are crucial to task offloading and computing resource allocation decisions.

[0054] Construct an action space that covers the key decision actions that the agent can perform. The unloading decision action is The uninstall decision is partial uninstall, , where 0 means that the task is not offloaded to the RSU, and the vehicle only relies on local computing resources and twin-assisted computing; 1 means that the task is offloaded to the RSU, and the offloading ratio needs to be further determined.

[0055] Construct a reward function. The vehicle, as an intelligent agent, makes corresponding decisions by maximizing the reward of the algorithm through interaction with the environment. In order to minimize the total delay and total cost of the system, the reward function is defined r for .

[0056] Step S42, initializing the global Actor network parameters and the global Critic network parameters, and initializing the local Actor network parameters and the local Critic network parameters in each parallel asynchronous thread to the global Actor network parameters and the global Critic network parameters.

[0057] Specifically, initialize and initialize the global Actor network parameters and global critic network parameters , initialize the network parameters of multiple parallel asynchronous threads at the same time, each thread has independent local Actor network parameters and local critic network parameters , and set the local Actor network parameters and local critic network parameters Set to global Actor network parameters and global critic network parameters Consistent, that is , ; Furthermore, it is necessary to initialize the environment and experience buffer.

[0058] Step S43, execute all threads in parallel, and obtain the current state of each thread during its execution. , the local Actor network generates an action probability distribution based on the current state, and randomly samples the action from the action probability distribution ; Perform sampling actions in the environment and get timely rewards and the next state ; Experience for storage.

[0059] Specifically, when entering the asynchronous thread execution phase, each thread operates independently. For each thread loop, the current state of the thread is obtained. , the local Actor network outputs the action probability distribution according to the current state , and from the action probability distribution Random sampling is performed in the selection sampling action ; The agent performs actions in the environment , get instant rewards and the new state , storage experience will into the experience buffer.

[0060] Step S44: The local critic network calculates the temporal difference error based on experience.

[0061] Specifically, the experience is input into the local critic network, and the local critic network calculates the temporal difference error according to the temporal difference error algorithm. ,in, represents the time-series difference error, which measures the deviation of the current state value estimate from the actual return; Indicates that the agent is t Instant rewards for performing actions at all times; represents the discount factor, which is used to balance the importance of immediate rewards and future rewards; Represents the value estimate of the local Critic network in the next state, Represents the value estimate of the local Critic network in the current state.

[0062] Step S45, based on the time difference error, the Actor network loss and the Critic network loss are calculated respectively, and the local Actor network parameters and the local Critic network parameters are updated respectively. Specifically, Update the Actor Network: Calculate the loss function of the Actor Network ,in, Represents the output action probability distribution of the local Actor network, and uses the gradient ascent method to update the parameters of the local Actor network , calculate the gradient of the local Actor loss function with respect to the network parameters , update the parameters ,in is the learning rate of the Actor network; Update the Critic network: Calculate the loss function of the Critic network , use the gradient descent method to update the parameters of the local Critic network , calculate the gradient of the local Critic loss function with respect to the network parameters , update the parameters ,in is the learning rate of the Critic network.

[0063] Step S46, looping steps S43-S45 until the loop termination condition is met, and outputting the optimal strategy represented by the global Actor network.

[0064] Specifically, steps S43-S45 are looped until the training termination condition is met (such as reaching the set number of training steps or the agent performance reaches the expectation), and finally the optimal strategy represented by the global Actor network is output.

[0065] In the digital twin network architecture of the present invention, the vehicle twin collects and synchronizes the running speed of the user's vehicle in real time, monitors the computing resources, task information, network status and cost-effectiveness of the user's vehicle; the RSU twin collects and shares the roadside equipment status data in real time, and cooperates with the vehicle twin to complete data interaction. The vehicle twin and the RSU twin together constitute a digital twin network. The vehicle twin and the RSU twin act as intelligent agents and jointly execute the A3C algorithm to collaboratively achieve the optimal decision for task offloading. After obtaining the optimal decision, considering that the vehicle twin has abundant computing resources and all the information about the user's vehicle tasks, the vehicle twin is further used as a computing node to assist the user's vehicle in calculating the tasks that are not offloaded to the RSU.

[0066] The present invention combines digital twin technology with the A3C algorithm to achieve optimal offloading decisions and computing resource allocation. Digital twin technology maps the state of the physical system in real time by building a virtual model that is highly similar to the physical entity. The A3C algorithm, as an asynchronous and parallel deep reinforcement learning algorithm, can use the digital twin environment to evaluate and update strategies simultaneously in multiple threads based on the large amount of real-time data provided by different twins. The parallel learning characteristics of the A3C algorithm can significantly accelerate the convergence speed of the model and shorten the time for strategy optimization. Figure 3 , which is a simulation diagram of the optimal offloading solution and the minimum system cost obtained by the A3C algorithm under the digital twin framework in the digital twin-assisted vehicle task offloading and computing resource allocation optimization method provided in an embodiment of the present invention. Figure 4 , which is a comparison chart of system simulation costs of different calculation methods based on the optimal offloading solution provided in the embodiment of the present application, wherein the DT+A3C (Proposed) curve represents the system simulation cost curve of the calculation method combining the digital twin technology and the A3C algorithm provided in the embodiment of the present application, the DT+SAC curve represents the system simulation cost curve of the calculation method combining the digital twin technology and the SAC algorithm, the DT+DDPG curve represents the system simulation cost curve of the calculation method combining the digital twin technology and the DDPG algorithm, the No-DT+A3C represents the system simulation cost curve of a single A3C algorithm, the Local Only curve represents the system simulation cost curve of the local calculation method only, and the RSU Only represents the system simulation cost curve of the calculation method that only unloads to the RSU. Figure 4 As shown, digital twin technology can update the status information of users and various physical devices in the scene in real time. The twins in the digital twin network act as agents to make decisions with lower latency than traditional solutions. In scenarios with different vehicles, the objective function value of the proposed scheme is always at a lower level compared to the cases of only offloading to the RSU or only local calculation and no digital twins. This means that the technology can more accurately balance the delay and cost factors and effectively reduce the overall system loss. Similarly, although the DDPG and SAC algorithms are also significantly better than some traditional algorithms, highlighting the universal advantages of digital twin technology in optimizing objective functions under a multi-algorithm architecture, and can provide a more economical and efficient solution for system operation, its objective function is still higher than the objective function of the method of this application. Faced with the increased complexity of traffic scenarios brought about by the increase in the number of vehicles, digital twin technology gives the algorithm excellent scalability and adaptability. As Figure 4 As shown in the figure, as the number of vehicles increases from 5 to 30, the objective function values ​​of the algorithms based on digital twins have a gentle growth trend, while the objective function values ​​of the schemes without digital twins have a large increase. This shows that digital twin technology enables the algorithm to maintain stable performance when dealing with large-scale vehicles and complex traffic flows, ensuring that the system operates efficiently in a dynamically changing environment, and effectively expanding the application boundaries and scope of the algorithm.

[0067] Corresponding to the above embodiment, an embodiment of the present invention further provides an electronic device.

[0068] See also Figure 5 , is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 As shown, the electronic device 500 may include: a processor 501, a memory 502 and a communication unit 503. These components communicate via one or more buses. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not limit the embodiments of the present invention. It may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0069] The communication unit 503 is used to establish a communication channel so that the electronic device can communicate with other devices.

[0070] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 502, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 501 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0071] The memory 502 is used to store the execution instructions of the processor 501. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0072] When the execution instructions in the memory 502 are executed by the processor 501 , the electronic device 500 is enabled to execute part or all of the steps in the above method embodiment.

[0073] Corresponding to the above embodiment, the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program is running, the device where the computer-readable storage medium is located may be controlled to execute some or all of the steps in the above method embodiment. In a specific implementation, the computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0074] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes part or all of the steps in the above method embodiment.

[0075] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. A and B may be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c may be single or multiple.

[0076] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0078] In several embodiments provided by the present invention, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0079] The above is only a specific embodiment of the present invention. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A digital twin-assisted vehicle task offloading and computing resource allocation optimization method, characterized in that: include: Obtain vehicle and RSU information, build vehicle twins and RSU twins, and build a digital twin network; Determine the total delay and total cost of the system based on the digital twin network, wherein the total delay includes local computing delay, vehicle twin assisted computing delay, GAP error, uplink delay and RSU computing delay, and the total cost includes RSU computing cost and vehicle twin assisted computing cost; Establish an optimization problem model based on minimizing the total delay and total cost of the system; The digital twin network solves the optimization problem model.

2. The vehicle task offloading and computing resource allocation optimization method assisted by digital twin according to claim 1 is characterized in that: The total delay The calculation formula is: ， ， in, N represents the total number of vehicles, T Indicates the total number of time slots, Indicates the task calculation delay, Indicates the uplink delay. Indicates the RSU calculation delay, Indicates the total delay of vehicle users.

3. The vehicle task offloading and computing resource allocation optimization method assisted by digital twin according to claim 2 is characterized in that: Uplink delay for: ， in, Indicates the task transfer rate, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The size of the tasks generated within; RSU calculation delay for: ， in, Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates that RSU is allocated to the task computing resources.

4. The vehicle task offloading and computing resource allocation optimization method assisted by digital twin according to claim 2 is characterized in that: Total delay of vehicle users for: ， ， ， ， in, Indicates the local computing latency, represents the vehicle twin-assisted computing delay, Gap represents the GAP error, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates vehicle n The computing power of The vehicle twin maps the vehicle’s computational capabilities through a virtual representation, Represents the computational power error generated during the vehicle twin mapping process.

5. The digital twin-assisted vehicle task offloading and computing resource allocation optimization method according to claim 1 is characterized in that: Total cost for: ， ， ， in, Indicates the RSU calculation cost, represents the vehicle twin auxiliary computing cost, N represents the total number of vehicles, T Indicates the total number of time slots, Indicates vehicle n In time slot t Uninstall ratio within Indicates the completion of the vehicle n In time slot t The CPU cycles required for the tasks generated within, Indicates the unit price of RSU calculation, Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within Represents the unit price of vehicle twin assisted computing.

6. The digital twin-assisted vehicle task offloading and computing resource allocation optimization method according to claim 1 is characterized in that: The optimization problem model is: ， in, represents the delay weight factor, represents the price weight factor, x represents the uninstall decision, Indicates that RSU is allocated to the task of computing resources, Indicates vehicle n In time slot t Uninstall ratio within Indicates vehicle n In time slot t The proportion of vehicle twin-assisted calculations performed within represents the total delay, represents the total cost; The model satisfies the following constraints: ， ， ， ， ， in, Indicates the task calculation delay, Indicates the maximum tolerable delay, Indicates vehicle n No. m The offloading strategy of each task, Indicates the maximum computing resource of the RSU.

7. The digital twin-assisted vehicle task offloading and computing resource allocation optimization method according to claim 6 is characterized in that: The digital twin network solves the optimization problem model, specifically: Step S41, constructing state space, action space and reward function; Step S42, initializing the global Actor network parameters and the global Critic network parameters, and initializing the local Actor network parameters and the local Critic network parameters in each parallel asynchronous thread to the global Actor network parameters and the global Critic network parameters; Step S43, execute all threads in parallel, and obtain the current state of each thread during its execution. , the local Actor network generates an action probability distribution based on the current state, and randomly samples the action from the action probability distribution ; Perform sampling actions in the environment and get timely rewards and the next state ; Experience To store; Step S44, the local critic network calculates the temporal difference error based on experience; Step S45, calculating the Actor network loss and the Critic network loss respectively based on the temporal difference error, and updating the local Actor network parameters and the local Critic network parameters respectively; Step S46, looping steps S43-S45 until the loop termination condition is met, and outputting the optimal strategy represented by the global Actor network.

8. The vehicle task offloading and computing resource allocation optimization method assisted by digital twin according to claim 7 is characterized in that: The state space is: ,in, Indicates vehicle n In time slot t The tasks generated within Indicates vehicle n The twins of Indicates n Vehicle and k The distance between RSUs; The action space is: ; The reward function is: .

9. The vehicle task offloading and computing resource allocation optimization method assisted by digital twin according to claim 7 is characterized in that: Timing Differential Error for: ， in, Indicates that the agent is t The instant reward after performing the action at any time, represents the discount factor, Represents the value estimate of the local Critic network in the next state, Represents the value estimate of the local Critic network in the current state.

10. The digital twin-assisted vehicle task offloading and computing resource allocation optimization method according to claim 9, characterized in that: The calculation formula for updating the local Actor network parameters is: ， ， in, Represents local Actor network parameters, represents the learning rate of the Actor network, represents the gradient of the Actor network loss with respect to the network parameters, Indicates Actor network loss, Represents the output action probability distribution of the local Actor network; The calculation formula for updating the local Critic network parameters is: ， ， in, Represents the local Critic network parameters, represents the learning rate of the Critic network, Represents the gradient of the Critic network loss with respect to the network parameters, Represents the Critic network loss.

Images