Multi-dimensional resource optimization method for vehicle-road cooperation digital twin system

Abstract

The invention discloses a multi-dimensional resource optimization method for a vehicle-road cooperation digital twin system. The method comprises the steps of scene construction, task division and construction of a channel model and a calculation model. Through joint optimization of dynamic task division, channel resource allocation, vehicle transmitting power adjustment and RSU computing resource allocation, minimization of digital twinborn model construction and updating cost is realized; the invention further provides a deep reinforcement learning algorithm Noisy Net-HyAR integrated with a noise network Noisy Net, and the Noisy Net is utilized to replace a traditional exploration strategy, so that the agent can explore more states and action spaces, and the reinforcement learning performance is improved. According to the method, the time delay and the energy consumption for constructing the digital twinning are used as the system cost, the system cost is used as an index, the effectiveness of the method is verified in a simulation result, and the adaptive capacity to the environment is shown in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation systems, specifically a multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure, and these statements may constitute prior art. In the process of developing this invention, the inventors discovered at least the following problems in the prior art.

[0003] With the acceleration of urbanization and the surge in car ownership, traditional transportation systems face severe challenges in terms of efficiency, safety, and sustainability. Intelligent Transportation Systems (ITS), as a core paradigm of next-generation traffic management, integrates advanced technologies such as the Internet of Things (IoT), artificial intelligence (AI), and 5G communication to achieve comprehensive digital and intelligent management of traffic elements. According to statistics from the International Transport Forum (ITF), the application of intelligent transportation technologies globally can improve urban road traffic efficiency by 35%-50%, reduce traffic accident rates by more than 60%, and reduce carbon emissions by 20%-30%. Its core value lies in breaking through the fragmented and passive response model of traditional traffic management, constructing an integrated dynamic optimization system of "perception-decision-control," and providing a revolutionary solution for the efficient, safe, and green development of modern urban transportation.

[0004] As a key enabling technology for ITS, digital twins (DTs) can improve traffic management efficiency and enhance driving safety through real-time connectivity between physical systems and digital spaces. Numerous studies have focused on using DTs to optimize ITS operations and resource management. However, most existing work completely delegates the task of building / updating DT models to the Roadside Unit (RSU) and assumes that the constructed DTs are always real-time and effective. In real-world scenarios, to achieve real-time DT model updates, vehicles on the road may need to transmit megabytes of high-definition video streams or LiDAR point cloud data to the RSU in each time slot for fusion computation. However, the limited bandwidth in current and future ITS systems cannot support the transmission of large-scale raw data, and the limited computing resources of the RSU cannot build / update DT models for data uploaded by all vehicles within its coverage area. Furthermore, traditional convex optimization methods cannot effectively solve the data transmission problem under dynamic environmental changes. Traditional deep reinforcement learning (DRL) algorithms, such as the patent application number 202411091402.7 entitled "A method for allocating sensory computing resources and deploying vehicle digital twins in the Internet of Vehicles", are prone to getting stuck in local optima when facing high-dimensional mixed action spaces, resulting in a large policy bias problem.

[0005] Some technologies attempt to address these issues through algorithmic innovation. For example, patent application number 202411091398.4, entitled "Resource Allocation Method for Collaborative Perception and Edge Cooperation Assisted by Digital Twins," proposes a multi-agent deep deterministic policy gradient (HAS-MADDPG) algorithm based on a hybrid action space. However, simply innovating the algorithm without changing existing solutions (or failing to provide adaptable solutions) ultimately cannot adequately solve the problems mentioned above.

[0006] Therefore, the problem to be solved by this invention is how to design optimization schemes and adapt and innovate algorithms to solve the optimization problem under the vehicle-road cooperative digital twin system, minimize the cost of building and updating the road DT model, and ensure the real-time performance of the DT model built in complex scenarios. Summary of the Invention

[0007] In view of the above problems, the purpose of this invention is to solve some of the problems in the prior art, or at least alleviate these problems.

[0008] A multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems includes the following steps:

[0009] A transmission model for a vehicle-road cooperative digital twin system is constructed, whose network framework includes a physical layer and a digital twin layer. Based on the process of building / updating the DT, the preprocessing tasks are divided into vehicle end and RSU end in three task division methods to construct a task division model. Under the system model, the transmission model, power consumption model and computing model of the system are established. Finally, the total cost model is constructed.

[0010] With the aim of minimizing the cost of building / updating road DTs, task partitioning is selected through joint optimization. Uplink channel allocation Vehicle transmission power and RSU computing power allocation To construct an optimization problem;

[0011] The NoisyNet network is integrated into the Actor network of the Hybrid Action Representation (HyAR) algorithm. The optimization problem is solved using the NoisyNet-HyAR algorithm with integrated noise network, and a resource allocation scheme that minimizes the cost of building / updating road DT is learned.

[0012] Furthermore, the optimization problem is formulated as follows:

[0013]

[0014] in: For roadside units The cost of building / updating road DT; Maximum allowable delay; For roadside units Total latency to complete DT build / update; Each RSU is used as a set , The vehicles are used as a set Channel set ; Indicates roadside unit Assigned to vehicles Computing power used to build / update DT; Indicates roadside unit For vehicles Selected task allocation method This indicates the task allocation method selected by the vehicle; For vehicles Channel allocation; Indicates time slot Vehicle Selected transmit power, This is the maximum transmission power. This is the minimum transmit power. Let be the decision variable, representing the channel. Whether to assign to a vehicle ;

[0015] Constraint C1 means that the latency of building / updating the road DT should be less than the maximum allowable latency. Constraints C2 and C3 are constraints on RSU computational resources; constraint C4 means that the RSU must select and can only select one task partitioning method for the vehicle; constraint C5 is a capacity limit for channel resources; constraint C6 is a constraint on decision variables; and constraint C7 is a constraint on vehicle transmit power.

[0016] Furthermore, the physical layer consists of vehicles on the road, each equipped with four 120° field-of-view cameras and communication and computing devices; the digital twin layer is deployed in RSUs on the road, each RSU is configured with an edge server responsible for providing computing resources for building / updating the DT; the RSU connects to the vehicles within its coverage area via wireless communication and acts as a central processing unit to collect vehicle parameters, data collected by vehicle sensors, and channel status information within its coverage area.

[0017] Furthermore, the DT consists of a static layer and a dynamic layer; the static layer contains three-dimensional physical models of static objects within the RSU coverage area, with information provided by high-definition maps; the dynamic layer contains dynamic objects on the road, with information derived from environmental information collected in real time by vehicles on the road and vehicle parameters.

[0018] Furthermore, the three task division methods are as follows:

[0019] Method 1: The vehicle does not perform any local processing, but directly transmits four real-time video frames captured by the onboard cameras around the vehicle and the vehicle's own coordinates. The entire task of building / updating the DT is completed only by the RSU.

[0020] Method 2: The vehicle and RSU collaborate to complete the preprocessing task. The vehicle uses a generalized reference representation segmentation model to process real-time video frames captured by the onboard camera, identifying dynamic objects in the perceived image information. Only the vehicle's own coordinates and the images of the dynamic objects are transmitted, while the RSU completes the remaining calculation steps. The ratio of the remaining data size to the original data size after using dynamic foreground and static background separation techniques is [not specified in the original text].

[0021] Method 3: The vehicle independently completes the preprocessing task, that is, based on Method 2, it further processes the images of dynamic objects in the scene, extracts the 3D feature parameters of the dynamic objects, and transmits the vehicle's own coordinates and the 3D feature parameters of other dynamic objects in the scene.

[0022] The NoisyNet-HyAR algorithm, which employs a hybrid action representation using an integrated noise network, is used to solve the optimization problem, and includes the following steps:

[0023] The optimization problem is modeled as a Markov decision process; the corresponding elements are defined as follows:

[0024] state space :definition For time slots The status consists of the amount of data the vehicle needs to upload, the computing resources allocated by the base station, and channel status information. ;in This refers to the amount of data that the vehicle needs to upload. These are the computing resources already allocated to the base station. It is a channel state information vector;

[0025] Action space : Defined as a time slot The actions involved include task partitioning and selection, uplink channel allocation, vehicle transmit power allocation, and RSU computing power allocation. ;in, It is a task allocation selection. It is the uplink channel allocation matrix. It is the vehicle's transmission power. It is the allocation of RSU computing power;

[0026] reward function The reward function is used to measure the agent's performance and guide its decision-making process; in the first... Within each time slot, the instant reward can be defined as: ;in This is a penalty constant; a penalty mechanism is triggered when the agent's decision does not meet the time delay constraint. For time slots The lower the cost of building / updating DT, the greater the reward value.

[0027] NoisyNet is integrated into the Actor network of the traditional HyAR algorithm, and the NoisyNet-HyAR algorithm is used to solve the optimization problem by employing the hybrid action representation of the integrated noise network.

[0028] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multi-dimensional resource optimization method for a vehicle-road cooperative digital twin system.

[0029] The present invention has the following beneficial effects:

[0030] 1. Refine the process of building / updating DT: In view of the contradiction between the large amount of real-time data that needs to be uploaded and processed when building / updating DT and the limited network and computing resources in the Internet of Vehicles, the task of building / updating DT has been refined, and a multi-dimensional resource collaborative optimization scheme that considers the division of the task of building / updating DT has been designed.

[0031] 2. Multi-dimensional Resource Joint Optimization: Vehicle-Road Cooperative Digital Twin (DT) systems face core challenges such as dynamic scenario adaptation, multi-objective trade-offs, and heterogeneous resource integration, lacking a balance between real-time performance and energy consumption. This invention jointly considers the heterogeneity of the RSU (Road Service Unit), vehicle computing resources, and network conditions in the DT system, and jointly optimizes the selection of vehicle transmit power and the allocation of RSU communication and computing resources. This minimizes system cost while meeting the latency requirements for building / updating the DT.

[0032] 3. Algorithm Innovation and Performance Improvement: This invention combines NoisyNet with the HyAR algorithm, effectively enhancing the algorithm's exploration capabilities and optimizing the balance between exploration and exploitation, thereby significantly improving decision-making efficiency. Furthermore, the proposed NoisyNet-HyAR algorithm exhibits stronger robustness and higher learning efficiency in high-dimensional action spaces.

[0033] By synergistically combining the aforementioned improvements, this invention proposes a multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems. It combines NoisyNet and a novel hybrid action reinforcement learning framework—HyAR—to solve the optimization problem in vehicle-road cooperative digital twin systems, minimize the cost of building and updating the road DT model, and ensure the real-time performance of the constructed DT model in complex scenarios. Attached Figure Description

[0034] Figure 1 This is a scene diagram of the vehicle cooperative digital twin system of the present invention;

[0035] Figure 2 This is a detailed diagram illustrating the DT construction / update process of this invention;

[0036] Figure 3 A graphical representation of the NoisyNet noisy linear layer;

[0037] Figure 5 This is a convergence trend diagram of the reward values ​​of the algorithm proposed in this invention and the comparison algorithm;

[0038] Figure 6 This diagram illustrates the impact of different numbers of vehicles on the algorithm's reward value. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings. The embodiments of the present invention are only used to illustrate the present invention and not to limit the present invention. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the technical concept of the present invention should be included within the scope of the present invention.

[0040] This invention aims to solve the problem of efficient construction and updating of digital twin models under resource-constrained conditions. It hopes to minimize the cost of constructing and updating road digital twin (DT) models in vehicle-road cooperative digital twin systems by optimizing task partitioning, channel allocation, vehicle transmit power, and roadside unit (RSU) computing resource allocation.

[0041] To address the problems existing in current technologies, this invention proposes a multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems. Under the constraint of the maximum tolerable latency for autonomous vehicles, this method minimizes the cost of building and updating the road DT model by jointly optimizing task partitioning, channel allocation, vehicle transmit power, and RSU computational resource allocation. To achieve the above objectives, this invention employs the following technical solution.

[0042] S1: Building the model.

[0043] In this embodiment, the system model structure is composed as follows: Figure 1 As shown, the network framework considered in this invention has two layers: a physical layer and a digital twin layer. The physical layer mainly consists of vehicles on the road. Each vehicle has four 120° field of view (FoV) cameras that display environmental information in front, behind, to the left, and to the right of the vehicle, thereby capturing a complete 360° view of the environment around the vehicle. It is also equipped with communication and computing devices to upload the captured environmental information to the RSU.

[0044] The digital twin layer is deployed in the RSUs on the road (each RSU is configured with an edge server to provide computing resources for building / updating the DT), and the scenario includes... RSUs are used in sets It indicates that the vehicles are located near the RSU, the first ( There are a total of ) RSU coverage areas A collection of vehicles The RSU can connect to vehicles within its coverage area via wireless communication and act as a central processing unit to collect vehicle parameters, data collected by vehicle sensors, and channel status information within its coverage area.

[0045] Construct a task partitioning model. Based on the process of building / updating the DT (Driver Module), consider partitioning the preprocessing tasks, assigning them to the vehicle end and RSU (Roadside Unit) end using three different partitioning methods. For vehicles The selected task allocation method is: In the formula Indicates roadside unit The vehicles that got off This indicates the task allocation method selected for the vehicle. Specifically:

[0046] The DT consists of a static layer and a dynamic layer. The static layer of the DT contains 3D physical models of static objects such as roads, buildings, traffic infrastructure, and vegetation within the RSU coverage area. This layer's information is provided by high-definition maps. This layer changes at a low frequency and therefore does not require real-time updates. The dynamic layer of the DT contains dynamic objects such as vehicles, pedestrians, and traffic lights on the road. This layer's information is derived from environmental information collected in real time by vehicles on the road, as well as vehicle parameters (location, size, and model).

[0047] Figure 2 This paper demonstrates the refined DT (Digital Twin) construction / update process. Since this invention only considers the construction and update of the dynamic layer of the digital twin, the real-time video frames captured by the vehicle's onboard sensors need to be processed by a Generalized Referring Expression Segmentation (GRES) model to identify dynamic objects in the captured real-time video frames. Then, the image data of the identified dynamic objects undergoes self-supervised 3D reconstruction to extract the 3D feature parameters (camera, light, shape, texture) of the dynamic objects. Because these two steps are prerequisites for creating the digital twin model, this scheme collectively refers to the process of dynamic object identification and extraction of 3D feature parameters as preprocessing. After preprocessing, 3D modeling is performed based on the extracted 3D feature parameters of the dynamic objects to complete the creation of the digital twin model of the dynamic objects. Based on the DT construction / update process, the preprocessing task is divided into three parts, with three different task division methods assigned to the vehicle end and the RSU (Remote Subsystem Unit). Indicates roadside unit For vehicles Choose a task partitioning method. The three partitioning methods are as follows:

[0048] Method 1: The vehicle does not perform any local processing, but directly transmits four real-time video frames captured by the onboard cameras around the vehicle and the vehicle's own coordinates. The entire task of building / updating the DT is completed only by the RSU.

[0049] Method 2: The vehicle and RSU collaborate to complete the preprocessing task. The vehicle uses the Generalized Referring Expression Segmentation (GRES) model to process the real-time video frames captured by the onboard camera, identifying dynamic objects in the perceived image information. Only the vehicle's own coordinates and the images of the dynamic objects are transmitted (the ratio of the remaining data size to the original data size after using dynamic foreground and static background separation techniques). The remaining calculation steps are completed by the RSU end;

[0050] Method 3: The vehicle independently completes the preprocessing task, that is, based on Method 2, it further processes the images of dynamic objects in the scene, extracts the 3D feature parameters (camera, light, shape, texture) of the dynamic objects, and transmits the vehicle's own coordinates and the 3D feature parameters of other dynamic objects in the scene.

[0051] exist Figure 1 Under the system model shown, establish the system's transmission model. Assume the channel set is... The channel bandwidth is set to The time slot index set is At the start of each time slot, the Roadside Unit (RSU) collects driving parameters and channel gain information uploaded by the vehicles, and then makes a channel allocation decision. The channels allocated to vehicles within the coverage area are determined by a matrix. This indicates that the vehicles... The channel allocation is represented as If the channel Vehicles were assigned ,but ,otherwise In the time slot Roadside unit The vehicles that got off ( The uplink transmission rate is:

[0052]

[0053] In the formula This represents the bandwidth of each sub-channel. Indicates noise power density. Indicates time slot Vehicle with RSU Channel gain between Indicates time slot Vehicle Selected transmit power. Assume the maximum transmit power is... The minimum value is Therefore, the transmit power constraint is .

[0054] Because each time slot is very short, we assume that the channel gain remains the same in each time slot, but will vary between different time slots. We assume different task partitioning methods for vehicles... The amount of data to be uploaded is In the time slot Vehicle The uplink transmission delay is:

[0055]

[0056] Construct a power consumption model. Assume that under different task partitioning methods, the vehicle... The amount of data to be uploaded is Then in the time slot Vehicle The uplink transmission delay is In the formula Indicates road test unit For vehicles The selected task allocation method. Based on transmission delay. and vehicles Selected transmit power It can calculate the vehicle The energy consumed by uplink transmission is .

[0057] Construct a computational model. Assume that each vehicle has a constant computing power, and that when local preprocessing of collected environmental information is required, the vehicle always dedicates all its computing resources to performing the task. Assume the road test unit... The vehicles that got off Its computing power is When the roadside unit The vehicles that got off Choose option 1, i.e. In this method, four real-time video streams captured by the onboard cameras around the vehicle are directly transmitted without processing. Therefore, the vehicle... The computational cost of local computation is When the roadside unit The vehicles that got off Choosing method 2 requires the vehicle to use a trained GRES model to identify dynamic objects in the collected environmental information. Assuming the computational complexity of image and language feature extraction is... Therefore, vehicles under this method The computational cost of local computation is When the roadside unit The vehicles that got off Choosing method 3 requires extracting the feature parameters of the dynamic object based on method 2 before transmission. Assume the computational complexity of extracting the feature parameters of the dynamic object is O(n). Therefore, vehicles under this method The computational cost of local computation is .

[0058] Therefore, according to the road test unit For vehicles Different task allocation methods selected, vehicles The computational load for local computation is as follows: , , ,in , , These represent the vehicles when selecting task allocation method 1, method 2, and method 3, respectively. The computational cost of local calculations. In the formula... This represents the computational complexity of performing image and language feature extraction. This indicates the initial image data size captured by the vehicle. This represents the computational complexity of extracting feature parameters from dynamic objects. This represents the ratio of the remaining data size to the original data size. Therefore, in a time slot... At the same time, vehicles under different task division modes The latency for local preprocessing is In the formula Indicates vehicle The computational cost of local computation Indicates vehicle Its computing power.

[0059] Therefore, in time slots At the same time, vehicles under different task division modes The latency for local preprocessing is:

[0060]

[0061] In the time slot Time-side unit For vehicles The latency for edge preprocessing of the uploaded information is:

[0062]

[0063] In the formula Indicates roadside unit For vehicles The computational load required to preprocess the uploaded information Indicates roadside unit Assigned to vehicles Computational capabilities used for building / updating the DT. Assuming roadside units... Its computing power is Therefore, roadside unit Assigned to vehicles The computational power constraints for building / updating DT are: .

[0064] Finally, the total cost model is constructed. Since each RSU builds / updates the Data Center (DT) for the roads within its coverage area, the RSU needs to receive data from all vehicles before building / updating the DT, as the DT is created by incorporating many attributes of the physical system. Therefore, the latency caused by uploading and preprocessing the environmental information collected by the vehicles is:

[0065]

[0066] In the formula This indicates the vehicle's local calculation latency. Indicates the uplink transmission delay of the vehicle. Indicates roadside unit For vehicles The latency of edge preprocessing of the uploaded information.

[0067] The latency for RSU to build a dynamic 3D model of a vehicle is:

[0068]

[0069] In the formula Represented as vehicles Creating 3D models of dynamic objects in real-time video frames requires computational resources.

[0070] Therefore, roadside units The total latency to complete DT build / update is:

[0071]

[0072] This invention argues that the cost of RSU (Roadside Unit) for constructing / renewing road DT (Driver Targeting) consists of two parts: latency and energy consumption. Therefore, the roadside unit... The cost of building / updating road DT is:

[0073]

[0074] In the formula , These represent the weights of latency cost and energy cost for building / updating DT, respectively.

[0075] S2: Construction optimization problem.

[0076] Based on the established model, in order to minimize the cost of RSU construction / updating road DT, this invention proposes a joint optimization of task partitioning selection. Uplink channel allocation Vehicle transmission power and RSU computing power allocation To minimize the cost of building / updating the road DT, the specific optimization problem is as follows:

[0077]

[0078] in: For roadside units The cost of building / updating road DT; Maximum allowable delay; For roadside units Total latency to complete DT build / update; Each RSU is used as a set , The vehicles are used as a set Channel set ; Indicates roadside unit Assigned to vehicles Computing power used to build / update DT; Indicates roadside unit For vehicles Selected task allocation method This indicates the task allocation method selected by the vehicle; For vehicles Channel allocation; Indicates time slot Vehicle Selected transmit power, This is the maximum transmission power. This represents the minimum transmission power. Let be the decision variable, representing the channel. Whether to assign to a vehicle .

[0079] Constraint C1 means that the latency of building / updating the road DT should be less than the maximum allowable latency. Constraints C2 and C3 are constraints on RSU computational resources; constraint C4 means that the RSU must select and can only select one task partitioning method for the vehicle; constraint C5 is a capacity limit for channel resources; constraint C6 is a constraint on decision variables; and constraint C7 is a constraint on vehicle transmit power.

[0080] S3: Solve the optimization problem.

[0081] In our designed system, the proposed joint optimization problem involving task partitioning, uplink channel allocation, vehicle transmit power, and base station computing power allocation is a hybrid action space problem encompassing both continuous and discrete actions. Traditional reinforcement learning algorithms cannot directly handle this heterogeneous action space. If the heterogeneous space is simply converted to a homogeneous space through discretization or continuousization, the size of the converted action space will obviously grow exponentially with the increase in the number of discrete actions, leading to scalability issues. On the other hand, mapping all discrete actions to a continuous dimension generates a piecewise function action subspace, resulting in additional difficulties in approximation and generalization.

[0082] To effectively solve the planned problem, this invention employs the Hybrid Action Representation (HyAR) algorithm from the DRL algorithm for solution. This addresses the limitations of conventional HyAR algorithms. The low efficiency of traditional exploration methods leads to slow convergence of reinforcement learning policies and the potential for getting trapped in local optima. This invention integrates NoisyNet into the Actor network of the HyAR algorithm. NoisyNet drives exploration through learnable perturbations of neural network weights. This method allows a single modification to the weight vector to induce consistent and potentially highly complex state-dependent policy changes across multiple time steps, enabling the agent to naturally generate exploratory behavior during the learning process without the need for additional perturbations. Such exploration strategies allow the agent to explore more states and action spaces, thereby improving reinforcement learning performance and effectively solving the above problems.

[0083] First, the original optimization problem is modeled as a Markov decision process.

[0084] The original cost minimization problem can be modeled as a Markov decision process. RSU is an agent where the system's state in each time slot depends only on the system's current state and the action taken by RSU. The agent interacts with the environment as follows: in each training set time period, the agent first obtains the state from the environment. Then select a strategy from the action space. Then, the environment updates its current state to... and receive corresponding rewards. Empirical tuples It is then stored in the replay memory buffer. middle.

[0085] The corresponding element is defined as follows:

[0086] state space :definition For time slots The status consists of the amount of data that the vehicle needs to upload, the computing resources allocated by the base station, and the channel status information; ,in This refers to the amount of data that the vehicle needs to upload. These are the computing resources already allocated to the base station. It is a channel state information vector.

[0087] Action space : Defined as a time slot The actions involved include task partitioning and selection, uplink channel allocation, vehicle transmit power allocation, and RSU computing power allocation. ,in, It is a task allocation selection. It is the uplink channel allocation matrix. It is the vehicle's transmission power. This refers to the allocation of RSU computing power.

[0088] reward function The reward function is used to measure the agent's performance and guide its decision-making process; in the first... Within each time slot, the instant reward can be defined as: ,in This is a penalty constant; a penalty mechanism is triggered when the agent's decision does not meet the time delay constraint. For time slots The lower the cost of building / updating DT, the greater the reward value.

[0089] Secondly, NoisyNet is integrated into the Actor network of the traditional HyAR algorithm, such as... Figure 4 As shown.

[0090] Due to the use of HyAR The current exploration strategy still relies on random perturbations of the agent's policy to induce new behaviors. However, such localized, "jittering" perturbations are unlikely to generate the large-scale behavioral patterns required for efficient exploration in many environments, leading to reinforcement learning strategies getting stuck in local optima. Therefore, this invention considers a simple alternative called NoisyNet, which drives exploration through learnable perturbations of neural network weights. This method allows a single modification to the weight vector to elicit consistent and potentially highly complex state-dependent policy changes across multiple time steps, enabling the agent to naturally generate exploratory behavior during the learning process without introducing additional perturbations. Such exploration strategies allow the agent to explore more states and action spaces, thereby improving reinforcement learning performance.

[0091] NoisyNet is a type of neural network, such as Figure 3 The image shows a graphical representation of the NoisyNet noisy linear layer, whose weights and biases are affected by a noise parameter function. These parameters are adjusted using gradient descent. More precisely, let... It is a parameter of noise A vector-parameterized neural network that accepts input and output We will use noise parameters Represented as ,in It is a vector of learnable parameters. It is a zero-mean noise vector with a fixed statistic. This represents element-wise multiplication. Consider having... Input and A linear layer of a neural network with output is represented by the following formula:

[0092]

[0093] in It is a layer input. It is a weight matrix. It is a bias.

[0094] The corresponding noise linear layer is defined as follows:

[0095]

[0096] in and Replaced the formula In and .parameter and It is learnable, and and It is a noisy random variable.

[0097] Then a learnable embedding table is constructed. , where parameters Used to represent Each discrete action. Each row of the embedded table. ( (Row index) corresponds to discrete actions of 3D continuous vector representation.

[0098] Then, a conditional variational autoencoder (VAE) is used to model the latent representation space of continuous parameters:

[0099] Encoder: (parameters are) ) in state and discrete action embedding As a condition, continuous parameters Mapping to latent variables .

[0100] Decoder: (parameters are) Under the same conditions, from the latent variables Reconstructing continuous parameters .

[0101] Conditional VAEs are trained by maximizing the variational lower bound to ensure that the latent space can effectively represent the dependencies of mixed actions. Specifically, the encoder... Gaussian latent distribution , where the mean and standard deviation Output by the encoder. For any latent variable sampled from this distribution. The decoder decodes it into continuous parameters in a deterministic manner. ,Right now:

[0102]

[0103] in, This is a transformation network. This is a fully connected layer used for reconstruction. Here, This represents the set of parameters for the decoder network, the prediction network, and the transform network shared by both. This is derived from the empirical playback buffer. A batch of state and raw mixed action data is sampled, and the following loss function is minimized. Joint training embedding table Conditional variational autoencoder , :

[0104]

[0105] For any latent variable and It can be conveniently decoded into hybrid actions by combining nearest neighbor lookup of the embedded table with a VAE decoder. The specific encoding and decoding process is as follows:

[0106]

[0107] HyAR in the transform network of a conditional VAE decoder A subnet was then cascaded. This is used to predict the residuals of state transition dynamics. For any state transition sample... Its state residual is defined as In cases where the distinction between symbols is not strictly defined (i.e., here) ), predict residual The generation process, thereby constructing the formula:

[0108]

[0109] Then minimize the L2 norm squared prediction error:

[0110]

[0111] Therefore, the final training loss for the hybrid action representation is derived as follows:

[0112]

[0113] in, It is a hyperparameter used to adjust the loss of dynamic prediction representation, and the final loss function depends on the unrewarded data that is dynamically related to the environment.

[0114] After constructing a compact, decodable, and semantically smooth hybrid action representation space, the next step is to learn a latent reinforcement learning policy within this representation space. The Actor network parameterizes the latent policy within the learned hybrid action representation space. (Parameters are) Output latent action vectors ,in Then the potential action is decoded into the original hybrid action. Dual Critic Network Using potential actions as input, approximate the value function of mixed actions. ,Right now Transfer samples were sampled from the experience playback buffer. The Critic network is trained using Clipped Double Q-Learning, and its loss function (for...) )as follows:

[0115]

[0116] in These are the parameters of the target Critic network and the target policy network, respectively. The Actor network is updated using the deterministic policy gradient, according to the following update rules:

[0117]

[0118] This method drives exploration through learnable perturbations of neural network weights. A single modification to the weight vector can induce consistent and potentially highly complex state-dependent policy changes across multiple time steps, allowing the agent to naturally generate exploratory behavior during the learning process.

[0119] Based on the above design, a HyAR algorithm integrating NoisyNet was designed and integrated into existing HyAR algorithms. Finally, to verify the effectiveness of the proposed scheme, the method was evaluated, demonstrating that the proposed NoisyNet-HyAR is effective and robust in solving the problem of minimizing the cost of RSU construction / updating road DT.

[0120] Algorithm 1 illustrates the specific implementation of this process.

[0121]

[0122] To demonstrate the effectiveness of the proposed solution, a comprehensive evaluation of the method was conducted, and the effectiveness and robustness of the proposed NoisyNet-HyAR in minimizing the cost of building / updating the Data Transmission Unit (DT) were verified. This example provides a description of the simulation setup, including environmental details, model design, and benchmarks used to evaluate the performance of the proposed method. In this invention, a scenario consisting of one independent Responsible Unit (RSU) and 4-6 autonomous vehicles is considered, with the RSU at a height of 25m. To fully validate the advantages of the proposed solution, the following comparative scheme was set up:

[0123] Comparison with Option 1: Simultaneously consider different task division methods and different vehicle transmission powers for data uploading.

[0124] Comparison Scheme 2: Ignoring different task division methods, data is uploaded only based on different vehicle transmission powers.

[0125] Comparison Scheme 3: Ignoring different vehicle transmission powers, data is uploaded only based on different task division methods.

[0126] By comparing the simulation results with the above schemes, this example provides a specific simulation and the conclusions drawn.

[0127] Figure 5 The convergence trends of the reward values ​​of the proposed algorithm and the comparison algorithms are shown. The results demonstrate that the NoisyNet-HyAR algorithm outperforms the baseline algorithms HyAR and NoisyNet-PDQN-TD3 in both convergence speed and final convergence performance. This advantage is mainly due to the fact that, compared to the HyAR algorithm, NoisyNet-HyAR introduces a noisy network instead of a HyAR network. The greedy strategy improves exploration efficiency and allows for the exploration of more states and actions, enabling the network to converge faster and find the optimal policy, thus increasing the convergence reward. Compared to the NoisyNet-PDQN-TD3 algorithm, NoisyNet-HyAR learns within a unified latent space, resulting in better learning performance and allowing the network to learn better policies.

[0128] Figure 6 The impact of different numbers of vehicles on the algorithm's reward value is demonstrated. Results show that the NoisyNet-HyAR algorithm used in this invention has a wider applicability and better robustness, maintaining good convergence performance even as the action space increases. In contrast, the performance of the NoisyNet-PDQN-TD3 algorithm is highly sensitive to the increase in the number of vehicles, and its strategy may be less adaptable to more complex multi-agent coordination.

[0129] This invention refines the DT (Digital Twin) construction process while considering vehicle channel conditions and available RSU (Resource Unit) computing resources. It dynamically allocates the DT construction / updating tasks rationally to vehicles and RSUs, and jointly optimizes channel resource allocation, vehicle transmit power, and RSU computing resource allocation to improve resource utilization, thereby achieving more efficient DT construction / updating. To address the established optimization problem, a novel hybrid action reinforcement learning framework—hybrid action representation integrating a noisy network—is proposed. This method integrates NoisyNet into the Actor network of the HyAR algorithm. Since NoisyNet drives exploration through learnable perturbations of neural network weights, the agent naturally generates exploratory behavior during the learning process, effectively solving the problem of low learning efficiency in conventional HyAR algorithms that leads to reinforcement learning strategies getting trapped in local optima. Finally, using the latency and energy consumption of constructing a digital twin as the system cost, and using system cost as an indicator, the effectiveness of the method is verified in simulation results, demonstrating its adaptability to complex scenarios.

[0130] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multi-dimensional resource optimization method for a vehicle-road cooperative digital twin system.

[0131] The conventional techniques and solutions not described in detail in the above embodiments are all well known in the art, and therefore will not be elaborated upon here. The above embodiments and / or experimental examples describe the preferred embodiments of the present invention in detail. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

Claims

1. A multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems, characterized in that, Includes the following steps: A transmission model for a vehicle-road cooperative digital twin system is constructed, whose network framework includes a physical layer and a digital twin layer. Based on the process of building / updating the DT, the preprocessing tasks are divided into vehicle end and RSU end in three task division methods to construct a task division model. Under the system model, the transmission model, power consumption model and computing model of the system are established. Finally, the total cost model is constructed. With the aim of minimizing the cost of building / updating road DTs, task partitioning is selected through joint optimization. Uplink channel allocation Vehicle transmission power and RSU computing power allocation To construct an optimization problem; The NoisyNet network is integrated into the Actor network of the Hybrid Action Representation (HyAR) algorithm. The optimization problem is solved using the NoisyNet-HyAR algorithm with integrated noise network, and a resource allocation scheme that minimizes the cost of building / updating road DT is learned.

2. The multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems according to claim 1, characterized in that, The optimization problem is described as follows: in: roadside unit The cost of building / updating road DT; Maximum allowable delay; For roadside units Total latency to complete DT build / update; Each RSU is used as a set , The vehicles are used as a set Channel set ; Indicates roadside unit Assigned to vehicles Computing power used to build / update DT; Indicates roadside unit For vehicles Selected task allocation method This indicates the task allocation method selected by the vehicle; For vehicles Channel allocation; Indicates time slot Vehicle Selected transmit power, This is the maximum transmission power. This is the minimum transmit power. Let be the decision variable, representing the channel. Whether to assign to a vehicle ; Constraint C1 means that the latency of building / updating the road DT should be less than the maximum allowable latency. Constraints C2 and C3 are constraints on RSU computational resources; constraint C4 means that the RSU must select and can only select one task partitioning method for the vehicle; constraint C5 is a capacity limit for channel resources; constraint C6 is a constraint on decision variables; and constraint C7 is a constraint on vehicle transmit power.

3. The multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems according to claim 1, characterized in that, The physical layer consists of vehicles on the road, each equipped with four 120° field-of-view cameras and communication and computing devices; the digital twin layer is deployed in RSUs on the road, and each RSU is configured with an edge server to provide computing resources for building / updating the DT. The RSU connects to vehicles within its coverage area via wireless communication and acts as a central processing unit to collect vehicle parameters, data collected by vehicle sensors, and channel status information within its coverage area.

4. The multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems according to claim 1 or 3, characterized in that, The DT consists of a static layer and a dynamic layer; the static layer contains three-dimensional physical models of static objects within the RSU coverage area, with information provided by high-definition maps; The dynamic layer contains dynamic objects on the road, and the information is derived from environmental information collected in real time by vehicles on the road, as well as vehicle parameters.

5. The multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems according to claim 4, characterized in that, The three task division methods are as follows: Method 1: The vehicle does not perform any local processing, but directly transmits four real-time video frames captured by the onboard cameras around the vehicle and the vehicle's own coordinates. The entire task of building / updating the DT is completed only by the RSU. Method 2: The vehicle and RSU collaborate to complete the preprocessing task. The vehicle uses a generalized reference representation segmentation model to process real-time video frames captured by the onboard camera, identifying dynamic objects in the perceived image information. Only the vehicle's own coordinates and the images of the dynamic objects are transmitted, while the RSU completes the remaining calculation steps. The ratio of the remaining data size to the original data size after using dynamic foreground and static background separation techniques is [not specified in the original text]. Method 3: The vehicle independently completes the preprocessing task, that is, based on Method 2, it further processes the images of dynamic objects in the scene, extracts the 3D feature parameters of the dynamic objects, and transmits the vehicle's own coordinates and the 3D feature parameters of other dynamic objects in the scene.

6. The multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems according to claim 1, characterized in that, The NoisyNet-HyAR algorithm, which employs a hybrid action representation using an integrated noise network, is used to solve the optimization problem, and includes the following steps: The optimization problem is modeled as a Markov decision process; the corresponding elements are defined as follows: state space :definition For time slots The status consists of the amount of data the vehicle needs to upload, the computing resources allocated by the base station, and channel status information. ;in This refers to the amount of data that the vehicle needs to upload. These are the computing resources already allocated to the base station. It is a channel state information vector; Action space : Defined as a time slot The actions involved include task partitioning and selection, uplink channel allocation, vehicle transmit power allocation, and RSU computing power allocation. ;in, It is a task allocation selection. It is the uplink channel allocation matrix. It is the vehicle's transmission power. It is the allocation of RSU computing power; reward function The reward function is used to measure the agent's performance and guide its decision-making process; in the first... Within each time slot, the instant reward can be defined as: ;in This is a penalty constant; a penalty mechanism is triggered when the agent's decision does not meet the time delay constraint. For time slots The lower the cost of building / updating DT, the greater the reward value. NoisyNet is integrated into the Actor network of the traditional HyAR algorithm, and the NoisyNet-HyAR algorithm is used to solve the optimization problem by employing the hybrid action representation of the integrated noise network.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-dimensional resource optimization method for vehicle-road cooperative digital twin systems as described in any one of claims 1 to 6.

Images