Intelligent driving edge computing resource cooperative configuration method and system for intelligent networked automobile

Abstract

The application provides a kind of intelligent driving edge computing resource coordination configuration method and system for intelligent networked vehicle, the method includes the following steps: with the total time delay of all vehicles received signal and completed computing task minimization as optimization goal, transmission power, transmission rate, channel, vehicle moving speed, perception accuracy, energy consumption, local computing resource as constraint condition, Markov decision process is constructed to solve Markov decision process, improved TD3 algorithm is used to solve model problem, and the best transmission power and computing resource allocation strategy of roadside unit are obtained.The application reasonably allocates transmission power and vehicle computing resources, and realizes the minimization of overall time delay of resource allocation.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving computing power networks and multi-dimensional computing power collaboration, specifically relating to a method and system for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles. Background Technology

[0002] With the rapid development of technology, the rapid advancement of large multimodal language models (MLLMs) has greatly promoted research in artificial intelligence. They have demonstrated outstanding performance in various fields, including intelligent connected vehicles and automatic control, especially in the field of in-vehicle intelligence. As technology continues to advance, the application prospects of large multimodal language models are becoming increasingly broad. However, directly deploying these large-scale MLLMs to customer devices is limited by the sheer size of the model parameters, leading to a significant decrease in the model's generalization ability when deployed on the device side. Therefore, a new approach to model deployment has been considered: combining cloud computing services. By deploying the model in the cloud, the in-vehicle system can fully utilize the powerful computing capabilities of the cloud to process and analyze massive amounts of voice data. This deployment method not only overcomes the problem of limited computing resources on the device side but also enables the in-vehicle system to perform real-time intelligent voice interaction, greatly promoting the development of autonomous driving technology and facilitating people's daily lives.

[0003] Autonomous vehicle technology, as its core support, is undergoing profound changes. However, facing increasingly demanding tasks requiring low latency and high computing power, traditional cloud computing models are proving inadequate due to remote transmission delays and bandwidth constraints. This is where the rise of mobile edge computing (MEC) has injected new vitality into autonomous vehicles. By deploying servers in locations such as roadside units, MEC can process data locally, significantly reducing transmission latency, improving resource scheduling flexibility, and alleviating the burden on cloud centers.

[0004] While this technology can alleviate the problems of high computational load and latency requirements in in-vehicle technology, its unique communication and perception radar operating modes limit the upper limit of intelligence when pursuing applications in new autonomous vehicles such as higher levels of autonomous driving. For example, the perception equipment required for autonomous driving (such as millimeter-wave radar and lidar) has limitations in detection range, environmental adaptability, and wireless resource utilization. At the same time, the rapid channel changes and Doppler frequency shift caused by the high-speed movement of vehicles also affect the accuracy and reliability of signal transmission. In summary, the goal is to solve the problems of low spectrum utilization and insufficient perception accuracy.

[0005] To address the aforementioned issues, a proposed solution is integrated sensing and communication, which uses a single hardware device to simultaneously implement communication and sensing functions. This not only reduces hardware costs and improves resource utilization but also enables communication and sensing to complement each other, achieving more accurate channel estimation, beam design, and multi-target management. However, in existing research on integrated sensing and communication, the complexity of orthogonal time-frequency multiplexing (OFDM) technology is too high, making it unsuitable for time-delay-sensitive tasks. Furthermore, in scenarios involving high-speed autonomous vehicles, OFDM technology causes Doppler frequency shifts, altering the carrier frequency and interfering with the beam, resulting in inaccurate information reception at the receiver. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the aforementioned background technology and provide a method and system for collaborative configuration of edge computing resources for intelligent connected vehicles. Based on orthogonal time-frequency-space integrated communication and sensing technology, resources are effectively allocated. Under constraints such as sensing accuracy and energy consumption, the goal is to minimize the total delay in task completion, and the transmission power and vehicle computing resources are allocated accordingly.

[0007] The technical solution adopted in this invention is: a method for collaborative configuration of edge computing resources for intelligent driving in intelligent connected vehicles, comprising the following steps:

[0008] In scenarios where edge layer, roadside unit, and vehicle layer collaborate to achieve intelligent driving, roadside unit uses orthogonal time-frequency-space technology to transmit integrated sensing and communication signals. With the optimization objective of minimizing the overall latency for all vehicles to receive signals and complete computational tasks, a Markov decision process is constructed with constraints such as transmission power, transmission rate, channel, vehicle speed, sensing accuracy, energy consumption, and local computing resources.

[0009] The Markov decision process

[0010] The state space is for the roadside unit to obtain the state information of the environment, the channel gain of each transmission channel, as well as interference information and vehicle information in each time interval.

[0011] The action space refers to the allocation method of roadside unit transmission power and computing resources;

[0012] The reward function includes a negative function of the average delay of multiple vehicles completing the task, and a penalty set during the training process when the policy given by the roadside unit as an agent fails to meet the constraints.

[0013] Solve the Markov decision process to obtain the optimal transmit power and computational resource allocation strategy for the roadside unit.

[0014] In the above technical solution, the Markov decision process is solved by an improved TD3 algorithm, wherein the solution process of the TD3 algorithm is optimized by controlling the update frequency of Critic and Actor, thereby obtaining the optimal transmission power and computing resource allocation strategy for the roadside unit.

[0015] In the above technical solution, during the process of solving the Markov decision process, the update frequency of the Critic network is reduced only when the absolute value of the temporal difference signal of the Critic network is greater than a set value; each time the Actor is updated, the Actor is updated only when the absolute value of the temporal difference signal of the Critic network is less than or equal to a set value; otherwise, the Actor update is skipped.

[0016] In the above technical solution, during the process of solving the Markov decision process, the selection probability of a sample is calculated based on its priority when updating the experience pool. A structured binary tree (Sum-tree) is considered for storage, where the weight of the parent node is the sum of the weights of its child nodes. Samples are stored in the Sum-tree according to their priority parameters. When collecting k samples, the experience pool is divided into k equidistant intervals, with each interval having a weight equal to 1 / k of the root node's weight. A random number is generated within each interval, and the leaf node in the Sum-tree closest to this random number is searched to obtain the corresponding sample. Each sample is assigned an importance sampling weight and normalized. Each new record added to the experience pool is given a higher initial priority to ensure that it is selected first in subsequent sampling processes. When the experience pool reaches its capacity limit, the lowest priority sample is removed.

[0017] In the above technical solution, the probability P(j) of selecting the j-th sample is calculated using the following formula:

[0018]

[0019] Where δ j Let ε be the temporal difference error of the j-th sample, B be the number of samples in the experience pool, and ε be the noise level to prevent the selection probability from being zero due to the temporal difference error being zero. α is a parameter that controls the sampling. When α = 0, it is the same as random sampling. This indicates the priority of the j-th sample, controlled by parameter α.

[0020]

[0021] y m This indicates that the parameter θ is based on the empirical value of the Q function. i The calculated Q value of the m-th sample in the experience pool;

[0022] The parameter θ represents the usage parameters of the m-th sample under state s and action a in the current time slot t. i Q-network predicted return value

[0023] w j These are importance weights, where K is the number of mini-batches sampled from the experience pool. δ represents the squared error between the predicted return and the current return for the m-th time. i＝1,2 This represents any two samples that are randomly selected, from which the optimal time-series difference error is chosen.

[0024] In the above technical solution, the importance weight w is defined by the following formula. j :

[0025]

[0026] Where K is the number of Mini-batch samples in the experience pool, and η∈(0,1) plays the role of bias correction, gradually approaching 1 as the training process progresses.

[0027] In the above technical solution, the constraints include:

[0028] The power allocated to the transmitter of the roadside unit must not exceed its maximum power;

[0029] The transmission rate of a single vehicle must not be less than its minimum transmission rate.

[0030] The number of DD grids assigned to a single vehicle must not exceed the maximum tolerance number;

[0031] The vehicle's speed is maintained within a certain range;

[0032] The lower bounds of angle and distance for Cramerlow are smaller than their minimum tolerance sensing accuracy;

[0033] The energy consumption generated locally by the vehicle must not exceed its maximum tolerance limit;

[0034] The computing resources allocated to the vehicle locally must not exceed the maximum computing power of the autonomous vehicle.

[0035] This invention provides a collaborative configuration system for intelligent driving edge computing resources for intelligent connected vehicles, used to implement the collaborative configuration method for intelligent driving edge computing resources for intelligent connected vehicles described in the above technical solution.

[0036] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles as described in the above technical solution.

[0037] The present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles as described above.

[0038] The beneficial effects of this invention are as follows: The resource collaborative configuration optimization method based on integrated communication and sensing proposed in this invention uses OTFS technology during downlink transmission, considers dynamic environmental changes and local resource constraints from multiple perspectives, establishes various models, proposes a latency optimization problem, and uses an improved TD3 algorithm to solve the resource collaborative configuration. This invention can achieve efficient resource utilization and precise intelligent driving under highly dynamic environments, diverse service requirements, multi-dimensional resource matching, and the need for intelligent adaptation to environmental changes. Under the constraints of roadside unit sensing accuracy requirements and vehicle energy consumption, this invention jointly allocates transmission power and vehicle computing resources for collaborative configuration, thereby minimizing the total latency of task transmission and local computing.

[0039] Furthermore, the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm excels in solving reinforcement learning tasks with continuous action spaces. Compared to traditional reinforcement learning methods, the TD3 algorithm introduces a dual Critic network and a delayed update mechanism, effectively reducing the bias in Q-value estimation, thereby improving policy stability and convergence speed. Using the improved TD3 algorithm to solve Markov decision processes in intelligent driving edge computing scenarios can more accurately optimize resource allocation strategies, thus more efficiently balancing communication, computing resources, and perception accuracy, ultimately reducing the overall latency of vehicle tasks.

[0040] Furthermore, this invention controls the update frequency of the Critic and Actor by controlling the change in the temporal difference (TD) signal. This delays Actor updates when the Critic network has a large bias, and only performs Actor updates when the Critic network is stable. This effectively reduces the variance in Q-value estimation, improving the model's learning performance and convergence speed. In addition, this dynamic adjustment mechanism avoids instability caused by frequent updates, making the model more robust and adaptable in complex edge computing scenarios.

[0041] Furthermore, this invention introduces a Sum-tree structure for priority sampling. This scheme effectively improves the utilization rate of high-priority samples without neglecting other samples. The Sum-tree structure reduces the time complexity of priority sampling, ensuring frequent use of high-priority samples while further reducing the computational overhead of sampling operations. This efficient priority sampling mechanism can improve the utilization efficiency of the experience replay pool, making the reinforcement learning process more efficient and accelerating the optimization of resource allocation strategies.

[0042] Furthermore, this invention defines the sample selection probability using temporal difference error (TD-error), ensuring that samples with larger errors are preferentially selected for training. TD-error reflects the impact of samples on Q-value estimation. Using this error to set the sample selection probability can enhance the learning of key samples, thereby improving the effectiveness of network parameter updates. In addition, by introducing noise and control parameters, overfitting caused by excessive sample concentration is avoided, improving the robustness and generalization ability of the model.

[0043] Furthermore, importance sampling weights can correct for biases caused by priority sampling, making model updates more accurate and preventing high-priority samples from excessively influencing parameter updates. Weight normalization limits weight values ​​to a certain range, further ensuring training stability. In this way, the model can utilize high-priority key samples during training while avoiding biases caused by high-priority samples excessively influencing training, thereby improving the overall accuracy and reliability of the model.

[0044] Furthermore, the constraints set by this invention, such as transmit power, transmission rate, DD grid limitations, vehicle speed, and energy consumption, ensure the practical applicability of the resource allocation strategy. These constraints make the optimization problem more aligned with the requirements of actual intelligent driving edge computing scenarios, thereby ensuring that computing and communication resources are allocated to meet the physical constraints and service needs of the vehicle and environment. Ultimately, these constraints ensure the completion rate of perception and computing tasks, reduce the risk of task failure, and guarantee resource utilization and safety in driving scenarios. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating an application scenario of the present invention;

[0046] Figure 2 This is a schematic diagram of the method and system architecture of the present invention;

[0047] Figure 3 This is a schematic diagram of the integrated communication and sensing scenario of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but these descriptions do not constitute a limitation on the present invention.

[0049] like Figure 1-3 As shown, this invention provides a method for collaborative configuration of edge computing resources for intelligent driving in intelligent connected vehicles, comprising the following steps:

[0050] In scenarios where edge layer, roadside unit, and vehicle layer collaborate to achieve intelligent driving, roadside unit uses orthogonal time-frequency-space technology to transmit integrated sensing and communication signals. With the optimization objective of minimizing the overall latency for all vehicles to receive signals and complete computational tasks, a Markov decision process is constructed with constraints such as transmission power, transmission rate, channel, vehicle speed, sensing accuracy, energy consumption, and local computing resources.

[0051] The Markov decision process

[0052] The state space is for the roadside unit to obtain the state information of the environment, the channel gain of each transmission channel, as well as interference information and vehicle information in each time interval.

[0053] The action space refers to the allocation method of roadside unit transmission power and computing resources;

[0054] The reward function includes a negative function of the average delay of multiple vehicles completing the task, and a penalty set during the training process when the policy given by the roadside unit as an agent fails to meet the constraints.

[0055] Solve the Markov decision process to obtain the optimal transmit power and computational resource allocation strategy for the roadside unit.

[0056] The principles of the present invention will be further explained below with reference to specific embodiments.

[0057] The application scenario of this invention is to integrate ISAC technology into the edge computing of autonomous vehicles, and establish a sensor-integrated vehicle edge computing scenario framework.

[0058] The framework for enabling in-vehicle computing scenarios through sensor integration, such as Figure 1 As shown, achieving intelligent driving requires collaboration among three layers: the edge layer, roadside units, and the vehicle layer. The core network of the edge layer globally controls all task allocation, and its network also provides computing power, processing information from different vehicles and sending it to the roadside units via downlink. The roadside units, acting as intermediaries, link the edge layer and the vehicle layer, and perceive the surrounding environment, acquiring real-time information about their surroundings. The communicating vehicles primarily perform local computing tasks after receiving signals from the integrated communication and sensing downlink, and autonomously make intelligent driving decisions.

[0059] Intelligent driving demands high safety standards and has a low fault tolerance rate. The key to achieving high-level autonomous driving lies in quickly and accurately completing communication, perception, and computation tasks. Current research on integrated communication and perception largely utilizes OFDM modulation methods. However, in the edge computing environment of autonomous vehicles, high-speed movement of the vehicle generates Doppler frequency shifts that interfere with the signal, posing a potential hazard to autonomous driving.

[0060] To address the above issues, this invention proposes that roadside units use OTFS technology to transmit ISAC signals. Taking into account factors such as vehicle movement, transmission power, sensing accuracy, local energy consumption, and computing resources, the invention rationally allocates transmission power and computing resources to reduce the overall latency for vehicle users to receive and process signals.

[0061] like Figure 2 As shown: The process of the resource collaborative configuration optimization scheme based on ISAC is as follows: (1) The edge layer sends task information such as route planning and accident avoidance generated by the data provider to the roadside unit. (2) The roadside unit calculates the resource collaborative configuration method according to the current environmental information and allocates appropriate transmission power and computing resources of the target vehicle. (3) Based on OTFS technology, the integrated communication and perception signal is modulated to perceive the surrounding environment while transmitting the task. (4) After receiving the communication signal, the target vehicle processes the task locally according to the allocated resources and completes the autonomous driving decision. (5) After receiving the echo of the perceived target, the roadside unit updates the environmental information locally.

[0062] Based on the above system framework, communication model, vehicle movement model, perception model, time delay model, and energy consumption model are established as constraints, and an optimization problem is established.

[0063] In communication models, such as Figure 3 The diagram shows multiple signal transmission paths. The line-of-sight link has the shortest transmission delay, and its Doppler frequency is V. The Doppler frequency of the non-line-of-sight link i is V. i * A discretized DD domain plane is constructed, divided into M·N grids. Each grid can represent a channel. The processed integrated inductive signal is obtained through OTFS modulation and demodulation. During signal transmission, pilot information transmission symbols are typically inserted at the transmitter. Based on OTFS, the transmitter converts the DD domain signal into a time domain signal for transmission in the channel, and the receiver converts the time domain signal back into a DD domain signal. For different users, the RSU transmits signals of different power levels by superimposing them according to non-orthogonal multiple access, and uses serial interference cancellation technology to eliminate interference between users.

[0064] In the vehicle movement model, the vehicle movement time is divided into n time intervals, with i representing each vehicle. The speed of each vehicle at each moment is recorded, and the speed is kept between its maximum and minimum values ​​at all times.

[0065] In the perception model, the Cramer-Rao Lower Bound (CRLB) is used as an indicator of perception performance. The main parameters that need to be perceived are the vehicle's direction of motion and the angle between the radar wave and the vehicle's distance.

[0066] In the latency model, the optimization objective is to reduce the overall latency for all users to receive signals and complete computation tasks. This latency can be divided into transmission latency and computation latency. Transmission latency is the latency t that the RSU transmits to the user. i tran The computational latency is the latency t for processing tasks originating from RSU tasks and additional tasks generated locally. i comp The total delay for the entire cycle is:

[0067] t i total =t i tran +t i comp

[0068] In the energy consumption model, after receiving the ISAC signal, the autonomous vehicle needs to demodulate the required signal locally and use it as input to execute the next decision-making task. The energy consumption required for local computation is expressed as follows: There will also be energy consumption during the acceptance process. At the same time, in order to ensure the safety of the vehicle itself, energy consumption must have an upper limit, let's assume that the upper limit is... The maximum energy consumption is expressed as follows:

[0069]

[0070] The optimization problem of this invention is how to maximize user work efficiency while ensuring constraints such as sensing accuracy, energy consumption, transmission power, and local computing resources. The optimization objective and constraints are defined as follows:

[0071]

[0072] C4:v min ≤v i,n ≤v max

[0073]

[0074] C8:f i,local ≤f i,max

[0075] C1 indicates that the power allocated to the RSU transmitter must not exceed the maximum power; C2 indicates that the transmission rate for a single user must not be less than the minimum transmission rate; C3 indicates that the number of DD grids allocated to a single user must not exceed the maximum tolerable number; C4 indicates that the vehicle's speed must be maintained within a certain range; C5 and C6 indicate that the Cramer-Rao lower bounds for angle and distance must be less than the minimum tolerable perception accuracy; C7 indicates that the energy consumption generated locally by the vehicle must not exceed the maximum tolerable limit; C8 indicates that the computing resources allocated locally to the vehicle must not exceed the maximum computing power of the autonomous vehicle. Here, "user" refers to the vehicle.

[0076] Specifically, optimize the target This means that the total latency of U users (i.e., vehicles) is summed to obtain the total latency of all users, and then minimized.

[0077] M*N represents the RSU transmit power corresponding to user i in the grid represented by k=1 and l=0 in the DD domain; M*N represents the total number of grids in the DD domain, and k and l represent the grid numbers in the DD domain, respectively.

[0078] This indicates that the transmission rate Ri of user i has a minimum value R. min .

[0079] VU = {1,2,...,i,...U} represents the set of vehicle users.

[0080] δ kl,i This indicates whether user i has been allocated a channel in the k, i coordinates of the DD domain. A value of 0 indicates that no channel has been allocated, while a value of 0 indicates that a signal has been allocated.

[0081] This represents the channels allocated to users across the entire DD domain, with added constraints w. max This makes the number have a maximum value.

[0082] ν min ≤ν i，n ≤ν max The motion time is divided into n time slots, ν i，n This represents the speed of vehicle i in the nth time slot, which has a maximum and a minimum value.

[0083] China F i,n F represents the beamforming matrix of user i. n ={F 1,n ,F 2,n ,...,F K,n},in p i,n N represents the transmit power allocated to user i within time slot n.t denoted by , where is the number of transmitting antennas, and a(·) is the array steering vector for transmitting and receiving.

[0084] Specifically represented as

[0085] θ i,n Let CRLB(θ) be the angle between the direction of vehicle i's movement in the nth time slot and the line-of-sight link. i,n ,F i,n ) represents the minimum variance bound of the parameter estimates for user i in the nth time slot, which depends specifically on its beamforming matrix F. i,n .

[0086] Similarly, it also has a constraint. Similarly to the previous one, θ i,n Change to d i,n , which is the distance between user i and the link terminal in the nth time slot.

[0087] This represents the local computing power consumption of the i-th user. and transmission energy consumption The sum has a maximum value.

[0088] f i,local ≤f i,max This represents the computing power f allocated to local vehicle i. i,local It must not exceed the maximum value f i,max .

[0089] Analysis of the problem reveals that the optimization problem proposed in this invention involves continuous variables and binary variables in the channel allocation process. This optimization problem belongs to the nonlinear programming problem of mixed integers, which is difficult to solve using conventional methods.

[0090] Therefore, this invention employs deep reinforcement learning to solve this MINLP problem. Specifically, it constructs a Markov decision process based on the optimization objective and corresponding constraints.

[0091] In an MDP (Multi-Level Processing), the unit that executes decisions is called an agent; in this invention, the agent is an RSU (Responsive Unit). The part that interacts with the agent is called the environment. An MDP typically consists of a quadruple (S, A, R, P). Where s... t ∈S is a finite set of states in the environment at time t, a t ∈A is the set of actions that the agent can perform at time r. t It is based on the agent in state s t Make the following action a t The AI ​​will use the rewards it receives to adjust its strategy for the next action. And p tIt is the state transition probability, representing the transition from the current state s. t Take action a t The probability of entering the next state. Typically p t The environment is unknown, but the state transition probability remains constant; this property is called the Markov property. The reward obtained by RSU and the next state depend only on the current RSU state and the action chosen at that time. Only in this way can RSU learn the patterns of environmental changes.

[0092] In this invention, the state space, action space, and reward function are defined as follows:

[0093] (1) State space: The RSU can obtain state information in the environment in each time interval, including channel allocation information of the RSU to mobile vehicle users, channel gain and interference information of each transmission channel and vehicle information, including vehicle information such as its location, speed, maximum allocable computing resources, maximum tolerable energy consumption and received power.

[0094] Action space: The RSU's transmit power and computing resource allocation method.

[0095] (2) Define a t =(P(t),F(t)),a t ∈A. Transmit power P=(p kl,i ,i∈VU,k∈[0,M-1],l∈[0,N-1]),p kl,i For positive continuous variables and Σp < P max Vehicle computing resources F = (f i,local ,i∈VU),f i,local For continuous variables and 0≤f i,local ≤f i,max Reward Function: Since the optimization objective of this problem is to minimize latency, it is known that the instantaneous reward obtained by RSU is negatively correlated with the average latency of multiple users completing the task. Furthermore, during training, if the agent's policy fails to meet energy consumption and perception accuracy constraints, appropriate penalties must be imposed. The reward function r of the agent is defined as follows: t for:

[0096]

[0097] Among them, p1, p2, and p3 are the penalties for exceeding the energy consumption limit, angle perception accuracy, and distance perception accuracy, respectively.

[0098] When an agent makes a decision that satisfies the constraints, it receives a reward based on the total system latency. When the constraints are not met, it incurs a corresponding penalty. The optimization objective of this chapter is to maximize the cumulative discounted reward of the long-term system.

[0099] Reinforcement learning is essentially a Markov decision process. Given an MDP (S, A, R, P), the agent determines the outcome based on the current state s. t To perform an action a according to a certain pattern t It then enters the next state, and the intelligent system receives the reward r for that action. t By continuously exploring and adjusting its behavior, an intelligent agent adapts to its environment, ultimately developing a strategy that yields the maximum long-term reward. In summary, reinforcement learning is about finding an optimal policy π. * This maximizes the value of the action.

[0100]

[0101] Where Q is called the behavioral value function Q. π (s,a)=E π [G t |S t =s,A t =a], representing the expected benefit gained from taking action a in state s. This invention obtains the allocation method of roadside unit transmission power and computing resources by finding the optimal strategy.

[0102] To solve the aforementioned Markov decision process, this invention proposes a dynamically adaptive delayed update strategy and a priority experience selection method, improving the experience selection rules and delayed update frequency in the TD3 algorithm. The aim is to flexibly adjust the update frequency based on the actual situation during training and prioritize the use of high-value samples, thereby further optimizing the performance of the TD3 algorithm and improving the efficiency and accuracy of solving the resource collaborative allocation problem.

[0103] A dynamically adaptive delayed update strategy should reduce the frequency of Critic updates based on changes in the target network. This reduces the variance of Q-value updates, yields more accurate Q-values, and accelerates convergence. The temporal difference signal TD-error reflects the range of changes in the Critic network; a larger value indicates more changes to the Critic network and a greater bias in Q-value estimation. In this case, the Actor network update needs to be delayed.

[0104] This invention employs a dynamically adaptive delayed update strategy to reduce the update frequency of the Critic network based on changes in the target network, thereby reducing the variance of Q-value updates, obtaining more accurate Q-values, and accelerating convergence. Specifically, it includes the following steps:

[0105] 1) Monitoring time-series differential signals:

[0106] During each update, the temporal difference (TD) signal of the Critic network is calculated, representing the difference between the Q-value estimate in the current state and the target Q-value.

[0107] 2) Evaluate the Q-value deviation:

[0108] The degree of change in the Critic network is assessed based on the magnitude of the time-series differential signal.

[0109] When the absolute value of the time-series difference signal is large, it indicates that the estimation bias of the Critic network is large, and the update of the delayed Actor network needs to be considered.

[0110] 3) Dynamically adjust the Critic update frequency:

[0111] Set a threshold; when the absolute value of the sequential differential signal exceeds this threshold, reduce the update frequency of the Critic network.

[0112] Specifically, this could be achieved by skipping the Critic network update during Actor network updates, or by extending the Critic network update cycle.

[0113] 4) Delay Actor updates:

[0114] If updates to the Critic network are delayed, ensure that the Actor network is updated only during this period when the state estimates are relatively stable.

[0115] A parameter d can be introduced to represent the update period of the Actor network. The Actor is only updated when the deviation of the Critic network is small, that is, when the absolute value of the time-series difference signal is less than the threshold.

[0116] 5) Feedback mechanism:

[0117] Regularly check the performance of the Critic network to ensure that updated strategies are effective.

[0118] Based on the convergence of the Critic network and the accuracy of the Q-value, the threshold and Actor update period d are dynamically adjusted.

[0119] Regarding the priority selection of experience samples, due to the limited capacity of the experience pool, discarding high-value data when there is a large amount of sample data is detrimental to training. Therefore, it is necessary to perform ordered and structured operations on the experience pool. The priority selection method uses a greedy strategy to select samples with high rewards, and the rewards are evaluated using TD-error. However, simply selecting samples with high rewards can lead to overfitting of TD-error and loss of sample diversity.

[0120] Therefore, this invention uses a combination of random priority sampling, offset sampling, and importance sampling. This ensures that even the lowest priority samples have a certain probability of being selected, while making the most of high-priority data samples. The probability of each sample being selected is determined by the TD-error. The probability of sample j being selected is:

[0121]

[0122] Where δ j Let TD-error be the number of samples in the experience pool, B be the number of samples in the pool, ε be the noise level to prevent the selection probability from being zero when TD-error is zero, and α be the parameter that controls the sampling. When α = 0, it is the same as random sampling.

[0123] Each sample has a priority parameter, and samples are selected according to priority during each sampling, which undoubtedly increases additional computational and time overhead. A time-saving tree structure could be considered to store this data.

[0124] For the priority parameter of each sample, a structured binary tree called a Sum-tree can be used for storage. This structure is characterized by the parent node's weight being the sum of its child node weights. Samples are stored in the Sum-tree according to their priority parameters. When k samples need to be collected, the empirical samples are divided into k equally spaced intervals. This is done by dividing the root node by k, randomly generating a number for each interval, and then searching the structured Sum-tree for the leaf node that most closely matches that number. This allows for the selection of suitable data with minimal time overhead and ensures that samples with low probability are not selected simultaneously.

[0125] While adding selection probabilities to samples improves the utilization of high-priority samples, this high utilization can also lead to biases in neural network parameter updates.

[0126] To eliminate bias, an update rule for the importance sampling weights is used. The importance weights w of the samples are defined. j :

[0127]

[0128] Where K is the number of mini-batches sampled in the experience pool, and η∈(0,1) serves as a bias correction factor, gradually approaching 1 during training. Importance weight w j The purpose is to utilize high-priority samples more frequently without significantly impacting each parameter update. To ensure the stability of parameter updates, the importance weight coefficient w is... j Normalized to the (0,1) interval. The modified TD-error calculation rule is as follows:

[0129]

[0130] From this point on, each new record added to the experience replay pool is assigned a higher priority, ensuring that the record has a greater probability of appearing in subsequent sampling processes. When the experience pool reaches its maximum capacity, the lowest priority record is discarded.

[0131] This algorithm can effectively solve the proposed optimization problem, thereby minimizing the total task completion delay. The pseudocode is as follows:

[0132]

[0133]

[0134] To analyze the performance of the improved TD3 algorithm proposed in this invention, this specific embodiment selects DDQN, MADDPG, MRL-DDPG, and other algorithms that also consider the collaborative allocation of edge computing resources for intelligent connected vehicles as comparison algorithms. The performance evaluation metrics selected by this invention are as follows:

[0135] (1) Training reward: refers to the reward obtained by the agent during the training process of the deep reinforcement learning algorithm. It is determined according to the reward function formula and determines the convergence of the algorithm during the training process.

[0136] (2) Bit error rate (BER): The smaller the BER, the more stable the signal transmission and the higher the communication quality.

[0137] (3) Average latency: Average latency refers to the average latency of all vehicle users completing their tasks. The lower the average latency, the better the effect of the resource collaborative allocation system.

[0138] (4) Average energy consumption: Average energy consumption refers to the average energy consumption of all users of the vehicle when receiving signals and processing computing tasks locally.

[0139] (5) Maximum user latency: Maximum user latency reflects the time cost for the user with the longest service response latency to complete a task among all users in the system. Average latency reflects the overall system situation, while maximum user latency reflects the worst-case scenario in the system.

[0140] (6) Channel transmission rate: The channel transmission rate measures the efficiency of information transmission within a system, and its unit is bit / s / Hz. The higher the channel transmission rate, the higher the information transmission efficiency and the better the performance.

[0141] (7) Cramerlow Lower Bound (CRLB): CRLB reflects the lower bound of radar's detection accuracy for targets. The smaller the CRLB, the higher the radar's detection accuracy and the better the detection effect.

[0142] Experimental results show that, in terms of the number of vehicles, when the number of vehicles is 10, the improved TD3 algorithm reduces average latency by 38.2%, 24.2%, and 13.0% respectively compared to DQN, MADDPG, and MRL-DDPG, and reduces average energy consumption by 27.3%, 19.7%, and 14.3% respectively. The channel transmission rate decreases as the number of vehicles increases. In terms of task load, when the task load is 0.8 Mbit, the ITD3 algorithm reduces latency by 13.2%, 8.1%, and 6.2% compared to DDQN, MADDPG, and MRL-DDPG, respectively. When the task load is higher, such as 3.2 Mbit, the ITD3 algorithm reduces latency distribution by 19.3%, 12.3%, and 8.1% compared to DDQN, MADDPG, and MRL-DDPG. With increasing task load, the ITD3 algorithm performs better in reducing the average task completion latency. In terms of transmission power, at a transmission power of 20dBm, the ITD3 algorithm reduces latency by 18.1%, 11.5%, and 8.3% compared to DDQN, MADDPG, and MRL-DDPG, respectively. At a transmission power of 30dBm, the ITD3 algorithm reduces latency by 19.2%, 12.4%, and 9.7% compared to DDQN, MADDPG, and MRL-DDPG, respectively. It can be seen that as the transmit power increases, the channel transmission rate also increases.

[0143] In summary, the improved TD3 algorithm proposed in this invention can accelerate the convergence speed, and in the integrated communication and sensing environment, the resource collaborative configuration scheme based on OTFS technology is better suited to high-speed moving vehicle scenarios compared to the traditional OFDM-based scheme. It can be seen that, from different perspectives, the improved TD3 algorithm proposed in this invention has greater advantages than other algorithms in reducing the average latency and average energy consumption of task completion.

[0144] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for intelligent driving edge computing resource collaborative configuration for intelligent vehicles, characterized in that: The method comprises the following steps: In the scenario of intelligent driving realized by cooperation of an edge layer, a roadside unit and a vehicle layer, the roadside unit transmits integrated sensing and communication signals using orthogonal time-frequency-space technology; A Markov decision process is constructed with a minimum total time delay of all vehicles receiving signals and completing a computing task as an optimization objective and with a transmission power, a transmission rate, a channel, a vehicle moving speed, a sensing accuracy, an energy consumption, and a local computing resource as constraint conditions; In the Markov decision process, A state space is state information obtained by the roadside unit in each time interval, channel gain of each transmission channel, interference information and vehicle information in an environment; An action space is a distribution mode of the transmission power and the computing resource of the roadside unit; A reward function includes a negative function of an average time delay of multiple vehicles completing a task and a set penalty when a strategy given by the roadside unit as an intelligent agent does not meet the constraint condition in a training process; The Markov decision process is solved to obtain an optimal transmission power and computing resource distribution strategy of the roadside unit; The Markov decision process is solved by using an improved TD3 algorithm, in which an updating frequency of a Critic and an Actor is controlled to optimize a solving process of the TD3 algorithm, so as to obtain the optimal transmission power and computing resource distribution strategy of the roadside unit; In the process of solving the Markov decision process, the updating frequency of the Critic network is reduced only when an absolute value of a time series difference signal of the Critic network is greater than a set value; and the updating of the Actor is executed only when the absolute value of the time series difference signal of the Critic network is less than or equal to the set value; otherwise, the updating of the Actor is skipped. 2.The intelligent driving edge computing resource cooperative configuration method for intelligent vehicles according to claim 1, wherein: In the process of solving the Markov decision process, a selection probability of a sample is calculated according to a priority of the sample when an experience pool is updated; a Sum-tree structured binary tree is used for storage, and the Sum-tree has a feature that a parent node weight is a sum of child node weights; when k samples are collected, the experience pool is divided into k equidistant intervals, and a weight of each interval is 1 / k of a root node weight; a random number is generated in each interval, a leaf node closest to the random number in the Sum-tree is searched, and a corresponding sample is obtained; each sample is assigned an importance sampling weight and is normalized; each new record is given a high initial priority when added to the experience pool, so as to be preferentially selected in a subsequent sampling process; when the experience pool reaches an upper limit of a capacity, a sample with a lowest priority is removed. 3.The method of claim 2, wherein: A selection probability P(j) of the jth sample is calculated by using the following formula: ； wherein is the timing difference error of the jth sample, B is the number of samples in the experience pool, and is the noise is used to prevent the timing difference error being zero leading to the selection probability being zero, is the parameter controlling the sampling, when is the same as random sampling; represents the priority of the jth sample controlled by the parameter ； y m represents the empirical value of the Q function parameter the current Q value of the mth sample in the empirical pool represents the usage parameter of the mth sample under state s and action a at the current time slot t the Q-network predicted return value of is the importance weight, K is the number of Mini-batch sampled from the experience pool, represents the square error between the predicted return of the mth and the current return; represents any two samples extracted, and from the two samples, an optimal time difference error is selected. 4.The method of claim 3, wherein the method further comprises: Importance weights are defined using the following formula : ； where K is the number of Mini-batches sampled from the experience pool, acts as a bias correction, gradually tending towards 1 as the training process progresses. 5.The method of claim 1, wherein: The constraint conditions include: The power distributed by the roadside unit transmission end cannot exceed a maximum power; The transmission rate of a single vehicle cannot be less than a minimum transmission rate; The number of distributed DD grids of a single vehicle cannot exceed a maximum tolerance number; The moving speed of the vehicle is kept within a certain range; The Cramer-Rao lower bound of the angle and the distance is less than a minimum tolerance sensing accuracy; The energy consumption generated by the vehicle locally cannot exceed a maximum tolerance limit; The computing resources allocated to the vehicle locally shall not exceed the maximum computing capacity of the autonomous vehicle.

6. An intelligent driving edge computing resource cooperative configuration system for intelligent networked vehicles, characterized in that: The method for implementing the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles according to any one of claims 1-5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles according to any one of claims 1-5.

8. An electronic device, comprising: Comprise: A memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles according to any one of claims 1-5.

Images