Intelligent driving edge computing resource collaborative configuration method and system for intelligent connected automobile

Abstract

The invention provides an intelligent driving edge computing resource collaborative configuration method and system for an intelligent networked automobile, and the method comprises the following steps: minimizing the total time delay of all vehicles receiving signals and completing a computing task as an optimization target; the transmitting power, the transmission rate, the channel, the vehicle moving speed, the sensing precision, the energy consumption and the local computing resources are used as constraint conditions, a Markov decision process is constructed to solve the Markov decision process, an improved TD3 algorithm is used to solve a model problem, and the optimal transmitting power and computing resource allocation strategy of the roadside unit is obtained. According to the method, the transmitting power and the computing resources of the vehicles are reasonably allocated, and the total time delay of resource allocation is minimized.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving computing power network and multi-dimensional computing power collaboration, and specifically relates to a method and system for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles. Background Art

[0002] With the rapid development of science and technology, the rapid development of multimodal large language models (MLLMs) has greatly promoted the research of artificial intelligence technology, and has shown excellent performance in the fields of intelligent connected vehicles, automatic control and other fields. Especially in the field of in-vehicle intelligence, with the continuous advancement of technology, the application prospects of multimodal large language models are becoming increasingly broad. However, the direct deployment of such large-scale MLLMs on customer devices is limited by the huge model parameters, resulting in a significant decrease in the generalization ability of the model when deployed on the device side. Therefore, people have considered a new idea for model deployment: combining cloud computing services, that is, by deploying the model on the cloud, the vehicle system can make full use of the powerful computing power of the cloud to process and analyze massive voice data. This deployment method not only overcomes the problem of limited computing resources on the device side, but also enables the vehicle system to perform intelligent voice interaction in real time, greatly promoting the development of autonomous driving technology and facilitating people's daily lives.

[0003] As its core support, autonomous vehicle technology is undergoing profound changes. However, faced with the growing number of low-latency, high-computing tasks, the traditional cloud computing model is unable to cope with the long-distance transmission delay and bandwidth pressure. At this time, the rise of mobile edge computing has injected new vitality into autonomous vehicles. By deploying servers in locations such as roadside units, MEC can process data nearby, significantly reduce transmission latency, improve resource scheduling flexibility, and reduce the burden on cloud centers.

[0004] Although this technology can alleviate the problems of large amount of calculation and high latency requirements of vehicle-mounted technology, its unique communication and perception radar working mode limits the upper limit of intelligent work when pursuing the application of new autonomous driving vehicles such as higher-level autonomous driving. For example, the perception equipment required for autonomous driving (such as millimeter wave radar and lidar) has limitations in detection range, environmental adaptability and wireless resource utilization. At the same time, the rapid change of channels and Doppler frequency shift caused by high-speed vehicle movement also affect the accuracy and reliability of signal transmission. In summary, it is necessary to solve the problems of low spectrum utilization and insufficient perception accuracy.

[0005] In response to the above problems, a proposed solution is to integrate perception and communication, that is, to realize communication and perception functions simultaneously through a set of hardware equipment, which not only reduces hardware costs and improves resource utilization, but also enables communication and perception to assist each other to achieve more accurate channel estimation, beam design and multi-target management. However, in the existing research work on integrated communication and perception, the complexity of orthogonal time-frequency multiplexing technology is too high and is not suitable for delay-sensitive tasks. Orthogonal frequency division multiplexing technology produces Doppler frequency shift in the scenario of high-speed autonomous driving vehicles, causing the carrier frequency to change, which has a certain interference with the beam, resulting in the inability of the receiving end to accurately obtain information. Summary of the invention

[0006] The purpose of the present invention is to address the shortcomings of the above-mentioned background technology and to provide a method and system for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles. This method effectively allocates resources based on the integrated communication perception technology of orthogonal time-frequency-space. Under the constraints of perception accuracy, energy consumption, etc., the transmission power and vehicle computing resources are allocated with the optimization goal of minimizing the total delay in task completion.

[0007] The technical solution adopted by the present invention is: a method for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles, comprising the following steps:

[0008] In the scenario where the edge layer, roadside unit and vehicle layer cooperate to realize intelligent driving, the roadside unit uses orthogonal time-frequency-space technology to transmit integrated perception and communication signals; with the optimization goal of minimizing the overall delay for all vehicles to receive signals and complete computing tasks, the transmission power, transmission rate, channel, vehicle moving speed, perception accuracy, energy consumption, and local computing resources are used as constraints to construct a Markov decision process;

[0009] In the Markov decision process,

[0010] The state space is for the roadside unit to obtain the state information in the environment and the channel gain of each transmission channel as well as the interference information and vehicle information in each time interval;

[0011] The action space is the allocation of transmission power and computing resources of roadside units;

[0012] The reward function includes the negative function of the average delay of multiple vehicles to complete the task, and the set penalty when the strategy given by the roadside unit as an intelligent agent does not meet the constraints during the training process;

[0013] Solve the Markov decision process to obtain the optimal transmission power and computing resource allocation strategy for roadside units.

[0014] In the above technical solution, the solution of the Markov decision process is implemented by an improved TD3 algorithm, wherein the update frequency of Critic and Actor is controlled to optimize the solution process of the TD3 algorithm, thereby obtaining the optimal transmission power and computing resource allocation strategy of the roadside unit.

[0015] In the above technical solution, in the process of solving the Markov decision process, the update frequency of the Critic network is reduced only when the absolute value of the time difference signal of the Critic network is greater than the set value; each time the Actor is updated, the Actor is updated only when the absolute value of the time difference signal of the Critic network is less than or equal to the set value; otherwise, the update of the Actor is skipped.

[0016] In the above technical scheme, in the process of solving the Markov decision process, when updating the experience pool, the selection probability of the sample is calculated according to its priority; the structured binary tree storage of Sum-tree is considered, and the structural feature of the structure is that the weight of the parent node is the sum of the weights of the child nodes, and the samples are stored in the Sum-tree according to the priority parameters; when k samples need to be collected, the experience pool needs to be divided into k equidistant intervals, and the weight of each interval is 1\k of the root node weight; a random number is generated in each interval, and the leaf node closest to the random number in the Sum-tree is searched to obtain the corresponding sample; each sample is assigned an importance sampling weight and normalized; each time a new record is added to the experience pool, a higher initial priority is given to ensure that the record is selected first in the subsequent sampling process; when the experience pool reaches the upper limit of capacity, the sample with the lowest priority is removed.

[0017] In the above technical solution, the following formula is used to calculate the probability of selecting the jth sample P(j):

[0018]

[0019] where δ j is the temporal difference error of the jth sample, B is the number of samples in the experience pool, the noise ε is used to prevent the temporal difference error from being zero, resulting in the probability of selection being zero, α is the parameter controlling the sampling, when α = 0, it is the same as random sampling; P(i) is the priority of the i-th sample; P(j) is the priority of the j-th sample; represents the priority of the jth sample controlled by parameter α;

[0020]

[0021] y m Indicates that according to the empirical value parameter θ of the Q function i The calculated current Q value of the mth sample in the experience pool;

[0022] represents the usage parameter θ of the mth sample in state s and action a at the current time slot t i The return value predicted by the Q network

[0023] w j is the importance weight, K is the number of Mini-batches sampled from the experience pool, Indicates the calculation of the square error between the m-th predicted return and the current return; δ i＝1,2 Represents two arbitrarily selected samples, from which an optimal timing difference error is selected.

[0024] In the above technical solution, the importance weight w is defined by the following formula: j :

[0025]

[0026] Among them, K is the number of mini-batches sampled by the experience pool, and η∈(0,1) plays the role of bias correction and gradually tends to 1 during the training process.

[0027] In the above technical solution, the constraints include:

[0028] The power allocated to the transmitter of the roadside unit shall not exceed its maximum power;

[0029] The transmission rate of a single vehicle shall not be less than its minimum transmission rate;

[0030] The number of DD grids allocated to a single vehicle shall not exceed the maximum tolerance;

[0031] The vehicle's movement speed is kept within a certain range;

[0032] The Cramer-Rao lower bounds of angles and distances must be smaller than their minimum tolerable perceptual accuracy;

[0033] The energy consumption generated locally by the vehicle must not exceed its maximum tolerable limit;

[0034] The computing resources allocated to the vehicle must not exceed the maximum computing capacity of the autonomous vehicle.

[0035] The present invention provides a DNN model partitioning system for vehicle-mounted edge intelligence, which is used to implement the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles described in the above technical solution.

[0036] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles as described in the above technical solution is implemented.

[0037] The present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles as described in the above technical solution.

[0038] The beneficial effects of the present invention are as follows: the resource collaborative configuration optimization method based on communication perception integration proposed in the present invention uses OTFS technology in the downlink transmission process, considers dynamic environmental changes and local resource limitations in many aspects, establishes multiple models, proposes delay optimization problems, and uses an improved TD3 algorithm to solve resource collaborative configuration. The present invention can achieve efficient resource utilization and precise intelligent driving under highly dynamic environments and diversified business needs, as well as multi-dimensional resource matching and multiple factors that require intelligent adaptation to environmental changes. Under the constraints of the perception accuracy requirements of roadside units, vehicle energy consumption, etc., the present invention jointly allocates transmission power and vehicle computing resources for collaborative configuration, thereby minimizing the total delay of task transmission and local computing.

[0039] Furthermore, the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm performs well in solving reinforcement learning tasks in continuous action spaces. Compared with traditional reinforcement learning methods, the TD3 algorithm introduces a dual critic network and a delayed update mechanism to effectively reduce the deviation of Q-value estimation, thereby improving the stability and convergence speed of the strategy. Using the improved TD3 algorithm to solve the Markov decision process in the intelligent driving edge computing scenario can more accurately optimize the resource allocation strategy, thereby more efficiently balancing communication, computing resources and perception accuracy, and ultimately reducing the overall latency of vehicle tasks.

[0040] Furthermore, the present invention controls the update frequency of Critic and Actor by the change of the time difference (TD) signal, so that the update of Actor is delayed when the Critic network deviation is large, and the update of Actor is performed only when the Critic network is stable. This can effectively reduce the variance in Q value estimation and improve the learning effect and convergence speed of the model. In addition, the dynamic adjustment mechanism can avoid the instability caused by frequent updates, making the model more robust and adaptable in complex edge computing scenarios.

[0041] Furthermore, the present invention introduces a Sum-tree structure for priority sampling, which can effectively improve the utilization rate of high-priority samples without neglecting other samples. The Sum-tree structure reduces the time complexity of priority sampling, while ensuring the frequent utilization of high-priority samples, further reducing the computational overhead of the sampling operation. This efficient priority sampling mechanism can improve the utilization efficiency of the experience replay pool, make the reinforcement learning process more efficient, and accelerate the optimization of resource allocation strategies.

[0042] Furthermore, the present invention defines the probability of sample selection by using the time series difference error (TD-error) to ensure that samples with larger errors are preferentially selected for training. TD-error reflects the influence of samples on Q-value estimation. Using this error to set the probability of sample selection can enhance the learning of key samples, thereby improving the effectiveness of network parameter updating. In addition, by introducing noise and controlling parameters, overfitting caused by excessive sample concentration is avoided, and the robustness and generalization ability of the model are improved.

[0043] Furthermore, importance sampling weights can correct the deviation caused by priority sampling, making model updates more accurate and preventing high-priority samples from overly affecting parameter updates. Weight normalization limits the weight value to a certain range, further ensuring the stability of training. In this way, the model can not only utilize key samples with higher priority during training, but also avoid the deviation caused by excessive influence of high-priority samples on training, thereby improving the accuracy and reliability of the overall model.

[0044] Furthermore, the constraints set by the present invention, such as transmission power, transmission rate, DD grid limit, vehicle speed and energy consumption, can ensure the practical applicability of the resource allocation strategy. These constraints make the optimization problem more in line with the requirements of the actual intelligent driving edge computing scenario, thereby ensuring that computing resources and communication resources can meet the physical constraints and service requirements of the vehicle and environment when allocated. The setting of such constraints can ultimately ensure the completion rate of perception and computing tasks, reduce the risk of task failure, and ensure resource utilization and safety in driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of an application scenario of the present invention;

[0046] Figure 2 It is a schematic diagram of the method and system architecture of the present invention;

[0047] Figure 3 It is a schematic diagram of the communication perception integration scenario of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but they do not constitute a limitation on the present invention.

[0049] like Figure 1-3 As shown, the present invention provides a method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles, comprising the following steps:

[0050] In the scenario where the edge layer, roadside unit and vehicle layer cooperate to realize intelligent driving, the roadside unit uses orthogonal time-frequency-space technology to transmit integrated perception and communication signals; with the optimization goal of minimizing the overall delay for all vehicles to receive signals and complete computing tasks, the transmission power, transmission rate, channel, vehicle moving speed, perception accuracy, energy consumption, and local computing resources are used as constraints to construct a Markov decision process;

[0051] In the Markov decision process,

[0052] The state space is for the roadside unit to obtain the state information in the environment and the channel gain of each transmission channel as well as the interference information and vehicle information in each time interval;

[0053] The action space is the allocation of transmission power and computing resources of roadside units;

[0054] The reward function includes the negative function of the average delay of multiple vehicles to complete the task, and the set penalty when the strategy given by the roadside unit as an intelligent agent does not meet the constraints during the training process;

[0055] Solve the Markov decision process to obtain the optimal transmission power and computing resource allocation strategy for roadside units.

[0056] The principle of the present invention is further explained below with reference to specific embodiments.

[0057] The application scenario of the present invention is to integrate ISAC technology into the edge computing of autonomous driving vehicles to establish a synaesthesia-integrated vehicle-enabled edge computing scenario framework.

[0058] The framework of synaesthesia integration empowering vehicle computing scenarios is as follows Figure 1 As shown in the figure, to achieve intelligent driving, the edge layer, roadside unit and vehicle layer need to cooperate. The core network of the edge layer globally controls all task allocations. Its network also provides computing performance, processes information from different vehicles, and sends it to the roadside unit through the downlink. The roadside unit acts as an intermediary, linking the edge layer and the vehicle layer, and perceives the surrounding environment to obtain surrounding information in real time. The communication vehicle mainly performs local computing tasks after receiving the signal of the communication perception integrated downlink, and makes intelligent driving decisions autonomously.

[0059] Intelligent driving has high safety requirements and low fault tolerance. How to quickly and accurately complete communication tasks, perception tasks, and computing tasks is the key to achieving high-level autonomous driving. In the current research on integrated communication and perception, OFDM modulation methods are mostly used. However, in the edge computing environment of autonomous driving vehicles, high-speed moving vehicles will produce Doppler frequency shifts that interfere with the signal, thus posing a hidden danger to autonomous driving.

[0060] To solve the above problems, the present invention proposes to allow roadside units to use OTFS technology to transmit ISAC signals, consider limiting factors such as vehicle movement, transmission power, perception accuracy, local energy consumption, and computing resources, reasonably allocate transmission power and computing resources, and reduce the overall delay of vehicle users receiving and processing signals.

[0061] like Figure 2 As shown in the figure, the ISAC-based resource collaborative configuration optimization solution process is as follows: (1) The edge layer sends the task information such as route planning and accident avoidance generated by the data provider to the roadside unit. (2) The roadside unit calculates the resource collaborative configuration method based on the current environmental information and allocates the appropriate transmission power and computing resources to the target vehicle. (3) Based on the OTFS technology, the communication and perception integrated signal is modulated to perceive the surrounding environment while transmitting the task. (4) After receiving the communication signal, the target vehicle processes the task locally according to the allocated resources and completes the autonomous driving decision. (5) After receiving the echo from the perceived target, the roadside unit updates the environmental information locally.

[0062] Based on the above system framework, the communication model, vehicle movement model, perception model, delay model, and energy consumption model are established as constraints, and the optimization problem is established.

[0063] In the communication model, Figure 3 There are multiple signal transmission paths shown, among which the line-of-sight link has the shortest transmission delay and its Doppler frequency is V. The Doppler frequency of the non-line-of-sight link i is V i * . Construct a discretized DD domain plane, which is divided into M·N grids. Each grid can represent a channel. The processed synaesthesia integrated signal is obtained by modulation and demodulation through OTFS technology. When transmitting a signal, the transmission symbol of the pilot information is usually inserted at the transmitting end. Based on the OTFS technology, the transmitting end converts the DD domain signal into a time domain signal for transmission in the channel, and the receiving end converts the time domain signal back to a DD domain signal. For different users, the RSU superimposes and transmits signals of different powers according to non-orthogonal multiple access, and uses serial interference cancellation technology to eliminate interference between users.

[0064] In the vehicle mobility model, the vehicle movement time is divided into n time intervals, i represents each vehicle, and its speed at each moment is recorded to ensure that the speed always remains between the maximum values.

[0065] In the perception model, Cramer-Rao Lower Bound (CRLB) is used as an indicator of perception performance. The main parameters that need to be perceived are the vehicle's movement direction, the angle of the radar wave, and the vehicle's movement distance.

[0066] In the delay model, the optimization goal is to reduce the overall delay for all users to receive signals and complete computing tasks. This delay can be divided into transmission delay and computing delay. Transmission delay is the delay t of RSU transmission to users. i tran , the computational delay is the delay of processing tasks from RSU and additional tasks generated locally i comp The total delay of the whole cycle is:

[0067] t i total =t i tran +t i comp

[0068] In the energy consumption model, after receiving the ISAC signal, the autonomous driving vehicle needs to demodulate the required signal locally and use the signal as input to perform the next decision task. The energy consumption required for local calculation is expressed as There will also be acceptance energy consumption during the acceptance process At the same time, in order to ensure the safety of the vehicle itself, the energy consumption must have an upper limit, assuming that its upper limit is The maximum energy consumption is expressed as follows:

[0069]

[0070] The optimization problem of the present invention is how to maximize the user's work efficiency under the constraints of perception accuracy, energy consumption, transmission power, local computing resources, etc. The optimization objectives and constraints are defined as follows:

[0071]

[0072]

[0073] C4:v min ≤v i,n ≤v max

[0074]

[0075] C8:f i,local ≤f i,max

[0076] C1 indicates that the power allocated by the RSU transmitter must not exceed the maximum power, C2 indicates that the transmission rate of a single user must not be less than the minimum transmission rate, C3 indicates that the number of DD grids allocated to a single user must not exceed the maximum tolerance number, and C4 indicates that the vehicle's movement speed is kept within a certain range. C5 and C6 indicate that the Cramer-Rao lower bounds of angles and distances must be less than the minimum tolerance perception accuracy, and C7 indicates that the energy consumption generated locally by the vehicle must not exceed the maximum tolerance limit. C8 indicates that the computing resources allocated to the local vehicle must not exceed the maximum computing capacity of the autonomous driving vehicle. Among them, user represents the vehicle.

[0077] Specifically, the optimization goal The total delay of U users (i.e., vehicles) is added together to obtain the total delay of all users and minimized.

[0078] It represents the RSU transmission power corresponding to user i in the grid represented by k=1 and l=0 in the DD domain; M*N represents the total number of grids in the DD domain, and k and l represent the grid numbers in the DD domain respectively.

[0079] Indicates that the transmission rate Ri of user i has a minimum value R min .

[0080] VU={1,2,...,i,...U} represents the vehicle user set.

[0081] Medium Delta kl,i Indicates whether user i is assigned a channel in the k, i coordinate of the DD domain. A value of 0 indicates that no channel is assigned, otherwise it indicates that a signal is assigned.

[0082] represents the channels allocated to users in the entire DD domain, adding the constraint w max So that the number has the maximum value.

[0083] ν min ≤ν i，n ≤ν max The movement time is divided into n time slots, ν i，n It represents the speed of vehicle i in the nth time slot, which has maximum and minimum values.

[0084] Medium F i,n represents the beamforming matrix F of user i n ={F 1,n ,F 2,n ,...,F K,n},in p i,n is the transmission power allocated to user i in time slot n, Nt is the number of transmit antennas and a(·) is the array steering vector for transmit and receive.

[0085] Specifically expressed as

[0086] θ i,n is the angle between the moving direction of vehicle i in the nth time slot and the line-of-sight link, CRLB(θ i,n ,F i,n ) represents the minimum variance bound of the parameter estimation of user i in the nth time slot, which depends on its beamforming matrix F i,n .

[0087] Similarly, it also has a constraint The same as above, θ i,n Change to d i,n , that is, the distance between user i and the link terminal in the nth time slot.

[0088] represents the local computing energy consumption of the i-th user and transmission energy consumption The sum has a maximum value

[0089] f i,local ≤f i,max represents the computing power f allocated to local vehicle i i,local The maximum value f must not be exceeded i,max .

[0090] From the analysis of the problem, it can be seen that there are continuous variables in the optimization problem proposed by the present invention and binary variables in the channel allocation process. The optimization problem belongs to a mixed integer nonlinear programming problem and is difficult to solve using conventional methods.

[0091] Therefore, the present invention adopts a deep reinforcement learning method to solve the MINLP problem, that is, a Markov decision process is constructed based on the optimization objective and the corresponding constraints.

[0092] The unit that makes decisions in MDP is called an agent. In this invention, the agent is an RSU, and the part that interacts with the agent is called the environment. MDP is usually composed of a four-tuple (S, A, R, P). t ∈S is a finite set of states in the environment at time t, a t ∈A is the set of actions that the agent can take at time, r t is based on the agent in state s t Next action a t The reward obtained, the intelligent body will adjust the strategy of taking action next time according to the reward feedback. tis the state transition probability, indicating that from the current state s t Take action a t The probability of entering the next state. Usually p t The state transition probability is fixed in the environment, which is called the Markov characteristic. The reward and next state obtained by the RSU are only related to the current state of the RSU and the action selected at that time. Only in this way can the RSU learn the law of environmental changes.

[0093] The definitions of state space, action space, and reward function in the present invention are as follows:

[0094] (1) State space: The RSU can obtain the state information of the environment in each time interval, including the channel allocation information of the RSU to the mobile vehicle users, the channel gain of each transmission channel, the interference information and the vehicle information. The vehicle information includes its position, speed, maximum allocatable computing resources, maximum tolerable energy consumption and receiving power.

[0095] Action space: The RSU’s transmit power and computing resource allocation method.

[0096] (2)Define a t =(P(t),F(t)),a t ∈A. Transmitting power P = (p kl,i ,i∈VU,k∈[0,M-1],l∈[0,N-1]),p kl,i is a positive continuous variable and ∑p<P max . Vehicle computing resources F = (f i,local ,i∈VU),f i,local is a continuous variable and 0≤f i,local ≤f i,max . Reward function: Since the optimization goal of this problem is to minimize the delay, it can be seen that the instantaneous reward obtained by the RSU is negatively correlated with the average delay of multiple users completing the task. In addition, during the training process, if the strategy given by the agent does not meet the energy consumption limit and the perception accuracy limit, a corresponding penalty should be given. Define the reward function r of the agent t for:

[0097]

[0098] Among them, p1, p2, and p3 are the penalties for exceeding the energy consumption limit, angle perception accuracy, and distance perception accuracy, respectively.

[0099] When the decision made by the agent meets the constraints, it will be rewarded according to the total delay of the system. When the constraints are not met, it will be punished accordingly. The optimization goal of this chapter is to maximize the cumulative discounted reward of the long-term system.

[0100] The process of reinforcement learning is a Markov decision process. Given an MDP (S, A, R, P), the agent makes a decision based on the current state s. t , perform an action a according to a certain rule t , and enter the next state, then the agent will get the reward r for this behavior t The agent continuously explores, adjusts its behavior, and adapts to the environment, eventually forming a strategy to obtain the maximum long-term reward. In summary, reinforcement learning is to find an optimal strategy π * , so that the value of this behavior is maximized.

[0101]

[0102] Where Q is called the behavior value function Q π (s,a)=E π [G t |S t =s,A t =a], which represents the expected benefit obtained by taking action a in state s. The present invention obtains the allocation mode of the transmission power and computing resources of the roadside unit by finding the optimal strategy.

[0103] In order to solve the above Markov decision process, the present invention proposes a dynamically adaptive delayed update strategy and a priority experience selection method, which improves the experience selection rules and the frequency of delayed updates in the TD3 algorithm. It aims to flexibly adjust the update frequency according to the actual situation during the training process, and give priority to the use of high-value samples, so as to further optimize the performance of the TD3 algorithm and improve the efficiency and accuracy of solving the resource collaborative configuration problem.

[0104] The dynamically adaptive delayed update strategy should reduce the frequency of critic updates according to the changes in the target network. This can reduce the variance of Q-value updates, obtain more accurate Q-values, and accelerate convergence. The time difference signal TD-error reflects the range of changes in the critic network. The larger its value, the more the critic network has been changed, and the greater the deviation of the Q-value estimation. At this time, the update of the actor network needs to be delayed.

[0105] The present invention reduces the update frequency of the Critic network according to the changes of the target network through a dynamically adaptive delayed update strategy, so as to reduce the variance of the Q value update, obtain a more accurate Q value and accelerate convergence, and specifically includes the following steps:

[0106] 1) Monitor the timing differential signal:

[0107] During each update, the temporal difference (TD) signal of the Critic network is calculated, which represents the difference between the estimated Q value in the current state and the target Q value.

[0108] 2) Evaluate Q value deviation:

[0109] The degree of change of the Critic network is evaluated based on the size of the time difference signal.

[0110] When the absolute value of the time series difference signal is large, it means that the estimation deviation of the Critic network is large, and it is necessary to consider delaying the update of the Actor network.

[0111] 3) Dynamically adjust the critic update frequency:

[0112] Set a threshold. When the absolute value of the time series difference signal is greater than the threshold, reduce the update frequency of the Critic network.

[0113] The specific implementation can be: skipping the update of the Critic network when the Actor network is updated, or extending the update cycle of the Critic network.

[0114] 4) Delay Actor Update:

[0115] If updates to the Critic network are delayed, ensure that the Actor network is updated only during this period when the state estimate is relatively stable.

[0116] A parameter d can be introduced to represent the update period of the Actor network. The Actor is updated only when the deviation of the Critic network is small, that is, when the absolute value of the time series difference signal is less than the threshold.

[0117] 5) Feedback mechanism:

[0118] Regularly check the performance of the critic network to ensure that the update strategy is effective.

[0119] According to the convergence of the Critic network and the accuracy of the Q value, the threshold and Actor update period d are dynamically adjusted.

[0120] In terms of priority experience selection, due to the limited capacity of the experience pool, when there are many sample data, eliminating high-value data is not conducive to training. Therefore, it is necessary to operate the experience pool in an orderly and structured manner. The priority experience selection method uses a greedy strategy to select samples with large rewards, and the rewards are judged using TD-error. However, simply selecting samples with high rewards will lead to overfitting of TD-error and loss of sample diversity.

[0121] Therefore, the present invention uses a combination of random priority sampling, offset sampling and importance sampling. This ensures that the lowest priority sample has a certain probability of being selected, and uses high priority data samples as much as possible. The probability of each sample being selected is determined by TD-error. The probability of sample j being extracted is:

[0122]

[0123] where δ j is the TD-error, B is the number of samples in the experience pool, the noise ε is used to prevent the TD-error from being zero, resulting in the selection probability being zero, and α is the parameter that controls the sampling. When α=0, it is the same as random sampling.

[0124] Each sample has a priority parameter, and each time the sample is sampled, the sample is selected according to the priority, which undoubtedly increases the additional computational overhead and time overhead. You can consider using a time-saving tree structure to store this data.

[0125] For the priority parameter of each sample, we can consider using the structured binary tree storage of Sum-tree. The structural feature of this structure is that the weight of the parent node is the sum of the weights of the child nodes. When storing samples, they are stored in the Sum-tree according to the priority parameter P(j). When k samples need to be collected, the empirical samples are divided into k equally spaced intervals, that is, the root node is divided by k, a number is randomly generated for each interval, and the corresponding leaf node closest to the number is found in the structured Sum-tree. In this way, appropriate data can be selected within a small time cost, and it is ensured that samples with a small probability will not be selected at the same time.

[0126] Although the method of adding selection probability to samples improves the utilization rate of high-priority samples, high utilization rate will also lead to deviations in the update of neural network parameters.

[0127] In order to eliminate the bias, the update rule of importance sampling weight is used. Define the importance weight w of the sample j :

[0128]

[0129] Where K is the number of mini-batches sampled from the experience pool, and η∈(0,1) plays a role in bias correction, which gradually approaches 1 during training. Importance weight w j The role of is to use high-priority samples more frequently, but will not affect each parameter update too much. In order to ensure the stability of the updated parameters, the importance weight coefficient w j Normalized to the interval (0,1). The modified TD-error calculation rule is:

[0130]

[0131] So far, each time a new record is added to the experience playback pool, the new record will be given a higher priority to ensure that the record has a higher probability in the subsequent sampling process. When the experience pool capacity reaches the maximum limit, the record with the lowest priority will be discarded.

[0132] Using this algorithm, the proposed optimization problem can be effectively solved, thereby minimizing the total delay in task completion. The pseudo code is as follows:

[0133]

[0134]

[0135] In order to analyze the performance of the improved TD3 algorithm proposed in the present invention, this specific embodiment selects DDQN, MADDPG, MRL-DDPG and other intelligent driving edge computing resource collaborative configuration algorithms that also consider intelligent connected vehicles as comparison algorithms. The performance evaluation indicators selected by the present invention are as follows:

[0136] (1) Training reward: refers to the reward obtained by the agent during the training process of the deep reinforcement learning algorithm. It is determined according to the reward function formula and determines the convergence of the algorithm training process.

[0137] (2) Bit error rate (BER): The smaller the BER, the more stable the signal transmission and the higher the communication quality.

[0138] (3) Average delay: The average delay refers to the average delay of all vehicle users to complete the task. The smaller the average delay, the better the effect of the resource collaborative configuration system.

[0139] (4) Average energy consumption: Average energy consumption refers to the average value of energy consumption generated by all users of the vehicle when receiving signals and processing computing tasks locally.

[0140] (5) Maximum user delay: The maximum user delay reflects the time it takes for the user with the longest service response delay to complete a task among all users in the system. The average delay reflects the overall situation of the system, while the maximum user delay can reflect the worst situation in the system.

[0141] (6) Channel transmission rate: The channel transmission rate can measure the efficiency of information transmission within a system, and its unit is bit / s / Hz. The higher the channel transmission rate, the higher the information transmission efficiency and the better the performance.

[0142] (7) Cramer-Rao lower bound (CRLB): CRLB reflects the lower bound of the radar’s perception accuracy for a target. The smaller the CRLB, the higher the radar’s perception accuracy and the better the perception effect.

[0143] The experimental results show that from the perspective of the number of vehicles, when the number of vehicles is 10, the improved TD3 algorithm reduces the average delay by 38.2%, 24.2%, and 13.0% respectively compared with DQN, MADDPG, and MRL-DDPG, and reduces the average energy consumption by 27.3%, 19.7%, and 14.3% respectively. As the number of vehicles increases, the channel transmission rate is decreasing. From the perspective of the task volume, when the task volume is 0.8Mbit, the ITD3 algorithm reduces the delay by 13.2%, 8.1%, and 6.2% respectively compared with DDQN, MADDPG, and MRL-DDPG. When the task volume is 3.2Mbit, the ITD3 algorithm reduces the delay allocation by 19.3%, 12.3%, and 8.1% compared with DDQN, MADDPG, and MRL-DDPG. With the increase of the task volume, the ITD3 algorithm has a better performance in reducing the average delay of task completion. From the perspective of transmission power, when the transmission power is 20dBm, the ITD3 algorithm reduces the delay by 18.1%, 11.5%, and 8.3% compared with DDQN, MADDPG, and MRL-DDPG, respectively. When the transmission power is 30dBm, the ITD3 algorithm reduces the delay allocation by 19.2%, 12.4%, and 9.7% compared with DDQN, MADDPG, and MRL-DDPG. It can be seen that as the transmission power increases, the channel transmission rate will also increase.

[0144] In summary, the improved TD3 algorithm proposed in the present invention can accelerate the convergence speed, and in the integrated communication and perception environment, the proposed resource collaborative configuration scheme based on OTFS technology can better adapt to high-speed moving vehicle scenarios than the traditional OFDM-based scheme. It can be seen that compared from different angles, the improved TD3 algorithm proposed in the present invention is more advantageous than other algorithms in reducing the average delay and average energy consumption of task completion.

[0145] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles, characterized in that: The following steps are involved: In the scenario where the edge layer, roadside unit, and vehicle layer cooperate to realize intelligent driving, the roadside unit uses orthogonal time-frequency-space technology to transmit integrated perception and communication signals; With the optimization goal of minimizing the overall delay for all vehicles to receive signals and complete computing tasks, the Markov decision process is constructed with the transmission power, transmission rate, channel, vehicle moving speed, perception accuracy, energy consumption, and local computing resources as constraints. In the Markov decision process, The state space is for the roadside unit to obtain the state information in the environment and the channel gain of each transmission channel as well as the interference information and vehicle information in each time interval; The action space is the allocation of transmission power and computing resources of roadside units; The reward function includes the negative function of the average delay of multiple vehicles to complete the task, and the set penalty when the strategy given by the roadside unit as an intelligent agent does not meet the constraints during the training process; Solve the Markov decision process to obtain the optimal transmission power and computing resource allocation strategy for roadside units.

2. The method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles according to claim 1, characterized in that: The solution of the Markov decision process is achieved through an improved TD3 algorithm, wherein the update frequency of Critic and Actor is controlled to optimize the solution process of the TD3 algorithm, thereby obtaining the optimal transmission power and computing resource allocation strategy of the roadside unit.

3. The method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles according to claim 2, characterized in that: In the process of solving the Markov decision process, the update frequency of the Critic network is reduced only when the absolute value of the time difference signal of the Critic network is greater than the set value; each time the Actor is updated, the Actor is updated only when the absolute value of the time difference signal of the Critic network is less than or equal to the set value; otherwise, the update of the Actor is skipped.

4. The method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles according to claim 2, characterized in that: In the process of solving the Markov decision process, when updating the experience pool, the selection probability of the sample is calculated according to its priority; Consider using Sum-tree structured binary tree storage. The structural feature of this structure is that the weight of the parent node is the sum of the weights of the child nodes. When storing samples, they are stored in the Sum-tree according to the priority parameters. When k samples need to be collected, the experience pool needs to be divided into k equidistant intervals, and the weight of each interval is 1\k of the root node weight. A random number is generated in each interval, and the leaf node closest to the random number in the Sum-tree is searched to obtain the corresponding sample. Each sample is assigned an importance sampling weight and normalized. Each time a new record is added to the experience pool, a higher initial priority is given to ensure that the record is selected first in the subsequent sampling process. When the experience pool reaches the upper limit of capacity, the sample with the lowest priority is removed.

5. The method for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles according to claim 4, characterized in that: The following formula is used to calculate the probability of selecting the jth sample P(j): where δ j is the temporal difference error of the jth sample, B is the number of samples in the experience pool, the noise ε is used to prevent the temporal difference error from being zero, resulting in the probability of selection being zero, α is the parameter controlling the sampling, when α = 0, it is the same as random sampling; P(i) is the priority of the i-th sample; P(j) is the priority of the j-th sample; represents the priority of the jth sample controlled by parameter α; y m Indicates that according to the empirical value parameter θ of the Q function i The calculated current Q value of the mth sample in the experience pool; represents the usage parameter θ of the mth sample in state s and action a at the current time slot t i The return value predicted by the Q network; w j is the importance weight, K is the number of Mini-batches sampled from the experience pool, Indicates the calculation of the square error between the m-th predicted return and the current return; δ i＝1,2 Represents two arbitrarily selected samples, from which an optimal timing difference error is selected.

6. The method for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles according to claim 5, characterized in that: The importance weight w is defined as follows: j : Among them, K is the number of mini-batches sampled by the experience pool, and η∈(0,1) plays the role of bias correction and gradually tends to 1 during the training process.

7. The method for collaborative configuration of intelligent driving edge computing resources for intelligent connected vehicles according to claim 1, characterized in that: The constraints include: The power allocated to the transmitter of the roadside unit shall not exceed its maximum power; The transmission rate of a single vehicle shall not be less than its minimum transmission rate; The number of DD grids allocated to a single vehicle shall not exceed the maximum tolerance; The vehicle's movement speed is kept within a certain range; The Cramer-Rao lower bounds of angles and distances must be smaller than their minimum tolerable perceptual accuracy; The energy consumption generated locally by the vehicle must not exceed its maximum tolerable limit; The computing resources allocated to the vehicle must not exceed the maximum computing capacity of the autonomous vehicle.

8. A DNN-based model partitioning system for vehicle-mounted edge intelligence, characterized by: Used to implement the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for collaboratively configuring intelligent driving edge computing resources for intelligent connected vehicles as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the intelligent driving edge computing resource collaborative configuration method for intelligent connected vehicles as described in any one of claims 1 to 7 by executing the computer instructions.

Images