RIS-assisted resource allocation optimization method for vehicle-to-everything communication based on SAC-MHSA algorithm

Abstract

This invention belongs to the field of vehicle-to-everything (V2X) communication and mobile edge computing technology, and relates to a RIS-assisted V2X communication resource allocation optimization method based on the SAC-MHSA algorithm. The method includes: establishing a V2X-MEC collaborative optimization model; initializing scene parameters and constructing a high-speed experience replay pool, using a low-level tensor matrix pre-allocation mechanism to store interaction experience; extracting global collaborative features of each agent; generating joint discrete actions using an Actor network and interacting with the environment to generate immediate rewards; executing a global target action pre-computation mechanism during the training and update phase, and inferring action tensors in parallel; applying a Critic network based on a cross-attention mechanism to evaluate policy value, and updating network parameters using smoothed L1 loss; updating the Actor network and simultaneously performing adaptive target entropy adjustment to optimize the temperature coefficient; and determining whether the maximum number of iterations has been reached. This method aims to minimize the average latency and average energy consumption of system task processing, thereby improving the overall collaborative performance and algorithm training efficiency of the V2X system.

Description

Technical Field

[0001] This invention belongs to the field of vehicle-to-everything (V2X) communication and mobile edge computing technology, specifically involving an improved multi-agent discrete flexible action evaluation (Discrete MASAC) algorithm based on multi-head self-attention mechanism (MHSA), which is used for collaborative offloading decision of computing tasks and joint allocation optimization of communication resources in complex V2X environments. Background Technology

[0002] With the rapid development of wireless communication technology, autonomous driving, and intelligent transportation systems, vehicle-to-everything (V2X) has gradually become a key cornerstone of future intelligent transportation. In actual V2X operation scenarios, vehicles need to process massive amounts of latency-sensitive and computationally intensive tasks in real time while driving. Traditional centralized cloud server architectures, due to long data transmission distances and high link losses, are unable to meet the stringent requirements of V2X for ultra-low latency and high reliability. Therefore, multi-access edge computing (MEC) technology has emerged. MEC technology deploys computing resources near roadside units (RSUs) close to the vehicle, allowing the vehicle to offload high-load tasks to edge nodes, thereby effectively reducing processing latency, alleviating local computing power bottlenecks, and reducing the energy consumption of vehicle equipment.

[0003] However, in complex road conditions with high concurrency and multiple vehicles, the computing resources of MEC devices and the communication channel resources of RSUs are extremely limited. When a large number of vehicles simultaneously request task offloading, it can easily lead to severe channel resource contention and co-channel signal interference. The allocation of vehicle-to-everything (V2X) resources is essentially a highly coupled, multi-dimensional optimization process. It involves not only the selection of offloading modes in the computing dimension, but also the partitioning of channel resource blocks (RBs) in the communication dimension, as well as the phase scheduling configuration of auxiliary communication links introduced to avoid physical obstruction and fading. Traditional static rule allocation or heuristic algorithms often struggle to adapt quickly to this dynamically changing high-dimensional mixed state space, leading to improper joint allocation, increased communication congestion, and consequently, a surge in overall system task processing latency, excessive power consumption, and severe resource waste.

[0004] Currently, Multi-Agent Reinforcement Learning (MARL) has shown some potential in solving the problem of coordinated scheduling in vehicle-to-everything (V2X) networks. However, existing single-agent or traditional multi-agent reinforcement learning algorithms still face significant challenges in practical applications: First, in terms of feature perception, traditional network structures are weak in extracting global co-frequency interference features caused by multiple concurrent vehicles, making it difficult for agents to accurately perceive the coupling effects of neighboring vehicle actions, and they are prone to getting trapped in local optima; Second, in terms of algorithm training and execution efficiency, traditional multi-agent algorithms suffer from severe nested inference curse when evaluating global value, resulting in excessively long model training latency, huge underlying memory read and write overhead, and network exploration often collapses when dealing with high-dimensional discrete action spaces due to the forced high target entropy.

[0005] Therefore, in complex coupled scenarios involving multiple vehicles and multiple tasks, how to overcome the feature perception bottleneck and high-dimensional inference latency of traditional algorithms, and design a joint allocation method for vehicle-to-everything (V2X) communication and computing resources with a high-speed optimization mechanism and global collaborative capabilities to minimize the average latency and average energy consumption of system task processing is a technical challenge that urgently needs to be solved in the current field of vehicle-to-everything (V2X) MEC. Summary of the Invention

[0006] The purpose of this invention is to provide a method for optimizing the allocation of vehicle network communication resources based on discrete SAC-MHSA and RIS-assisted algorithm, so as to solve the problems of communication resource competition and computation offloading coordination in multi-vehicle concurrent scenarios of vehicle network.

[0007] This invention is achieved through the following technical solution:

[0008] A RIS-assisted vehicular network communication resource allocation optimization method based on the SAC-MHSA algorithm includes the following steps:

[0009] Step 1: Establish a V2X-MEC collaborative optimization model using a multi-agent reinforcement learning framework, including an environmental simulation model and a multi-dimensional computation and communication model;

[0010] Step 2: Initialize scene parameters based on Step 1 and build a high-speed experience replay pool, using a low-level tensor matrix pre-allocation mechanism to store interactive experiences;

[0011] Step 3: Extract global collaborative features of each agent through a state token encoder and a multi-head self-attention mechanism (MHSA);

[0012] Step 4: Utilize the Actor network to generate joint discrete actions that include computation patterns, communication resource blocks, and auxiliary link configurations, and interact with the environment to generate instant rewards.

[0013] Step 5: Execute a global target action pre-computation mechanism during the training and update phase to infer action tensors in parallel and eliminate computational bottlenecks.

[0014] Step 6: Apply the Critic network based on the cross-attention mechanism to evaluate the value of the strategy, and update the network parameters using smooth L1 loss.

[0015] Step 7: Update the Actor network based on the policy gradient and simultaneously perform adaptive target entropy adjustment to optimize the temperature coefficient;

[0016] Step 8: Determine if the number of iterations has reached the maximum number of rounds. If the iteration is complete, end the training and output the optimal collaborative resource allocation strategy model.

[0017] Furthermore, in step 1, the environmental simulation model is a road of a set length, with RSUs (Roadside Units) deployed along the roadside. Each RSU is connected to a MEC (Mobile Edge Computing) server via a wired connection. The total computing resources of the MEC server are... ; on the road The set of intelligent vehicles equipped with local computing power devices is denoted as V={1,2, ...,N}. The vehicles communicate with the RSU via a wireless network and can choose to execute tasks locally or offload them to the MEC server for computation. In the scenario, all vehicles simultaneously generate a task to be processed in each time slot. The generated task characteristics are set as follows ,in, For task data size, The number of CPU cycles required per unit of task data. The maximum acceptable latency tolerance for completing the task. To maximize energy consumption tolerance, auxiliary communication surfaces are also deployed within the scenario to optimize signal quality in non-line-of-sight environments.

[0018] Furthermore, the communication model is as follows: when the vehicle chooses to communicate with the RSU for offloading, the total system communication bandwidth is B, which is divided into M resource blocks in the frequency domain; when the vehicle transmits mission data, it is affected by its own transmit power. The dynamic change in the signal-to-interference-plus-noise ratio (SNR) at the receiver is denoted as the effect of joint channel fading and co-channel interference power generated by sharing the same resource block. .

[0019] Furthermore, decision variables include unloading decision variables. Resource block allocation decision variables and auxiliary link configuration decision variables ;in This indicates the task calculation mode for vehicle i, taking the value 0 or 1. When the task is calculated locally on the vehicle, This indicates that the task is unloaded to the MEC server for computation; This indicates the index of the resource block occupied when vehicle i is unloaded; This represents the auxiliary communication phase scheduling configuration index assigned to vehicle i by the system;

[0020] When the vehicle selects local calculation ( The formulas for processing delay and energy consumption at that time are:

[0021] ;

[0022] ;

[0023] in, The number of CPU cycles required per unit of task data. Where k is the local CPU frequency of the vehicle, and k is the local calculation energy consumption coefficient. When the vehicle selects edge unloading ( The formulas for processing delay and energy consumption at that time are:

[0024] ;

[0025] ;

[0026] in, Total computing resources for MEC servers The actual transmission rate affected by interference from multiple vehicles. Uplink transmission power for vehicles;

[0027] A fitness function is established with the objective of minimizing system overhead, and its mathematical expression is as follows:

[0028] ;

[0029] in, and Assigning weights to latency and energy consumption, Timeout penalties are set for timed-out tasks and co-frequency interference of resource blocks;

[0030] Constraints:

[0031] ;

[0032] ;

[0033] ;

[0034] .

[0035] Among them, This indicates the task calculation mode for vehicle i, and can be either 0 or 1. This indicates the index of the resource block occupied when vehicle i is unloaded; This indicates the minimum permissible value for the vehicle's local CPU frequency. This indicates the maximum allowed value for the vehicle's local CPU frequency.

[0036] Furthermore, in step 2, the specific process of initializing scene parameters and constructing the rapid experience replay pool is as follows:

[0037] Based on the vehicle-to-everything (V2X) MEC system model constructed in step 1, the simulation scenario parameters are first initialized, setting the number of vehicle agents to N, the number of communication resource blocks to M, and the total number of auxiliary reflector phases to L; subsequently, a high-speed experience playback pool is constructed, with a maximum storage capacity of C and a batch processing sampling size of [missing information]. A low-level tensor matrix pre-allocation mechanism is used in memory to establish contiguous memory blocks for each interaction dimension, with the initial state shown in the following equation:

[0038] ;

[0039] ;

[0040] Where E represents a set of matrices including the state observation pool S, the joint action pool A, the immediate reward pool R, the next state pool S', and the termination flag pool D, and d is the feature depth of each dimension.

[0041] In the experience storage phase, the experience tuple generated in a single interaction is defined as follows: As shown in the following formula:

[0042] ;

[0043] Establish storage update logic based on cursor pointer ptr and real-time scale size, when new interactive experiences are obtained. At this time, data overwriting is performed directly by locating the matrix index through the pointer; the update formula for the pointer and pool size is as follows:

[0044] ;

[0045] ;

[0046] This mechanism operates through pre-allocated contiguous memory blocks in the NumPy underlying layer, ensuring that the data writing process has a constant time complexity;

[0047] During the algorithm update phase, a bottom-level slice sampling method based on a random uniform distribution is used to first generate a sample containing... A set I of random indices, The batch size for a single slice sampling in the high-speed experience playback pool is calculated using the following formula:

[0048] ;

[0049] Subsequently, the pre-allocated contiguous memory matrix is ​​sliced ​​directly according to the index set I to obtain the training samples for the current batch. :

[0050] ;

[0051] The operation of the high-speed experience playback pool is subject to physical constraints related to capacity overflow and sampling stability, as shown in the following formula:

[0052] ;

[0053] ;

[0054] Among them, constraint C5 ensures that memory addressing does not go out of bounds and uses modular arithmetic to implement circular storage to isolate the negative impact of stale data; constraint C6 specifies the sampling trigger threshold, that is, only when the total amount of experience accumulated in the playback pool reaches the set batch size is the subsequent network update process allowed to be executed.

[0055] Furthermore, the state token encoder and multi-head self-attention mechanism in step 3 aim to extract global cooperative interference features in a multi-vehicle concurrent environment. They transform the local observation states of each vehicle agent into a high-dimensional feature token sequence, calculate the correlation weights between different feature tokens, and finally fuse them to output a cooperative feature vector with a global perspective. The mapping formula for the state tokens is:

[0056] ;

[0057] in, The representation layer normalization operation is used to normalize the feature vector, where s is the normalized state observation vector. For scalar mapping functions, The feature embedding matrix is ​​used; the core formula for calculating the association weights in the multi-head self-attention mechanism is:

[0058] ;

[0059] in, For querying the matrix, The key matrix, For value matrices, The dimension scaling factor for the attention head. , , All are learnable linear mapping weight matrices.

[0060] Furthermore, in step 4, the Actor network takes the extracted collaborative features as input and generates discrete probability distributions for the corresponding resource dimensions through three independent fully connected prediction heads. The formula for calculating the joint discrete action probability distribution is as follows:

[0061] ;

[0062] in, To generate the computational pattern decision probability for whether to perform edge unloading, Assign decision probabilities to generate communication resource blocks that occupy specific channels. The auxiliary link is configured with decision probabilities to generate a specific phase; the formula for calculating the joint discrete action log probability of the Actor network is:

[0063] ;

[0064] in, The minimum value of the stabilizing washer term, representing the prevention of gradient collapse in the discrete action space during logarithmic operations, is taken as [value missing]. The agent performs classification and sampling based on the calculated joint action probability distribution to obtain the final joint discrete action and sends it to the vehicle network environment for execution. After receiving the action, the environment updates its state for the next moment and outputs an instant reward value to evaluate the merits of the action.

[0065] Furthermore, in step 5, the formula for inferring the action tensor using the global target action pre-computation mechanism is:

[0066] ;

[0067] in, For the j-th agent, there is a gradient-free objective Actor policy network. Let j be the state observation value of the j-th agent at the next time step. This is the set of tensors for the next target actions of all agents in parallel inference; the pre-computation mechanism extracts the target action sequences of all vehicle agents in batches at once outside the traversal loop of the Critic value network update.

[0068] Furthermore, in step 6, the Critic network based on the cross-attention mechanism evaluates the value of the current policy action by aggregating global state features and the action information of neighboring agents, as follows:

[0069] First, the Critic network receives the global collaborative feature vector extracted in step 3 and the joint discrete actions performed by each agent. After performing high-dimensional embedding mapping on the action sequence, it uses a cross-attention mechanism to fuse the action features of neighboring agents. The current agent's state features serve as the query matrix. The action embedding features of neighboring agents are used as the key matrix. Sum matrix ; Critic network output comprehensive action value assessment value The calculation formula is:

[0070] ;

[0071] in, This is the cross-attention operation function. This is the mapping function for the fully connected multilayer perceptron in the network backend; This represents two independent Critic evaluation network branches set up in parallel to alleviate the Q-value overestimation problem in traditional reinforcement learning;

[0072] Subsequently, the soft target value calculated in parallel by the target network in step 5 is obtained. This is used as the true label for the current Critic network update; the time difference error is calculated using a smoothed L1 loss function. The formula for the Critic network's loss function is:

[0073] ;

[0074] in, This refers to the batch size of a single slice sample in the high-speed experience replay pool. and This is a sample of states and actions in the current batch; The piecewise function for smoothing error is specifically defined mathematically as follows:

[0075] ;

[0076] in, The absolute value of the error;

[0077] The system achieves extremely fast and stable updates to the weights of the multi-agent Critic value network by calculating the smooth L1 loss function and using the optimizer to perform gradient backpropagation.

[0078] Furthermore, in step 7, the specific process of updating the Actor network based on the policy gradient and simultaneously performing adaptive target entropy adjustment is as follows:

[0079] First, the system observes the state samples of the current batch. Recalculate the probability distribution of joint discrete actions using the Actor network. Combined with the action value assessment benchmarks given by the two Critic evaluation networks in step 6. The optimization objective of the Actor network is to maximize the weighted sum of the expected value of the action and the policy entropy; the corresponding policy gradient loss function is calculated as follows:

[0080] ;

[0081] in, This refers to the batch size of a single slice sampled in the high-speed experience playback pool. To balance the temperature coefficient between exploration and utilization, the system updates the weight parameters of the Actor network by calculating the loss function and performing gradient backpropagation.

[0082] Subsequently, to address the allocation characteristics of discrete resources in the Internet of Vehicles (IoV), a dynamic target entropy biased towards utilization was set. The calculation formula is shown below:

[0083] ;

[0084] in, The total dimension of the discrete action space of the vehicle-to-everything (V2X) system;

[0085] Based on the set dynamic target entropy For learnable temperature coefficient Perform synchronous gradient optimization; the Alpha loss function for optimizing the temperature coefficient is calculated as follows:

[0086] ;

[0087] When the true information entropy of the current strategy is lower than the target entropy, the loss function drives the temperature coefficient. Increase the size of the policy space to encourage agents to explore unknown policy spaces; conversely, when policies become too divergent, the driving force... Reduce to accelerate network convergence;

[0088] Each iteration calculates a new action probability distribution based on the policy network, and updates the Actor network parameters and temperature coefficient simultaneously; it checks whether the number of iterations has reached the maximum number of rounds. If not, it returns to step 3 to continue updating; after reaching the maximum number of iterations, it outputs the optimal Actor policy network weights, which is the optimal collaborative resource allocation strategy model.

[0089] Compared with the prior art, the beneficial effects of the present invention are:

[0090] This invention proposes an optimization method for communication resource allocation in vehicular network systems based on a multi-head self-attention mechanism. This method, through a joint architecture of local computation, MEC edge offloading, and auxiliary communication links, combined with the advantages of an improved multi-agent discrete SAC algorithm, can optimize the joint allocation of computing power and communication channel resources in complex scenarios with concurrent interference from multiple vehicles. By introducing a global collaborative feature extraction mechanism of multi-head self-attention (MHSA), a low-level continuous memory experience replay pool, and a global target action pre-computation mechanism, the method effectively improves the system's resource utilization efficiency against co-channel interference, significantly reduces the average processing latency of vehicle task data and the total system energy consumption, and increases system rewards. Furthermore, comparative verification with mainstream benchmark algorithms such as multi-agent D3QN and A2C, as well as ablation experiments within the system, demonstrate that the improved algorithm proposed in this invention not only has significant advantages in convergence speed and global optimization capability, but also fully verifies the necessity of new components such as multi-head self-attention collaborative feature extraction and dynamic target entropy for improving system robustness, greatly enhancing the collaborative scheduling performance of the entire vehicular network edge computing system. Attached Figure Description

[0091] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 A detailed flowchart illustrating the method of this invention;

[0093] Figure 2 This is a model diagram of a vehicle-to-everything (V2X) communication system.

[0094] Figure 3 This is a diagram of the Actor-Critic network architecture based on a multi-head self-attention mechanism.

[0095] Figure 4 The average task reward convergence curve during the training process of this method is shown in the figure.

[0096] Figure 5 This is a graph showing the average task latency during the training process of this method.

[0097] Figure 6 This is a graph showing the average energy consumption of the task during the training process of this method.

[0098] Figure 7 An annealing curve is used to explore the loss function and policy of the Critic network during the training process of this method.

[0099] Figure 8 This is a graph comparing the average task reward of ablation experiments using different variant modules of this method.

[0100] Figure 9 This is a comparison curve of the average ablation time under different variant modules of this method;

[0101] Figure 10 This is a comparison curve of the average energy consumption of ablation experiments using different variant modules of this method;

[0102] Figure 11 A graph showing the average task reward for collaborative unloading of this method compared to other mainstream benchmark algorithms;

[0103] Figure 12 This is a comparison curve of the average delay of cooperative unloading with other mainstream benchmark algorithms;

[0104] Figure 13 This is a curve comparing the average energy consumption of our method with other mainstream benchmark algorithms for cooperative unloading. Detailed Implementation

[0105] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0106] This invention provides an improved multi-agent discrete flexible action evaluation (Discrete MASAC) algorithm based on multi-head self-attention (MHSA) for discrete action spaces. By constructing an intelligent algorithm with a high-speed optimization mechanism and global coordination capability, it overcomes the bottleneck of feature perception and high-dimensional action inference latency in complex environments, and realizes the scientific allocation of communication and computing resources in vehicle-to-everything (V2X) systems. It aims to solve the problems of communication resource competition and computation offloading coordination in multi-vehicle concurrent scenarios in V2X systems, so as to minimize the average latency and average energy consumption of system task processing, and improve the overall coordination performance and algorithm training efficiency of V2X systems.

[0107] This invention relates to a RIS-assisted vehicular network communication resource allocation optimization method based on the SAC-MHSA algorithm, comprising the following steps:

[0108] Step 1: Establish a V2X-MEC collaborative optimization model using a multi-agent reinforcement learning framework, including an environmental simulation model and a multi-dimensional computation and communication model;

[0109] Step 2: Initialize scene parameters based on Step 1 and build a high-speed experience replay pool, using a low-level tensor matrix pre-allocation mechanism to store interactive experiences;

[0110] Step 3: Extract global collaborative features of each agent through a state token encoder and a multi-head self-attention mechanism (MHSA);

[0111] Step 4: Utilize the Actor network to generate joint discrete actions that include computation patterns, communication resource blocks, and auxiliary link configurations, and interact with the environment to generate instant rewards.

[0112] Step 5: Execute a global target action pre-computation mechanism during the training and update phase to infer action tensors in parallel and eliminate computational bottlenecks.

[0113] Step 6: Apply the Critic network based on the cross-attention mechanism to evaluate the value of the strategy, and update the network parameters using smooth L1 loss.

[0114] Step 7: Update the Actor network based on the policy gradient and simultaneously perform adaptive target entropy adjustment to optimize the temperature coefficient;

[0115] Step 8: Determine if the number of iterations has reached the maximum number of rounds. If the iteration is complete, end the training and output the optimal collaborative resource allocation strategy model.

[0116] Furthermore, in step 1, the environmental simulation model is a road of a set length, with RSUs (Roadside Units) deployed along the roadside. Each RSU is connected to a MEC (Mobile Edge Computing) server via a wired connection. The total computing resources of the MEC server are... ; on the road The set of intelligent vehicles equipped with local computing power is denoted as V={1,2, ...,N}. These vehicles communicate with the RSU (Resource Unit) via a wireless network and can choose to execute tasks locally or offload them to the MEC (Multi-access Edge Computing) server for computation. In this scenario, all vehicles simultaneously generate a task to be processed in each time slot. The generated task characteristics are set as follows ,in For task data size, The number of CPU cycles required per unit of task data. The maximum acceptable latency tolerance for completing the task. To maximize energy consumption tolerance, auxiliary communication surfaces are also deployed within the scenario to optimize signal quality in non-line-of-sight environments.

[0117] Communication Model: When the vehicle chooses to communicate with the RSU for offloading, the total system communication bandwidth is B, which is divided into M resource blocks in the frequency domain; when the vehicle transmits mission data, it is affected by its own transmit power. The dynamic change in the signal-to-interference-plus-noise ratio (SNR) at the receiver is denoted as the effect of joint channel fading and co-channel interference power generated by sharing the same resource block. ;

[0118] Decision variables include unloading decision variables. Resource block allocation decision variables and auxiliary link configuration decision variables ;in This indicates the task calculation mode for vehicle i, taking the value 0 or 1. When the task is calculated locally on the vehicle, This indicates that the task is unloaded to the MEC server for computation; This indicates the index of the resource block occupied when vehicle i is unloaded; This represents the auxiliary communication phase scheduling configuration index assigned to vehicle i by the system;

[0119] When the vehicle selects local calculation ( The formulas for processing delay and energy consumption at that time are:

[0120] ;

[0121] ;

[0122] in, The number of CPU cycles required per unit of task data. Where is the local CPU frequency of the vehicle, and k is the local calculation energy consumption coefficient;

[0123] When the vehicle selects edge unloading ( The formulas for processing delay and energy consumption at that time are:

[0124] ;

[0125] ;

[0126] in, Total computing resources for MEC servers The actual transmission rate affected by interference from multiple vehicles. Uplink transmission power for vehicles;

[0127] Considering the system's tolerance limits for latency and energy consumption, this invention establishes a fitness function with the objective of minimizing system overhead (maximizing joint reward), the mathematical expression of which is as follows:

[0128] ;

[0129] in, and The weights for latency and energy consumption. Timeout penalties are set for timed-out tasks and resource block interference.

[0130] Constraints:

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] Among them, This indicates the task calculation mode for vehicle i, and can be either 0 or 1. This indicates the index of the resource block occupied when vehicle i is unloaded; This indicates the minimum permissible value for the vehicle's local CPU frequency. This indicates the maximum allowed value for the vehicle's local CPU frequency.

[0136] In step 2, the specific process of initializing scene parameters and constructing the rapid experience replay pool is as follows:

[0137] Based on the vehicle-to-everything (V2X) MEC system model constructed in step 1, the simulation scenario parameters are first initialized. The number of vehicle agents is set to N, the number of communication resource blocks to M, and the total number of auxiliary reflector phases to L. Subsequently, a high-speed experience playback pool is constructed, with a maximum storage capacity of C and a batch sampling size of [value missing]. A low-level tensor matrix pre-allocation mechanism is used in memory to establish contiguous memory blocks for each interaction dimension, with the initial state shown in the following equation:

[0138] ;

[0139] ;

[0140] Where E represents a set of matrices including the state observation pool S, the joint action pool A, the immediate reward pool R, the next state pool S', and the termination flag pool D, and d is the feature depth of each dimension.

[0141] In the experience storage phase, the experience tuple generated in a single interaction is defined as follows: As shown in the following formula:

[0142] ;

[0143] The system establishes storage update logic based on the cursor pointer ptr and the real-time scale size. When acquiring new interactive experiences... At this time, data overwriting is performed directly by locating the matrix index through the pointer. The update formula for the pointer and pool size is shown below:

[0144] ;

[0145] ;

[0146] This mechanism operates through pre-allocated contiguous memory blocks in the NumPy underlying layer, ensuring that the data writing process has a constant time complexity.

[0147] During the algorithm update phase, a low-level slice sampling technique based on random uniform distribution is employed. The system first generates a slice containing... A set I of random indices, The batch size for a single slice sampling in the high-speed experience playback pool is calculated using the following formula:

[0148] ;

[0149] Subsequently, the pre-allocated contiguous memory matrix is ​​sliced ​​directly according to the index set I to obtain the training samples for the current batch. :

[0150] ;

[0151] The operation of the high-speed experience playback pool is subject to physical constraints related to capacity overflow and sampling stability, as shown in the following formula:

[0152] ;

[0153] ;

[0154] Among them, constraint C5 ensures that memory addressing does not go out of bounds and uses modular arithmetic to implement circular storage to isolate the negative impact of stale data; constraint C6 specifies the sampling trigger threshold, that is, only when the total amount of experience accumulated in the playback pool reaches the set batch size, the system allows the subsequent network update process to be executed, so as to ensure the stability of model convergence.

[0155] Furthermore, the state token encoder and multi-head self-attention mechanism in step 3 aim to extract global cooperative interference features in a multi-vehicle concurrent environment. They transform the local observation states of each vehicle agent into a high-dimensional feature token sequence, calculate the correlation weights between different feature tokens, and finally fuse them to output a cooperative feature vector with a global perspective. The mapping formula for the state tokens is:

[0156] ;

[0157] in, The representation layer normalization operation is used to normalize the feature vector, where s is the normalized state observation vector. For scalar mapping functions, The feature embedding matrix is ​​used; the core formula for calculating the association weights in the multi-head self-attention mechanism is:

[0158] ;

[0159] in, For querying the matrix, The key matrix, For value matrices, The dimension scaling factor for the attention head. , , All are learnable linear mapping weight matrices.

[0160] Furthermore, in step 4, the Actor network takes the extracted collaborative features as input and generates discrete probability distributions for the corresponding resource dimensions through three independent fully connected prediction heads. The formula for calculating the joint discrete action probability distribution is as follows:

[0161] ;

[0162] in, To generate the computational pattern decision probability for whether to perform edge unloading, Assign decision probabilities to generate communication resource blocks that occupy specific channels. The auxiliary link is configured with decision probabilities to generate a specific phase; the formula for calculating the joint discrete action log probability of the Actor network is:

[0163] ;

[0164] in, The minimum value of the stabilizing washer term (with a value of ) represents the value of the term that prevents gradient collapse in the discrete action space during logarithmic operations. The agent performs classification and sampling based on the calculated joint action probability distribution to obtain the final joint discrete action and sends it to the vehicle network environment for execution. After receiving the action, the environment updates its state at the next moment and outputs an instant reward value to evaluate the merits of the action.

[0165] Furthermore, in step 5, the formula for inferring the action tensor using the global target action pre-computation mechanism is:

[0166] ;

[0167] in, For the j-th agent, there is a gradient-free objective Actor policy network. Let j be the state observation value of the j-th agent at the next time step. The set of tensors representing the next target actions of all agents in parallel inference; the pre-computation mechanism extracts the target action sequences of all vehicle agents in batches at once outside the traversal loop of the Critic value network update, avoiding redundant inference of the neighbor agent network when each agent updates, thereby eliminating the computational bottleneck generated in traditional multi-agent cross-evaluation.

[0168] Furthermore, in step 6, the Critic network based on the cross-attention mechanism evaluates the value of the current policy action by aggregating global state features and the action information of neighboring agents, as follows:

[0169] First, the Critic network receives the global collaborative feature vector extracted in step 3 and the joint discrete actions performed by each agent. To accurately isolate action interference in a multi-vehicle environment, the action sequences are mapped using high-dimensional embedding, and then a cross-attention mechanism is used to fuse the action features of neighboring agents. The current agent's state features serve as the query matrix. The action embedding features of neighboring agents are used as the key matrix. Sum matrix The Critic network outputs a comprehensive action value assessment. The calculation formula is:

[0170] ;

[0171] in, This is the cross-attention operation function. This is the mapping function for the fully connected multilayer perceptron in the network backend; This represents two independent Critic evaluation network branches set up in parallel to alleviate the Q-value overestimation problem in traditional reinforcement learning;

[0172] Subsequently, the soft target value calculated in parallel by the target network in step 5 is obtained. This is used as the true label for the current Critic network update. Since vehicular network tasks may generate abnormally large latency penalties under extreme congestion, a smoothed L1 loss function is used to calculate the time difference error to prevent network gradient explosion caused by excessive absolute error. The formula for calculating the loss function of the Critic network is:

[0173] ;

[0174] in, This refers to the batch size of a single slice sample in the high-speed experience replay pool. and This is a sample of states and actions in the current batch; The piecewise function for smoothing error is specifically defined mathematically as follows:

[0175] ;

[0176] in, This represents the absolute value of the error.

[0177] The system achieves extremely fast and stable updates to the weights of the multi-agent Critic value network by calculating the smooth L1 loss function and using the optimizer to perform gradient backpropagation.

[0178] Furthermore, in step 7, the specific process of updating the Actor network based on the policy gradient and simultaneously performing adaptive target entropy adjustment is as follows:

[0179] First, in order to update the Actor policy network, the system uses the state observation samples of the current batch. Recalculate the probability distribution of joint discrete actions using the Actor network. Combining the action value assessment benchmarks given by the two Critic evaluation networks in step 6. The optimization objective of the Actor network is to maximize the weighted sum of the expected value of an action and the policy entropy. The corresponding policy gradient loss function is calculated as follows:

[0180] ;

[0181] in, This refers to the batch size of a single slice sampled in the high-speed experience playback pool. To balance the temperature coefficient between exploration and utilization, the system updates the weight parameters of the Actor network by calculating the loss function and performing gradient backpropagation.

[0182] Subsequently, to prevent multi-vehicle intelligent agents from falling into purely random and blind divergent exploration in a high-dimensional discrete action space due to forced high target entropy, this invention introduces an adaptive dynamic target entropy adjustment mechanism. Based on the allocation characteristics of discrete resources in the vehicle network, a dynamic target entropy biased towards utilization is set. The calculation formula is shown below:

[0183] ;

[0184] in, The total dimension of the discrete action space of the vehicle-to-everything (V2X) system;

[0185] Based on the set dynamic target entropy The system has a learnable temperature coefficient. Perform synchronous gradient optimization. The formula for calculating the alpha loss function for optimizing the temperature coefficient is shown below:

[0186] ;

[0187] By optimizing the above formula, the system constructs a negative feedback adjustment closed loop for exploration and utilization: when the true information entropy of the current strategy is lower than the target entropy, the loss function drives the temperature coefficient. Increase the size of the policy space to encourage agents to explore unknown policy spaces; conversely, when policies become too divergent, the driving force... Reduce to accelerate network convergence;

[0188] Each iteration calculates a new action probability distribution based on the policy network, and updates the Actor network parameters and temperature coefficient simultaneously to balance the algorithm's exploration and utilization capabilities, avoid getting trapped in local optima, and improve the algorithm's convergence. It checks whether the number of iterations has reached the maximum number of rounds. If not, it returns to step 3 to continue updating. After reaching the maximum number of iterations, it outputs the optimal Actor policy network weights, which is the optimal collaborative resource allocation strategy model.

[0189] Example 1

[0190] A RIS-assisted vehicular network communication resource allocation optimization method based on the SAC-MHSA algorithm includes the following steps:

[0191] Step 1: Establish a V2X-MEC collaborative optimization model using a multi-agent reinforcement learning framework, including an environment simulation model and a multi-dimensional computation and communication model. A schematic diagram of the scenario model is shown below. Figure 2 As shown;

[0192] The environmental simulation model is a city road of a set length, with Roadside Units (RSUs) deployed at certain intervals along the roadside. Each RSU is individually connected to a Mobile Edge Computing (MEC) server via a wired connection. The total computing resources of a single MEC server are... Let V = {1, 2, ..., N} be the set of N intelligent vehicles on the road, each equipped with a local computing device and an onboard communication module. Vehicles and their local RSUs (Roadside Units) within their communication range communicate via wireless channels. Vehicles can choose to compute tasks locally or offload them to a MEC (Multi-access Edge Computing) server matched to their RSUs. In this scenario, all vehicles simultaneously generate one task to be processed in each time slot. Let the task generated by vehicle i be denoted as... ,in For task data size, The number of CPU cycles required per unit of task data. The maximum acceptable latency tolerance for completing the task. To achieve maximum energy consumption tolerance. In addition, auxiliary communication reflectors (RIS) are deployed on the surfaces of buildings within the scenario to enhance the transmission quality of wireless signals in non-line-of-sight environments;

[0193] Communication Model: When a vehicle chooses to wirelessly communicate with the RSU to offload a task, the total system communication bandwidth is B, which is divided into M equal-bandwidth resource blocks in the frequency domain. Different vehicles may occupy the same resource blocks when offloading tasks, resulting in co-channel interference. The uplink transmission rate of vehicle i when transmitting data is... Depends on its own transmission power Channel fading gain and co-channel interference power are calculated using Shannon's formula.

[0194] Decision variables include: calculating unloading decision variables , indicates the calculation mode selected by vehicle i, and can be either 0 or 1. When it means the task is calculated only locally in the vehicle, This indicates that the task is completely offloaded to the edge server for computation; communication resource block allocation decision variables. This represents the communication resource block index occupied during vehicle unloading, with values ​​{0, 1, ..., M-1}; and the auxiliary link configuration decision variable. , which represents the index of the specific phase scheduling scheme assigned to the vehicle by the smart surface;

[0195] When the vehicle selects local calculation (i.e.) When ), the task processing delay formula is:

[0196] ;

[0197] in, The number of CPU cycles required per unit of task data. This refers to the vehicle's local CPU frequency.

[0198] The corresponding local energy consumption formula is:

[0199] ;

[0200] in, Calculate the energy consumption coefficient locally;

[0201] When the vehicle selects edge unloading (i.e. When ), the formula for the offloading delay of transmission and computation coordination is:

[0202] ;

[0203] in, Total computing resources for MEC servers The actual transmission rate affected by interference from multiple vehicles;

[0204] The corresponding local energy consumption formula is:

[0205] ;

[0206] To ensure the system's reliable handling of latency-sensitive tasks and to uniformly evaluate the immediate rewards of the reinforcement learning environment, the actual comprehensive processing latency of vehicle i is considered in conjunction with the unloading decision variables. Limited by the maximum latency tolerance, the actual processing latency is calculated using the following formula:

[0207] ;

[0208] Based on this, the system constructs a fitness evaluation function for multi-agent reinforcement learning, which drives the subsequent training of the network and optimization of collaborative strategies.

[0209] Step 2: Initialize scene parameters based on Step 1 and build a high-speed experience replay pool, using a low-level tensor matrix pre-allocation mechanism to store interactive experiences;

[0210] Based on the V2X-MEC collaborative optimization model constructed in step 1, the scenario parameters and algorithm operation-related parameters are first initialized, including the number of vehicle intelligent agents N, the number of communication resource blocks M, the total number of auxiliary reflector phases L, the maximum storage capacity of the experience playback pool C, and the batch processing sampling size. And the initial weight parameters of each neural network, etc.

[0211] To address the massive amounts of high-dimensional interaction data generated in multi-agent environments and eliminate the computational bottleneck caused by traditional list appending, this invention employs a low-level tensor matrix pre-allocation mechanism to construct a high-speed experience replay pool. Contiguous storage space is allocated in memory for each interaction dimension, and an initial set of experience matrices is created. Represented as:

[0212] ;

[0213] Where E represents a set of matrices including the state observation pool S, the joint action pool A, the immediate reward pool R, the next state pool S', and the termination flag pool D, and d is the feature depth of each dimension.

[0214] In the model-environment interaction and storage phase, the experience tuple generated in a single interaction is defined as follows: , represented as:

[0215] ;

[0216] The system establishes storage update logic based on the cursor pointer ptr and the real-time data size size. When acquiring new interactive experiences... At that time, the underlying overwrite is performed directly by locating the matrix index through the pointer. The update formulas for the cursor pointer, the playback cursor pointer, and the real-time size of the playback pool are as follows:

[0217] ;

[0218] ;

[0219] During the training and update phase of the algorithm network, the system employs a low-level slice sampling technique based on random uniform distribution. The system first generates a slice containing... The set I of random indices is calculated as follows:

[0220] ;

[0221] Subsequently, the training samples for the current batch are obtained by slicing the pre-allocated contiguous memory tensor matrix directly according to the index set I. :

[0222] ;

[0223] Furthermore, the access and operation of the high-speed experience playback pool must strictly meet the following physical space and sampling stability constraints:

[0224] ;

[0225] ;

[0226] Among them, constraint C5 represents the capacity constraint of memory addressing, which uses the modulo operation of Equation 8 to implement circular storage to ensure that the pointer does not go out of bounds and automatically overwrites the oldest experience data; constraint C6 represents the sampling trigger threshold constraint, that is, only when the total number of experience data accumulated in the playback pool reaches the set single batch sampling size. Only when the system allows the opening of the underlying tensor slicing interface for batch data extraction and network parameter updates, can the stability of model gradient updates be guaranteed.

[0227] Step 3: Extract global collaborative features of each agent through a state token encoder and a multi-head self-attention mechanism (MHSA);

[0228] Furthermore, in step 3, the goal is to extract global cooperative interference features in a multi-vehicle concurrent environment through a state token encoder and a multi-head self-attention mechanism. This transforms the local observation states of each vehicle agent into a high-dimensional feature token sequence, calculates the correlation weights between different feature tokens, and finally fuses and outputs a cooperative feature vector with a global perspective. The mapping formula for the state tokens is:

[0229] ;

[0230] in, The representation layer normalization operation is used to normalize the feature vector, where s is the normalized state observation vector. For scalar mapping functions, For feature embedding matrix;

[0231] The generated feature token sequence is then input into the Multi-Head Self-Attention (MHSA) module. The query matrix is ​​then used... Key matrix Sum matrix It captures the complex coupling relationships and dynamic interference features between different state feature dimensions to achieve deep feature fusion; the core formula for calculating the association weights in the multi-head self-attention mechanism is:

[0232] ;

[0233] in, For querying the matrix, The key matrix, For value matrices, This is a scaling factor for the dimension of the attention head, used to prevent gradient vanishing. , , All are learnable linear mapping weight matrices.

[0234] Finally, the output of the attention mechanism is deeply fused with the feedforward neural network through residual connections, and layer normalization is performed to extract a highly abstract collaborative feature vector. :

[0235] ;

[0236] Collaborative feature vectors The system comprehensively characterizes the current vehicle's task load, computing power status, and degree of interference from the same frequency, and uses these as the underlying input features for the Actor network to generate joint decision-making actions and the Critic network to perform value assessments.

[0237] Step 4: Utilize the Actor network to generate joint discrete actions that include computation patterns, communication resource blocks, and auxiliary link configurations, and interact with the environment to generate instant rewards.

[0238] Furthermore, in step 4, the Actor network uses the global collaborative feature vector extracted in step 3. As input, discrete probability distributions for the corresponding resource dimensions are generated through three independent fully connected layer prediction heads. The formula for calculating the joint discrete action probability distribution is as follows:

[0239] ;

[0240] in, To generate the computational pattern decision probability for whether to perform edge unloading, Assign decision probabilities to generate communication resource blocks that occupy specific channels. To generate the auxiliary link configuration decision probability for a specific phase; since the joint optimization problem of vehicle networking involves three orthogonal action subspaces of computation, communication and auxiliary links, the above formula multiplies the three independent marginal probabilities to obtain the comprehensive decision probability of the current agent in the joint action space.

[0241] To ensure the stability of numerical computation during gradient backpropagation in network training, the formula for calculating the joint discrete action log probability in the Actor network is as follows:

[0242] ;

[0243] in, The minimum value of the stabilizing washer term (with a value of ) represents the value of the term that prevents gradient collapse in the discrete action space during logarithmic operations. When the probability of an Actor network predicting a certain action is extremely close to 0 in the early or late stages of exploration, The existence of this avoids the logarithmic function calculation result from outputting negative infinity, thus ensuring the stable update of the Actor network weight parameters.

[0244] The agent performs classification distribution sampling based on the calculated joint action probability distribution to obtain exploratory deterministic joint discrete actions. Subsequently, the action sets generated by all vehicle agents are sent in parallel to the vehicle network environment simulation model for execution. After receiving the actions, the environment model performs state transitions and calculates the task processing latency and energy consumption penalty caused by the current strategy based on the fitness evaluation function defined in step 1. Finally, it feeds back the agent with an instant reward value that evaluates the merits of the action for storage in the experience replay pool.

[0245] Step 5: Execute a global target action pre-computation mechanism during the training and update phase to infer action tensors in parallel and eliminate computational bottlenecks.

[0246] Furthermore, in step 5, to efficiently calculate the target value assessment value of each agent in the discrete action space, the system performs a global target action pre-calculation mechanism outside the network update loop. The formula for calculating the parallel inference action tensor is:

[0247] ;

[0248] in, Let j-th vehicle agent be a target Actor policy network without gradients. Let j be the state observation value of the j-th agent at the next time step. This is the set of tensors representing the next target action of all agents, inferred in parallel.

[0249] Step 6: Apply the Critic network based on the cross-attention mechanism to evaluate the value of the strategy, and update the network parameters using smooth L1 loss.

[0250] Further, in step 6, as Figure 3 The Critic network, based on the cross-attention mechanism, evaluates the value of the current policy action by aggregating global state features and the action information of neighboring agents, as follows:

[0251] First, the Critic network receives the global collaborative feature vector extracted in step 3 and the joint discrete actions performed by each agent. To accurately isolate action interference in a multi-vehicle concurrent environment, the action sequence is mapped using a high-dimensional embedding method, and then a cross-attention mechanism is used to fuse the action features of neighboring agents. The current agent's state features serve as the query matrix. The action embedding features of neighboring agents are used as the key matrix. Sum matrix The Critic network outputs a comprehensive action value assessment. The calculation formula is:

[0252] ;

[0253] in, This is the cross-attention operation function. This is the mapping function for the fully connected multilayer perceptron in the network backend; This represents two independent Critic evaluation network branches set up in parallel to alleviate the Q-value overestimation problem in traditional reinforcement learning.

[0254] Subsequently, the target action tensor calculated in parallel by the target network in step 5 is obtained, and the soft target value is calculated by combining it with the immediate reward from environmental feedback. This is used as the true label for the current Critic network update. Since vehicular network tasks may generate abnormally large latency penalties under extreme congestion, to prevent network gradient explosion caused by excessive absolute error, this invention uses a smoothed L1 loss function to calculate the time difference error. The formula for calculating the loss function of the Critic network is:

[0255] ;

[0256] in, This refers to the batch size of a single slice sample in the high-speed experience replay pool. and This is a sample of states and actions in the current batch; The piecewise function for smoothing error is specifically defined mathematically as follows:

[0257] ;

[0258] in, This represents the absolute value of the error.

[0259] Step 7: Update the Actor network based on the policy gradient and simultaneously perform adaptive target entropy adjustment to optimize the temperature coefficient;

[0260] Furthermore, in step 7, the specific process of updating the Actor network based on the policy gradient and simultaneously performing adaptive target entropy adjustment is as follows:

[0261] First, in order to update the Actor policy network, the system uses the state observation samples of the current batch. Recalculate the probability distribution of joint discrete actions using the Actor network. Combining the action value assessment benchmarks given by the two Critic evaluation networks in step 6. The optimization objective of an Actor network is to maximize the weighted sum of the expected value of an action and the policy entropy. The corresponding policy gradient loss function is calculated as follows:

[0262] ;

[0263] in, This refers to the batch size of a single slice sampled in the high-speed experience playback pool. To balance the temperature coefficient between exploration and utilization, the system updates the weight parameters of the Actor network by calculating the loss function and performing gradient backpropagation.

[0264] Subsequently, to prevent multi-vehicle intelligent agents from falling into purely random and blind divergent exploration in a high-dimensional discrete action space due to forced high target entropy, this invention introduces an adaptive dynamic target entropy adjustment mechanism. Based on the allocation characteristics of discrete resources in the vehicle network, a dynamic target entropy biased towards utilization is set. Its calculation formula is:

[0265] ;

[0266] in, The total dimension of the discrete action space of the vehicle-to-everything (V2X) system;

[0267] Based on the set dynamic target entropy The system has a learnable temperature coefficient. Synchronous gradient optimization is performed. The formula for calculating the alpha loss function for optimizing the temperature coefficient is shown in Equation 24:

[0268] ;

[0269] Based on the above equation, the system constructs a negative feedback adjustment closed loop for exploration and exploitation: when the true information entropy of the current strategy is lower than the target entropy, the loss function drives the temperature coefficient. Increase the size of the policy space to encourage agents to explore unknown policy spaces; conversely, when policies become too divergent, the driving force... Reduce to accelerate network convergence;

[0270] Step 8: Determine if the number of iterations has reached the maximum number of rounds. If the iterations are complete, end the training and output the optimal collaborative resource allocation strategy model.

[0271] The system completes the Actor network, Critic network, and temperature coefficient calculation for the current round. After updating the parameters, the current training iteration round number is incremented by 1. In each iteration, the vehicle agent interacts with the vehicle network environment again based on the updated policy network weights, uses the multi-head self-attention mechanism to extract dynamically changing global collaborative features again, and continuously corrects the joint allocation actions of the computing mode, communication resource blocks, and auxiliary links, thereby improving the convergence and global optimization ability of the algorithm in complex co-frequency interference environments.

[0272] The system then determines whether the current number of training iterations has reached the preset maximum number of rounds. If the maximum number of rounds has not been reached, return to step 3, and control the agent to continue collecting and sampling new interaction experiences from the rapid experience replay pool, entering the next round of iterative optimization; if the current iteration has reached the maximum number of rounds... If the multi-agent network training is completed, the system ends the loop and outputs the optimal Actor policy network weights that have converged at this time. These weights are the optimal vehicle network computing and communication collaborative resource allocation strategy model required for the problem of this invention.

[0273] In embodiments of the present invention, such as Figure 1 As shown, the specific steps are as follows:

[0274] Initialize V2X-MEC scenario parameters: set the number of vehicles, number of communication resource blocks, number of auxiliary link phases, maximum number of iteration rounds, and batch size;

[0275] Initialize the underlying tensor matrix rapid experience replay pool;

[0276] Initialize the Actor policy network, two Critic evaluation networks, and the corresponding Target network for each vehicle agent, and initialize the adaptive temperature coefficient;

[0277] While t < maximum number of iterations Tmax:

[0278] Reset the vehicle network environment and obtain the initial observation status of all intelligent agents;

[0279] For each environmental interaction time slot, do:

[0280] The application of a state token encoder and a multi-head self-attention mechanism extracts global collaborative features;

[0281] Utilize Actor networks to generate joint discrete actions based on feature-based computational patterns, communication resource blocks, and auxiliary link configurations;

[0282] The agent executes joint actions, the environmental state transitions to the next state, and provides immediate task rewards and termination flags.

[0283] A low-level pointer overwriting mechanism is used to store interactive experience tuples into the ultra-fast experience replay pool;

[0284] If the amount of data in the replay pool is greater than or equal to the batch size:

[0285] Batch empirical data were sampled from the playback pool based on randomly uniformly distributed slices.

[0286] Execute a global target action pre-calculation mechanism and infer the target action tensor for the next state in parallel.

[0287] Calculate the target Q value and apply smoothed L1 loss to update the parameters of the two Critic networks;

[0288] The policy gradient and Alpha loss are calculated by combining the dynamic target entropy, and the Actor network parameters and temperature coefficient are updated synchronously.

[0289] Smoothly update the target network weights using a soft update mechanism;

[0290] The state is updated to the next state;

[0291] If the termination condition is met, the current round of interaction ends;

[0292] t = t + 1;

[0293] Training ends after reaching the maximum number of iterations, and the optimal Actor collaborative resource allocation strategy model is output.

[0294] In addition, the Dual Deep Q Network (D3QN) and the Advantageous Actor-Critic (A2C) algorithm were selected as comparison algorithms for performance verification.

[0295] In this invention, the maximum number of training rounds for the algorithm is 2500, the batch size for a single slice sampling is 256, and the initial temperature coefficient is set to 1.0. Specific scene simulation parameters are shown in Table 1.

[0296] Table 1 Simulation Parameters

[0297]

[0298] After 2500 training experiments, the average result was obtained. The experimental data of the algorithm of this invention are as follows: Figures 4-7 As shown, the comparison between the system ablation experiment and the benchmark algorithm is as follows: Figures 8-13 As shown.

[0299] As the number of training experiments increases, Figure 4 The results show that by learning optimal offloading decisions, resource allocation, and RIS phase, the average task reward score of multi-agent systems steadily increases. Figure 5 , Figure 6 The average task latency and energy consumption were significantly reduced, demonstrating the effectiveness of the edge computing collaborative offloading strategy.

[0300] Figure 7 This study reveals the underlying training dynamics of reinforcement learning, with Alpha demonstrating the decay process of the adaptive temperature parameter. This decay curve illustrates that the agent actively explores the complex joint action space in the early stages of training, and gradually converges and shifts to utilizing the learned optimal network resource scheduling strategy as training progresses.

[0301] Figures 8-10 The graphs show the ablation experiment performance comparison curves of the algorithm of this invention under different variant modules. When the RIS-assisted communication link and the MHSA global collaborative feature extraction module are introduced into the system, the high-dimensional action space is reduced to a low-dimensional action space through the multi-head self-attention mechanism, and the agent's ability to resist co-frequency interference is significantly enhanced. Figure 8 The report indicates that the average task reward for the complete architecture during convergence in the later stages of training jumps significantly to over 145 points. Figure 9 , Figure 10 The figures indicate that the average task latency and average energy consumption are significantly lower than those of the DSAC and RIS architectures without attention, as well as the basic pure DSAC architecture. In high-dimensional action spaces, the DSAC and RIS architectures without attention cannot handle high-dimensional actions, resulting in system dimensionality collapse. The pure DSAC architecture, in the presence of severe Ricean fading and multipath scattering, experiences severe physical collisions of multipath signals at the receiver, leading to system paralysis relying solely on basic reinforcement learning algorithms for RB scheduling and offloading decisions. This fully validates the necessity and effectiveness of the multi-head self-attention collaborative feature extraction mechanism and the intelligent surface-assisted module in complex high-dimensional environments.

[0302] Figures 11-13 The graph shows the performance comparison curves of the cooperative offloading of the algorithm of this invention (DSAC Proposed) with other mainstream benchmark algorithms (D3QN, A2C). Under the same vehicle network communication resource conditions and task data scale, traditional reinforcement learning algorithms such as D3QN and A2C are prone to getting trapped in local optima in complex discrete action spaces due to the lack of an effective multi-agent global action evaluation mechanism. Figure 11 This indicates that the optimized DSAC algorithm yields significantly higher rewards than the A2C and D3QN communication resource allocation algorithms. Figure 12 This indicates that the optimized DSAC algorithm has significantly lower latency than the A2C and D3QN communication resource allocation algorithms. Figure 13 The results show that the energy consumption of the optimized DSAC algorithm is significantly lower than that of the A2C and D3QN communication resource allocation algorithms. In comparison, the optimized algorithm model of this invention can quickly escape local optima, resulting in a sharp decrease in both average task latency and energy consumption, with all performance indicators completely outperforming the compared algorithms. Therefore, the improved multi-agent discrete SAC algorithm has an absolute leading advantage in the joint optimization problem of communication and computing resources aimed at reducing latency and energy consumption.

[0303] In summary, the multi-agent communication resource allocation optimization method based on SAC-MHSA for RIS-assisted vehicle networking proposed in this invention can effectively reduce the processing latency of tasks in multi-vehicle concurrent scenarios, achieve the global optimal allocation of edge computing power and communication channel resources, and significantly improve the overall collaborative performance of the vehicle networking system. Its model converges very quickly, is robust, and is easy to deploy and implement in real-world complex edge computing environments.

[0304] It will be understood by those skilled in the art that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A RIS-assisted vehicular network communication resource allocation optimization method based on the SAC-MHSA algorithm, characterized in that, Includes the following steps: Step 1: Establish a V2X-MEC collaborative optimization model using a multi-agent reinforcement learning framework, including an environmental simulation model and a multi-dimensional computation and communication model; Step 2: Initialize scene parameters based on Step 1 and build a high-speed experience replay pool, using a low-level tensor matrix pre-allocation mechanism to store interactive experiences; Step 3: Extract global collaborative features of each agent through a state token encoder and a multi-head self-attention mechanism; Step 4: Utilize the Actor network to generate joint discrete actions that include computation patterns, communication resource blocks, and auxiliary link configurations, and interact with the environment to generate instant rewards; Step 5: Execute a global target action pre-computation mechanism during the training and update phase to infer action tensors in parallel and eliminate computational bottlenecks. Step 6: Apply the Critic network based on the cross-attention mechanism to evaluate the value of the strategy, and update the network parameters using smooth L1 loss. Step 7: Update the Actor network based on the policy gradient and simultaneously perform adaptive target entropy adjustment to optimize the temperature coefficient; Step 8: Determine if the number of iterations has reached the maximum number of rounds. If the iteration is complete, end the training and output the optimal collaborative resource allocation strategy model.

2. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that: In step 1, the environmental simulation model is a road of a set length, with Roadside Units (RSUs) deployed along the roadside. Each RSU is connected to a Mobile Edge Computing (MEC) server via a wired connection. The total computing resources of the MEC server are... ; on the road The set of intelligent vehicles equipped with local computing power devices is denoted as V={1,2, ...,N}. The vehicles communicate with the RSU via a wireless network and can choose to execute tasks locally or offload them to the MEC server for computation. In the scenario, all vehicles simultaneously generate a task to be processed in each time slot. The generated task characteristics are set as follows ,in, For task data size, The number of CPU cycles required per unit of task data. The maximum acceptable latency tolerance for completing the task. To maximize energy consumption tolerance, auxiliary communication surfaces are also deployed within the scenario to optimize signal quality in non-line-of-sight environments.

3. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 2, characterized in that, The communication model is as follows: when the vehicle chooses to communicate with the RSU for offloading, the total system communication bandwidth is B, which is divided into M resource blocks in the frequency domain; when the vehicle transmits mission data, it is affected by its own transmit power. The dynamic change in the signal-to-interference-plus-noise ratio (SNR) at the receiver is denoted as the effect of joint channel fading and co-channel interference power generated by sharing the same resource block. .

4. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 3, characterized in that, Decision variables include unloading decision variables. Resource block allocation decision variables and auxiliary link configuration decision variables ;in This indicates the task calculation mode for vehicle i, taking the value 0 or 1. When the task is calculated locally on the vehicle, This indicates that the task is unloaded to the MEC server for computation; This indicates the index of the resource block occupied when vehicle i is unloaded; This represents the auxiliary communication phase scheduling configuration index assigned to vehicle i by the system; When the vehicle selects local calculation ( The formulas for processing delay and energy consumption at that time are: ； ； in, Where is the local CPU frequency of the vehicle, and k is the local calculation energy consumption coefficient; When the vehicle selects edge unloading ( The processing delay and energy consumption formulas are as follows: ； ； in, The actual transmission rate affected by interference from multiple vehicles. Uplink transmission power for vehicles; A fitness function is established with the objective of minimizing system overhead, and its mathematical expression is as follows: ； in, and Assigning weights to latency and energy consumption, Timeout penalties are set for timed-out tasks and co-frequency interference of resource blocks; Constraints: ； ； ； ； Among them, This indicates the task calculation mode for vehicle i, and can be either 0 or 1. This indicates the index of the resource block occupied when vehicle i is unloaded; This indicates the minimum permissible value for the vehicle's local CPU frequency. This indicates the maximum allowed value for the vehicle's local CPU frequency.

5. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, In step 2, the specific process of initializing scene parameters and constructing the rapid experience replay pool is as follows: Based on the vehicle-to-everything (V2X) MEC system model constructed in step 1, the simulation scenario parameters are first initialized, setting the number of vehicle agents to N, the number of communication resource blocks to M, and the total number of auxiliary reflector phases to L; subsequently, a high-speed experience playback pool is constructed, with a maximum storage capacity of C and a batch processing sampling size of [missing information]. A low-level tensor matrix pre-allocation mechanism is used in memory to establish contiguous memory blocks for each interaction dimension, with the initial state shown in the following equation: ； ； Where E represents a set of matrices including the state observation pool S, the joint action pool A, the immediate reward pool R, the next state pool S', and the termination flag pool D, and d is the feature depth of each dimension. In the experience storage phase, the experience tuple generated in a single interaction is defined as follows: As shown in the following formula: ； Establish storage update logic based on cursor pointer ptr and real-time scale size, when new interactive experiences are obtained. At this time, data overwriting is performed directly by locating the matrix index through the pointer; the update formula for the pointer and pool size is as follows: ； ； This mechanism operates through pre-allocated contiguous memory blocks in the NumPy underlying layer, ensuring that the data writing process has a constant time complexity; During the algorithm update phase, a bottom-level slice sampling method based on a random uniform distribution is used to first generate a sample containing... The set I of random indices is calculated using the following formula: ； Subsequently, the pre-allocated contiguous memory matrix is ​​sliced ​​directly according to the index set I to obtain the training samples for the current batch. : ； The operation of the high-speed experience playback pool is subject to physical constraints related to capacity overflow and sampling stability, as shown in the following formula: ； ； Among them, constraint C5 ensures that memory addressing does not go out of bounds and uses modular arithmetic to implement circular storage to isolate the negative impact of stale data; constraint C6 specifies the sampling trigger threshold, that is, only when the total amount of experience accumulated in the playback pool reaches the set batch size is the subsequent network update process allowed to be executed.

6. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, The state token encoder and multi-head self-attention mechanism in step 3 aim to extract global cooperative interference features in a multi-vehicle concurrent environment. They transform the local observation states of each vehicle agent into a high-dimensional feature token sequence, calculate the correlation weights between different feature tokens, and finally fuse them to output a cooperative feature vector with a global perspective. The mapping formula for the state tokens is: ； in, The representation layer normalization operation is used to normalize the feature vector, where s is the normalized state observation vector. For scalar mapping functions, The feature embedding matrix is ​​used; the core formula for calculating the association weights in the multi-head self-attention mechanism is: ； in, For querying the matrix, The key matrix, For value matrices, The dimension scaling factor for the attention head. , , All are learnable linear mapping weight matrices.

7. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, In step 4, the Actor network takes the extracted collaborative features as input and generates discrete probability distributions for the corresponding resource dimensions through three independent fully connected prediction heads. The formula for calculating the joint discrete action probability distribution is as follows: ； in, To generate the computational pattern decision probability for whether to perform edge unloading, Assign decision probabilities to generate communication resource blocks that occupy specific channels. The auxiliary link is configured with decision probabilities to generate a specific phase; the formula for calculating the joint discrete action log probability of the Actor network is: ； in, The minimum value of the stabilizing washer term, representing the prevention of gradient collapse in the discrete action space during logarithmic operations, is taken as [value missing]. The agent performs classification and sampling based on the calculated joint action probability distribution to obtain the final joint discrete action and sends it to the vehicle network environment for execution. After receiving the action, the environment updates its state for the next moment and outputs an instant reward value to evaluate the merits of the action.

8. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, In step 5, the formula for inferring the action tensor using the global target action pre-computation mechanism is: ； in, For the j-th agent, there is a gradient-free objective Actor policy network. Let j be the state observation value of the j-th agent at the next time step. This is the set of tensors for the next target actions of all agents in parallel inference; the pre-computation mechanism extracts the target action sequences of all vehicle agents in batches at once outside the traversal loop of the Critic value network update.

9. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, In step 6, the Critic network based on the cross-attention mechanism evaluates the value of the current policy action by aggregating global state features and the action information of neighboring agents, as follows: First, the Critic network receives the global collaborative feature vector extracted in step 3 and the joint discrete actions performed by each agent. After performing high-dimensional embedding mapping on the action sequence, it uses a cross-attention mechanism to fuse the action features of neighboring agents. The current agent's state features serve as the query matrix. The action embedding features of neighboring agents are used as the key matrix. Sum matrix ; Critic network output comprehensive action value assessment value The calculation formula is: ； in, This is the cross-attention operation function. This is the mapping function for the fully connected multilayer perceptron in the network backend; This represents two independent Critic evaluation network branches set up in parallel to alleviate the Q-value overestimation problem in traditional reinforcement learning; Subsequently, the soft target value calculated in parallel by the target network in step 5 is obtained. This is used as the true label for the current Critic network update; the time difference error is calculated using a smoothed L1 loss function. The formula for the Critic network's loss function is: ； in, This refers to the batch size of a single slice sample in the high-speed experience replay pool. and This is a sample of states and actions in the current batch; The piecewise function for smoothing error is specifically defined mathematically as follows: ； in, The absolute value of the error; The system achieves extremely fast and stable updates to the weights of the multi-agent Critic value network by calculating the smooth L1 loss function and using the optimizer to perform gradient backpropagation.

10. The RIS-assisted vehicle-to-everything (V2X) communication resource allocation optimization method based on the SAC-MHSA algorithm according to claim 1, characterized in that, In step 7, the specific process of updating the Actor network based on the policy gradient and simultaneously performing adaptive target entropy adjustment is as follows: First, the system observes the state samples of the current batch. Recalculate the probability distribution of joint discrete actions using the Actor network. Combined with the action value assessment benchmarks given by the two Critic evaluation networks in step 6. The optimization objective of the Actor network is to maximize the weighted sum of the expected value of the action and the policy entropy; the corresponding policy gradient loss function is calculated as follows: ； in, This refers to the batch size of a single slice sampled in the high-speed experience playback pool. To balance the temperature coefficient between exploration and utilization, the system updates the weight parameters of the Actor network by calculating the loss function and performing gradient backpropagation. Subsequently, to address the allocation characteristics of discrete resources in the Internet of Vehicles (IoV), a dynamic target entropy biased towards utilization was set. The calculation formula is shown below: ； in, The total dimension of the discrete action space of the vehicle-to-everything (V2X) system; Based on the set dynamic target entropy For learnable temperature coefficient Perform synchronous gradient optimization; the Alpha loss function for optimizing the temperature coefficient is calculated as follows: ； When the true information entropy of the current strategy is lower than the target entropy, the loss function drives the temperature coefficient. Increase the size of the policy space to encourage agents to explore unknown policy spaces; conversely, when policies become too divergent, the driving force... Reduce to accelerate network convergence; Each iteration calculates a new action probability distribution based on the policy network, and updates the Actor network parameters and temperature coefficient simultaneously; it checks whether the number of iterations has reached the maximum number of rounds. If not, it returns to step 3 to continue updating; after reaching the maximum number of iterations, it outputs the optimal Actor policy network weights, which is the optimal collaborative resource allocation strategy model.

Images