A Vehicle Network Edge Buffer Transfer Method Based on Reinforcement Learning and PID Control

By combining reinforcement learning and PID control, the vehicle-to-everything (V2X) buffer transmission rate is adaptively adjusted, solving the data frame loss problem caused by wireless interference and achieving low-latency, high-throughput transmission in environments with wireless loss.

CN123123848APending Publication Date: 2026-11-17IANGSU COLLEGE OF ENG & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611598143.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing vehicle-to-everything (V2X) buffering and transmission methods fail to effectively handle data frame loss caused by wireless interference, resulting in increased transmission latency and decreased throughput, especially poor performance in scenarios with severe packet loss.

Method used

A method based on reinforcement learning and PID control is adopted to detect interference and packet loss in the wireless channel and the data frame queue length of the transmission buffer. The task transmission rate is adaptively adjusted, and the application layer rate is dynamically adjusted when there is heavy packet loss. The buffer scheduling transmission is achieved through the designed ACK confirmation frame and broadcast frame.

Benefits of technology

It effectively reduces end-to-end latency of data frames, maintains high throughput, and preserves transmission reliability and efficiency in wireless loss environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123848A_ABST
    Figure CN123123848A_ABST
Patent Text Reader

Abstract

This invention discloses a vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control, relating to the fields of V2X and data transmission technology. It solves the technical problem of increased latency in vehicle-to-everything (V2X) buffer file transmission under wireless interference and packet loss environments. The method includes the following steps: S1: latency modeling and transmission buffer queue length detection for V2X buffer transmission; S2: detecting the data frame loss rate of wireless communication between the vehicle and the base station; S3: detecting throughput; S4: executing rate control based on dynamic dual-Q reinforcement learning; S5: executing rate control and buffer scheduling based on proportional-integral-derivative (PID) control. This invention uses reinforcement learning algorithms under mild and moderate packet loss conditions and PID control algorithms under severe packet loss conditions to dynamically adjust the application layer transmission rate, and achieves buffer scheduling transmission through designed ACK acknowledgment frames and broadcast frames.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of vehicle networking and data transmission technology, and in particular to a vehicle networking edge buffer transmission method based on reinforcement learning and PID control. Background Technology

[0002] By deploying user files and applications on vehicle terminals or edge servers, vehicle-to-everything (V2X) edge caching technology can accelerate file delivery and reduce redundant file transfers, thereby speeding up computational response and reducing backbone network bandwidth consumption. Considering a scenario where vehicles communicate with roadside edge servers via base stations, after a vehicle terminal generates a computational task, the task can be scheduled to be completed either locally on the vehicle or on the edge server. If the decision is made to complete the computation on the edge server, the vehicle needs to transmit the task data to the edge server via a wireless channel.

[0003] Task data can be specifically divided into user parameter files and application files. The former consists of personalized data generated by vehicle users, while the latter is a relatively generic application. Applications are usually large, and for ease of transmission, they can be further broken down into several program components. When a vehicle user needs to complete a task on an edge server, both the user parameter file and the application file need to be transmitted to the server via the base station. However, if the server does not store all or part of the application components required for the computation task, the vehicle needs to transmit the missing components of the application.

[0004] Existing vehicular network (V2N) buffer transmissions are directly delivered to the lower layer at the application layer according to the set task generation rate. The transmission rate of the Media Access Control (MAC) layer is usually calculated using the Shannon formula. However, due to the characteristics of wireless channels, V2N transmission buffer files may experience two types of data frame loss. The first is collision loss between data frames, which occurs when terminal nodes simultaneously contend for a single shared channel. Data frames from different source nodes collide on the channel, and the superposition of the two data frame signals prevents the receiving node from properly parsing the data frame, resulting in data loss (hereinafter referred to as collision loss). The second is when the signal transmitted by the vehicle is interfered with by wireless devices outside the V2N network or blocked by obstacles, preventing the data frame from being correctly parsed, thus resulting in data frame loss (hereinafter referred to as interference loss). If the source node does not receive an ACK confirmation frame, it can be determined that the transmitted data frame has been lost. To ensure reliable data transmission, wireless communication protocols usually have a data frame retransmission mechanism to retransmit lost data frames. This brings two problems. On the one hand, retransmitted data frames need to be temporarily stored in the transmission buffer before receiving an ACK confirmation from the destination node, which makes it easier for subsequent data frames arriving in the buffer to queue up. On the other hand, retransmitted data frames are equivalent to adding extra load to the MAC layer, further consuming MAC bandwidth. Both of these factors increase queuing latency, which in turn leads to an increase in end-to-end latency.

[0005] The paper "Edge Caching Strategy Based on Multi-Agent Deep Reinforcement Learning in Cloud-Edge-Device Scenarios" proposes an edge caching strategy based on a multi-agent deep reinforcement learning algorithm, aiming to maximize cumulative rewards. However, the communication overhead modeled only from the perspective of transmitted data, and the transmission delay was only given as a fixed interval without deriving or using quantitative indicators such as channel capacity. Therefore, the research lacks a communication context. The paper "A Task Cooperative Offloading Algorithm Based on Service Caching for Vehicle Edge Computing" proposes a cache offloading (JACTOP) algorithm based on deep reinforcement learning. This study uses Shannon's formula to model wireless communication, considering the impact of factors such as vehicle transmit power and signal spectrum on MAC layer bandwidth, and derives communication delay. Because this study uses independent channels, each vehicle has its own dedicated wireless bandwidth, eliminating the problem of data frame collision and loss between vehicles. However, the study also ignores interference and packet loss caused by external interference in the electromagnetic space (such as speed radar, fixed or moving obstacles). Therefore, this paper conducts research based on the assumption of no packet loss and does not consider the pressure on the transmission queue and bandwidth caused by the data frame retransmission mechanism. Therefore, the JACTOP buffered transmission algorithm causes data frames to queue in the vehicle's transmission buffer for a long time, resulting in poor transmission latency and performance.

[0006] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0007] The purpose of this invention is to provide a vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control. For mild and moderate packet loss caused by wireless interference, the method detects interference packet loss in the wireless channel and the data frame queue length in the transmission buffer, and adaptively adjusts the task transmission rate using a reinforcement learning mechanism. For severe packet loss, when the data frame loss rate due to wireless interference exceeds the packet loss threshold, the method uses classic PID control to dynamically adjust the application layer rate. In addition, the method implements buffered scheduling transmission, which effectively reduces the end-to-end latency of data frames and can maintain high throughput even in environments with wireless data frame loss.

[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: a vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control, comprising the following steps:

[0009] S1: Delay modeling and transmit buffer queue length detection for vehicle-to-everything (V2X) cache transmission;

[0010] S2: Detects the data frame loss rate of wireless communication between the vehicle and the base station;

[0011] S3: Detection throughput;

[0012] S4: Perform rate control based on dynamic double-Q reinforcement learning;

[0013] S5: Performs rate control and buffer scheduling based on proportional-integral-derivative PID control.

[0014] Further, step S1 includes the following steps:

[0015] S11, Delay Modeling of Vehicle Network Buffer Transmission

[0016] Since wireless communication systems have a retransmission mechanism, let the maximum number of retransmissions be m. The source node will attempt to send the data frame frame at most m+1 times. If the source node does not receive an ACK confirmation frame after the data frame frame has been sent m+1 times, the source node will abandon the transmission and discard the data frame frame. Let the data frame loss rate be p. Then the probability of a single successful transmission of the data frame frame is 1-p, and the failure probability is p. Let K be the total number of transmission attempts for the data frame frame, and each transmission is independent of the others. The probability of success on the k-th transmission attempt is... for:

[0017] (1);

[0018] in, ;

[0019] Assuming all m+1 transmissions fail, the data frame is discarded by the source node. The probability of the data frame being discarded is... for:

[0020] (2);

[0021] Substituting formula (1) into formula (2), we get:

[0022] (3);

[0023] Let n be a non-negative integer variable, and let the subtrahend term in formula (3) be:

[0024] (4);

[0025] According to the formula for the sum of a geometric series, we have:

[0026] (5);

[0027] Based on formulas (4) and (5), formula (3) is:

[0028] (6);

[0029] Let the average number of transmission attempts for a data frame be... ,have:

[0030]

[0031] = (7);

[0032] Based on the formula for infinite series and the finite term truncation, the second term on the right side of formula (7) is:

[0033] (8);

[0034] Then the average number of sending attempts for:

[0035] (9);

[0036] Simplify formula (9) to calculate the average number of sending attempts. for:

[0037] (10);

[0038] Data sent from the application layer of the source node is encapsulated into data frames at the Media Access Control (MAC) layer and then transmitted to the wireless channel via the wireless network card. External interference or shielding on the wireless channel can cause data frame loss, preventing the destination node from sending an ACK confirmation frame and triggering the source node to initiate a retransmission mechanism. Let the length of each data frame be L (in bits), the application layer's transmission rate be rate (in megabits per second, Mb / s), and the duration of a timeslot be Duration (in seconds). Within one Duration, the total number of data frames received by the MAC layer is:

[0039] (11);

[0040] Let the vehicle's wireless communication bandwidth be B, measured in megabits per second (Mb / s). Within the duration of one time slot (Duration), the total number of data frames (Output) sent by the Media Access Control (MAC) layer is:

[0041] (12);

[0042] Considering the reception and transmission of data at the Media Access Control (MAC) layer, within the duration of one time slot (Duration), the queue length q of data frames in the vehicle's transmission buffer is:

[0043] (13);

[0044] The queuing delay occurs after a data frame arrives at the transmit buffer. for:

[0045] ;

[0046] (14);

[0047] Substitute formula (10) into formula (14). for:

[0048] (15);

[0049] When the time slot duration, vehicle wireless communication bandwidth B, maximum retransmission count m, and data frame length L are fixed values, the queuing delay depends on the data frame loss rate p and the application layer transmission rate rate.

[0050] When the queue ends, the data frame transmission delay... for:

[0051] (16);

[0052] The end-to-end delay of the data frame is:

[0053] (17);

[0054] S12. Send buffer queue length detection

[0055] When the transmit buffer of the Media Access Control (MAC) layer receives a newly arrived data frame, the receive function is triggered. Read the current queue length curq_, the pseudocode is as follows:

[0056] ;

[0057] .

[0058] Further, S2 includes:

[0059] S2 includes:

[0060] In the t-th sampling period, the vehicle The total number of data frames sent is recorded. The base station counts the number of data frames received. Design a response frame such that the base station constructs a total of Y response frames within the (t+1)th sampling period, and sends the y-th response frame. Enter "0001" in the "Type" field for positive integers. , representing a response frame Its function is to statistically analyze the frame loss rate in response frames. In the "Number of Received Frames" field, enter the number of received data frames obtained by the base station in the t-th period. In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The base station sends Y response frames sequentially to the MAC address of the vehicle. Receive any response frame within the (t+1)th sampling period That is, to know the number of data frames received by the base station in the t-th period. Then the data frame loss rate p is:

[0061] (18).

[0062] Further, S3 includes:

[0063] S3 includes:

[0064] Based on the statistics in S2, the throughput Thr between the vehicle and the base station in the t-th period is calculated as follows:

[0065] (19).

[0066] Further, S4 includes:

[0067] S41, State Space Design

[0068] In terms of observation dimensions, a two-dimensional environment observation vector is adopted, which includes the queue length q of the vehicle transmission buffer and the data frame loss rate p. Through normalization, boundary clipping, and binning operations, the two-dimensional environment observation vector is transformed into a one-dimensional discrete state index, and operators are defined. For floor operations, the fractional part of the floating-point binned value is truncated to obtain the integer index. The queue length hyperparameter is used. The total number of bins is the queue length, and the loss rate hyperparameter is... To define the total number of bins for the data frame loss rate, and to ensure that the index falls within a valid value range, a boundary pruning function is defined. for:

[0069] (20);

[0070] in, Enter the values ​​to be cropped;

[0071] S411, Queue Length Normalization and Binning

[0072] Perform queue length normalization, mapping the queue length index to the value range [0,1]. Set the maximum queue length. Normalized queue length for:

[0073] (twenty one);

[0074] Perform binning operation to obtain the queue length index. for:

[0075] (twenty two);

[0076] S412, Numerical clamping and binning of data frame loss rate p

[0077] Perform numerical clamping, and set the loss rate threshold as follows: Define the clipping loss probability. for:

[0078] (twenty three);

[0079] Perform binning operations to obtain the data frame loss rate index. :

[0080] (twenty four);

[0081] S413, Index Concatenation

[0082] The two-dimensional discrete index is mapped to a discrete state s using the following dimension-weighted concatenation formula:

[0083] (25);

[0084] The range of the number of discrete states s is as follows: ,in The upper limit of discrete states;

[0085] S42, Motionspace Design

[0086] A low-dimensional discrete action space is used to reduce the overhead of table lookup calculations, and three types of rate actions are set. These correspond to three different transmission rate adjustment amounts. Discrete action value range Define the absolute value of the transmission rate adjustment amount as ,action This represents reducing the transmission rate; the rate adjustment amount. ,action This represents maintaining the current sending rate, with the rate adjustment amount. =0, action This represents increasing the sending rate; the rate adjustment amount. for After each strategy iteration completes the action selection, the sending rate is set based on the action offset. To avoid network overload or zero throughput caused by unlimited increases or decreases in the sending rate, the following rules apply to the sending rate: If the current sending rate is less than the lower limit... Then the updated sending rate will be set to... If the current sending rate is greater than the rate limit Then the updated sending rate will be set to... ;

[0087] Let the current transmission rate be... After executing action a, the intermediate process rate is obtained. for:

[0088] (26);

[0089] For intermediate process rate Apply boundary constraints and update the transmission rate. for:

[0090] (27);

[0091] S43. Reward Function Design

[0092] Using statistical throughput as the positive reward, and introducing queues and packet loss as penalties, a reward function is constructed. The reward is a linear weighted sum of throughput gain and penalty. At the same time, upper and lower boundary constraints are set to suppress reward explosion under extreme network conditions and ensure stable iteration.

[0093] S431, Calculate Rewards

[0094] Throughput calculated according to formula (19) The data frame loss rate is calculated using formula (18). The queue length q of the send buffer obtained in S12, and the reward. The calculation formula is:

[0095] (28);

[0096] in, The reward weight for throughput, The reward weight of the queue, The reward weight for the data frame loss rate;

[0097] S432, Reward Truncation Constraint

[0098] Set the maximum threshold for rewards to The minimum threshold for rewards is To prevent extreme observations from causing a reward explosion, the following measures are taken:

[0099] (29);

[0100] S44, Action Selection Strategy

[0101] Given the current discrete state To balance environmental exploration and strategy utilization, an exploration probability of is adopted. Greedy strategy output action ;

[0102] Set up two independent "state-action" value matrices Value Matrix With value matrix The elements in the scalar are denoted as scalars. , representing the estimated long-term cumulative return corresponding to action a in discrete state s, the value matrix. With value matrix Initialize all values ​​to 0 and iterate alternately to mitigate overestimation bias;

[0103] Define the transfer storage set Any "state-action" pair corresponds to a transition unit ,in This represents the next discrete state obtained after the environment transition following the execution of action a; To effectively identify Boolean values, This represents a "state-action" pair with historical interaction samples, used for offline planning;

[0104] S441, Generation Interval Uniform random number variables within ;

[0105] S442, Exploration Branch

[0106] like Randomly select actions ;

[0107] S443, Utilizing Branches

[0108] Calculate the mean of the two Q values ​​for each action. Then select the action with the largest mean:

[0109] (30);

[0110] S45, Algorithm Iterative Calculation

[0111] S451, Matrix Element Update

[0112] matrix and The elements are updated alternately to decouple the selection of the optimal action from the evaluation of the target value. The input parameters are the current state s, the action to be performed a, the reward, and the next state. Next action ;

[0113] S4511, Value Matrix Update elements;

[0114] S45111, in In China, a value matrix is ​​used. Select the optimal action ;

[0115] (31);

[0116] S45112, Target Value From the value matrix The corresponding action value calculation includes:

[0117] (32);

[0118] in, Discount factor;

[0119] S45113, Update ;

[0120] (33);

[0121] in, The learning rate;

[0122] S4512, Value Matrix Update elements

[0123] S45121, in In China, a value matrix is ​​used. Select the optimal action :

[0124] (34);

[0125] S45122, Target Value From the value matrix The corresponding action value calculation includes:

[0126] (35);

[0127] S45123, Update :

[0128] (36);

[0129] S452, Virtual planning process:

[0130] The plan reuses historical interaction samples stored in the transfer storage set M for offline iteration, with a maximum of 100 executions per iteration. The specific steps for sample playback are as follows: S4521 to S4524.

[0131] S4521, Traverse all states All actions ;

[0132] S4522. If the transfer unit satisfies the valid identifier Take out the sample Call the matrix element update step of S451 to complete offline training;

[0133] S4523, The cumulative number of executions has reached... When the time comes, terminate the traversal process;

[0134] S4524. After each round of real-world environment interaction, the results obtained from this interaction will be... Store in transfer storage collection and set a valid identifier This provides a sample for subsequent offline planning.

[0135] Further, step S5 includes the following steps:

[0136] S51. Effectiveness Judgment of Rate Control Based on Reinforcement Learning

[0137] When the vehicle's data frame loss rate p is greater than the loss rate threshold If the system determines that the rate control based on reinforcement learning cannot cope with severe packet loss, it will enter the PID control step; otherwise, it will continue to execute the rate control based on reinforcement learning S4.

[0138] S52, Rate Control Based on PID Control

[0139] set up This represents the expected queue length. Let be the queue length in the z-th period, then the queue length deviation in the z-th period is... for:

[0140] (37);

[0141] and The first Period and the Periodic queue length deviation, incremental transmission rate of PID control output. for:

[0142] (38);

[0143] in, and These are the proportional coefficient, integral coefficient, and differential coefficient, respectively.

[0144] In the z-th cycle, the application layer transmission rate of the PID controller output for:

[0145] (39);

[0146] in, It is the first The periodic transmission rate;

[0147] S53, Rate Change Detection of PID Controller

[0148] If the Sending rate of a periodic PID controller Less than the Sending rate of a periodic PID controller This indicates that if the PID controller is implementing a speed reduction strategy, then the S54 buffer scheduling and transmission strategy will be executed; otherwise, the S51 strategy will be executed.

[0149] S54, cache scheduling and transmission;

[0150] S541, Vehicles Continue sending user parameter D, pause sending program components. To adapt to the deceleration strategy of the PID controller;

[0151] S542. Based on the designed broadcast frame structure, the base station constructs broadcast frame B1 and fills the auxiliary transmission threshold into the "auxiliary threshold" field of broadcast frame B1. Enter the program component number in the "Program Component Number" field of broadcast frame B1. number In broadcast frame B1, a 0 is entered in the "successful transmission flag" field, indicating that this is a search for a buffered sender component. The base station sends the broadcast frame B1, and the vehicle that receives the broadcast frame B1... If its own data frame loss rate p is less than the auxiliary transmission threshold And the cache contains program components Then the vehicle Send response frame immediately Response frame Enter "1111" in the "Type" field to indicate that the response frame Used for buffering auxiliary transmission, vehicles Assisted transmission components In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The MAC address; if a message destined for a vehicle is intercepted. response frames The remaining vehicles will no longer send cache program components. If none of the vehicles have cached components Then the vehicle Within the specified time If the vehicle still does not receive a response frame with "1111" entered in the "Type" field, After the user parameter D has been transmitted, restore the transmission program component. ;

[0152] S543, When the vehicle Successfully transmitted program components to the base station Afterwards, the vehicle Construct broadcast frame B2, and fill in the "Program Component Number" field in broadcast frame B2. In broadcast frame B2, a 1 is entered in the "successful transmission flag" to indicate that the program component was successfully sent. By sending the broadcast frame B2, a notification is sent to surrounding vehicles. "Component transmission successful," after receiving broadcast frame B2, vehicle V j Program component number awaiting transmission ,vehicle Continue executing S53.

[0153] Meanwhile, this invention proposes a vehicle-to-everything (V2X) edge buffer transmission system based on reinforcement learning and PID control. The system, applying the method described in this invention, includes the following steps:

[0154] The detection module is configured to perform the following processes: send buffer queue length detection, throughput detection;

[0155] Mathematical modeling module: configured to perform the following process: mathematical modeling of the latency of buffered transmission;

[0156] The transmission module is configured to perform the following processes: perform rate control based on dynamic double-Q learning reinforcement learning, and perform rate control and buffer scheduling based on proportional-integral-derivative (PID).

[0157] Meanwhile, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the steps of the method described in the present invention.

[0158] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method described in the present invention when invoked by a processor.

[0159] Finally, the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the present invention.

[0160] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0161] (1) Existing vehicle-to-everything (V2X) buffer transmission algorithms treat the wireless channel as an ideal communication medium with no data loss, without considering the negative impact of external wireless interference and shielding on communication, which leads to increased transmission delay. This invention mathematically models the data frame transmission delay and derives the delay expression for data frame transmission. The mathematical model shows that the queuing delay mainly depends on the data frame loss rate p and the application layer transmission rate rate. Therefore, it is necessary to adjust the application layer transmission rate rate to effectively control the delay index.

[0162] (2) This invention designs and implements a vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control. For mild to moderate packet loss caused by wireless interference, the invention adaptively adjusts the task transmission rate by detecting interference packet loss in the wireless channel and the data frame queue length in the transmission buffer using a reinforcement learning mechanism. For severe packet loss, when the data frame loss rate exceeds the packet loss threshold, the invention dynamically adjusts the application layer rate using classic PID control, and implements buffered scheduling transmission through designed ACK confirmation frames and broadcast frames. Ultimately, this invention can effectively reduce end-to-end latency of data frames and maintain high throughput. Attached Figure Description

[0163] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0164] Figure 1 The response frame structure diagram is designed for this invention.

[0165] Figure 2 This is a diagram of the broadcast frame structure designed for this invention.

[0166] Figure 3 This is a comparison chart of the throughput of the JACTOP algorithm and the edge caching transmission scheme in a scenario with mild packet loss.

[0167] Figure 4 This is a comparison chart of end-to-end latency between the JACTOP algorithm and the edge buffer transmission scheme in a scenario with mild packet loss.

[0168] Figure 5 This is a comparison chart of the throughput of the JACTOP algorithm and the edge caching transmission scheme under moderate packet loss scenarios.

[0169] Figure 6 This is a comparison chart of the end-to-end latency of the JACTOP algorithm and the edge buffer transmission scheme under moderate packet loss scenarios.

[0170] Figure 7 This is a comparison chart of the throughput of the JACTOP algorithm and the edge caching transmission scheme in a scenario with heavy packet loss.

[0171] Figure 8 This is a comparison chart of the end-to-end latency of the JACTOP algorithm and the edge buffer transmission scheme under heavy packet loss scenarios.

[0172] Figure 9 This is a flowchart of the edge buffer transmission method for vehicle-to-everything (V2X) networks based on reinforcement learning and PID control, according to the present invention.

[0173] Figure 10The flowchart illustrates the execution of proportional-integral-derivative (PID) rate control and cache scheduling based on the present invention. Detailed Implementation

[0174] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0175] Example 1

[0176] See Figure 1 and Figure 7 The present invention provides a technical solution for a vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control, comprising the following steps:

[0177] S1: Delay modeling and transmit buffer queue length detection for vehicle-to-everything (V2X) cache transmission;

[0178] S2: Detects the data frame loss rate of wireless communication between the vehicle and the base station;

[0179] S3: Detection throughput;

[0180] S4: Perform rate control based on dynamic double-Q reinforcement learning;

[0181] S5: Performs rate control and buffer scheduling based on proportional-integral-derivative PID control.

[0182] Furthermore, step S1 includes the following steps:

[0183] S11, Delay Modeling of Vehicle Network Buffer Transmission

[0184] Since wireless communication systems have a retransmission mechanism, let the maximum number of retransmissions be m. The source node will attempt to send the data frame frame at most m+1 times. If the source node does not receive an ACK confirmation frame after the data frame frame has been sent m+1 times, the source node will abandon the transmission and discard the data frame frame. Let the data frame loss rate be p. Then the probability of a single successful transmission of the data frame frame is 1-p, and the failure probability is p. Let K be the total number of transmission attempts for the data frame frame, and each transmission is independent of the others. The probability of success on the k-th transmission attempt is... for:

[0185] (1);

[0186] in, ;

[0187] Assuming all m+1 transmissions fail, the data frame is discarded by the source node. The probability of the data frame being discarded is... for:

[0188] (2);

[0189] Substituting formula (1) into formula (2), we get:

[0190] (3);

[0191] Let n be a non-negative integer variable, and let the subtrahend term in formula (3) be:

[0192] (4);

[0193] According to the formula for the sum of a geometric series, we have:

[0194] (5);

[0195] Based on formulas (4) and (5), formula (3) is:

[0196] (6);

[0197] Let the average number of transmission attempts for a data frame be... ,have:

[0198]

[0199] = (7);

[0200] Based on the formula for infinite series and the finite term truncation, the second term on the right side of formula (7) is:

[0201] (8);

[0202] Then the average number of sending attempts for:

[0203] (9);

[0204] Simplify formula (9) to calculate the average number of sending attempts. for:

[0205] (10);

[0206] Data sent from the application layer of the source node is encapsulated into data frames at the Media Access Control (MAC) layer and then transmitted to the wireless channel via the wireless network card. External interference or shielding on the wireless channel can cause data frame loss, preventing the destination node from sending an ACK confirmation frame and triggering the source node to initiate a retransmission mechanism. Let the length of each data frame be L (in bits), the application layer's transmission rate be rate (in megabits per second, Mb / s), and the duration of a timeslot be Duration (in seconds). Within one Duration, the total number of data frames received by the MAC layer is:

[0207] (11);

[0208] Let the vehicle's wireless communication bandwidth be B, measured in megabits per second (Mb / s). Within the duration of one time slot (Duration), the total number of data frames (Output) sent by the Media Access Control (MAC) layer is:

[0209] (12);

[0210] Considering the reception and transmission of data at the Media Access Control (MAC) layer, within the duration of one time slot (Duration), the queue length q of data frames in the vehicle's transmission buffer is:

[0211] (13);

[0212] The queuing delay occurs after a data frame arrives at the transmit buffer. for:

[0213] ;

[0214] (14);

[0215] Substitute formula (10) into formula (14). for:

[0216] (15);

[0217] When the time slot duration, vehicle wireless communication bandwidth B, maximum retransmission count m, and data frame length L are fixed values, the queuing delay depends on the data frame loss rate p and the application layer transmission rate rate.

[0218] When the queue ends, the data frame transmission delay... for:

[0219] (16);

[0220] The end-to-end delay of the data frame is:

[0221] (17);

[0222] S12. Send buffer queue length detection

[0223] When the transmit buffer of the Media Access Control (MAC) layer receives a newly arrived data frame, the receive function is triggered. Read the current queue length curq_, the pseudocode is as follows:

[0224] ;

[0225] .

[0226] Further, S2 includes:

[0227] S2 includes:

[0228] In the t-th sampling period, the vehicle The total number of data frames sent is recorded. The base station counts the number of data frames received. Design a response frame such that the base station constructs a total of Y response frames within the (t+1)th sampling period, and sends the y-th response frame. Enter "0001" in the "Type" field for positive integers. , representing a response frame Its function is to statistically analyze the frame loss rate in response frames. In the "Number of Received Frames" field, enter the number of received data frames obtained by the base station in the t-th period. In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The base station sends Y response frames sequentially to the MAC address of the vehicle. Receive any response frame within the (t+1)th sampling period That is, to know the number of data frames received by the base station in the t-th period. Then the data frame loss rate p is:

[0229] (18).

[0230] Further, S3 includes:

[0231] S3 includes:

[0232] Based on the statistics in S2, the throughput Thr between the vehicle and the base station in the t-th period is calculated as follows:

[0233] (19).

[0234] Further, S4 includes:

[0235] S41, State Space Design

[0236] In terms of observation dimensions, a two-dimensional environment observation vector is adopted, which includes the queue length q of the vehicle transmission buffer and the data frame loss rate p. Through normalization, boundary clipping, and binning operations, the two-dimensional environment observation vector is transformed into a one-dimensional discrete state index, and operators are defined. For floor operations, the fractional part of the floating-point binned value is truncated to obtain the integer index. The queue length hyperparameter is used. The total number of bins is the queue length, and the loss rate hyperparameter is... To define the total number of bins for the data frame loss rate, and to ensure that the index falls within a valid value range, a boundary pruning function is defined. for:

[0237] (20);

[0238] in, Enter the values ​​to be cropped;

[0239] S411, Queue Length Normalization and Binning

[0240] Perform queue length normalization, mapping the queue length index to the value range [0,1]. Set the maximum queue length. Normalized queue length for:

[0241] (twenty one);

[0242] Perform binning operation to obtain the queue length index. for:

[0243] (twenty two);

[0244] S412, Numerical clamping and binning of data frame loss rate p

[0245] Perform numerical clamping, and set the loss rate threshold as follows: Define the clipping loss probability. for:

[0246] (twenty three);

[0247] Perform binning operations to obtain the data frame loss rate index. :

[0248] (twenty four);

[0249] S413, Index Concatenation

[0250] The two-dimensional discrete index is mapped to a discrete state s using the following dimension-weighted concatenation formula:

[0251] (25);

[0252] The range of the number of discrete states s is as follows: ,in The upper limit of discrete states;

[0253] S42, Motionspace Design

[0254] A low-dimensional discrete action space is used to reduce the overhead of table lookup calculations, and three types of rate actions are set. These correspond to three different transmission rate adjustment amounts. Discrete action value range Define the absolute value of the transmission rate adjustment amount as ,action This represents reducing the transmission rate; the rate adjustment amount. ,action This represents maintaining the current sending rate, with the rate adjustment amount. =0, action This represents increasing the sending rate; the rate adjustment amount. for After each strategy iteration completes the action selection, the sending rate is set based on the action offset. To avoid network overload or zero throughput caused by unlimited increases or decreases in the sending rate, the following rules apply to the sending rate: If the current sending rate is less than the lower limit... Then the updated sending rate will be set to... If the current sending rate is greater than the rate limit Then the updated sending rate will be set to... ;

[0255] Let the current transmission rate be... After executing action a, the intermediate process rate is obtained. for:

[0256] (26);

[0257] For intermediate process rate Apply boundary constraints and update the transmission rate. for:

[0258] (27);

[0259] S43. Reward Function Design

[0260] Using statistical throughput as the positive reward, and introducing queues and packet loss as penalties, a reward function is constructed. The reward is a linear weighted sum of throughput gain and penalty. At the same time, upper and lower boundary constraints are set to suppress reward explosion under extreme network conditions and ensure stable iteration.

[0261] S431, Calculate Rewards

[0262] Throughput calculated according to formula (19) The data frame loss rate is calculated using formula (18). The queue length q of the send buffer obtained in S12, and the reward. The calculation formula is:

[0263] (28);

[0264] in, The reward weight for throughput, The reward weight of the queue, The reward weight for the data frame loss rate;

[0265] S432, Reward Truncation Constraint

[0266] Set the maximum threshold for rewards to The minimum threshold for rewards is To prevent extreme observations from causing a reward explosion, the following measures are taken:

[0267] (29);

[0268] S44, Action Selection Strategy

[0269] Given the current discrete state To balance environmental exploration and strategy utilization, an exploration probability of is adopted. Greedy strategy output action ;

[0270] Set up two independent "state-action" value matrices Value Matrix With value matrix The elements in the scalar are denoted as scalars. , representing the estimated long-term cumulative return corresponding to action a in discrete state s, the value matrix. With value matrix Initialize all values ​​to 0 and iterate alternately to mitigate overestimation bias;

[0271] Define the transfer storage set Any "state-action" pair corresponds to a transition unit ,in This represents the next discrete state obtained after the environment transition following the execution of action a; To effectively identify Boolean values, This represents a "state-action" pair with historical interaction samples, used for offline planning;

[0272] S441, Generation Interval Uniform random number variables within ;

[0273] S442, Exploration Branch

[0274] like Randomly select actions ;

[0275] S443, Utilizing Branches

[0276] Calculate the mean of the two Q values ​​for each action. Then select the action with the largest mean:

[0277] (30);

[0278] S45, Algorithm Iterative Calculation

[0279] S451, Matrix Element Update

[0280] matrix and The elements are updated alternately to decouple the selection of the optimal action from the evaluation of the target value. The input parameters are the current state s, the action to be performed a, the reward, and the next state. Next action ;

[0281] S4511, Value Matrix Update elements;

[0282] S45111, in In China, a value matrix is ​​used. Select the optimal action ;

[0283] (31);

[0284] S45112, Target Value From the value matrix The corresponding action value calculation includes:

[0285] (32);

[0286] in, Discount factor;

[0287] S45113, Update ;

[0288] (33);

[0289] in, The learning rate;

[0290] S4512, Value Matrix Update elements

[0291] S45121, in In China, a value matrix is ​​used. Select the optimal action :

[0292] (34);

[0293] S45122, Target Value From the value matrix The corresponding action value calculation includes:

[0294] (35);

[0295] S45123, Update :

[0296] (36);

[0297] S452, Virtual planning process:

[0298] The plan reuses historical interaction samples stored in the transfer storage set M for offline iteration, with a maximum of 100 executions per iteration. The specific steps for sample playback are as follows: S4521 to S4524.

[0299] S4521, Traverse all states All actions ;

[0300] S4522. If the transfer unit satisfies the valid identifier Take out the sample Call the matrix element update step of S451 to complete offline training;

[0301] S4523, The cumulative number of executions has reached... When the time comes, terminate the traversal process;

[0302] S4524. After each round of real-world environment interaction, the results obtained from this interaction will be... Store in transfer storage collection and set a valid identifier This provides a sample for subsequent offline planning.

[0303] Further, step S5 includes the following steps:

[0304] S51. Effectiveness Judgment of Rate Control Based on Reinforcement Learning

[0305] When the vehicle's data frame loss rate p is greater than the loss rate threshold If the system determines that the rate control based on reinforcement learning cannot cope with severe packet loss, it will enter the PID control step; otherwise, it will continue to execute the rate control based on reinforcement learning S4.

[0306] S52, Rate Control Based on PID Control

[0307] set up This represents the expected queue length. Let be the queue length in the z-th period, then the queue length deviation in the z-th period is... for:

[0308] (37);

[0309] and The first Period and the Periodic queue length deviation, incremental transmission rate of PID control output. for:

[0310] (38);

[0311] in, and These are the proportional coefficient, integral coefficient, and differential coefficient, respectively.

[0312] In the z-th cycle, the application layer transmission rate of the PID controller output for:

[0313] (39);

[0314] in, It is the first The periodic transmission rate;

[0315] S53, Rate Change Detection of PID Controller

[0316] If the Sending rate of a periodic PID controller Less than the Sending rate of a periodic PID controller This indicates that if the PID controller is implementing a speed reduction strategy, then the S54 buffer scheduling and transmission strategy will be executed; otherwise, the S51 strategy will be executed.

[0317] S54, cache scheduling and transmission;

[0318] S541, Vehicles Continue sending user parameter D, pause sending program components. To adapt to the deceleration strategy of the PID controller;

[0319] S542. Based on the designed broadcast frame structure, the base station constructs broadcast frame B1 and fills the auxiliary transmission threshold into the "auxiliary threshold" field of broadcast frame B1. Enter the program component number in the "Program Component Number" field of broadcast frame B1. number In broadcast frame B1, a 0 is entered in the "successful transmission flag" field, indicating that this is a search for a buffered sender component. The base station sends the broadcast frame B1, and the vehicle that receives the broadcast frame B1... If its own data frame loss rate p is less than the auxiliary transmission threshold And the cache contains program components Then the vehicle Send response frame immediately Response frame Enter "1111" in the "Type" field to indicate that the response frame Used for buffering auxiliary transmission, vehicles Assisted transmission components In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The MAC address; if a message destined for a vehicle is intercepted. response frames The remaining vehicles will no longer send cache program components. If none of the vehicles have cached components Then the vehicle Within the specified time If the vehicle still does not receive a response frame with "1111" entered in the "Type" field, After the user parameter D has been transmitted, restore the transmission program component. ;

[0320] S543, When the vehicle Successfully transmitted program components to the base station Afterwards, the vehicle Construct broadcast frame B2, and fill in the "Program Component Number" field in broadcast frame B2. In broadcast frame B2, a 1 is entered in the "successful transmission flag" to indicate that the program component was successfully sent. By sending the broadcast frame B2, a notification is sent to surrounding vehicles. "Component transmission successful," after receiving broadcast frame B2, vehicle V j Program component number awaiting transmission ,vehicle Continue executing S53.

[0321] Example 2: Based on Example 1, the following comparative experiment was conducted: The experimental parameters are as follows: the communication bandwidth of each vehicle is 10MHz; the in-vehicle application consists of 20 components, and the data size of each component is 1Mbits to 5Mbits; the user parameter data size of the vehicle task is 1Mbits to 5Mbits; the maximum number of transmissions of the MAC layer data frame is set to 6, i.e., 1 initial transmission and 5 retransmissions; specific communication parameters are shown in Table 1.

[0322] Table 1 Vehicle-to-Everything (V2X) Communication and Task Parameter Configuration Table

[0323]

[0324] The reinforcement learning algorithm parameters are set as follows: learning rate α is 0.40, discount factor γ is 0.92, greedy exploration probability ε is 0.07, and the number of virtual sample replay steps per RL update is N. plan The queue length is 3, the number of bins after normalization is 12, the number of bins with packet loss rate is 8, and the weight w Thr The weight is 10, and the weight w qlen The value is 1.2, and the weight w loss 38, N a The value is 3, and the sampling period for reinforcement learning is 0.001. The absolute value of the transmission rate adjustment, ΔR, is 5000, and RATE... MIN For 1,000,000, RATE MAX It is 3,000,000. th It is 0.8, p a The value is 0.5. Specific reinforcement learning parameters are shown in Table 2.

[0325] Table 2 Hyperparameter Settings for Reinforcement Learning Algorithms

[0326]

[0327] The experiment sets up mild, moderate, and severe packet loss scenarios, and compares and analyzes the performance of the JACTOP algorithm and the method of this invention (hereinafter referred to as the "edge buffer transmission scheme") in terms of throughput and average end-to-end latency. For detailed performance comparisons, see [link to relevant documentation]. Figures 3 to 8 .

[0328] Figure 3 This study compares the throughput of the JACTOP algorithm and the edge caching transmission scheme in scenarios with mild packet loss. Figure 4 This study compares the end-to-end latency of the JACTOP algorithm and the edge buffer transmission scheme in scenarios with mild packet loss.

[0329] Figure 5 This study compares the throughput of the JACTOP algorithm and the edge caching transmission scheme in a moderate packet loss scenario. Figure 6 This study compares the end-to-end latency of the JACTOP algorithm and the edge caching transmission scheme in a moderate packet loss scenario.

[0330] Figure 7 This study compares the throughput of the JACTOP algorithm and the edge caching transmission scheme in scenarios with severe packet loss. Figure 8 This study compares the end-to-end latency of the JACTOP algorithm and the edge caching transmission scheme in scenarios with severe packet loss.

[0331] Based on the above experiments, the following conclusions can be drawn. Firstly, as packet loss worsens, the throughput of both schemes decreases. This is because increased packet loss in wireless transmission leads to a significant increase in the number of data frame retransmissions, resulting in a continuous decrease in the amount of effective transmitted data. However, under heavy packet loss conditions, the throughput difference between the two schemes significantly decreases, demonstrating the effectiveness of the edge caching transmission scheme in dealing with heavy packet loss. Secondly, as the packet loss rate increases, the latency performance of both schemes increases. However, the latency of the edge caching scheme is lower than that of the JACTOP algorithm, especially in scenarios with mild packet loss, where the latency of the edge caching scheme is effectively controlled, demonstrating the effectiveness of the reinforcement learning algorithm in dealing with mild packet loss.

[0332] In summary, the edge buffering transmission scheme, at the cost of slightly reduced throughput, significantly reduces end-to-end latency; and the lower the network packet loss rate, the more significant the latency optimization effect. As channel packet loss worsens, the throughput performance disadvantage of the edge buffering transmission scheme gradually disappears compared to the JACTOP algorithm, and its end-to-end latency remains lower than that of the JACTOP algorithm. Therefore, the overall transmission performance of the edge buffering transmission scheme is superior to that of the JACTOP algorithm.

[0333] Example 3: This example proposes a vehicle-to-everything (V2X) edge buffer transmission system based on reinforcement learning and PID control, including:

[0334] The detection module is configured to perform the following processes: send buffer queue length detection, throughput detection;

[0335] Mathematical modeling module: configured to perform the following process: mathematical modeling of the latency of buffered transmission;

[0336] The transmission module is configured to perform the following processes: perform rate control based on dynamic double-Q learning reinforcement learning, and perform rate control and buffer scheduling based on proportional-integral-derivative (PID).

[0337] Example 4: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.

[0338] Example 5: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.

[0339] Example 6: This example proposes a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in this invention, which will not be repeated here.

[0340] It should be noted that the processing flow of embodiments 3-6 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.

[0341] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0342] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A vehicular internet of things edge caching and transmission method based on reinforcement learning and PID control, characterized in that, Includes the following steps: S1: Delay modeling and transmit buffer queue length detection for vehicle-to-everything (V2X) cache transmission; S2: Detects the data frame loss rate of wireless communication between the vehicle and the base station; S3: Detection throughput; S4: Perform rate control based on dynamic double-Q reinforcement learning; S5: Performs rate control and buffer scheduling based on proportional-integral-derivative PID control. 2.The vehicular edge caching and transmission method based on reinforcement learning and PID control according to claim 1, wherein, S1 includes the following steps: S11, Delay Modeling of Vehicle Network Buffer Transmission Since wireless communication systems have a retransmission mechanism, let the maximum number of retransmissions be m. The source node will attempt to send the data frame frame at most m+1 times. If the source node does not receive an ACK confirmation frame after the data frame frame has been sent m+1 times, the source node will abandon the transmission and discard the data frame frame. Let the data frame loss rate be p. Then the probability of a single successful transmission of the data frame frame is 1-p, and the failure probability is p. Let K be the total number of transmission attempts for the data frame frame, and each transmission is independent of the others. The probability of success on the k-th transmission attempt is... for: (1); wherein ; Assuming all m+1 transmissions fail, the data frame is discarded by the source node. The probability of the data frame being discarded is... for: (2); Substituting formula (1) into formula (2), we get: (3); Let n be a non-negative integer variable, and let the subtrahend term in formula (3) be: (4); According to the formula for the sum of a geometric series, we have: (5); Based on formulas (4) and (5), formula (3) is: (6); Let the average number of transmission attempts for a data frame frame be There are: (7); Based on the formula for infinite series and the finite term truncation, the second term on the right side of formula (7) is: (8); the average number of transmission attempts is: (9); Simplify formula (9) to calculate the average number of sending attempts. for: (10); Data sent from the application layer of the source node is encapsulated into data frames at the Media Access Control (MAC) layer and then transmitted to the wireless channel via the wireless network card. External interference or shielding on the wireless channel can cause data frame loss, preventing the destination node from sending an ACK confirmation frame and triggering the source node to initiate a retransmission mechanism. Let the length of each data frame be L (in bits), the application layer's transmission rate be rate (in megabits per second, Mb / s), and the duration of a timeslot be Duration (in seconds). Within one Duration, the total number of data frames received by the MAC layer is: (11); Let the vehicle's wireless communication bandwidth be B, measured in megabits per second (Mb / s). Within one time slot's duration (Duration), the total number of data frames (Output) sent by the Media Access Control (MAC) layer is: (12); Considering the reception and transmission of data at the Media Access Control (MAC) layer, within the duration of one time slot (Duration), the queue length q of data frames in the vehicle's transmission buffer is: (13); The queuing delay of waiting for the data frame frame to arrive at the sending buffer is: ; (14); Substitute equation (10) into equation (14), is: (15); When the time slot duration, vehicle wireless communication bandwidth B, maximum retransmission count m, and data frame length L are fixed values, the queuing delay depends on the data frame loss rate p and the application layer transmission rate rate. When the queue is finished, the transmission of the data frame frame is delayed is: (16); The end-to-end delay of the data frame is: (17); S12. Send buffer queue length detection When the transmit buffer of the Media Access Control (MAC) layer receives a newly arrived data frame, the receive function is triggered. Read the current queue length curq_, the pseudocode is as follows: ; 。 3. The edge buffer transmission method for vehicle-to-everything (V2X) networks based on reinforcement learning and PID control according to claim 1, characterized in that, S2 includes: In the t-th sampling period, the vehicle The total number of data frames sent is recorded. The base station counts the number of data frames received. Design a response frame such that the base station constructs a total of Y response frames within the (t+1)th sampling period, and sends the y-th response frame. Enter "0001" in the "Type" field for positive integers. , representing a response frame Its function is to statistically analyze the frame loss rate in response frames. In the "Number of Received Frames" field, enter the number of received data frames obtained by the base station in the t-th period. In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The base station sends Y response frames sequentially to the MAC address of the vehicle. Receive any response frame within the (t+1)th sampling period That is, to know the number of data frames received by the base station in the t-th period. Then the data frame loss rate p is: (18)。 4. The vehicle-to-everything (V2X) edge buffer transmission method based on reinforcement learning and PID control according to claim 1, characterized in that, S3 includes: Based on the statistics in S2, the throughput Thr between the vehicle and the base station in the t-th period is calculated as follows: (19)。 5. The edge buffer transmission method for vehicle-to-everything (V2X) networks based on reinforcement learning and PID control according to claim 1, characterized in that, S4 includes: S41, State Space Design In terms of observation dimensions, a two-dimensional environment observation vector is adopted, which includes the queue length q of the vehicle transmission buffer and the data frame loss rate p. Through normalization, boundary clipping, and binning operations, the two-dimensional environment observation vector is transformed into a one-dimensional discrete state index, and operators are defined. For floor operations, the fractional part of the floating-point binned value is truncated to obtain the integer index. The queue length hyperparameter is used. The total number of bins is the queue length, and the loss rate hyperparameter is... To define the total number of bins for the data frame loss rate, and to ensure that the index falls within a valid value range, a boundary pruning function is defined. for: (20); in, Enter the values ​​to be cropped; S411, Queue Length Normalization and Binning Perform queue length normalization, mapping the queue length index to the value range [0,1]. Set the maximum queue length. Normalized queue length for: (21); Perform binning operation to obtain the queue length index. for: (22); S412, Numerical clamping and binning of data frame loss rate p Perform numerical clamping, and set the loss rate threshold as follows: Define the clipping loss probability. for: (23); Perform binning operations to obtain the data frame loss rate index. : (24); S413, Index Concatenation The two-dimensional discrete index is mapped to a discrete state s using the following dimension-weighted concatenation formula: (25); The range of the number of discrete states s is: ,in The upper limit of discrete states; S42, Motionspace Design A low-dimensional discrete action space is used to reduce the overhead of table lookup calculations, and three types of rate actions are set. These correspond to three different transmission rate adjustment amounts. Discrete action value range Define the absolute value of the transmission rate adjustment amount as ,action This represents reducing the transmission rate; the rate adjustment amount. ,action This represents maintaining the current sending rate, with the rate adjustment amount. =0, action This represents increasing the sending rate; the rate adjustment amount. for After each strategy iteration completes the action selection, the sending rate is set based on the action offset. To avoid network overload or zero throughput caused by unlimited increases or decreases in the sending rate, the following rules apply to the sending rate: If the current sending rate is less than the lower limit... Then the updated sending rate will be set to... If the current sending rate is greater than the rate limit Then the updated sending rate will be set to... ; Let the current transmission rate be... After executing action a, the intermediate process rate is obtained. for: (26); For intermediate process rate Apply boundary constraints and update the transmission rate. for: (27); S43. Reward Function Design Using statistical throughput as the positive reward, and introducing queues and packet loss as penalties, a reward function is constructed. The reward is a linear weighted sum of throughput gain and penalty. At the same time, upper and lower boundary constraints are set to suppress reward explosion under extreme network conditions and ensure stable iteration. S431, Calculate Rewards Throughput calculated according to formula (19) The data frame loss rate is calculated using formula (18). The queue length q of the send buffer obtained in S12, and the reward. The calculation formula is: (28); in, The reward weight for throughput, The reward weight of the queue, The reward weight for the data frame loss rate; S432, Reward Truncation Constraint Set the maximum threshold for rewards to The minimum threshold for rewards is To prevent extreme observations from causing a reward explosion, the following measures are taken: (29); S44, Action Selection Strategy Given the current discrete state To balance environmental exploration and strategy utilization, an exploration probability of is adopted. Greedy strategy output action ; Set up two independent "state-action" value matrices Value Matrix With value matrix The elements in the scalar are denoted as scalars. , representing the estimated long-term cumulative return corresponding to action a in discrete state s, the value matrix. With value matrix Initialize all values ​​to 0 and iterate alternately to mitigate overestimation bias; Define the transfer storage set Any "state-action" pair corresponds to a transition unit ,in This represents the next discrete state obtained after the environment transition following the execution of action a; To effectively identify Boolean values, This represents a "state-action" pair with historical interaction samples, used for offline planning; S441, Generation Interval Uniform random number variables within ; S442, Exploration Branch like Randomly select actions ; S443, Utilizing Branches Calculate the mean of the two Q values ​​for each action. Then select the action with the largest mean: (30); S45, Algorithm Iterative Calculation S451, Matrix Element Update matrix and The elements are updated alternately to decouple the selection of the optimal action from the evaluation of the target value. The input parameters are the current state s, the action to be performed a, the reward, and the next state. Next action ; S4511, Value Matrix Update elements; S45111, in In China, a value matrix is ​​used. Select the optimal action ; (31); S45112, Target Value From the value matrix The corresponding action value calculation includes: (32); in, Discount factor; S45113, Update ; (33); in, The learning rate; S4512, Value Matrix Update elements S45121, in In China, a value matrix is ​​used. Select the optimal action : (34); S45122, Target Value From the value matrix The corresponding action value calculation includes: (35); S45123, Update : (36); S452, Virtual planning process: The plan reuses historical interaction samples stored in the transfer storage set M for offline iteration, with a maximum of 100 executions per iteration. The specific steps for sample playback are as follows: S4521 to S4524. S4521, Traverse all states All actions ; S4522. If the transfer unit satisfies the valid identifier Take out the sample Call the matrix element update step of S451 to complete offline training; S4523, The cumulative number of executions has reached... When the time is right, terminate the traversal process; S4524. After each round of real-world environment interaction, the results obtained from this interaction will be... Store in transfer storage collection and set a valid identifier This provides a sample for subsequent offline planning.

6. The edge buffer transmission method for vehicle-to-everything (V2X) networks based on reinforcement learning and PID control according to claim 1, characterized in that, S5 includes the following steps: S51. Effectiveness Judgment of Rate Control Based on Reinforcement Learning When the vehicle's data frame loss rate p is greater than the loss rate threshold If the system determines that the rate control based on reinforcement learning cannot cope with severe packet loss, it will enter the PID control step; otherwise, it will continue to execute the rate control based on reinforcement learning S4. S52, Rate Control Based on PID Control set up This represents the expected queue length. Let be the queue length in the z-th period, then the queue length deviation in the z-th period is... for: (37); and The first Period and the Periodic queue length deviation, incremental transmission rate of PID control output. for: (38); in, and These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. In the z-th cycle, the application layer transmission rate of the PID controller output for: (39); in, It is the first The periodic transmission rate; S53, Rate Change Detection of PID Controller If the Sending rate of a periodic PID controller Less than the Sending rate of a periodic PID controller If the PID controller is executing a speed-reducing strategy, then the S54 buffer scheduling and transmission strategy will be executed; otherwise, the S51 strategy will be executed. S54, cache scheduling and transmission; S541, Vehicles Continue sending user parameter D, pause sending program components. To adapt to the deceleration strategy of the PID controller; S542. Based on the designed broadcast frame structure, the base station constructs broadcast frame B1 and fills the auxiliary transmission threshold into the "auxiliary threshold" field of broadcast frame B1. Enter the program component number in the "Program Component Number" field of broadcast frame B1. number In broadcast frame B1, a 0 is entered in the "Successful Transmission Flag" field, indicating that this is a search for a buffered sender component. The base station sends the broadcast frame B1, and the vehicle that receives the broadcast frame B1... If its own data frame loss rate p is less than the auxiliary transmission threshold And the cache contains program components Then the vehicle Send response frame immediately Response frame Enter "1111" in the "Type" field to indicate that the response frame Used for buffering auxiliary transmission, vehicles Assisted transmission components In the response frame In the "Destination MAC Address" field, fill in the vehicle's... The MAC address; if a message destined for a vehicle is intercepted. response frames The remaining vehicles will no longer send cache program components. If none of the vehicles have cached components Then the vehicle Within the specified time If the vehicle still does not receive a response frame with "1111" entered in the "Type" field, After the user parameter D has been transmitted, restore the transmission program component. ; S543, When the vehicle Successfully transmitted program components to the base station Afterwards, the vehicle Construct broadcast frame B2, and fill in the "Program Component Number" field in broadcast frame B2. In broadcast frame B2, a 1 is entered in the "successful transmission flag" to indicate that the program component was successfully sent. By sending the broadcast frame B2, a notification is sent to surrounding vehicles. "Component transmission successful," after receiving broadcast frame B2, vehicle V j Program component number awaiting transmission ,vehicle Continue executing S53.

7. A vehicle-to-everything (V2X) edge buffer transmission system based on reinforcement learning and PID control, characterized in that, The system comprising the steps of applying the method according to any one of claims 1 to 6, wherein the system includes: The detection module performs send buffer queue length detection and throughput detection. Mathematical modeling module: configured to perform the following process: mathematical modeling of the latency of buffered transmission; The transmission module is configured to perform the following process: perform rate control based on dynamic double-Q learning reinforcement learning; The rate control module and the buffer scheduling module are configured to perform the following process: perform rate control and buffer scheduling based on proportional-integral-derivative PID control.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is configured to implement the steps of the method according to any one of claims 1 to 6 when invoked by a processor.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.