Wireless transmission congestion control method and system based on DDPG model

Abstract

The invention discloses a wireless transmission congestion control method based on a DDPG model, and the method comprises the following steps: obtaining a congestion window value and an acknowledgement data packet in a bottom layer TCP protocol stack in each round-trip delay period, and generating network statistical information; preprocessing the network statistical information by using a convolution matrix, and generating a network state vector in combination with a recursive structure; generating a reward value corresponding to the network state vector through a reward function; based on the network state vector and the reward value, adopting a DDPG model to output a maximum value of a congestion window; and determining the size of the congestion window by adopting a maximum filter according to the maximum value of the congestion window, and feeding back the size of the congestion window to a bottom TCP protocol stack. According to the method, a deep reinforcement learning mechanism and dynamic adjustment of the congestion window are combined through the DDPG model, the problem of performance limitation of a traditional control method in a complex and changeable wireless environment is solved, and the real-time performance of wireless transmission is improved on the premise that high throughput is not lost.

Description

Technical Field

[0001] The present invention relates to the field of wireless transmission congestion control, and more particularly to a wireless transmission congestion control method and system based on a DDPG model. Background Art

[0002] With the rapid advancement of information technology, wireless communication has become an indispensable part of modern life. It not only supports the data transmission needs of mobile devices such as smartphones and tablets, but is also one of the key infrastructures for Internet of Things (IoT) applications. Mobile devices and Internet of Things (IoT) applications have put forward higher requirements for network transmission efficiency and quality. In addition, wireless networks have the characteristics of high bit error rate, highly dynamic network topology, large fluctuations in link bandwidth and delay, and high random packet loss rate, which can easily lead to insufficient or excessive utilization of link resources and even cause network congestion. Therefore, how to dynamically adjust the congestion control strategy to adapt to the highly changing wireless network link environment is the key to achieving efficient transmission.

[0003] Currently, the traditional TCP congestion control algorithm performs congestion control by maintaining a congestion window (cwnd). Packet loss-based algorithms, such as Cubic, cannot effectively distinguish between congestion-induced packet loss and packet loss caused by random errors, resulting in frequent window reduction and performance degradation. Delay-based algorithms, such as Vegas, are difficult to accurately measure link congestion in wireless networks with large delay fluctuations. The above traditional congestion control algorithms all use fixed rules to adjust the congestion window, and have good convergence, but in a highly dynamic link environment, it is difficult to determine the optimal congestion window, resulting in problems such as reduced throughput and increased transmission delay, affecting the efficiency and real-time performance of service transmission.

[0004] At the same time, the congestion control algorithm based on machine learning can dynamically adjust the congestion control strategy according to the real-time network status, and has better adaptability to the changing network environment. For example, Remy adopts a Monte Carlo-based strategy to generate a mapping from state to action through offline pre-training, but it relies on prior assumptions about the network. Aurora uses deep reinforcement learning technology to extract network features in complex link environments, learn from past network status and experience, and explore behaviors that make rewards higher to adjust the congestion window size. However, the machine learning-based method has problems such as slow learning speed and difficult convergence, and it is difficult to adapt to the dynamically changing wireless environment.

[0005] In addition, existing congestion control algorithms focus on improving throughput, but in practical applications, especially in delay-sensitive service scenarios such as video streaming and online games, low latency is also crucial. They often cannot meet these two goals at the same time.

[0006] Therefore, how to design a wireless transmission congestion control method based on the DDPG model to effectively solve the performance limitations of the existing TCP congestion control algorithm in complex and changeable wireless environments is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0007] In view of this, the present invention provides a wireless transmission congestion control method based on the DDPG model, which combines deep reinforcement learning technology with traditional congestion control algorithms, obtains and processes network statistical information in real time, and uses convolution matrices and recursive structures to generate feature-rich network state vectors. In combination with the designed reward function and deep reinforcement learning mechanism, adaptive congestion control of dynamic changes in wireless networks is achieved.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a wireless transmission congestion control method based on a DDPG model, comprising the following steps:

[0010] S1. In each round-trip delay period, obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack to generate network statistics;

[0011] S2. Preprocessing the network statistical information using a convolution matrix and generating a network state vector in combination with a recursive structure;

[0012] S3. Generate a reward value corresponding to the network state vector through a reward function;

[0013] S4. Based on the network state vector and the reward value, a DDPG model is used to output the maximum value of the congestion window; the DDPG model includes: a policy network and a value network;

[0014] S5. According to the maximum value of the congestion window, a maximum filter is used to determine the size of the congestion window and the result is fed back to the underlying TCP protocol stack.

[0015] Preferably, the S1 comprises:

[0016] S11, generating data packet statistics based on the confirmed data packet;

[0017] S12, using the data packet statistical information to calculate the average data packet delay in each round trip delay period, the average throughput in each round trip delay period, and the number of data packets with the average data packet delay;

[0018] S13. The average data packet delay, average throughput and number of data packets are combined with the congestion window value in the underlying TCP protocol stack as network statistical information.

[0019] Preferably, in S2, preprocessing the network statistical information using a convolution matrix includes:

[0020] S21. Define the convolution matrix:

[0021]

[0022] Where, d is the average packet delay, d 0 is the target delay;

[0023] S22, using the convolution matrix to perform filtering processing, and the average packet delay d and the target delay d 0 To encode:

[0024]

[0025] Where p is the average throughput and n is the number of packets;

[0026] S23. Based on the filtered and encoded data, obtain the observation statistical information at time t:

[0027]

[0028] Among them, cwnd t It is the congestion window value in the underlying TCP protocol stack.

[0029] Preferably, in S2, the network state vector is expressed as:

[0030] s t =[o t ,o t-1 ,…,o t-m+1 ]

[0031] Among them, t-1 is the observation statistics at time t-1, o t-m+1 is the observation statistics at time t-m+1, and m is the length of the network state vector.

[0032] Preferably, in S3, the reward function is expressed as:

[0033]

[0034] Where ω(n,d) is the moving average of the two most recent values ​​of d:

[0035]

[0036] n pre ,d pre are the values ​​of n and d at the previous moment respectively.

[0037] Preferably, S4 includes:

[0038] S41, initializing time step and related network parameters;

[0039] S42, through the policy network π θ , determine action a t And the maximum value of the congestion window cwnd at time t max ;

[0040] The action a t It is expressed as:

[0041] a t =π θ (s t )+N t

[0042]

[0043] Among them, π θ (s t ) is the policy function, N t is the noise term, a t The value range is [-1,1];

[0044] S43, execute action a t , get the reward value r t and the new state s t+1 ; Combined with the value network Qω, define the Q value function Q π (s t ,a t )=E[R t |s t ,a t ], for the action a t Evaluation; among them, R t To accumulate expected returns;

[0045] S44, combined with the experience replay mechanism, the experience tuple (s t ,a t ,r t ,s t+1 ), and randomly select N experience tuples (s i ,a i ,r i ,s i+1 );

[0046] S45, based on the experience tuple (s i ,a i ,r i ,s i+1 ) Update network parameters;

[0047] S46, repeat the above S42 to S45 until the output satisfies the preset condition and outputs the maximum congestion window cwnd at the corresponding time max .

[0048] Preferably, in S43, the cumulative expected return R t It is expressed as:

[0049] R t =r t +γ 1 r t+1 +γ 2 r t+2 +…+γ n r t+n

[0050] Among them, γ 1 , γ 2 …γ n is the discount factor, and its value range is [0,1].

[0051] Preferably, the S45 includes:

[0052] S451. Define the optimal strategy π θ The expected return J(θ) under:

[0053]

[0054] Use the policy gradient ascent strategy to improve the policy network π θ Perform parameter update, and the gradient of the expected return J(θ) is expressed as:

[0055]

[0056] Among them, η θ is the learning rate of the policy network;

[0057] S452, update the parameters of the value network Qω by minimizing the mean square error between the target Q value and the current Q value:

[0058]

[0059]

[0060] Among them, N represents the number of data, y i represents the target Q value, y i =r i +γQ ω' (s i+1 ,π θ' (s i+1 )), π θ' and Q ω'They are target strategy network and target value network respectively;

[0061] S453, updating the network parameters of the target strategy network and the target value network:

[0062] θ′←τθ+(1-τ)θ′

[0063] ω′←τω+(1-τ)ω′

[0064] Among them, τ is the speed control parameter.

[0065] Preferably, S5 includes:

[0066] The maximum value of the congestion window is compared with the congestion window value calculated by the underlying TCP protocol stack, and the smaller congestion window value is selected and fed back to the underlying TCP protocol stack.

[0067] In a second aspect, the present invention provides a wireless transmission congestion control system based on a DDPG model, comprising:

[0068] Monitoring module: used to obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack in each round-trip delay period and generate network statistics;

[0069] State generation module: used to pre-process the network statistical information using a convolution matrix and generate a network state vector in combination with a recursive structure;

[0070] Reward generation module: used to generate a reward value corresponding to the network state vector through a reward function;

[0071] DRL agent module: used to output the maximum value of the congestion window using the DDPG model based on the network state vector and the reward value; the DDPG model includes: a policy network and a value network;

[0072] Maximum filter module: used to determine the size of the congestion window according to the maximum value of the congestion window using the maximum filter, and feed it back to the underlying TCP protocol stack.

[0073] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0074] Beneficial effects:

[0075] 1. Obtaining the congestion window value and confirmation data packet in the underlying TCP protocol stack in each round-trip time (RTT) cycle can timely reflect the current network status. It helps to generate accurate network statistics, which enables the system to respond quickly to network changes and ensures accurate capture of network status, thereby improving the accuracy of congestion control.

[0076] 2. The convolution operation can identify local dependencies in the data, while the recursive structure can effectively capture long-term dependencies in the time series. This combination not only enhances the model's ability to understand the dynamics of complex networks, but also provides richer and more structured inputs for subsequent decision-making processes, thereby improving the adaptability and robustness of the entire system.

[0077] 3. The reward function generates a reward value corresponding to the network state vector, which can effectively guide the learning process of the DDPG model. The agent will first keep the average delay below the target delay and get less punishment. Then, it will maximize its throughput and the number of packets received to get more rewards. This adaptive reward mechanism encourages the DDPG model to explore the optimal strategy and learn to take the most appropriate congestion control measures under different network conditions to achieve efficient data transmission.

[0078] 4. By outputting the maximum value of the congestion window for the continuous action space, the continuous action space allows for more subtle adjustments compared to discrete action selection. The policy network is responsible for proposing action suggestions, while the value network evaluates the quality of these actions. The two work together to ensure the scientificity and effectiveness of the congestion window adjustment. At the same time, the target network is used to ensure the stability of the training process; the experience replay pool is used to reduce the strong correlation between data. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0080] Figure 1 A flow chart of a wireless transmission congestion control method based on a DDPG model provided in an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of the process of outputting the maximum value of the congestion window using the DDPG model provided in an embodiment of the present invention;

[0082] Figure 3 A framework diagram of a wireless transmission congestion control system based on a DDPG model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Embodiment 1;

[0085] like Figure 1 As shown, the present invention provides a wireless transmission congestion control method based on the DDPG model, comprising the following steps:

[0086] S1. In each round-trip delay period, obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack to generate network statistics;

[0087] S2. Preprocessing the network statistical information using a convolution matrix and generating a network state vector in combination with a recursive structure;

[0088] S3. Generate a reward value corresponding to the network state vector through a reward function;

[0089] S4. Based on the network state vector and the reward value, a DDPG model is used to output the maximum value of the congestion window; the DDPG model includes: a policy network and a value network;

[0090] S5. According to the maximum value of the congestion window, a maximum filter is used to determine the size of the congestion window and the result is fed back to the underlying TCP protocol stack.

[0091] This method improves the adaptability to complex wireless environments by processing network statistics in real time and generating feature-rich state vectors. Using a carefully designed reward function and deep reinforcement learning mechanism, this method maximizes throughput while ensuring low latency, solving the performance problems of existing TCP algorithms under high bit error rates and dynamic changes. The policy network works in conjunction with the value network and adjusts the congestion window in real time in combination with the maximum filter, ensuring efficient and stable data transmission.

[0092] The above steps and related technical features are further described in detail below:

[0093] This embodiment S1 includes:

[0094] S11, generating data packet statistics based on the confirmed data packet;

[0095] S12, using the data packet statistical information to calculate the average data packet delay in each round trip delay period, the average throughput in each round trip delay period, and the number of data packets with the average data packet delay;

[0096] S13. The average data packet delay, average throughput and number of data packets are combined with the congestion window value in the underlying TCP protocol stack as network statistical information.

[0097] It generates packet statistics based on confirmed packets, calculates the average packet delay, average throughput and corresponding number of packets in each round-trip delay period, and combines the congestion window value to form comprehensive network statistics. It provides real-time and detailed network status perception, enhances the ability to adapt to complex wireless environments, and provides a solid data foundation for subsequent intelligent decision-making.

[0098] In this embodiment S2, the network statistical information is preprocessed using a convolution matrix, and a network state vector is generated in combination with a recursive structure;

[0099] Specific:

[0100] The network statistical information is preprocessed using a convolution matrix, including:

[0101] S21. Define the convolution matrix:

[0102]

[0103] Where, d is the average packet delay, d 0 is the target delay; the target delay represents the expected data transmission speed and response time, which aims to maximize TCP throughput while minimizing end-to-end delay.

[0104] S22, using the convolution matrix to perform filtering processing, and the average packet delay d and the target delay d 0 To encode:

[0105]

[0106] Where p is the average throughput and n is the number of packets;

[0107] S23. Based on the filtered and encoded data, obtain the observation statistical information at time t:

[0108]

[0109] Among them, cwnd t It is the congestion window value in the underlying TCP protocol stack.

[0110] Furthermore, the network state vector is expressed as:

[0111] s t =[o t ,o t-1 ,…,o t-m+1]

[0112] Among them, t-1 is the observation statistics at time t-1, o t-m+1 is the observation statistics at time t-m+1, and m is the length of the network state vector.

[0113] In this step, the network statistics are preprocessed by defining a convolution matrix, and the average packet delay and target delay are filtered and encoded using the convolution matrix, and the observation statistics at time t are generated by combining the average throughput and the number of packets. Furthermore, combined with the recursive structure, the observation statistics of the current moment and m historical adjacent moments are combined into a network state vector, thereby extracting rich feature representations, providing comprehensive and structured input for subsequent deep reinforcement learning decisions.

[0114] In this embodiment S3, the reward function is expressed as:

[0115]

[0116] Where ω(n,d) is the moving average of the two most recent values ​​of d:

[0117]

[0118] n pre ,d pre are the values ​​of n and d at the previous moment respectively.

[0119] The reward function is designed to balance the two key factors of throughput and latency. 0 When the delay is less than or equal to d, a negative incentive is given to punish high delay; on the contrary, when the delay is less than or equal to d 0 When , positive incentives are given to encourage low latency. At the same time, the reward function takes into account the moving average ω(n,d) to smooth the impact of latency fluctuations. This helps to ensure that the average latency of data packets is lower than the target value while maximizing throughput, thereby improving user experience.

[0120] In this embodiment S4, Figure 2 As shown, based on the network state vector and the reward value, the DDPG model is used to output the maximum value of the congestion window;

[0121] Specifically:

[0122] S41, initializing time step and related network parameters;

[0123] S42, through the policy network π θ , determine action a t And the maximum value of the congestion window at time t cwnd max ;

[0124] The action a t It is expressed as:

[0125] a t =π θ (s t )+N t

[0126]

[0127] Among them, π θ (s t ) is the policy function, N t is the noise term, a t The value range is [-1,1]; adding noise terms helps to explore different actions and prevent falling into local optimal solutions.

[0128] S43, execute action a t , get the reward value r t and the new state s t+1 ; Combined with the value network Qω, define the Q value function Q π (s t ,a t )=E[R t |s t ,a t ], for the action a t Evaluation; among them, R t is the cumulative expected return; the cumulative expected return R t It is expressed as:

[0129] R t =r t +γ 1 r t+1 +γ 2 r t+2 +…+γ n r t+n

[0130] Among them, γ 1 , γ 2 …γ n is the discount factor, and its value range is [0,1].

[0131] Cumulative expected return R t Combining immediate rewards and future discounted rewards, it not only focuses on current benefits but also considers long-term effects, thus achieving more robust decision-making.

[0132] S44. Combined with the experience replay mechanism, the experience tuple s is stored in the experience replay buffer D. t ,a t ,r t ,st+1 ), and randomly select N experience tuples (s i ,a i ,r i ,s i+1 ); it breaks the time correlation of data and improves the diversity of samples and the stability of training.

[0133] S45, based on the experience tuple (s i ,a i ,r i ,s i+1 ) to update network parameters; specifically including:

[0134] S451. Define the optimal strategy π θ The expected return J(θ) under:

[0135]

[0136] Use the policy gradient ascent strategy to improve the policy network π θ Perform parameter update, and the gradient of the expected return J(θ) is expressed as:

[0137]

[0138] Among them, η θ is the learning rate of the policy network;

[0139] S452, update the parameters of the value network Qω by minimizing the mean square error between the target Q value and the current Q value:

[0140]

[0141]

[0142] Among them, N represents the number of data, y i represents the target Q value, y i =r i +γQ ω' (s i+1 ,π θ' (s i+1 )), π θ' and Q ω' They are target strategy network and target value network respectively;

[0143] S453, updating the network parameters of the target strategy network and the target value network:

[0144] θ′←τθ+(1-τ)θ′

[0145] ω′←τω+(1-τ)ω′

[0146] Among them, τ is the speed control parameter.

[0147] S46, repeat the above S42 to S45 until the output satisfies the preset condition and outputs the maximum congestion window cwnd at the corresponding time max .

[0148] In this step, the target policy network and the policy network come from the same network, but the update frequency of the target policy network lags behind the policy network, and the same is true for the value network. Using a variety of historical data for training improves the generalization ability and robustness of the model, making the learning process smoother and accelerating the convergence speed.

[0149] Through the DDPG model, combined with the policy network and the value network, the maximum value of the congestion window is dynamically adjusted in each time step. By introducing noise terms and experience replay mechanisms, the exploration ability and training stability are enhanced. The combined use of the policy gradient ascent method and minimization of the Q value error ensures efficient optimization of the policy and value networks. Ultimately, this method can maximize throughput while maintaining low latency, significantly improving the performance of wireless transmission and user experience.

[0150] This embodiment S5 includes:

[0151] The maximum value of the congestion window is compared with the congestion window value calculated by the underlying TCP protocol stack, and the smaller congestion window value is selected and fed back to the underlying TCP protocol stack. This ensures the real-time transmission while maximizing the throughput, and realizes efficient and adaptive wireless transmission congestion control.

[0152] The above steps collect network statistics in the TCP protocol stack in real time and use convolutional matrices and recursive structures to generate feature-rich network state vectors. Combined with the reward function that balances throughput and latency, the policy network and value network work together to output the optimal maximum value of the congestion window. The experience replay mechanism and soft update strategy are used to ensure the stability and efficiency of the training process.

[0153] Finally, by comparing the DDPG model output with the congestion window calculated by the TCP protocol stack and selecting the smaller value to feed back to the underlying layer, low-latency, high-throughput data transmission is achieved in a high bit error rate and dynamically changing environment, improving the overall transmission performance and user experience.

[0154] Embodiment 2;

[0155] like Figure 3 As shown, this embodiment provides a wireless transmission congestion control system based on the DDPG model, including:

[0156] Monitoring module: used to obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack in each round-trip delay period and generate network statistics;

[0157] State generation module: used to pre-process the network statistical information using a convolution matrix and generate a network state vector in combination with a recursive structure;

[0158] Reward generation module: used to generate a reward value corresponding to the network state vector through a reward function;

[0159] DRL agent module: used to output the maximum value of the congestion window using the DDPG model based on the network state vector and the reward value; the DDPG model includes: a policy network and a value network;

[0160] Maximum filter module: used to determine the size of the congestion window according to the maximum value of the congestion window using the maximum filter, and feed it back to the underlying TCP protocol stack.

[0161] The system improves its adaptability to complex wireless environments by processing network statistics in real time and generating feature-rich state vectors. Using a carefully designed reward function and deep reinforcement learning mechanism, this method maximizes throughput while ensuring low latency, solving the performance problems of existing TCP algorithms under high bit error rates and dynamic changes. Deep reinforcement learning technology is combined with traditional congestion control, the policy network and the value network work together, and the congestion window is adjusted in real time in combination with the maximum filter, ensuring efficient and stable data transmission.

[0162] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0163] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A wireless transmission congestion control method based on DDPG model, characterized in that: The following steps are involved: S1. In each round-trip delay period, obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack to generate network statistics; S2. Preprocessing the network statistical information using a convolution matrix and generating a network state vector in combination with a recursive structure; S3. Generate a reward value corresponding to the network state vector through a reward function; S4. Based on the network state vector and the reward value, the DDPG model is used to output the maximum value of the congestion window; The DDPG model includes: a policy network and a value network; S5. According to the maximum value of the congestion window, a maximum filter is used to determine the size of the congestion window and the result is fed back to the underlying TCP protocol stack.

2. According to claim 1, a wireless transmission congestion control method based on the DDPG model is characterized in that: The S1 includes: S11, generating data packet statistics based on the confirmed data packet; S12, using the data packet statistical information to calculate the average data packet delay in each round trip delay period, the average throughput in each round trip delay period, and the number of data packets with the average data packet delay; S13. The average data packet delay, average throughput and number of data packets are combined with the congestion window value in the underlying TCP protocol stack as network statistical information.

3. According to claim 1, a wireless transmission congestion control method based on the DDPG model is characterized in that: In S2, the network statistical information is preprocessed using a convolution matrix, including: S21. Define the convolution matrix: Where, d is the average packet delay, d0 is the target delay; S22, using the convolution matrix to perform filtering processing, and encoding the average data packet delay d and the target delay d0: Where p is the average throughput and n is the number of packets; S23. Based on the filtered and encoded data, obtain the observation statistical information at time t: Among them, cwnd t It is the congestion window value in the underlying TCP protocol stack.

4. The wireless transmission congestion control method based on the DDPG model according to claim 1, characterized in that: In S2, the network state vector is expressed as: s t =[the t ,the t-1 ,…,the t-m+1 ] Among them, t-1 is the observation statistics at time t-1, o t-m+1 is the observation statistics at time t-m+1, and m is the length of the network state vector.

5. The wireless transmission congestion control method based on the DDPG model according to claim 1, characterized in that: In S3, the reward function is expressed as: Where ω(n,d) is the moving average of the two most recent values ​​of d: n pre ,d pre are the values ​​of n and d at the previous moment respectively.

6. The wireless transmission congestion control method based on the DDPG model according to claim 1, characterized in that: The S4 includes: S41, initializing time step and related network parameters; S42, through the policy network π θ , determine action a t And the maximum value of the congestion window cwnd at time t max ; The action a t It is expressed as: a t =π θ (s t )+N t Among them, π θ (s t ) is the policy function, N t is the noise term, a t The value range is [-1,1]; S43, execute action a t , get the reward value r t and the new state s t+1 ; Combined with the value network Qω, define the Q value function Q π (s t ,a t )=E[R t |s t ,a t ], for the action a t Evaluation; among them, R t To accumulate expected returns; S44. Combined with the experience replay mechanism, the experience tuple s is stored in the experience replay buffer D. t ,a t ,r t ,s t+1 ), and randomly select N experience tuples (s i ,a i ,r i ,s i+1 ); S45, based on the experience tuple (s i ,a i ,r i ,s i+1 ) Update network parameters; S46, repeat the above S42 to S45 until the output satisfies the preset condition and outputs the maximum congestion window cwnd at the corresponding time max .

7. A wireless transmission congestion control method based on DDPG model according to claim 6, characterized in that: In S43, the cumulative expected return R t It is expressed as: R t =r t +g 1 r t+1 +g 2 r t+2 +…+c n r t+n Among them, γ 1 , γ 2 …γ n is the discount factor, and its value range is [0,1].

8. The wireless transmission congestion control method based on the DDPG model according to claim 6, characterized in that: The S45 includes: S451. Define the optimal strategy π θ The expected return J(θ) under: Use the policy gradient ascent strategy to improve the policy network π θ Perform parameter update, and the gradient of the expected return J(θ) is expressed as: Among them, η θ is the learning rate of the policy network; S452, update the parameters of the value network Qω by minimizing the mean square error between the target Q value and the current Q value: Among them, N represents the number of data, y i represents the target Q value, y i =r i +γQ ω' (s i+1 ,π θ' (s i+1 )), π θ' and Q ω' They are target strategy network and target value network respectively; S453, updating the network parameters of the target strategy network and the target value network: θ'←τθ+(1-τ)θ' ω′←τω+(1-τ)ω′ Among them, τ is the speed control parameter.

9. The wireless transmission congestion control method based on the DDPG model according to claim 1, characterized in that: The S5 includes: The maximum value of the congestion window is compared with the congestion window value calculated by the underlying TCP protocol stack, and the smaller congestion window value is selected and fed back to the underlying TCP protocol stack.

10. A wireless transmission congestion control system based on DDPG model, characterized in that: include: Monitoring module: used to obtain the congestion window value and confirmation data packet in the underlying TCP protocol stack in each round-trip delay period and generate network statistics; State generation module: used to pre-process the network statistical information using a convolution matrix and generate a network state vector in combination with a recursive structure; Reward generation module: used to generate a reward value corresponding to the network state vector through a reward function; DRL agent module: used to output the maximum value of the congestion window using the DDPG model based on the network state vector and the reward value; The DDPG model includes: a policy network and a value network; Maximum filter module: used to determine the size of the congestion window according to the maximum value of the congestion window using the maximum filter, and feed it back to the underlying TCP protocol stack.

Images