User mode multipath transmission method based on reinforcement learning

Abstract

The invention provides a user mode multi-path transmission method based on reinforcement learning. The user mode multi-path transmission method comprises the steps of multi-path management based on a user mode, real-time network state monitoring and self-adaptive flow control based on reinforcement learning. Firstly, based on multi-path management of a user mode, parallel transmission of a plurality of network paths is realized through a user mode protocol stack, and data is received and transmitted by using a UDP (User Datagram Protocol), so that the transmission efficiency is improved; secondly, monitoring the network state in real time, dynamically collecting the state information of each path, including time delay, packet loss rate, bandwidth and the like, and monitoring the transmission performance of each path in real time; and finally, based on self-adaptive flow regulation and control of reinforcement learning, the distribution of data streams among paths is dynamically adjusted according to the network by utilizing a reinforcement learning model, and the transmission performance and the network resource utilization rate are maximized. The method is suitable for a multi-path transmission scene, is especially suitable for a multi-path communication scene of the mobile equipment in a weak network environment, can optimize the communication performance in the weak network environment, and improves the transmission efficiency and stability of a system.

Description

Technical Field

[0001] The present invention relates to the field of multi-path transmission networks, and in particular to a multi-path transmission method based on reinforcement learning and a user-mode transmission protocol. Background Art

[0002] In the digital age, the Internet is an important infrastructure that connects devices and users around the world, and has promoted the development of remote work, live streaming, e-commerce and other fields. In network protocols, stream sockets (TCP) and datagram sockets (UDP) are important protocols in the transport layer. TCP provides reliable, sequence-guaranteed data transmission, while UDP transmits data in a fast, connectionless manner, which is suitable for application scenarios with high real-time requirements.

[0003] However, with the development of software and hardware technologies and the improvement of people's expectations for network performance, especially in the context of bandwidth upgrades and latency optimization brought about by 5G technology, the traditional transport layer socket protocol TCP is somewhat weak in the rapidly developing mobile Internet. In theory, the mobile Internet is affected by unstable network signals, frequent changes in network access, and high-frequency fragmentation of transmitted data, and has the characteristics of high latency, low bandwidth, and easy jitter. Such a mobile weak network environment is a great challenge for what methods and strategies to use to control data transmission to improve user experience.

[0004] Although TCP performs well in reliability, its congestion control mechanism and complex connection management show performance bottlenecks in the modern mobile Internet environment, especially in the weak ground mobile network environment with unstable network, high user mobility and burst traffic. On this basis, user-mode transmission has gradually become a hot spot in protocol design. This approach is to move the execution of the protocol from the kernel to the user space, which makes the protocol more flexible and lightweight, and can achieve efficient data transmission through the UDP protocol, and avoid the kernel-mode control overhead of network communication. In comparison, it is more adaptable to the modern and changing network environment. In existing work, user-mode protocols represented by QUIC (QUIC: A UDP-based multiplexed and secure transport) and KCP (KCP: AFast and Reliable ARQ Protocol) have performed well in different network scenarios.

[0005] With the advancement of hardware, mobile devices can connect to multiple networks (such as Wi-Fi and cellular networks) at the same time, which provides a basis for multipath transmission. Multipath transmission refers to the ability of devices to use multiple network paths in parallel, thereby improving bandwidth utilization and network fault tolerance, especially in weak network environments, which can effectively reduce transmission delay and packet loss rate. At present, the widely used and deployed transport layer multipath network protocols are MPTCP (Multipath TCP) and MPQUIC (Multipath QUIC), both of which improve communication efficiency by using multiple communication lines in parallel, but they are still in the initial stage, and there is no better solution for the trade-off and regulation between multiple paths.

[0006] At the same time, the progress of machine learning technology has also provided new solutions for network transmission, mainly by introducing machine learning models through training to provide better adjustment solutions for network transmission. In recent years, works such as PCC (PCC: Re-architecting Congestion Control for Consistent High Performance), PCCVivace (PCC Vivace: Online-Learning Congestion Control), TCP-RL (Dynamic TCP initial windows and congestion control schemes through reinforcement learning) have been introduced to dynamically adjust TCP congestion control through learning. There are also some optimization solutions for channel selection in wireless scenarios based on learning traffic and channel prediction. However, there is currently no reinforcement learning practice on multipath transmission protocols.

[0007] In summary, the traditional TCP protocol has many limitations in mobile weak network environments, and the multi-path transmission method based on user state can effectively overcome the lack of flexibility of the kernel state protocol and the performance bottleneck of single-path transmission. However, the existing multi-path transmission method still faces challenges in complex network environments. Therefore, dynamic optimization of traffic distribution through reinforcement learning can more effectively cope with network fluctuations and achieve more efficient communication performance improvement. Summary of the invention

[0008] In view of the shortcomings of existing network transmission protocol technology, the present invention designs a multi-path weak network transmission method based on user state, and combines reinforcement learning to realize adaptive control of multi-path traffic. The purpose of the present invention is to provide a multi-path transmission method based on reinforcement learning, which improves the communication efficiency in a weak network environment by combining the fast transmission capability of UDP and the bandwidth advantage of multi-path parallelism.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A multi-path transmission method based on reinforcement learning is characterized in that the method includes: multi-path management based on user state, real-time network status monitoring and adaptive flow control based on reinforcement learning.

[0011] As a preferred method, user-state based multi-path management is to manage multiple network paths through a user-state protocol stack, establish multiple sub-paths for parallel data transmission, use the UDP protocol as the underlying transmission protocol, and send, receive and select paths for data packets in user space.

[0012] Preferably, real-time network status monitoring is to collect status information of each network path in real time through a path manager, including parameters such as path delay, packet loss rate, jitter and bandwidth utilization, and independently manage and monitor each sub-path.

[0013] Preferably, the traffic control based on reinforcement learning is to dynamically adjust the distribution of traffic between different paths according to the network status information of each sub-path by utilizing the reinforcement learning model, so as to optimize the overall transmission performance and network resource utilization.

[0014] In the above technical solution, the present invention provides a multi-path transmission method based on reinforcement learning, which has the following beneficial effects:

[0015] 1. The present invention adopts a multi-path transmission method based on user state, fully utilizing multiple network paths for data transmission, significantly improving bandwidth utilization and transmission stability, and realizing reliable control of data through flexible control of user state protocol, optimizing the response speed of the system, and solving the transmission bottleneck problem to a certain extent.

[0016] 2. The present invention designs a real-time network status monitoring mechanism that can continuously collect information such as latency, packet loss rate, bandwidth, etc., provide data support for subsequent path selection and traffic allocation, and ensure that transmission decisions can be optimized and adjusted according to real-time network conditions.

[0017] 3. The present invention introduces a reinforcement learning model for adaptive traffic control. By dynamically adjusting the traffic allocation strategy through real-time analysis of network status, reinforcement learning enables multi-path transmission to adaptively select the optimal path according to current network conditions, further improving transmission efficiency and network resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1An architecture diagram of a multiplexing method based on reinforcement learning provided by the present invention;

[0020] Figure 2 A schematic diagram of a multi-path transmission method based on reinforcement learning provided by the present invention;

[0021] Figure 3 A schematic diagram of an application scenario to which the present invention is applicable;

[0022] Figure 4 An architectural diagram of the user-mode based multi-path transmission method provided by the present invention;

[0023] Figure 5 A schematic diagram of complete data encapsulation in the OSI reference model of a user-mode based multiplexing method provided by the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0025] The present invention discloses a user-mode multi-path transmission method based on reinforcement learning, which aims to improve the transmission efficiency in a weak network environment and realize adaptive flow control. Figure 1 , Figure 2 As shown in the architecture diagram and schematic diagram of the present invention, the core content of the method includes three aspects: first, multi-path management based on user state, which realizes parallel transmission of multiple network paths through the user state protocol stack, and uses the UDP protocol to send and receive data, thereby reducing transmission bottlenecks and significantly improving transmission performance. Secondly, real-time network status monitoring, dynamically collects status information of each path, such as latency, packet loss rate and bandwidth, monitors the transmission performance of each path in real time, and provides reliable data support for subsequent path selection and traffic allocation. Finally, adaptive traffic control based on reinforcement learning dynamically adjusts the traffic allocation strategy according to real-time network feedback through the reinforcement learning model, ensures the optimal allocation of data flows among multiple paths, and maximizes transmission performance and network resource utilization. The present invention can adaptively select the best path in a complex network environment and optimize the overall performance of multi-path transmission. It is particularly suitable for Figure 3 The mobile device communication in the weak network scenario shown significantly improves the transmission efficiency and stability of the system.

[0026] Combination Figure 1 The multi-path transmission method based on reinforcement learning of the present invention comprises the following steps:

[0027] Step S1: User-state-based multi-path management implements the management of multiple network paths through the user-state protocol stack, establishes multiple sub-paths to support parallel data transmission, uses the UDP protocol as the underlying transmission protocol, and flexibly controls the sending, receiving, and path selection of data packets in the user space to ensure efficient data transmission and improve overall transmission performance.

[0028] Step S2: Network status monitoring The path manager collects the status information of each path in real time, including key parameters such as latency, packet loss rate, jitter and bandwidth utilization. The path manager independently manages and monitors each sub-path to ensure that the transmission performance can be optimized and adjusted according to the real-time network status.

[0029] Step S3: Traffic control based on reinforcement learning Through the reinforcement learning model, the distribution of traffic between different paths is dynamically adjusted according to the real-time network status information of each sub-path. According to the path performance feedback, the data allocation strategy is optimized to maximize the overall transmission performance and improve the utilization of network resources.

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Example 1

[0032] According to the user-mode based multi-path management method provided by the present invention, the specific implementation method is as follows:

[0033] Step S1.1: Multipath environment creation phase. Figure 4 As shown in the figure, the multi-path transmission protocol is implemented in the user state based on the multi-path management in the user state, and the sub-flow nodes are controlled and managed by the path manager inside the protocol. Therefore, it is necessary to create a path manager and multiple control sockets. The path manager is used to coordinate and manage the communication activities of all sub-paths to ensure the effective data distribution between multiple paths. The control socket is responsible for transmitting control information, which is used to maintain the state management and synchronization of the path in the subsequent communication process. By creating and initializing the path manager and socket, communication resources can be flexibly allocated on different network lines, and the basic configuration of multi-path parallel transmission can be realized, thereby ensuring that all available network resources can be effectively utilized in different network environments.

[0034] Step S1.2: In the multi-path transmission connection phase, the communication ends are controlled by the path manager to establish a communication link, and the relevant control information required for connection establishment and subsequent communication will be transmitted through the UDP protocol. This process ensures that both parties can establish a stable communication connection between multiple network paths, realize the initialization configuration of multi-path transmission, and provide guarantee for subsequent reliable data transmission.

[0035] The specific steps for establishing multi-path transmission are as follows:

[0036] Step S1.2.1: The client path manager sends a connection request via the UDP protocol, which includes the protocol type used, the client sub-path address and port information.

[0037] Step S1.2.2: After receiving the connection request, the server-side path manager establishes a mapping according to the client's sub-path information for subsequent multi-path transmission communication, and sends a connection confirmation message.

[0038] Step S1.2.3: After the connection is established successfully, both parties will perform data transmission based on UDP. The reliability and congestion control of the data will be controlled by a custom user-mode protocol to ensure the reliability and stability of multi-path communication.

[0039] Step S1.3: In the multi-path transmission communication phase, the path manager divides the data packets to be transmitted according to the real-time network status, such as latency, packet loss rate, etc., and distributes them to different sub-paths for transmission. During this process, the path manager dynamically adjusts the traffic distribution on each path to ensure that the data can be transmitted through the path with the best network conditions, thereby improving transmission efficiency and reducing latency and packet loss. After receiving the data, the server unpacks and reassembles the data through the sub-path protocol header to ensure the integrity and correctness of the data.

[0040] Regarding the protocol header part Figure 5 As shown, the present invention performs reliability control of related multi-path transmission by adding a custom user-space protocol header CUH (custom user-space protocol head) to the data at the transport layer, and at the same time calls the UDP protocol at the transport layer to add a UDP protocol header to it so as to use UDP to perform faster data transmission than TCP.

[0041] Taking the client sending data to the server as an example, the specific multi-path transmission communication steps are as follows:

[0042] Step S1.3.1: Client data packetization and encapsulation: The client path manager packets the data to be transmitted, encapsulates the data according to the corresponding user mode protocol, and adds a custom transmission protocol header to the packet to ensure that the data can be correctly identified and processed between multiple paths.

[0043] Step S1.3.2: Allocate sub-packets to sub-communication nodes: The client path manager dynamically analyzes the network status, selects appropriate sub-communication nodes, and distributes the encapsulated sub-packets to different network lines for transmission, so as to improve bandwidth utilization and reduce latency.

[0044] Step S1.3.3: Server-side data reception and unpacking: The server-side subcommunication node receives the data packet from the client, unpacks it according to the user-mode protocol header information, extracts the valid data, and hands it over to the server-side path manager.

[0045] Step S1.3.4: Returning data packets and feedback: The server-side path manager sends the response data packet back to the client sub-communication node through the sub-communication node according to the protocol processing flow and the saved client line information, ensuring that the client can obtain the communication feedback from the server in a timely manner.

[0046] Step S1.3.5: Client data unpacking and processing: After receiving the response packet sent back by the server, the client sub-communication node unpacks the data and passes it to the path manager, which further processes the data.

[0047] Step S1.3.6: Sending and allocating control information: The client path manager adds control information to the newly generated data packet according to the status of the data to be sent in the buffer, and distributes it to different sub-communication nodes for sending by dynamically analyzing the network situation.

[0048] Step S1.4: In the multi-path transmission disconnection phase, when the data transmission is completed, the client path manager actively initiates the disconnection operation, sends a disconnection request to each sub-communication node, and notifies it to close the corresponding socket and release the occupied resources. After receiving the disconnection request, the server gradually closes all sub-path connections, releases related resources, and completes the end operation of the multi-path transmission session. The purpose of this step is to ensure the reasonable release of resources, avoid redundant connections occupying system resources, and ensure the long-term stable operation of the system.

[0049] Example 2

[0050] According to the network status monitoring method provided by the present invention, the specific implementation method is as follows:

[0051] Step S2.1: Initialization monitoring of network status. In the initialization phase of transmission, the external monitoring component is started to perform initial status monitoring on each subpath P. Basic network status information is collected, including delay (d i ), packet loss rate (l i ), jitter (j i ) and bandwidth utilization (b i ) to ensure that the system has basic network reference data before the transmission starts. This data will serve as the basis for subsequent optimization and adjustment.

[0052] Step S2.2: Real-time network status collection. During the transmission process, the external monitoring component continuously monitors the real-time status of each sub-path, collects and records the delay (d i (t)), packet loss rate (li (t)) and bandwidth utilization (b i (t)) and other key performance indicators. All data will form a time series of the path. Figure 1 , Figure 2 The architecture diagram and schematic diagram are connected with the traffic allocation method based on reinforcement learning and the multi-path transmission method based on user state, providing a real-time basis for subsequent traffic allocation.

[0053] Step S2.3: Network status information recording. The external monitoring component stores the network status information collected at each time point in the path status table. The path status matrix is ​​expressed as:

[0054]

[0055] Among them, S(t) represents the global path state of the system at time t, and n is the number of sub-paths. The system updates historical records to ensure that subsequent analysis and optimization can effectively cope with dynamic changes in the network.

[0056] Step S2.4: Regular network status update. The external monitoring component updates the path status at a preset time interval to ensure that the recorded data can reflect the current network status. The real-time update of the state matrix S(t) provides accurate basic data for subsequent path selection and traffic control, ensuring the stability of transmission.

[0057] Step S2.5: Abnormal state detection and marking. During real-time monitoring, the external monitoring component detects changes in network status. Once a path P i If the delay or packet loss rate exceeds the preset threshold, the path will be marked as abnormal. The judgment basis for abnormal state detection is as follows:

[0058] Step S2.5.1: Delay anomaly detection (D th ). Path delay d i The abnormal marking of (t) is based on the delay threshold D th The threshold is usually set based on historical statistical data and application requirements. i (t) exceeds D th , the system marks the path as having abnormal latency:

[0059] E i (t) = {d i (t)>D th}

[0060] Among them, the delay threshold D th It can be set according to the following formula:

[0061] D th =d avg +k·σ

[0062] Among them, d avg is the historical average delay, σ is the standard deviation, and k is the adjustment coefficient. Select an appropriate value based on the application's sensitivity to delay.

[0063] Step S2.5.2: Packet loss rate anomaly detection (L th ). Path packet loss rate l i The abnormal marking of (t) is based on the packet loss rate threshold L th If the packet loss rate of a path is l i (t) exceeds L th , the system marks the path as abnormal packet loss:

[0064] E i (t) = {l i (t)>L th}

[0065] Packet loss rate threshold L th The setting can be based on historical packet loss data and network requirements, the formula is as follows:

[0066] L th = l avg +k·σ

[0067] Among them, l avg is the historical average packet loss rate, σ is the standard deviation of the packet loss rate, and k is adjusted according to the network environment and application requirements.

[0068] Through the above steps, the system can collect and record the status information of each network path in real time, including key parameters such as delay, packet loss rate and bandwidth. When it is detected that the path delay or packet loss rate exceeds the preset threshold, the system can mark the abnormal path in time and record the abnormal status. This network status monitoring method not only provides accurate real-time data support for the dynamic adjustment of the system, but also can effectively improve the response speed to network abnormalities and ensure the stability and controllability of the network transmission process.

[0069] Embodiment 3:

[0070] The present invention also provides a traffic control method based on reinforcement learning, and the specific implementation method is as follows:

[0071] Step S3.1: Construction and initialization of DQN-based reinforcement learning model. First, receive the network status information provided by S2 real-time network status acquisition, including the delay of each path (d i (t)), packet loss rate (l i (t)) and bandwidth utilization (b i Based on this information, a deep Q network (DQN) model is constructed, which includes the following sub-steps:

[0072] Step S3.1.1: Action space initialization. The action space is defined as the traffic allocation ratio for each sub-path. The actions that the model can choose in different states are to allocate data flows to different paths. For the path set P, the action space can be expressed as a set of allocation strategies:

[0073] A={a1,a2,...,a n}

[0074] Among them, each action a i Corresponding to a set of traffic distribution strategies, for example, when there are two network paths, 50% of the data flow is allocated to path P1 and 50% to path P2.

[0075] Step S3.1.2: State space construction. The state space contains the real-time state information of the network, such as path delay, packet loss rate and bandwidth. The network state S(t) at each moment is expressed as:

[0076]

[0077] Among them, d i (t), l i (t), b i (t) represent the delay, packet loss rate and bandwidth utilization of the i-th path at time t respectively.

[0078] Step S3.1.3: Reward function design. The reward function is defined based on transmission performance, with the goal of maximizing data transmission efficiency and network resource utilization. After the model performs an action, it will receive rewards or penalties based on factors such as transmission success rate, latency, and packet loss rate:

[0079] R(t)=-α·d avg (t)-β·l avg (t)+γ·b avg (t)

[0080] Among them, d avg (t), l avg (t), b avg (t) is the weighted average of the delay, packet loss rate and bandwidth of each path, α, β,

[0081] γ is the corresponding weight coefficient, which is used to balance the influence of various factors.

[0082] Step S3.2: Model training and strategy optimization. The DQN model is trained in the experimental environment, multi-path transmission tasks are simulated, and the path selection strategy of the model is gradually optimized, which specifically includes the following sub-steps:

[0083] Step S3.2.1: Initialize the experimental environment. The simulated network state includes the delay, packet loss rate and bandwidth of each path. Assuming there are n paths, the model is initialized according to the path set P of these states, and the state vector S of each path i (t) contains three key parameters:

[0084] S i (t) = {d i (t), l i (t), b i (t)}

[0085] Among them, d i (t) is the time delay, l i (t) is the packet loss rate, b i (t) is the bandwidth.

[0086] Step S3.2.2: Model training process. The DQN model optimizes path selection by selecting different traffic distribution strategies. After each action a(t) is selected, the system adjusts the Q value of the model based on the reward function feedback. The Q value update formula is:

[0087]

[0088] Among them: α is the learning rate, R(t) is the reward at the current moment, γ is the discount factor, indicating the importance of future rewards, max a′ Q(S(t+1), a′) is the maximum Q value of the next state.

[0089] Step S3.2.3: Experience replay. During the training process, the model introduces an experience replay mechanism to store the past state-action-reward-new state (SARS) quadruple in the memory pool. Each time the model is updated, samples in the memory pool are randomly selected for training to break the time correlation and improve training efficiency and stability.

[0090] Step S3.3: Real-time traffic allocation and path selection decision. The trained DQN model is deployed in the actual multi-path transmission system to make real-time decisions on the traffic allocation scheme for each transmission task. The specific process is as follows:

[0091] Step S3.3.1: Real-time state evaluation. At the beginning of each transmission task, the system generates the current state vector S(t) = {S1(t), S2(t), ..., S n (t)}, the delay, packet loss rate and bandwidth of each path constitute the current global network status.

[0092] Step S3.3.2: Path selection and traffic allocation. The DQN model selects the optimal action based on the current state vector S(t), that is, selects the optimal traffic allocation scheme a(t), and allocates the data flow to multiple paths in different proportions. During this process, the model dynamically adjusts the traffic allocation to ensure that the data is transmitted through the optimal path.

[0093] The actual flow rate f of each path i (t) are allocated as follows:

[0094] f i (t) = a(t)·b I (t)

[0095] Among them, a(t) represents the allocation ratio, b I (t) is the path P I bandwidth.

[0096] Step S3.4: Transmission feedback and model adaptive update. In the actual transmission process, the DQN model adaptively adjusts its strategy according to the feedback results of each transmission, as follows:

[0097] Step S3.4.1: Feedback collection. The model calculates the actual reward value R(t) based on the feedback information after transmission (such as delay, packet loss rate, etc.) and adjusts the path selection strategy according to the transmission results.

[0098] The system calculates the actual reward value R(t) based on the feedback information such as the delay and packet loss rate after the transmission is completed, and updates the Q value of the model. The reward function is the same as before:

[0099] R(t)=-α·d avg (t)-β·l avg (t)+γ·b avg (t)

[0100] Step S3.4.2: Online learning and policy update. The model uses the online learning mechanism to update the Q value in real time using the newly collected data and optimize the policy, so that the system can adapt to different network conditions and continuously optimize the traffic distribution plan. The Q value update is still performed according to the Q learning formula.

[0101] Step S3.5: Long-term strategy optimization and model retraining. To maintain the long-term effectiveness of the model, regular retraining and strategy optimization are performed, including:

[0102] Step S3.5.1: Historical data accumulation and analysis. The system stores the long-term network status and transmission result data in the historical database, and regularly analyzes these data to detect the changing trend of the network status, forming a complete historical data set for model retraining.

[0103] Step S3.5.2: Regular retraining and model optimization. The model is regularly retrained based on historical data to maintain its adaptability to changes in the network environment. During the retraining process, the model continues to optimize the path selection strategy through the Q value update mechanism:

[0104] Q(S(t), a(t))←Q(S(t), a(t))+α[R(t)+γ·a′maxQ(S(t+1), a′)-Q(S(t), a(t))]

[0105] As described above, the transmission method of the present invention can adaptively optimize traffic distribution in a complex network environment and improve the efficiency and stability of multi-path transmission.

Claims

1. A user-mode multipath transmission method based on reinforcement learning, characterized in that: include: User-mode multi-path management, real-time network status monitoring, and adaptive traffic control based on reinforcement learning. Step S1: The user-mode based multi-path management is to manage multiple network paths through a user-mode protocol stack, establish multiple sub-paths for parallel data transmission, use the UDP protocol as the underlying transmission protocol, and perform data packet sending, receiving and path selection operations in the user space; Step S2: The real-time network status monitoring is to collect the status information of each network path in real time through the path manager, including parameters such as path delay, packet loss rate, jitter and bandwidth utilization, and independently manage and monitor each sub-path; Step S3: The reinforcement learning-based traffic control is to dynamically adjust the distribution of traffic between different paths according to the network status information of each sub-path by utilizing the reinforcement learning model to optimize the overall transmission performance and network resource utilization.

2. According to the user-mode multi-path transmission method based on reinforcement learning in claim 1, it is characterized in that: The step S1 adopts: Step S1.1: In the multi-path environment creation phase, a path manager and multiple control sockets are created. The path manager is used to coordinate and manage the communication activities of all sub-paths to ensure effective data distribution among multiple paths. The control socket is responsible for transmitting control information to maintain the state management and synchronization of the path in the subsequent communication process. By creating and initializing the path manager and sockets, communication resources can be flexibly allocated on different network lines to achieve the basic configuration of multi-path parallel transmission, thereby ensuring that all available network resources can be effectively utilized in different network environments. Step S1.2: In the multi-path transmission connection phase, the path manager sends a multi-path connection request to the server through the UDP protocol, which contains the client's sub-path information and its corresponding communication parameters, such as IP address and port number; on this basis, the server-side path manager receives and analyzes the information, establishes the corresponding sub-path mapping, and feeds back confirmation information to the client; this process ensures that both parties can establish a stable communication connection between multiple network paths, realize the initialization configuration of multi-path transmission, and provide guarantee for subsequent reliable data transmission; Step S1.3: In the multi-path transmission communication phase, the path manager divides the data packets to be transmitted according to the real-time network status, such as latency, packet loss rate, etc., and allocates them to different sub-paths for transmission; During this process, the path manager will dynamically adjust the traffic distribution on each path to ensure that data can be transmitted through the path with the best network conditions, thereby improving transmission efficiency and reducing latency and packet loss. After receiving the data, the server will unpack and reassemble the data through the sub-path protocol header to ensure the integrity and correctness of the data. Step S1.4: During the multi-path transmission disconnection phase, when data transmission is completed, the client path manager actively initiates a disconnection operation and sends a disconnection request to each sub-communication node, notifying it to close the corresponding socket and release the occupied resources; after receiving the disconnection request, the server gradually closes all sub-path connections, releases related resources, and completes the termination operation of the multi-path transmission session; the purpose of this step is to ensure the reasonable release of resources, avoid redundant connections occupying system resources, and ensure the long-term stable operation of the system.

3. According to the user-mode multi-path transmission method based on reinforcement learning in claim 1, it is characterized in that: The step S2 adopts: Step S2.1: Initialization monitoring of network status. The external monitoring component is started when the transmission is initialized, and the initial status of each sub-path is monitored. The basic network status information of each path, such as delay, packet loss rate, jitter and bandwidth, is collected to ensure that basic network reference data is available for subsequent transmission. Step S2.2: Real-time network status collection. The external monitoring component continuously monitors the real-time status of each sub-path during the transmission process, collects key performance indicators such as latency, packet loss rate and bandwidth, and promptly feeds back the current network status to the system, providing real-time basis for path management and traffic allocation; Step S2.3: Network status information recording, the external monitoring component stores the collected real-time network status information in the path status table and regularly updates the historical records of each sub-path to ensure that the system can perform effective analysis and subsequent optimization decisions based on changes in network status; Step S2.4: Regular network status update: the external monitoring component updates the path status information at a preset time interval to ensure that the recorded data can reflect the latest status of the current network and provide accurate basic data for the transmission method to adjust the flow distribution and path selection in real time; Step S2.5: Abnormal state detection and marking. The external monitoring component detects abnormal network conditions during the monitoring process, such as a surge in latency or an abnormal increase in packet loss rate. Once an abnormality is found, the monitoring component marks the path as unstable and records the specific abnormal information for subsequent adjustment and optimization operations.

4. The user-mode multi-path transmission method based on reinforcement learning according to claim 1, characterized in that: The step S3 adopts: Step S3.1: Construction and initialization of the reinforcement learning model based on DQN, receiving the network status information transmitted by part S2, including key parameters such as latency, packet loss rate and bandwidth, building a reinforcement learning model based on the deep Q network (DQN) algorithm, and initializing its action space, state space and reward function; Step S3.2: Model training and strategy optimization. Traffic distribution is modeled as a path selection problem. The constructed DQN model is trained through multi-path transmission tasks in the experimental environment. The model gradually optimizes its path selection strategy by deciding the priority of the path to achieve the optimal allocation of data flow. Step S3.3: Real-time traffic allocation and path selection decision. The trained DQN model is deployed in the communication user end to make real-time decisions on the traffic allocation plan for each transmission task. The model dynamically adjusts the data flow allocation model training and optimization of each path based on the real-time network status information transmitted by part S2. Step S3.4: Transmission feedback and model adaptive update. The model continuously updates its path selection decision by monitoring the feedback results of each transmission, such as delay changes and packet loss. According to the transmission feedback results, the model adjusts the path selection strategy in real time through the online learning mechanism to ensure that the system can adapt to network fluctuations and continuously optimize the traffic distribution plan. Step S3.5: Long-term strategy optimization and model retraining. Through long-term transmission tasks, the model accumulates a large amount of network transmission data, and the DQN model is retrained and optimized regularly to ensure that it continues to maintain optimal performance in different dynamic network environments.

Images