End system access delay jitter optimization method based on reinforcement learning

Abstract

The invention provides an end system access delay jitter optimization method based on reinforcement learning, and belongs to the technical field of terminal access, and the method comprises the steps: accessing a 5G access device to a network access point through employing a random access algorithm, and obtaining a reinforcement learning model based on an access selection algorithm; the reinforcement learning model is trained, and the trained reinforcement learning model is transmitted downwards to all 5G terminal devices using the access node for network transmission; network information of network access points is collected and quantized, a network state matrix is formed through combination, the network state matrix is input to the reinforcement learning model, a quantized value is output, the network access point corresponding to the optimal delay jitter performance is selected for network switching, and network service is obtained; the problems that access judgment is not carried out by taking delay jitter as the main performance in the existing access technology and complex and changeable network scenes of the 5G mobile terminal are difficult to deal with are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of terminal access, and in particular relates to a method for optimizing terminal system access delay jitter based on reinforcement learning. Background Art

[0002] In recent years, people have an increasing demand for high-quality broadband wireless network access anytime and anywhere, and various wireless network technologies that adapt to different scenarios have developed rapidly. Mobile cellular networks have evolved from the Global System for Mobile Communications (GSM) to the Universal Mobile Telecommunications System (UMTS), and then to the Long Term Evolution (LTE), providing a wide range of network coverage and seamless mobility guarantees; in addition, the establishment of a series of 802.11 wireless local area network (WLAN) standards and 802.16 wireless metropolitan area network (WiMAX) standards has provided users with high-speed wireless connections. Within the signal coverage of the GSM base station, adding wireless access points such as LTE, WLAN, and WiMAX to facilitate data transmission has become a common mode in the industry, gradually forming heterogeneous wireless networks HWNs where multiple networks coexist and signal coverage overlaps. At the same time, the signal coverage, uplink and downlink transmission rates, and the most suitable types of services supported by various networks are different, and no network technology can support all different user services at the same time. With the development of HWNs, these networks compete, complement, and promote each other in their respective evolution processes, which ultimately makes the integration of HWNs inevitable. In the process of HWNs convergence, each wireless network is used as an access network and user connection through different architectures and protocols. After convergence, each wireless network is interconnected through a public IP core network. Multi-mode mobile terminal users can select a certain access point and access the Internet through a public IP core network. Network access selection is one of the key technologies of HWNs convergence. Its main function is to control the user's access request and select a certain network to provide connection services for the user. How to utilize the characteristics of HWNs in different access technologies, overlapping network architectures, and multi-service traffic loads to provide users with access options while ensuring the service quality QOS of user services and optimizing the use of wireless resources has become a hot topic in HWNs research.

[0003] The design of network access selection algorithm is directly related to user experience and network resource utilization. At present, a lot of research has been done in the existing technology on HWNs access selection algorithm. Looking at various algorithms, this paper divides them into access selection algorithms based on received signal strength (RSS), load balancing and service QOS according to the judgment criteria. According to the mathematical model adopted by the algorithm, it is further divided into access selection algorithms based on multi-attribute decision-making, utility function, fuzzy logic, game theory, etc.

[0004] As network terminal applications continue to develop and progress, the demand for network performance is also increasing. At present, due to the emergence of more and more high-real-time traffic, 5G terminal devices have higher and higher requirements for the delay jitter provided by the network. Under the requirement of low delay, the access node is also required to provide very low delay jitter performance. At present, 5G terminal access algorithms have made significant progress in achieving high speed, low latency and high reliability. 5G networks use a variety of advanced technologies, such as massive multiple-input multiple-output MIMO, beamforming and millimeter wave bands to optimize terminal access performance. Access algorithms have been improved in resource allocation, interference management and load balancing, enabling 5G networks to support more terminal devices and higher data rates. However, there is no intelligent access technology that uses delay jitter as the main performance for access judgment. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for optimizing terminal system access delay jitter based on reinforcement learning, which solves the problem that the existing access technology does not use delay jitter as the main performance for access judgment and the problem that the complex and changeable network scenarios of 5G mobile terminals are difficult to cope with.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is: a method for optimizing end system access delay jitter based on reinforcement learning, comprising the following steps:

[0007] S1. Use a random access algorithm to access a 5G access device to a network access point, and use an access selection algorithm to obtain a reinforcement learning model based on the access selection algorithm;

[0008] S2. Train a reinforcement learning model based on the access selection algorithm, and transmit the trained reinforcement learning model to all 5G terminal devices that use the access node for network transmission;

[0009] S3. Collect network information of network access points using 5G terminal equipment;

[0010] S4. quantize all collected network information and combine them into a network state matrix, input the network state matrix into the trained reinforcement learning model, output the quantized value, select the network access point corresponding to the optimal delay jitter performance according to the output quantized value, and perform network switching;

[0011] S5. Connect the 5G terminal device to the network, obtain the network service of the network access point corresponding to the optimal delay and jitter performance, and complete the end system access delay and jitter optimization.

[0012] The beneficial effects of the present invention are as follows: the present invention realizes access judgment with delay jitter as the main performance by using access selection algorithm, reinforcement learning model and data collection, and enables 5G mobile terminals to effectively cope with complex and changeable network scenarios.

[0013] Furthermore, the specific steps of S1 are as follows:

[0014] Using the random access algorithm, a network access point with the operator access node identifier is randomly generated in the 5G access device. According to the operator access node identifier, the 5G terminal device is used to connect to the 5G network of the corresponding operator, and the access selection algorithm is used according to the network access point to obtain a reinforcement learning model based on the access selection algorithm.

[0015] Furthermore, the S2 comprises the following steps:

[0016] S201, obtaining a reinforcement learning model and a training data set based on an access selection algorithm for each access node in a 5G access device, and initializing the reinforcement learning model and the training data set based on the access selection algorithm using a Markov model;

[0017] S202, using the access node to collect network data uploaded by the 5G terminal, and access device parameters and device performance, and combine them to obtain data training samples, and based on the data training samples, use Q learning to train the parameters of the reinforcement learning model;

[0018] S203, determine whether the time reaches the training cycle or whether there is a terminal requesting model parameters, if so, execute S204, otherwise execute S202;

[0019] S204, using the parameters of the trained reinforcement learning model to optimize the objective function of the reinforcement learning model, training the reinforcement learning model based on the access selection algorithm;

[0020] S205, transmitting the trained reinforcement learning model downward to all 5G terminal devices that use the access node for network transmission;

[0021] The objective function of the reinforcement learning model is expressed as follows:

[0022]

[0023] Among them, L(θ) represents the objective function of the reinforcement learning model, r t (θ) represents the reward ratio, ε represents the estimation factor of the advantage function, represents a positive number in the estimation factor of the advantage function, clip(·) represents contrastive language-image pre-training, θ represents the model parameters of reinforcement learning, and θ old represents the model parameters of the previous round of reinforcement learning, a t Indicates execution of an action, s t Indicates the state, π θ Represents the algorithmic strategy of the reinforcement learning model parameters, π θold An algorithmic policy representing the model parameters of the previous round of reinforcement learning.

[0024] The beneficial effect of the above further solution is that the present invention utilizes the Markov model to learn and strengthen the previous access status and network performance, and trains the network parameters, thereby improving the network delay jitter performance for node selection.

[0025] Furthermore, the S3 comprises the following steps:

[0026] S301, dividing the network status information into terminal node network information, access node status information and flow status information;

[0027] S302, using 5G terminal equipment to collect terminal node network information;

[0028] S303, using 5G terminal equipment to collect access node status information;

[0029] S304. Collect flow status information using 5G terminal equipment.

[0030] The beneficial effects of the above further scheme are: the present invention collects clear network status information, including terminal node network information, access node status information and flow status information, thereby improving the accuracy of the network status matrix composed of the network status information, and enabling the reinforcement learning access selection model to have more accurate actual data, thereby improving the ability of the present invention to cope with complex and changing 5G mobile network scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The figure is a flow chart of the method of the present invention.

[0032] Figure 2 The figure is a flow chart of the process of model training and issuing to access nodes in this embodiment.

[0033] Figure 3 This is a schematic diagram of access selection in this embodiment. DETAILED DESCRIPTION

[0034] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0035] Example

[0036] like Figure 1 As shown, the present invention provides a method for optimizing end system access delay jitter based on reinforcement learning, and its implementation method is as follows:

[0037] S1. Use the random access algorithm to access the 5G access device to the network access point, and use the access selection algorithm to obtain a reinforcement learning model based on the access selection algorithm. The specific steps are as follows:

[0038] Using the random access algorithm, a network access point with the operator access node identifier is randomly generated in the 5G access device. According to the operator access node identifier, the 5G terminal device is used to connect to the 5G network of the corresponding operator, and the access selection algorithm is used according to the network access point to obtain a reinforcement learning model based on the access selection algorithm.

[0039] In this embodiment, a random access algorithm is used to randomly generate an operator access node identifier. The 5G terminal device connects to the corresponding operator's 5G network based on the identifier, and obtains a reinforcement learning model based on the access selection algorithm from the access point. At this time, the 5G access device has not obtained the parameters of the trained reinforcement learning model, so it is impossible to directly use the reinforcement learning model to select the access point with the best delay jitter performance and obtain network services. Therefore, a random access algorithm is first used to access an access point that can provide a network.

[0040] S2. Train the reinforcement learning model based on the access selection algorithm, and transmit the trained reinforcement learning model to all 5G terminal devices that use the access node for network transmission. The specific steps are as follows:

[0041] S201, obtaining a reinforcement learning model and a training data set based on an access selection algorithm for each access node in a 5G access device, and initializing the reinforcement learning model and the training data set based on the access selection algorithm using a Markov model;

[0042] S202, using the access node to collect network data uploaded by the 5G terminal, and access device parameters and device performance, and combine them to obtain data training samples, and based on the data training samples, use Q learning to train the parameters of the reinforcement learning model;

[0043] S203, determine whether the time reaches the training cycle or whether there is a terminal requesting model parameters, if so, execute S204, otherwise execute S202;

[0044] S204, using the parameters of the trained reinforcement learning model to optimize the objective function of the reinforcement learning model, training the reinforcement learning model based on the access selection algorithm;

[0045] S205: Transmit the trained reinforcement learning model downward to all 5G terminal devices that use the access node for network transmission.

[0046] In this embodiment, the access node is requested to perform a reinforcement learning model parameter request. From the perspective of the terminal, only the trained model parameters are obtained, but the steps of model training and data set collection are specifically included;

[0047] like Figure 2 As shown, the reinforcement learning model and training data set based on the access selection algorithm of each access node in the 5G access device are obtained, and the Markov model is used for initialization. The parameters include<S,A,P,R,γ> ; Where S represents the state space, the set of all possible states, representing the different conditions or positions that the agent can be in the environment, and the agent will migrate between these states; A represents the action space, the set of all possible behaviors or actions that the agent can choose to perform; for each state s∈S, the agent will select an action a from the action space A to perform; P represents the state transition probability, the probability distribution of transferring from a state s to the next state s′ after executing action a, and P(s′|s,a) is used to represent the probability of the agent reaching a new state s′ given the current state s and the action a taken; R represents the reward function, which defines a function that maps from state-action pairs to immediate rewards. After taking action a, the immediate reward obtained when the agent transfers from state s to is recorded as R(s,a,s′) or abbreviated r; γ represents the discount factor, which is a value between 0 and 1, which determines the importance of future rewards. The closer the value of the discount factor γ is to 1, the longer the agent looks at future rewards and the more it pays attention to long-term interests; the closer the value is to 0, the more the agent prefers short-term gains.

[0048] In this embodiment, the access node is used to collect network data uploaded by the 5G terminal, and the device parameters and device performance are accessed to obtain new data training samples. The collected data is used to continuously train parameters based on Q learning, and the optimal decision sequence of the Markov decision process is solved by the Bellman equation. The state value function V(s) is used to evaluate the quality of the current state. Each state value is determined not only by the current state but also by the subsequent state. Therefore, the state value function V(s) of the current state can be obtained by calculating the expectation of the cumulative reward of the state.

[0049] The expression of the state value function V(s) is as follows:

[0050] V π (s) = E π [R t+1 +γV(s′)|s t =s]

[0051] Among them, π represents the algorithm strategy, E represents the expectation of the reward function, and R t+1 Represents the t+1th reward function; sort out E π It indicates the expected size of the reward that can be obtained by reinforcement learning when executing the access strategy of the algorithm;

[0052] According to the state value function V(s), the state action value function Q(S,A) is derived;

[0053] The expression of the state action value function Q(S,A) is as follows:

[0054] Q π (s,a)=E π [R t+1 +γQ(S t+1 ,A t+1 )|A t =a,s t =s]

[0055] Among them, A t+1 represents the t+1th action; the Q value can be calculated according to the state action value function Q(S,A), and the Q value is learned using the Q-table update process, where α represents the learning rate, β represents the reward decay coefficient, and the time difference method is used for updating; the update expression is as follows:

[0056] Q(s,a)←Q(s,a)+α[β+γmax a′ Q(s′,a′)-Q(s,a)]

[0057] Use L1 regularization parameter reverse derivation to optimize the reinforcement learning model parameters. The expression is as follows:

[0058]

[0059] Among them, L data Represents the data loss of the reinforcement learning model, which is usually the error between the predicted value of the reinforcement learning model and the true label, including mean square error (MSE) or cross-entropy loss. λ represents the regularization parameter, which is used to control the strength of the regularization term. i | represents the absolute value of the weight of the reinforcement learning model.

[0060] In this embodiment, the reinforcement learning model based on the access selection algorithm is trained by continuously optimizing the objective function specially designed for reinforcement learning, that is, repeatedly optimizing the reinforcement learning model parameters. The objective function of the reinforcement learning model is expressed as follows:

[0061]

[0062] Among them, L(θ) represents the objective function of the reinforcement learning model, r t (θ) represents the reward ratio, ε represents the estimation factor of the advantage function, represents a positive number in the estimation factor of the advantage function, clip(·) represents contrastive language-image pre-training, θ represents the model parameters of reinforcement learning, and θ old represents the model parameters of the previous round of reinforcement learning, a t Indicates execution of an action, s t Indicates the state, π θ Represents the algorithmic strategy of the reinforcement learning model parameters, π θold The algorithmic strategy representing the model parameters of the previous round of reinforcement learning;

[0063] The trained reinforcement learning model is transmitted downward to all 5G terminal devices that use the access node for network transmission.

[0064] S3. Use 5G terminal equipment to collect network information of network access points. The specific steps are as follows:

[0065] In this embodiment, the 5G terminal access device collects network information such as the location, number, available bandwidth, background flow information, number of hops of the upper layer connection, basic delay bandwidth requirements, and its own location direction of the network access point; the network status information that needs to be collected is mainly divided into three parts, terminal node network information, access node status information, and flow status information: the network status is defined as a three-part combination expression as follows:

[0066] S network =[S end ,S access ,S flow ]

[0067] Among them, S network Indicates network status information, S end Represents the terminal node network information, S access Indicates access node status information, S flow Indicates flow status information;

[0068] Collect terminal node network information, such as the location of the 5G terminal device, mobile speed, device type, and the types and number of operators supported by the device; the expression of the terminal node network information is as follows:

[0069] S end =[POS end ,RATE,TYPE,OPERAT]

[0070] Among them, S end Indicates the network information of the terminal node, RATE indicates the mobile rate, TYPE indicates the device type, and OPERAT indicates the types and number of operators supported by the device;

[0071] Collect access node status information, such as the access node number, the available bandwidth of the access node, the number of background flows of the access node, the size of the background flows of the access node, and the operator and location of the access node; the expression of the access node status information is as follows:

[0072] S access =[ID,BandWidth,NUM access ,SIZE access ,POS access ]

[0073] Among them, S flow Indicates access node status information, ID indicates the access node number, BandWidth indicates the available bandwidth of the access node, NUM access Indicates the number of background flows accessing the node, SIZE access Indicates the background flow size of the access node, POS access Indicates the operator and location of the access node;

[0074] Collect flow status information, such as the receiver of the flow, the number of routing hops, the delay required by the flow, the bandwidth and delay jitter requirements required by the flow, the bit size of the flow, and the duration of the flow; the expression of the flow status information is as follows:

[0075] S flow =[RECV,HOP,DELAY,JUTTER,BYTES,DURATION]

[0076] Among them, S flowIt indicates the flow status information, RECV indicates the receiver of the flow, HOP indicates the number of routing hops, DELAY indicates the delay required by the flow, JUTTER indicates the bandwidth and delay jitter requirements required by the flow, BYTES indicates the bit size of the flow, and indicates the duration of the flow.

[0077] S4. Quantify all collected network information and combine them into a network state matrix, input the network state matrix into the trained reinforcement learning model, output the quantized value, select the network access point corresponding to the optimal delay jitter performance according to the output quantized value, and perform network switching.

[0078] In this embodiment, all collected network information is quantified and combined into a network state matrix as the input of the reinforcement learning access selection model. According to the quantized value output by the reinforcement learning model, the corresponding access point is found and the network switching is performed.

[0079] S5. Connect the 5G terminal device to the network, obtain the network service of the network access point corresponding to the optimal delay and jitter performance, and complete the end system access delay and jitter optimization.

[0080] In this embodiment, the access network provides services for upper-layer applications and completes the end system access delay jitter optimization based on reinforcement learning.

[0081] In this embodiment, Figure 3 As shown, the access selection includes: a core network device 103, a first access network device 101 and a second access network device 102 connected to the core network device 103; the first access network device 101 is connected to a first terminal device 104 and a second terminal device 105 respectively; the second access network device 101 is connected to a third terminal device 106 and a fourth terminal device 107 respectively;

[0082] The core network device is connected to a plurality of access network devices, and each access network device is connected to a plurality of terminal devices; the access selection of the end system is realized; and the access delay jitter of the end system is optimized by using the present invention.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A method for optimizing end system access delay jitter based on reinforcement learning, characterized in that: The following steps are involved: S1. Use a random access algorithm to access a 5G access device to a network access point, and use an access selection algorithm to obtain a reinforcement learning model based on the access selection algorithm; S2. Train a reinforcement learning model based on an access selection algorithm in the access node, and transmit the trained reinforcement learning model downward to all 5G terminal devices that use the access node for network transmission; S3. Collect network information of network access points using 5G terminal equipment; S4. quantize all collected network information and combine them into a network state matrix, input the network state matrix into the trained reinforcement learning model, output the quantized value, select the network access point corresponding to the optimal delay jitter performance according to the output quantized value, and perform network switching; S5. Connect the 5G terminal device to the network, obtain the network service of the network access point corresponding to the optimal delay and jitter performance, and complete the end system access delay and jitter optimization.

2. The end system access delay jitter optimization method based on reinforcement learning according to claim 1 is characterized in that: The specific steps of S1 are as follows: Using the random access algorithm, a network access point with the operator access node identifier is randomly generated in the 5G access device. According to the operator access node identifier, the 5G terminal device is used to connect to the 5G network of the corresponding operator, and the access selection algorithm is used according to the network access point to obtain a reinforcement learning model based on the access selection algorithm.

3. The end system access delay jitter optimization method based on reinforcement learning according to claim 1 is characterized in that: The S2 comprises the following steps: S201, obtaining a reinforcement learning model and a training data set based on an access selection algorithm for each access node in a 5G access device, and initializing the reinforcement learning model and the training data set based on the access selection algorithm using a Markov model; S202, using the access node to collect network data uploaded by the 5G terminal, and access device parameters and device performance, and combine them to obtain data training samples, and based on the data training samples, use Q learning to train the parameters of the reinforcement learning model; S203, determine whether the time reaches the training cycle or whether there is a terminal requesting model parameters, if so, execute S204, otherwise execute S202; S204, using the parameters of the trained reinforcement learning model to optimize the objective function of the reinforcement learning model, training the reinforcement learning model based on the access selection algorithm; S205: Transmit the trained reinforcement learning model downward to all 5G terminal devices that use the access node for network transmission.

4. The end system access delay jitter optimization method based on reinforcement learning according to claim 3 is characterized in that: The objective function of the reinforcement learning model is expressed as follows: Among them, L(θ) represents the objective function of the reinforcement learning model, r t (θ) represents the reward ratio, ε represents the estimation factor of the advantage function, represents a positive number in the estimation factor of the advantage function, clip(·) represents contrastive language-image pre-training, θ represents the model parameters of reinforcement learning, and θ old represents the model parameters of the previous round of reinforcement learning, a t Indicates execution of an action, s t Indicates the state, π θ Represents the algorithmic strategy of the reinforcement learning model parameters, π θold An algorithmic policy representing the model parameters of the previous round of reinforcement learning.

5. The method for optimizing end system access delay jitter based on reinforcement learning according to claim 1, characterized in that: The S3 comprises the following steps: S301, dividing the network status information into terminal node network information, access node status information and flow status information; S302, using 5G terminal equipment to collect terminal node network information; S303, using 5G terminal equipment to collect access node status information; S304. Collect flow status information using 5G terminal equipment.

Images