Adaptive base station quasi-network service data networking method based on reinforcement learning

Abstract

The invention relates to the technical field of communication, in particular to an adaptive base station quasi-network service data networking method based on reinforcement learning, which comprises the following steps: initializing parameters of a deep Q network, and acquiring state information of each base station in an initial state; the intelligent agent obtains a state vector; the intelligent agent selects and executes a proper action, and adjusts a topological structure of base station networking; then observing a new state after the action is executed, and calculating an obtained reward according to a reward function; the state, the action, the reward and the new state are stored in an experience playback buffer area for subsequent network training; according to the method, the change of the network environment can be sensed in real time, the topological structure of base station networking is autonomously adjusted through reinforcement learning, the continuously changing application requirements are met, and the flexibility and adaptability of the network are improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, specifically to an adaptive base station quasi-network service data networking method based on reinforcement learning. Background Technology

[0002] In current communication networks, regional cross-network converged services face numerous challenges. With the widespread adoption of various smart devices and the development of diversified applications, the network environment is becoming increasingly complex, placing higher demands on the flexibility and adaptability of base station networking. Traditional base station networking strategies are often based on pre-defined rules and fixed topologies, making it difficult to perceive and adapt to dynamic changes in the network environment in real time, such as changes in user distribution, fluctuations in service traffic, and the access of new services. This leads to problems such as high data transmission latency, unstable service quality, and low network resource utilization in regional cross-network converged services. Summary of the Invention

[0003] The purpose of this invention is to provide an adaptive base station quasi-network service data networking method based on reinforcement learning, which enables the autonomous establishment and adjustment of the topology of the base station quasi-network service data networking to better adapt to ever-changing application requirements.

[0004] To achieve the above objectives, the present invention provides the following technical solution: An adaptive base station quasi-network service data networking method based on reinforcement learning includes: (1) Initialization: Initialize the parameters of the deep Q network, including weights and biases, and initialize the experience replay buffer; at the same time, obtain the state information of each base station in the initial state as the input of the agent; (2) State acquisition: The agent observes the current state of the base station network in real time and acquires the state vector. State vector S represents the state vector S of the observed base station network at time t; (3) Model parameter optimization: The agent optimizes the model parameters according to the current state. Based on the output of the deep Q-network, select the appropriate action. ;action This represents the action vector for observing the base station network at time t. ; (4) Network strategy adjustment: Execute the selected action. Adjustments are made to the topology of the base station network, such as establishing or disconnecting connections and adjusting transmission parameters. (5) Execution of observation and reward calculation: the new state after the observation action is executed. And calculate the reward obtained based on the reward function. ; S represents the state vector S of the observed base station network at time t+1; The reward r represents the reward for observing the base station network at time t; (6) Experience storage: storing state ,action ,award and new status Stored in the experience replay buffer for subsequent network training; (7) Network state update: Randomly select a batch of experience samples from the experience replay buffer and update the network parameters using the training algorithm of deep Q network; During the training process, target network technology is used to stabilize the training process; (8) Repeated iteration: Repeat the above process of state observation, action selection, execution and reward acquisition, experience storage and network update until the topology of the base station network reaches the optimal or meets the preset termination conditions.

[0005] Preferably, the intelligent agent is constructed through a deep Q network (DQN), which is used to process the state input, action output and value assessment during the networking process of base station quasi-network service data; Preferably, the deep Q-network includes an input layer, a hidden layer, and an output layer. The input layer receives the status information of the base station network, the hidden layer performs feature extraction and processing on the status information, and the output layer outputs the Q-value of each possible action.

[0006] Preferably, the state of each base station is represented as a state vector. ,in This represents the various state parameters of the i-th base station, including its location coordinates. Received signal power Bandwidth utilization And the connection status with other base stations, the state vector of the i-th base station. ,in This indicates the connection status between base station i and base station j.

[0007] Preferably, each action is represented as an action vector. ,in This represents the j-th action; for example, establishing a connection between the k-th base station and the l-th base station can be represented by a specific action code; actions include establishing a new base station connection, disconnecting an existing base station connection, and adjusting the transmission parameters of the connection (such as transmission power and bandwidth allocation). Preferably, the action selection adopts an ε-greedy strategy. In order to balance exploration and utilization, an ε-greedy strategy is adopted, that is, under a certain probability ε, actions are randomly selected for exploration to discover new effective strategies; under a probability 1-ε, the action with the largest current Q value is selected for utilization to take advantage of the existing optimal strategy.

[0008] Preferably, the reward function is used to evaluate the impact of the agent's action on the base station networking effect, so as to guide the agent to learn the optimal networking strategy; the reward function is expressed as follows: Where ΔR represents the change in data transmission rate, ΔD represents the change in latency, and ΔL represents the change in load balancing. , , These are weighting coefficients, which can be adjusted according to actual application needs. The reward function is designed to comprehensively consider factors such as network transmission efficiency, service quality, and resource utilization. For example, when an agent takes an action, if the network's average data transmission rate increases, latency decreases, or load balancing improves, a positive reward is given; if network congestion, service interruption, or resource waste occurs, a negative reward is given.

[0009] Preferably, using target network technology to stabilize the training process includes setting up a target network and periodically copying the parameters of the current network into the target network for calculating the target Q value.

[0010] Preferably, the formula for calculating the target Q value is as follows: ,in This is a discount factor, representing the importance of future rewards. For the target network, These are the parameters of the target network. The parameters of the current network are updated by minimizing the mean squared error between the predicted Q-value and the target Q-value. Optimize the performance of deep Q-networks. This represents the target Q value at time t; Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Strong adaptability: It can perceive changes in the network environment in real time and autonomously adjust the topology of the base station network through reinforcement learning to adapt to constantly changing application requirements and improve the flexibility and adaptability of the network.

[0011] 2. Optimize network performance: Design a reward function that comprehensively considers factors such as network transmission efficiency, service quality, and resource utilization to guide the agent to learn the optimal networking strategy, thereby improving the overall network performance, reducing transmission latency, and increasing resource utilization.

[0012] 3. Stable and efficient training: By employing experience replay and target network techniques, sample correlation during training is reduced, the training process is stabilized, and the convergence speed and stability of the algorithm are improved, enabling it to be effectively applied in complex real-world network environments. Attached Figure Description

[0013] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0014] Figure 2 This is a structural diagram of the DQN algorithm of the present invention. Detailed Implementation

[0015] like Figure 1 As shown, a base station quasi-network service data networking method based on adaptive control theory and reinforcement learning specifically includes the following steps: 1) Constructing a deep Q-network intelligent agent Adopting such Figure 2 The deep Q-network structure shown includes an input layer, hidden layers, and an output layer. The number of neurons in the input layer is determined by the dimension of the state space. For example, if the state space includes the location coordinates (2D), signal strength (1D), load status (1D), and connection states with other base stations (assuming a maximum of 5 connections, each connection state being 1D), then for a network with n base stations, the dimension of the state space is n×(2+1+1+5)=n×9, and the number of neurons in the input layer is n×9. The hidden layers are fully connected layers. The first hidden layer contains 256 neurons, and the second hidden layer contains 128 neurons. The ReLU activation function is used. The number of neurons in the output layer is equal to the dimension of the action space. For example, if the action space includes establishing 10 possible connections, disconnecting 10 possible connections, and adjusting 5 transmission parameters, then the dimension of the action space is 25, the number of neurons in the output layer is 25, and the output layer outputs the Q-values ​​for each action.

[0016] 2) Define the state space, action space, and reward function. (1) State Space: Taking a network with 3 base stations as an example, the state vector of each base station includes its location coordinates (x, y), received signal power (rssi), bandwidth utilization (λ), and connection status with the other two base stations (1 for connected, 0 for disconnected). The state vector is then: , in This indicates the connection status between base station i and base station j.

[0017] (2) Action space: Actions include establishing a connection between base station 1 and base station 2, establishing a connection between base station 1 and base station 3, establishing a connection between base station 2 and base station 3, disconnecting the connection between base station 1 and base station 2, disconnecting the connection between base station 1 and base station 3, disconnecting the connection between base station 2 and base station 3, and adjusting the transmission power (high, medium and low levels) and bandwidth allocation (high, medium and low levels) of each connection. There are a total of 3 (establishing a connection) + 3 (disconnecting a connection) + 3×2 (adjusting transmission parameters) = 12 actions. The action vector is represented by one-hot encoding.

[0018] (3) Reward function: set weight coefficients , , Once a valid connection is established, if the average data transmission rate of the network increases by 10%, then ΔR = 10; if the latency decreases by 5%, then ΔD = 5; and if the load balancing improves by 8%, then ΔL = 8. The reward value r = 0.4⁻¹⁰ + 0.3⁻¹(-(-5)) + 0.3⁻¹⁸ = 4 + 1.5 + 2.4 = 7.9.

[0019] 3) Autonomous establishment and adjustment of network topology (1) Initialization: Randomly initialize the weights and biases of the deep Q network, and set the size of the experience replay buffer to 10000. Obtain the initial state, assuming that no connection is established between the three base stations at the beginning. The state vector is the initial position, signal strength, load status and disconnected state of each base station.

[0020] (2) State acquisition: The agent obtains the state vectors of the three base stations through the state perception module. .

[0021] (3) Model parameter optimization: An ε-greedy strategy is adopted, with ε=0.9 initially and gradually decaying to 0.1 as training progresses. In a certain iteration, ε=0.8, and actions are randomly selected with an 80% probability, such as choosing to establish a connection between base station 1 and base station 2; actions with the largest current Q value are selected with a 20% probability.

[0022] (4) Network strategy adjustment: Execute the action of establishing a connection between base station 1 and base station 2, and adjust the network topology.

[0023] (5) Perform observation and reward calculation: Observe the new state Calculate the reward value Assume that due to the increased transmission rate after the connection is established, a positive reward of 5 is obtained.

[0024] (6) Experience storage: will , , , It is stored in the experience playback buffer.

[0025] (7) Network state update: Extract a batch of empirical samples of size 32 from the empirical replay buffer and calculate the target Q value. ,in =0.95, the target network updates its parameters every 100 iterations. The parameters of the current network are optimized using the mean squared error loss function, with the Adam optimizer used and the learning rate set to 0.001.

[0026] (8) Repeated iteration: After multiple iterations of training, the agent can autonomously select the optimal action, establish and adjust the topology of the base station network to adapt to the ever-changing network environment.

[0027] The specific embodiment of the present invention has been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the embodiments described above. For those skilled in the art, various changes, modifications, substitutions, and variations made to these embodiments without departing from the principles and ideas of the present invention should still fall within the protection scope of the present invention.

Claims

1. A method for adaptive base station quasi-network service data networking based on reinforcement learning, characterized in that, include: (1) Initialization: Initialize the parameters of the deep Q network, including weights and biases, and initialize the experience replay buffer; at the same time, obtain the state information of each base station in the initial state as the input of the agent; (2) State acquisition: The agent observes the current state of the base station network in real time and acquires the state vector. ; (3) Model parameter optimization: The agent optimizes the model parameters according to the current state. Based on the output of the deep Q-network, select the appropriate action. ; (4) Network strategy adjustment: Execute the selected action. Adjustments were made to the topology of the base station network. (5) Execution of observation and reward calculation: the new state after the observation action is executed. And calculate the reward obtained based on the reward function. ; (6) Experience storage: storing state ,action ,award and new status Stored in the experience replay buffer for subsequent network training; (7) Network state update: Randomly select a batch of experience samples from the experience replay buffer and update the network parameters using the training algorithm of deep Q network; During the training process, target network technology is used to stabilize the training process; (8) Repeated iteration: Repeat the above process of state observation, action selection, execution and reward acquisition, experience storage and network update until the topology of the base station network reaches the optimal or meets the preset termination conditions.

2. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 1, characterized in that, The agent is constructed through a deep Q network and is used to process the state input, action output, and value assessment during the base station quasi-network service data networking process.

3. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 2, characterized in that, The deep Q-network includes an input layer, a hidden layer, and an output layer. The input layer receives the network status information of the base station, the hidden layer performs feature extraction and processing on the status information, and the output layer outputs the Q value of each possible action.

4. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 1, characterized in that, State vector ,in This represents the various state parameters of the i-th base station, including its location coordinates. Received signal power Bandwidth utilization And the connection status with other base stations, the state vector of the i-th base station. ,in This indicates the connection status between base station i and base station j.

5. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 1, characterized in that, Action vectors ,in This represents the j-th action; actions include establishing a new base station connection, disconnecting an existing base station connection, and adjusting the transmission parameters of the connection.

6. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 5, characterized in that, The action selection adopts an ε-greedy strategy.

7. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 1, characterized in that, The reward function is used to evaluate the impact of the agent's actions on the base station networking effect, so as to guide the agent to learn the optimal networking strategy; the reward function is expressed as follows: Where ΔR represents the change in data transmission rate, ΔD represents the change in latency, and ΔL represents the change in load balancing. , , These are weighting coefficients, which can be adjusted according to actual application needs.

8. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 1, characterized in that, Using target network techniques to stabilize the training process involves setting up a target network and periodically copying the parameters of the current network into the target network to calculate the target Q-value.

9. The adaptive base station quasi-network service data networking method based on reinforcement learning according to claim 8, characterized in that, Formula for calculating the target Q value ,in This is a discount factor, representing the importance of future rewards. For the target network, These are the parameters of the target network.

Images