A physical cell identification self-configuration method based on reinforcement learning

Abstract

The present invention is a physical cell identifier (PCI) self-configuration method based on reinforcement learning. The first step is to establish a system model, modeling the physical cell identifier (PCI) self-configuration problem as a Markov decision process, converting the physical problem into a mathematical derivation. The second step is to use Q learning to solve the PCI allocation problem in scenarios where the wireless network topology changes based on the modeling in the first step. The third step is to use dual Q learning to accelerate the convergence speed of solving the PCI allocation problem in scenarios where the wireless network topology changes based on the modeling in the first step. Compared with traditional PCI self-configuration methods, this method reduces the probability of PCI conflicts and confusion, and reduces the time overhead of updating PCI allocation schemes in scenarios where the future 6G wireless network topology frequently changes.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent wireless networks in wireless communications, and in particular relates to a physical cell identity (PCI) self-configuration method based on reinforcement learning. Background Art

[0002] In wireless communication networks, a Global Cell Identity (GCI) is used to uniquely identify a base station. It consists of an absolute Location Area Identity (LAI) and a relative Physical Cell Identity (PCI). The PCI distinguishes individual cells in the network and has a one-to-one correspondence with the frequency positions of the cell's orthogonal signals, reference signals, and pseudo-random signals. It is also related to the physical channel scrambling code. There are 504 PCIs in 4G systems, up to 1008 in 5G systems, and potentially even more in future 6G systems. However, with increasing network density and heterogeneity, the risk of PCI conflicts and confusion is increasing. PCI self-configuration is a core component of self-configuration technology. It requires the rapid and accurate allocation of PCIs to network cells based on known wireless network topology, following PCI allocation constraints and the principle of maximum reuse. PCI allocation constraints are divided into hard constraints and soft constraints. Hard constraints refer to situations that should be strictly avoided during the PCI allocation process. Violating these constraints may affect normal user communications during network operation. They mainly include ensuring no conflict and no confusion. Soft constraints refer to situations that should be avoided as much as possible during the PCI allocation process. They are higher-level requirements for PCI allocation from the perspective of preventing signal interference. They are mainly module constraints and are difficult to achieve in actual applications.

[0003] In future 6G wireless networks, the number and density of network elements will increase dramatically. Furthermore, the introduction of infrastructure such as drones and aerial platforms will enable mobile deployment based on user needs. This will lead to frequent changes in network topology, necessitating changes in the PCIs of each cell within the network to avoid conflicts and confusion. Previous research has rarely considered the potential PCI conflicts and confusion issues associated with frequent topological changes in wireless networks, and has also struggled to balance the time overhead of PCI reallocation with the degree of PCI reuse.

[0004] To address these issues, this paper proposes a reinforcement learning-based PCI self-configuration method. This method leverages the strengths of Q-learning (QL) to address PCI conflicts and confusion in scenarios where wireless network topologies frequently change, reducing the time required to reassign PCIs and making it suitable for practical communication applications. Summary of the Invention

[0005] Technical Problem: The purpose of this invention is to propose a reinforcement learning-based physical cell identifier self-configuration method. On the one hand, the wireless network PCI allocation problem is modeled as a Markov decision process (MDP), with the optimization goal of minimizing the probability of PCI conflicts and confusion. On the other hand, a Q-learning model is used to solve the PCI allocation optimization problem, and a dual-Q learning strategy is used to accelerate convergence. Compared with traditional PCI self-configuration methods, this method reduces the probability of PCI conflicts and confusion, and reduces the time overhead of updating PCI allocation schemes in future 6G wireless network topology scenarios with frequent changes.

[0006] Technical solution: The physical cell identity self-configuration method based on reinforcement learning of the present invention includes the following steps:

[0007] Step 1: Establish a system model. Model the PCI self-configuration problem as a Markov decision process, transforming the physical problem into a mathematical derivation.

[0008] Step 2: Based on the modeling in the first step, Q-learning is used to solve the PCI allocation problem in scenarios with changing wireless network topology. Q-learning is a reinforcement learning method based on the Markov decision process. First, the PCI allocation constraint is used to define the Q-learning reward function. Next, a two-dimensional Q-table is selected as the Q-learning policy function. The Q-table is a data structure that records the expected returns of possible actions for each state in a reinforcement learning environment. By defining a list of optional cells, the update range of the Q-table is limited to accelerate algorithm convergence. Then, a two-step action decision is performed to adjust the wireless network PCI allocation plan and calculate the reward. Finally, the Q-table is updated based on the reward value. The above steps are iterated to train Q-learning until convergence, gradually updating the wireless network PCI allocation plan. The PCI with the highest evaluation score for each cell in the final two-dimensional Q-table is selected as the final PCI allocation plan.

[0009] Step 3: Building on the modeling in Step 1, dual Q-learning is used to accelerate the convergence of the PCI allocation problem in scenarios with wireless network topology changes. First, the Q-learning reward function is defined using PCI allocation constraints. Two two-dimensional Q-tables are then used as the Q-learning policy function. A list of selectable cells is defined to limit the Q-table update range to accelerate algorithm convergence. A two-step action decision is then performed to adjust the wireless network PCI allocation plan and calculate the reward. Finally, the two Q-tables are updated based on the reward value. The Q-learning process is iteratively trained until convergence, gradually updating the wireless network PCI allocation plan. The PCI with the highest total evaluation score for each cell in the two final Q-tables is selected as the final PCI allocation plan. Dual Q-learning uses two Q-functions to learn the optimal value function from different experiences, avoiding the overestimation of the optimal value function that can occur with a single Q-function in complex random environments, thereby accelerating convergence. This step differs from Step 2 only in terms of Q-table design and Q-table update; the remaining steps are identical to those in Step 2.

[0010] The first step, system model establishment, is specifically as follows:

[0011] The physical cell identifier PCI self-configuration problem is modeled as a Markov decision process, which corresponds one-to-one to each element of the Markov decision process MDP. Specifically, the agent in the MDP corresponds to the wireless network operation and maintenance management system OAM, which controls the allocation of PCIs to each base station in the wireless network. The environment corresponds to the wireless communication network. The current network should have a certain neighbor relationship and a feasible PCI allocation scheme under the current wireless network topology. The agent will allocate PCIs to base stations newly added to the network based on the current PCI allocation scheme according to changes in the network topology and adjust the PCIs of existing base stations in the network. The state corresponds to the PCI allocation scheme of the current network. The action decision is designed to be divided into two steps. The first step is to select the base station BS where the PCI adjustment occurs. c , is it a new base station moved into the network or an existing base station in the network; the second step is to adjust the selected base station BS c The agent will make action decisions based on the current environment. Action decisions will cause the wireless network to change the PCI of a base station based on the current PCI allocation plan. Each action actually represents a PCI allocation plan. The reward is defined as a score for the action decision of allocating PCI according to the PCI allocation constraints. The reward will guide the agent to optimize the PCI allocation strategy and accumulate as a reward over time, leading the agent to ultimately solve the optimal PCI allocation plan for the wireless network.

[0012] The second step includes the following process:

[0013] Step 2.1, define the reward function. The reward function is the return, which consists of the immediate reward of the action decision in the current state and all subsequent persistent rewards, as shown below:

[0014] G t =R t+1 +γR t+2 +γ 2 R t+3 +...=R t+1 +γG t+1 (1)

[0015] Among them G t represents the reward obtained from time t, R t+1 Indicates making action decision A t The next moment state S t+1 The reward, return G t and reward R t All are expressed in evaluation scores, and the reward R t+1 It represents the evaluation score of the wireless network PCI allocation scheme after adjustment at time t, and the return G t The cumulative evaluation score obtained by continuously adjusting the wireless network PCI allocation plan starting from time t is a number related to the number of network nodes that violate the PCI allocation constraints. The specific calculation method is defined in step 2.6. Its value only reflects the degree of match between the current PCI allocation plan and the PCI allocation constraints. γ is the discount factor, a number between 0 and 1, whose value is determined by experimental adjustment. It helps PCI allocation focus more on the rationality of the current plan. Time t only represents the state transition time and has nothing to do with the actual operation time of the wireless network.

[0016] The objective function of PCI self-configuration is maxG t , that is, the decision with the largest reward is considered to be the optimal PCI allocation scheme;

[0017] Step 2.2, define the policy function and select the Q table as the random policy function π of the PCI self-configuration method. For the two-step action strategy, a complete action decision only changes the PCI of one base station in the network. Use the two-dimensional Q table to record the cumulative reward of each PCI allocated to each base station in the network; at time t, the two-dimensional Q table only reflects the current network PCI allocation scheme, that is, the state S t , the reward in the Q table is calculated by t Iterative update: After the iteration is completed, the PCI with the highest return on each base station is selected as the final PCI allocation scheme. The horizontal header in the Q table represents the optional PCI index, and the vertical header represents the base station index. The return value in the table has no practical meaning and only indicates the preference of each base station in the network for selecting different PCIs.

[0018] Step 2.3, initialize the Q table and the optional cell list List, read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, for the original base station set {BS original}, set a positive integer q in the corresponding PCI position in the Q table. The value of the positive integer q is determined by the proportion of base stations newly added to the network. The remaining PCI positions are initialized to 0. For the new base station set {BS new}, all its PCI positions in the Q table are initialized to 0;

[0019] Initialize the optional cell list List to List = {BS new}, the base stations in the list will adjust PCI first;

[0020] Step 2.4, the action decision A t Split into two steps; execute the first step action decision; at time t, action decision A t First, randomly select the base station BS that needs to adjust the PCI from the list c ;

[0021] Step 2.5, execute the second step action decision: at time t, action decision A t In the selected BS c Then adjust BS according to the ε greedy strategy c PCI; ε greedy strategy is an action selection strategy that allows the agent to balance between exploration and utilization. The agent randomly gives BS with probability ε c Select a PCI and give it to BS with probability 1-ε c Select the PCI with the highest return in the Q table;

[0022] Step 2.6, calculate A t The reward is calculated according to formula (2) at time t, the action decision A t In state S t The action reward R obtained t , where r represents the basic reward, p conflict Indicates A t PCI conflict penalty caused, p confuse Indicates A t PCI confusion penalty caused, p modm Indicates A t The above parameters are all positive integers, and their values ​​are determined by the network's tolerance for the PCI allocation scheme.

[0023] R t =rp conflict -p confuse -p modm (2)

[0024] Step 2.7, update the Q table, and update the BS in the Q table according to formula (3). c PCI return value of the corresponding column;

[0025]

[0026] where Q t+1 (BS c ,PCI c ) indicates the Q table corresponding to the base station BS at time t+1 c Allocate PCI c The state return, Q t (BS c ,PCI c ) represents the Q table corresponding to the base station BS at time t c Allocate PCI c Status return, Indicates the corresponding base station BS on the Q table at time t c Assign the optimal state return of the optimal PCI, α represents the learning rate, and its value is determined by experimental adjustment to obtain the fastest convergence speed, γ represents the discount factor, R t Represents the action decision A calculated according to formula (2) t Action rewards;

[0027] Step 2.8, evaluate action decision A t ,

[0028] If R t ≥0, the current PCI adjustment is retained, that is, the state is transferred from the wireless network PCI allocation scheme S at time t t , transfer to the PCI allocation scheme S at time t+1 t+1 ;

[0029] If R t <0, only the Q table is updated, no state transfer occurs, and BS c Add the neighboring base stations to the available cell list List and repeat steps 2.5 to 2.8;

[0030] Step 2.9: Repeat steps 2.4 to 2.8 within the set number of training iterations N; after each round of training, calculate the number of violations of constraint C by the current network PCI allocation scheme. conflict 、C confuse and C modm If the probability does not change within M consecutive iterations, the training is considered to have converged and the training is terminated early.

[0031] In step 2.10, based on the final Q table, the PCI with the highest value on each base station is output as the final wireless network PCI allocation solution.

[0032] The PCI allocation constraints are divided into three types: no conflict, no confusion and different modules m. The no conflict constraint C conflict It means that two neighboring base stations cannot be assigned the same PCI, otherwise users in the common coverage area of ​​the two base stations will not be able to identify the serving base station; no confusion constraint C confuse It means that two different neighboring base stations of a base station cannot be assigned the same PCI, otherwise the service users of the base station will not be able to identify the target base station during the handover process; the module m is not the same constraint C modm It means that the PCI of two neighboring base stations divided by a positive integer m should have different remainders, where m is determined by the sequence of the reference signal. There is a one-to-one correspondence between PCI and reference signal. If the reference signal is composed of m base sequences, then the different remainders of the PCI of adjacent base stations modulo m can reduce the interference of the reference signal. For the action decision A at time t, t After that, check whether all base stations in the network comply with the PCI allocation constraints. If A t If the probability of PCI conflicts, confusion and modulo m congruence events in the network is reduced, the reward R t+1 Is a positive number; if A t If the PCI allocation constraint is violated, then R t+1 The penalty value should be subtracted based on the impact on the operation of the wireless network, and the violation of the hard constraint C conflict and C confuse The penalty value for violating the soft constraint C should be set higher than modm .

[0033] The third step is: Dual Q learning uses two Q functions to learn the optimal value function from different experiences, avoiding the problem of "overestimation" of the optimal value function that may occur with a single Q function in a complex random environment, and accelerating the convergence speed. This step is different from the second step only in the Q table design and Q table update part, and the rest of the steps are the same as the second step.

[0034] Step 3.1 to step 3.2 are the same as steps 2.1 to step 2.2;

[0035] Step 3.3, initialize the Q table and the optional cell list List;

[0036] Read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, and initialize two Q tables, respectively denoted as Q A and Q B ; For the original base station set {BS original}, set a positive integer q to the corresponding PCI position in the Q table, and initialize the remaining PCI positions to 0; for the new base station set {BS new}, initialize all PCI positions in the Q table to 0; initialize the optional cell list List to List = {BS new}, the base stations in the list will adjust PCI first;

[0037] Steps 3.4 to 3.6 are the same as steps 2.4 to 2.6;

[0038] Step 3.7, update the Q table,

[0039] Update BS in Q table according to formula (4) and (5) c PCI return value of the corresponding column;

[0040]

[0041] where Q A t+1 (BS c ,PCI c ) represents Q at time t+1 A The table corresponds to the base station BS c Allocate PCI c The state return, Q A t (BS c ,PCI c ) represents Q at time t A The table corresponds to the base station BS c Allocate PCI c Status return, represents Q at time t A The corresponding base station BS in the table c The optimal state reward for allocating the optimal PCI, Q B t+1 (BS c ,PCI c ) represents Q at time t+1 B The table corresponds to the base station BS c Allocate PCI c The state return, Q B t (BS c ,PCI c ) represents Q at time t B The table corresponds to the base station BS c Allocate PCI c Status return, represents Q at time t B The corresponding base station BS in the table cThe optimal state return of the optimal PCI is assigned; α represents the learning rate, γ represents the discount factor, R t Represents the action decision A calculated according to formula (2) t Action rewards;

[0042] Steps 3.8 to 3.10 are the same as steps 2.8 to 2.10.

[0043] Beneficial Effects: The reinforcement learning-based PCI self-configuration method proposed in this invention has the following advantages: Compared with traditional PCI self-configuration methods, this invention uses Q-learning to reduce the probability of PCI conflicts and confusion in scenarios where wireless network topology changes, while maintaining the overall PCI occupancy by adhering to the principle of maximum reuse. In addition, this invention uses dual Q-learning to further improve the speed of PCI allocation. DETAILED DESCRIPTION

[0044] Step one: Establishing the system model.

[0045] The PCI self-configuration problem is modeled as a Markov decision process, which corresponds to each element of the MDP one-to-one. Specifically, the agent in the MDP corresponds to the wireless network operation and maintenance management system (OAM), which controls the allocation of PCIs to each base station in the wireless network. The environment corresponds to the wireless communication network. The current network should have a certain neighbor relationship and a PCI allocation scheme that is feasible under the current wireless network topology. The agent will allocate PCIs to base stations newly added to the network based on the current PCI allocation scheme according to changes in the network topology and adjust the PCIs of existing base stations in the network. The action decision is designed to be divided into two steps. The first step is to select the base station BS where the PCI adjustment occurs. c , which may be a new base station moved into the network or an existing base station in the network. The second step is to adjust the selected base station BS c The agent will make action decisions based on the current environment. These decisions will cause the wireless network to change the PCI of a base station based on the current PCI allocation plan. Each action actually represents a PCI allocation plan. Rewards are defined as the scores assigned to PCI allocation action decisions based on the PCI allocation constraints. Rewards guide the agent in optimizing its PCI allocation strategy and accumulate over time as rewards, ultimately guiding the agent to find the optimal PCI allocation plan for the wireless network.

[0046] Step 2: Based on the modeling in the first step, Q learning is used to solve the PCI allocation problem under the scenario of wireless network topology changes.

[0047] This step includes the following processes:

[0048] Step 2.1, define the reward function.

[0049] The reward function is the return, which consists of the immediate reward for the action decision in the current state and all subsequent persistent rewards, as shown below:

[0050] G t =R t+1 +γR t+2 +γ 2 R t+3 +...=R t+1 +γG t+1 (6)

[0051] Among them G t represents the reward obtained from time t, R t+1 Indicates making action decision A t The next moment state S t+1 The reward, return G t and reward R t All are expressed in evaluation scores, and the reward R t+1 It represents the evaluation score of the wireless network PCI allocation scheme after adjustment at time t, and the return G t The cumulative evaluation score obtained by continuously adjusting the wireless network PCI allocation plan starting at time t is a number related to the number of network nodes violating the PCI allocation constraints. The specific calculation method is defined in step 2.6. Its value only reflects the degree to which the current PCI allocation plan matches the PCI allocation constraints. γ represents the discount factor, a number between 0 and 1, whose value is determined through experimental adjustments. It helps PCI allocation focus more on the rationality of the current plan. Time t only represents the state transition moment and is unrelated to the actual operation time of the wireless network.

[0052] PCI allocation constraints are divided into three types: no conflict, no confusion, and modulo 30 non-remainder. conflict It means that two neighboring base stations cannot be assigned the same PCI, otherwise users in the common coverage area of ​​the two base stations will not be able to identify the serving base station; no confusion constraint C confuse It means that two different neighboring base stations of a base station cannot be assigned the same PCI, otherwise the service users of the base station will not be able to identify the target base station during the handover process; the modulo 30 non-complementary constraint C mod30 The PCI of two neighboring base stations should have different remainders when divided by 30, where 30 is determined by the sequence of the reference signal. The uplink demodulation reference signal is composed of 30 different base sequences. The PCI and the reference signal have a one-to-one correspondence. The PCI of adjacent base stations must have different remainders modulo 30. mod30Reference signal interference can be reduced, and this constraint is possible in practical applications. t After that, check whether all base stations in the network comply with the PCI allocation constraints. If A t The probability of PCI conflict, confusion and modulo 30 congruence events in the network is reduced, so the reward R is set t+1 is 2; if A t If the PCI allocation constraint is violated, then R t+1 The penalty value should be subtracted based on the impact on the operation of the wireless network, and the violation of the hard constraint C conflict and C confuse The penalty value is set to 1, violating the soft constraint C mod30 The penalty value is set to 0.5.

[0053] The objective function of PCI self-configuration is maxG t , that is, the decision with the largest reward is considered to be the optimal PCI allocation scheme.

[0054] Step 2.2, define the policy function.

[0055] The Q table is selected as the random strategy function π of the PCI self-configuration method. For the two-step action strategy, a complete action decision only changes the PCI of one base station in the network. A two-dimensional Q table is used to record the cumulative reward of each base station in the network for each PCI. At time t, the two-dimensional Q table only reflects the current network PCI allocation scheme, that is, the state S t , the rewards in the table are calculated by reward R t The iterative update process selects the PCI with the highest reported value from each base station as the final PCI allocation scheme. The horizontal header in the Q-table represents the available PCI index, while the vertical header represents the base station index. The reported values ​​in the table are meaningless and only indicate the preference of each base station in the network for different PCIs.

[0056] Step 2.3: Initialize the Q table and the optional cell list List.

[0057] Read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, and for the original base station set {BS original}, the corresponding PCI position in the Q table is set to 5, and the other PCI positions are initialized to 0; for the new base station set {BS new}, with all its PCI positions in the Q table initialized to 0.

[0058] Initialize the optional cell list List to List = {BS new}, the base stations in the list will adjust the PCI first.

[0059] Step 2.4, perform the first action decision.

[0060] At time t, action decision A t First, randomly select the base station BS that needs to adjust the PCI from the list c .

[0061] Step 2.5, perform the second action decision.

[0062] At time t, action decision A t Adjust BS according to ε greedy strategy c The ε greedy strategy is an action selection strategy that allows the agent to balance exploration and exploitation. The agent randomly gives BS c Select a PCI and give it to BS with probability 1-ε c Select the PCI with the highest reward in the Q table and set ε = 0.1.

[0063] Step 2.6, calculate A t rewards.

[0064] According to formula (7), at time t, the action decision A t In state S t The action reward R obtained t , where r represents the basic reward, r = 2, p conflict Indicates A t PCI conflict penalty caused by p conflict =1, p confuse Indicates A t PCI confusion penalty caused, p confuse =1, p mod30 Indicates A t PCI mode 30 conflict penalty caused by mod30 =0.5.

[0065] R t =rp conflict -p confuse -p mod30 (7)

[0066] Step 2.7, update the Q table.

[0067] Update BS in Q table according to formula (8) c The PCI return value of the corresponding column.

[0068]

[0069] where Q t+1 (BS c ,PCIc ) indicates the Q table corresponding to the base station BS at time t+1 c Allocate PCI c The state return, Q t (BS c ,PCI c ) represents the Q table corresponding to the base station BS at time t c Allocate PCI c Status return, Indicates the corresponding base station BS on the Q table at time t c Assign the optimal state return of the optimal PCI, α represents the learning rate, α = 0.1, γ represents the discount factor, γ = 0.9, and the values ​​of α and γ are determined by experimental adjustment to obtain the fastest convergence speed, R t Denotes the action decision A calculated according to formula (7) t action rewards.

[0070] Step 2.8, evaluate action decision A t .

[0071] If R t ≥0, the current PCI adjustment is retained, that is, the state is transferred from the wireless network PCI allocation scheme S at time t t , transfer to the PCI allocation scheme S at time t+1 t+1 .

[0072] If R t <0, only the Q table is updated, no state transfer occurs, and BS c Add the neighboring base station to the available cell list List and repeat steps 2.5 to 2.8.

[0073] Step 2.9: Repeat steps 2.4 to 2.8 within the set training iteration number N = 100. After each round of training, calculate the number of violations of the current network PCI allocation scheme C conflict 、C confuse and C mod30 If the probability does not change within M = 5 consecutive iterations, the training is considered to have converged and terminated early.

[0074] In step 2.10, based on the final Q table, the PCI with the highest value on each base station is output as the final wireless network PCI allocation solution.

[0075] Step 3: Based on the modeling in the first step, dual-Q learning is used to accelerate the convergence speed of solving the PCI allocation problem in the scenario of wireless network topology changes.

[0076] Dual Q learning uses two Q functions to learn the optimal value function from different experiences. This avoids the overestimation of the optimal value function that can occur with a single Q function in complex random environments, accelerating convergence. This step differs from the second step only in terms of Q-table design and update; the rest of the steps are identical.

[0077] This step includes the following processes:

[0078] Steps 3.1 to 3.2 are the same as steps 2.1 to 2.2.

[0079] Step 3.3: Initialize the Q table and the optional cell list List.

[0080] Read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, and initialize two Q tables, respectively denoted as Q A and Q B For the original base station set {BS original}, set a positive integer q to the corresponding PCI position in the Q table, and initialize the remaining PCI positions to 0; for the new base station set {BS new}, and initialize all PCI positions in the Q table to 0. Initialize the optional cell list List to List = {BS new}, the base stations in the list will adjust the PCI first.

[0081] Steps 3.4 to 3.6 are the same as steps 2.4 to 2.6.

[0082] Step 3.7, update the Q table.

[0083] Update BS in Q table according to formula (4) and (5) c The PCI return value of the corresponding column.

[0084]

[0085]

[0086] where Q A t+1 (BS c ,PCI c ) represents Q at time t+1 A The table corresponds to the base station BS c Allocate PCI c The state return, Q A t (BS c ,PCI c ) represents Q at time t A The table corresponds to the base station BS c Allocate PCIc Status return, represents Q at time t A The corresponding base station BS in the table c Assign the optimal state reward of the optimal PCI to Q B The same is true for the table. α represents the learning rate, α = 0.1, γ represents the discount factor, γ = 0.9, and the values ​​of α and γ are determined by dynamic adjustment of the experiment to obtain the fastest convergence speed. t Denotes the action decision A calculated according to formula (7) t action rewards.

[0087] Steps 3.8 to 3.10 are the same as steps 2.8 to 2.10.

[0088] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A physical cell identification self-configuration method based on reinforcement learning, characterized in that The method comprises the following steps: Step 1: Establish a system model. Model the physical cell identifier (PCI) self-configuration problem as a Markov decision process (MDP), transforming the physical problem into a mathematical derivation. Step 2: Based on the modeling in the first step, Q-learning is used to solve the PCI allocation problem in scenarios where the wireless network topology changes. Q-learning is a reinforcement learning method based on the Markov decision process (MDP). First, the PCI allocation constraint is used to define the Q-learning reward function. Next, a two-dimensional Q-table is selected as the Q-learning policy function. The Q-table is a data structure that records the expected returns of possible actions for each state in the reinforcement learning environment. By defining a list of optional cells, the update range of the Q-table is limited to accelerate algorithm convergence. Then, a two-step action decision is performed to adjust the wireless network PCI allocation plan and calculate the reward. Finally, the Q-table is updated based on the reward value. The above steps are iterated to train Q-learning until convergence, gradually updating the wireless network PCI allocation plan. The PCI with the highest evaluation score for each cell in the final two-dimensional Q-table is selected as the final PCI allocation plan. The specific process includes the following: Step 2.1, define the reward function. The reward function is the return, which consists of the immediate reward of the action decision in the current state and all subsequent persistent rewards, as shown below: G t =R t+1 +γR t+2 +γ 2 R t+3 +...=R t+1 +γG t+1 (1) Among them G t represents the reward obtained from time t, R t+1 Indicates making action decision A t The next moment state S t+1 The reward, return G t and reward R t All are expressed in evaluation scores, and the reward R t+1 It represents the evaluation score of the wireless network PCI allocation scheme after adjustment at time t, and the return G t represents the cumulative evaluation score obtained by continuously adjusting the wireless network PCI allocation plan starting from time t. The evaluation score is a number related to the number of network nodes that violate the PCI allocation constraints. Its value only reflects the degree of match between the current PCI allocation plan and the PCI allocation constraints. γ represents the discount factor, a number between 0 and 1. Its value is determined by experimental adjustment and helps PCI allocation pay more attention to the rationality of the current plan. Time t only represents the state transition moment. The objective function of PCI self-configuration is maxG t , that is, the decision with the largest reward is considered to be the optimal PCI allocation scheme; Step 2.2, define the policy function and select the Q table as the random policy function π of the PCI self-configuration method. For the two-step action strategy, a complete action decision only changes the PCI of one base station in the network. Use the two-dimensional Q table to record the cumulative reward of each PCI allocated to each base station in the network; at time t, the two-dimensional Q table only reflects the current network PCI allocation scheme, that is, the state S t , the reward in the Q table is calculated by t Iterative update: After the iteration is completed, the PCI with the highest return on each base station is selected as the final PCI allocation scheme. The horizontal header in the Q table represents the optional PCI index, and the vertical header represents the base station index. The return value in the table only represents the preference of each base station in the network for selecting different PCIs. Step 2.3, initialize the Q table and the optional cell list List, read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, for the original base station set {BS original }, set a positive integer q in the corresponding PCI position in the Q table. The value of the positive integer q is determined by the proportion of base stations newly added to the network. The remaining PCI positions are initialized to 0. For the new base station set {BS new }, initialize all its PCI positions in the Q table to 0; Initialize the optional cell list List to List = {BS new }, the base stations in the list will adjust PCI first; Step 2.4, the action decision A t Split into two steps; execute the first step action decision; at time t, action decision A t First, randomly select the base station BS that needs to adjust the PCI from the list c ; Step 2.5, execute the second step action decision: at time t, action decision A t In the selected BS c Then adjust BS according to the ε greedy strategy c PCI; ε greedy strategy is an action selection strategy that allows the agent to balance between exploration and utilization. The agent randomly gives BS with probability ε c Select a PCI and give it to BS with probability 1-ε c Select the PCI with the highest return in the Q table; Step 2.6, calculate A t The reward is calculated according to formula (2) at time t, the action decision A t In state S t The action reward R obtained t , where r represents the basic reward, p conflict Indicates A t PCI conflict penalty caused by p confuse Indicates A t PCI confusion penalty caused, p modm Indicates A t The parameters of this step are all positive integers, and their values ​​are determined by the network's tolerance for the PCI allocation scheme. R t =r-p conflict -p confuse -p modm (2) Step 2.7, update the Q table, and update the BS in the Q table according to formula (3). c PCI return value of the corresponding column; where Q t+1 (BS c ,PCI c ) indicates the Q table corresponding to the base station BS at time t+1 c Allocate PCI c Status return, Q t (BS c ,PCI c ) represents the Q table corresponding to the base station BS at time t c Allocate PCI c Status return, Indicates the base station BS corresponding to the Q table at time t c Assign the optimal state return of the optimal PCI, α represents the learning rate, and its value is determined by experimental adjustment to obtain the fastest convergence speed, γ represents the discount factor, R t Represents the action decision A calculated according to formula (2) t Action rewards; Step 2.8, evaluate action decision A t , If R t ≥0, the current PCI adjustment is retained, that is, the state is transferred from the wireless network PCI allocation scheme S at time t t , transfer to the PCI allocation scheme S at time t+1 t+1 ; If R t <0, only the Q table is updated, no state transfer occurs, and BS c Add the neighboring base stations to the available cell list List and repeat steps 2.5 to 2.8; Step 2.9: Repeat steps 2.4 to 2.8 within the set number of training iterations N; after each round of training, calculate whether the current network PCI allocation scheme violates the conflict-free constraint C. conflict , no confusion constraint C confuse and modulo m are not identical constraints C modm If the probability does not change within M consecutive iterations, the training is considered to have converged and the training is terminated early. In step 2.10, based on the final Q table, the PCI with the highest value on each base station is output as the final wireless network PCI allocation solution.

2. The physical cell identification self-configuration method based on reinforcement learning according to claim 1 is characterized in that The first step, system model establishment, is specifically as follows: The physical cell identifier (PCI) self-configuration problem is modeled as a Markov decision process, with a one-to-one correspondence between each element of the Markov decision process (MDP). Specifically, the agent in the MDP corresponds to the wireless network operation and maintenance management system (OAM), which controls the PCI allocation of each base station in the wireless network. The environment corresponds to the wireless communication network. The current network should have a certain neighbor relationship and a feasible PCI allocation scheme under the current wireless network topology. The agent will allocate PCIs to newly added base stations based on the current PCI allocation scheme and adjust the PCIs of existing base stations in the network according to changes in the network topology. The state corresponds to the current network PCI allocation scheme. The action decision is designed to be divided into two steps. The first step is to select the base station BS where the PCI adjustment occurs. c , the BS c Is it a new base station moved into the network or an existing base station in the network? The second step is to adjust the selected base station BS c The agent will make action decisions based on the current environment. Action decisions will cause the wireless network to change the PCI of a base station based on the current PCI allocation plan. Each action represents a PCI allocation plan. The reward is defined as a score for the action decision of allocating PCI according to the PCI allocation constraints. The reward will guide the agent to optimize the PCI allocation strategy and accumulate as a reward over time, leading the agent to ultimately solve the optimal PCI allocation plan for the wireless network.

3. The physical cell identification self-configuration method based on reinforcement learning according to claim 2 is characterized in that The method uses Q learning to solve the PCI allocation problem in a scenario where the wireless network topology changes, and uses dual Q learning to accelerate the convergence speed of solving the PCI allocation problem in a scenario where the wireless network topology changes. First, the PCI allocation constraint is used to define the Q learning reward function. Then, two two-dimensional Q tables are used as the policy function of Q learning. The update range of the Q table is limited by defining an optional cell list to accelerate the convergence of the algorithm. Then, a two-step action decision is performed to adjust the wireless network PCI allocation plan and calculate the reward. Finally, the two Q tables are updated according to the reward value. The above steps are iterated to train Q learning until convergence, and the wireless network PCI allocation plan is gradually updated. The PCI with the highest total evaluation score for each cell in the two final Q tables is selected as the final PCI allocation plan. Double Q-learning uses two Q-functions to learn the optimal value function from different experiences.

4. The physical cell identification self-configuration method based on reinforcement learning according to claim 3 is characterized in that The PCI allocation constraints are divided into three types: no conflict, no confusion and different modules m. The no conflict constraint C conflict It means that two neighboring base stations cannot be assigned the same PCI, and there is no confusion constraint C confuse It means that two different neighboring base stations of a base station cannot be assigned the same PCI, and the module m is not the same constraint C modm It means that the PCI of two neighboring base stations divided by a positive integer m should have different remainders, where m is determined by the sequence of the reference signal. There is a one-to-one correspondence between PCI and reference signal. If the reference signal is composed of m base sequences, the PCI of adjacent base stations modulo m is different to reduce the reference signal interference; make action decision A at time t t After that, check whether all base stations in the network comply with the PCI allocation constraints. If A t If the probability of PCI conflicts, confusion and modulo m congruence events in the network is reduced, the reward R t+1 Is a positive number; if A t If the PCI allocation constraint is violated, then R t+1 The penalty value should be subtracted based on the impact on the operation of the wireless network, violating the conflict-free constraint C conflict and the no-confusion constraint C confuse The penalty value of should be set higher than that of violating the modulo m difference constraint C modm The penalty value.

5. The physical cell identification self-configuration method based on reinforcement learning according to claim 3 is characterized in that The third step is: dual Q learning uses two Q functions to learn the optimal value function from different experiences, and the rest are the same as the second step; Step 3.1 to step 3.2 are the same as steps 2.1 to step 2.2; Step 3.3, initialize the Q table and the optional cell list List; Read the existing network PCI allocation plan, initialize the Q table after the wireless network topology changes, and initialize two Q tables, respectively denoted as Q A and Q B ; For the original base station set {BS original }, set a positive integer q to the corresponding PCI position in the Q table, and initialize the remaining PCI positions to 0; for the new base station set {BS new }, initialize all PCI positions in the Q table to 0; initialize the optional cell list List to List = {BS new }, the base stations in the list will adjust PCI first; Steps 3.4 to 3.6, for Q A and Q B The execution steps are the same as steps 2.4 to 2.6; Step 3.7, Update Q A and Q B surface, Update BS in Q table according to formula (4) and (5) c PCI return value of the corresponding column; where Q A t+1 (BS c ,PCI c ) represents Q at time t+1 A The table corresponds to the base station BS c Allocate PCI c The state return, Q A t (BS c ,PCI c ) represents Q at time t A The table corresponds to the base station BS c Allocate PCI c Status return, represents Q at time t A The corresponding base station BS in the table c The optimal state reward for allocating the optimal PCI, Q B t+1 (BS c ,PCI c ) represents Q at time t+1 B The table corresponds to the base station BS c Allocate PCI c The state return, Q B t (BS c ,PCI c ) represents Q at time t B The table corresponds to the base station BS c Allocate PCI c Status return, represents Q at time t B The corresponding base station BS in the table c The optimal state return of the optimal PCI is assigned; α represents the learning rate, γ represents the discount factor, R t Represents the action decision A calculated according to formula (2) t Action rewards; Steps 3.8 to 3.10, for Q A and Q B The execution steps are the same as steps 2.8 to 2.10.

Images