A dual-q network-based dynamic optimization method and system for frequency switching threshold

Abstract

This invention relates to a method and system for dynamic optimization of inter-frequency handover thresholds based on a dual-Q network. The method identifies problematic cells based on acquired real-time MRO data, real-time OMC data, and real-time threshold data. An optimization request is sent to the dual-Q network, along with the current state vector of the problematic cell. The Q1 network selects preferred actions based on the current state vector, and the Q2 network performs a long-term value evaluation on these actions. If the long-term value does not meet the expected results, the Q1 network is dynamically updated based on this value. The updated Q1 network then dynamically updates the Q2 network. The updated Q2 network performs a long-term value evaluation on the new preferred actions selected by the updated Q1 network until the new long-term value meets the expected results. Finally, the corresponding new preferred action is executed to achieve the inter-frequency handover threshold for the problematic cell. Therefore, this invention improves network coverage quality and user experience.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method and system for dynamic optimization of inter-frequency switching thresholds based on dual-Q networks. Background Technology

[0002] In 4G / 5G heterogeneous networks, inter-frequency handover relies on the A2 / A3 / A5 event triggering mechanism. The specific triggering rules are as follows: A2 event: neighbor cell measurement is triggered when the signal quality of the serving cell is lower than a preset threshold; A3 event: handover is triggered when the signal quality of the neighboring cell is better than that of the serving cell by a certain offset; A5 event: handover is triggered when the signal quality of the serving cell is lower than threshold 1 and the signal quality of the neighboring cell is higher than threshold 2. That is, the existing inter-frequency handover mainly relies on manual static configuration, which cannot adapt to dynamic changes in the network and affects network coverage quality and user experience. Summary of the Invention

[0003] The technical problem to be solved by the present invention is: to provide a method and system for dynamic optimization of inter-frequency handover threshold based on dual-Q network, so that the inter-frequency handover threshold can adapt to complex network dynamic changes, thereby improving network coverage quality and user experience.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network, comprising:

[0006] Acquire real-time MR data, real-time OMC data, and real-time threshold data. Determine whether there are problem cells based on the real-time MR data, real-time OMC data, and real-time threshold data. If there are, send an optimization request to the dual-Q network and simultaneously send the current state vector of the problem cell to the dual-Q network. The current state vector includes: current weak coverage percentage, current weak coverage strong neighbor cell percentage, current PRB utilization, A5 threshold, A3 threshold, forward CIO, and reverse CIO. The dual-Q network includes Q1 network and Q2 network.

[0007] The dual-Q network receives the optimization request and the current state vector. The Q1 network selects the preferred action based on the current state vector. The Q2 network performs a long-term value evaluation on the preferred action to obtain the long-term evaluation value.

[0008] If the long-term evaluation value does not meet the expected results, the Q1 network is dynamically updated based on the long-term evaluation value to obtain an updated Q1 network. New preferred actions are then selected through the updated Q1 network. The Q2 network is dynamically updated based on the new preferred actions and the updated Q1 network to obtain an updated Q2 network. The long-term value of the new preferred actions is then re-evaluated through the updated Q2 network to obtain new long-term evaluation values ​​until the new long-term evaluation value meets the expected results. Finally, the corresponding new preferred actions are executed to achieve the inter-frequency handover threshold for the problematic cell.

[0009] The beneficial effects of this invention are as follows: Problem cells are identified based on real-time MR data, real-time OMC data, and real-time threshold data, i.e., problem cells are identified based on multi-source data, improving the accuracy of problem cell identification. A dual-Q network structure is used, employing a Q1 network to select preferred actions, and then a Q2 network to perform long-term value evaluation of these preferred actions, avoiding overestimation bias in the preferred actions selected by the Q1 network. Furthermore, when selecting preferred actions, the Q1 network considers the current state vectors, including the current weak coverage ratio, the current weak coverage strong neighbor ratio, the current PRB utilization rate, the A5 threshold, the A3 threshold, and the positive and negative CIO, taking into account complex network dynamic changes and improving the accuracy of the selected preferred actions. To ensure accuracy, and for preferred actions that fail to achieve the expected results in long-term value assessment, the Q1 network will be updated based on the long-term value assessment, new preferred actions will be selected, and the Q2 network will be updated based on the new preferred actions and the updated Q1 network. The long-term value of the new preferred actions will then be reassessed using the updated Q2 network until a new preferred action is obtained that corresponds to the expected results in long-term value assessment. While updating the Q1 network, the Q2 network is also updated using the updated Q1 network and the new preferred actions, thereby improving the accuracy of the calculated long-term value assessment and ensuring the accuracy of the new preferred actions. This improves the network coverage quality and user experience of the problematic cells after the preferred actions are implemented.

[0010] Optionally, determining whether a problematic cell exists based on the real-time MR data, the real-time OMC data, and the real-time threshold data includes:

[0011] The uplink signal interference-to-noise ratio and power margin report for each serving cell are obtained from the real-time MR data.

[0012] All neighboring cells corresponding to each serving cell are obtained from the real-time OMC data and the real-time threshold data. The neighboring cell with the strongest signal strength is selected from all the neighboring cells as the first neighboring cell. The difference between the serving RSRP value of each serving cell and the neighboring cell RSRP value of the corresponding first neighboring cell is obtained from the real-time MR data and calculated.

[0013] Determine whether the uplink signal interference-to-noise ratio is lower than a first threshold, whether the power margin report is lower than a second threshold, and whether the difference is greater than or equal to a third threshold. If the uplink signal interference-to-noise ratio is lower than the first threshold, the power margin report is lower than the second threshold, and the difference is greater than or equal to the third threshold, then the serving cell is selected as a candidate cell.

[0014] The weak coverage percentage is calculated based on the candidate RSRP value of the candidate cells, and problem cells are identified from the candidate cells based on the weak coverage percentage.

[0015] Optionally, the problem cells include moderately problematic cells and severely problematic cells, and the step of calculating the weak coverage ratio based on the candidate RSRP values ​​of the candidate cells, and identifying problem cells from the candidate cells based on the weak coverage ratio, includes:

[0016] Determine whether there are candidate RSRP values ​​less than the fifth threshold. If so, obtain the first number of candidate RSRP values ​​less than the fifth threshold and use the first number as the first weak coverage ratio. Determine whether the first weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the first weak coverage ratio greater than or equal to the fourth threshold are designated as moderately problematic cells.

[0017] Determine whether there are candidate RSRP values ​​less than the sixth threshold. If so, obtain the second number of candidate RSRP values ​​less than the sixth threshold and use the second number as the second weak coverage ratio. Determine whether the second weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the second weak coverage ratio greater than or equal to the fourth threshold are designated as severely problem cells.

[0018] As described above, combining signal strength, uplink signal interference-to-noise ratio, and power margin report to identify problem cells improves the accuracy of problem cell identification compared to the traditional single service RSRP value judgment. Furthermore, it calculates the weak coverage ratio based on candidate RSRP values, identifies problem cells based on the weak coverage ratio, and further classifies problem cells into severely problematic cells and extremely problematic cells. This not only achieves fine-grained management of problem cells but also ensures the comprehensiveness of the identified problem cells.

[0019] Optionally, the Q1 network selects preferred actions based on the current state vector, including:

[0020] The Q1 network calculates the Q1 value of each action in the preset action space based on the current state vector, adopts the ε-greedy strategy, randomly selects an action in the preset action space with probability ε to start exploration, and calculates the Q1 value of the corresponding action for each exploration based on the current state vector, thus obtaining the set of Q1 values ​​for each action.

[0021] With probability 1-ε, the action corresponding to the largest Q1 value is selected from the set of Q1 values ​​of each action as a candidate action, and the candidate action with the largest Q1 value is selected from the candidate action set as the preferred action.

[0022] As described above, when selecting the preferred action, an ε-greedy strategy is adopted. The action in the preset action space is explored with probability ε to find better actions, ensuring the comprehensiveness of the obtained Q1 value set. Then, the action corresponding to the largest Q1 value is selected from the Q1 value set of each action as a candidate action with probability 1-ε, avoiding getting trapped in local optima, and thus ensuring the accuracy of the preferred action selected from the candidate action set.

[0023] Optionally, the long-term value assessment of the preferred action through the Q2 network to obtain the long-term value assessment includes:

[0024] The Q2 network predicts the next state vector corresponding to the preferred action using the state prediction model and the current state vector. It also predicts future environmental data after the preferred action is executed. The future environmental data includes: future user rate, future ping-pong switching rate, and future disconnection rate.

[0025] Obtain the current environment data corresponding to the current state vector, and perform a long-term value assessment of the preferred action based on the current environment data, the future environment data, the current state vector, and the next state vector to obtain the long-term assessment value.

[0026] As described above, when the Q2 network evaluates the long-term value of the preferred action, it not only combines the next state vector corresponding to the preferred action, but also future environmental data, current environmental data, and the current state vector, including future user rate, future ping-pong switching rate, and future disconnection rate. This ensures the comprehensiveness of the long-term value evaluation and improves the accuracy of the calculated long-term evaluation value.

[0027] Optionally, the step of performing a long-term value assessment of the preferred action based on the current environmental data, the future environmental data, the current state vector, and the next state vector to obtain the long-term assessment value includes:

[0028] Substituting the current environmental data, the future environmental data, the current state vector, and the next state vector into the reward function formula, the reward function value is obtained. The reward function formula is:

[0029] ;

[0030] Where, r t This represents the reward function value at time t. The current weight representing the percentage of weak coverage at time t+1. This represents the current percentage of weak coverage at time t. This indicates the percentage of weak coverage in the future at time t+1. This represents the current weight of the proportion of weak-coverage strong-neighbor areas at time t+1. This represents the current percentage of weak-coverage strong-neighbor areas at time t. This indicates the percentage of weak-coverage strong-neighbor areas at time t+1 in the future. This represents the current weight of the ping-pong switching rate at time t+1. This represents the current ping-pong switching rate at time t. This represents the future ping-pong switching rate at time t+1. The current weight representing the user rate at time t+1. This represents the current user rate at time t. This represents the future user rate at time t+1. The current weight representing the utilization rate at time t+1. This represents the current utilization rate at time t. This represents the future utilization rate at time t+1. This represents the current weight of the disconnection rate at time t+1. This represents the current disconnection rate at time t. This represents the future disconnection rate at time t+1;

[0031] Based on the next state vector and the Q1 network, the next optimal action corresponding to the next state vector is selected. The next optimal action and the reward function value are then substituted into the value assessment formula for long-term value assessment, yielding the long-term assessed value. The value assessment formula is as follows:

[0032] ;

[0033] Among them, P t r represents the long-term valuation value at time t. t This represents the reward function value at time t, and β represents the discount factor. This indicates the next preferred action at time t+1.

[0034] As described above, when conducting long-term value assessment, the Q2 network incorporates a discount factor and a reward function value. This balances short-term coverage and long-term network performance while enhancing the network's long-term stability and improving the accuracy of long-term value assessment.

[0035] Optionally, the step of dynamically updating the Q1 network based on the long-term evaluation value to obtain an updated Q1 network, re-selecting new preferred actions using the updated Q1 network, dynamically updating the Q2 network based on the new preferred actions and the updated Q1 network to obtain an updated Q2 network, and re-evaluating the long-term value of the new preferred actions using the updated Q2 network includes:

[0036] The long-term evaluation value is input into the first update formula to dynamically update the Q1 network, resulting in the updated Q1 network. The first update formula is:

[0037] ;

[0038] in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q1 network before the update at time t. This represents the online learning rate of the Q1 network. Indicates the long-term valuation value at time t;

[0039] New preferred actions are re-selected using the updated Q1 network. The Q2 network is then dynamically updated based on these new preferred actions, the updated Q1 network, and the second update formula, resulting in an updated Q2 network. The new preferred actions are then re-evaluated for long-term value using the updated Q2 network. The second update formula is:

[0040] ;

[0041] in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q2 network before the update at time t. This represents the offline learning rate of the Q2 network. Indicates the discount factor. This represents the updated Q1 network at time t+1, s t+1 This represents the current state vector at time t+1. This indicates the new preferred action at time t+1.

[0042] Optionally, the step of re-selecting new preferred actions through the updated Q1 network includes:

[0043] The actions in the preset action space are updated and adjusted according to the priority update and adjustment rules to obtain the updated preset action space. New preferred actions are then selected from the updated preset action space.

[0044] The preset action space includes: Action 1: Adjust the A3 event trigger threshold, Action 2: Adjust the A5-1 event trigger threshold, Action 3: Adjust the A5-2 event trigger threshold, Action 4: Adjust the forward neighbor cell CIO, and Action 5: Adjust the reverse neighbor cell CIO;

[0045] The priority update adjustment rules include: action 4 has a higher priority than action 5, action 5 has a higher priority than action 2 and action 3, and action 2 and action 3 have a higher priority than action 1.

[0046] As described above, when the updated Q1 network re-selects new preferred actions, it will update and adjust the actions in the preset space according to the priority update and adjustment rules. The updates and adjustments of forward CIO and reverse CIO take precedence over the adjustments of various thresholds, ensuring the rationality of the update and adjustment while improving the adjustment efficiency.

[0047] In a second aspect, the present invention provides a dynamic optimization system for inter-frequency handover threshold based on a dual-Q network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the dynamic optimization method for inter-frequency handover threshold based on a dual-Q network described in the first aspect.

[0048] The technical effects of the inter-frequency handover threshold dynamic optimization system based on dual-Q network provided in the second aspect are described in the relevant description of the inter-frequency handover threshold dynamic optimization method based on dual-Q network provided in the first aspect. Attached Figure Description

[0049] Figure 1 This is a flowchart of a dynamic optimization method for inter-frequency handover threshold based on a dual-Q network provided in this embodiment;

[0050] Figure 2 This is a schematic diagram of the overall process of a dynamic optimization method for inter-frequency handover threshold based on a dual-Q network provided in this embodiment;

[0051] Figure 3 This is a schematic diagram of the structure of a frequency switching threshold dynamic optimization system based on a dual-Q network provided in this embodiment.

[0052] [Explanation of Labels in the Attached Image]

[0053] 1. A dynamic optimization system for inter-frequency handover threshold based on dual-Q network;

[0054] 2. Processor;

[0055] 3. Memory. Detailed Implementation

[0056] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0057] Example 1

[0058] Please refer to Figures 1 to 2 This invention provides a method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network, comprising the following steps:

[0059] S1. Obtain real-time MR data, real-time OMC data, and real-time threshold data. Determine whether there is a problem cell based on the real-time MR data, real-time OMC data, and real-time threshold data. If there is, send an optimization request to the dual-Q network and simultaneously send the current state vector of the problem cell to the dual-Q network. The current state vector includes: current weak coverage percentage, current weak coverage strong neighbor cell percentage, current PRB utilization, A5 threshold, A3 threshold, forward CIO, and reverse CIO. The dual-Q network includes Q1 network and Q2 network.

[0060] In this embodiment, as Figure 2As shown, real-time MR (Measurement Report) data, real-time OMC (Operations and Maintenance Center) data, and real-time threshold data are acquired. The real-time MR data includes MR.LteScRSRP (Serving Cell Reference Signal Received Power), MR.LteNcRSRP (Neighboring Cell Reference Signal Received Power), MR.LteScRSRQ (Serving Cell Reference Signal Received Quality), MR.LteNcRSRQ (Neighboring Cell Reference Signal Received Quality), MR.LteScTadv (Serving Cell Event Lead), MR.LteScPHR (Serving Cell Power Margin Report), MR.LteScAOA (Serving Cell Angle of Arrival), and MR.Lt... eScSinrUL (serving cell uplink signal interference-to-noise ratio), MR.LteScEarfcn (serving cell frequency), MR.LteScPci (serving cell physical cell identifier), MR.LteNcEarfcn (neighboring cell frequency), and MR.LteNcPci (neighboring cell physical cell identifier) ​​interact with the network management system via HTTP protocol when collecting real-time MR data. The data format is JSON nested CSV, and multiple batches of data are associated with cell IDs based on timestamps accurate to the second. The key fields of the collected real-time MR data are converted according to the rules in Table 1.

[0061] Table 1. Conversion Rules for Key Fields in MR Data

[0062] Fields Conversion rules Physical meaning and reasonable range RSRP actual value Actual RSRP value = Original RSRP value - 156 LTE signal strength, range [-140dBm, -44dBm] SINR actual value Actual SINR value = Original SINR value / 2 - 23 Signal-to-noise ratio, range [-20dB, 30dB] PHR actual value Actual PHR value = Original PHR value - 23 Terminal power margin, range [-23dB, 32dB]

[0063] Real-time OMC data includes call completion rate, call drop rate, handover success rate, cell uplink and downlink speeds, and ping-pong handover counts. The data format is CS. Real-time threshold data includes A3 event trigger threshold, A5 event trigger threshold, and neighbor cell CIO configuration parameters. The acquired real-time MR data undergoes filtering and standardization. The filtering rules include: Filtering Rule 1: Filtering data where MR.LteScRSRP exceeds the range [16, 112]; Filtering Rule 2: Filtering null values; Filtering Rule 3: Filtering data using Isolation... Data with MR.LteScRSRP < -140 and MR.LteScPHR > 15 calculated by the Forest algorithm are filtered and standardized. The standardization process includes: Standardization rule 1: Calculate the mean μ and standard deviation σ for continuous features such as MR.LteScRSRP and MR.LteScSinrUL; Standardization rule 2: Calculate the difference sequence of MR.LteScRSRP with a sliding window of 5, such as the mean difference between adjacent values ​​within the window; Standardization rule 3: Calculate the standard deviation / mean of MR.LteScSinrUL within the window. The above filtering and standardization processes can be adjusted according to the actual situation. Based on the obtained real-time MR data, real-time OMC data, and real-time threshold data, it is determined whether there is a problem in the West Campus. If so, an optimization request is sent to the Double Deep Q-Network, and the current state vector of the problem cell is also sent to the Double Deep Q-Network. The current state vector includes the current weak coverage ratio, the current weak coverage strong neighbor cell ratio, the current PRB utilization rate, the A5 threshold, the A3 threshold, the positive CIO, and the negative CIO.

[0064] At this point, step S1, which involves determining whether a problem cell exists based on the real-time MR data, the real-time OMC data, and the real-time threshold data, includes:

[0065] S11. Obtain the uplink signal interference-to-noise ratio and power margin report for each serving cell from the real-time MR data;

[0066] S12. Obtain all neighboring cells corresponding to each serving cell from the real-time OMC data and the real-time threshold data, select the neighboring cell with the strongest signal strength from all the neighboring cells as the first neighboring cell, and obtain and calculate the difference between the serving RSRP value of each serving cell and the neighboring cell RSRP value of the corresponding first neighboring cell from the real-time MR data.

[0067] S13. Determine whether the uplink signal interference-to-noise ratio is lower than a first threshold, whether the power margin report is lower than a second threshold, and whether the difference is greater than or equal to a third threshold. If the uplink signal interference-to-noise ratio is lower than the first threshold, the power margin report is lower than the second threshold, and the difference is greater than or equal to the third threshold, then the serving cell is selected as a candidate cell.

[0068] S14. Calculate the weak coverage ratio based on the candidate RSRP value of the candidate cells, and identify problem cells from the candidate cells based on the weak coverage ratio.

[0069] At this point, step S14 includes:

[0070] S141. Determine whether there are candidate RSRP values ​​less than the fifth threshold. If there are, obtain the first number of candidate RSRP values ​​less than the fifth threshold and use the first number as the first weak coverage ratio. Determine whether the first weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the first weak coverage ratio greater than or equal to the fourth threshold are designated as moderately problematic cells.

[0071] S142. Determine whether there are candidate RSRP values ​​less than the sixth threshold. If so, obtain the second number of candidate RSRP values ​​less than the sixth threshold and use the second number as the second weak coverage ratio. Determine whether the second weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the second weak coverage ratio greater than or equal to the fourth threshold are designated as severely problematic cells.

[0072] In this embodiment, as Figure 2 As shown, all neighboring cells corresponding to each serving cell are obtained from real-time OMC data and real-time threshold data. When identifying neighboring cells, neighboring cells are identified according to geographical coverage distance using neighboring cell frequency points and neighboring cell PCIs from MR data. The identified neighboring cells are monitored and evaluated in real time, and the data of the neighboring cells are dynamically updated. Missing configuration checks are performed on the identified neighboring cells. The identification of the neighboring cells is confirmed by comparing them with the neighboring cell relationship table of the base station. The accuracy of the configuration parameters of the neighboring cells is also checked. The configuration parameters include: base station configuration, frequency point configuration, PCI configuration, TAC configuration, etc.

[0073] The cell with the strongest signal strength is selected from all neighboring cells as the first neighboring cell. The difference between the serving RSRP value of each serving cell and the corresponding neighboring cell RSRP value of the first neighboring cell is obtained and calculated from real-time MR data. The uplink interference-to-noise ratio (IRR) and power margin report of each serving cell are also obtained from real-time MR data. Serving cells with an IRR below a first threshold, a power margin report below a second threshold, and a difference greater than or equal to a third threshold are selected as candidate cells. The first threshold is 0 dB, the second threshold is 2 dB, and the third threshold is 3 dB. The weak coverage percentage is calculated based on the candidate RSRP values ​​of the candidate cells. If there are candidate RSRP values ​​less than a fifth threshold, the first number of candidate RSRP values ​​less than the fifth threshold is obtained. The first quantity is used as the first weak coverage percentage. If the first weak coverage percentage is greater than or equal to the fourth threshold, it indicates that there are problematic cells among the candidate cells. Candidate cells with the first weak coverage percentage greater than or equal to the fourth threshold are classified as moderately problematic cells. If there are candidate RSRP values ​​less than the sixth threshold, the second quantity of candidate RSRP values ​​less than the sixth threshold is obtained and used as the second weak coverage percentage. If the second weak coverage percentage is greater than or equal to the fourth threshold, it indicates that there are problematic cells among the candidate cells. Candidate cells with the second weak coverage percentage greater than or equal to the fourth threshold are classified as severely problematic cells. Moderately problematic cells and severely problematic cells are marked. The fifth threshold is -105dBm, the sixth threshold is -110dBm, and the fourth threshold is 5%.

[0074] S2. The dual-Q network receives the optimization request and the current state vector. The Q1 network selects the preferred action based on the current state vector. The Q2 network performs a long-term value assessment on the preferred action to obtain the long-term assessment value.

[0075] At this point, the Q1 network in step S2 selects the preferred action based on the current state vector, including:

[0076] S21. The Q1 network calculates the Q1 value of each action in the preset action space based on the current state vector, adopts the ε-greedy strategy, randomly selects an action in the preset action space with probability ε to start exploration, and calculates the Q1 value of the corresponding action for each exploration based on the current state vector to obtain the set of Q1 values ​​for each action.

[0077] S22. Select the action corresponding to the largest Q1 value from the set of Q1 values ​​of each action with a probability of 1-ε as a candidate action to obtain a candidate action set. Select the candidate action with the largest Q1 value from the candidate action set as the preferred action.

[0078] In this embodiment, the Q1 network is an online network. Its main task is to calculate the Q1 value of each action in the preset action space based on the current state vector using an ε-greedy strategy, so as to obtain the Q1 value set of each action. Then, with a probability of 1-ε, the action corresponding to the largest Q1 value is selected from the Q1 value set of each action as a candidate action, so as to obtain a candidate action set. That is, each action in the candidate action set is the action corresponding to the largest Q1 value in all exploration calculations. Then, the candidate action with the largest Q1 value among all candidate actions is selected as the preferred action. The preset action space includes: Action 1: Adjust the A3 event trigger threshold, Action 2: Adjust the A5-1 event trigger threshold, Action 3: Adjust the A5-2 event trigger threshold, Action 4: Adjust the forward neighbor CIO, and Action 5: Adjust the reverse neighbor CIO.

[0079] At this point, the long-term value assessment of the preferred action obtained in step S2 through the Q2 network includes:

[0080] S23. The Q2 network predicts the next state vector corresponding to the preferred action through the state prediction model and the current state vector, and at the same time predicts the future environmental data after the preferred action is executed. The future environmental data includes: future user rate, future ping-pong switching rate and future disconnection rate.

[0081] In this embodiment, as Figure 2 As shown, the Q2 network predicts the next state vector corresponding to the preferred action through the state prediction model and the current state vector, and at the same time predicts the future environmental data after the preferred action is executed. The state prediction model is pre-constructed based on historical MR data, historical OMC data and historical threshold data using Markov decision-making.

[0082] S24. Obtain the current environment data corresponding to the current state vector, and perform a long-term value assessment on the preferred action based on the current environment data, the future environment data, the current state vector, and the next state vector to obtain the long-term assessment value.

[0083] At this point, the long-term value assessment of the preferred action based on the current environmental data, the future environmental data, the current state vector, and the next state vector in step S24, to obtain the long-term assessment value, includes:

[0084] S241. Substitute the current environment data, the future environment data, the current state vector, and the next state vector into the reward function formula to calculate the reward function value. The reward function formula is:

[0085] ;

[0086] Where, r t This represents the reward function value at time t. The current weight representing the percentage of weak coverage at time t+1. This represents the current percentage of weak coverage at time t. This indicates the percentage of weak coverage in the future at time t+1. This represents the current weight of the proportion of weak-coverage strong-neighbor areas at time t+1. This represents the current percentage of weak-coverage strong-neighbor areas at time t. This indicates the percentage of weak-coverage strong-neighbor areas at time t+1 in the future. This represents the current weight of the ping-pong switching rate at time t+1. This represents the current ping-pong switching rate at time t. This represents the future ping-pong switching rate at time t+1. The current weight representing the user rate at time t+1. This represents the current user rate at time t. This represents the future user rate at time t+1. The current weight representing the utilization rate at time t+1. This represents the current utilization rate at time t. This represents the future utilization rate at time t+1. This represents the current weight of the disconnection rate at time t+1. This represents the current disconnection rate at time t. This represents the future disconnection rate at time t+1;

[0087] In this embodiment, the initial value of the current weight of the weak coverage strong neighbor cell ratio at time t+1 in the reward function varies depending on the label of the problem cell. When the problem cell is a moderately problematic cell, the initial value of the current weight of the current weight of the weak coverage strong neighbor cell ratio at time t+1 is 0.6; when the problem cell is a severely problematic cell, the initial value of the current weight ... The percentage of weak-coverage strong neighboring areas achieved = PRB utilization rate achievement rate = User rate achievement rate = Disconnection success rate = The preset adjustment cycle is 24 hours, meaning that the KPI achievement rate is calculated every 24 hours, and the corresponding weights are dynamically adjusted according to preset weighting rules based on the KPI achievement rate. For example, if the calculated result of the weak coverage percentage achievement rate is negative, its corresponding weight is increased. If the calculated percentage of weak-coverage strong neighboring areas is negative, then its corresponding weight is increased. If the calculated PRB utilization rate is greater than 70%, its corresponding weight will be reduced. If the calculated user rate achievement rate is negative, its corresponding weight will be increased. If the calculated ping-pong switching achievement rate is negative, its corresponding weight will be reduced. If the calculated disconnection rate is negative, its corresponding weight will be reduced. .

[0088] In one specific embodiment, the current weight of the weak coverage rate percentage at time t+1 is 0.2, the current weak coverage rate percentage at time t is 0.05, the future weak coverage rate percentage at time t+1 is 0.05, the current weight of the weak coverage strong neighbor cell percentage at time t+1 is 0.4, the current weak coverage strong neighbor cell percentage at time t is 0.07, the future weak coverage strong neighbor cell percentage at time t+1 is 0.05, the current weight of the ping-pong handover rate at time t+1 is 0.2, indicating that the current ping-pong handover rate at time t is 0.09, the future ping-pong handover rate at time t+1 is 0.08, and the current weight of the user rate at time t+1 is 0.1. The utilization rate is 48, the future user rate at time t+1 is 45, the current weight of utilization rate at time t+1 is 0.1, the current utilization rate at time t is 0.35, the future utilization rate at time t+1 is 0.35, the current weight of disconnection rate at time t+1 is 0.1, the current disconnection rate at time t is 0.5, and the future disconnection rate at time t+1 is 0.3. Then the reward function value at time t = 0.2×(0.05-0.05)+0.4×(0.07-0.05)+0.2×(0.09-0.08)+0.1×(48-45)+0.1×(0.35-0.35)+0.1×(0.5-0.3)=0.33.

[0089] S242. Based on the next state vector and the Q1 network, the next preferred action corresponding to the next state vector is selected. The next preferred action and the reward function value are substituted into the value assessment formula to perform a long-term value assessment, and the long-term value assessment is obtained. The value assessment formula is:

[0090] ;

[0091] Among them, P t r represents the long-term valuation value at time t. t This represents the reward function value at time t, and β represents the discount factor. This indicates the next preferred action at time t+1.

[0092] In this embodiment, the discount factor ranges from [0.8, 0.99], preferably 0.9.

[0093] S3. Determine whether the long-term evaluation value has achieved the expected effect. If not, dynamically update the Q1 network based on the long-term evaluation value to obtain the updated Q1 network. Re-select new preferred actions through the updated Q1 network. Dynamically update the Q2 network based on the new preferred actions and the updated Q1 network to obtain the updated Q2 network. Re-evaluate the long-term value of the new preferred actions through the updated Q2 network to obtain new long-term evaluation values ​​until the new long-term evaluation value achieves the expected effect. Execute the corresponding new preferred actions to realize the inter-frequency handover threshold for the problem cell.

[0094] At this point, step S3 involves dynamically updating the Q1 network based on the long-term evaluation value to obtain an updated Q1 network. New preferred actions are then selected using the updated Q1 network. The Q2 network is then dynamically updated based on the new preferred actions and the updated Q1 network to obtain an updated Q2 network. Finally, the long-term value of the new preferred actions is re-evaluated using the updated Q2 network, including:

[0095] S31. Input the long-term evaluation value into the first update formula to dynamically update the Q1 network, obtaining the updated Q1 network. The first update formula is:

[0096] ;

[0097] in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q1 network before the update at time t. This represents the online learning rate of the Q1 network. Indicates the long-term valuation value at time t;

[0098] S32. New preferred actions are re-selected using the updated Q1 network. The Q2 network is dynamically updated based on the new preferred actions, the updated Q1 network, and the second update formula to obtain the updated Q2 network. The long-term value of the new preferred actions is re-evaluated using the updated Q2 network. The second update formula is:

[0099] ;

[0100] in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q2 network before the update at time t. This represents the offline learning rate of the Q2 network. Indicates the discount factor. This represents the updated Q1 network at time t+1, s t+1 This represents the current state vector at time t+1. This indicates the new preferred action at time t+1.

[0101] In this embodiment, the online learning rate of the Q1 network and the offline learning rate of the Q2 network are in the range of (0,1), preferably 0.01. When the Q2 network is dynamically updated according to the new preferred action, the updated Q1 network, and the second update formula, the update cycle of the Q2 network is shorter than that of the Q1 network. For example, the update cycle of the Q1 network is 15 minutes / time, and the update cycle of the Q2 network is 10 hours / time. Furthermore, the Q2 network is also updated in conjunction with the number of iterations. For example, if the number of iterations exceeds 40, the Q2 network is updated, thereby achieving the decoupling of the Q1 network and the Q2 network.

[0102] At this point, the step S32, which involves re-selecting new preferred actions using the updated Q1 network, includes:

[0103] S321. Update and adjust the actions in the preset action space according to the priority update and adjustment rules to obtain the updated and adjusted preset action space, and re-select new preferred actions from the updated and adjusted preset action space.

[0104] The preset action space includes: Action 1: Adjust the A3 event trigger threshold, Action 2: Adjust the A5-1 event trigger threshold, Action 3: Adjust the A5-2 event trigger threshold, Action 4: Adjust the forward neighbor cell CIO, and Action 5: Adjust the reverse neighbor cell CIO;

[0105] The priority update adjustment rules include: action 4 has a higher priority than action 5, action 5 has a higher priority than action 2 and action 3, and action 2 and action 3 have a higher priority than action 1.

[0106] In this embodiment, as Figure 2 As shown, when the updated Q1 network re-selects new preferred actions, it updates and adjusts the actions in the preset space according to the priority update rule. The adjustment of the forward neighboring cell CIO takes precedence over the adjustment of the reverse neighboring cell CIO, and the adjustment of the reverse neighboring cell CIO takes precedence over the A5-1 event trigger threshold and the A5-2 event trigger threshold. The A5-1 event trigger threshold and the A5-2 event trigger threshold will only be updated and adjusted when the CIO adjustment is saturated. Among them, when the CIO range is [-6dB, 6dB], that is, when the CIO is 6dB, it means that the forward CIO is saturated. When CIO is -6dB, it indicates that the reverse CIO is saturated. The A5-1 event trigger threshold and the A5-2 event trigger threshold take precedence over the A3 event trigger threshold. When the updated Q1 network updates and adjusts the actions in the preset space, it makes different adjustments based on the label of the problem cell. When the problem cell is a moderately problematic cell, the updated Q1 network updates and adjusts the actions in the preset space by ±1dB. When the problem cell is a severely problematic cell, the updated Q1 network updates and adjusts the actions in the preset space by ±2dB.

[0107] In this embodiment, the method also includes real-time monitoring of problematic cells that fail to meet inter-frequency handover thresholds, specifically:

[0108] KPI achievement rate is monitored in the short term or in the long term. If the reward function value calculated based on the KPI achievement rate decreases for three consecutive times, the rollback mechanism is triggered to roll back to the previous stable dual-Q network. The short term monitoring is 0-3 hours of monitoring, and the long term monitoring is 24 hours of monitoring.

[0109] Example 2

[0110] Please refer to Figure 3 The present invention provides a dynamic optimization system 1 for inter-frequency switching threshold based on dual-Q network, including a memory 3, a processor 2, and a computer program stored in the memory 3 and run on the processor 2. When the processor 2 executes the computer program, it implements the steps in Embodiment 1.

[0111] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0114] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0115] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0116] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0117] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A dynamic optimization method for inter-frequency handover threshold based on a dual-Q network, characterized in that, include: Acquire real-time MR data, real-time OMC data, and real-time threshold data. Determine whether there are problem cells based on the real-time MR data, real-time OMC data, and real-time threshold data. If there are, send an optimization request to the dual-Q network and simultaneously send the current state vector of the problem cell to the dual-Q network. The current state vector includes: current weak coverage percentage, current weak coverage strong neighbor cell percentage, current PRB utilization, A5 threshold, A3 threshold, forward CIO, and reverse CIO. The dual-Q network includes Q1 network and Q2 network. The dual-Q network receives the optimization request and the current state vector. The Q1 network selects the preferred action based on the current state vector. The Q2 network performs a long-term value evaluation on the preferred action to obtain the long-term evaluation value. If the long-term evaluation value does not meet the expected results, the Q1 network is dynamically updated based on the long-term evaluation value to obtain an updated Q1 network. New preferred actions are re-selected through the updated Q1 network. The Q2 network is dynamically updated based on the new preferred actions and the updated Q1 network to obtain an updated Q2 network. The long-term value of the new preferred actions is re-evaluated through the updated Q2 network to obtain a new long-term evaluation value until the new long-term evaluation value meets the expected results. The corresponding new preferred actions are then executed to achieve the inter-frequency handover threshold for the problematic cell. The Q1 network selects preferred actions based on the current state vector, including: The Q1 network calculates the Q1 value of each action in the preset action space based on the current state vector, adopts the ε-greedy strategy, randomly selects an action in the preset action space with probability ε to start exploration, and calculates the Q1 value of the corresponding action for each exploration based on the current state vector, thus obtaining the set of Q1 values ​​for each action. With probability 1-ε, select the action corresponding to the largest Q1 value from the set of Q1 values ​​of each action as a candidate action to obtain a candidate action set. Select the candidate action with the largest Q1 value from the candidate action set as the preferred action. The long-term value assessment of the preferred action through the Q2 network, resulting in a long-term value assessment, includes: The Q2 network predicts the next state vector corresponding to the preferred action using the state prediction model and the current state vector. It also predicts future environmental data after the preferred action is executed. The future environmental data includes: future user rate, future ping-pong switching rate, and future disconnection rate. Obtain the current environment data corresponding to the current state vector, and perform a long-term value assessment of the preferred action based on the current environment data, the future environment data, the current state vector, and the next state vector to obtain the long-term assessment value.

2. The method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network as described in claim 1, characterized in that, The step of determining whether a problematic cell exists based on the real-time MR data, the real-time OMC data, and the real-time threshold data includes: The uplink signal interference-to-noise ratio and power margin report for each serving cell are obtained from the real-time MR data. All neighboring cells corresponding to each serving cell are obtained from the real-time OMC data and the real-time threshold data. The neighboring cell with the strongest signal strength is selected from all the neighboring cells as the first neighboring cell. The difference between the serving RSRP value of each serving cell and the neighboring cell RSRP value of the corresponding first neighboring cell is obtained from the real-time MR data and calculated. Determine whether the uplink signal interference-to-noise ratio is lower than a first threshold, whether the power margin report is lower than a second threshold, and whether the difference is greater than or equal to a third threshold. If the uplink signal interference-to-noise ratio is lower than the first threshold, the power margin report is lower than the second threshold, and the difference is greater than or equal to the third threshold, then the serving cell is selected as a candidate cell. The weak coverage percentage is calculated based on the candidate RSRP value of the candidate cells, and problem cells are identified from the candidate cells based on the weak coverage percentage.

3. The method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network as described in claim 2, characterized in that, The problematic cells include moderately problematic cells and severely problematic cells. The step of calculating the weak coverage percentage based on the candidate RSRP values ​​of the candidate cells, and identifying problematic cells from the candidate cells based on the weak coverage percentage, includes: Determine whether there are candidate RSRP values ​​less than the fifth threshold. If so, obtain the first number of candidate RSRP values ​​less than the fifth threshold and use the first number as the first weak coverage ratio. Determine whether the first weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the first weak coverage ratio greater than or equal to the fourth threshold are designated as moderately problematic cells. Determine whether there are candidate RSRP values ​​less than the sixth threshold. If so, obtain the second number of candidate RSRP values ​​less than the sixth threshold and use the second number as the second weak coverage ratio. Determine whether the second weak coverage ratio is greater than or equal to the fourth threshold. If so, there are problem cells among the candidate cells, and the candidate cells with the second weak coverage ratio greater than or equal to the fourth threshold are designated as severely problem cells.

4. The method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network as described in claim 1, characterized in that, The long-term value assessment of the preferred action based on the current environmental data, the future environmental data, the current state vector, and the next state vector yields the following long-term value assessment: Substituting the current environmental data, the future environmental data, the current state vector, and the next state vector into the reward function formula, the reward function value is obtained. The reward function formula is: ； Where, r t This represents the reward function value at time t. This represents the current weight of the percentage of weak coverage at time t+1. This represents the current percentage of weak coverage at time t. This indicates the percentage of weak coverage in the future at time t+1. This represents the current weight of the proportion of weak-coverage strong-neighbor areas at time t+1. This represents the current percentage of weak-coverage strong-neighbor areas at time t. This indicates the percentage of weak-coverage strong-neighbor areas at time t+1 in the future. This represents the current weight of the ping-pong switching rate at time t+1. This represents the current ping-pong switching rate at time t. This represents the future ping-pong switching rate at time t+1. The current weight representing the user rate at time t+1. This represents the current user rate at time t. This represents the future user rate at time t+1. The current weight representing the utilization rate at time t+1. This represents the current utilization rate at time t. This represents the future utilization rate at time t+1. This represents the current weight of the disconnection rate at time t+1. This represents the current disconnection rate at time t. This represents the future disconnection rate at time t+1; Based on the next state vector and the Q1 network, the next optimal action corresponding to the next state vector is selected. The next optimal action and the reward function value are then substituted into the value assessment formula for long-term value assessment, yielding the long-term assessed value. The value assessment formula is as follows: ； Among them, P t r represents the long-term valuation value at time t. t This represents the reward function value at time t, and β represents the discount factor. This indicates the next preferred action at time t+1.

5. The method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network as described in claim 1, characterized in that, The process involves dynamically updating the Q1 network based on the long-term evaluation value to obtain an updated Q1 network. New preferred actions are then selected using the updated Q1 network. The Q2 network is then dynamically updated based on the new preferred actions and the updated Q1 network to obtain an updated Q2 network. Finally, the long-term value of the new preferred actions is re-evaluated using the updated Q2 network, including: The long-term evaluation value is input into the first update formula to dynamically update the Q1 network, resulting in the updated Q1 network. The first update formula is: ； in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q1 network before the update at time t. This represents the online learning rate of the Q1 network. Indicates the long-term valuation value at time t; New preferred actions are re-selected using the updated Q1 network. The Q2 network is then dynamically updated based on these new preferred actions, the updated Q1 network, and the second update formula, resulting in an updated Q2 network. The new preferred actions are then re-evaluated for long-term value using the updated Q2 network. The second update formula is: ； in, s represents the updated Q1 network at time t. t This represents the current state vector at time t. This represents the candidate action i at time t. This represents the Q2 network before the update at time t. This represents the offline learning rate of the Q2 network. Indicates the discount factor. This represents the updated Q1 network at time t+1, s t+1 This represents the current state vector at time t+1. This indicates the new preferred action at time t+1.

6. The method for dynamic optimization of inter-frequency handover threshold based on a dual-Q network as described in claim 5, characterized in that, The process of re-selecting new preferred actions through the updated Q1 network includes: The actions in the preset action space are updated and adjusted according to the priority update and adjustment rules to obtain the updated preset action space. New preferred actions are then selected from the updated preset action space. The preset action space includes: Action 1: Adjust the A3 event trigger threshold, Action 2: Adjust the A5-1 event trigger threshold, Action 3: Adjust the A5-2 event trigger threshold, Action 4: Adjust the forward neighbor cell CIO, and Action 5: Adjust the reverse neighbor cell CIO; The priority update adjustment rules include: action 4 has a higher priority than action 5, action 5 has a higher priority than action 2 and action 3, and action 2 and action 3 have a higher priority than action 1.

7. A dynamic optimization system for inter-frequency switching thresholds based on a dual-Q network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

Images