Model training method and device and channel quality indicator value prediction method and device

Abstract

The invention provides a model training method and device and a channel quality indicator value prediction method and device, and relates to the technical field of communication. The method comprises: obtaining state information according to a channel quality indicator (CQI) value reported by a terminal and hybrid automatic repeat request (HARQ) feedback information, the state information being training data of a deep Q network (DQN) model; and training the DQN model according to the training data and a reward function to obtain a target DQN model, the reward function comprising reward values corresponding to three types of prediction effects, and the target DQN model being used for predicting a CQI value. According to the scheme, the number of reward values is increased, the DQN model can obtain more reward values under the prediction effect based on the reward function, the algorithm prediction effect can be improved, and network convergence is accelerated.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method and device for model training and channel quality indicator value prediction. Background Art

[0002] The measurement, scheduling and processing delays from the user to the base station introduce the Channel Quality Indicator (CQI) delay effect. Using outdated CQI values ​​may lead to the selection of incorrect modulation and coding schemes, which will have an adverse impact on performance. Therefore, it is necessary to accurately predict and estimate CQI. Existing technologies all use the traditional Deep Q-Network (DQN) model to predict CQI, but the traditional DQN model has the problems of slow convergence speed and poor prediction effect. Summary of the invention

[0003] The purpose of this application is to provide a model training, channel quality indicator value prediction method and device to solve the problems of slow convergence speed and poor prediction effect of the traditional DQN model when predicting CQI.

[0004] To achieve the above objectives, an embodiment of the present application provides a model training method, comprising:

[0005] According to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, the state information is obtained, and the state information is the training data of the deep Q network DQN model;

[0006] The DQN model is trained according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0007] Optionally, the method of the embodiment of the present application further includes:

[0008] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is less than or equal to a first preset threshold, determining that the prediction effect of the DQN model is a first type of prediction effect, and the reward value corresponding to the first type of prediction effect is a first value; or

[0009] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the first preset threshold and less than or equal to the second preset threshold, it is determined that the prediction effect of the DQN model is a second type of prediction effect, and the reward value corresponding to the second type of prediction effect is a second value, wherein the second preset threshold is greater than the first preset threshold; or

[0010] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the second preset threshold, it is determined that the prediction effect of the DQN model is a third type of prediction effect, and the reward value corresponding to the third type of prediction effect is a third value.

[0011] Optionally, the third value is obtained based on the ratio of a first value to a second value, wherein the first value is the difference between a block error rate at a first moment and a block error rate at a second moment, and the second value is a larger value between the block error rate at the first moment and the block error rate at the second moment, and the second moment is the moment before the first moment.

[0012] Optionally, the third value satisfies the following formula:

[0013]

[0014] Among them, r DQN_t Represents the reward value at the first moment, BLER t-1 Indicates the block error rate at the second moment, BLER t Indicates the block error rate at the first moment.

[0015] Optionally, training the DQN model according to the training data and the reward function to obtain a target DQN model includes:

[0016] The DQN model is trained according to the training data, the reward function and the action-value function to obtain a target DQN model, wherein the action-value function is a function for obtaining the value Q corresponding to an action, the action-value function is negatively correlated with the number of times a target action is selected under the state information, and the target action corresponds to a CQI value.

[0017] Optionally, the action-value function satisfies the following formula:

[0018]

[0019] Among them, s DQN Indicates status information, the a DQN Indicates that based on the preset strategy in s DQN The target action selected is , γ represents the learning rate, and M is s DQN Next select a DQN The number of times.

[0020] The embodiment of the present application also provides a method for predicting a channel quality indicator value using the target DQN model as described above, including:

[0021] Acquire a channel quality indicator CQI value reported by the terminal and a CQI prediction value of the target DQN model;

[0022] A target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value, and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

[0023] Optionally, the target CQI prediction value satisfies the following formula:

[0024] CQI pre =αCQI DQN +(1-α)CQI UE ;

[0025] Among them, CQI pre represents the target CQI prediction value, α represents the weight value, CQI DQN Indicates the CQI prediction value, CQI UE Indicates the CQI value reported by the terminal.

[0026] The present application also provides a model training device, including:

[0027] A first acquisition module is used to obtain state information according to a channel quality indication CQI value reported by a terminal and hybrid automatic repeat request HARQ feedback information, wherein the state information is training data of a deep Q network DQN model;

[0028] The second acquisition module is used to train the DQN model according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0029] The embodiment of the present application also provides a channel quality indicator value prediction device using the target DQN model, including:

[0030] A third acquisition module is used to obtain a channel quality indication CQI value reported by the terminal and a CQI prediction value of the target DQN model;

[0031] The fourth acquisition module is used to acquire a target CQI prediction value according to the CQI value reported by the terminal, the CQI prediction value and the weight value of the CQI prediction value, wherein the weight value is obtained according to the prediction effect of the target DQN model.

[0032] The embodiment of the present application also provides a model training device, including: a transceiver and a processor;

[0033] The processor is used to obtain state information according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, wherein the state information is the training data of the deep Q network DQN model; according to the training data and the reward function, the DQN model is trained to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0034] The embodiment of the present application also provides a channel quality indicator value prediction device using a target DQN model, characterized in that it includes: a transceiver and a processor;

[0035] The processor is used to obtain a channel quality indication CQI value reported by the terminal and a CQI prediction value of the target DQN model;

[0036] A target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value, and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

[0037] An embodiment of the present application also provides a model training device, including: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the steps of the model training method described above are implemented.

[0038] An embodiment of the present application also provides a channel quality indicator value prediction device using the target DQN model, including: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the steps of the channel quality indicator value prediction method as described above are implemented.

[0039] An embodiment of the present application also provides a readable storage medium having a program or instruction stored thereon, characterized in that when the program or instruction is executed by a processor, the steps of the model training method described above are implemented, or the steps of the channel quality indicator value prediction method described above are implemented.

[0040] The beneficial effects of the above technical solution of the present application are as follows:

[0041] In an embodiment of the present application, state information is obtained according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, and the state information is the training data of the deep Q network DQN model; the DQN model is trained according to the training data and the reward function to obtain the target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects. Compared with the reward function that only includes two types of reward values ​​in the existing solution, this reward function increases the number of reward values. Based on this reward function, the DQN model can obtain more reward values ​​under more prediction effects, thereby improving the prediction effect of the algorithm and accelerating the convergence of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a model training method according to an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of the deep neural network of this application;

[0044] Figure 3 This is a structural block diagram of the DQN algorithm implemented in the embodiment of the present application;

[0045] Figure 4 A flow chart of a method for predicting a channel quality indicator value according to an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a module of a model training device according to an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a module of a channel quality indicator value prediction device according to an embodiment of the present application;

[0048] Figure 7 This is a structural block diagram of a model training device according to an embodiment of the present application;

[0049] Figure 8 This is one of the structural block diagrams of the channel quality indicator value prediction device according to an embodiment of the present application;

[0050] Fig. 9 This is the second structural block diagram of the model training device or channel quality indicator value prediction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and advantages to be solved by the application clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the application. In addition, for clarity and brevity, the description of known functions and structures has been omitted.

[0052] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0053] In various embodiments of the present application, it should be understood that the order of execution of the following processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the terms "system" and "network" are often used interchangeably in this article.

[0054] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0055] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In cellular systems, channel conditions can seriously affect communication quality. Channel Quality Indication (CQI) is one of the important parameters that determine the multimedia transmission rate. However, the CQI delay effect, that is, the mismatch between the current channel state and its CQI representation, is introduced due to the measurement, scheduling, and processing delays from the user to the base station. Using outdated CQI values ​​may lead to the selection of the wrong modulation and coding scheme, which will have an adverse impact on performance. The channel state in the fifth-generation mobile communication technology (5G) new radio (New Radio, NR) is complex and changeable. The CQI delay effect will seriously limit the throughput of the 5G system, thereby negatively affecting the control of reliability in ultra-reliable and low-latency communication (Ultra-Reliable Low-Latency Communication, URLLC). This effect is more obvious when the channel changes rapidly. This makes it more challenging to accurately estimate the CQI.

[0057] The deep learning methods used in the prior art for CQI prediction are all supervised learning, which requires reliance on a large number of labeled data sets. The reinforcement learning technology solution adopted in this application belongs to the category of unsupervised learning and does not require the data set to be labeled.

[0058] In addition, this application has made two improvements to address the problems of slow convergence speed and poor prediction effect of the traditional DQN algorithm.

[0059] First, the Q-value update method in traditional DQN leads to poor algorithm exploration ability. If a suboptimal action is overestimated and the overestimated Q-value exceeds the Q-value of the optimal action, the optimal action can never be found. At the same time, the efficiency of network training is low, which affects the network convergence speed and CQI prediction effect.

[0060] This application improves the Q value calculation method (i.e., action-value function) to reduce the number of times the agent enters repeated states during training, allowing it to better explore new states. At the same time, it improves the speed of network convergence and optimizes the effect of CQI prediction.

[0061] Secondly, in view of the situation in which the intelligent agent in the traditional reward function can obtain reward values ​​only when the prediction is correct or incorrect, but cannot obtain effective feedback in other cases, the present application redesigns the reward function to add a third reward situation in addition to correct prediction and incorrect prediction, thereby simultaneously improving the speed of network convergence and optimizing the effect of CQI prediction.

[0062] In addition, the final CQI prediction result in the present application is composed of two parts, the CQI value predicted by the model and the CQI value fed back by the UE, according to different weights, and the prediction effect is evaluated according to the downlink block error rate (Block Error Rate, BLER), and the weights of each part of the CQI final value are adjusted with a certain duration (such as 5 minutes) as a cycle to improve the accuracy of the model.

[0063] The model training method provided in the embodiment of the present application is described in detail below through some embodiments and their application scenarios in combination with the accompanying drawings.

[0064] like Figure 1 As shown, the embodiment of the present application provides a model training method, including:

[0065] Step 101: Obtain state information according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, where the state information is the training data of the deep Q network DQN model.

[0066] Optionally, the CQI value reported by the terminal at the current moment and the HARQ feedback information constitute the state information s DQN .

[0067] In the embodiment of the present application, the above-mentioned status information s DQN Determined as the input data of the DQN model, that is, training data.

[0068] Step 102: Train the DQN model according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0069] In an embodiment of the present application, state information is obtained according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, and the state information is the training data of the deep Q network DQN model; the DQN model is trained according to the training data and the reward function to obtain the target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects. Compared with the reward function that only includes two types of reward values ​​in the existing solution, this reward function increases the number of reward values. Based on this reward function, the DQN model can obtain more reward values ​​under more prediction effects, thereby improving the prediction effect of the algorithm and accelerating the convergence of the network.

[0070] Optionally, the method of the embodiment of the present application further includes:

[0071] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is less than or equal to a first preset threshold, determining that the prediction effect of the DQN model is a first type of prediction effect, and the reward value corresponding to the first type of prediction effect is a first value; or

[0072] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the first preset threshold and less than or equal to the second preset threshold, it is determined that the prediction effect of the DQN model is a second type of prediction effect, and the reward value corresponding to the second type of prediction effect is a second value, wherein the second preset threshold is greater than the first preset threshold; or

[0073] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the second preset threshold, it is determined that the prediction effect of the DQN model is a third type of prediction effect, and the reward value corresponding to the third type of prediction effect is a third value.

[0074] Optionally, the third value is obtained based on the ratio of a first value to a second value, wherein the first value is the difference between a block error rate at a first moment and a block error rate at a second moment, and the second value is a larger value between the block error rate at the first moment and the block error rate at the second moment, and the second moment is the moment before the first moment.

[0075] Optionally, the third value satisfies the following formula:

[0076]

[0077] Among them, r DQN_t Represents the reward value at the first moment, BLER t-1 Indicates the block error rate at the second moment, BLER t Indicates the block error rate at the first moment.

[0078] In the embodiment of the present application, when the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the second preset threshold, the BLER at the previous moment is t-1 and the BLER at the next moment t The difference between t and BLER t-1 The reward value is the ratio of the largest one among them. If the BLER at the previous moment is smaller than that at the next moment, it proves that the prediction effect of the intelligent agent has become worse, and the item is a negative number. If the BLER at the previous moment is larger than that at the next moment, it proves that the prediction effect of the intelligent agent has become better, and the item is a positive number.

[0079] In reinforcement learning, the agent needs to take actions to interact with the environment to obtain reward values. The network parameters are updated and better behavior strategies are learned based on the reward values. The reward function is an important part of DQN, and its setting will affect the prediction effect of the algorithm and the convergence of the network. The reward function in the traditional DQN algorithm is set as follows:

[0080]

[0081] Among them, r DQN_t Represents the reward value, and a correct prediction is CQI DQN =CQI UE , and the prediction error refers to CQI DQN ≠CQI UE , A is a positive number, B is a negative number, CQI DQN is the CQI prediction value output by the DQN model, CQI UE Indicates the CQI value reported by the terminal. In this reward function, the agent's reward is divided into two types: correct prediction and wrong prediction. The granularity of the division is relatively coarse. Although the agent can gradually make correct predictions through training, this process requires a large number of rounds.

[0082] The embodiment of the present application redesigns the reward function, which is as follows:

[0083]

[0084] Wherein, M1 represents the first preset threshold, M2 represents the second preset threshold, M2 is greater than M1, A represents the first value, B represents the second value, Represents the third value, A is an integer and B is a negative number.

[0085] Optionally, training the DQN model according to the training data and the reward function to obtain a target DQN model includes:

[0086] The DQN model is trained according to the training data, the reward function and the action-value function to obtain a target DQN model, wherein the action-value function is a function for obtaining the value Q corresponding to an action, the action-value function is negatively correlated with the number of times a target action is selected under the state information, and the target action corresponds to a CQI value.

[0087] Optionally, the action-value function satisfies the following formula:

[0088]

[0089] Among them, s DQN Indicates status information, the a DQN Indicates that based on the preset strategy in sDQN The target action selected is , γ represents the learning rate, and M is s DQN Next select a DQN The number of times.

[0090] The DQN model in the embodiment of the present application uses a deep neural network (DNN) to approximate the Q-value function (ie, the above-mentioned action-value function).

[0091] like Figure 2 As shown, the CQI value reported by the UE at the current moment and the HARQ feedback information constitute the state s DQN , and as the input of DNN, after DNN analysis, according to its strategy The action-value function of the above preset strategy predicts the value of each action and takes the action with the greatest value. DQN , that is, the CQI value CQI predicted by DQN DQN , reward r DQN Related to the downlink block error rate (Block Error Rate, BLER).

[0092] The detailed implementation process of the DQN algorithm is as follows Figure 3 As shown in the figure, the neural network of the DQN algorithm is divided into two parts: the evaluation network and the target network. The difference between the two is the network parameters. The algorithm estimates the Q value of all actions in the action space based on the current state through the evaluation network, and calculates the loss in coordination with the output value of the target network. The evaluation network learns and updates the network parameters in real time, and copies the parameters to the target network every certain rounds (certain duration) to achieve the update of the target network. During training, a batch of samples and training samples need to be uniformly and randomly selected from the experience replay pool and mixed together as training data to destroy the correlation of the samples. The DQN error function Loss is the square of the difference between the Q value output by the target network and the evaluation network:

[0093] Loss=(Q DQN (s DQN_t ,a DQN_t θ DQN )-targetQ DQN_t ) 2 (5)

[0094] Among them, Q DQN (s DQN_t ,a DQN_t θ DQN ) is the Q value of the evaluation network output, targetQ DQN_t is the Q value output by the target network. DQN Represents the network parameters of the evaluation network.

[0095] Among them, targetQ DQN_t The calculation formula is as follows:

[0096]

[0097] in, Represents the network parameters of the target network.

[0098] in addition, Figure 3 s′ DQN represents the state at the next moment, a′ DON Represents s′ DQN The action corresponding to the optimal Q value under .

[0099] After calculating the loss value, DQN updates the network parameters θ through the gradient descent algorithm DQN , the gradient descent formula is as follows:

[0100]

[0101] in, represents the gradient, and E represents the expected function.

[0102] The Q value update method in traditional DQN is as follows:

[0103]

[0104] Among them, γ is the learning rate, μ is the discount factor (used to reduce the impact of new values), both range between 0 and 1, r DQN For state s DQN Take action a DQN The reward obtained after this method, due to the max function, causes the strategy to be based on the next moment state s′ DQN To choose a DQN Let the next moment Q DQN This method will make the algorithm exploration ability poor, the network training efficiency low, and affect the network convergence speed and CQI prediction effect. This application proposes to improve the Q value calculation method (i.e., action-value function) to reduce the number of times the intelligent agent enters repeated states during training, so that it can better explore new states. At the same time, it improves the speed of network convergence and optimizes the effect of CQI prediction.

[0105] In the embodiment of the present application, the Q value updating method satisfies the above formula (4). The larger M is, the larger Q is. DQN (s DQN ,a DQN ) is smaller. When this value decreases, the agent tends to choose other actions in the action space, thereby exploring new states, improving the network convergence speed, and avoiding convergence to the local optimum.

[0106] The embodiment of the present application redesigns the Q value update and improves the reward function, so that the model can converge at a faster speed and explore the optimal strategy, thereby improving the prediction effect of CQI.

[0107] like Figure 4 As shown, the embodiment of the present application also provides a channel quality indicator value prediction method using the target DQN model as described above, including:

[0108] Step 401: Acquire a channel quality indicator CQI value reported by a terminal and a CQI prediction value of the target DQN model.

[0109] Step 402: Obtain a target CQI prediction value according to the CQI value reported by the terminal, the CQI prediction value, and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

[0110] In the embodiment of the present application, at the target time, the target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value and the weight value of the CQI prediction value. The target time refers to the time other than the time when the UE feeds back the CQI.

[0111] Optionally, the target CQI prediction value satisfies the following formula:

[0112] CQI pre =αCQI DQN +(1-α)CQI UE ;

[0113] Among them, CQI pre represents the target CQI prediction value, α represents the weight value, CQI DQN Indicates the CQI prediction value, CQI UE Indicates the CQI value reported by the terminal.

[0114] Wherein, 0<α<1, α=0.3 at initialization, and the value of α is dynamically adjusted to the above weights at intervals of a preset time length (e.g., 5 minutes) according to the prediction effect of the CQI prediction value. When the CQI prediction effect is good, the value of α is increased, and when the CQI prediction effect is poor, the value of α is reduced, so as to give full play to the effect of the CQI value fed back by the UE and improve the accuracy of the model prediction.

[0115] In the embodiment of the present application, when the UE feeds back the CQI, the CQI value fed back by the UE, i.e., CQI UE , as CQI pre , and at other times (i.e. the target time mentioned above), CQI DQN and CQI UEPerform weighted summation to obtain the final CQI prediction value CQI pre .

[0116] The solution of the embodiment of the present application uses the value obtained by weighted summing the current CQI prediction value of the model and the CQI value fed back by the UE as the final CQI prediction value of the model, and adjusts the weight according to the prediction effect, so as to give full play to the effect of the CQI value fed back by the UE and improve the accuracy of the model prediction.

[0117] like Figure 5 As shown, the embodiment of the present application provides a model training device, comprising:

[0118] A first acquisition module 501 is used to obtain state information according to a channel quality indication CQI value reported by a terminal and hybrid automatic repeat request HARQ feedback information, wherein the state information is training data of a deep Q network DQN model;

[0119] The second acquisition module 502 is used to train the DQN model according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0120] Optionally, the device of the embodiment of the present application further includes:

[0121] A determination module, configured to determine that the prediction effect of the DQN model is a first type of prediction effect, and the reward value corresponding to the first type of prediction effect is a first value when the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is less than or equal to a first preset threshold; or

[0122] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the first preset threshold and less than or equal to the second preset threshold, it is determined that the prediction effect of the DQN model is a second type of prediction effect, and the reward value corresponding to the second type of prediction effect is a second value, wherein the second preset threshold is greater than the first preset threshold; or

[0123] When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the second preset threshold, it is determined that the prediction effect of the DQN model is a third type of prediction effect, and the reward value corresponding to the third type of prediction effect is a third value.

[0124] Optionally, the third value is obtained based on the ratio of a first value to a second value, wherein the first value is the difference between a block error rate at a first moment and a block error rate at a second moment, and the second value is a larger value between the block error rate at the first moment and the block error rate at the second moment, and the second moment is the moment before the first moment.

[0125] Optionally, the third value satisfies the following formula:

[0126]

[0127] Among them, r DQN_t Represents the reward value at the first moment, BLER t-1 Indicates the block error rate at the second moment, BLER t Indicates the block error rate at the first moment.

[0128] Optionally, the second acquisition module is used to:

[0129] The DQN model is trained according to the training data, the reward function and the action-value function to obtain a target DQN model, wherein the action-value function is a function for obtaining the value Q corresponding to an action, the action-value function is negatively correlated with the number of times a target action is selected under the state information, and the target action corresponds to a CQI value.

[0130] Optionally, the action-value function satisfies the following formula:

[0131]

[0132] Among them, s DQN Indicates status information, the a DQN Indicates that based on the preset strategy in s DQN The target action selected is , γ represents the learning rate, and M is s DQN Next select a DQN The number of times.

[0133] In an embodiment of the present application, state information is obtained according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, and the state information is the training data of the deep Q network DQN model; the DQN model is trained according to the training data and the reward function to obtain the target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects. Compared with the reward function that only includes two reward values ​​in the existing solution, this reward function increases the number of reward values. Based on this reward function, the DQN model can obtain more reward values ​​under more prediction effects, thereby improving the prediction effect of the algorithm and accelerating the convergence of the network.

[0134] like Figure 6 As shown, the embodiment of the present application also provides a channel quality indicator value prediction device using the target DQN model as described above, including:

[0135] The third acquisition module 601 is used to obtain the channel quality indicator CQI value reported by the terminal and the CQI prediction value of the target DQN model;

[0136] The fourth acquisition module 602 is used to acquire a target CQI prediction value according to the CQI value reported by the terminal, the CQI prediction value and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

[0137] Optionally, the target CQI prediction value satisfies the following formula:

[0138] CQI pre =αCQI DQN +(1-α)CQI UE ;

[0139] Among them, CQI pre represents the target CQI prediction value, α represents the weight value, CQI DQN Indicates the CQI prediction value, CQI UE Indicates the CQI value reported by the terminal.

[0140] The solution of the embodiment of the present application uses the value obtained by weighted summing the current CQI prediction value of the model and the CQI value fed back by the UE as the final CQI prediction value of the model, and adjusts the weight according to the prediction effect, so as to give full play to the effect of the CQI value fed back by the UE and improve the accuracy of the model prediction.

[0141] like Figure 7 As shown, the embodiment of the present application further provides a model training device 700, including: a transceiver 720 and a processor 710;

[0142] The processor 710 is used to obtain state information according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, where the state information is the training data of the deep Q network DQN model; according to the training data and the reward function, the DQN model is trained to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

[0143] The device can implement all implementation methods of the above-mentioned model training method embodiment and can achieve the same technical effect, which will not be repeated here.

[0144] like Figure 8As shown, the embodiment of the present application also provides a channel quality indicator value prediction device 800 using the target DQN model as described above, including: a transceiver 820 and a processor 810;

[0145] The processor 810 is configured to obtain a channel quality indicator CQI value reported by a terminal and a CQI prediction value of the target DQN model;

[0146] A target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

[0147] The device can implement all implementation methods of the above-mentioned channel quality indicator value prediction method embodiment and can achieve the same technical effect, which will not be repeated here.

[0148] like Fig. 9 As shown, an embodiment of the present application also provides a model training device, including: a transceiver 910, a processor 900, a memory 920, and a program or instruction stored in the memory 920 and executable on the processor 900; when the processor 900 executes the program or instruction, the steps in the model training method as described above are implemented.

[0149] The transceiver 910 is used to receive and send data under the control of the processor 900 .

[0150] Among them, Fig. 9 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 900 and memory represented by memory 920. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 may store data used by the processor 900 when performing operations.

[0151] The present application also provides a channel quality indicator value prediction device, such as Fig. 9 As shown, it includes: a transceiver 910, a processor 900, a memory 920, and a program or instruction stored in the memory 920 and executable on the processor 900; when the processor 900 executes the program or instruction, the steps in the channel quality indicator value prediction method as described above are implemented.

[0152] An embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the model training method or channel quality indicator value prediction method as described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0153] The processor is a processor in the model training device or channel quality indicator value prediction described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0155] In the embodiment of the present application, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0156] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0157] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0158] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present application. Therefore, the present application should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present application will be perfect and complete, and the scope of the present application will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "comprising" and / or "including" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0159] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: According to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, the state information is obtained, and the state information is the training data of the deep Q network DQN model; The DQN model is trained according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

2. The method according to claim 1, characterized in that Also includes: When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is less than or equal to a first preset threshold, determining that the prediction effect of the DQN model is a first type of prediction effect, and the reward value corresponding to the first type of prediction effect is a first value; or When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the first preset threshold and less than or equal to the second preset threshold, it is determined that the prediction effect of the DQN model is a second type of prediction effect, and the reward value corresponding to the second type of prediction effect is a second value, wherein the second preset threshold is greater than the first preset threshold; or When the absolute value of the difference between the CQI prediction value output by the DQN model and the CQI value reported by the terminal is greater than the second preset threshold, it is determined that the prediction effect of the DQN model is a third type of prediction effect, and the reward value corresponding to the third type of prediction effect is a third value.

3. The method according to claim 2, characterized in that The third value is obtained based on the ratio of the first value to the second value, the first value is the difference between the block error rate at the first moment and the block error rate at the second moment, the second value is the larger value of the block error rate at the first moment and the block error rate at the second moment, and the second moment is the moment before the first moment.

4. The method according to claim 3, characterized in that: The third value satisfies the following formula: Among them, r DQN_t Represents the reward value at the first moment, BLER t-1 Indicates the block error rate at the second moment, BLER t Indicates the block error rate at the first moment.

5. The method according to claim 1, characterized in that The step of training the DQN model according to the training data and the reward function to obtain a target DQN model includes: The DQN model is trained according to the training data, the reward function and the action-value function to obtain a target DQN model, wherein the action-value function is a function for obtaining the value Q corresponding to an action, the action-value function is negatively correlated with the number of times a target action is selected under the state information, and the target action corresponds to a CQI value.

6. The method according to claim 5, characterized in that The action-value function satisfies the following formula: Among them, s DQN Indicates status information, a DQN Indicates that based on the preset strategy in s DQN The target action selected is , γ represents the learning rate, and M is s DQN Next select a DQN The number of times.

7. A method for predicting a channel quality indicator value using the target DQN model according to any one of claims 1 to 6, characterized in that: include: Acquire a channel quality indicator CQI value reported by the terminal and a CQI prediction value of the target DQN model; A target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

8. The method according to claim 7, characterized in that The target CQI prediction value satisfies the following formula: CQI pre =αCQI DQN +(1-α)CQI UE ; Among them, CQI pre represents the target CQI prediction value, α represents the weight value, CQI DQN Indicates the CQI prediction value, CQI UE Indicates the CQI value reported by the terminal.

9. A model training device, characterized in that: include: A first acquisition module is used to obtain state information according to a channel quality indication CQI value reported by a terminal and hybrid automatic repeat request HARQ feedback information, wherein the state information is training data of a deep Q network DQN model; The second acquisition module is used to train the DQN model according to the training data and the reward function to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

10. A channel quality indicator value prediction device using the target DQN model according to any one of claims 1 to 6, characterized in that: include: A third acquisition module is used to obtain a channel quality indication CQI value reported by the terminal and a CQI prediction value of the target DQN model; The fourth acquisition module is used to acquire a target CQI prediction value according to the CQI value reported by the terminal, the CQI prediction value and the weight value of the CQI prediction value, wherein the weight value is obtained according to the prediction effect of the target DQN model.

11. A model training device, characterized in that: include: transceivers and processors; The processor is used to obtain state information according to the channel quality indication CQI value reported by the terminal and the hybrid automatic repeat request HARQ feedback information, wherein the state information is the training data of the deep Q network DQN model; according to the training data and the reward function, the DQN model is trained to obtain a target DQN model, wherein the reward function includes reward values ​​corresponding to three types of prediction effects, and the target DQN model is used to predict the CQI value.

12. A channel quality indicator value prediction device using the target DQN model according to any one of claims 1 to 6, characterized in that: include: transceivers and processors; The processor is used to obtain a channel quality indication CQI value reported by the terminal and a CQI prediction value of the target DQN model; A target CQI prediction value is obtained according to the CQI value reported by the terminal, the CQI prediction value and a weight value of the CQI prediction value, wherein the weight value is obtained according to a prediction effect of the target DQN model.

13. A model training device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the steps of the model training method as described in any one of claims 1 to 6 are implemented.

14. A channel quality indicator value prediction device using the target DQN model according to any one of claims 1 to 6, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the steps of the channel quality indicator value prediction method as described in any one of claims 7 to 8 are implemented.

15. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by the processor, the steps of the model training method as described in any one of claims 1 to 6 are implemented, or the steps of the channel quality indicator value prediction method as described in any one of claims 7 to 8 are implemented.

Images