An Adaptive Pilot Method Based on Deep Reinforcement Learning

Abstract

The present invention discloses an adaptive pilot method based on deep reinforcement learning. The method includes: the transmitting end obtains the channel statistical information returned by the receiving end; based on the channel statistical information, a pre-trained pilot decision model based on deep reinforcement learning is used for decision-making to obtain a pilot decision result; according to the pilot decision result, the pilot spacing and pilot power in the time-frequency domain are adjusted; wherein the pilot decision model based on deep reinforcement learning includes two evaluation neural networks and a target neural network with the same structure; the ε-greedy greedy algorithm is used for training until a preset condition is reached to obtain a trained pilot decision model based on deep reinforcement learning. This model can adjust the pilot distribution and power according to the statistical characteristics of the time-varying channel to adapt to the non-stationary channel characteristics. While ensuring the channel estimation accuracy, it effectively reduces the pilot quantity and power overhead, and improves the working efficiency of the communication system.

Description

Technical Field

[0001] The present invention belongs to the field of information and communication technologies and relates to an adaptive pilot method based on deep reinforcement learning in an orthogonal frequency division multiplexing (OFDM) system. Background Art

[0002] Channel estimation plays a key role in improving the performance of wireless communication systems. To overcome the adverse effects brought about by channel conditions, a communication system needs to perform link adaptation, channel equalization, adaptive transmission, etc. according to channel state information to improve system performance. In an actual OFDM system, a transmitting party inserts pilot symbols into data symbols, and a receiving party can obtain complete channel state information in the time-frequency domain through a channel estimation algorithm. However, in a highly dynamic channel environment, such as a vehicle-to-vehicle channel or an air-to-ground channel, due to complex terrain environments, electromagnetic propagation environments, obstacles, and the mobility of both the transceiver and the receiver, the wireless channel exhibits significant dynamic change characteristics and non-stationary characteristics. The channel transition probability, the number and power of paths, the maximum multipath delay, and the maximum Doppler shift are all uncertain. Therefore, a pre-determined pilot pattern is difficult to adapt to changes in the channel environment, and more effective and flexible pilot patterns need to be explored to meet the dynamic channel environment.

[0003] Currently, most pilot patterns are designed based on the worst channel conditions, trading a large pilot density and pilot power for the accuracy of channel estimation. For example, in the LTE standard, the pilot interval and power are fixed, and it is specifically designed for the case where the center frequency is 2 GHz, the root mean square delay is τ rms = 991 ns, and the terminal speed is v = 500 km / h. However, for a time-frequency flat channel, the correlation between channels is relatively strong, and not many pilots are required for estimation. Since data information cannot be transmitted at pilot positions, too many pilots will cause a large system overhead, reducing the spectral efficiency and the performance of the communication system.

[0004] The principle of channel estimation mainly relies on a certain correlation between pilot symbols and data symbols. The change in correlation in the time domain is mainly caused by the time-varying coherence time due to the change in Doppler shift. The correlation in the frequency domain is due to the change in the multipath delay spread caused by the change in the scattering environment around the transceiver during movement, resulting in a time-varying coherence bandwidth. In addition, when there are obstacles between the transceiver and the receiver, the line-of-sight link disappears, and the channel will be in a deep fading state. At this time, affected by the deep fading, the energy of the received reference signal is extremely low, and the amplitude and phase information of the channel cannot be accurately estimated, affecting the channel estimation performance. Therefore, under deep fading channel conditions, pilot symbols with higher power need to be configured to ensure channel estimation performance.

[0005] In summary, the pilot spacing is affected by the channel coherence time and coherence bandwidth, while the power allocation of the pilot is affected by the channel fading degree. Therefore, the requirements for the pilot interval and power are different under different channel conditions. Too few pilots will lead to a decline in channel estimation performance. However, too many pilots will occupy the limited time-frequency resources, resulting in a decline in the performance of the communication system. Under the condition of limited total power, the greater the power allocated to the pilot symbol, the lower the power of the corresponding data symbol, which affects the decoding and demodulation of the actual information bits. The traditional fixed pilot pattern does not consider the characteristics of the dynamically changing non-stationary channel, so it cannot adaptively adjust the pilot pattern to meet the performance requirements of the system. Summary of the Invention

[0006] Objective: To overcome the deficiencies in the prior art and address the problem of the mismatch between the existing pilot pattern and the channel environment, the present invention provides an adaptive pilot method based on deep reinforcement learning, in which the transmitter can adaptively select different pilot intervals and powers according to the channel statistical information estimated by the receiver.

[0007] Technical Solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides an adaptive pilot method based on deep reinforcement learning, including:

[0009] The transmitter obtains the channel statistical information returned by the receiver;

[0010] Based on the channel statistical information, a pilot decision model based on deep reinforcement learning that has been pre-trained is used for decision-making to obtain a pilot decision result;

[0011] According to the pilot decision result, the pilot spacing and pilot power in the time-frequency domain are adjusted;

[0012] The pilot decision model based on deep reinforcement learning includes two evaluation neural networks and a target neural network with the same structure; the ε-greedy greedy algorithm is used for training until a preset condition is reached to obtain a trained pilot decision model based on deep reinforcement learning.

[0013] In some embodiments, obtaining the channel statistical information returned by the receiver includes:

[0014] The state in the k-th frame is represented as where respectively represent the time-domain correlation vector, frequency-domain correlation vector, and signal-to-noise ratio statistical characteristics in the k-th frame;

[0015] The calculation formula for the channel statistical information is as follows:

[0016] Correlation coefficient in the time domain:

[0017]

[0018] Correlation coefficient in the frequency domain:

[0019]

[0020] Statistical characteristics of signal-to-noise ratio:

[0021]

[0022] Among them, represents the non-stationary channel matrix in the nth frame, which contains N D OFDM symbols and N S subcarriers; diag m vectorizes the mth diagonal in matrix X; {diag m []} q represents the qth element in the mth diagonal vector; p k and represent the power of the kth frame and the average channel gain respectively;

[0023] According to the correlation coefficients of the channel in the time and frequency domains, obtain the correlation vectors of the non-stationary channel in the time and frequency domains:

[0024]

[0025] In some embodiments, a training method for a pilot decision-making model based on deep reinforcement learning includes:

[0026] Step S1. Initialize the parameters of the deep reinforcement learning DQN network and the iteration number k = 1, and set the size of the experience replay unit and the maximum number of frames k max ;[[ID=?]]

[0027] Step S2. The agent receives the channel statistical information s k in the kth frame, and randomly selects an action a k = a k ' with probability 1-ε, otherwise selects the action with the largest output Q value where θ is the evaluation network parameter;

[0028] Step S3. Execute the action a k and update the next state to s k+1 ;

[0029] Step S4. Evaluate the channel normalized estimation error function NMSE k and the cost function Based on the channel normalized estimation error function NMSE It should be noted that there seems to be an unclear or incorrect tag in the original text at line 49 (tag ). This might cause some confusion in the translation. If possible, it would be beneficial to clarify or correct the original text for a more accurate translation.k Sum cost function Obtain the reward function r k (s k , a k );

[0030] Step S5. Store (s k , a k , r k , s k+1 ) into the experience replay unit;

[0031] Step S6. Randomly sample historical data of mini - batch size from the experience replay unit to calculate the target value, use the RMSProp optimizer to minimize the loss function, and iteratively update the evaluation network parameter θ; after 1000 iterations, replace the target network parameter θ with the evaluation network parameter θ - ;

[0032] Step S7. k = k + 1, if k > k max , then the training ends; otherwise, return to step 2.

[0033] Furthermore, in some embodiments, represent the action of the k - th frame as a k = {D f (k), D t (k), ρ(k)}, where D f (k) and D t (k) respectively represent the intervals of pilot symbols in the time domain and frequency domain; based on the possible coherence bandwidth and coherence time, use and to respectively represent the optional sets of time - frequency domain pilot intervals; ρ(k) is the ratio of the pilot symbol power to the data symbol power, and its optional space is Therefore, the size of the agent's action space is T * F * Q, where T and F are the numbers of optional pilot intervals in the time domain and frequency domain respectively, and Q is the number of optional power ratios.

[0034] Furthermore, in some embodiments, the channel normalized estimation error function NMSE k , includes:

[0035]

[0036] Among them, and respectively represent the actual channel response and the estimated channel response at the v - th sub - carrier of the u - th OFDM symbol, N D and N S respectively represent the number of OFDM symbols and sub - carriers.

[0037] Further, in some embodiments, a cost function is used to represent the channel estimation overhead brought by the pilot number and power, and the expression is:

[0038]

[0039] where p k represents the pilot power in the k-th frame, c b and N k respectively represent the bandwidth transmission cost and the pilot number of the pilot in the k-th frame; p max and c p respectively represent the maximum transmission power and its transmission cost.

[0040] Further, in some embodiments, based on the channel normalized estimation error function NMSE k and the cost function a reward function r k (s k , a k ) is obtained, including:

[0041] The reward function r k (s k , a k ) in the k-th frame:

[0042] [[ID=F3]]

[0043] where when NMSE k is less than or equal to the threshold T th , it is considered that the performance requirement is met, and a positive reward u1 is obtained; otherwise, it indicates that the estimation effect is not good, and a penalty -u2 is obtained.

[0044] Further, in some embodiments, calculating the target value includes:

[0045]

[0046] where, and θ - are the Q function and weight parameters of the target network, and A represents the set of actions;

[0047] The expression of the loss function Loss(θ) is as follows:

[0048]

[0049] where E[.] is the mean function, Q(s k , a k |θ) is the state - action Q value, and θ is the weight parameter of the evaluation network.

[0050] In a second aspect, the present invention provides an adaptive pilot device based on deep reinforcement learning, including:

[0051] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented. Beneficial effects: The adaptive pilot method based on deep reinforcement learning provided by the present invention has the following advantages: The design problem of pilot patterns is modeled into a Markov decision process (MDP) process, and a method based on deep reinforcement learning is proposed to solve the pilot design decision problem in an actual wireless time-varying and non-stationary channel environment. The deep Q-network (DQN) can adjust the pilot distribution and power according to the statistical characteristics of channel time variation to adapt to the non-stationary channel characteristics. While ensuring the channel estimation accuracy, it effectively reduces the number of pilots and power overhead, and improves the working efficiency of the communication system. Description of the Drawings

[0052] Figure 1 Schematic diagram of pilot configuration under different channel conditions according to an embodiment of the present invention;

[0053] Figure 2 Flowchart of the DQN-based adaptive pilot configuration algorithm according to an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of the DQN structure according to an embodiment of the present invention.

[0055] Figure 4 Flowchart of the adaptive pilot method based on deep reinforcement learning according to an embodiment of the present invention. Detailed Embodiments

[0056] The present invention will be further described below in conjunction with the drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and cannot be used to limit the protection scope of the present invention.

[0057] In the description of the present invention, the meaning of several is more than one, the meaning of multiple is more than two, greater than, less than, exceeding, etc. are understood as not including the present number, and above, below, within, etc. are understood as including the present number. If there is a description of first and second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0058] In the description of the present invention, the description with reference to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0059] Embodiment 1

[0060] An adaptive pilot method based on deep reinforcement learning, comprising:

[0061] The transmitter obtains the channel statistical information returned by the receiver;

[0062] Based on the channel statistical information, a pre-trained pilot decision model based on deep reinforcement learning is used for decision-making to obtain a pilot decision result;

[0063] According to the pilot decision result, the pilot spacing and pilot power in the time-frequency domain are adjusted;

[0064] Wherein the pilot decision model based on deep reinforcement learning includes two evaluation neural networks and a target neural network with the same structure; the ε-greedy greedy algorithm is used for training until a preset condition is reached to obtain a trained pilot decision model based on deep reinforcement learning.

[0065] In some embodiments, an adaptive pilot method based on deep reinforcement learning, the adaptive pilot pattern is as shown in the appendix Figure 1 As shown. In frame (a), when the channel correlation is weak and the fading degree is strong, it is necessary to reduce the pilot interval and configure high-power pilots at this time; in frame (b), when the channel correlation increases and the fading degree decreases, the pilot interval should be increased and low-power pilots should be configured at this time.

[0066] In the method of this embodiment, the pilot pattern design problem in each OFDM frame is considered. The receiver extracts statistical information such as the channel correlation vector and the channel gain power through the preamble symbol in the frame header and sends it back to the transmitter through the feedback link. Based on the statistical information, the transmitter adjusts the time-frequency pilot spacing and pilot power so that the channel estimation accuracy meets the requirements of the performance index and minimizes the performance loss caused by the pilot overhead to the system. Since the link state transition probability in the wireless channel is unknown, the actual channel characteristics are time-varying, so a dynamic pilot configuration design problem arises.

[0067] To solve this dynamic decision-making problem, the present invention models the pilot design problem in a time-varying and non-stationary channel environment as a Markov Decision Process (MDP). Using the three elements of state S, action A, and reward function R to represent the Markov decision process, the present invention defines the triple (S, A, R) as follows:

[0068] (1) State space

[0069] The state in the k-th frame is represented as where respectively represent the time-domain correlation vector, frequency-domain correlation vector, and signal-to-noise ratio statistical characteristics in the k-th frame. The calculation formulas for the channel statistical information are as follows:

[0070] Correlation coefficient in the time domain:

[0071]

[0072] Correlation coefficient in the frequency domain:

[0073]

[0074] Signal-to-noise ratio statistical characteristics:

[0075]

[0076] where represents the non-stationary channel matrix in the n-th frame, which contains N D OFDM symbols and N S sub-carriers. diag m [·] vectorizes the m-th diagonal in matrix X; {diag m []} q represents the q-th element in the m-th diagonal vector; p k and respectively represent the power in the k-th frame and the average channel gain; According to the correlation coefficient in the time domain and the correlation coefficient in the frequency domain the correlation vectors of the channel in the time-frequency domain can be further obtained:

[0077]

[0078] (2) Action space

[0079] The action in the k-th frame is represented as a k ={D f (k), D t (k), ρ(k)}, where D f(k) and D t (k) represent the intervals of pilot symbols in the time domain and frequency domain respectively. Based on the possible coherence bandwidth and coherence time, use and to represent the optional sets of pilot intervals in the time-frequency domain respectively. ρ(k) is the ratio of the power of pilot symbols to the power of data symbols, and its optional space is Therefore, the size of the agent's action space is T*F*Q, where T and F are the numbers of optional pilot intervals in the time domain and frequency domain respectively, and Q is the number of optional power ratios.

[0080] (3) Reward function

[0081] In state s k , execute action a k , and the corresponding reward value r k (s k , a k ) is obtained. Under dynamically changing channel conditions, while ensuring the accuracy of channel estimation, the system overhead should also be reduced. Here, the channel normalized estimation error function for the k-th frame is defined as:

[0082]

[0083] Among them, and represent the actual channel response and the estimated channel response at the v-th subcarrier of the u-th OFDM symbol respectively. When the accuracy of channel estimation is less than or equal to the threshold T th , it is considered that the performance requirements are met, and a positive reward u1 is obtained; otherwise, it indicates that the estimation effect is not good, and a penalty -u2 is obtained.

[0084] Since pilot symbols will occupy the bandwidth for data transmission, the more pilot symbols are configured, the larger the bandwidth occupied by pilot symbols, and the available bandwidth for data transmission decreases. In addition, due to the limited system power, the higher the power of the configured pilot symbols will lead to a decrease in the power of data symbols, which will further cause a decrease in the system transmission capacity. Therefore, the present invention designs a cost function to represent the channel estimation overhead caused by the number of pilots and power, and its expression can be written as:

[0085]

[0086] Among them, p k represents the pilot power in the k-th frame, c b and N k represent the bandwidth transmission cost and the number of pilots of the k-th frame respectively. p max and c p represent the maximum transmission power and its transmission cost respectively. Therefore, the reward function r k (sk , a k ) is jointly determined by the estimated performance metric and the cost function. Therefore, the reward function in the k-th frame can be written as

[0087]

[0088] This method uses a Deep Q Network (DQN) to optimize the pilot interval and power to maximize the estimation performance while reducing the system overhead. During the learning process, the agent continuously interacts with the unknown wireless channel environment, explores the complex variation laws of the channel, and makes decisions using the obtained channel characteristics to obtain the optimal pilot design strategy, so that the pilot pattern matches the channel characteristics, thereby maximizing the long-term cumulative reward function.

[0089]

[0090] where γ k is the discount factor, which is used to characterize the impact of future rewards on the current reward. The flow chart is as shown in the appendix Figure 2 .

[0091] The DQN contains two neural networks with the same structure, namely the evaluation neural network and the target neural network, as shown in the appendix Figure 3 . The observed environmental state s k is fed as input into the evaluation neural network, and the state-action Q-value Q(s k , a k |θ) is output, where θ is the parameter of the evaluation neural network. To avoid falling into local optimization, the ε-greedy algorithm is used to select the action a k to balance the relationship between exploration and exploitation. The agent executes the action a k , and the state transfers to the next state s k+1 , and the benefit function r k (s k , a k ) is obtained. To improve the sample utilization rate, the DQN algorithm stores (s k , a k , r k , s k+1 ) in the experience replay unit. The target value can be calculated by the formula:

[0092]

[0093] where, and θ -are the Q-function and weight parameters of the target network. In each iteration of the training phase, the DQN algorithm randomly samples a certain number of samples from the experience replay unit to update the neural network parameters, aiming to ensure the independence between samples. The network parameters θ are iteratively optimized by minimizing the loss function between the state-action value and the target value. The expression of the loss function is as follows:

[0094]

[0095] After 1000 iterations, the weight parameters of the evaluation network θ are used to replace the weight parameters θ of the target network - .

[0096] (4) The algorithm is as shown in the appendix Figure 4 as follows. The specific steps are as follows:

[0097] 1. Initialize the DQN network parameters and the iteration number k = 1, and set the size of the experience replay unit and the maximum number of frames k max ,

[0098] 2. The agent receives the channel statistics information in the k-th frame Randomly select an action a with probability 1-ε k ', otherwise select the action with the maximum Q value

[0099] 3. Execute the action a k , and update the next state s k+1

[0100] 4. Evaluate the NMSE and system overhead to obtain the reward function r k (s k ,a k ), that is

[0101]

[0102] 5. Store (s k ,a k ,r k ,s k+1 ) into the experience replay unit

[0103] 6. Randomly sample historical data of mini-batch size from the experience replay unit to calculate the target value The loss function Loss(θ), use the RMSProp optimizer to minimize the loss function, and iteratively update the weight parameters in the evaluation network. After 1000 iterations, use the evaluation network parameters to replace the target network parameters.

[0104] 7. k = k + 1, if k > k max , then the training ends; otherwise, return to step 2.

[0105] Example 2

[0106] Second, this embodiment provides an adaptive pilot device based on deep reinforcement learning, including a processor and a storage medium;

[0107] The storage medium is used to store instructions;

[0108] The processor is configured to operate according to the instructions to execute the steps of the method according to Embodiment 1.

[0109] Example 3

[0110] Third, this embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to Embodiment 1 are implemented.

[0111] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one or more of the processes Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0113] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in one or more of the processes Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps for implementing the functions specified in one block or a plurality of blocks.

[0115] The above are only the preferred embodiments of the present invention, and it should be pointed out that: for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. An adaptive pilot method based on deep reinforcement learning, characterized in that Including: The sending end obtains the channel statistical information returned by the receiving end; The channel statistical information includes time-domain correlation vectors, frequency-domain correlation vectors, and signal-to-noise ratio statistical characteristics in each frame; Based on the channel statistical information, a pre-trained pilot decision model based on deep reinforcement learning is used for decision-making to obtain a pilot decision result; According to the pilot decision result, the pilot spacing and pilot power in the time-frequency domain are adjusted; The pilot decision model based on deep reinforcement learning includes two evaluation neural networks and a target neural network with the same structure; The ε-greedy greedy algorithm is used for training until a preset condition is reached to obtain a trained pilot decision model based on deep reinforcement learning; A training method for a pilot decision model based on deep reinforcement learning includes: Step S1. Initialize the parameters of the deep reinforcement learning DQN network and the iteration number k = 1, and set the size of the experience replay unit and the maximum number of frames k max ; Step S2. The agent receives the channel statistics s in the k-th frame k , and randomly selects an action a with probability 1 - ε k = a k ', otherwise select the action with the largest Q value output where θ is the evaluation network parameter; Step S3. Perform action a k and update the next state to s k+1 ; Step S4. Evaluate the channel normalized estimation error function NMSE of the k-th frame k and the cost function Based on the channel normalized estimation error function NMSE k and the cost function Obtain the reward function r k (s k ,a k ); Step S5. Store (s k , a k , r k , s k+1 ) into the experience replay unit; Step S6. Randomly sample historical data of mini-batch size from the experience replay unit to calculate the target value, use the RMSProp optimizer to minimize the loss function, and iteratively update the evaluation network parameters θ; after 1000 iterations, replace the target network parameters θ with the evaluation network parameters θ - ; Step S7. k = k + 1. If k > k max , then the training ends; otherwise, return to Step 2; Denote the action of the k-th frame as a k ={D f (k), D t (k), ρ(k)}, where D f (k) and D t (k) represent the intervals of pilot symbols in the time domain and the frequency domain respectively; and represent the optional sets of pilot intervals in the time-frequency domain respectively based on the possible coherence bandwidth and coherence time; ρ(k) is the ratio of the power of pilot symbols to the power of data symbols, and its optional space is Therefore, the size of the action space of the agent is T * F * Q, where T and F are the numbers of optional pilot intervals in the time domain and the frequency domain respectively, and Q is the number of optional power ratios; Channel normalized estimation error function NMSE of the k-th frame k , including: Among them, and respectively represent the actual channel response and the estimated channel response at the v-th subcarrier in the u-th OFDM symbol, N D and N S respectively represent the number of OFDM symbols and the number of subcarriers; Using a cost function represents the channel estimation overhead caused by the number of pilots and power, and the expression is: where p k represents the pilot power in the k-th frame, c b and N k represent the bandwidth transmission cost and the number of pilots of the pilot in the k-th frame respectively; p max and c p represent the maximum transmission power and its transmission cost respectively; Based on the channel normalization estimation error function NMSE k and the cost function the reward function r is obtained k (s k , a k ) includes: The reward function r in the k-th frame k (s k , a k ): where NMSE k is less than or equal to the threshold T th then it is considered that the performance requirements are met, and a positive reward u1 is obtained; otherwise, it indicates that the estimation effect is poor, and a penalty -u2 is obtained.

2. The adaptive pilot method based on deep reinforcement learning according to claim 1, wherein Obtaining the channel statistical information returned by the receiving end, including: Denote the state in the k-th frame as where represent the time-domain correlation vector, the frequency-domain correlation vector, and the SNR statistical characteristics in the k-th frame, respectively; The calculation formula of the channel statistical information is as follows: Correlation coefficient in the time domain: Correlation coefficient in the frequency domain: Signal-to-noise ratio statistical characteristics: Among them, represents the non-stationary channel matrix in the nth frame, which contains N D OFDM symbols and N S subcarriers; diag m [X] represents the vectorization of the mth diagonal in matrix X; {diag m [X]} q represents the qth element in the mth diagonal vector; p k and respectively represent the transmission power and average channel gain of the kth frame; According to the correlation coefficients in the time and frequency domains of the channel, the correlation vectors of the non-stationary channel in the time and frequency domains are obtained; 3. The adaptive pilot method based on deep reinforcement learning according to claim 1, wherein Calculate the target value including: where, and θ - are the Q function and weight parameters of the target network, and A represents the set of actions; The expression of the loss function Loss(θ) is as follows: where E[.] is the mean function, Q(s k , a k |θ) is the state-action Q-value, and θ is the weight parameter of the evaluation network.

4. An adaptive pilot device based on deep reinforcement learning, characterized in that, Including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images