Radio frequency front end guiding method based on deep reinforcement learning

Abstract

The invention discloses a radio frequency front end guiding method based on deep reinforcement learning, and belongs to the technical field of radio frequency front end intelligent frequency sweeping in a complex electromagnetic environment. According to the invention, the problem of low frequency sweeping efficiency of the radio frequency front end in the existing method is solved. According to the method, nonlinear frequency sweeping and deep reinforcement learning are adopted, learning training is performed on a deep reinforcement learning agent, and a trained radio frequency front-end intelligent guide frame is obtained; through the trained deep reinforcement learning agent and the nonlinear frequency sweeping mode, the radio frequency front end can be intelligently guided to reach the important frequency band of the complex electromagnetic environment, the frequency sweeping speed, the frequency sweeping precision and the intelligence of the radio frequency front end are improved, the radio frequency front end is guided to rapidly sweep frequency to align the important frequency band, and autonomous tuning of the radio frequency front end is completed. The method can be applied to radio frequency front-end intelligent frequency sweeping in a complex electromagnetic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of radio frequency front-end intelligent frequency scanning in complex electromagnetic environments, and specifically relates to a radio frequency front-end guidance method based on deep reinforcement learning. Background Art

[0002] As the electromagnetic environment in which communication equipment is located becomes increasingly complex, the RF front-end is required to acquire frequency bands quickly and intelligently. The frequency sweeping equipment needs to be able to locate the frequency band where the signal is located faster and more accurately, ensure the accuracy of the frequency sweeping, and be intelligent. However, relying solely on the traditional linear frequency sweeping technology of the RF front-end can no longer meet the needs of rapid acquisition of key frequency bands. To solve such problems, artificial intelligence technology has become very important for the intelligence of the RF front-end.

[0003] In order to improve the guided frequency scanning capability of the RF front-end, both traditional methods and artificial intelligence methods try their best to demonstrate their advantages while making up for their disadvantages. The traditional method considers the tuning capability of the RF front-end from a hardware perspective, and improves and optimizes the internal components to complete the tuning. However, the tuning range of the circuit improvement is only about 30MHz, which is unable to achieve large-band adjustment in the GHz range, and the linear frequency scanning method is still used, which cannot meet the user's fast and intelligent needs. In contrast, the use of deep reinforcement learning algorithm design is one of the key technologies to effectively improve the intelligence of equipment. Artificial intelligence technology targets the RF front-end, and through continuous interactive learning with the external environment, it improves the ability to quickly acquire key frequency bands, and supports a wide range of frequency band adjustments to achieve intelligent guidance of the RF front-end. However, the traditional RF front-end uses a linear frequency scanning method, which takes a long time and is inefficient.

[0004] In summary, the existing technology still has the problem of long RF front-end frequency scanning time and low frequency scanning efficiency. Therefore, it is still impossible to achieve the requirement of fast frequency scanning in a complex electromagnetic environment. Proposing a new RF front-end guidance method is an issue that needs to be urgently solved. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of low efficiency of RF front-end frequency scanning in existing methods, and propose a RF front-end guidance method based on deep reinforcement learning.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a radio frequency front-end guidance method based on deep reinforcement learning, the method specifically comprises the following steps:

[0007] Step 1: After the data center divides the frequency band information obtained by the RF front end into sub-bands, it performs spectrum sensing on each sub-band to obtain feature extraction results and importance evaluation results of each sub-band;

[0008] Step 2: The feature extraction results and importance evaluation results of each sub-frequency band in step 1 are used as the environment of the deep reinforcement learning agent. The state of the deep reinforcement learning agent is composed of the sub-frequency band center frequency, the feature extraction results of the sub-frequency band, and the sweep index. The action of the deep reinforcement learning agent is the center frequency adjustment value and the sweep index adjustment value.

[0009] The RF front end performs frequency sweep based on the actions output by the deep reinforcement learning agent at each step and the importance evaluation results in step one.

[0010] Furthermore, in step 1, the data center divides the frequency band information obtained by the RF front end into sub-bands, specifically:

[0011] The frequency range of the frequency band information obtained by the RF front end is recorded as 0 to L, where the unit of L is GHz. The frequency band information obtained by the RF front end is then divided into m sub-bands, and the length of each sub-band is T.

[0012] Furthermore, the spectrum sensing is performed on each sub-frequency band to obtain the feature extraction result of each sub-frequency band, specifically:

[0013] Take any sub-band as an example, and denote the signal in the sub-band as s T (t), for the signal s T (t) Perform Fourier transform to obtain the frequency domain signal s T (f), and according to the frequency domain signal s T (f) calculating the energy spectral density E(f) of the sub-band;

[0014] Then take the limit of the energy spectral density E(f) to obtain the power spectral density P(f) of the sub-band:

[0015]

[0016] Similarly, the energy spectral density and power spectral density of each sub-band are obtained respectively;

[0017] The energy spectrum density of each sub-band is normalized, and then the power spectrum density of each sub-band is normalized to obtain the normalized energy spectrum density and normalized power spectrum density of each sub-band respectively. The normalized energy spectrum density and normalized power spectrum density are used as the characteristics of the sub-band.

[0018] Furthermore, the specific process of performing spectrum sensing on each sub-frequency band to obtain the importance evaluation result is as follows:

[0019] The received signal energy of each sub-band is normalized, and then the energy detection threshold is compared with the normalized received signal energy value of each sub-band. If the normalized received signal energy value in a sub-band is greater than or equal to the threshold value, the importance of the sub-band is 1, otherwise, the importance of the sub-band is 0.

[0020] Furthermore, the initial state of the deep reinforcement learning agent consists of the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency sweep index.

[0021] Furthermore, the radio frequency front end adopts an exponential nonlinear frequency sweeping method.

[0022] Furthermore, the training method of the deep reinforcement learning agent is:

[0023] Step 1: Divide the frequency band information in the training set into sub-bands, and the lengths of the sub-bands obtained by division are equal; then extract the characteristics and importance evaluation results of each sub-band respectively;

[0024] The center frequency of the first sub-band, the initial frequency sweep index and the importance of the first sub-band form the initial state s0 of the agent;

[0025] Step 2, initialize t=0;

[0026] Step 3: The agent determines the state s t Select action a t , and obtain the agent's selected action a t The reward received t ;

[0027]

[0028] Calculate action a t The center frequency adjustment value in state s t If the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 1, then execute step 4; if the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 0, then directly execute step 5;

[0029] Step 4: According to action a t The sweep index adjustment value in state s t The sweep frequency index value in is adjusted, that is, the state s t The frequency sweep index value and action a in t The frequency sweep index adjustment value in the state s is added, and the RF front end tThe sub-frequency band corresponding to the center frequency of is taken as the starting sub-frequency band, the sub-frequency band corresponding to the sum of the calculated center frequencies is taken as the target sub-frequency band, and the frequency is swept according to the adjusted sweep index value to reach the target sub-frequency band;

[0030] Status t+1 The center frequency in is the sum of the calculated center frequencies, state s t+1 The sweep frequency index value in is the adjusted sweep frequency index value, and the state s t+1 The importance of is the importance of the sub-band corresponding to the sum of the calculated center frequencies, and the experience playback sample (s t ,a t ,r t ,s t+1 ) into the experience replay pool, and then proceed to step 6;

[0031] Step 5: Set state s t+1 The center frequency of state s t The center frequency, state s t+1 The sweep index value is state s t The sweep frequency index value and action a in t The sum of the sweep index adjustment values ​​in state s t+1 The importance of state s t The importance of; then proceed to step 6;

[0032] Step 6: Calculate the cumulative reward R, and then determine whether the network parameter update frequency has been reached based on t;

[0033] If not reached, go directly to step 7;

[0034] If it is reached, a fixed number of experience replay samples are taken from the experience replay pool, and the network parameters are updated using the taken experience replay samples, and then step 7 is executed;

[0035] Step 7, let t=t+1, and then determine whether t>T is satisfied;

[0036] If satisfied, the training ends;

[0037] If not satisfied, return to step 3.

[0038] Furthermore, the agent is configured according to the state s t Select action a t The ε-greedy strategy is adopted.

[0039] Furthermore, the cumulative reward R is:

[0040] R=r+r0+γr1+γ 2 r2+···+γ t rt

[0041] Among them, r is the initial reward value, r t represents the reward obtained in the tth step, and γ is the discount factor.

[0042] Furthermore, in step 2, the RF front end performs frequency sweeping according to the action output by the deep reinforcement learning agent at each step and the importance evaluation result in step 1, specifically:

[0043] Step (1), using the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency sweep index to form an initial state, using the initial state as the initial state of the trained deep reinforcement learning agent, and inputting the initial state into the deep reinforcement learning agent;

[0044] Step (2), the agent selects an action in the current state;

[0045] Step (3), calculating the sum L of the center frequency adjustment value in the selected action and the center frequency in the current state, and calculating the sum α of the sweep frequency index in the current state and the sweep frequency index adjustment value in the action;

[0046] If the importance of the sub-frequency band corresponding to the center frequency L is 1, the RF front end uses an exponential nonlinear frequency sweeping method to reach the sub-frequency band corresponding to the center frequency L, and the frequency sweeping index during the frequency sweeping is α; then execute step (4);

[0047] If the importance of the sub-band corresponding to the center frequency L is 0, execute step (5);

[0048] Step (4), update the agent state: replace the center frequency in the updated state with L, replace the sweep frequency index in the updated state with α, and the importance in the updated state with 1;

[0049] Then execute step (6);

[0050] Step (5), update the agent state: replace the sweep frequency index in the updated state with α, and the center frequency and importance in the updated state remain unchanged;

[0051] Then execute step (6);

[0052] Step (6): Use the updated state as the input of the deep reinforcement learning agent, and then return to step (2) until there is no sub-band with an importance of 1 after the sub-band currently scanned.

[0053] The beneficial effects of the present invention are:

[0054] The present invention adopts nonlinear frequency sweeping and deep reinforcement learning. By training the deep reinforcement learning agent, a trained RF front-end intelligent guidance framework is obtained. Through the trained deep reinforcement learning agent and nonlinear frequency sweeping method, the RF front-end can be intelligently guided to reach important frequency bands in complex electromagnetic environments, thereby improving the frequency sweeping speed, frequency sweeping accuracy and intelligence of the RF front-end, guiding the RF front-end to quickly sweep to align with important frequency bands, and completing autonomous tuning of the RF front-end. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the intelligent RF front-end system model;

[0056] It includes two stages: data perception and processing and learning and training feedback. The RF front-end cognitive engine is the deep reinforcement learning part.

[0057] Figure 2 This is a schematic diagram of the RF front-end intelligent guidance framework;

[0058] Figure 3 It is a schematic diagram of the relationship between the sweep index parameter, the sweep position and the sweep frequency;

[0059] Figure 4 This is a schematic diagram of the deep reinforcement learning module;

[0060] Figure 5 It is a flow chart of the RF front-end intelligent guidance method;

[0061] Figure 6 Schematic diagram of the convergence of the reward curve of the RF front-end intelligent guidance method. DETAILED DESCRIPTION

[0062] Specific implementation method 1: Combination Figure 1 and Figure 2 The present embodiment describes a method for guiding a radio frequency front end based on deep reinforcement learning, and the method specifically comprises the following steps:

[0063] Step 1: Using a complex electromagnetic environment to provide a frequency band environment obtained by the RF front end, the data center divides the frequency band information obtained by the RF front end into sub-bands, and then performs spectrum sensing on each sub-band to obtain feature extraction results and importance evaluation results for each sub-band;

[0064] Step 2: The feature extraction results and importance evaluation results of each sub-frequency band in step 1 are used as the environment of the deep reinforcement learning agent. The state of the deep reinforcement learning agent is composed of the sub-frequency band center frequency, the feature extraction results of the sub-frequency band, and the sweep index. The action of the deep reinforcement learning agent is the center frequency adjustment value and the sweep index adjustment value.

[0065] The RF front end performs frequency sweeping according to the actions output by the deep reinforcement learning agent at each step and the importance evaluation result in step one, so as to improve the frequency sweeping speed, frequency sweeping accuracy and intelligence of the RF front end.

[0066] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that, in step 1, the data center divides the frequency band information obtained by the RF front end into sub-bands, specifically:

[0067] The frequency range of the frequency band information obtained by the RF front end is recorded as 0 to L, where the unit of L is GHz. The frequency band information obtained by the RF front end is then divided into m sub-bands, and the length of each sub-band is T.

[0068] The other steps and parameters are the same as those in the first embodiment.

[0069] For example, assuming there is a frequency band of 0-40 GHz, and the length of each sub-band is 200 MHz, the entire frequency band is divided into 200 sub-band intervals.

[0070] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the spectrum sensing is performed on each sub-frequency band to obtain the feature extraction results of each sub-frequency band, specifically:

[0071] Take any sub-band as an example, and denote the signal in the sub-band as s T (t), for the signal s T (t) Perform Fourier transform to obtain the frequency domain signal s T (f), and according to the frequency domain signal s T (f) calculating the energy spectral density E(f) of the sub-band;

[0072] Then the energy spectral density E(f) of this sub-band is:

[0073] E(f)=|s T (f)| 2

[0074] Then take the limit of the energy spectral density E(f) to obtain the power spectral density P(f) of the sub-band:

[0075]

[0076] Similarly, the energy spectral density and power spectral density of each sub-band are obtained respectively;

[0077] The energy spectrum density of each sub-band is normalized, and then the power spectrum density of each sub-band is normalized to obtain the normalized energy spectrum density and normalized power spectrum density of each sub-band respectively. The normalized energy spectrum density and normalized power spectrum density are used as the characteristics of the sub-band.

[0078] The other steps and parameters are the same as those in the first or second embodiment.

[0079] Specific implementation mode 4: This implementation mode is different from any one of the specific implementation modes 1 to 3 in that the specific process of performing spectrum sensing on each sub-frequency band to obtain the importance evaluation result is as follows:

[0080] The received signal energy of each sub-band is normalized, and then the energy detection threshold is compared with the normalized received signal energy value of each sub-band. If the normalized received signal energy value in a sub-band is greater than or equal to the threshold value, the importance of the sub-band is 1, otherwise, the importance of the sub-band is 0.

[0081] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0082] Energy detection is used to determine whether there is a signal in the sub-band, and then the importance of the sub-band is determined. The presence or absence of a signal in a sub-band is assumed to be a binary judgment problem, and the representation method of the frequency band signal is:

[0083]

[0084] Where s(t) is the signal; n(t) is the noise; H0 and H1 represent the noise state hypothesis and the signal and noise superposition state hypothesis respectively; in the observation time T, the relationship between the received signal energy and the threshold value th is used to determine whether there is a signal in the sub-band; after the signal is digitized, the received signal is represented as follows:

[0085]

[0086] Among them, s(i) is the signal; n(i) is the noise; N is the number of sample points; the detection statistic can be expressed as:

[0087]

[0088] Where D is the detection statistic and obeys Gaussian distribution. By using the false alarm probability to detect the threshold condition, the false alarm probability can be expressed as:

[0089]

[0090] Among them, p f is the false alarm probability, σ is the variance, th is the threshold, Q(·) is the Gaussian distribution function, and under the assumption of H1, the detection probability obtained by the normalization method can be expressed as:

[0091]

[0092] in, is the average power of the signal; therefore, the normalized energy can be used to determine whether there is a signal to determine the importance of the sub-band, that is, to obtain the band importance evaluation result of each sub-band.

[0093] Specific implementation method five: This implementation method is different from any one of specific implementation methods one to four in that the initial state of the deep reinforcement learning agent is composed of the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency scanning index.

[0094] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0095] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that the radio frequency front end adopts an exponential nonlinear frequency sweeping method.

[0096] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0097] Specific implementation method seven: Combination Figure 5 This embodiment is different from the first to sixth embodiments in that the training method of the deep reinforcement learning agent is:

[0098] Step 1: Divide the frequency band information in the training set into sub-bands, and the lengths of the sub-bands obtained by division are equal; then extract the characteristics and importance evaluation results of each sub-band respectively;

[0099] The center frequency of the first sub-band, the initial frequency sweep index and the importance of the first sub-band form the initial state s0 of the agent;

[0100] Step 2, initialize t=0;

[0101] Step 3: The agent determines the state s t Select action a t , and obtain the agent's selected action a t The reward received t ;

[0102]

[0103] When no important frequency band is found, the minimum reward r is given min , when an important frequency band is found, the maximum reward r is given max ;

[0104] Calculate action a t The center frequency adjustment value in state s tIf the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 1, then execute step 4; if the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 0, then directly execute step 5;

[0105] Step 4: According to action a t The sweep index adjustment value in state s t The sweep frequency index value in is adjusted, that is, the state s t The sweep frequency index value and action a in t The frequency sweep index adjustment value in the state s is added, and the RF front end t The sub-frequency band corresponding to the center frequency of is taken as the starting sub-frequency band, the sub-frequency band corresponding to the sum of the calculated center frequencies is taken as the target sub-frequency band, and the frequency is swept according to the adjusted sweep index value to reach the target sub-frequency band;

[0106] Status t+1 The center frequency in is the sum of the calculated center frequencies, state s t+1 The sweep frequency index value in is the adjusted sweep frequency index value, and the state s t+1 The importance of is the importance of the sub-band corresponding to the sum of the calculated center frequencies, and the experience playback sample (s t ,a t ,r t ,s t+1 ) into the experience replay pool, and then proceed to step 6;

[0107] Step 5: Set state s t+1 The center frequency of state s t The center frequency, state s t+1 The sweep index value is state s t The sweep frequency index value and action a in t The sum of the sweep index adjustment values ​​in state s t+1 The importance of state s t The importance of; then proceed to step 6;

[0108] Step 6: Calculate the cumulative reward R, and then determine whether the network parameter update frequency has been reached based on t (update the network parameters once every fixed number of steps);

[0109] If not reached, go directly to step 7;

[0110] If it is reached, a fixed number of experience replay samples are taken from the experience replay pool, and the network parameters are updated using the taken experience replay samples, and then step 7 is executed;

[0111] Step 7, let t=t+1, and then determine whether t>T is satisfied;

[0112] If satisfied, the training ends;

[0113] If not satisfied, return to step 3.

[0114] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0115] When the sweep index parameters are different, the relationship curve between the sweep position and the sweep frequency is as follows: Figure 3 As shown; where α takes different values ​​to affect the rate of exponential frequency sweep; when α=1, the exponential frequency sweep becomes a linear frequency sweep, the frequency increases with a stable step length, and the frequency points are evenly distributed; when α=0.5, the frequency sweep interval gradually becomes smaller, and the frequency point distribution gradually becomes denser; when α=1.5, the frequency sweep interval gradually becomes larger, and the frequency point distribution gradually becomes sparser; therefore, when 0<α<1, the step interval decreases, the frequency point distribution gradually becomes denser, and the frequency sweep rate continues to decrease; when α>1, the step interval becomes larger, the frequency point distribution gradually becomes sparser, and the frequency sweep rate increases; therefore, the exponential parameter α can be adjusted dynamically, the step interval can be changed dynamically, and the frequency sweep rate can be controlled. The frequency sweep rate can be increased by changing the frequency sweep interval between two sweeps.

[0116] At time t, the deep reinforcement learning agent obtains state s from the environment t ; Select action a according to the ε-greedy strategy t And feedback to the environment; the environment according to the given a t To give the agent a reward r t and the next time series state s t+1 ; and store it in the experience replay pool; after the network has been trained for a period of time, the experience replay pool will randomly select an experience replay sample to update the main Q network and the target Q network to calculate the Q value; the Q values ​​calculated by the two neural networks are calculated using the loss function, and the main Q network is updated using the gradient descent method; after N steps of training, the network parameters of the main Q network are copied to the target Q network to speed up the convergence of the network; after multiple rounds of training, learning is completed.

[0117] like Figure 4 As shown in the figure, the composition of the deep reinforcement learning model and the process of interaction with the environment are shown. Deep reinforcement learning consists of four parts: the main Q network, the target Q network, the experience replay pool and the loss function, namely:

[0118] The main Q network is composed of a fully connected neural network; it is used to receive the state s of the current time sequence t t , select action a according to the ε-greedy strategy t , and calculate the Q value of the state-action pair; perform Q value update iteration and back propagation process during training;

[0119] The target Q network has the same structure as the main Q network; it is used to calculate the state s of the next time sequence t+1 In the current action a t After a period of training, the parameters in the main Q network are copied to keep the target Q value stable.

[0120] The experience replay pool is used to store experience replay samples; after a period of time, the samples in the experience replay pool are used for network training;

[0121] The loss function uses the gradient descent method to update the network parameters;

[0122] Among them, the Q value is used to measure the value of the current state action pair; the Q value is expressed in the network update process as follows:

[0123]

[0124] Among them, Q t (s t ,a t ) is the Q value corresponding to the state-action pair at time sequence t; α t is the learning rate of the Q network at time t, which is used to control the update probability of the state-action pair; γ is the discount factor; after each iteration, the current state, action, reward and next state are stored as a sample in the experience replay pool; the experience replay sample m t It can be expressed as:

[0125] m t =(s t ,a t ,r t ,s t+1 )

[0126] Among them, s t is the current state; a t For action; t For reward;s t+1 is the state at the next moment; after a period of time, the samples in the experience replay pool are used to train the network; the target Q network uses the gradient descent method to update the network parameters, and the mean square error model is used to characterize the loss function; the loss function can be expressed as:

[0127]

[0128] Where E(·) is the expected function; R is the cumulative reward; γ is the discount factor; is the target Q network parameter; Q(s t ,a t ; ω) is the main Q network parameter; after the loss function is calculated, the gradient descent method is used to update the Q network.

[0129] Specific implementation eight: This implementation differs from any one of the specific implementations one to seven in that the agent t Select action a t The ε-greedy strategy is adopted.

[0130] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0131] The specific ε-greedy strategy is:

[0132]

[0133] Among them, ε is the ratio of exploration and exploitation, ε∈[0,1]; p is the probability between exploration and trade-off; a is an action in the randomly selected action set; is the action with the largest Q value.

[0134] Specific implementation method 9: This implementation method is different from any one of the specific implementation methods 1 to 8 in that the cumulative reward R is:

[0135] R=r+r0+γr1+γ 2 r2+···+γ t r t

[0136] Among them, r is the initial reward value, r t represents the reward obtained in the tth step, γ is the discount factor, γ∈(0,1].

[0137] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0138] The method of the eighth specific implementation method is used to perform multiple rounds of training. After multiple rounds of training, the cumulative rewards obtained in each round of training will be attenuated to a certain extent compared with the cumulative rewards obtained in the initial round.

[0139] Specific implementation method ten: This implementation method is different from any one of specific implementation methods one to nine in that, in step two, the RF front end performs frequency sweeping according to the action outputted at each step by the deep reinforcement learning agent and the importance evaluation result in step one, specifically:

[0140] Step (1), using the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency sweep index to form an initial state, using the initial state as the initial state of the trained deep reinforcement learning agent, and inputting the initial state into the deep reinforcement learning agent;

[0141] Step (2), the agent selects an action in the current state;

[0142] Step (3), calculating the sum L of the center frequency adjustment value in the selected action and the center frequency in the current state, and calculating the sum α of the sweep frequency index in the current state and the sweep frequency index adjustment value in the action;

[0143] If the importance of the sub-frequency band corresponding to the center frequency L is 1, the RF front end uses an exponential nonlinear frequency sweeping method to reach the sub-frequency band corresponding to the center frequency L, and the frequency sweeping index during the frequency sweeping is α; then execute step (4);

[0144] If the importance of the sub-band corresponding to the center frequency L is 0, execute step (5);

[0145] Step (4), update the agent state: replace the center frequency in the updated state with L, replace the sweep frequency index in the updated state with α, and the importance in the updated state with 1;

[0146] Then execute step (6);

[0147] Step (5), update the agent state: replace the sweep frequency index in the updated state with α, and the center frequency and importance in the updated state remain unchanged;

[0148] Then execute step (6);

[0149] Step (6): Use the updated state as the input of the deep reinforcement learning agent, and then return to step (2) until there is no sub-band with an importance of 1 after the sub-band currently scanned.

[0150] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0151] Simulation part

[0152] The simulation parameters are set as follows: frequency band interval 0~40GHz; number of sub-bands 200; sub-band interval 200MHz; key frequency band interval 21.8GHz~22GHz, i.e. the 109th sub-band; initial sweep frequency f0=100MHz; resolution bandwidth B r =10MHz; initial sweep index α=2; initial reward value -100; minimum reward value 1; maximum reward value 100; center frequency adjustment amount Δf sk ={+100,0,-100}MHz; bandwidth adjustment ΔB k ={0}MHz; frequency sweep index adjustment value Δα k={-0.5, 0, +0.5}, where the exponential parameter range is α∈[0,2]; the experience replay pool capacity is 500; the number of training rounds is 500; the number of iterations per round is 2000; the number of neural network layers is 2; the number of neurons in the first layer is 100; the number of neurons in the second layer is 1; the network learning rate is 0.001; the discount factor is 0.8; and the number of tests is 10 times.

[0153] Simulation environment: MATLAB 2018b

[0154] The simulation results are as follows Figure 6 As shown. Figure 6 It can be seen that with the increase of training rounds, the average reward obtained by the agent gradually increases and tends to be relatively stable; the average reward value obtained when it is stable is about 49.2527; in the first rounds, the agent finds the key frequency band less often, so the average reward obtained is less; with the addition of experience replay and the update of the network, the reward that the agent can obtain gradually increases, and after about 250 rounds of learning, the reward basically tends to be stable; the network learns based on the accumulation of experience, and the agent can find the key frequency band faster; therefore, the convergence of the algorithm is also proved.

[0155] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A radio frequency front-end guidance method based on deep reinforcement learning, characterized in that: The method specifically comprises the following steps: Step 1: After the data center divides the frequency band information obtained by the RF front end into sub-bands, it performs spectrum sensing on each sub-band to obtain feature extraction results and importance evaluation results of each sub-band; Step 2: The feature extraction results and importance evaluation results of each sub-frequency band in step 1 are used as the environment of the deep reinforcement learning agent. The state of the deep reinforcement learning agent is composed of the sub-frequency band center frequency, the feature extraction results of the sub-frequency band, and the sweep index. The action of the deep reinforcement learning agent is the center frequency adjustment value and the sweep index adjustment value. The RF front end performs frequency sweep based on the actions output by the deep reinforcement learning agent at each step and the importance evaluation results in step one.

2. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 1, characterized in that: In step 1, the data center divides the frequency band information obtained by the RF front end into sub-bands, specifically: The frequency range of the frequency band information obtained by the RF front end is recorded as 0 to L, where the unit of L is GHz. The frequency band information obtained by the RF front end is then divided into m sub-bands, and the length of each sub-band is T.

3. The RF front-end guidance method based on deep reinforcement learning according to claim 1, characterized in that: The spectrum sensing is performed on each sub-frequency band to obtain the feature extraction result of each sub-frequency band, specifically: Take any sub-band as an example, and denote the signal in the sub-band as s T (t), for the signal s T (t) Perform Fourier transform to obtain the frequency domain signal s T (f), and according to the frequency domain signal s T (f) calculating the energy spectral density E(f) of the sub-band; Then take the limit of the energy spectral density E(f) to obtain the power spectral density P(f) of the sub-band: Similarly, the energy spectral density and power spectral density of each sub-band are obtained respectively; The energy spectrum density of each sub-band is normalized, and then the power spectrum density of each sub-band is normalized to obtain the normalized energy spectrum density and normalized power spectrum density of each sub-band respectively. The normalized energy spectrum density and normalized power spectrum density are used as the characteristics of the sub-band.

4. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 3, characterized in that: The specific process of performing spectrum sensing on each sub-frequency band to obtain the importance evaluation result is as follows: The received signal energy of each sub-band is normalized, and then the energy detection threshold is compared with the normalized received signal energy value of each sub-band. If the normalized received signal energy value in a sub-band is greater than or equal to the threshold value, the importance of the sub-band is 1, otherwise, the importance of the sub-band is 0.

5. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 4, characterized in that: The initial state of the deep reinforcement learning agent consists of the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency sweep index.

6. The RF front-end guidance method based on deep reinforcement learning according to claim 5, characterized in that: The radio frequency front end adopts an exponential nonlinear frequency sweeping method.

7. The RF front-end guidance method based on deep reinforcement learning according to claim 6, characterized in that: The training method of the deep reinforcement learning agent is: Step 1: Divide the frequency band information in the training set into sub-bands, and the lengths of the sub-bands obtained by division are equal; then extract the characteristics and importance evaluation results of each sub-band respectively; The center frequency of the first sub-band, the initial frequency sweep index and the importance of the first sub-band form the initial state s0 of the agent; Step 2, initialize t=0; Step 3: The agent determines the state s t Select action a t , and obtain the agent's selected action a t The reward received t ; Calculate action a t The center frequency adjustment value in state s t If the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 1, then execute step 4; if the importance of the sub-frequency band corresponding to the calculated sum of the center frequencies is 0, then directly execute step 5; Step 4: According to action a t The sweep index adjustment value in state s t The sweep frequency index value in is adjusted, that is, the state s t The frequency sweep index value and action a in t The frequency sweep index adjustment value in the state s is added, and the RF front end t The sub-frequency band corresponding to the center frequency of is taken as the starting sub-frequency band, the sub-frequency band corresponding to the sum of the calculated center frequencies is taken as the target sub-frequency band, and the frequency is swept according to the adjusted sweep index value to reach the target sub-frequency band; Status t+1 The center frequency in is the sum of the calculated center frequencies, state s t+1 The sweep frequency index value in is the adjusted sweep frequency index value, and the state s t+1 The importance of is the importance of the sub-band corresponding to the sum of the calculated center frequencies, and the experience playback sample (s t ,a t ,r t ,s t+1 ) into the experience replay pool, and then proceed to step 6; Step 5: Set state s t+1 The center frequency of state s t The center frequency, state s t+1 The sweep index value is state s t The frequency sweep index value and action a in t The sum of the sweep index adjustment values ​​in state s t+1 The importance of state s t The importance of; then proceed to step 6; Step 6: Calculate the cumulative reward R, and then determine whether the network parameter update frequency has been reached based on t; If not reached, go directly to step 7; If it is reached, a fixed number of experience replay samples are taken from the experience replay pool, and the network parameters are updated using the taken experience replay samples, and then step 7 is executed; Step 7, let t=t+1, and then determine whether t>T is satisfied; If satisfied, the training ends; If not satisfied, return to step 3.

8. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 7, characterized in that: The agent is based on the state s t Select action a t The ε-greedy strategy is adopted.

9. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 8, characterized in that: The cumulative reward R is: R=r+r0+γr1+γ 2 r2+···+γ t r t Among them, r is the initial reward value, r t represents the reward obtained in the tth step, and γ is the discount factor.

10. The method for guiding a radio frequency front end based on deep reinforcement learning according to claim 9, characterized in that: In step 2, the RF front end performs frequency sweeping according to the action output by the deep reinforcement learning agent at each step and the importance evaluation result in step 1, specifically: Step (1), using the center frequency of the first sub-frequency band, the importance of the first sub-frequency band and the initial frequency sweep index to form an initial state, using the initial state as the initial state of the trained deep reinforcement learning agent, and inputting the initial state into the deep reinforcement learning agent; Step (2), the agent selects an action in the current state; Step (3), calculating the sum L of the center frequency adjustment value in the selected action and the center frequency in the current state, and calculating the sum α of the sweep frequency index in the current state and the sweep frequency index adjustment value in the action; If the importance of the sub-frequency band corresponding to the center frequency L is 1, the RF front end uses an exponential nonlinear frequency sweeping method to reach the sub-frequency band corresponding to the center frequency L, and the frequency sweeping index during the frequency sweeping is α; then execute step (4); If the importance of the sub-band corresponding to the center frequency L is 0, execute step (5); Step (4), update the agent state: replace the center frequency in the updated state with L, replace the sweep frequency index in the updated state with α, and the importance in the updated state with 1; Then execute step (6); Step (5), update the agent state: replace the sweep frequency index in the updated state with α, and the center frequency and importance in the updated state remain unchanged; Then execute step (6); Step (6): Use the updated state as the input of the deep reinforcement learning agent, and then return to step (2) until there is no sub-band with an importance of 1 after the sub-band currently scanned.

Images