Wavelet noise reduction direct positioning algorithm based on Q learning

Abstract

The invention provides a wavelet denoising direct positioning algorithm based on Q learning, and the algorithm is characterized in that the algorithm comprises the following steps: S1, selecting a proper denoising threshold value according to an exploration coefficient and a Q value table; s2, performing wavelet noise reduction processing on the received signal by using the selected noise reduction threshold value; s3, performing spatial spectrum estimation by using the noise-reduced signal; s4, combining the estimation result of the spatial spectrum and the action of the observation station to obtain a reward matrix R; and S5, updating the Q value table by using the reward matrix, and obtaining an optimal direct positioning result after learning is completed. According to the method, the Q learning algorithm is used for directly positioning the noisy signal, and the positioning result is used as feedback for learning, so that the method is more reasonable than other noise reduction algorithms using the signal-to-noise ratio as feedback, the intelligence of the positioning algorithm is effectively improved, and the method has higher positioning precision under the condition of low signal-to-noise ratio.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and more specifically, to a direct positioning algorithm based on enhanced wavelet denoising. Background Art

[0002] With the continuous development of communication technology, people are becoming more and more demanding on the accuracy of communication equipment. In recent years, the target direct positioning algorithm has received widespread attention from scholars. Compared with the traditional two-step positioning algorithm, the direct positioning method does not introduce additional errors, effectively improves the positioning accuracy, and reduces the algorithm complexity. However, most direct positioning algorithms are not smart enough, and various parameters need to be set manually during use, and the positioning effect is poor in an environment with a low signal-to-noise ratio.

[0003] In 2008, Oispuu et al. generalized the multiple signal classification algorithm and proposed a direct positioning algorithm based on subspace data fusion. The algorithm calculates the noise subspace and searches for the location of the minimum noise value in the geographic grid as the target location estimate. The algorithm can achieve decoupled estimation of multiple target locations using a single motion array.

[0004] In 2013, Zhong et al. introduced the idea of ​​least mean square filtering and proposed a direct positioning algorithm based on adaptive filtering by using the time difference from the target to different observation stations. This method coordinates the information of different observation stations and uses iterative operations to reduce the computational complexity of the multi-station direct positioning algorithm. However, the positioning accuracy is somewhat lost under low signal-to-noise ratio conditions, and only a single target can be located.

[0005] In 2017, Lu Zhiyu proposed a variable-speed multi-station joint direct positioning algorithm based on segmented signal correlation accumulation. This algorithm uses multiple observation stations with variable speeds to more accurately locate target information and solve the phase ambiguity problem caused by coherent accumulation. It has good robustness under low signal-to-noise ratio conditions. However, multi-station positioning requires the coordination of information from all stations, and its complexity is higher than that of single-station positioning, so it still needs some improvement.

[0006] Wavelet denoising is a signal processing technique that eliminates the effects of noise. Since data has strong correlation in wavelet transform, after the noisy signal is decomposed by wavelet to obtain a series of wavelet coefficients, the energy of the signal is concentrated in some large wavelet coefficients, while the energy of the noise is distributed throughout the entire wavelet domain. Therefore, by retaining large wavelet coefficients for wavelet reconstruction, the signal can be denoised. Due to its simple operation and good denoising effect, wavelet denoising has attracted widespread attention in the fields of image and audio.

[0007] A wavelet denoising direct positioning algorithm based on reinforcement learning is disclosed, which includes the following steps: receiving a signal, determining a filtering threshold according to an exploration coefficient and a Q value table, performing wavelet decomposition on the signal, filtering the signal according to the threshold, reconstructing the signal, using the reconstructed signal to perform spatial spectrum estimation on the target position, obtaining a reward matrix R according to the reconstruction result, and updating the Q value table using the reward matrix R. Summary of the invention

[0008] The present invention provides a wavelet denoising direct positioning algorithm based on Q learning, which can obtain better positioning effect in a low signal-to-noise ratio environment.

[0009] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0010] A wavelet denoising direct positioning algorithm based on Q learning includes the following steps:

[0011] S1: Select an appropriate noise reduction threshold based on the exploration coefficient and Q value table.

[0012] S2: Use the selected noise reduction threshold to perform wavelet noise reduction on the received signal.

[0013] S3: Use the signal of the denoised signal to perform spatial spectrum estimation.

[0014] S4: Combine the estimated results of the spatial spectrum and the actions of the observation station to obtain the reward matrix R.

[0015] S5: Use the reward matrix to update the Q value table, and the optimal direct positioning result can be obtained after learning is completed.

[0016] Preferably, when the observation station receives a noisy signal, it will decide to use a random threshold for filtering according to the exploration coefficient or select a threshold for filtering according to the Q value table, where states in the Q value table represent the threshold, denoted as sta, and action represents three actions, denoted as act, such as increasing the threshold, decreasing the threshold, or keeping the threshold unchanged. The optional range of the threshold is [0, λ max ], so the Q value table has a total of λ max states. The larger the exploration coefficient, the greater the probability that the observatory will adopt a random threshold. The exploration coefficient will decrease with iterations and is set as follows:

[0017] P=β r

[0018] Where β is the exploration factor, β∈[0,1]. r is the number of iterations. If the threshold is selected according to the Q value table, the threshold with the largest Q value will be selected for filtering. The current threshold is denoted as λ.

[0019] Preferably, the noisy signal received by the observation station can be expressed as:

[0020] x(t)=A·s(t)+n(t)

[0021] Where s(t) is the signal emitted by the target, A is the array steering matrix, n(t) is Gaussian additive white noise, and x(t) is the signal received by the observation station.

[0022] The observation station performs wavelet decomposition on the received signal, and the amplitude of the wavelet transform coefficient is calculated as follows:

[0023]

[0024] Where a is the scale factor, b is the translation factor, * represents the complex conjugate, Ψ(t) is the wavelet function, Ψ (a.b) (t) is the wavelet basis function obtained by scaling and translating the wavelet mother function:

[0025]

[0026] The sym4 wavelet is used to transform the received signal. The support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and it also has good regularity, continuity and symmetry. To a certain extent, it can reduce the phase distortion when analyzing and reconstructing the signal.

[0027] Since the signal has a strong correlation, the signal will be concentrated on the large wavelet coefficients. The wavelet coefficients are processed using the threshold obtained in step S1, and the threshold function for noise reduction is:

[0028]

[0029] where w j,k represents the kth wavelet coefficient on the jth scale before denoising, represents the kth wavelet coefficient on the jth scale after denoising. The signal is then reconstructed using the denoised wavelet coefficient to obtain the denoised signal.

[0030] The formula for signal reconstruction is:

[0031]

[0032] in, is the signal after noise reduction.

[0033] Preferably, using the noise reduction signal, we use the subspace data fusion method to directly locate the target. The covariance matrix of the noise reduction signal is subjected to eigenvalue decomposition, and the obtained eigenvector can be divided into two parts, 1 large eigenvalue and M-1 small eigenvalues. M is the number of observation antenna array elements.

[0034] Covariance matrix R l The formula is:

[0035]

[0036] Perform eigenvalue decomposition on the covariance matrix to obtain the signal subspace and the noise subspace The cost function of the subspace data fusion method is:

[0037]

[0038] Preferably, the observation station performs rough wavelet denoising and spatial spectrum estimation on the received signals of the last five times, and compares them with the rough positioning results of the previous time to obtain the reward matrix R. The previous threshold is recorded as The current threshold is denoted as λ. According to the action of the observation station and the positioning comparison, it can be divided into the following 6 cases.

[0039] 1) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0040]

[0041] Where θ is the reward step size, which is a positive value.

[0042] 2) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows:

[0043]

[0044] 3) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows:

[0045]

[0046] 4) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0047]

[0048] 5) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows:

[0049]

[0050] 6) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows:

[0051]

[0052] Preferably, after the observation station obtains the reward matrix R, it updates the Q value table. The Q value table update formula is as follows:

[0053] newQ (sta,act) =(1-α)Q (sta,act) +α(R (sta,act) +γ*maxQ′(sta′,act′))

[0054] Where α is the learning rate, γ is the discount factor, and newQ (sta,act) It is the new Q value based on the state and action, and maxQ′(sta′,act′) is the maximum reward that may be obtained in the expected new state. After the learning is completed, the state with the largest Q value is the optimal threshold at the moment. Using this threshold can obtain the best positioning result.

[0055] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0056] The present invention uses a Q learning algorithm to directly locate noisy signals to improve positioning accuracy. Wavelet denoising is performed on noisy signals to effectively improve positioning accuracy under low signal-to-noise ratio conditions. The Q learning algorithm is used to reasonably select the noise reduction threshold, which improves the intelligence of the positioning algorithm and can be applied in different scenarios. Under the premise of unchanged performance, the algorithm gives priority to small threshold filtering. On the one hand, it can prevent signal distortion caused by excessive filtering, and on the other hand, it keeps the phase amplitude of the signal unchanged as much as possible, ensuring that the waveform after filtering conforms to the trend of the original signal, making the result more credible. Using the positioning result as feedback for learning is more reasonable than other algorithms using the signal-to-noise ratio as feedback, and has higher positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0058] Figure 2 This is a comparison chart of RMSE before and after noise reduction at different signal-to-noise ratios in the embodiment.

[0059] Figure 3 1 is a comparison diagram of the signal-to-noise ratio of the received signal before and after noise reduction at different signal-to-noise ratios in the embodiment.

[0060] Figure 4 It is the improvement ratio of positioning accuracy under different signal-to-noise ratios in the embodiment. DETAILED DESCRIPTION

[0061] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0062] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0063] This embodiment provides a digital communication signal modulation recognition method based on pre-processing noise reduction, such as Figure 1 As shown, the following steps are included:

[0064] S1: Select an appropriate noise reduction threshold based on the exploration coefficient and Q value table.

[0065] S2: Use the selected noise reduction threshold to perform wavelet noise reduction on the received signal.

[0066] S3: Use the signal of the denoised signal to perform spatial spectrum estimation.

[0067] S4: Combine the estimated results of the spatial spectrum and the actions of the observation station to obtain the reward matrix R.

[0068] S5: Use the reward matrix to update the Q value table, and the optimal direct positioning result can be obtained after learning is completed.

[0069] In this embodiment, when the observation station receives the noisy signal in step S1, it will decide to use a random threshold for filtering according to the exploration coefficient or select a threshold for filtering according to the Q value table, where states in the Q value table represents the threshold, denoted as sta, and action represents three actions, namely, increasing the threshold, decreasing the threshold, or keeping the threshold unchanged, denoted as act. The optional range of the threshold is [0, λ max ], let λ max =100, so the threshold value can be selected in the range of [0, 100], and the Q value table has 100 states. The larger the exploration coefficient, the greater the probability that the observation station will use a random threshold. The exploration coefficient will decrease with iterations and is set as follows:

[0070] P=β r

[0071] Where β is the exploration factor, β∈[0,1]. r is the number of iterations. Let β=0.8, r=5, then the exploration coefficient is P=0.8 5 =0.33, indicating that the observation station has a probability of 0.33 to select a random threshold, and a probability of (1-0.33) to select a threshold according to the Q value table. If the threshold is selected according to the Q value table, the threshold with the largest Q value will be selected for filtering. Assume that in this embodiment, the observation station selects a random threshold strategy, the random selection is 85, and the current threshold is recorded as λ, then λ=85.

[0072] In this embodiment, the noisy signal received by the observation station in step S2 can be expressed as:

[0073] x(t)=A·s(t)+n(t)

[0074] Where s(t) is the signal emitted by the target, A is the array steering matrix, n(t) is Gaussian additive white noise, x(t) is the signal received by the observation station. In this embodiment, the signal-to-noise ratio of the noisy signal is 0 dB.

[0075] The observation station performs wavelet decomposition on the received signal, and the amplitude of the wavelet transform coefficient is calculated as follows:

[0076]

[0077] Where a is the scale factor, b is the translation factor, * represents the complex conjugate, Ψ(t) is the wavelet function, Ψ (a.b) (t) is the wavelet basis function obtained by scaling and translating the wavelet mother function:

[0078]

[0079] The sym4 wavelet is used to transform the received signal. The support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and it also has good regularity, continuity and symmetry. To a certain extent, it can reduce the phase distortion when analyzing and reconstructing the signal.

[0080] The discrete form of Sym4 wavelet is:

[0081]

[0082] Since the signal has a strong correlation, the signal will be concentrated on the large wavelet coefficients. The wavelet coefficients are processed using the threshold obtained in step S1, and the threshold function for noise reduction is:

[0083]

[0084] where w j,k represents the kth wavelet coefficient on the jth scale before denoising, represents the kth wavelet coefficient on the jth scale after denoising. The signal is then reconstructed using the denoised wavelet coefficient to obtain the denoised signal.

[0085] The formula for signal reconstruction is:

[0086]

[0087] in, is the signal after noise reduction.

[0088] In this embodiment, the observation station described in step S3 uses the noise reduction signal, and we use the subspace data fusion method to directly locate the target. Let the number of observation antenna array elements M = 16, perform eigenvalue decomposition on the covariance matrix of the noise reduction signal, and the obtained eigenvector can be divided into two parts, 1 large eigenvalue and 15 small eigenvalues.

[0089] Covariance matrix R l The formula is:

[0090]

[0091] Perform eigenvalue decomposition on the covariance matrix to obtain the signal subspace and the noise subspace The cost function of the subspace data fusion method is:

[0092]

[0093] In this embodiment, the observation station described in step S4 performs rough wavelet denoising and spatial spectrum estimation on the received signals for the last five times, and compares them with the rough positioning results of the previous time to obtain the reward matrix R. The previous threshold is recorded as The current threshold is denoted as λ.

[0094] According to the movement of the observation station and the positioning comparison, it can be divided into the following 6 situations.

[0095] 1) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0096]

[0097] Where θ is the reward step size, which is a positive value.

[0098] 2) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows:

[0099]

[0100] 3) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows:

[0101]

[0102] 4) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0103]

[0104] 5) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows:

[0105]

[0106] 6) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows:

[0107]

[0108] Set the previous threshold The current threshold λ=85, and the reward step length θ=50. This means that the observation station takes the action of increasing the threshold. If the rough positioning effect is better than before, the reward matrix R is set as follows:

[0109]

[0110] In this embodiment, when the observation station obtains the reward matrix R in step S5, the Q value table is updated. The Q value table update formula is as follows:

[0111] newQ (sta,act) =(1-α)Q (sta,act) +α(R (sta,act) +γ*maxQ′(sta′,act′))

[0112] Where α is the learning rate, γ is the discount factor, and newQ (sta,act) is the new Q value based on the state and action. maxQ′(ta′,act′) is the maximum reward that may be obtained in the expected new state. Let the learning rate α=0.3 and the discount factor γ=0. Then the update formula of the Q value table is as follows:

[0113]

[0114] After learning, the state with the maximum Q value is the current optimal threshold, which can obtain the best positioning result.

[0115] Figure 2 The RMSE comparison diagram before and after noise reduction under different signal-to-noise ratios in the embodiment is shown in FIG. It can be seen that the RMSE of the present invention is lower than that of the positioning algorithm before noise reduction under different signal-to-noise ratios. The effect improvement is more obvious under low signal-to-noise ratio conditions.

[0116] Figure 31 is a comparison diagram of the received signal noise ratio before and after noise reduction under different signal noise ratios in the embodiment. It can be seen that the present invention can improve the signal noise ratio of the received signal under different noise intensity conditions.

[0117] Figure 4 It is the improvement ratio of positioning accuracy under different signal-to-noise ratios in the embodiment. It can be seen that the improvement ratio of the algorithm is higher under low signal-to-noise ratio conditions, and can even reach 10%.

[0118] The present invention uses the Q learning algorithm to directly locate the noisy signal to improve the positioning accuracy. Using the positioning result as feedback for learning is more reasonable than other algorithms using the signal-to-noise ratio as feedback, and has higher positioning accuracy.

[0119] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0120] Obviously, the above embodiments of the present invention are examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A wavelet denoising direct positioning algorithm based on Q learning, characterized in that: The following steps are involved: S1: Select an appropriate noise reduction threshold based on the exploration coefficient and Q value table. S2: Use the selected noise reduction threshold to perform wavelet noise reduction on the received signal. S3: Use the signal of the denoised signal to perform spatial spectrum estimation. S4: Combine the estimated results of the spatial spectrum and the actions of the observation station to obtain the reward matrix R. S5: Use the reward matrix to update the Q value table, and the optimal direct positioning result can be obtained after learning is completed.

2. The wavelet denoising direct positioning algorithm based on Q learning according to claim 1 is characterized in that: In step S1, when the observation station receives the noisy signal, it will decide to use a random threshold for filtering according to the exploration coefficient or select a threshold for filtering according to the Q value table, where states in the Q value table represent the threshold, denoted as sta, and action represents three actions, denoted as act, such as increasing the threshold, decreasing the threshold, or keeping the threshold unchanged. The optional range of the threshold is [0, λ max ], so the Q value table has a total of λ max states. The larger the exploration coefficient, the greater the probability that the observatory will adopt a random threshold. The exploration coefficient will decrease with iterations and is set as follows: P＝β r Where β is the exploration factor, β∈[0,1]. r is the number of iterations. If the threshold is selected according to the Q value table, the threshold with the largest Q value will be selected for filtering. The current threshold is denoted as λ.

3. The wavelet denoising direct positioning algorithm based on Q learning according to claim 2 is characterized in that: The noisy signal received by the observation station in step S2 can be expressed as: x(t)=A·s(t)+n(t) Where s(t) is the signal emitted by the target, A is the array steering matrix, n(t) is Gaussian additive white noise, and x(t) is the signal received by the observation station. The observation station performs wavelet decomposition on the received signal, and the amplitude of the wavelet transform coefficient is calculated as follows: Where a is the scale factor, b is the translation factor, * represents the complex conjugate, Ψ(t) is the wavelet function, Ψ (a.b) (t) is the wavelet basis function obtained by scaling and translating the wavelet mother function: The sym4 wavelet is used to transform the received signal. The support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and it also has good regularity, continuity and symmetry. To a certain extent, it can reduce the phase distortion when analyzing and reconstructing the signal. Since the signal has a strong correlation, the signal will be concentrated on the large wavelet coefficients. The wavelet coefficients are processed using the threshold obtained in step S1, and the threshold function for noise reduction is: where w j，k represents the kth wavelet coefficient on the jth scale before denoising, represents the kth wavelet coefficient on the jth scale after denoising. The signal is then reconstructed using the denoised wavelet coefficient to obtain the denoised signal. The formula for signal reconstruction is: in, is the signal after noise reduction.

4. The wavelet denoising direct positioning algorithm based on Q learning according to claim 3 is characterized in that: In step S3, we use the subspace data fusion method to directly locate the target using the noise reduction signal. The covariance matrix of the noise reduction signal is subjected to eigenvalue decomposition, and the obtained eigenvector can be divided into two parts, 1 large eigenvalue and M-1 small eigenvalues. M is the number of observation antenna array elements. Covariance matrix R l The formula is: Perform eigenvalue decomposition on the covariance matrix to obtain the signal subspace and the noise subspace The cost function of the subspace data fusion method is:

5. The wavelet denoising direct positioning algorithm based on Q learning according to claim 4 is characterized in that: In step S4, the observation station performs rough wavelet denoising and spatial spectrum estimation on the received signals of the last five times, and compares them with the rough positioning results of the previous time to obtain the reward matrix R. The previous threshold is recorded as The current threshold is recorded as λ. According to the action of the observation station and the positioning comparison, it can be divided into the following 6 cases. 1) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows: Where θ is the reward step size, which is a positive value. 2) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows: 3) If It means that the observation station takes the action of increasing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows: 4) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is better than the previous one, the reward matrix R is set as follows: 5) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect is worse than before, the reward matrix R is set as follows: 6) If It means that the observation station takes the action of reducing the threshold. If the rough positioning effect changes little from the previous one, the reward matrix R is set as follows:

6. The Q-learning-based wavelet denoising direct positioning algorithm according to claim 5, characterized in that: In step S5, when the observation station obtains the reward matrix R, it updates the Q value table. The Q value table update formula is as follows: newQ (sta，act) =(1-α)Q (sta，act) +α(R (sta，act) +γ*maxQ′(sta′,act′)) where α is the learning rate, γ is the discount factor, newQ (sta，act) is the new Q value based on the state and action. maxQ′(sta′, act′) is the maximum reward that may be obtained in the expected new state. After learning, the state with the maximum Q value is the optimal threshold at the moment. Using this threshold can obtain the best positioning result.

Images