Wavelet denoising direct positioning algorithm based on q-learning

Abstract

The application provides a wavelet denoising direct positioning algorithm based on Q learning, and is characterized by comprising the following steps: S1, selecting a suitable denoising threshold according to an exploration coefficient and a Q value table; S2, using the selected denoising threshold to perform wavelet denoising processing on a received signal; S3, using the denoised signal to perform spatial spectrum estimation; S4, obtaining a reward matrix R by combining the estimation result of the spatial spectrum and the action of an observation station; and S5, updating the Q value table by using the reward matrix, so that an optimal direct positioning result can be obtained after learning is completed. The application uses the Q learning algorithm to perform direct positioning on a noisy signal, uses a positioning result as feedback for learning, is more reasonable than other denoising algorithms which use a signal-to-noise ratio as feedback, effectively improves the intelligence of the positioning algorithm, and has higher positioning precision under a low signal-to-noise ratio condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, more particularly, to a direct positioning algorithm based on reinforcement and wavelet denoising. BACKGROUND

[0002] With the continuous development of communication technology, people's precision of communication equipment is also increasingly demanding. In recent years, target direct positioning algorithm has been widely concerned by scholars. Compared with the traditional two-step positioning algorithm, direct positioning method can effectively improve the positioning accuracy and reduce the algorithm complexity without introducing additional errors. However, most of the direct positioning algorithms are not intelligent enough, and the parameters need to be set manually, and the positioning effect is not good in low signal-to-noise ratio environment.

[0003] In 2008, Oispuu et al. extended the multiple signal classification algorithm and proposed a direct positioning algorithm based on subspace data fusion. The algorithm calculates the noise subspace and searches for the position of the minimum noise value in the geographical grid as the position estimate of the target. This algorithm can realize decoupled estimation of multiple target positions using a single motion array.

[0004] In 2013, Zhong et al. introduced the least mean square filtering idea and proposed a direct positioning algorithm based on adaptive filtering using the time difference of the target to different observation stations. This method can reduce the computational complexity of multi-station direct positioning algorithm by integrating the information of different observation stations and using iterative operation. However, the positioning accuracy is lost under low signal-to-noise ratio conditions, and only a single target can be positioned.

[0005] In 2017, Lu Zhiyu proposed a variable-speed multi-station joint direct positioning algorithm based on segmented signal correlation accumulation. This algorithm can more accurately locate the target information using multiple variable-speed observation stations, and can solve the phase ambiguity problem caused by coherent accumulation, and has good robustness under low signal-to-noise ratio conditions. However, multi-station positioning needs to integrate the information of all stations, and the complexity is higher than that of single-station positioning, which still needs to be improved.

[0006] Wavelet denoising is a signal processing technology to eliminate the influence of noise. Since the data in wavelet transform has strong correlation, after wavelet decomposition of the noisy signal, the energy of the signal is concentrated in some large wavelet coefficients, while the energy of the noise is distributed throughout the wavelet domain. Therefore, by retaining large wavelet coefficients for wavelet reconstruction, the signal can be denoised. Due to the simple operation and good denoising effect of wavelet denoising, it has attracted widespread attention in the fields of image, audio, etc.

[0007] Disclosed is a wavelet denoising direct positioning algorithm based on reinforcement learning. The algorithm comprises the following steps: receiving a signal, determining a filtering threshold according to an exploration coefficient and a Q value table, performing wavelet decomposition on the signal, filtering the signal according to the threshold, reconstructing the signal, performing spatial spectrum estimation on the target position using the reconstructed signal, obtaining a reward matrix R according to the reconstructed result, and updating the Q value table using the reward matrix R. SUMMARY

[0008] The application provides a wavelet denoising direct positioning algorithm based on Q learning, which can achieve better positioning effect in a low signal-to-noise ratio environment.

[0009] To solve the above technical problems, the technical solution of the application is as follows:

[0010] A wavelet denoising direct positioning algorithm based on Q learning comprises the following steps:

[0011] S1: selecting a suitable denoising threshold according to an exploration coefficient and a Q value table.

[0012] S2: performing wavelet denoising processing on the received signal using the selected denoising threshold.

[0013] S3: performing spatial spectrum estimation using the denoised signal.

[0014] S4: obtaining a reward matrix R by combining the estimation result of the spatial spectrum and the action of the observation station.

[0015] S5: updating the Q value table using the reward matrix, so that the optimal direct positioning result can be obtained after learning.

[0016] Preferably, when the observation station receives a noisy signal, it will determine whether to use a random threshold for filtering according to an exploration coefficient or select a threshold for filtering according to a Q value table, wherein the Q value table states represents the threshold, denoted as sta, and the action represents three actions of increasing the threshold, decreasing the threshold or keeping the threshold unchanged, denoted as act. The selectable range of the threshold is [0, λ max ], so the Q value table has λ max states. The greater the exploration coefficient, the greater the probability that the observation station uses a random threshold. The exploration coefficient decreases with iterations and is set as follows:

[0017] P=β r

[0018] wherein β is an exploration factor, β∈[0, 1]. r is the number of iterations. If a threshold is selected according to the Q value table, the threshold with the maximum Q value is selected for filtering. The current threshold is denoted as λ.

[0019] Preferably, the noisy signal received by the observation station can be expressed as:

[0020] x(t) = A-s(t) + n(t)

[0021] Where s(t) is the signal emitted by the target, A is the array steering matrix, n(t) is the Gaussian additive white noise. x(t) is the signal received by the observation station.

[0022] The observation station carries out wavelet decomposition on the received signal, and the calculation method of the wavelet transform coefficient amplitude is as follows:

[0023]

[0024] In the formula, a is a scale factor, b is a translation factor, * represents a complex conjugate, Ψ(t) is a wavelet function, and Ψ (a.b) (t) is a wavelet basis function obtained by scaling and translating the mother wavelet function:

[0025]

[0026] The sym4 wavelet is selected for wavelet transformation of the received signal. The support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and the sym4 wavelet also has good regularity, continuity and symmetry. To some extent, it can reduce the phase distortion when analyzing and reconstructing the signal.

[0027] Because the signal has strong correlation, the signal will concentrate on large wavelet coefficients. The threshold value obtained in step S1 is used to process the wavelet coefficients, and the threshold function for noise reduction is:

[0028]

[0029] Where w j，k represents the jth scale kth wavelet coefficient before noise reduction, represents the jth scale kth wavelet coefficient after noise reduction. Then the wavelet coefficients after noise reduction are used to reconstruct the signal to obtain the noise reduction signal.

[0030] The formula for signal reconstruction is:

[0031]

[0032] Where, is the signal after noise reduction.

[0033] Preferably, using the noise reduction signal, we use the subspace data fusion method to directly locate the target. The eigenvalue decomposition of the covariance matrix of the noise reduction signal is carried out, and the obtained eigenvalue can be divided into two parts, one large eigenvalue and M-1 small eigenvalues. M is the number of observation antenna elements.

[0034] The formula of the covariance matrix R l is:

[0035]

[0036] Eigenvalue decomposition of the covariance matrix to obtain the signal subspace and noise subspace The cost function of the subspace data fusion method is:

[0037]

[0038] Preferably, the observation station performs rough wavelet denoising and spatial spectrum estimation on the received signal for the last 5 times, and compares with the rough positioning result of the previous time to obtain the reward matrix R. The threshold of the previous time is denoted as The current threshold is denoted as λ. According to the action of the observation station and the positioning comparison, it can be divided into the following 6 cases.

[0039] 1) If , it means that the observation station takes the action of increasing the threshold, and if the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0040]

[0041] Where θ is the reward step, which is a positive value.

[0042] 2) If , it means that the observation station takes the action of increasing the threshold, and if the rough positioning effect is worse than the previous one, the reward matrix R is set as follows:

[0043]

[0044] 3) If , it means that the observation station takes the action of increasing the threshold, and if the rough positioning effect changes little compared with the previous one, the reward matrix R is set as follows:

[0045]

[0046] 4) If , it means that the observation station takes the action of reducing the threshold, and if the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0047]

[0048] 5) If , it means that the observation station takes the action of reducing the threshold, and if the rough positioning effect is worse than the previous one, the reward matrix R is set as follows:

[0049]

[0050] 6) If If the action of the observation station is to reduce the threshold value, and the change of the rough positioning effect is small compared with the previous one, the reward matrix R is set as follows:

[0051]

[0052] Preferably, the Q value table is updated after the observation station obtains the reward matrix R. The Q value table updating formula is as follows:

[0053] newQ (sta，act) =(1-α)Q (sta，act) +α(R (sta，act) +γ*maxQ′(sta′,act′))

[0054] Wherein, a is the learning rate, γ is the discount factor, newQ (sta，act) is the new Q value based on the state and action, and maxQ′(sta′,act′) is the maximum reward that can be obtained in the expected new state. After learning is completed, the state with the maximum Q value is the optimal threshold value at present. The threshold value can obtain the best positioning result.

[0055] Compared with the prior art, the technical scheme of the present application has the beneficial effects that:

[0056] The present application uses the Q learning algorithm to directly position the noisy signal, so as to improve the positioning accuracy. The wavelet denoising of the noisy signal can effectively improve the positioning accuracy under the condition of low signal-to-noise ratio. The Q learning algorithm is used to reasonably select the denoising threshold value, so as to improve the intelligence of the positioning algorithm and can be applied in different scenes. On the premise that the performance is unchanged, the algorithm preferentially selects small threshold value filtering, which can prevent signal distortion caused by excessive filtering on the one hand, and can keep the phase and amplitude of the signal unchanged as much as possible on the other hand, so as to ensure that the waveform after filtering conforms to the original signal trend, and make the result more reliable. Using the positioning result as feedback for learning is more reasonable than using the signal-to-noise ratio as feedback in other algorithms, and has higher positioning accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 It is a method flowchart of the present application.

[0058] Figure 2 It is an RMSE comparison chart before and after denoising under different signal-to-noise ratios in the embodiment.

[0059] Figure 3 It is a comparison chart of the received signal-to-noise ratio before and after denoising under different signal-to-noise ratios in the embodiment.

[0060] Figure 4 It is the positioning accuracy improvement ratio under different signal-to-noise ratios in the embodiment. DETAILED DESCRIPTION

[0061] The accompanying drawings are only used for illustrative description and cannot be understood as a limitation of the patent;

[0062] The technical solutions of the present application will be further described below in combination with the drawings and examples.

[0063] The present embodiment provides a digital communication signal modulation recognition method based on pre-processing noise reduction, as shown in the following steps: Figure 1

[0064] S1: Select a suitable noise reduction threshold according to the exploration coefficient and the Q value table.

[0065] S2: Use the selected noise reduction threshold to perform wavelet noise reduction processing on the received signal.

[0066] S3: Perform spatial spectrum estimation using the noise-reduced signal.

[0067] S4: Obtain the reward matrix R by combining the estimation results of the spatial spectrum and the actions of the observation station.

[0068] S5: Update the Q value table using the reward matrix, and after learning, the optimal direct positioning result can be obtained.

[0069] In the present embodiment, after the observation station receives the noisy signal, step S1 determines to use a random threshold for filtering according to the exploration coefficient or selects a threshold for filtering according to the Q value table, wherein the Q value table states represents the threshold, denoted as sta, and the action represents three actions of increasing the threshold, decreasing the threshold or keeping the threshold unchanged, denoted as act. The selectable range of the threshold is [0, λ max ], and let λ max = 100, so the selectable range of the threshold is [0, 100], and the Q value table has 100 states. The larger the exploration coefficient, the greater the probability of the observation station using a random threshold. The exploration coefficient will decrease with iteration, and is set as follows:

[0070] P = β r

[0071] Where β is the exploration factor, β ∈ [0, 1]. r is the number of iterations. Let β = 0.8 and r = 5, then the exploration coefficient P = 0.8 5 = 0.33, which means that the observation station has a probability of 0.33 of selecting a random threshold, and a probability of (1-0.33) of selecting a threshold according to the Q value table. If a threshold is selected according to the Q value table, the threshold with the maximum Q value will be selected for filtering. Assuming that in the present embodiment, the observation station selects a random threshold strategy, the random selection is 85, and the current threshold is denoted as λ, then λ = 85.

[0072] In the present embodiment, the noisy signal received by the observation station in step S2 can be represented as:​

[0073] x(t) = A-s(t) + n(t)

[0074] Where s(t) is the signal emitted by the target, A is the array steering matrix, n(t) is the Gaussian additive white noise. x(t) is the signal received by the observation station. In this embodiment, the signal-to-noise ratio of the noisy signal is 0 dB.

[0075] The observation station performs wavelet decomposition on the received signal, and the calculation method of the wavelet transform coefficient amplitude is as follows:

[0076]

[0077] In the formula, a is a scale factor, b is a translation factor, * represents a complex conjugate, Ψ(t) is a wavelet function, and Ψ (a.b) (t) is a wavelet basis function obtained by scaling and translating the mother wavelet function:

[0078]

[0079] The sym4 wavelet is selected for wavelet transformation of the received signal. The support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and the sym4 wavelet also has good regularity, continuity and symmetry. To some extent, it can reduce the phase distortion when analyzing and reconstructing the signal.

[0080] The discrete form of the sym4 wavelet is:

[0081]

[0082] Because the signal has strong correlation, the signal will be concentrated on the large wavelet coefficients. The threshold value obtained in step S1 is used to process the wavelet coefficients, and the threshold function for noise reduction is:

[0083]

[0084] Where w j，k represents the jth scale kth wavelet coefficient before noise reduction, represents the jth scale kth wavelet coefficient after noise reduction. Then, the wavelet coefficients after noise reduction are used to reconstruct the signal to obtain the noise-reduced signal.

[0085] The formula for signal reconstruction is:

[0086]

[0087] Where, is the signal after noise reduction.

[0088] In this embodiment, the observation station in step S3 uses the noise reduction signal, and we use the subspace data fusion method to directly locate the target. Let the number of observation antenna elements M = 16, and perform eigenvalue decomposition on the covariance matrix of the noise reduction signal to obtain the eigenvectors, which can be divided into two parts, one large eigenvalue and 15 small eigenvalues.

[0089] Covariance matrix R l The formula is:

[0090]

[0091] Perform eigenvalue decomposition on the covariance matrix to obtain the signal subspace and the noise subspace The cost function of the subspace data fusion method is:

[0092]

[0093] In this embodiment, the observation station in step S4 performs rough wavelet noise reduction and spatial spectrum estimation on the received signal for nearly 5 times, and compares the rough positioning result with the previous rough positioning result to obtain the reward matrix R. The threshold of the previous time is denoted as The current threshold is denoted as λ.

[0094] According to the action of the observation station and the positioning comparison, it can be divided into the following six cases.

[0095] 1) If It indicates that the observation station takes the action of increasing the threshold, and if the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0096]

[0097] Where θ is the reward step, which is a positive value.

[0098] 2) If It indicates that the observation station takes the action of increasing the threshold, and if the rough positioning effect is worse than the previous one, the reward matrix R is set as follows:

[0099]

[0100] 3) If It indicates that the observation station takes the action of increasing the threshold, and if the rough positioning effect changes little compared with the previous one, the reward matrix R is set as follows:

[0101]

[0102] 4) If It indicates that the observation station takes the action of reducing the threshold, and if the rough positioning effect is better than the previous one, the reward matrix R is set as follows:

[0103]

[0104] 5) If , it means that the observation station takes the action of reducing the threshold value, and if the rough positioning effect is worse than before, the reward matrix R is set as follows:

[0105]

[0106] 6) If , it means that the observation station takes the action of reducing the threshold value, and if the rough positioning effect changes little compared with before, the reward matrix R is set as follows:

[0107]

[0108] Let the threshold value of the previous time be The current threshold value λ = 85, and the reward step θ = 50. It means that the observation station takes the action of increasing the threshold value, and if the rough positioning effect is better than before, the reward matrix R is set as follows:

[0109]

[0110] In the embodiment, after the observation station obtains the reward matrix R in step S5, the Q value table is updated. The Q value table updating formula is as follows:

[0111] newQ (sta，act) = (1-α)Q (sta，act) + α (R (sta，act) + γ * maxQ' (sta', act'))

[0112] Wherein, α is the learning rate, γ is the discount factor, newQ (sta，act) is the new Q value based on the state and action. maxQ' (sta', act') is the maximum reward that can be obtained in the expected new state. Let the learning rate α = 0.3 and the discount factor γ = 0. Then the Q value table updating formula is as follows:

[0113]

[0114] After learning is completed, the state with the maximum Q value is the optimal threshold value at present. The threshold value can obtain the best positioning result.

[0115] Figure 2 The figure is a comparison of RMSEs before and after noise reduction under different signal-to-noise ratios in the embodiment. It can be seen that the RMSE of the present application is lower than that of the positioning algorithm before noise reduction under different signal-to-noise ratios. The effect is improved more obviously under the condition of low signal-to-noise ratio.

[0116] Figure 3The signal-to-noise ratio of the received signal before and after noise reduction under different signal-to-noise ratios in the embodiment is compared in the figure. It can be seen that the signal-to-noise ratio of the received signal can be improved under different noise intensities.

[0117] Figure 4 The positioning accuracy improvement ratio under different signal-to-noise ratios in the embodiment is shown in the figure. It can be seen that the improvement ratio is higher under low signal-to-noise ratio conditions, and can even reach 10%.

[0118] The Q-learning algorithm is used to directly position the noisy signal in the embodiment, so as to improve the positioning accuracy. The positioning result is used as feedback for learning, which is more reasonable than using the signal-to-noise ratio as feedback in other algorithms, and has higher positioning accuracy.

[0119] The accompanying drawings are only used for illustrative description, and cannot be understood as a limitation on the patent;

[0120] Obviously, the above embodiments of the present application are examples for clearly illustrating the present application, and are not a limitation on the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments cannot be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A Q-learning based wavelet denoising direct positioning algorithm, characterized in that, The method comprises the following steps: S1: selecting a suitable noise reduction threshold according to an exploration coefficient and a Q value table; S2: performing wavelet noise reduction processing on the received signal using the selected noise reduction threshold; S3: performing spatial spectrum estimation using the signal after noise reduction; S4: obtaining a reward matrix R by combining the estimation result of the spatial spectrum and the action of the observation station; S5: updating the Q value table using the reward matrix, and obtaining the optimal direct positioning result after learning is completed; After the observation station receives the noisy signal in step S1, it will determine whether to use a random threshold for filtering according to the exploration coefficient or select a threshold for filtering according to the Q-value table, where the Q-value table states represents the threshold, denoted as sta, and action represents three actions of increasing the threshold, decreasing the threshold, or keeping the threshold unchanged, denoted as act; the selectable range of the threshold is [0, λ max ], so there are λ max states in the Q-value table; the greater the exploration coefficient P, the greater the probability that the observation station will use a random threshold; where the exploration coefficient P will continuously decrease with iteration, and is set as follows: P＝β r Wherein β is an exploration factor, β ∈ [0, 1]; r is the number of iterations; If the threshold is selected according to the Q value table, the threshold with the maximum Q value will be selected for filtering; the current threshold is denoted as λ; The noisy signal received by the observation station is denoted as: x(t) = A·s(t) + n(t) Where s(t) is the signal transmitted by the target, A is the array steering matrix, n(t) is the Gaussian additive white noise; x(t) is the signal received by the observation station; The received signal is decomposed by the observation station, and the calculation method of the wavelet transform coefficient amplitude WT(a, b) is as follows: where a is a scaling factor, b is a translation factor, * denotes complex conjugate, Ψ(t) is a wavelet function, Ψ (a，b) (t) is a wavelet basis function obtained by scaling and translating the mother wavelet function The sym4 wavelet is selected for wavelet transformation of the received signal; the support range of the sym4 wavelet is 2*4-1, the vanishing moment is 4, and it also has good regularity, continuity and symmetry; to a certain extent, it can reduce the phase distortion when analyzing and reconstructing the signal; Because the signal has strong correlation, the signal will be concentrated on the large wavelet coefficients; the threshold obtained in step S1 is used to process the wavelet coefficients, and the threshold function for noise reduction is: where w j，k denotes the kth wavelet coefficient on the jth scale before denoising, denotes the kth wavelet coefficient on the jth scale after denoising; the current threshold is denoted as λ; the wavelet coefficients after denoising are then used to reconstruct the signal to obtain a denoised signal; The formula for signal reconstruction is: wherein, is the signal after noise reduction.

2. The Q-learning based wavelet denoising direct positioning algorithm according to claim 1, characterized in that, In step S3, the subspace data fusion method is used for direct positioning of the target; the covariance matrix of the noise reduction signal is subjected to eigenvalue decomposition, and the obtained characteristic vector is divided into two parts, one large eigenvalue and M-1 small eigenvalues; M is the number of observation antenna elements; Covariance matrix R L The formula is: Eigenvalue decomposition of the covariance matrix yields the signal subspace and the noise subspace The cost function for the subspace data fusion approach is:

3. The Q-learning based wavelet denoising direct positioning algorithm according to claim 2, characterized in that, In step S4, the observation station performs rough wavelet noise reduction and spatial spectrum estimation on the received signal in the last 5 times, and compares the rough positioning result with the previous rough positioning result to obtain the reward matrix R; The threshold of the previous time is denoted as The current threshold is denoted as λ; according to the action of the observation station and the positioning contrast, it is divided into the following 6 cases: 1) If R = 1, if the rough positioning is better than the previous one, otherwise R = 0. Wherein θ is a reward step, which is a positive value; 2) If R = -1 if the rough positioning is worse than the previous one, and the reward matrix R is set as follows: 3) If indicates that the observation station takes the action of increasing the threshold, and if the change in the coarse positioning effect is small compared to the previous one, the reward matrix R is set as follows: 4) If R = 1, if the rough positioning is better than the previous one, otherwise R = 0. 5) If R = -1 if the station takes the action of reducing the threshold and the coarse positioning is worse than before. 6) If R = 0 if the station takes the action of reducing the threshold and the effect of the coarse positioning is very small compared to the previous one.

4. The Q-learning based wavelet denoising direct positioning algorithm according to claim 3, characterized in that, In step S5, the Q value table is updated after the observation station obtains the reward matrix R; the Q value table updating formula is as follows: newQ (sta，act) = (1 - a) * Q (sta，act) + a * (R (sta，act) + g * maxQ'(sta', act')) where a is the learning rate, g is the discount factor, newQ (sta，act) is the new Q value based on state and action; maxQ'(sta', act') is the maximum reward that can be obtained in the expected new state, and the state with the maximum Q value after learning is the threshold value of the optimal threshold value at present; the best positioning result can be obtained by using the threshold value.

Images