Underwater acoustic channel multipath sparseness estimation method based on deep reinforcement learning

Through the multipath sparsity estimation method of water acoustic channel based on deep reinforcement learning, the sparsity of water acoustic channels is adaptively adjusted, solving the problem that the impact of sparsity in the prior art is not effectively considered, and improving the accuracy and equalization performance of channel estimation.

CN120223468APending Publication Date: 2025-06-27SOUTH CHINA UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510325602.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-19
Publication Date
2025-06-27

AI Technical Summary

Technical Problem

The existing water acoustic channel estimation method based on orthogonal matching tracking algorithm fails to effectively consider the impact of sparseness on estimation performance, especially when the time-variability of the water acoustic channel is obvious, resulting in low channel estimation accuracy.

Method used

The multipath sparsity estimation method of water acoustic channel based on deep reinforcement learning is adopted, and the sparsity of water acoustic channel is adaptively adjusted using the deep reinforcement learning algorithm DQN to optimize channel estimation.

Benefits of technology

It improves the accuracy of water acoustic channel estimation and channel equalization performance, reduces the bit error rate, and performs better especially in complex water acoustic environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120223468A_ABST
    Figure CN120223468A_ABST
Patent Text Reader

Abstract

The invention discloses an underwater acoustic channel multipath sparseness estimation method based on deep reinforcement learning, which utilizes the characteristic that an underwater acoustic channel has time-varying characteristics, especially the sparseness of the underwater acoustic channel changes along with time. However, sparseness is an important parameter for a compressed sensing algorithm and has direct influence on estimation performance. Therefore, the deep reinforcement learning algorithm DQN is introduced to track the sparseness of the underwater acoustic channel, and then the channel impulse response is accurately estimated by adopting the OMP algorithm. Compared with the traditional LS and OMP algorithms, the method can effectively track the time-varying characteristics of the underwater acoustic channel, better estimate the sparseness of the underwater acoustic channel, and significantly improve the decoding performance. Compared with a Q-learning algorithm based on a table form, the method can achieve convergence more quickly.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater acoustic communication, and particularly relates to an underwater acoustic channel estimation method based on the orthogonal matching pursuit algorithm, which is applicable to underwater acoustic channels with obvious multipath sparsity and time-variation, and specifically relates to an underwater acoustic channel multipath sparsity estimation method based on deep reinforcement learning. Background Art

[0002] Due to the movement of internal waves, platforms, the sea surface, etc., the underwater acoustic channel has characteristics such as large delay spread, sparsity, and time-variation. In view of the sparsity characteristic of the underwater acoustic channel, compression sensing algorithms such as the orthogonal matching pursuit algorithm are usually used to reconstruct the channel, so that the transmitted information bits can be better recovered at the receiving end.

[0003] In the current communication system, when using the orthogonal matching pursuit algorithm to estimate the underwater acoustic channel, the sparsity is unknown and is often selected according to empirical values. However, different selections of sparsity have a great impact on the channel estimation accuracy. In addition, due to the time-variation of the underwater acoustic channel, the sparsity of the underwater acoustic channel is not fixed.

[0004] Currently, the underwater acoustic channel estimation methods based on the orthogonal matching pursuit algorithm rarely consider the influence of sparsity on the estimation performance. Therefore, optimizing the sparsity parameter during the channel estimation process can adapt to the time-variation of the underwater acoustic channel, thereby improving the channel estimation accuracy. For this reason, there is an urgent need to propose an underwater acoustic channel multipath sparsity estimation method based on deep reinforcement learning for underwater acoustic channels with obvious sparsity and time-variation. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the disadvantages and deficiencies of the prior art, and provide an underwater acoustic channel multipath sparsity estimation method based on deep reinforcement learning.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions: An underwater acoustic channel multipath sparsity estimation method based on deep reinforcement learning, the underwater acoustic channel multipath sparsity estimation method includes the following steps: S1. Construct the state space of the underwater acoustic channel multipath sparsity estimation system and the action space as well as the reward function ; construct the evaluation network in the deep reinforcement learning algorithm and the target network where represents the underwater acoustic channel state, and represents the action taken in this state, and respectively represent the evaluation network and the target network The network parameters and action space are defined as , corresponding to decreasing, maintaining, or increasing the current sparsity respectively; S2. Set the parameters of the underwater acoustic channel multipath sparsity estimation system, including the discount factor , initial exploration rate , minimum exploration rate , decay factor , threshold of the experience pool , total number of training times , current training times , maximum number of orthogonal frequency division multiplexing symbols transmitted each time during training ; S3. In the underwater acoustic channel multipath sparsity estimation system, initialize the underwater acoustic channel state , and use the underwater acoustic channel state as the current multipath sparsity of the underwater acoustic channel. According to the current training times , initial exploration rate , minimum exploration rate , and decay factor , calculate the exploration rate for the th training ; S4. Based on the exploration rate calculated in step S3, select an action according to the greedy policy under the current underwater acoustic channel state . The action is selected from the action space ={+1, -1, 0}, where +1 means increasing by 1 based on the current sparsity, -1 means decreasing by 1 based on the current sparsity, 0 means keeping the current sparsity unchanged; in addition, initialize the number of transmitted orthogonal frequency division multiplexing symbols ; S5. Adjust the multipath sparsity of the current underwater acoustic channel according to the selected action , estimate the channel, perform channel equalization processing based on the estimated channel to calculate the bit error rate, and calculate the reward value using the bit error rate according to the preset reward function , and record the adjusted underwater acoustic channel state ; S6. Store the quadruple sample obtained in S5, that is into the experience pool, and judge whether the number of quadruple samples in the experience pool exceeds the threshold , if it exceeds, then go to step S7; otherwise, enter the judgment of step S10; S7. Randomly extract quadruple samples to form a training data set. For each quadruple sample in the training data set, input it into the evaluation network, and output the cumulative reward expectation value for executing each action in this underwater acoustic channel state , and take out the corresponding value as the predicted cumulative reward expectation value ; S8. For each quadruple sample in the training data set , input it into the target network to output the cumulative reward expectation value corresponding to each action in this underwater acoustic channel state , select the maximum value from it and combine the discount factor with the reward value in this quadruple sample to calculate the target cumulative reward expectation value ; S9. Calculate the mean square error between , and then minimize the mean square error to update the parameters of the evaluation network . Whenever the parameters of the evaluation network are updated times, the current parameters are assigned to the target network, that is, let ; S10. Judge whether the number of currently transmitted orthogonal frequency division multiplexing symbols is equal to . If , then let , , go to step S4 to continue transmitting the next orthogonal frequency division multiplexing symbol. If , then further judge whether the current training times is equal to . If , then let go to step S3 for the next training. If , then end the training and execute step S11; S11. Utilize the trained evaluation network , based on the input initial underwater acoustic channel state , output the estimated value of the multipath sparsity of the underwater acoustic channel.

[0007] Furthermore, in step S1, the state space is defined as , where represents the maximum value of the multipath sparsity of the underwater acoustic channel, and the reward function has the following calculation formula , where represents a constant represents the bit error rate. It can be seen that a lower bit error rate corresponds to a higher reward, thus guiding the underwater acoustic channel multipath sparsity estimation system to learn strategies for reducing the bit error rate; the structures of the constructed evaluation network and target network are the same, using a fully connected neural network to reduce oscillations during training and improve convergence

[0008] Furthermore, in step S3, the exploration rate for the th training is calculated according to the following formula

[0009] where is the initial exploration rate is the minimum exploration rate is the decay factor is the current number of training times; it can be seen from the formula that as the number of training times increases decreases accordingly, so that the exploration rate is higher in the initial stage of training, effectively avoiding falling into local optima, and the exploration rate is lower in the later stage of training, which can accelerate the convergence speed and improve the accuracy of underwater acoustic channel multipath sparsity estimation

[0010] Furthermore, in step S4 the greedy policy process is as follows: randomly select an action with the exploration rate , and select the action with the maximum expected value of the current cumulative reward with probability . The operation formula is as follows

[0011] where represents that under the current underwater acoustic channel state , it is input into the evaluation network and the action that maximizes the expected value of the cumulative reward is selected The greedy policy is mainly used to explore the relationship between exploration and exploitation during training to ensure that the underwater acoustic channel multipath sparsity estimation system fully explores in the initial stage of training and tends to exploit the learned optimal strategy in the later stage

[0012] Furthermore, in step S5, the orthogonal matching pursuit algorithm is a traditional underwater acoustic channel estimation method. The underwater acoustic channel multipath sparsity is an important parameter affecting its estimation accuracy. Inputting the current underwater acoustic channel multipath sparsity adjusted according to the action into this algorithm can estimate the current underwater acoustic channel

[0013] Further, in the step S8, the expected value of the target cumulative reward is calculated by the formula:

[0014] where represents the reward obtained by executing the action under the current underwater acoustic channel state ; represents the discount factor, represents that under the next underwater acoustic channel state , select the action that maximizes the value of the target network ; The calculation of the expected value of the target cumulative reward is mainly used to update the evaluation network parameters in step S9, and at the same time can effectively improve the convergence speed during the training of the underwater acoustic channel multipath sparsity estimation system, enabling it to more accurately adjust the multipath sparsity value of the underwater acoustic channel, thereby improving the estimation accuracy.

[0015] Further, in order to test the performance of the underwater acoustic channel multipath sparsity estimation system, in the step S11, use the trained evaluation network , based on the input initial underwater acoustic channel state , adaptively adjust the sparsity of the underwater acoustic channel, and the steps of adaptive adjustment are as follows: S11.1. Input the initial underwater acoustic channel state of the underwater acoustic channel, and initialize the number of transmitted orthogonal frequency division multiplexing symbols , and establish an underwater acoustic channel multipath sparsity record list P to store the estimated values of the underwater acoustic channel multipath sparsity during the transmission process; S11.2. Input the current underwater acoustic channel state into the evaluation network, and output the expected values of the cumulative rewards corresponding to each action . Select the one corresponding to the maximum value among them as the optimal action under this underwater acoustic channel state , and execute this action to update the underwater acoustic channel state to . Store the new underwater acoustic channel state into the underwater acoustic channel multipath sparsity record list P; S11.3. Judge whether the current number of transmitted orthogonal frequency division multiplexing symbols has reached the maximum number of transmitted symbols . If not, let , continue to transmit the next orthogonal frequency division multiplexing symbol, and return to step S11.2; If so, execute step S11.4; S11.4. Output the estimated value of the multipath sparsity of the underwater acoustic channel during the entire transmission process according to the sparsity record list P.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) In the present invention, the deep reinforcement learning algorithm DQN adaptively adjusts the sparsity of the underwater acoustic channel for underwater acoustic channel estimation during the interaction with the underwater acoustic channel, overcoming the time-varying nature of the underwater acoustic channel to a certain extent. (2) The present invention uses the deep reinforcement learning algorithm DQN to estimate the sparsity of the underwater acoustic channel, and then uses the orthogonal matching pursuit algorithm to estimate the channel. Compared with traditional methods, this method can achieve higher estimation accuracy and lower bit error rate in a complex underwater acoustic environment. The simulation results show that the bit error rate is about one-tenth of that of traditional methods. Even in a simple underwater acoustic environment, such as a pool or a calm lake with weak time-varying characteristics, compared with traditional models, it will not cause a decrease in estimation accuracy and performance loss. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a flowchart of the steps of a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning disclosed in Embodiment 1 of the present invention; Figure 2 It is a flowchart of step S11 in a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning disclosed in Embodiment 1 of the present invention; Figure 3 It is a simulation diagram of the convergence performance of a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning disclosed in Embodiment 1 of the present invention; Figure 4 It is a simulation diagram of the mean square error performance of a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning disclosed in Embodiment 2 of the present invention; Figure 5 It is a simulation diagram of the bit error rate performance of a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning disclosed in Embodiment 2 of the present invention. Detailed Embodiments

[0019] To enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solution in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of this application.

[0020] In this application, the mention of "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of this application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0021] Embodiment 1 The underwater acoustic channel adopted in this embodiment is 100 time-varying underwater acoustic channels generated by combining the BELLOP acoustic simulation tool and the autoregressive model. Since this channel has time-varying characteristics and its sparsity changes dynamically with time, traditional methods are difficult to effectively track the change of channel sparsity, resulting in a decrease in channel estimation accuracy and affecting the equalization performance. The present invention adaptively tracks the time-variability of the underwater acoustic channel through a deep reinforcement learning algorithm, accurately estimates the channel sparsity, thereby improving the accuracy of channel estimation and enhancing the performance of channel equalization.

[0022] This embodiment discloses a method for estimating the multipath sparsity of an underwater acoustic channel based on deep reinforcement learning, as Figure 1 shown, including the following steps: S1. Construct the state space , action space and reward function of the multipath sparsity estimation system for the underwater acoustic channel; construct the evaluation network and target network in the deep reinforcement learning algorithm, represents the state where the underwater acoustic channel is located, represents the action taken in this state, and respectively represent the network parameters of the evaluation network and the target network . The state space is defined as , where represents that the maximum value of the multipath sparsity of the underwater acoustic channel is 20, and the action space is defined as , corresponding to reducing, maintaining, or increasing the current sparsity respectively, the reward function is defined as , where represents the bit error rate. Both the evaluation network and the target network are fully connected neural networks. The structure of the network includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is the dimension of the state 1, the number of neurons in the hidden layer is 128, and the number of neurons in the output layer is the number of action values in the action space 3; S2. Set the parameters of the underwater acoustic channel multipath sparsity estimation system, including the discount factor , the initial exploration rate , the minimum exploration rate , the decay factor , the threshold of the experience pool , the total number of training times , the current training times , the maximum number of orthogonal frequency division multiplexing symbols transmitted each time during training ; S3. In the underwater acoustic channel multipath sparsity estimation system, initialize the initial channel state of the underwater acoustic channel as , indicating that the multipath sparsity of the current underwater acoustic channel is 7. According to the current training times , the initial exploration rate , the minimum exploration rate , and the decay factor According to the formula calculate the exploration rate for the th training; S4. Based on the exploration rate calculated in step S3, according to the greedy policy, select an action under the current underwater acoustic channel state , the action is selected from the action space ={+1, -1, 0}, where +1 means increasing by 1 on the basis of the current sparsity, -1 means decreasing by 1 on the basis of the current sparsity, 0 means keeping the current sparsity unchanged; in addition, initialize the number of transmitted orthogonal frequency division multiplexing symbols ; S5. Adjust the multipath sparsity of the current underwater acoustic channel according to the selected action , and use the orthogonal matching pursuit algorithm to estimate the channel. Based on the estimated channel, perform channel equalization processing to calculate the bit error rate. According to the preset reward function , calculate the reward value using the bit error rate , and record the adjusted underwater acoustic channel state ; S6. Put the quadruple samples obtained in S5, i.e., into the experience pool, and determine whether the number of quadruple samples in the experience pool exceeds the threshold . If it exceeds, go to step S7; otherwise, enter the judgment in step S10; S7. Randomly extract quadruple samples from the experience pool to form a training data set. For each quadruple sample in the training data set, input it into the evaluation network, and output the cumulative reward expectation value for each action executed under this underwater acoustic channel state , and take out the corresponding to this quadruple sample value as the predicted cumulative reward expectation value ; S8. For each quadruple sample in the training data set , input it into the target network to output the cumulative reward expectation value for each action corresponding to this underwater acoustic channel state , select the maximum value from it and combine it with the discount factor and the reward value in this quadruple sample to calculate the target cumulative reward expectation value . The calculation formula is ; S9. Calculate the mean square error between and , then minimize the mean square error and update the parameters of the evaluation network . Every time the parameters of the evaluation network are updated times, assign the current parameters to the target network, i.e., let ; S10. Determine whether the number of currently transmitted orthogonal frequency division multiplexing symbols is equal to . If , then let , , go to step S4 to continue transmitting the next orthogonal frequency division multiplexing symbol. If , then further determine whether the current training times is equal to . If , then let go to step S3 for the next training. If , then end the training and execute step S11; S11. Use the trained evaluation network , based on the input initial underwater acoustic channel state , output the estimated value of the multipath sparsity of the underwater acoustic channel. The specific steps are as Figure 2 shown as follows: S11.1. Input the initial underwater acoustic channel state of the underwater acoustic channel , and initialize the number of transmitted orthogonal frequency division multiplexing symbols , establish a multipath sparsity record list P of the underwater acoustic channel to store the estimated values of the multipath sparsity of the underwater acoustic channel during the transmission process; S11.2. Input the current underwater acoustic channel state into the evaluation network, and output the expected cumulative reward values corresponding to each action . Select the one corresponding to the maximum value from them as the optimal action in this underwater acoustic channel state , and execute this action to update the underwater acoustic channel state to , and store the new underwater acoustic channel state into P; S11.3. Judge whether the current number of transmitted orthogonal frequency division multiplexing symbols has reached the maximum number of transmitted symbols . If not, let , continue to transmit the next orthogonal frequency division multiplexing symbol, and return to step S11.2; if so, execute step S11.4; S11.4. According to the sparsity record list P, output the estimated value of the multipath sparsity of the underwater acoustic channel during the entire transmission process.

[0023] During the training process of this example, the change of the cumulative reward with the number of training times is as Figure 3 shown. The simulation results show that compared with the Q-learning algorithm, the multipath sparsity estimation algorithm of the underwater acoustic channel based on deep reinforcement learning (OMP-DQN) reaches convergence after 40 times of training, and the convergence speed is significantly improved. At the same time, the cumulative reward value of the algorithm tends to be stable after convergence, with almost no obvious fluctuation, indicating that its training process is more stable and the policy optimization effect is better.

[0024] Embodiment 2 To verify the performance of the underwater acoustic channel multipath sparsity estimation system in a noisy environment, this embodiment refers to steps S1 to S11 in Embodiment 1, outputs the estimated value of the underwater acoustic channel multipath sparsity, and inputs it into the OFDM baseband simulation communication system for testing. The specific steps are as follows: within the signal-to-noise ratio range of 0 - 30 dB, 1000 simulations are carried out, and 100 OFDM symbols are transmitted each time. The channel estimation mean square error graph and the bit error rate graph are obtained as shown in Figure 4 , Figure 5 respectively. The simulation results show that compared with the traditional least squares method (LS) and the orthogonal matching pursuit (OMP) algorithm, the underwater acoustic channel multipath sparsity estimation method based on deep reinforcement learning (OMP-DQN) exhibits lower estimation accuracy and lower bit error rate under different signal-to-noise ratio conditions, verifying its superiority.

[0025] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0026] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning, characterized in that: The underwater acoustic channel multipath sparsity estimation method comprises the following steps: S1. Constructing the state space of underwater acoustic channel multipath sparsity estimation system , Action Space And the reward function ; Constructing the evaluation network in deep reinforcement learning algorithm and the target network , Indicates the state of the underwater acoustic channel, Indicates the action taken in this state. and Represent the evaluation network With the target network The network parameters and action space Defined as , corresponding to reducing, maintaining or increasing the current sparsity respectively; S2. Set the parameters of the underwater acoustic channel multipath sparsity estimation system, including the discount factor , initial exploration rate , minimum exploration rate , attenuation factor , the threshold of the experience pool , total number of training sessions , Current training times , the maximum number of OFDM symbols transmitted during each training ; S3. In the underwater acoustic channel multipath sparsity estimation system, initialize the underwater acoustic channel state , the underwater acoustic channel state As the multipath sparsity of the current underwater acoustic channel, according to the current number of training , initial exploration rate , minimum exploration rate And the attenuation factor Calculate the The exploration rate of training ; S4, based on the exploration rate calculated in step S3 ,according to The greedy strategy is used in the current state of the underwater acoustic channel. Next select action , the action From the action space ={+1, -1, 0}, where +1 means adding 1 to the current sparsity. -1 means reducing the current sparsity by 1. 0 means keeping the current sparsity unchanged; in addition, initialize the number of transmitted OFDM symbols ; S5. According to the selected action Adjust the multipath sparsity of the current underwater acoustic channel, estimate the channel, perform channel equalization based on the estimated channel to calculate the bit error rate, and calculate the error rate according to the preset reward function. , using the bit error rate to calculate the reward value , and record the adjusted underwater acoustic channel status ; S6, the four-tuple sample obtained in S5, that is, Store in the experience pool and determine whether the number of quadruple samples in the experience pool exceeds the threshold If it exceeds, then go to step S7, otherwise go to step S10; S7, randomly draw from the experience pool Four-tuple samples To form a training data set, for each four-tuple sample in the training data set, Input into the evaluation network, and output the expected value of the cumulative reward for each action performed under the underwater acoustic channel state , and take out the one that matches the sample in the four-tuple Corresponding The value is used as the expected value of the predicted cumulative reward ; S8. For each quadruple sample in the training dataset ,Will Input into the target network and output to get the expected value of the cumulative reward corresponding to each action in the underwater acoustic channel state , select the maximum value and combine it with the discount factor And the reward value in the four-tuple sample Calculate the expected value of the target cumulative reward ; S9. Calculation and The mean square error between them is then minimized and the parameters of the evaluation network are updated , every time the parameters of the evaluation network are updated Then the current parameters are assigned to the target network, that is, ; S10, determine the number of OFDM symbols currently being transmitted Is it equal to ,like , then let , , go to step S4 to continue transmitting the next OFDM symbol. Then further determine the current number of training times Is it equal to ,like Then Go to step S3 for the next training. Then the training ends and step S11 is executed; S11. Using the trained evaluation network , based on the initial underwater acoustic channel state input , output the multipath sparsity estimation value of the underwater acoustic channel.

2. According to the method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning according to claim 1, it is characterized in that: In step S1, the state space Defined as ,in represents the maximum value of the multipath sparsity of the underwater acoustic channel, and the reward function The calculation formula is ,in represents a constant, Represents the bit error rate; the constructed evaluation network has the same structure as the target network, and a fully connected neural network is used.

3. According to the method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning in claim 1, it is characterized in that: In step S3, The exploration rate of training Calculated according to the following formula: in, is the initial exploration rate, is the minimum exploration rate, is the attenuation factor, is the current number of training times.

4. According to the method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning according to claim 1, it is characterized in that: In step S4 The greedy strategy process is as follows: with the exploration rate Randomly select actions with probability Select the action with the largest expected value of the current cumulative reward. The operation formula is as follows: in, Indicates the current underwater acoustic channel status Next, input it into the evaluation network and choose the action that maximizes the expected value of the cumulative reward .

5. According to the method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning in claim 1, it is characterized in that: In step S8, the target cumulative reward expected value The calculation formula is: in, Indicates the current underwater acoustic channel status Next action The rewards received, represents the discount factor, Indicates the next underwater acoustic channel status , select Make the target network The action with the largest value .

6. According to the method for estimating multipath sparsity of underwater acoustic channels based on deep reinforcement learning in claim 1, it is characterized in that: In step S11, the trained evaluation network is used , based on the initial underwater acoustic channel state input , adaptively adjust the sparsity of the underwater acoustic channel. The steps of adaptive adjustment are as follows: S11.

1. Initial channel state of the input underwater acoustic channel , and initialize the number of transmitted OFDM symbols , establish an underwater acoustic channel multipath sparsity record list P for storing the estimated value of the underwater acoustic channel multipath sparsity during the transmission process; S11.

2. Current underwater acoustic channel status Input into the evaluation network and output each action The corresponding cumulative reward expectation value, select the maximum value corresponding to In the underwater acoustic channel state The optimal action under the condition is taken and executed, so that the underwater acoustic channel state is updated to , the new underwater acoustic channel state Stored in the underwater acoustic channel multipath sparsity record list P; S11.3 Determine the number of OFDM symbols currently transmitted Whether the maximum number of transmission symbols has been reached , if otherwise , Continue to transmit the next OFDM symbol and return to step S11.2; if yes, execute step S11.4; S11.

4. According to the sparsity record list P, output the multipath sparsity estimation value of the underwater acoustic channel during the entire transmission process.