Compensation method and system for optimizing underwater UUV communication Doppler effect

Abstract

The invention discloses a compensation method and system for optimizing the Doppler effect of underwater UUV communication, and the method comprises the steps: initializing a Q table, defining a state space and an action space of a UUV, enabling the state space to comprise Doppler frequency shift, a signal-to-noise ratio and an error rate, enabling the action space to be the adjustment action of the frequency of the UUV, enabling the Q table to store the value of a state-action pair, and enabling the Q table to store the value of the UUV; q values represent long-term rewards possibly obtained after a certain action is selected, and all Q values are set to be 0 during initialization; based on the current state, selecting actions in the action space by using an epsilon-greedy strategy; executing the action and updating the state, and calculating an award according to an award function which considers the signal-to-noise ratio, the bit error rate and the energy consumption; updating the Q value, and updating the Q value in the Q table according to the calculated reward and the new state; and outputting an optimal strategy until the Q table converges.

Description

Technical Field

[0001] The present invention relates to the field of UUV communication, and in particular to a compensation method and system for optimizing the Doppler effect of underwater UUV communication. Background Art

[0002] In underwater environments, communication between UUVs relies on acoustic signals, which are affected by many factors during transmission, including changes in water flow, depth, temperature, etc. In particular, when there is relative motion between UUVs, the Doppler effect will cause the signal frequency to shift, causing the signal frequency received by the receiver to change, which may cause communication failure or signal attenuation. Therefore, how to maintain communication quality in this dynamic environment and reduce the negative impact of the Doppler effect has become an urgent problem to be solved in underwater communication technology.

[0003] In the prior art, compensation for Doppler frequency shift usually relies on manual settings or rule-based algorithms. These methods often require pre-setting of channel models and changes in Doppler effect. Traditional methods usually rely on fixed frequency adjustment rules, which cannot achieve optimal compensation under different circumstances and have poor adaptability. Summary of the invention

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention discloses a compensation method for optimizing the Doppler effect of underwater UUV communication, comprising: a. Initialize the Q table and define the state space and action space of the UUV. The state space includes Doppler frequency shift, signal-to-noise ratio, and bit error rate. The action space is the adjustment action of the UUV frequency. The Q table stores the value of the state-action pair. The Q value represents the long-term reward that may be obtained after selecting a certain action. At initialization, all Q values ​​are set to 0; b. Based on the current state, use the ε-greedy strategy to select an action in the action space; c. Execute the action and update the state, calculating the reward according to the reward function, where the reward function takes into account the signal-to-noise ratio, bit error rate and energy consumption; d. Update the Q value, and update the Q value in the Q table according to the calculated reward and the new state; e. Repeat steps b to d until the Q table converges and output the optimal strategy.

[0005] Furthermore, the selecting an action in the action space using the ε-greedy strategy based on the current state includes: When selecting an action based on the state, a random value r is randomly generated; When r is less than ε, the action with the largest Q value corresponding to the state is selected with probability ε; Otherwise, an action in the action space is randomly selected with probability 1-ε.

[0006] Furthermore, the reward function is set as: R t =α×SNR-β×BER-γ×EnergyCost; Among them, α, β, γ are weight coefficients, SNR is signal-to-noise ratio, BER is bit error rate, and EnergyCost is energy consumption.

[0007] Further, the updating Q value includes: According to the reward R corresponding to the previous state t and the new state S t+1 , update the Q value in the Q table, including: Q(S t ,A t ) is updated to Q(S t ,A t )+η×(R t+1 +δ×maxQ(S t+1 ,A')-Q(S t ,A t )); Among them, S t is the current state, A t is the currently selected action, R t+1 is the reward value after taking the action, Q(S t ,A t ) indicates the current state S t Next, select Action A t Q value, that is, the expected benefit of the current action; maxQ(S t+1 ,A') means in the next state S t+1 In the above formula, the maximum Q value of the best action A′ is selected, that is, the maximum future reward that may be obtained in the next state. η is the learning rate, which determines the ratio of immediate reward to historical experience. η∈[0,1], when η=0, it completely relies on historical Q values ​​and does not learn new information. When η=1, it completely ignores historical Q values ​​and only relies on current rewards. δ is the discount factor, which determines the ratio of future rewards to current rewards. When δ=0, it focuses entirely on immediate rewards and ignores future gains. When δ=1, it focuses entirely on future rewards and ignores immediate rewards.

[0008] Furthermore, the output optimal strategy includes: When the Q table converges, the optimal strategy is output: ; Among them, arg represents the independent variable of the selection function, arg(maxQ(S,A)) represents the action with the largest Q value, S is the state, A is the action, is the optimal strategy.

[0009] Furthermore, the UUV frequency adjustment actions include: increasing the frequency offset, decreasing the frequency offset, and keeping the current frequency unchanged.

[0010] In another aspect, the present invention provides a compensation system for optimizing the Doppler effect of underwater UUV communications, comprising: Initialization module, used to initialize the Q table, define the state space and action space of UUV, where the state space includes Doppler frequency shift, signal-to-noise ratio and bit error rate, the action space is the adjustment action of UUV frequency, the Q table stores the value of state-action pair, the Q value represents the long-term reward that may be obtained after selecting a certain action, and all Q values ​​are set to 0 during initialization; A reinforcement learning module that performs the following steps: Based on the current state, an ε-greedy strategy is used to select an action in the action space; Execute actions and update states, calculating rewards based on a reward function that takes into account signal-to-noise ratio, bit error rate, and energy consumption; Update the Q value, and update the Q value in the Q table according to the calculated reward and the new state; The strategy output module is used to repeat the steps in the reinforcement learning module until the Q table converges and output the optimal strategy.

[0011] By adopting the above technical solutions, the frequency adjustment strategy can be automatically optimized, greatly reducing human intervention. Q-Learning can automatically adjust the frequency through repeated interactions with the environment and find the best frequency adjustment action without human intervention, reducing the complexity of manually setting frequency adjustment parameters. The reward mechanism evaluates the effect of each frequency adjustment, and optimizes the frequency adjustment strategy based on the feedback of state-action-reward. It can adaptively adjust the frequency compensation strategy to adapt to complex and changing environmental conditions. The Q-Learning algorithm uses reinforcement learning, without the need for complex channel models, and only relies on feedback signals for adjustment. It does not rely on accurate modeling of the environment, but gradually optimizes through interaction with the environment, thereby simplifying the calculation and modeling complexity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which: Figure 1 Flowchart of this application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0014] Reference Figure 1 , an embodiment of the present invention provides a compensation method for optimizing the Doppler effect of underwater UUV communication, comprising: a. Initialize the Q table and define the state space and action space of the UUV. The state space includes Doppler shift, signal-to-noise ratio, and bit error rate. The action space is the adjustment action of the UUV frequency. The Q table stores the value of the state-action pair. The Q value represents the long-term reward that may be obtained after selecting a certain action. At initialization, all Q values ​​are set to 0.

[0015] Among them, the adjustment actions of the UUV frequency include: increasing the frequency offset, decreasing the frequency offset, and keeping the current frequency unchanged. The Doppler shift is the frequency change caused by the relative speed and direction of the UUV. The signal-to-noise ratio (SNR) indicates the signal quality. A low SNR will cause communication failure. The bit error rate (BER) is an indicator of communication quality. When the bit error rate is too high, the communication performance will decrease.

[0016] This application adopts the Q-Learning algorithm to combat the Doppler effect by adjusting communication parameters in real time. The frequency shift caused by the Doppler effect will cause the frequency offset of the communication signal, so the communication frequency must be dynamically adjusted to maintain signal quality.

[0017] b. Based on the current state, use the ε-greedy strategy to select an action in the action space.

[0018] Specifically, they include: When selecting an action based on the state, a random value r is randomly generated; When r is less than ε, the action with the largest Q value corresponding to the state is selected with probability ε; Otherwise, an action in the action space is randomly selected with probability 1-ε.

[0019] The specific value of ε is set by those skilled in the art according to actual needs and will not be described in detail here.

[0020] c. Execute the action and update the state, calculating the reward according to the reward function, where the reward function takes into account the signal-to-noise ratio, bit error rate, and energy consumption.

[0021] Specifically, the reward function is set as: R t =α×SNR-β×BER-γ×EnergyCost; Among them, α, β, γ are weight coefficients, SNR is signal-to-noise ratio, BER is bit error rate, and EnergyCost is energy consumption.

[0022] The reward function is used to evaluate the effect of each frequency adjustment action. The following factors are considered when designing the reward function: If the bit error rate (BER) decreases by adjusting the frequency, a positive reward is given, and if the bit error rate increases or does not change, a negative reward is given.

[0023] If the signal-to-noise ratio (SNR) increases, a positive reward is given, and if the bit error rate decreases or remains unchanged, a negative reward is given.

[0024] If the energy consumption decreases (EnergyCost), a positive reward is given, and if the bit error rate increases or does not change, a negative reward is given.

[0025] This reward function will guide the Q-Learning algorithm to select the best frequency adjustment strategy.

[0026] d. Update the Q value and update the Q value in the Q table according to the calculated reward and new status.

[0027] The specific steps for updating the Q value include: According to the reward R corresponding to the previous state t and the new state S t+1 , update the Q value in the Q table, including: Q(S t ,A t ) is updated to Q(S t ,A t )+η×(R t+1 +δ×maxQ(S t+1 ,A')-Q(S t ,A t )); Among them, S t is the current state, A t is the currently selected action, R t+1 is the reward value after taking the action, Q(S t ,A t ) indicates the current state S t Next, select Action A t Q value, that is, the expected benefit of the current action; maxQ(S t+1 ,A') means in the next state S t+1In the above formula, the maximum Q value of the best action A′ is selected, that is, the maximum future reward that may be obtained in the next state. η is the learning rate, which determines the ratio of immediate reward to historical experience. η∈[0,1], when η=0, it completely relies on historical Q values ​​and does not learn new information. When η=1, it completely ignores historical Q values ​​and only relies on current rewards. δ is the discount factor, which determines the ratio of future rewards to current rewards. When δ=0, it focuses entirely on immediate rewards and ignores future gains. When δ=1, it focuses entirely on future rewards and ignores immediate rewards.

[0028] e. Repeat steps b to d until the Q table converges and output the optimal strategy.

[0029] By looping the above steps, the Q-Learning algorithm will continuously optimize the Q table. Whenever the environmental state changes, the Q-Learning algorithm will calculate the new reward value and update the Q value, learn how to better combat frequency shift, and finally learn the strategy of selecting the optimal action under different environmental states.

[0030] When the Q table converges, the optimal strategy is output: ; Among them, arg represents the independent variable of the selection function, arg(maxQ(S,A)) represents the action with the largest Q value, S is the state, A is the action, is the optimal strategy.

[0031] Through training, the optimal strategy learned by Q-Learning can guide how to adjust the frequency in different states to minimize the bit error rate (BER). In practical applications, the algorithm selects an optimal action from the Q table, that is, the frequency increment, based on the current frequency offset and signal-to-noise ratio and other state information. For example, assuming that the bit error rate is high in the current state, the Q-Learning algorithm may choose to reduce the frequency offset, reduce interference in communication by adjusting the frequency, and optimize signal quality. After each training, the algorithm will gradually converge to the optimal frequency adjustment strategy based on the update of the Q value.

[0032] As mentioned above, the Q-Learning algorithm can automatically optimize the frequency adjustment strategy, greatly reducing human intervention. Specifically, Q-Learning can automatically adjust the frequency through repeated interactions with the environment and find the best frequency adjustment action without human intervention, reducing the complexity of manually setting frequency adjustment parameters. The Q-Learning algorithm evaluates the effect of each frequency adjustment (such as SNR improvement, BER reduction, etc.) through a reward mechanism, and optimizes the frequency adjustment strategy based on the feedback of state-action-reward. It can adaptively adjust the frequency compensation strategy to adapt to complex and changing environmental conditions.

[0033] The Q-Learning algorithm uses reinforcement learning, without the need for complex channel models, and only relies on feedback signals (such as frequency offset, BER, SNR) for adjustment. It does not rely on accurate modeling of the environment, but gradually optimizes through interaction with the environment, thereby simplifying the calculation and modeling complexity of the system.

[0034] As a state-space based reinforcement learning method, Q-Learning is highly adaptable and can be continuously applied in multi-UUV communication networks without the need for large-scale hardware support or highly complex software algorithms. Through this efficient learning process, Q-Learning can be extended to multi-UUV scenarios, where each UUV can independently learn how to compensate for Doppler shift, thereby improving the efficiency of the overall system.

[0035] These advantages give the Q-Learning algorithm significant technical advantages in solving the Doppler frequency shift problem, which can significantly improve system performance, reduce operating costs, and improve communication stability and efficiency.

[0036] An embodiment of the present invention further provides a system, comprising: Initialization module, used to initialize the Q table, define the state space and action space of UUV, where the state space includes Doppler frequency shift, signal-to-noise ratio and bit error rate, the action space is the adjustment action of UUV frequency, the Q table stores the value of state-action pair, the Q value represents the long-term reward that may be obtained after selecting a certain action, and all Q values ​​are set to 0 during initialization; A reinforcement learning module that performs the following steps: Based on the current state, an ε-greedy strategy is used to select an action in the action space; Execute actions and update states, calculating rewards based on a reward function that takes into account signal-to-noise ratio, bit error rate, and energy consumption; Update the Q value, and update the Q value in the Q table according to the calculated reward and the new state; The strategy output module is used to repeat the steps in the reinforcement learning module until the Q table converges and output the optimal strategy.

[0037] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0038] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0039] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the compensation of Doppler effect of underwater UUV communication, characterized in that: include: a. Initialize the Q table and define the state space and action space of the UUV. The state space includes Doppler frequency shift, signal-to-noise ratio, and bit error rate. The action space is the adjustment action of the UUV frequency. The Q table stores the value of the state-action pair. The Q value represents the long-term reward that may be obtained after selecting a certain action. At initialization, all Q values ​​are set to 0; b. Based on the current state, use the ε-greedy strategy to select an action in the action space; c. Execute the action and update the state, calculating the reward according to the reward function, where the reward function takes into account the signal-to-noise ratio, bit error rate and energy consumption; d. Update the Q value, and update the Q value in the Q table according to the calculated reward and the new state; e. Repeat steps b to d until the Q table converges and output the optimal strategy.

2. The method according to claim 1, characterized in that The method of selecting an action in the action space using the ε-greedy strategy based on the current state includes: When selecting an action based on the state, a random value r is randomly generated; When r is less than ε, the action with the largest Q value corresponding to the state is selected with probability ε; Otherwise, an action in the action space is randomly selected with probability 1-ε.

3. The method according to claim 1, characterized in that The reward function is set as: R t =α×SNR-β×BER-γ×EnergyCost; Among them, α, β, γ are weight coefficients, SNR is signal-to-noise ratio, BER is bit error rate, and EnergyCost is energy consumption.

4. The method according to claim 1, characterized in that: The updating Q value comprises: According to the reward R corresponding to the previous state t and the new state S t+1 , update the Q value in the Q table, including: Q(S t ,A t ) is updated to Q(S t ,A t )+η×(R t+1 +δ×maxQ(S t+1 ,A')-Q(S t ,A t )); Among them, S t is the current state, A t is the currently selected action, R t+1 is the reward value after taking the action, Q(S t ,A t ) indicates the current state S t Next, select Action A t Q value, that is, the expected benefit of the current action; maxQ(S t+1 ,A') means in the next state S t+1 In the above formula, the maximum Q value of the best action A′ is selected, that is, the maximum future reward that may be obtained in the next state. η is the learning rate, which determines the ratio of immediate reward to historical experience, η∈[0,1]. When η=0, it completely relies on historical Q values ​​and does not learn new information. When η=1, it completely ignores historical Q values ​​and only relies on current rewards. δ is the discount factor, which determines the ratio of future rewards to current rewards, δ∈[0,1]. When δ=0, it focuses entirely on immediate rewards and ignores future gains. When δ=1, it focuses entirely on future rewards and ignores immediate rewards.

5. The method according to claim 1, characterized in that: The output optimal strategy includes: When the Q table converges, the optimal strategy is output: ； Among them, arg represents the independent variable of the selection function, arg(maxQ(S,A)) represents the action with the largest Q value, S is the state, A is the action, is the optimal strategy.

6. The method according to claim 1, characterized in that The UUV frequency adjustment actions include: increasing the frequency offset, decreasing the frequency offset, and keeping the current frequency unchanged.

7. A compensation system for optimizing the Doppler effect of underwater UUV communication, characterized in that: include: Initialization module, used to initialize the Q table, define the state space and action space of UUV, where the state space includes Doppler frequency shift, signal-to-noise ratio and bit error rate, the action space is the adjustment action of UUV frequency, the Q table stores the value of state-action pair, the Q value represents the long-term reward that may be obtained after selecting a certain action, and all Q values ​​are set to 0 during initialization; A reinforcement learning module that performs the following steps: Based on the current state, an ε-greedy strategy is used to select an action in the action space; Execute actions and update states, calculating rewards based on a reward function that takes into account signal-to-noise ratio, bit error rate, and energy consumption; Update the Q value, and update the Q value in the Q table according to the calculated reward and the new state; The strategy output module is used to repeat the steps in the reinforcement learning module until the Q table converges and output the optimal strategy.

Images