A method for underwater acoustic jamming decision based on deep reinforcement learning

Abstract

The application discloses a kind of underwater acoustic interference decision-making methods based on deep reinforcement learning, it is related to the field underwater acoustic communication technology field, for using deep reinforcement learning optimization interference resource allocation and improve interference energy efficiency ratio, including constructing three nodes underwater acoustic confrontation physical model carries out underwater acoustic confrontation experiment simulation, the confrontation process of jammer and target sending end is modeled as Markov decision process, generates action vector including interference power, interference duration and emission time using the deep reinforcement learning optimal decision algorithm of proximal policy optimization, reward function is constructed by introducing physical perception heuristic term, with the strategy pruning mechanism of PPO algorithm, it is realized in non-stationary ocean current environment under stable policy iteration.The application solves the problem of "monitoring missing" caused by long time delay in underwater acoustic communication and the problem of non-stationary interference failure caused by ocean current jitter, realizes accurate interference and robust interference on target communication link in non-stationary underwater acoustic environment.

Description

Technical Field

[0001] This invention relates to the field of underwater acoustic communication technology, and more particularly to the field of underwater acoustic communication countermeasures technology. Specifically, it refers to an underwater acoustic interference decision-making method based on deep reinforcement learning, which is used to optimize the allocation of interference resources, compensate for sound speed delay, and improve energy efficiency ratio by utilizing deep reinforcement learning. Background Technology

[0002] In complex underwater acoustic warfare scenarios, underwater monitoring networks or autonomous underwater vehicles (AUVs) frequently need to upload critical intelligence. However, enemy intrusion nodes often attempt to intercept and decode our communication packets, seriously threatening our information security. Therefore, it is necessary to monitor and jam illegal signals. In underwater acoustic communication warfare, traditional jamming methods mainly face three major challenges: energy efficiency bottlenecks, asynchronicity of perception caused by long latency, and disturbances from non-stationary environments.

[0003] To address the energy efficiency bottleneck, underwater jamming nodes typically have limited energy and cannot directly obtain the communication status of the target link. They need to passively listen to infer the target link's communication status and then execute jamming actions based on that status, which consumes a significant amount of energy. Therefore, minimizing one's own energy consumption and maximizing the enemy's retransmission cost while jamming enemy communication is a complex multi-objective trade-off problem. Furthermore, the underwater acoustic propagation speed is only 1500 m / s. By the time the jammer detects the signal, the target signal already has a significant spatial propagation delay, causing a misalignment between the observed state and the actual physical situation, resulting in a "detection miss" problem of asynchronicity. The underwater acoustic communication environment in the ocean is affected by ocean currents and tides, and the underwater acoustic communication channel is a non-stationary channel with random jitter, making temporal overlap of jamming extremely difficult.

[0004] Therefore, there is an urgent need to develop an underwater acoustic interference decision-making method that achieves precise and robust interference to target communication links in non-stationary underwater acoustic environments by listening to the repeating packet sequence number of the intrusion signal to provide feedback, while significantly reducing energy consumption. Summary of the Invention

[0005] The purpose of this invention is to provide a deep reinforcement learning-based underwater acoustic interference decision-making method to solve the problems of low interference accuracy and low energy utilization in the existing technology under long time delay and non-stationary underwater acoustic environments.

[0006] To address the above objectives, this invention provides a deep reinforcement learning-based underwater acoustic interference decision-making method, comprising: S1. Construct a physical model for underwater acoustic countermeasures to simulate the underwater acoustic signal countermeasure process; S2. Model the underwater acoustic signal countermeasure process as a Markov decision process, obtain and quantify the state vector of the underwater acoustic countermeasure physical model, and construct the state space; S3. Construct an underwater acoustic anti-interference decision model based on the near-end policy optimization algorithm, using the state vector as the model input, and using the near-end policy optimization algorithm to generate action vectors and construct the action space; S4. Construct a reward function, coupling communication interference reward term, energy efficiency cost penalty term, retransmission cost benefit term, and physical perception heuristic term; S5. Initialize the policy network and value network. Calculate the probability distribution of the state vector mapping to the action vector through the policy network, and estimate the value of the state vector based on the reward function through the value network. S6. Iterate the policy network and value network using the policy pruning mechanism of the generalized advantage estimation combined with the proximal policy optimization algorithm until the network converges and the optimal disturbance decision is obtained.

[0007] In S1, the underwater acoustic countermeasures physical model consists of a target transmitter, a target receiver, and a jammer. The signal transmitted by the target transmitter is the target signal, the signal transmission link between the target transmitter and the target receiver is the target link, the signal transmitted by the jammer is the jamming signal, and the signal transmission link between the target receiver and the jammer is the jamming link. In accordance with the JANUS underwater acoustic communication standard protocol, the underwater acoustic countermeasures simulation parameters are set in the physical model of underwater acoustic countermeasures, and the target signal transmission distance is defined as the straight-line distance between the target transmitter and the target receiver. The transmission distance of the jamming signal is the straight-line distance between the target receiver and the jammer. The jammer transmits an acoustic signal at the same frequency as the target transmitter, with a center frequency of [missing value]. The data packet length is The synchronization preamble length is .

[0008] The jammer adopts a responsive jamming mode, which acquires multi-dimensional state vectors by listening and constructs a state space. The state space includes the equivalent signal-to-interference-plus-noise ratio, time-domain overlap deviation, time delay jitter variance, communication feedback identifier, and the remaining capability status of the target transmitter. Based on underwater acoustic propagation theory, a path propagation loss calculation model is constructed to calculate the path loss. : ; ; In the formula, Represents the logarithm to base 10. Represents the geometric diffusion factor. Indicates the signal transmission distance. This represents the frequency absorption coefficient, which is calculated using the Thorp empirical model. Based on the strength of the target signal measured by the jammer, and combined with the path propagation loss calculation model, the equivalent signal-to-interference-plus-noise ratio at the target receiver is calculated. : ; In the formula, The target's estimated transmit power is obtained by the jammer through reverse calculation based on the signal strength of the passively monitored signal, and is equivalent to the target's nominal transmit power calculated by the jammer through reverse calculation based on the passively monitored signal strength. This indicates the actual transmission power of the jammer. This represents the estimated ambient background noise measured by the interference device.

[0009] A random process is introduced to simulate time delay jitter in a non-stationary environment caused by local ocean currents, internal waves, and nodal micro-displacements, to determine the moment when the jammer detects the target signal's synchronization preamble. Using the timeline anchor point, extract the actual transmission timestamp of the jammer's jamming signal. Calculate the time-domain overlap deviation : ; In the formula, Indicates the physical delay of the interfering link. Indicates geometric correction amount; Based on continuous monitoring by the jammer, the arrival times of each synchronization preamble detected during the monitoring process are statistically recorded to construct a sequence of continuously intercepted signal arrival times. Based on the time intervals between these times, a sliding window is used to calculate the variance of the time delay jitter. : ; In the formula, Indicates the length of the sliding window. This represents the total number of times in the sequence of continuously intercepted signal arrival times. Indicates the first A time interval, This represents the average time interval.

[0010] The target link operates using the Automatic Repeat Request (ARQ) protocol. Based on the ARQ protocol, the jammer identifies and records the packet sequence of the target signal it has been monitoring. If the packet sequence is found to be identical to the historical record, the jammer determines that the previous round of interference was successful and sets a communication interference feedback flag. Set to 1, otherwise set the communication feedback flag. Set to 0; when When =1, the jammer calculates the energy consumption for target signal retransmission based on the data packet length and signal center power, and estimates the remaining capacity status of the target transmitter. .

[0011] In S3, the action space is a joint decision triplet, including the interference power level, interference duration, and transmission time offset. The interference power level adopts adaptive power adjustment logic, and the interference duration is based on the delay jitter variance. Dynamic adjustment: When the jammer detects increased non-stationarity in the environment, it adaptively increases the jamming duration to provide temporal coverage redundancy, ensuring that the jamming signal can compensate for the predicted temporal overlap deviation caused by random jitter. The transmission time offset adopts prior compensation logic.

[0012] Construct a multi-objective composite reward function that couples a communication interference reward term, an energy efficiency cost penalty term, a retransmission cost benefit term, and a physical sensing heuristic term: ; ; ; In the formula, Indicates a reward for communication interference; This represents the cost-benefit item for retransmission. This indicates a penalty item for energy efficiency costs. , , These represent the weights of the communication interference reward, retransmission cost benefit, and energy efficiency cost penalty, respectively. Represents physical perception heuristics, Indicates the heuristic intensity coefficient. This represents the width of the potential energy field.

[0013] In S5, a parameterized mapping model based on a deep neural network is constructed, utilizing the policy network parameters. Define the probability distribution: ; In the formula, The state vector is represented as When choosing the first The probability of an action. Indicates the first One action vector, The output of the policy network indicates the first... The unnormalized log probability of the output of each action. This represents the total dimension of the action space; Utilizing value network parameters Define the value function, for Multidimensional state vector at time step Valuation is based on the expected cumulative return after discount: ; In the formula, express The multidimensional state vector at time step. The state vector is represented as Time-state value, Represents the expectation operator. Indicates the discount factor. express Multi-objective composite rewards at any time. Indicates the summation index.

[0014] A pruning objective function is used to impose a hard constraint on the update step size of the policy network: ; ; ; In the formula, This represents the loss term of the pruning strategy. This represents the ratio of the probabilities of the new and old strategies. The state vector in the new strategy is... When selecting the action vector as The probability, The state vector in the old policy is When selecting the action vector as The probability, This represents the estimated value of the advantage function, when... limit The expansion, when limit shrinkage, Represents the smoothing factor. This represents the clipping hyperparameter. This represents the clipping function. express The estimated value of the empirical expectation at time t. express The time difference residual at any given moment.

[0015] Introducing the prediction bias and sampling variance of the generalized advantage estimation balanced proximal policy optimization algorithm, the time difference residual is calculated: ; In the formula, express Time difference residuals at time points express Multi-objective composite rewards at any given time; Constructing a joint loss function Achieve synchronous gradient backpropagation for the policy network and value network: ; ; ; In the formula, Indicates a value loss item. Indicates the value loss coefficient. express Discounted return value at any moment This represents the entropy regularization term. Represents the entropy regularization coefficient. express Action vector at any given moment.

[0016] Compared with the prior art, the present invention has the following advantages: This invention provides an interference decision-making method based on near-end policy optimization deep reinforcement learning. Employing prior compensation logic, it compensates for propagation delay by adjusting the transmission time offset, thus offsetting perception lag and solving the problem of "missed detection" caused by low sound speeds. This achieves accurate collision detection of standard JANUS signals. Simultaneously, it dynamically adjusts the dynamic interference duration based on the time delay jitter variance, achieving time redundancy coverage of non-stationary jitter and enhancing the model's fault tolerance to random ocean current jitter. This significantly improves interference robustness and resource allocation efficiency in non-stationary underwater acoustic channels. Especially when facing perception asynchrony caused by low sound speeds of 1500 m / s and random time delay jitter caused by ocean current disturbances, it exhibits excellent adaptive capabilities. It solves the problems of "missed detection" caused by long time delays and inaccurate interference caused by ocean current jitter in underwater acoustic communication, providing key technical support for underwater acoustic information protection and electronic countermeasures research. This invention utilizes a multi-objective reward mechanism to construct a multi-objective composite reward function that couples a communication interference reward term, an energy efficiency cost penalty term, a retransmission cost benefit term, and a physical perception heuristic term. Combined with the policy pruning mechanism of the PPO algorithm, it achieves stable policy iteration in non-stationary ocean current environments. While interfering with the opponent's communication and increasing the opponent's energy consumption, it significantly reduces our own energy consumption, achieving precise and robust interference with the target's communication link in non-stationary underwater acoustic environments. Furthermore, the introduction of a physical perception heuristic term into the reward function greatly reduces the number of samples required for policy training convergence, providing a reliable technical approach for adversarial operations in complex dynamic scenarios. Attached Figure Description

[0017] Figure 1 This is the overall technical roadmap provided by the present invention; Figure 2 This is a diagram of the interference decision-making process in an underwater acoustic countermeasure environment provided by the present invention; Figure 3 This is the design diagram of the PPO algorithm provided by the present invention; Figure 4This is a comparison chart of the reward training curves of the six optimal strategy search algorithms provided by this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] A deep reinforcement learning-based underwater acoustic interference decision-making method includes: S1. Construct a three-node underwater acoustic countermeasure physical model consisting of a target transmitter, a target receiver, and a jammer to simulate the signal countermeasure process between the jammer and the target transmitter; S2. Model the signal countermeasures process as a Markov decision process, defining a state space... Action space State transition matrix Reward function and discount factor Markov decision quintuple Based on a three-node underwater acoustic countermeasure physical model, the state vector of the underwater acoustic countermeasure physical model is acquired and quantified by the passive monitoring of the jammer, and a state space is constructed. S3. Construct an underwater acoustic counter-jamming decision model based on the near-end policy optimization algorithm. Use the state vector as the model input, use the near-end policy optimization algorithm to generate action vectors, construct the action space, and guide the jammer to perform communication jamming on the target link. S4. Construct a reward function, coupling communication interference reward term, energy efficiency cost penalty term, retransmission cost benefit term, and physical perception heuristic term; S5. Initialize the policy network and value network. Calculate the probability distribution of the state vector mapping to the action vector through the policy network, and estimate the value of the state vector based on the reward function through the value network. S6. Iterate the policy network and value network using the policy pruning mechanism of the generalized advantage estimation combined with the proximal policy optimization algorithm until the network converges and the optimal disturbance decision is obtained.

[0020] In S1, a three-node underwater acoustic countermeasure physical model is constructed. First, an underwater acoustic communication environment model consisting of a target transmitter and a target receiver is constructed. Then, in the underwater acoustic communication environment model, a jammer with passive listening and adaptive decision-making capabilities is deployed for the target transmitter. Based on the three-node underwater acoustic countermeasure physical model, the underwater acoustic signal countermeasure process between the jammer and the target transmitter is simulated by benchmarking the underwater acoustic communication protocol JANUS.

[0021] The signal transmitted by the target transmitter is the target signal, and the signal transmission link between the target transmitter and the target receiver is the target link. The signal transmitted by the jammer is the jamming signal, and the signal transmission link between the target receiver and the jammer is the jamming link. Following the JANUS protocol, underwater acoustic countermeasures simulation parameters are set, and the target signal transmission distance is defined as the straight-line distance between the target transmitter and the target receiver. The transmission distance of the jamming signal is the straight-line distance between the target receiver and the jammer. The jammer transmits an acoustic signal at the same frequency as the target transmitter, with a center frequency of [missing value]. The length of the data packet sent by the target sender is The length of the synchronization preamble of the target signal is .

[0022] In S2, the Markov decision process is the process by which a jammer in an underwater acoustic communication environment changes its state and obtains a reward by taking jamming actions. It is usually represented by a quintuple. To represent. State space. The set of all possible states, with quantized state parameters including temporal overlap bias. Delay jitter variance Communication interference feedback indicator The equivalent signal-to-interference-plus-noise ratio at the target receiver The remaining capacity status of the target transmitter Action space It is the set of all jamming actions that the jammer can take, represented as a joint decision triple, including the jamming power level. Duration of interference and launch timing offset The action depends on the current state; state transition matrix By state transition probability composition, Indicates the current state Next, execute the action. Afterwards, the environment changed The probability, used to describe the dynamic characteristics of the environment, depends only on the current state and the current action and possesses Markov property; the reward function couples the opponent's communication interference reward term, our energy efficiency cost penalty term, the opponent's retransmission cost benefit term, and the physical perception heuristic term, and is used to reflect the jammer's state... Next, execute the action. Subsequently, the environment provides immediate feedback to the jammer regarding its benefits; The discount factor, a number between 0 and 1, is used to calculate the current value of future rewards. The signaling countermeasure process is modeled as a Markov decision process, and the optimal interference strategy is obtained based on a proximal policy optimization algorithm.

[0023] The jammer employs a reactive jamming mode, acquiring data through passive listening. Multidimensional state vector at time step : ; In the formula, express The time-domain overlap deviation between the target signal and the interference signal at a given time. express The variance of time delay jitter. express The communication feedback indicator at any given moment; Indicate Equivalent signal-to-interference-plus-noise ratio at any given moment; express The remaining capacity status of the target transmitter at any given time; Each multidimensional state vector in the state space is fixed and quantized by the jammer through local observation data combined with the underwater acoustic countermeasure physical model. Based on the jammer's continuous monitoring, the state space is constructed, which includes the equivalent signal-to-interference-plus-noise ratio, temporal overlap deviation, time delay jitter variance, communication feedback identifier, and the remaining capability state of the target transmitter.

[0024] Based on underwater acoustic propagation theory, a path propagation loss calculation model is constructed to calculate the path loss. : ; ; In the formula, Represents the logarithm to base 10. , representing the geometric diffusion factor, Indicates the signal transmission distance. The frequency absorption coefficient is calculated using the Thorp empirical model, which characterizes the energy attenuation of acoustic signals with distance and frequency during underwater propagation using a path propagation loss calculation model.

[0025] Based on the strength of the target signal measured by the jammer, and combined with the path propagation loss calculation model, the equivalent signal-to-interference-plus-noise ratio at the target receiver is calculated. : ; In the formula, The target's estimated transmit power is obtained by the jammer through reverse calculation based on the signal strength of the passively monitored signal, and is equivalent to the target's nominal transmit power calculated by the jammer through reverse calculation based on the passively monitored signal strength. This indicates the actual transmission power of the jammer. This represents the estimated ambient background noise measured by the interference device.

[0026] Since the jammer cannot directly read the true signal-to-interference-plus-noise ratio (SIR) at the target receiver, it employs a reactive jamming mode. Using the observed target signal strength and its own jamming power, it constructs an equivalent SIR estimation model to calculate the SIR at the target receiver. If the calculated SIR... Less than the communication failure success threshold If the communication interference is successful, the jammer determines that the communication interference was successful. The jammer indirectly confirms the success of the communication interference and verifies the accuracy of the inference model by monitoring whether duplicate packet sequence numbers appear in subsequent monitoring.

[0027] Considering the complex time-varying nature of the marine environment, a stochastic process is introduced to simulate the non-stationary environmental time delay jitter caused by local ocean currents, internal waves, and micro-displacements of nodes. By statistically analyzing the arrival patterns of locally intercepted signals from jammers, it is possible to achieve [the goal of controlling / intercepting] [signals]. Online estimation. The instant the target signal arrives at the target receiver. for: ; In the formula, Indicates the time when the target signal was transmitted. This represents the speed at which sound travels through water.

[0028] The core of jammer decision-making lies in achieving physical time-domain overlap between the jamming signal and the target signal's synchronization preamble. This precise time-domain overlap disrupts the target link's communication. Since the jammer cannot directly obtain the time-domain overlap at the target receiver, it employs asynchronous sampling logic to address the sensing asynchrony caused by long delays, calculating the timing when the jammer detects the target signal's synchronization preamble. Using the timeline anchor point, extract the actual transmission timestamp of the jammer's jamming signal. Calculate the time-domain overlap deviation : ; In the formula, Indicates the physical delay of the interfering link. This represents the geometric correction, which is equivalent to the estimated remaining time required for the signal to propagate from the jammer to the target receiver. The jammer must arrive at the jammer's instantaneous position by accurately extracting the synchronization preamble. This serves as the anchor point on the time axis. Through compensation calculations, it is ensured that the data packets of the interfering signal and the target signal achieve envelope overlap at the spatial convergence point. The propagation delay residual characterizes the temporal overlap deviation between the center of our interfering signal envelope and the center of the synchronization preamble of the target signal data packet on the time axis at the target receiver. Temporal overlap deviation The temporal overlap between the interference signal and the target signal in space is directly quantified. The smaller the temporal overlap deviation, the higher the temporal overlap accuracy.

[0029] The jammer continuously monitors and intercepts target signals, recording the time when the synchronization preamble of each signal is detected. Based on the times when the jammer continuously monitors the synchronization preamble of the target signals, a sequence of arrival times of the continuously intercepted signals is constructed. Based on the time intervals between moments within a sequence, the variance of time delay jitter is statistically calculated using a sliding window. : ; ; In the formula, Indicates the length of the sliding window. This represents the total number of times in the sequence of continuously intercepted signal arrival times. Indicates the first A time interval, Indicates the average time interval. This indicates that during the jammer's continuous monitoring process, the first [number]th ... At that moment, This indicates that during the jammer's continuous monitoring process, the first [number]th ... At any given moment. Delay jitter variance The variance of non-stationary small-scale time delay jitter is used to quantify the statistical intensity of random fluctuations in propagation time delay caused by local ocean current turbulence, internal waves, and micro-displacements of nodes.

[0030] The target link operates using the Automatic Repeat Request (ARQ) protocol. Based on the ARQ protocol, the jammer identifies and records the sequence number of data packets in the intercepted signal. If the sequence number of a detected data packet is the same as the historical record, the jamming operation is considered successful in the previous round, and a communication jamming feedback flag is set. Set to 1, otherwise communication feedback flag. =0; when When =1, the jammer calculates the energy consumption for target signal retransmission based on the data packet length and signal center power, and estimates the remaining capacity status of the target transmitter. The jammer's induced retransmission mechanism effectively deprives the target node of its limited battery power at the resource layer. This is achieved through communication feedback identification. This reflects whether the jamming action in the previous decision step was successful, leading to target data packet decoding failure and triggering retransmission. It provides sparse but crucial feedback signals to the jammer and is the core support for the closed loop of the reward function. The remaining capacity state of the target transmitter... This is used to assess the battery energy consumption trend of a target node under continuous interference and retransmission pressure. Since the battery level of the target transmitter cannot be directly obtained, the jammer maintains an accumulation counter for the target's retransmission behavior. Based on the known packet length and the target signal transmission power value, it calculates the additional energy consumption of the target signal due to retransmission. By countering the target through "energy deprivation," the jammer enables long-term resource consumption in decision-making.

[0031] Action space is a joint decision triplet , These correspond to the interference power level, interference duration, and transmission time offset, respectively. The interference power level employs adaptive power adjustment logic to achieve a balance between communication interference effectiveness and interference cost; the interference duration is based on the delay jitter variance. Dynamic adjustment is achieved by providing temporal coverage redundancy to offset random errors. When the jammer detects increased nonstationarity in the environment, it adaptively increases the jamming duration. To provide temporal coverage redundancy, ensuring that the interference signal can compensate for the predicted temporal overlap deviation caused by random jitter; the transmission time offset adopts prior compensation logic, which cancels the sensing time delay by transmitting the interference signal in advance, so as to achieve temporal overlap between the interference signal and the target signal.

[0032] To balance the effectiveness of communication interference with its energy consumption, a multi-objective composite reward function is constructed, which couples the opponent's communication interference reward term, our energy efficiency cost penalty term, the opponent's retransmission cost benefit term, and a physical perception heuristic term. ; ; ; In the formula, This indicates the reward for interference with the other party's communications; it is a basic reward for interference with communications. This represents the cost-benefit item for the target's retransmission, signifying the cost-benefit of depriving the enemy of their energy efficiency. It is equivalent to the additional energy consumption incurred by the target performing a retransmission due to our interference. This represents the energy efficiency cost penalty, equivalent to the product of the energy consumed in a single interference action. It serves as a negative reward, aiming to constrain the algorithm from blindly emitting high-power or excessively long-duration interference. This is the core logic for solving the problem of limited underwater energy. , , These represent the weights of the communication interference reward, retransmission cost benefit, and energy efficiency cost penalty, respectively. Represents physical perception heuristics, Indicates the heuristic intensity coefficient. The width of the potential field is used to adjust the sensitivity to bias. To address the slow convergence of reinforcement learning under sparse feedback, a Gaussian potential field is used to design a physical perception heuristic. Even if the perturbation fails, i.e. ,if only Despite the smaller size, the near-end strategy optimization (PPO) algorithm can still obtain continuous gradient guidance, guiding the interference signal to quickly approach the target center point and significantly accelerating convergence.

[0033] A multi-objective composite reward function is constructed to interfere with the communication of the intruder (target transmitter), increase the energy consumption of the intruder (target transmitter), and save the energy consumption of our jammer. A physical perception heuristic term is introduced into the multi-objective composite reward function, and the policy pruning mechanism of the PPO algorithm is used to achieve stable policy iteration in non-stationary ocean current environment to obtain the optimal jamming decision.

[0034] The core of the interference decision lies in the execution engine, which employs the Proximal Policy Optimization (PPO) algorithm to solve and iteratively optimize the Markov decision process. Addressing the long latency and non-stationary environment characteristics of underwater acoustic countermeasures, and adapting to the output requirements of the discrete action space interference policy, the PPO algorithm is specifically adapted and improved. A core pruning mechanism constrains the step size and magnitude of policy updates, avoiding gradient explosion and sudden parameter changes during policy updates, ensuring the stability and robustness of policy iteration. This effectively solves the technical pain points of traditional reinforcement learning algorithms in underwater acoustic interference scenarios, such as policy collapse, convergence oscillations, and poor adaptability to non-stationary environments. It enables robust iteration and continuous optimization of the interference policy in long latency, stochastic, and energy-constrained underwater acoustic countermeasures environments, outputting the optimal discrete interference action decision.

[0035] In S5, a parameterized mapping model based on a deep neural network is constructed, utilizing the policy network parameters. Define the probability distribution: ; In the formula, The state vector is represented as When choosing the first The probability of an action. Indicates the first One action vector, The output of the policy network indicates the first... The unnormalized log probability of the output of each action. This represents the total dimension of the action space. This represents the action space traversal index, used to traverse all actions in the action space. The policy network represents the action vector. The unnormalized score is used to reflect the state vector as Time action vector The level; A parameterized mapping model based on deep neural networks is constructed to realize the nonlinear transformation from a multidimensional physical perception state space to a probability distribution of disturbance actions. The output layer of the policy network Actor maps the features of the multidimensional state vector to the trigger probability distribution of each disturbance action branch through the Softmax function.

[0036] Utilizing value network parameters Define the value function, for Multidimensional state vector at time step Valuation is based on the expected cumulative return after discount: ; In the formula, express The multidimensional state vector at time step. The state vector is represented as Time-state value, Represents the expectation operator. Indicates the discount factor. express Multi-objective composite rewards at any time. This represents the summation index, equivalent to starting from the current time. The offset of the starting future time step. Discount factor. As a discount factor, it is used to calculate from the state The initial expected cumulative reward is used to offset future rewards to the current moment.

[0037] State transition matrix As an intrinsic attribute of the environment, it is used to describe the dynamic characteristics of underwater acoustics, that is, how the state changes after an interference action is performed. Implicit in the execution flow of the PPO algorithm, it is implicitly reflected through the sampling process and is the driver of the PPO algorithm's sampling process. Based on the sampling and policy update of the PPO algorithm, the policy network and value network are initialized, and the interference machine... Multidimensional state vector at time step Output action vector In a three-node underwater acoustic adversarial physics model, trajectory samples are executed and stored. Trajectory generation depends on state transition probabilities and policies. The underwater acoustic communication environment model, based on state transition probabilities and state vectors, executes and stores these samples. and action vectors Generate the state vector for the next time step. and instant rewards , sample Stored in the experience buffer, the sample contains the state transition matrix. Based on the sampling results, the PPO algorithm uses this sample data to obtain empirical approximation of the optimal strategy. State transition matrix. By influencing the sample distribution, the direction and magnitude of policy updates can be affected.

[0038] To avoid drastic oscillations in the policy during a single gradient update, a pruned objective function is used to impose a hard constraint on the update step size of the policy network: ; ; In the formula, This represents the loss term of the pruning strategy. This represents the ratio of the probabilities of the new and old strategies. The state vector in the new strategy is... When selecting the action vector as The probability, The state vector in the old policy is When selecting the action vector as The probability, This represents the estimated value of the advantage function, when... limit The expansion, when limit shrinkage, Represents the smoothing factor. This represents the clipping hyperparameter. This represents the clipping function. express The estimated value of the empirical expectation at time t. express The time difference residual at any given moment. This indicates that the current interference action is better than the average performance. This indicates that the current action has resulted in a decrease in interference effectiveness or an excessively low energy efficiency ratio. This asymmetric constraint mechanism ensures that the decision-making logic of the jammer will not collapse when faced with sudden changes in state caused by ocean current fluctuations.

[0039] Based on smoothing factor The recursive expression for the dominance function is: ; In the formula, Represents the recursive time parameter, smoothing factor The smoothing parameter for generalized dominance estimation is obtained by adjusting the parameter. The PPO algorithm can effectively smooth out the instantaneous observation noise introduced by the non-stationary characteristics of the underwater acoustic channel, and significantly improve the accuracy of judging the quality of interference actions.

[0040] To balance prediction bias and sampling variance, the PPO algorithm introduces generalized dominance estimation. First, the time difference residual is defined and calculated: ; In the formula, express Time difference residuals at time points express Multi-objective composite rewards at any given time, including physical perception heuristics. For the output of the value network; Constructing a joint loss function Achieve synchronous gradient backpropagation for the policy network and value network: ; ; ; In the formula, Indicates a value loss item. Indicates the value loss coefficient. express Discounted return value at any moment This represents the entropy regularization term. Represents the entropy regularization coefficient. express Action vector at any given moment.

[0041] Value loss item The mean squared error is used to force the value network to converge to the actual return, and the entropy regularization term is used. By maximizing the entropy of the policy distribution, the near-end policy optimization algorithm is encouraged to fully explore in the early stages of underwater acoustic signal countermeasures. The entropy regularization term, combined with the physical perception heuristic term, enables the jammer to still perform an effective global search for actions such as the launch time before it has acquired the target packet sequence number feedback.

[0042] In S1, based on the three-node underwater acoustic countermeasure physical model, the model parameters are set according to the underwater acoustic communication standard protocol JANUS, including signal transmission parameters, target link parameters, etc. Underwater acoustic countermeasure experiments are carried out based on the physical model to simulate the signal countermeasure process between the jammer and the target transmitter. The simulation parameters and parameter settings used in the underwater acoustic countermeasure experiment are shown in Table 1. Table 1. Comparison of underwater acoustic countermeasures simulation parameters; .

[0043] Figure 4The reward training curves of the optimal policy finding algorithms provided in this invention are compared to reflect the convergence efficiency and convergence effect of algorithms 1 to 6. Algorithms 1 to 5 are existing optimal policy finding algorithms based on deep reinforcement learning. Among them, algorithm 1 is the Policy Gradient (PG) algorithm, a policy-based reinforcement learning algorithm that solves for the optimal policy by directly optimizing the parameters of the policy function and maximizing the expected cumulative reward using gradient ascent. Algorithm 2 is the Deep Q-Network (DQN) algorithm, a value-based reinforcement learning algorithm that fits the state and action value functions through a neural network, uses a target network and experience replay mechanism for stable training, and indirectly learns the optimal policy by maximizing the Q value. Algorithm 3 is the D3QN algorithm, an improved version of the Deep Q-Network. It combines Double DQN (decoupling action selection and value evaluation to reduce overestimation) and Dueling DQN (decomposing the Q value into state value and advantage function), and uses priority experience replay and bidding network structure to improve training efficiency and stability. Algorithm 4 is a hybrid policy and value reinforcement learning algorithm (Advantage Actor-Critic, A2C). It outputs the action probability distribution through the Actor network, while the Critic network estimates the state value and calculates the advantage function. It utilizes the policy gradient and value function for collaborative optimization, reducing variance and improving training efficiency. PPO is a policy-based reinforcement learning algorithm that limits the difference between new and old policies by pruning the objective function, preventing performance crashes due to excessive update magnitude, and improving sample efficiency while ensuring training stability. Algorithm 5 is a proximal policy optimization (PPO) algorithm, a policy-based reinforcement learning algorithm that limits the magnitude of policy updates, maximizing the expected cumulative reward using gradient ascent while ensuring training stability, thereby solving for the optimal policy. Algorithm 6 is the optimal policy algorithm based on deep reinforcement learning provided in this invention. It introduces a physics-aware heuristic term on top of the proximal policy optimization algorithm, solving the problem of reward sparsity during policy training and optimization and accelerating the convergence speed of the optimal policy algorithm during training.

[0044] analyze Figure 4 The average reward training curves of each algorithm are shown. The number of training steps required for each optimal policy algorithm to reach steady-state convergence is statistically recorded and is equivalent to the number of convergence steps. Based on the comparison of the number of convergence steps, the convergence efficiency gap between the proximal policy optimization algorithm provided by this invention and the five existing optimal policy algorithms based on deep reinforcement learning, namely Algorithm 1 to Algorithm 5, is analyzed. The convergence efficiency comparison results are shown in Table 2.

[0045] Table 2. Comparison of convergence efficiency results; .

[0046] As shown in Table 2 and Figure 4 As shown, this algorithm utilizes the policy pruning mechanism of PPO to limit the policy update amplitude. It constructs a multi-objective composite reward function coupling communication interference reward, energy efficiency cost penalty, retransmission cost benefit, and physical perception heuristic. Generalized advantage estimation is introduced to balance the prediction bias and sampling variance of the near-end policy optimization algorithm, ensuring that interference decisions iterate smoothly within the confidence domain under the time delay caused by a sound speed of 1500 m / s. Compared to existing PG, DNQ, and A2C algorithms, the PPO algorithm provided by this invention improves convergence efficiency by 59%-92%, and exhibits extremely high stability after convergence. Compared to the ordinary PPO algorithm, the improved PPO algorithm of this invention improves convergence efficiency by 35.3%, and can provide continuous gradient guidance through the potential energy field, solving the "blind search" problem of the ordinary PPO algorithm.

[0047] Figure 1 The overall technical roadmap provided for this invention, such as Figure 1 As shown, an intrusion scenario is first constructed, including a target transmitter and a target receiver. The link between the target transmitter and the target receiver is the target link. An adaptive jammer is deployed for the target link. A three-node underwater acoustic countermeasure physical model is constructed, consisting of the target transmitter (Alice), the target receiver (Bob), and the jammer (Jammer). The model parameters are set according to the underwater acoustic communication standard protocol JANUS. Based on the underwater acoustic countermeasure physical model, the signal countermeasure process between the jamming signal and the target signal is simulated. Based on state awareness, the countermeasure process between the jammer and the intrusion target is modeled as a Markov decision process. A multi-dimensional perception state space is defined, including the equivalent signal-to-interference-plus-noise ratio, time-domain overlap deviation, historical delay jitter variance, communication interference feedback flag, and the remaining capability state of the target transmitter. Then, a near-end optimal policy algorithm based on deep reinforcement learning is used for decision generation optimization, generating an action vector containing interference power, interference duration, and interference transmission time. The jammer interferes with the target signal based on the action vector, detects and records the sequence number of the data packet. If the sequence number of the detected data packet is repeated with the historical record, it is determined that the previous round of interference was successful, and the communication interference feedback flag is set. Set to 1, otherwise communication feedback flag. =0; when When =1, the jammer calculates the energy consumption for target signal retransmission based on the data packet length and signal center power, and estimates the remaining capacity status of the target transmitter. ;when When =0, the policy is optimized and updated iteratively based on the optimal policy algorithm of near-end deep reinforcement learning.

[0048] Figure 2The interference decision-making process diagram provided by this invention in an underwater acoustic countermeasure environment is as follows: Figure 2 As shown, in an underwater acoustic communication environment, the target transmitter and receiver transmit data packets and return the packet sequence. The jammer obtains the multi-dimensional state vector at the current moment by listening to the data packets. Constructing a system that includes temporal overlap bias Delay jitter variance Communication interference feedback indicator The equivalent signal-to-interference-plus-noise ratio at the target receiver The remaining capacity status of the target transmitter state space , state space As input to the underwater acoustic countermeasures jamming decision model, the probability distribution of the state vector mapping to the action vector is calculated through a policy network, and the value of the state vector is estimated based on the reward function through a value network. The policy network and value network are iterated using a policy pruning mechanism combined with generalized advantage estimation and a near-end policy optimization algorithm until the network converges, obtaining the optimal jamming decision. An action vector containing the jamming power level, jamming time bias, and jamming duration is generated to perform jamming actions on the target receiver. In some observable non-cooperative countermeasure scenarios, each component of the state space is estimated and quantified by the jammer using local observation data combined with a physical model.

[0049] Figure 3 The PPO algorithm design diagram provided by this invention is as follows: Figure 3 As shown, in an underwater acoustic communication environment, the target transmitter and receiver transmit data packets and return the packet sequence. The jammer outputs the probability of jamming action based on the current state of the underwater acoustic communication environment through the policy network and executes jamming. It obtains the environmental state after the action is executed, calculates the composite reward by combining the current environmental state and the jamming action, and stores the current environmental state, the current jamming action, the reward function, and the environmental state after the action in the experience buffer pool. Then, it extracts experience from the pool, evaluates the state value through the value network, and updates the parameters of the policy network and value network through entropy regularization and optimizer. This process is repeated continuously to optimize the jamming strategy.

[0050] In S3, the core parameters of the proximal policy optimization algorithm are configured as follows: discount factor of 0.95, smoothing factor of 0.95, and pruning hyperparameter of 0.2. Training uses the Adam optimizer with a learning rate of [missing value]. .

[0051] The action state is a joint decision triplet. The interference power level in the action space set adopts adaptive power adjustment logic. The interference duration is dynamically adjusted based on the time delay jitter variance. When the jammer detects increased non-stationarity in the environment, it adaptively increases the interference duration to provide temporal coverage redundancy, ensuring that the interference signal can compensate for the prediction time-domain overlap deviation caused by random jitter. The transmission time offset adopts prior compensation logic. The selectable values ​​for the interference power level are [4W, 16W, 32W, 64W], the selectable values ​​for the interference duration are [0.1s, 0.2s, 0.4s], and the selectable range for the transmission time offset is 0 to 1.2 seconds with a step size of 0.1 seconds.

[0052] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A decision-making method for underwater acoustic interference based on deep reinforcement learning, characterized in that, include: S1. Construct a physical model for underwater acoustic countermeasures to simulate the underwater acoustic signal countermeasure process; S2. Model the underwater acoustic signal countermeasure process as a Markov decision process, obtain and quantify the state vector of the underwater acoustic countermeasure physical model, and construct the state space; S3. Construct an underwater acoustic anti-interference decision model based on the near-end policy optimization algorithm, using the state vector as the model input, and using the near-end policy optimization algorithm to generate action vectors and construct the action space; S4. Construct a reward function, coupling communication interference reward term, energy efficiency cost penalty term, retransmission cost benefit term, and physical perception heuristic term; S5. Initialize the policy network and value network. Calculate the probability distribution of the state vector mapping to the action vector through the policy network, and estimate the value of the state vector based on the reward function through the value network. S6. Iterate the policy network and value network using the policy pruning mechanism of the generalized advantage estimation combined with the proximal policy optimization algorithm until the network converges and the optimal disturbance decision is obtained.

2. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 1, characterized in that, In S1, the underwater acoustic countermeasures physical model consists of a target transmitter, a target receiver, and a jammer. The signal transmitted by the target transmitter is the target signal, the signal transmission link between the target transmitter and the target receiver is the target link, the signal transmitted by the jammer is the jamming signal, and the signal transmission link between the target receiver and the jammer is the jamming link. In accordance with the JANUS underwater acoustic communication standard protocol, the underwater acoustic countermeasures simulation parameters are set in the physical model of underwater acoustic countermeasures, and the target signal transmission distance is defined as the straight-line distance between the target transmitter and the target receiver. The transmission distance of the jamming signal is the straight-line distance between the target receiver and the jammer. The jammer transmits an acoustic signal at the same frequency as the target transmitter, with a center frequency of [missing value]. The data packet length is The synchronization preamble length is .

3. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 2, characterized in that, The jammer adopts a responsive jamming mode, which acquires multi-dimensional state vectors by listening and constructs a state space. The state space includes the equivalent signal-to-interference-plus-noise ratio, time-domain overlap deviation, time delay jitter variance, communication feedback identifier, and the remaining capability status of the target transmitter. Based on underwater acoustic propagation theory, a path propagation loss calculation model is constructed to calculate the path loss. : ； ； In the formula, Represents the logarithm to base 10. Represents the geometric diffusion factor. Indicates the signal transmission distance. This represents the frequency absorption coefficient, which is calculated using the Thorp empirical model. Based on the strength of the target signal measured by the jammer, and combined with the path propagation loss calculation model, the equivalent signal-to-interference-plus-noise ratio at the target receiver is calculated. : ； In the formula, The target's estimated transmit power is obtained by the jammer through reverse calculation based on the signal strength of the passively monitored signal, and is equivalent to the target's nominal transmit power calculated by the jammer through reverse calculation based on the passively monitored signal strength. This indicates the actual transmission power of the jammer. This represents the estimated ambient background noise measured by the interference device.

4. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 3, characterized in that, A random process is introduced to simulate time delay jitter in a non-stationary environment caused by local ocean currents, internal waves, and nodal micro-displacements, to determine the moment when the jammer detects the target signal's synchronization preamble. Using the timeline anchor point, extract the actual transmission timestamp of the jammer's jamming signal. Calculate the time-domain overlap deviation : ； In the formula, Indicates the physical delay of the interfering link. Indicates geometric correction amount; Based on continuous monitoring by the jammer, the arrival times of each synchronization preamble detected during the monitoring process are statistically recorded to construct a sequence of continuously intercepted signal arrival times. Based on the time intervals between these times, a sliding window is used to calculate the variance of the time delay jitter. : ； In the formula, Indicates the length of the sliding window. This represents the total number of times in the sequence of continuously intercepted signal arrival times. Indicates the first A time interval, This represents the average time interval.

5. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 4, characterized in that, The target link operates using the Automatic Repeat Request (ARQ) protocol. Based on the ARQ protocol, the jammer identifies and records the packet sequence of the target signal it has been monitoring. If the packet sequence is found to be identical to the historical record, the jammer determines that the previous round of interference was successful and sets a communication interference feedback flag. Set to 1, otherwise set the communication feedback flag. Set to 0; when When =1, the jammer calculates the energy consumption for target signal retransmission based on the data packet length and signal center power, and estimates the remaining capacity status of the target transmitter. .

6. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 5, characterized in that, In S3, the action space is a joint decision triplet, including the interference power level, interference duration, and transmission time offset. The interference power level adopts adaptive power adjustment logic, and the interference duration is based on the delay jitter variance. Dynamic adjustment: When the jammer detects increased non-stationarity in the environment, it adaptively increases the jamming duration to provide temporal coverage redundancy, ensuring that the jamming signal can compensate for the predicted temporal overlap deviation caused by random jitter. The transmission time offset adopts prior compensation logic.

7. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 6, characterized in that, Construct a multi-objective composite reward function that couples a communication interference reward term, an energy efficiency cost penalty term, a retransmission cost benefit term, and a physical sensing heuristic term: ； ； ； In the formula, Indicates a reward for communication interference; This represents the cost-benefit item for retransmission. This indicates a penalty item for energy efficiency costs. , , These represent the weights of the communication interference reward, retransmission cost benefit, and energy efficiency cost penalty, respectively. Represents physical perception heuristics, Indicates the heuristic intensity coefficient. This represents the width of the potential energy field.

8. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 7, characterized in that, In S5, a parameterized mapping model based on a deep neural network is constructed, utilizing the policy network parameters. Define the probability distribution: ； In the formula, The state vector is represented as When choosing the first The probability of an action. Indicates the first Each action vector Indicates the policy network output for the first The unnormalized log probability of the output of each action. This represents the total dimension of the action space; Utilizing value network parameters Define the value function, for Multidimensional state vector at time step Valuation is based on the expected cumulative return after discount: ； In the formula, express The multidimensional state vector at time step. The state vector is represented as Time-state value, Represents the expectation operator. Indicates the discount factor. express Multi-objective composite rewards at any time. Indicates the summation index.

9. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 8, characterized in that, A pruning objective function is used to impose a hard constraint on the update step size of the policy network: ； ； ； In the formula, This represents the loss term of the pruning strategy. This represents the ratio of the probabilities of the new and old strategies. The state vector in the new strategy is... When selecting the action vector as The probability, The state vector in the old policy is When selecting the action vector as The probability, This represents the estimated value of the advantage function, when... limit The expansion, when limit shrinkage, Represents the smoothing factor. This represents the clipping hyperparameter. This represents the clipping function. express The estimated value of the empirical expectation at time t. express The time difference residual at any given moment.

10. The underwater acoustic interference decision-making method based on deep reinforcement learning according to claim 9, characterized in that, Introducing the prediction bias and sampling variance of the generalized advantage estimation balanced proximal policy optimization algorithm, the time difference residual is calculated: ； In the formula, express Time difference residuals at time points express Multi-objective composite rewards at any given time; Constructing a joint loss function Achieve synchronous gradient backpropagation for the policy network and value network: ； ； ； In the formula, Indicates a value loss item. Indicates the value loss coefficient. express Discounted return value at any moment This represents the entropy regularization term. Represents the entropy regularization coefficient. express Action vector at any given moment.

Images