SAC-based intermittent sampling forwarding interference method, device and computer equipment

Abstract

The application relates to an SAC-based intermittent sampling and forwarding interference method, device and computer equipment. The method comprises the following steps: modeling a process of performing a penetration task on a radar by an interferer into a Markov model; constructing an SAC model; sampling experience data from a replay buffer, training a Q network in the SAC model by using the experience data, generating a target Q value by a target Q network to assist a Q network in updating, obtaining a trained Q network, obtaining a new action under the same state in the experience data according to a current policy network; inputting a new state-action pair corresponding to the state and the new action in the experience data into the trained Q network to obtain a corresponding Q estimation value, optimizing the Q estimation value, guiding the policy network to optimize according to the optimized Q estimation value, and obtaining a trained policy network; and making an interference decision according to the trained policy network. By using the method, a power-efficient interference strategy can be obtained while reducing the detection performance of the radar.

Description

Technical Field

[0001] This application relates to the field of intermittent sampling forwarding jamming technology, and in particular to an intermittent sampling forwarding jamming method, apparatus and computer equipment based on SAC. Background Technology

[0002] Intermittent sample-and-forward jamming (ISRJ) is an electronic countermeasure widely used in airborne self-defense jamming equipment. It generates discrete, narrow pulse trains through short-duration sample-and-forward techniques, achieving forward jamming and covering the carrier's penetration. This type of jamming can create multiple decoys, and the amplitude, location distribution, and other characteristics of the decoys are highly dependent on the jamming parameters.

[0003] The jamming performance of intermittent sampling forwarding jamming heavily depends on the settings of parameters such as sampling period, sampling duty cycle, and jamming power. To achieve a good suppression effect on radar, it is necessary to rationally set the jamming parameters for intermittent sampling. Furthermore, considering the energy limitations of airborne self-defense jamming equipment, it is also necessary to select an appropriate jamming power to obtain a power-efficient jamming strategy. In non-cooperative, dynamic adversarial scenarios, existing traditional jamming decision-making methods, which heavily rely on prior knowledge bases and accurate mathematical models, are prone to failure and struggle to achieve intelligent and adaptive jamming effects. Moreover, due to the multidimensional continuity of the parameter space, these methods converge slowly.

[0004] In recent years, many documents have mentioned intermittent sampling forwarding jamming. However, research on intermittent sampling forwarding jamming mainly focuses on the generation and distribution characteristics of false targets under different types of intermittent sampling forwarding jamming signals and the implementation of jamming systems. There is relatively little research on its impact on radar detection, and there is also a lack of quantitative analysis on the effectiveness of radar suppression jamming.

[0005] Throughout the dynamic penetration process, as the jamming distance changes, the originally set minimum effective jamming power and jamming parameters may not achieve satisfactory jamming effects, requiring real-time adjustment of the jamming parameters. Therefore, there is a jamming decision problem at different distances during the penetration process, and each jamming decision affects subsequent environmental conditions such as the operating status of non-friendly radars and jamming distances. Thus, the jamming decisions at each distance constitute a decision sequence.

[0006] However, traditional interference decision-making algorithms are often only suitable for solving small-scale problems. For combined optimization problems involving continuous variables such as interference power and parameters, the convergence speed of these algorithms drops significantly. Similarly, game theory-based interference decision-making methods struggle to construct a profit matrix in such adversarial environments. Secondly, considering the non-cooperative relationship between the opposing side, the interfering party lacks prior information about the environmental model, i.e., prior information about the non-friendly radar, such as radar operational state transition probabilities, radar system internal parameters, and performance indicators. Therefore, template-matching interference decision-making methods that heavily rely on prior knowledge bases, as well as traditional optimization methods that strictly depend on accurate mathematical models and require prior information about the environmental model, are prone to failure.

[0007] In recent years, reinforcement learning combined with deep neural networks has led to the emergence of many advanced deep reinforcement learning algorithms, such as Deep Q-Network (DQN), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO). These deep RL algorithms have achieved great success in a series of sequential decision problems, such as games, robotics, indoor navigation, and traffic light control. Meanwhile, reinforcement learning has been extensively studied in the application of communication jamming and anti-jamming, providing new insights for radar countermeasures. Compared with existing jamming decision-making methods, reinforcement learning is more suitable for solving the aforementioned sequential decision problems. It is a machine learning technique characterized by an agent learning through continuous interaction and trial and error with the environment, forming an action reward mechanism based on environmental feedback, and continuously adjusting and optimizing its strategy. Theoretically, it does not require much prior knowledge or learning based on labeled data, and it has a strong adaptability to unknown environments. Secondly, the algorithm uses a continuous action space, enabling finer-grained control over jamming power and parameters. Therefore, reinforcement learning algorithms have great potential in solving the problem of optimizing intermittent sampling and forwarding jamming strategies in dynamic and unknown environments.

[0008] Currently, there is limited research on designing radar jamming strategies using the RL method. Most scholars have introduced RL methods into radar jamming decision-making, enabling the jamming system to autonomously learn and determine the optimal jamming strategy for different radar operating modes. However, these studies primarily construct radar state transition diagrams for different radar operating modes and corresponding jamming patterns, focusing on macroscopic modeling of radar jamming and neglecting the fact that different parameter settings under a given jamming pattern can weaken the jamming effect. Building on this, Pan et al. considered the selection of jamming power and solved the joint optimization problem of jamming type selection and jamming power control based on a deep RL algorithm. Liu et al. proposed an adaptive radar jamming strategy optimization method based on dual Q-Learning, establishing a two-level jamming decision framework for multi-functional radars to determine the selection of jamming patterns and parameters. Zhang et al. proposed a jamming decision-making method based on an adaptive heuristic accelerated Q-learning algorithm, adaptively adjusting the jamming type and jamming power according to changes in radar threat level. However, these methods only consider non-coherent noise suppression jamming and do not consider that current airborne self-defense jamming equipment often uses coherent jamming methods such as intermittent sampling and forwarding jamming to obtain signal processing gain, and the corresponding jamming effect of such jamming depends heavily on the setting of jamming parameters. Summary of the Invention

[0009] Therefore, it is necessary to provide a method, apparatus, and computer device for intermittent sampling forwarding interference based on SAC to address the aforementioned technical problems.

[0010] An intermittent sampling forwarding interference method based on SAC, the method comprising:

[0011] The process of a jammer performing a penetration mission against a radar is modeled as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment.

[0012] A SAC model is constructed, comprising a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtaining interaction data between the jammer and the radar. This interaction data is then placed into a replay buffer as empirical data. The empirical data includes the current state, actions, reward value, and next state.

[0013] Empirical data is sampled from the playback buffer, and the Q network is trained using the empirical data. The target Q network generates a target Q value to assist the Q network in updating, thus obtaining the trained Q network.

[0014] Based on the current policy network, new actions are obtained in the same state in the empirical data. Based on the state in the empirical data and the new actions, new state-action pairs are obtained. The new state-action pairs are input into the trained Q network to obtain the corresponding Q estimates. The Q estimates are adjusted based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates. The optimized Q estimates guide the policy network to optimize, resulting in a trained policy network.

[0015] The trained policy network assists the jammer in making jamming decisions during radar penetration missions.

[0016] An intermittent sampling forwarding jamming device based on SAC, the device comprising:

[0017] The problem model construction module is used to model the process of a jammer performing a penetration mission against a radar as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment.

[0018] A solution model building module is used to construct the SAC model. The SAC model includes a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtains the interaction data between the jammer and the radar, and puts the interaction data as empirical data into the playback buffer. The empirical data includes the current state, action, reward value, and next state.

[0019] The Q-network training module is used to sample empirical data from the playback buffer, train the Q-network using the empirical data, and generate a target Q-value to assist the Q-network in updating, thereby obtaining the trained Q-network.

[0020] The policy network training module is used to obtain new actions in the same state in the empirical data based on the current policy network, obtain new state-action pairs based on the state in the empirical data and the new actions, input the new state-action pairs into the trained Q network to obtain the corresponding Q estimates, adjust the Q estimates based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates, and guide the policy network to optimize based on the optimized Q estimates to obtain the trained policy network.

[0021] The jamming decision module is used to assist the jammer in making jamming decisions during radar penetration missions based on the trained policy network.

[0022] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0023] The process of a jammer performing a penetration mission against a radar is modeled as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment.

[0024] A SAC model is constructed, comprising a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtaining interaction data between the jammer and the radar. This interaction data is then placed into a replay buffer as empirical data. The empirical data includes the current state, actions, reward value, and the next state.

[0025] Empirical data is sampled from the playback buffer, and the Q network is trained using the empirical data. The target Q network generates a target Q value to assist the Q network in updating, thus obtaining the trained Q network.

[0026] Based on the current policy network, new actions are obtained in the same state in the empirical data. Based on the state in the empirical data and the new actions, new state-action pairs are obtained. The new state-action pairs are input into the trained Q network to obtain the corresponding Q estimates. The Q estimates are adjusted based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates. The optimized Q estimates guide the policy network to optimize, resulting in a trained policy network.

[0027] The trained policy network assists the jammer in making jamming decisions during radar penetration missions.

[0028] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0029] The process of a jammer performing a penetration mission against a radar is modeled as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment.

[0030] A SAC model is constructed, comprising a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtaining interaction data between the jammer and the radar. This interaction data is then placed into a replay buffer as empirical data. The empirical data includes the current state, actions, reward value, and next state.

[0031] Empirical data is sampled from the playback buffer, and the Q network is trained using the empirical data. The target Q network generates a target Q value to assist the Q network in updating, thus obtaining the trained Q network.

[0032] Based on the current policy network, new actions are obtained in the same state in the empirical data. Based on the state in the empirical data and the new actions, new state-action pairs are obtained. The new state-action pairs are input into the trained Q network to obtain the corresponding Q estimates. The Q estimates are adjusted based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates. The optimized Q estimates guide the policy network to optimize, resulting in a trained policy network.

[0033] The trained policy network assists the jammer in making jamming decisions during radar penetration missions.

[0034] The aforementioned SAC-based intermittent sampling forwarding jamming method, device, and computer equipment, through Markov modeling of the dynamic penetration process of the self-defense jammer, and using the proposed prior-assisted SAC algorithm to solve the jamming decision problem of the self-defense jammer during the dynamic penetration process. First, introducing the SAC algorithm to solve the optimal jamming strategy can effectively solve the problems of unknown environmental model and multi-dimensional continuous jamming parameters, achieving finer-grained control and optimization of jamming parameters. Furthermore, the proposed prior-assisted SAC algorithm introduces certain prior knowledge on the basis of the SAC algorithm to assist and guide the optimization of the policy network, improving the learning speed of the algorithm, while avoiding getting trapped in local optima, which is conducive to improving the reliability of jamming decision and solution efficiency. Specifically, it obtains the jamming effect of the original state-action pair in the replay buffer, adjusts and optimizes the Q estimate of the new state-action pair according to the jamming effect, and uses the optimized Q estimate to guide and assist the agent's decision. This method can obtain a power-efficient jamming strategy while reducing radar detection performance. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating a scenario in one embodiment where a jamming device counters a search and surveillance radar.

[0036] Figure 2 This is a schematic diagram of intermittent sampling forwarding interference in one embodiment;

[0037] Figure 3 This is a flowchart illustrating an intermittent sampling forwarding interference method based on SAC in one embodiment;

[0038] Figure 4 This is a schematic diagram illustrating the interaction process between the jammer and the radar in one embodiment;

[0039] Figure 5 This is a schematic diagram of the SAC model in one embodiment;

[0040] Figure 6 This is a schematic diagram illustrating the training convergence of the SAC algorithm and the prior-assisted SAC algorithm in a non-cooperative scenario in one embodiment.

[0041] Figure 7 This is an example of the selection of interference parameters corresponding to different strategies during the penetration process in one embodiment, where (a) represents the selection of sampling pulse width, (b) represents the selection of sampling duty cycle, (c) represents the selection of sampling period, and (d) represents the selection of interference power utilization rate.

[0042] Figure 8 This is a structural block diagram of an intermittent sampling forwarding jamming device based on SAC in one embodiment;

[0043] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] like Figure 1 As shown, the jammer employs an intermittent sampling and forwarding jamming method. The early warning radar operates in search mode, transmitting a linear frequency modulated signal with relatively fixed parameters. The jammer flies along a predetermined route towards the non-friendly radar. Jamming resources are limited, and the maximum jamming power is [missing information]. The entire counter-penetration process is considered a single jamming round. In one round, the jammer can effectively interfere by optimizing parameters such as sampling pulse width, sampling period, and jamming power in real time. This prevents the target from being detected by non-friendly radars, reduces the effective detection range of non-friendly radars, and maximizes the jammer's effective range, thus minimizing the non-friendly radar's burn-through distance. Conversely, once the jamming fails and the target is detected by a non-friendly radar, the non-friendly radar immediately switches to tracking mode, signifying the end of this jamming round.

[0046] In the presence of self-defense interference, the signal received by a non-friendly radar can be represented as:

[0047] s(t) = x(t) + n(t) + j(t)

[0048] In the formula, x(t) represents the received target echo signal; n(t) represents a signal with a mean of 0 and a variance of σ. 2 Gaussian white noise; j(t) represents the received intermittent sampling forwarding interference signal, j(t) = A j x s (t-T0). A j This represents the amplitude of the interference signal, which is related to the interference power. s (t) refers to the interference signal obtained by intermittently sampling the radar signal, x s (t)=p s (t)·x(t). p s (t) represents a sampling period of T. s The sampling pulse width is τ s The corresponding sampling pulse train signal at that time, T0 represents the delay of the received intermittently sampled interference signal relative to the target echo signal, which typically depends on the sampling pulse width and the jammer's transmit / receive switching time t. d rect(·) represents a rectangular function. A schematic diagram of intermittent sampling forwarding interference is shown below. Figure 2 As shown.

[0049] According to the radar equation, the detection range R t The power P of the target echo signal received by the radar at the target location. rs and interference signal power P rj They are respectively: P rs =P t G t 2 λ 2 σ / ((4π) 3 R t 4 ) and P rj =P j G j G t λ 2 γ j / ((4π) 2 R t 2 In the formula, P t For radar transmit power, P j Given the jammer's transmit power, the gain of the radar antenna's main lobe in the target direction is G. t The jammer antenna gain is G. j , distance from radar antenna Rt The target scattering cross-section (RCS) at point σ is given by λ, where λ is the signal wavelength and γ is the signal wavelength. j This represents the polarization coefficient of the jamming signal on the radar antenna. With a constant radar transmit power, as the target approaches the radar, the power of the target echo received by the radar continuously increases. Similarly, the power of the received jamming signal also continuously increases, but the rate of increase is inconsistent. Therefore, the jammer's transmit power needs to be adjusted in real time according to changes in the jamming distance.

[0050] Modern radar systems typically employ Constant False Alarm Rate (CFAR) technology based on the Newman-Pearson (NP) criterion to detect targets. This technology calculates the detection threshold by estimating the average level of a reference cell. This paper considers the impact of intermittent sampling-forward interference on Cell Averaging CFAR (CA-CFAR) detection. When the radar performs CA-CFAR detection, the filtered peak of the intermittent sampling-forward interference signal enters the reference cell corresponding to the actual target detection cell, raising the detection level and thus suppressing target detection.

[0051] Based on the fundamental principle of intermittent sampling-forwarding jamming, the location and amplitude of multiple false targets formed by the jamming output are related to jamming parameters such as the sampling period, sampling pulse duty cycle, and jamming power. Therefore, by reasonably setting these parameters, most jamming components can fall within the reference cell range, increasing the detection threshold, affecting the detection performance of real targets, and simultaneously improving power utilization efficiency. However, generating a real-time and efficient intermittent sampling-forwarding jamming strategy is not easy.

[0052] In one embodiment, such as Figure 3 As shown, an intermittent sampling forwarding interference method based on SAC is provided, including the following steps:

[0053] Step 302: Model the process of the jammer performing a penetration mission against the radar as a Markov model.

[0054] Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment.

[0055] Treating the jammer as an agent and the non-friendly radar as the environment, the interaction process between the two is as follows: Figure 4 As shown, the entire interference decision-making process in a penetration mission is modeled as a Markov decision process (MDP), which is usually represented by a quadruple {S, A, R, P}, where S is the set of states, A is the set of actions, R is the state transition probability, and P is the reward value.

[0056] The state value represents the jammer agent's observation of the environment. One coherent processing time is used as a time unit. Considering the non-friendly radar transmits a linearly frequency-modulated signal with fixed parameters, only the distance D between the jammer and the radar is used at time t. t Let represent the state value s observed by the jammer at the current moment. t .

[0057] The action value represents the interference parameters selected by the jamming agent. In the current state s... t Below, the jammers jointly select the jamming power P. jt Intermittent sampling period T st Sampling pulse width τ st This corresponds to the action a. t =[P jt ,T st ,τ st ].

[0058] State transition probability P(s) t+1 |s t ,a t ) indicates that a non-friendly radar is performing jamming actions when the jammer takes action. t Then, based on its performance indicators such as test results, from the current state s t Transition to the next state s t+ The probability of 1. Current interference action a. t When effective, the non-friendly radar cannot detect the target, and the jammer continues to advance towards the non-friendly radar, gradually reducing the distance between them. When the jamming is ineffective, the non-friendly radar detects the target and switches to tracking mode, ending the jamming round. Typically, the jammer's knowledge of P is unknown.

[0059] The reward value reflects the characteristics of the task and is crucial for solving the strategy. The goal of solving the intermittent sampling forwarding interference strategy optimization problem is to find the optimal interference parameter 'a' under different states 's'. * The goal is to minimize the penetration distance of non-friendly radars, which is equivalent to maximizing the effective movement distance of the jammer, while attempting to achieve effective jamming with the lowest possible power. Therefore, a reward function is jointly designed from the perspectives of reducing target detection probability, improving the efficiency of jamming resource utilization, and minimizing the penetration distance of non-friendly radars.

[0060] In a collaborative scenario, the reward value at time t is expressed as:

[0061]

[0062] In the formula, P CFARt Defined as the ratio of the power of the reference element to the power of the actual target detection element in CFAR detection at the radar receiver. CFARt The larger the value, the higher the reward, aiming to guide more interference energy into the reference cell, improve the detection threshold, and achieve a better suppression effect on the radar. juset This represents the power utilization rate of the jammer at the current moment, which is the ratio of the current jamming transmission power to the maximum jamming power, and its value ranges from (0,1). Under the premise of effective jamming, the lower the transmission power, the better P... juset The smaller the value of β1, the greater the reward. β1 is a weighting coefficient that keeps the first reward value within a suitable range. t L represents the maximum effective forward range of the jammer. t The larger the value, the smaller the penetration distance of non-friendly radar, and the greater the reward value. t Used to indicate whether the current interference is effective, ξ t =1 indicates that the non-friendly radar has switched to tracking mode, signifying that the current jamming is ineffective, the non-friendly radar can detect the target, and this round of jamming has ended; ξ t =0 indicates that the non-friendly radar is still in the search state, and the current jamming is effective.

[0063] Furthermore, considering that in non-cooperative scenarios, the interfering party cannot obtain internal parameter information of the non-friendly radar system from its perspective, and therefore cannot obtain P... CFARt Value, adjust reward value:

[0064]

[0065] In the formula, β2 is also a weighting coefficient, with the same effect as β1, ξ t The value of this reward can be determined based on the changes in the PRF of the non-friendly radar. Typically, when a radar switches to tracking mode, its corresponding PRF also increases. In this way, a reward value that does not depend on the internal performance indicators of the non-friendly radar is designed from the perspective of the non-cooperative party, which improves the feasibility of the method to some extent.

[0066] Step 304: Construct the SAC model.

[0067] The goal of intelligent jammer optimization is to achieve the optimal strategy π. *Under the guidance of [unclear], the interference sampling parameters and power usage are jointly optimized to minimize the burn-through distance of non-friendly radars while improving power usage efficiency. For the problem presented in this paper, considering the unknown state transition probability P of the jammer to the environment, the Actor-Critic algorithm with a maximum entropy framework, namely the SAC algorithm, is adopted to solve this type of interference decision-making problem. The optimization objective of this algorithm is as follows:

[0068]

[0069] In the formula, the trajectory τ is a sequence of states, actions, and rewards obtained based on the current policy π, γ is the discount factor, α is the temperature hyperparameter, and H(π(·|s t )) indicates that the interference strategy π is in state s t The degree of randomness.

[0070] Compared to traditional reinforcement learning algorithms, this algorithm aims to maximize the cumulative reward of entropy regularization, rather than just the cumulative reward itself. In other words, besides maximizing the cumulative reward, it also aims to make the policy more random, increasing the exploration level of the reinforcement learning algorithm, helping to accelerate subsequent policy learning, and reducing the possibility of the policy getting trapped in poor local optima. This algorithm learns a random policy, allowing for highly random actions during training, thus having greater potential to handle unexpected perturbations during testing. Compared to other offline policy algorithms, the SAC algorithm is more robust and stable.

[0071] The SAC algorithm simultaneously establishes two Q-value functions (Critic), two corresponding objective Q-value functions, and one policy function (Actor). Considering the continuity of states and actions, these functions are approximated by a network. The structure of the SAC model is as follows: Figure 5 As shown, the SAC model includes a Q-network, a target Q-network, and a policy network. The Q-network comprises a first Q-network and a second Q-network, and the target Q-network comprises a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimates from the outputs of the first and second Q-networks. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtaining interaction data between the jammer and the radar. This interaction data is then placed into the playback buffer as empirical data, which includes the current state, actions, reward value, and next state.

[0072] The inputs to the Q-network and the corresponding target Q-network are state-action pairs, and the outputs are their corresponding Q-values. The Q-values ​​for state-action pairs are defined as follows:

[0073] Q(s,a)=Ε τ～π [G(τ)|S0=s,A0=a]

[0074] The Q-value can be simply understood as the value of taking action a in state s based on the current policy π. The Q-value changes accordingly with changes in the policy. The Actor policy network takes the current state value as input and outputs the probability distribution of actions selectable in the current state. The Q-network guides the learning and updating of the evaluation policy network; conversely, the policy network guides the action selection of the Q-network, thus affecting its updates. Therefore, the value network and policy network continuously optimize and learn alternately, eventually converging to the optimal policy. The specific algorithm is as follows:

[0075]

[0076] Step 306: Sample empirical data from the playback buffer, train the Q network using the empirical data, and use the target Q network to generate a target Q value to assist the Q network in updating, thus obtaining the trained Q network.

[0077] The SAC algorithm uses a single-step time-difference method for Q-value updates, and the target Q-value is generated by the target Q-network. The auxiliary Q-network is updated. Using the old parameters of the target Q-network makes training more stable. The loss function of the Q-network is given by the difference between its output Q-value and the target Q-value.

[0078]

[0079] The algorithm employs two Q-networks, selecting the network with the smaller output Q-value each time it's used to mitigate the problem of overestimating the Q-value. The target Q-network slowly approaches the Q-network through soft updates, stabilizing the learning process through this delayed update approach.

[0080]

[0081] Step 308: Obtain new actions in the same state from the empirical data based on the current policy network; obtain new state-action pairs based on the state and new actions in the empirical data; input the new state-action pairs into the trained Q network to obtain the corresponding Q estimates; adjust the Q estimates based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates; guide the policy network to optimize based on the optimized Q estimates to obtain the trained policy network.

[0082] Considering the lack of useful information in reward feedback during non-cooperative scenarios, the number of training iterations required for the algorithm will increase. To improve the learning speed of the SAC algorithm and avoid getting trapped in local optima, an improvement is proposed. A prior-assisted SAC algorithm is proposed by introducing certain prior knowledge for guidance and optimizing the constraint policy network. Based on the state-action pair (s) in the current experience buffer... t ,a tThe corresponding disturbance results are used to optimize and adjust the new state action pair to estimate the value Q(s). t ,a′ t This yields a new estimate Q′(s). t ,a′ t This guides the optimization of the Actor policy network.

[0083] The Q-value optimization adjustments are shown below:

[0084]

[0085] Among them, a t =[P jt ,T st ,τ st ],(s t ,a t ) represents the state-action pair currently sampled from the replay buffer for training the Q-network, while (s t ,a′ t This represents a new state-action pair obtained by reselecting actions under state st based on the current Actor policy network, used for subsequent policy network optimization. The original state-action pair (s) in the replay buffer is then retrieved. t ,a t Based on the interference effect, by comparing the original action a in the playback buffer under the same state... t With subsequent new action a′ t Determine whether the new action is suboptimal, and then adjust and optimize the Q estimate Q(s) of the new state-action pair. t ,a′ t This guides and assists the agent in decision-making, avoiding the exploration of suboptimal strategies and getting trapped in local optima, thus accelerating convergence.

[0086] Step 310: The jammer makes jamming decisions during the radar penetration mission based on the trained strategy network.

[0087] After the jammer performs its jamming based on the current decision, it stores the current empirical data in the replay buffer to further guide the optimization of the Q network.

[0088] In the aforementioned SAC-based intermittent sampling-forward jamming method, a Markov model is used to model the dynamic penetration process of the self-defense jammer. The proposed prior-assisted SAC algorithm is then used to solve the jamming decision problem during this dynamic penetration. First, introducing the SAC algorithm to solve for the optimal jamming strategy effectively addresses the problems of unknown environmental models and multi-dimensional continuous jamming parameters, enabling finer-grained control and optimization of these parameters. Furthermore, the proposed prior-assisted SAC algorithm introduces prior knowledge to assist and guide the optimization of the policy network, improving the algorithm's learning speed and avoiding getting trapped in local optima. This enhances the reliability and efficiency of the jamming decision. Specifically, it obtains the jamming effect of the original state-action pairs in the replay buffer, adjusts and optimizes the Q-estimate of the new state-action pairs based on the jamming effect, and uses the optimized Q-estimate to guide and assist the agent's decision-making. This method can achieve a power-efficient jamming strategy while reducing radar detection performance.

[0089] In one embodiment, adjusting the Q estimate based on whether the original state action in the empirical data is effective against the corresponding interference to obtain the optimized Q estimate includes: determining whether the original state action is effective against the corresponding interference; if the original action is effective, determining whether the new action is suboptimal; if the new action is suboptimal, obtaining the optimized Q estimate based on the difference between the interference power utilization rate of the original state action pair and the new state action pair and the Q estimate; if the original action is ineffective, obtaining the optimized Q estimate based on the interference power utilization rate corresponding to the new action and the Q estimate; obtaining the optimized Q estimate based on the difference between the interference power utilization rate of the original state action pair and the new state action pair and the Q estimate includes: obtaining the optimized Q estimate based on the difference between the interference power utilization rate of the original state action pair and the new state action pair and the Q estimate.

[0090] Q′(s t ,a′ t )=Q(s t ,a′ t )+min((P juset -P′ juset ),0)

[0091] Among them, (s t ,a′ t ) represents the new state action pair, Q(s) t ,a′ t ) represents the Q estimate corresponding to the new state-action pair, Q′(s) t ,a′ t ) represents the optimized Q estimate, P juset P′ represents the utilization rate of the interference power corresponding to the action in the original state. jusetThe new state action corresponds to the interference power utilization rate; if the original action interference fails, the optimized Q estimate is obtained based on the interference power utilization rate and Q estimate corresponding to the new action, including: If the original action interference fails, the optimized Q estimate is obtained based on the interference power utilization rate and Q estimate corresponding to the new action:

[0092]

[0093] Among them, (s t ,a′ t ) represents the new state action pair, Q(s) t ,a′ t ) represents the Q estimate corresponding to the new state-action pair, Q′(s) t ,a′ t P' is the optimized Q estimate. juset β3 represents the interference power utilization rate corresponding to the new state action, and β3 is the optimization coefficient.

[0094] Specifically, when the original action a t If the interference fails, then a new state action pair (s) should be added appropriately. t ,a′ t The Q value of ) is used to optimize the objective in conjunction with the interference strategy, based on the new action a′ t The increase in Q-value is determined by the corresponding interference power utilization rate; the smaller the selected interference power, the greater the increase in Q-value. Considering the randomness of the selection of new actions, a coefficient β3 is introduced to reduce its overall impact on Q-value and prevent it from affecting the agent's exploration of the optimal policy. If the original action a... t The interference was effective. To confirm whether the new action was suboptimal, the original action a was compared. t With new action a′ t Corresponding interference power P jt and P′ jt If P′ jt If it is larger, then a′ is considered to be larger. t This corresponds to the suboptimal action, where Q(s) is determined based on the difference in power utilization between the two. t ,a′ t The degree to which the value decreases ultimately guides the strategy to select fewer suboptimal actions a′. t Conversely, if P jt If it is larger, then it does not apply to Q(s) t ,a′ t The Q value is adjusted to avoid introducing additional errors. Considering that the Q value cannot be negative, the Q value is further processed: Q′(s) t ,a′ t )=max(Q′(s t ,a′ t ),0).

[0095] In one embodiment, the step of determining whether a new action is a suboptimal action includes: determining whether a new action is a suboptimal action based on the relationship between the interference power selected for the original action and the new action; if the interference power selected for the new action is greater than the interference power selected for the original action, then the new action is a suboptimal action; otherwise, the new action is not a suboptimal action; the method further includes: if the new action is not a suboptimal action, then the Q estimate is not adjusted.

[0096] In one embodiment, guiding the optimization of the policy network based on the optimized Q-estimate to obtain the trained policy network includes: constructing the loss function corresponding to the policy network based on the optimized Q-estimate as follows:

[0097]

[0098] Among them, L π (θ) is the loss function corresponding to the policy network, D KL (·) represents the KL divergence, πθ(·|s) t ) represents state s t The interference policy is as follows, where θ is the policy parameter, α is the entropy regularization coefficient, and Q′(s) is the interference policy. t ,a′ t )=max(Q′(s t ,a′ t Z(·) is the normalization constant. The policy network is trained using a loss function to obtain the trained policy network. In this embodiment, the estimated value Q(s) of the new state-action pair is optimized and adjusted. t ,a t ') is Q'(s t ,a t This leads to the optimization of the Actor policy network, ensuring that the policy output conforms to the optimized Q' value distribution.

[0099] In one specific embodiment, a simulation experiment was conducted based on a mission scenario of a penetration jamming aircraft countering a search and surveillance radar, and the results were analyzed. For the jamming aircraft, to avoid frequent decision-making, a jamming decision was made every 32 pulse groups to determine the optimal jamming parameters for the current state. The aim was to reduce the effective detection range of the non-friendly radar, prevent the target from being detected by the radar during flight, and minimize the burn-through distance of the non-friendly radar. To increase the algorithm's robustness to new situations, the initial distance between the target and the non-friendly radar was randomly generated within a certain interval in each run. The specific radar-jamming aircraft operating parameters are shown in Table 1.

[0100] Table 1 Radar-Jitter Operating Parameters

[0101]

[0102]

[0103] The algorithm's parameter settings are shown in Table 2. Both the Critic Q network and the Actor policy network are 4 layers, with 128 nodes in each hidden layer. The input dimension of the Critic Q network depends on the sum of the state and action dimensions, and its output is the Q-estimate for the corresponding input state-action pair, with a dimension of 1. The input dimension of the Actor policy network depends on the state dimension. Since action values ​​are multidimensional continuous variables, the Actor network outputs the probability distribution of action selection under the current input state. Typically, the network output is set to the mean and variance of this distribution. Therefore, the Actor network has two parallel output layers at the end, corresponding to the output mean and variance, respectively, with dimensions depending on the action dimension.

[0104] Table 2 Algorithm Simulation Parameter Settings

[0105]

[0106] The effectiveness of the proposed algorithm is verified through simulation experiments. First, we use average round reward as the evaluation metric to measure network performance. Average round reward refers to the cumulative reward generated throughout the entire process from the initial state to the final state. For non-cooperative scenarios, six random seeds are set to avoid random bias. The convergence performance of the SAC algorithm and the prior-assisted SAC algorithm in non-cooperative scenarios is tested. The test results are as follows: Figure 6 As shown, due to the introduction of certain prior knowledge, the learning speed of the prior-assisted SAC algorithm is significantly improved compared with the SAC algorithm in the first 5000 iterations. At the same time, its final convergence reward value is also higher than the corresponding reward value of the SAC algorithm, which to some extent proves that the method can effectively avoid getting trapped in local optima.

[0107] In one specific embodiment, we compare the interference effects of the interference strategies learned by the algorithm in the different scenarios described above with those of the fixed-parameter interference strategies. Figure 7 The diagram shows the selection of interference parameters for different strategies during penetration, where (a) represents the selection of sampling pulse width, (b) represents the selection of sampling duty cycle, (c) represents the selection of sampling period, and (d) represents the selection of interference power utilization rate. Figure 7The horizontal axis represents the distance the jammer travels towards the radar. An initial value of 0 corresponds to a distance of 0 travels, at which point the jammer and radar are 200 km apart. The point where the curve stops represents the maximum effective travel distance of the jammer under the current strategy. A larger value indicates that the jammer is closer to the non-friendly radar, and the non-friendly radar's penetration distance is smaller. Compared with a fixed strategy, the jamming strategies learned by the three different SAC algorithms all achieve a jamming effect with a maximum jammer travel distance of 176.94 km, corresponding to a non-friendly radar penetration distance of less than 23.06 km. This shows that the strategy learned based on the designed jamming effect evaluation scheme in non-cooperative scenarios can achieve the same jamming effect as the strategy learned in cooperative scenarios, proving the effectiveness of the proposed jamming effect evaluation scheme. While the maximum forward range of fixed jamming strategies is generally less than 176.94 km, corresponding to a non-friendly radar penetration range greater than 23.06 km, only the fixed strategy with a sampling pulse width of 0.5 μs achieves a maximum penetration range of 176.94 km, which is very close to the learned jamming parameters. However, its jamming power is much higher than the jamming power corresponding to the learned jamming strategies. Therefore, compared with the fixed strategy, it can be found that the jamming strategy learned based on SAC has better performance in terms of penetration range and power utilization efficiency, achieving similar or even better jamming effects with the least possible power, proving the feasibility of applying the SAC algorithm to optimize intermittent sampling and forwarding jamming strategies. Furthermore, from the perspective of comparing various SAC algorithms, the prior-assisted SAC algorithm has the best performance in non-cooperative scenarios. The strategy learned by this algorithm can achieve the same jamming effect as other strategies with the minimum jamming power, and the sampling pulse width value is the most stable throughout the entire jamming decision process, proving the superiority of the proposed prior-assisted SAC algorithm.

[0108] It should be understood that, although Figure 3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0109] In one embodiment, such as Figure 8 As shown, an intermittent sampling forwarding jamming device based on SAC is provided, comprising:

[0110] The problem model construction module 802 is used to model the process of the jammer performing a penetration mission against the radar as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the working performance index after the jammer takes a jamming action. The working performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative scenario or the non-cooperative scenario at the current moment.

[0111] The solution model construction module 804 is used to construct the SAC model; the SAC model includes a Q-network, a target Q-network, and a policy network; the Q-network includes a first Q-network and a second Q-network; the target Q-network includes a first target Q-network and a second target Q-network; the output of the Q-network is the smaller value of the Q-estimated values ​​output by the first Q-network and the second Q-network; under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtains the interaction data between the jammer and the radar, and puts the interaction data as empirical data into the playback buffer; the empirical data includes the current state, action, reward value, and next state.

[0112] The Q-network training module 806 is used to sample empirical data from the playback buffer, train the Q-network using the empirical data, and generate a target Q-value to assist the Q-network in updating, thereby obtaining the trained Q-network.

[0113] The policy network training module 808 is used to obtain new actions in the same state in the empirical data based on the current policy network, obtain new state-action pairs based on the state in the empirical data and the new actions, input the new state-action pairs into the trained Q network to obtain the corresponding Q estimates, adjust the Q estimates based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates, and guide the policy network to optimize based on the optimized Q estimates to obtain the trained policy network.

[0114] The jamming decision module 810 is used to assist the jammer in making jamming decisions during the radar penetration mission based on the trained policy network.

[0115] In one embodiment, the loss function corresponding to the policy network is further constructed based on the optimized Q estimate:

[0116]

[0117] Among them, L π(θ) is the loss function corresponding to the policy network, D KL (·) represents the KL divergence, πθ(·|s) t ) represents state s t The interference policy is as follows, where θ is the policy parameter, α is the entropy regularization coefficient, and Q′(s) is the interference policy. t ,a′ t )=max(Q′(s t ,a′ t Z(·) is the normalization constant. The policy network is trained using the loss function to obtain the trained policy network.

[0118] In one embodiment, it is further used to determine whether the interference effect of the original state action is effective. If the original action interference is effective, it is determined whether the new action is a suboptimal action. If the new action is a suboptimal action, an optimized Q estimate is obtained based on the difference between the interference power utilization rate of the original state action pair and the new state action pair and the Q estimate. If the original action interference is ineffective, an optimized Q estimate is obtained based on the interference power utilization rate corresponding to the new action and the Q estimate.

[0119] In one embodiment, an optimized Q estimate is further obtained based on the difference in interference power utilization between the original state action pair and the new state action pair, and the Q estimate:

[0120] Q′(s t ,a′ t )=Q(s t ,a′ t )+min((P juset -P′ juset ),0)

[0121] Among them, (s t ,a′ t ) represents the new state action pair, Q(s) t ,a′ t ) represents the Q estimate corresponding to the new state-action pair, Q′(s) t ,a′ t ) represents the optimized Q estimate, P juset P′ represents the utilization rate of the interference power corresponding to the action in the original state. juset The new state action corresponds to the interference power utilization rate.

[0122] In one embodiment, if the original action interference fails, an optimized Q estimate is obtained based on the interference power utilization rate corresponding to the new action and the Q estimate:

[0123]

[0124] Among them, (s t ,a′ t ) represents the new state action pair, Q(s) t ,a′ t ) represents the Q estimate corresponding to the new state-action pair, Q′(s) t ,a′ t ) represents the optimized Q estimate, -P′ juset β3 represents the interference power utilization rate corresponding to the new state action, and β3 is the optimization coefficient.

[0125] In one embodiment, it is further configured to determine whether the new action is a suboptimal action based on the relationship between the interference power selected for the original action and the new action. If the interference power selected for the new action is greater than the interference power selected for the original action, then the new action is a suboptimal action; otherwise, the new action is not a suboptimal action.

[0126] In one embodiment, the Q estimate is not adjusted if the new action is not a suboptimal action.

[0127] Specific limitations regarding the SAC-based intermittent sampling-forwarding jamming device can be found in the limitations of the SAC-based intermittent sampling-forwarding jamming method described above, and will not be repeated here. Each module in the aforementioned SAC-based intermittent sampling-forwarding jamming device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0128] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a SAC-based intermittent sampling and forwarding interference method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0129] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0131] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for intermittent sampling and forwarding interference based on SAC, characterized in that, The method includes: The process of a jammer performing a penetration mission against a radar is modeled as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The current state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment. A SAC model is constructed, comprising a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtaining interaction data between the jammer and the radar. This interaction data is then placed into a replay buffer as empirical data. The empirical data includes the current state, actions, reward value, and next state. Empirical data is sampled from the playback buffer, and the Q network is trained using the empirical data. The target Q network generates a target Q value to assist the Q network in updating, thus obtaining the trained Q network. Based on the current policy network, new actions are obtained in the same state in the empirical data. Based on the state in the empirical data and the new actions, new state-action pairs are obtained. The new state-action pairs are input into the trained Q network to obtain the corresponding Q estimates. The Q estimates are adjusted based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates. The optimized Q estimates guide the policy network to optimize, resulting in a trained policy network. The trained policy network assists the jammer in making jamming decisions during radar penetration missions. The Q estimate is adjusted based on whether the original state action in the empirical data is effective against the corresponding disturbance, resulting in an optimized Q estimate including: Determine whether the interference effect of the original state action is effective. If the original action is effective, determine whether the new action is suboptimal. If the new action is suboptimal, calculate the difference in interference power utilization between the original state action pair and the new state action pair. Based on the difference and the Q estimate, obtain the optimized Q estimate. If the original action interference fails, the optimized Q estimate is obtained based on the interference power utilization rate corresponding to the new action and the Q estimate.

2. The method according to claim 1, characterized in that, The optimized policy network is guided by the optimized Q-estimate to obtain the trained policy network, which includes: The loss function for the policy network is constructed based on the optimized Q estimate: ， in, The loss function corresponding to the policy network. Let KL divergence be the KL divergence. For state The following interference strategy For strategy parameters, The coefficient of the entropy regularization term, , This is the optimized Q estimate. For the new state action pair, The normalization constant is , For new actions; The policy network is trained using the loss function to obtain the trained policy network.

3. The method according to claim 1, characterized in that, Calculate the difference in interference power utilization between the original state action pair and the new state action pair, and obtain the optimized Q estimate based on the difference and the Q estimate, including: Calculate the difference in interference power utilization between the original state action pair and the new state action pair. Based on this difference and the Q estimate, obtain the optimized Q estimate: in, For the new state action pair, The Q-estimate for the new state action pair. This is the optimized Q estimate. The original state action corresponds to the interference power utilization rate. The new state action corresponds to the interference power utilization rate.

4. The method according to claim 1, characterized in that, If the original action interference fails, the optimized Q estimate value is obtained based on the interference power utilization rate corresponding to the new action and the Q estimate value, including: If the original action interference fails, the optimized Q estimate value is obtained based on the interference power utilization rate corresponding to the new action and the Q estimate value: in, For the new state action pair, The Q-estimate for the new state action pair. This is the optimized Q estimate. The new state action corresponds to the interference power utilization rate. To optimize the coefficients.

5. The method according to claim 1, characterized in that, The steps to determine whether a new action is a suboptimal action include: The new action is determined by the relationship between the interference power selected for the original action and the new action. If the interference power selected for the new action is greater than the interference power selected for the original action, then the new action is a suboptimal action; otherwise, the new action is not a suboptimal action.

6. The method according to claim 1 or 5, characterized in that, The method further includes: If the new action is not a suboptimal action, then the Q estimate is not adjusted.

7. A SAC-based intermittent sampling forwarding jamming device applied to the method described in any one of claims 1-6, characterized in that, The device includes: The problem model construction module is used to model the process of a jammer performing a penetration mission against a radar as a Markov model. Each time step of the Markov model includes the current state, action, state transition probability, and reward value. The current state includes the distance between the jammer and the radar at the current moment. The action includes the jamming parameters selected by the jammer. The jamming parameters include jamming power, intermittent sampling period, and sampling pulse width. The state transition probability includes the probability that the radar will transition from the current state to the next state according to the performance index after the jammer takes a jamming action. The performance index includes whether the jamming is effective. The reward value includes the reward function value corresponding to the cooperative or non-cooperative scenario at the current moment. A solution model building module is used to construct the SAC model. The SAC model includes a Q-network, a target Q-network, and a policy network. The Q-network includes a first Q-network and a second Q-network. The target Q-network includes a first target Q-network and a second target Q-network. The output of the Q-network is the smaller of the Q-estimated values ​​output by the first Q-network and the second Q-network. Under the guidance of the current policy network, the jammer makes jamming decisions during the penetration mission, obtains the interaction data between the jammer and the radar, and puts the interaction data as empirical data into the playback buffer. The empirical data includes the current state, action, reward value, and next state. The Q-network training module is used to sample empirical data from the playback buffer, train the Q-network using the empirical data, and generate a target Q-value to assist the Q-network in updating, thereby obtaining the trained Q-network. The policy network training module is used to obtain new actions in the same state in the empirical data based on the current policy network, obtain new state-action pairs based on the state in the empirical data and the new actions, input the new state-action pairs into the trained Q network to obtain the corresponding Q estimates, adjust the Q estimates based on whether the disturbances corresponding to the original state-action pairs in the empirical data are effective to obtain optimized Q estimates, and guide the policy network to optimize based on the optimized Q estimates to obtain the trained policy network. The jamming decision module is used to assist the jammer in making jamming decisions during radar penetration missions based on the trained policy network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images