Intelligent radar and jammer dynamic waveform gaming method based on curiosity driving

Abstract

The invention provides an intelligent radar and jammer dynamic waveform gaming method based on curiosity driving, and belongs to the field of intelligent radar and intelligent jammer waveform gaming. The problems that the intelligent agent is difficult to effectively learn from limited environment feedback due to lack of a waveform-level dynamic game mechanism close to an actual electronic warfare and a sparse reward problem, and the ability imbalance of the intelligent agent is easy to occur in a training process are solved. The method comprises the following steps: constructing a confrontation environment model, namely modeling a dynamic game of a radar and a jammer into a partially observable Markov decision process, and defining a state space, an observation space, an action space and a reward function; designing a complex field feature extraction network; establishing a decision-evaluation network based on an MADDPG framework; a curiosity exploration strategy is introduced; implementing an opponent agent sampling strategy; and alternately training the radar and jammer agents. The device is mainly used in the electronic warfare field.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent radar and intelligent jammer waveform game, and particularly relates to a method for dynamic waveform game between intelligent radar and jammer based on curiosity driving. BACKGROUND

[0002] The electronic warfare field has become a key field for obtaining tactical advantages, in which radar systems, as the core equipment for target detection and tracking, form a dynamic confrontation relationship with self-defense jammers. This confrontation is essentially a complex game process: the radar needs to maintain detection capability in the jamming environment, while the jammer tries to suppress the radar signal to protect its own target. In recent years, artificial intelligence technology, especially reinforcement learning methods, has been widely used to improve the situation awareness and autonomous decision-making capability of cognitive radar. For example, Li Kang's team of Xi'an University of Electronic Science and Technology uses Q-learning algorithm to optimize radar transmit power allocation, and later combines deep Q network (DQN) to propose an anti-jamming frequency hopping strategy, which realizes adaptive decision-making in a jamming environment by inputting radar transmission behavior and echo signal signal-to-noise ratio as state.

[0003] However, the existing research has limitations: first, most schemes only consider the scenario of intelligent radar against non-intelligent jammers, assuming that the jammer behavior follows a predefined fixed model (such as Markov chain). This setting causes the radar to actually learn a response scheme for a static jamming strategy, which cannot adapt to intelligent jammers with cognitive ability in real battlefield. With the application of machine learning technology in jamming strategy design, jammers can dynamically learn to select jamming frequency, power allocation mode or working mode, posing a significant threat to radar detection. Second, in the research of double-agent confrontation, although some scholars have tried to introduce multi-agent reinforcement learning (MARL) into the radar-jammer game system (such as the research on sub-pulse frequency agile radar against intelligent main lobe jammer based on game theory), these methods still face core challenges: (1) the environment modeling stays at the radar signal feature level, lacking a waveform-level dynamic game mechanism close to real combat electronic warfare; (2) the sparse reward problem makes it difficult for agents to effectively learn from limited environmental feedback; (3) the imbalance of agent capabilities (such as "stronger continuously suppressing the weaker") easily occurs during training, hindering the model from converging to the optimal strategy. SUMMARY

[0004] Therefore, the present application aims to provide a method for dynamic waveform game between intelligent radar and jammer based on curiosity driving, to solve the problems of lack of waveform-level dynamic game mechanism close to real combat electronic warfare, difficulty for agents to effectively learn from limited environmental feedback due to sparse reward problem, and imbalance of agent capabilities during training.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: A curiosity-driven dynamic waveform game method between an intelligent radar and a jammer, the method comprising: Constructing a confrontation environment model, including: modeling the dynamic game between radar and jammer as a partially observable Markov decision process, defining the state space, observation space, action space, and reward function; Design a complex domain feature extraction network; Establish a decision-evaluation network based on the MADDPG framework, including: a decision network outputs actions based on the private observations of the agents; a centralized evaluation network takes the observations and actions of all agents as input and outputs an action-value function; Introducing curiosity exploration strategies; Implementing adversary agent sampling strategies; Alternately train radar and jammer agents.

[0006] Furthermore, a preferred embodiment is proposed, wherein the complex domain feature extraction network includes: Split the linear frequency modulation signal emitted by the radar into real and imaginary parts in the complex domain; Three layers of one-dimensional convolution operations are performed on the real part and the imaginary part respectively. The number of channels of the three convolution layers is 32, 64, and 128 respectively, and the convolution kernel size is 9; the maximum pooling layer kernel size is 2 and the stride is 2; Generate 1×128 feature vectors corresponding to the real part or imaginary part respectively through adaptive average pooling; The real and imaginary features are concatenated to form a 256-dimensional fusion vector, which is then output through a fully connected layer as the environmental perception feature.

[0007] Furthermore, a preferred method is proposed, wherein the establishment of a decision-evaluation network based on the MADDPG framework further includes adopting a gradient update strategy:

[0008]

[0009] in: Representing an agent Experience replay buffer; Represents a buffer Observations and action transfers stored in ; Representation Strategy For its parameters gradient; Represents a centralized action-value function Action gradient.

[0010] Furthermore, a preferred method is proposed, wherein the curiosity strategy includes: Build the target network And a prediction network , both have the same structure; According to the observation of the next moment Calculate curiosity reward : .

[0011] Further, an optimal way is also proposed, the target network and the prediction network of the curiosity exploration strategy are both composed of convolutional neural network and fully connected layer, and the network weight is frozen in the training process.

[0012] Further, an optimal way is also proposed, the implementation of the opponent agent sampling strategy includes: Establish a strategy cache pool for radar and jammer respectively, and store different intelligent level models; According to the dynamic sampling probability Select the opponent model for training:

[0013] Among them, The average detection probability of the agent, j is the number of models in the strategy cache pool.

[0014] Further, an optimal way is also proposed, the alternately training radar and jammer agent includes: When the winning rate of one party is more than 55% for 100 consecutive games, fix its model parameters and store them in the cache pool; Update the model of the party that does not meet the standard until it reaches the same intelligent level.

[0015] Further, an optimal way is also proposed, the training process contains dynamic adjustment of parameters, the parameters include radar waveform parameters and jammer parameters; the radar waveform parameters include linear frequency modulation signal carrier frequency, frequency modulation rate, pulse width; the jammer parameters include narrowband aiming jamming bandwidth, dense false target jamming quantity.

[0016] Based on the same inventive concept, the present application also proposes a computer device, including a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method of the intelligent radar and jammer dynamic waveform game based on curiosity driving according to any one of the above.

[0017] Based on the same inventive concept, the present application also proposes a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is run by the processor to execute the steps of the method of the intelligent radar and jammer dynamic waveform game based on curiosity driving as claimed in any one of the above.

[0018] Compared with the prior art, the present application has the beneficial effects that: In view of the problem that the environment modeling is disconnected with actual combat, a complex domain feature extraction network structure is designed in the present application: by decomposing the linear frequency modulation signal into real and imaginary parts for double-channel processing, three layers of one-dimensional convolution (channel number 32 / 64 / 128, kernel size 9) and adaptive pooling are combined to realize deep mining of the complex domain features of the radar waveform. Figure 2 As shown in the network, the detection probability is still maintained at 80.4% under the random mode of the jammer (compared with the prior art simplified modeling scheme, which is improved by more than 20%), so that the game environment is closer to the real electronic warfare scene.

[0019] In view of the problem that the reward sparsity leads to low learning efficiency, a multi-agent curiosity exploration strategy is designed in the present application: a double-channel architecture of a target network and a prediction network is constructed to generate an intrinsic reward based on the prediction error. This mechanism drives the agent to actively explore unknown states in a sparse reward environment, and the number of training rounds is reduced by 40% compared with the prior art, and the average detection probability is improved to 0.897.

[0020] The present application solves the problem of strategy degradation caused by training imbalance, and proposes a dynamic opponent agent sampling strategy: a hierarchical model pool is constructed based on the detection probability weight, and a hybrid training mechanism of new and old versions is combined. This method makes the radar win rate against the L7 jammer 2.08 times, and completely avoids the phenomenon of "stronger suppressing weaker". Under the same training time, the detection probability of the present application reaches 0.946, which is improved by 8.1 percentage points compared with the optimal comparative algorithm MATD3 (0.875). BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of the present application, are used to provide a further understanding of the present application, and the schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in the present application; Figure 2 A complex domain feature extraction network diagram described in the present application; Figure 3 A curiosity exploration strategy structure diagram described in the present application; Figure 4 A radar complex domain feature extraction network diagram described in the present application under the random mode and the traversal mode of the jammer, wherein training episodes represent the number of training rounds, and supervised learning loss represents the loss function value of supervised learning; Figure 5A detection probability curve diagram of the radar described in the present application in a jammer random mode and a traversal mode, wherein the detection probability represents the detection probability of the radar; Figure 6 A detection probability curve diagram of the radar agent described in the present application in 6 different intelligent level jammer models; Figure 7 A winning rate curve diagram of the initial L1 radar described in the present application in 3 different intelligent level jammer models, wherein the winning rate represents the winning rate of the radar in each game round; Figure 8 A winning rate curve diagram of the initial L3 radar described in the present application in 3 different intelligent level jammer models; Figure 9 A detection probability comparison diagram of the method described in the present application and the classical deep reinforcement learning algorithms MADDPG, MAPPO and MATD3; Figure 10 A winning rate curve diagram of the random opponent sampling strategy described in the present application; Figure 11 A winning rate curve diagram of the opponent agent sampling strategy proposed by the method described in the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other, and the described embodiments are only a part of the embodiments of the present application, but not all the embodiments.

[0023] Embodiment one, the method described in the present embodiment is a curiosity-driven intelligent radar and jammer dynamic waveform game method, the method comprises: Constructing an adversarial environment model, comprising: modeling the dynamic game of the radar and the jammer as a partially observable Markov decision process, defining a state space, an observation space, an action space and a reward function; Designing a complex domain feature extraction network; Establishing a decision-evaluation network based on the MADDPG framework, comprising: a decision network outputs actions according to the private observation of the agent; and a centralized evaluation network takes the observation and action of all agents as input and outputs an action value function; Introducing a curiosity exploration strategy; Implementing an opponent agent sampling strategy; Alternately training the radar and jammer agents.

[0024] In this embodiment, the dynamic game between radar and jammer is modeled as a partially observable Markov decision process (POMDP), which more realistically simulates the actual electronic warfare environment and effectively deals with complex electromagnetic waveform game problems, and meets the actual operational requirements. In traditional reinforcement learning methods, due to the sparsity of environmental feedback, it is difficult for the agent to learn effective strategies through limited feedback. By introducing a curiosity exploration strategy, the agent can actively explore without explicit rewards, thereby compensating for the learning difficulties caused by sparse rewards and promoting the agent to effectively improve its performance during training.

[0025] In multi-agent game, the problem of imbalance between the abilities of agents may occur during training. By designing a decision-evaluation network based on the MADDPG framework, a centralized evaluation network is used to evaluate the behavior of all agents, which can balance the development of the abilities of different agents, so that the radar and jammer agents can co-evolve and avoid imbalance in ability. The alternating training of radar and jammer agents can allow the two agents to continuously optimize their strategies in the adversarial game and achieve higher game results. This approach helps to improve the convergence and efficiency of training and ensures that the agents can better cope with complex electromagnetic countermeasures environment in practical applications. By designing a complex domain feature extraction network, more accurate features can be extracted from the radar and jammer signals, improving the understanding and processing capabilities of the agents for complex signals, providing more accurate inputs for decision-making, and further improving the accuracy of game strategies. Designing an opponent agent sampling strategy can help the agent understand the behavior patterns of the opponent, thereby better predicting the actions of the opponent and responding accordingly, improving the adaptability and response speed of the agent in the adversarial environment.

[0026] Embodiment two, this embodiment is a further limitation of the method of embodiment one, the complex domain feature extraction network comprises: The linear frequency modulation signal transmitted by the radar is split into real and imaginary parts in the complex domain; The real and imaginary parts are respectively subjected to three one-dimensional convolution operations, the channel numbers of the three convolution layers are 32, 64 and 128 respectively, and the convolution kernel size is 9; the kernel size of the max pooling layer is 2 and the step is 2; Generate a 1x128 feature vector corresponding to the real or imaginary part through adaptive average pooling; The real and imaginary part features are spliced to form a 256-dimensional fusion vector, and the environment perception features are output through a fully connected layer.

[0027] The radar transmitted linear frequency modulation signal is essentially a complex signal, in which the real part and the imaginary part carry different information respectively. Splitting the signal into real and imaginary parts and processing them separately can capture more detailed information of the signal, thereby improving the feature extraction capability. This approach can avoid information loss that may be caused by a single processing method and fully utilize all features of the radar signal.

[0028] The three-layer one-dimensional convolution operation is performed on the real part and the imaginary part respectively, and the number of channels of each layer of convolution is gradually increased (32, 64, 128). This hierarchical design enables the network to gradually extract different levels of features in the signal from low to high, enhancing the expression ability of the model. The convolution kernel size is 9, the maximum pooling kernel size is 2 and the step is 2, which effectively reduces the dimension of the features and improves the computational efficiency. The adaptive average pooling can adaptively adjust the pooling output size according to the characteristics of the input signal, converting the real part or the imaginary part into a 1x128 feature vector. This approach not only can handle the variation of input features, but also can flexibly adjust the pooling operation according to the specific input size, improving the robustness and flexibility of feature extraction. By concatenating the real part and the imaginary part into a 256-dimensional fusion vector and outputting the environment perception features through the fully connected layer, this fusion strategy can effectively integrate the two parts of information and improve the final feature expression capability. This approach ensures that the agent can consider the complex characteristics of the signal in the decision-making process and make more accurate responses.

[0029] By improving the feature extraction network, the agent can better understand the environmental changes and quickly adapt to the countermeasures in the dynamic waveform game, thereby improving the countermeasure performance of the radar and the jammer.

[0030] Embodiment three, this embodiment is a further limitation of the method for a dynamic waveform game between an intelligent radar and a jammer based on curiosity driving according to embodiment one, wherein the decision-evaluation network based on the MADDPG framework further comprises updating the strategy through gradients:

[0031]

[0032] Wherein: represents the experience replay buffer of the agent ; represents the observation and action transition stored in the buffer ; represents the gradient of the strategy to its parameters ; represents the gradient of the centralized action value function to the action .

[0033] Embodiment four, the embodiment is one kind based on the further limitation of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in embodiment one, the curiosity strategy includes: The target network is constructed And the prediction network , both structures are same; According to the observation of next time Curiosity reward is calculated : .

[0034] Embodiment five, the embodiment is one kind based on the further limitation of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in embodiment four, the target network and prediction network of the curiosity exploration strategy are all composed of convolutional neural network and fully connected layer, and the network weight is frozen in the training process.

[0035] Embodiment six, the embodiment is one kind based on the further limitation of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in embodiment one, the implementation opponent agent sampling strategy includes: The strategy cache pool is established for radar and jammer respectively, and the model of different intelligent levels is stored; According to dynamic sampling probability Opponent model is selected for training:

[0036] Wherein, It represents the average detection probability of the intelligent agent, and j is the number of model in strategy cache pool.

[0037] Embodiment seven, the embodiment is one kind based on the further limitation of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in embodiment one, the radar and jammer intelligent agent are alternately trained, including: When the winning rate of one party is more than 55% for 100 consecutive games, the model parameters are fixed and stored in cache pool; The model of the party that does not reach the standard is updated until the same intelligent level is reached.

[0038] Embodiment eight, the embodiment is one kind based on the further limitation of the curiosity-driven intelligent radar and jammer dynamic waveform game method described in embodiment one, the parameter dynamic adjustment is contained in the training process, and the parameters include radar waveform parameters and jammer parameters; The radar waveform parameters include linear frequency modulation signal carrier frequency, frequency modulation rate, pulse width; The jammer parameters include narrowband aiming interference bandwidth and dense false target interference quantity.

[0039] Embodiment nine, a computer device, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the method of the embodiment one to eight.

[0040] Embodiment ten, a computer readable storage medium, the computer readable storage medium stores a computer program, when the processor runs the computer program, the processor executes the steps of the method of the embodiment one to eight.

[0041] Embodiment eleven, referring to Figures 1 to 11 This embodiment is to provide a specific embodiment of the method of the embodiment one, and also used to explain the embodiment two to eight, specifically: The confrontation process between the smart radar and the jammer can be regarded as a partially observable Markov decision process, the method of the embodiment can effectively solve the problem of uneven intelligence level in double-agent training, and avoid the phenomenon of "stronger suppressing the weaker". In addition, the curiosity strategy introduced in the embodiment can encourage the agent to actively observe and explore more possible interaction modes, so as to shorten the training time and improve the cumulative reward obtained by the agent.

[0042] In electronic confrontation, the perception of electronic environment and each other's information of the two parties is usually incomplete and inaccurate. Therefore, the confrontation environment in the embodiment is regarded as an extension of the multi-agent partially observable Markov decision process. The game process between the agents can be defined by a set of state sets , which describes all possible states of the agents. At each time , each agent will obtain its own observation value , and select an action according to its random strategy . The next state is determined by the state transition function . Each agent will obtain the corresponding reward according to the reward function , and receive the private observation information related to the current state . The goal of each agent is to maximize its private cumulative reward , where is the discount factor, is the total time step.

[0043] The embodiment proposes a curiosity-driven intelligent radar and jammer dynamic waveform game method. The method combines the agent sampling strategy and the curiosity-driven strategy, and the overall structure is as shown in Figure 1 .

[0044] Suppose that the pulse repetition interval of the linear frequency modulation signal transmitted by the radar in the initial environment is 50 microseconds, and if a sampling rate of 100 MHz is used, a 5000x1 discrete waveform feature will be obtained. The complex domain feature of the discrete waveform is divided into real and imaginary parts for extraction, and the internal network structure is as shown in Figure 2 .

[0045] Each one-dimensional convolution operation doubles the channel number to 32, 64 and 128 in turn, and the convolution kernel size is 9. After the ReLU activation layer, the maximum pooling operation is adopted, and the pooling kernel size is 2 and the step is 2. Finally, the feature with a size of 618x128 is converted into a 1x128 vector through adaptive average pooling. The feature fusion module splices the two 1x128 vectors from the real and imaginary part channels to form a 256x1 vector. After the vector passes through two fully connected layers and a global average pooling layer, it is output as a feature vector to the behavior decision network for perceiving the environment.

[0046] The curiosity-driven intelligent radar and jammer dynamic waveform game method proposed in the embodiment is based on a decision-evaluation architecture, and the basic decision module is consistent with the multi-agent deterministic policy gradient method. The multi-agent deterministic policy gradient method is a multi-agent deep reinforcement learning algorithm based on the "centralized training, distributed execution" paradigm. In the training phase, all agents share a centralized evaluation network, which obtains the state information of all agents and is used to guide the training of the decision network of each agent; in the evaluation phase, the decision network of each agent makes decisions independently. In the embodiment, the strategy of all agents can be represented as , and the parameters of the strategy network are denoted as . The decision network of each agent updates the strategy according to its own observation information. The centralized evaluation network takes the observations and actions of all agents as input, and outputs the value function of the action. The decision network guides the agent to explore the environment, and the evaluation network is used to evaluate the strategy effect of the decision network. Under the random strategy, the policy gradient of the expected return of each agent can be represented as:

[0047]

[0048] wherein: represents an agent 's experience replay buffer; represents the observations and action transitions stored in the buffer ; represents the policy 's parameters ; represents the centralized action-value function 's gradient with respect to the action . This policy gradient is used to increase the selection probability of the value action. The experience replay mechanism breaks the time correlation of the interaction between the intelligent radar and the jammer, thereby improving the stability and efficiency of the training. The evaluation network is updated by minimizing the loss function, thereby updating the parameters to optimize the value function. The loss function can be written as:

[0049] .

[0050] In deep reinforcement learning algorithms, the agent usually performs well in a reward-intensive environment. However, in the intelligent radar and jammer waveform game environment designed in the embodiment, the reward feedback is relatively sparse, and the effective interaction in the positive direction is difficult to explore, making it difficult for the agent to obtain an effective exploration strategy. In order to solve this problem, a curiosity exploration strategy is designed in the embodiment to encourage the agent to observe and explore more possible interactions. The curiosity exploration strategy takes the current observation state of the agent as input to generate a curiosity reward, thereby improving the exploration ability of the agent. The improved reward calculation method effectively optimizes the target exploration process of the agent, enabling the agent to obtain more appropriate rewards and prompting the algorithm to adopt a superior search strategy to guide the agent to complete the exploration task faster and more accurately. The random network distillation method is extended to a multi-agent system as a curiosity strategy module. The structure of the curiosity exploration strategy is shown in Figure 3 .

[0051] The curiosity exploration strategy module is composed of three neural networks: a target network for generating fixed features, a prediction network, and a policy network for prediction and decision-making based on the agent's observation. The target network generates a fixed output based on the agent's observation at the next time ; the prediction network is used to predict the output of the target network; and the policy network determines the action of the agent. The target network and the prediction network have the same structure and are composed of convolutional neural networks and fully connected layers for feature transformation. The output features of the two networks are represented as and The prediction network predicts the output of the target network at time The error generated by the output prediction of the target network is the curiosity reward The calculation formula is as follows:

[0052] The curiosity exploration strategy module generates a larger intrinsic reward for unfamiliar states, thereby driving the agent to minimize the error between the actual state and the predicted state. This mechanism can effectively solve the problem of sparse rewards, enabling the agent to explore the environment more quickly and efficiently, and accelerating the exploration process towards the target.

[0053] In the radar and jammer confrontation game environment, the agents are trained simultaneously in pairs. In one scenario, radars and jammers of different intelligence levels may be trained together. For example, an L1 level radar may be in a game against an L3 level jammer. However, the strategies adopted by different level agents in training have a significant impact on the rewards in the training process. In a competitive environment, previous studies have shown that if an agent always trains against the latest version of the opponent, it will lead to an imbalance in knowledge acquisition: one agent masters a more efficient strategy at the beginning of training, while the other may not be able to effectively improve due to continuous failure, ultimately failing to catch up with the opponent's ability. Therefore, in order to balance the learning ability of both agents, the present embodiment proposes an opponent sampling strategy that combines random old versions and high-level new versions of the opponent sample for mixed training. It should be noted that the strategy learned by the agent should have the ability to defeat the old version of the opponent, which is referred to as "lifelong learning" in the field of reinforcement learning.

[0054] Different intelligence levels of radar and jammer strategies need to be accurately classified and stored in the opponent strategy cache pool for sampling during training. AlphaGo Zero uses a self-play method in agent training. In order to evaluate the exploration and confrontation ability of the model, AlphaGo Zero uses the game score as the basis for reward: if the win rate of the new model is higher than 55%, the model will be selected as the current optimal model and used for subsequent training to ensure the quality of the training data.

[0055] In the confrontation environment of intelligent radars and jammers, the detection probability is used to measure the confrontation ability of both agents. The method proposed by us contains two opposing competitive agents, and each is provided with an independent agent buffer pool. To achieve an effective opponent selection mechanism, the strategy introduces a dynamic sampling probability for selecting opponents with different intelligence levels from the buffer pool for training. The dynamic sampling probability can be represented as:

[0056] wherein, represents the average detection probability of the intelligent agent.

[0057] The dynamic sampling strategy can avoid excessive dependence on high intelligent opponents and forgetting of old knowledge. The strategy preferentially selects weaker opponents for confrontation in the early stage of training and gradually turns to confrontation with stronger opponents. Meanwhile, the strategy introduces a random factor to retain a certain proportion of random sampling, thereby preventing the problem of insufficient opponent diversity.

[0058] In order to verify the effect of the method proposed in the embodiment, a simulation experiment is carried out by using the method proposed in the embodiment. The simulation experiment is realized by using Python, and the neural network and the deep reinforcement learning algorithm are developed based on the Pytorch framework. The GPU of the computer used in the experiment is NVIDIA RTX3090.

[0059] In the experiment, three types of jamming modes and two radar waveforms are designed. The related parameters are shown in Table 1.

[0060] Table 1. Parameters used in training the proposed method

[0061] It should be noted that different carrier frequencies will cause changes in radar cross section (RCS). Since the carrier frequency of the radar in the experiment varies in a large range, in order to reduce the calculation cost, the RCS is fixed as 3m². The hyperparameters in the training process are listed in Table 2.

[0062] The parameter settings of the deep reinforcement learning algorithm are shown in Table 3. The transition replay buffer of the radar and the jammer is 10 4 , and the replay buffer of the agent is 7.

[0063] Table 2. Hyperparameter settings in agent simulation

[0064] Table 3. Hyperparameter settings in deep reinforcement learning algorithm

[0065] In this experiment, the network parameters of the radar and the jammer are randomly initialized. The proposed method does not activate the curiosity exploration strategy module and the opponent agent sampling strategy module. The selectable parameters of the radar and the jammer are completely open to ensure the diversity of the confrontation space. It should be noted that in this environment, the radar has learnable parameters, while the parameters of the jammer are fixed.

[0066] In the experiment, the decision-making strategy of the jammer is divided into two types: random mode and traversal mode. We test the radar and jammer in the designed environment. As shown in Figure 4 , the loss value of the radar complex domain feature extraction network shows a downward trend in both random mode and traversal mode, indicating that the network can effectively extract feature parameters under different jamming types. Figure 5 It is shown that when the jammer is in random mode, the convergence speed of the network is lower than that in traversal mode, which indicates that when the jammer parameters change randomly, the interaction samples of the radar cannot efficiently cover all jamming behaviors, so the efficiency of the learning process is lower than that in traversal mode.

[0067] As shown in Figure 5 , when the jammer adopts the above two interference modes, the detection probability curve of the radar shows an upward trend. When the jammer is in traversal mode, the detection probability of the radar can reach 83.7% after 20,000 training times; when the jammer is in random mode, the detection probability of the radar can reach 80.4%.

[0068] In this experiment, the curiosity exploration strategy module and the opponent agent sampling module are activated. The selectable parameters of the radar and the jammer are open. During the training process, the detection probability of the radar and the jammer is evaluated independently. When the winning rate of one party is continuously higher than 55% in 100 consecutive games, the model parameters of that party will be fixed and saved. The saved model will be named as a certain level of agent, such as L1 radar or L2 jammer, etc. Assuming that in the initial game, the radar party first reaches a 55% win rate, the radar model will be saved and named as L1 radar, at this time the parameters of L1 radar are fixed and no longer updated. The network parameters of the jammer are trained unilaterally to improve the ability of the jammer until the jammer also obtains an L1 level model. Through the cross-training process, L1 to L7 level radar and jammer models are obtained.

[0069] After obtaining L1 to L7 level radar and jammer models, their abilities are tested in the experiment. As shown in Table 4, when the winning rates of the radar and the jammer are the same, i.e. the number of victories of the radar and the jammer is equal in a game, the value in the table is set to 1; when the number of victories of the radar is more than that of the jammer, the value in the table is greater than 1. The table shows the ability performance of different level radar and jammer models in the confrontation game. It can be seen that the radar confrontation ability obtained by this method is continuously improved, avoiding falling into local optimum under the condition of no control strategy.

[0070] Table 4 Relative competitive ability of different intelligent level radar and jammer

[0071] Further experiments are designed to demonstrate the learning ability of the radar agent, where the radar model parameters are not fixed and are continuously trained. The initial parameters are randomized, and the curiosity strategy module and adversary sampling module are activated. The previously obtained L1 to L7 jammer models are switched at different training stages, respectively, at 20,000, 40,000, …, 120,000 rounds. Figure 6 The radar detection probability curve under different jammer models is shown. In the case of no training control strategy, the final result shows that when the radar detects the L7 jammer, the detection probability reaches 50%.

[0072] In addition, additional test experiments are designed in this embodiment to verify the learning ability of the radar. Different radar models are selected, and the model parameters are not fixed. The experiments switch different levels of higher jammers at different stages to verify the final ability that the radar can achieve at different training stages.

[0073] As shown in Figure 7 , the L1 jammer is selected in the initial stage, and the L1, L3, and L5 jammers are switched at the 20,000th, 60,000th, and 100,000th rounds. The winning rate of the radar is always higher than that of the jammer during the continuous training process.

[0074] As shown in Figure 8 , the L3 jammer is selected in the initial stage, and the L3, L5, and L7 jammers are switched at the 40,000th, 80,000th, and 120,000th rounds. The winning rate of the radar is higher than that of the jammer. At the end of the training, the winning rate of the radar is 2.08 times that of the L7 jammer.

[0075] The method proposed in the present application is compared with the classical deep reinforcement learning algorithms MADDPG, MAPPO, and MATD3. The average detection probability of the radar of the four algorithms is shown in Figure 9 . The horizontal axis represents the number of training rounds, and the vertical axis represents the average detection probability of the radar in each round.

[0076] Compared with other algorithms, the detection probability curve of MADDPG performs poorly and grows slowly. Although its curve shows an overall upward trend, the detection probability hovers around 0.588 until the end of training. This is because in a high-dimensional state space, the agent is prone to fall into a local optimum, making it difficult to effectively explore and learn the optimal strategy. MADDPG lacks additional exploration driving force, and its robustness and generalization ability are weak, with a lower detection probability than other algorithms, requiring more training time and samples to achieve ideal convergence results.

[0077] The MAPPO algorithm performs poorly in the early stage, but shows a clear upward trend after about 7500 rounds and finally converges. The MAPPO algorithm has good convergence and training stability in the multi-agent environment. After the agent finds an effective strategy, it can gradually stabilize and improve the reward. However, in the later stage of training, the agent may become too conservative and adopt a fixed strategy, which cannot adapt to environmental changes. The decrease in exploration intensity will lead to a decrease in reward value.

[0078] The MATD3 algorithm shows a significant decrease in detection probability in the early 2500 rounds of training, and then starts to rise continuously. Compared with other algorithms, it performs poorly in the early stage of training, but as the number of training rounds increases, the strategy is constantly improved, showing strong late training ability. MATD3 uses the double discriminator mechanism of TD3 to avoid Q value overestimation, thereby obtaining a more stable exploration strategy.

[0079] The overall detection probability curve of the method proposed in the present application shows a stable upward trend, has the best performance and faster learning speed. The algorithm solves the problem of gradient explosion and gradient disappearance in the network, and can accelerate the convergence speed and find an effective strategy faster. The fluctuation in the training process is small, which improves the stability of the training. The curiosity exploration strategy module drives the agent to explore the environment in all directions, obtains more comprehensive experience and better strategy. When dealing with complex high-dimensional state space, the proposed method is more active in exploring new environment, and the algorithm has better robustness.

[0080] The maximum detection probability and average detection probability of the four algorithms are shown in Table 5, which also lists the training time required by the four algorithms to complete the same period of training.

[0081] Table 5 Detection probability and training time of the proposed method and three deep reinforcement learning algorithms

[0082] In the environment constructed in the present embodiment, the radar and jammer agents have models with different levels of intelligence. Different levels of opponents encountered during training can have a significant impact on the learning of the agent. In the present embodiment, the influence of two opponent sampling strategies in training is compared. Figure 10 The performance of the random opponent sampling strategy is shown. The win rate of the radar and jammer does not improve significantly, and the abilities of the two are similar. In this case, the radar and jammer are trapped in a local optimal state.

[0083] Figure 11The performance of the proposed opponent agent sampling strategy is demonstrated. The strategy makes both sides more likely to choose the newly generated advanced opponent during training, thereby ensuring the effectiveness and quality of the training interaction data. The sampling strategy also allows the agent to choose the old model of the opponent while ensuring the preservation of old knowledge that retains the ability of the old model. Under this training condition, the radar can continuously obtain reward improvement and avoid falling into the local optimal solution of the opponent.

[0084] The foregoing detailed description of the application has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed. Many modifications and variations are possible in light of this disclosure. The described embodiments were chosen in order to best explain the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application.

Claims

1. A curiosity-driven intelligent radar and jammer dynamic waveform game method, characterized in that: The method comprises: Constructing a confrontation environment model, including: modeling the dynamic game between radar and jammer as a partially observable Markov decision process, defining the state space, observation space, action space, and reward function; Design a complex domain feature extraction network; Establish a decision-evaluation network based on the MADDPG framework, including: a decision network outputs actions based on the private observations of the agents; a centralized evaluation network takes the observations and actions of all agents as input and outputs an action-value function; Introducing curiosity exploration strategies; Implementing adversary agent sampling strategies; Alternately train radar and jammer agents.

2. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The complex domain feature extraction network includes: Split the linear frequency modulation signal emitted by the radar into real and imaginary parts in the complex domain; Three layers of one-dimensional convolution operations are performed on the real part and the imaginary part respectively. The number of channels of the three convolution layers is 32, 64, and 128 respectively, and the convolution kernel size is 9; the maximum pooling layer kernel size is 2 and the stride is 2; Generate 1×128 feature vectors corresponding to the real part or imaginary part respectively through adaptive average pooling; The real and imaginary features are concatenated to form a 256-dimensional fusion vector, which is then output through a fully connected layer as the environmental perception feature.

3. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The establishment of a decision-evaluation network based on the MADDPG framework also includes a gradient update strategy: in: Representing an agent Experience replay buffer; Represents a buffer Observations and action transfers stored in ; Representation Strategy For its parameters gradient; Represents a centralized action-value function Action gradient.

4. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The curiosity strategies include: Build the target network and prediction network , the two have the same structure; According to the next moment observation Calculating curiosity rewards : 。 5. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 4 is characterized in that: The target network and prediction network of the curiosity exploration strategy are both composed of convolutional neural networks and fully connected layers, and the network weights are frozen during the training process.

6. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The implementation of the opponent agent sampling strategy includes: Establish separate strategy cache pools for radar and jammer to store models of different intelligence levels; According to dynamic sampling probability Select the adversary model for training: in, represents the average detection probability of the agent, and j is the number of models in the strategy cache pool.

7. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The alternate training of radar and jammer agents includes: When a team's winning rate exceeds 55% for 100 consecutive rounds, its model parameters are fixed and stored in the cache pool; Update the models of those that do not meet the standards until they reach the same level of intelligence.

8. The curiosity-driven intelligent radar and jammer dynamic waveform game method according to claim 1 is characterized in that: The training process includes dynamic parameter adjustment, and the parameters include radar waveform parameters and jammer parameters; the radar waveform parameters include linear frequency modulation signal carrier frequency, frequency modulation rate, and pulse width; the jammer parameters include narrowband aiming jamming bandwidth and the number of dense false target jammers.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a curiosity-driven intelligent radar and jammer dynamic waveform game method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a curiosity-driven intelligent radar and jammer dynamic waveform game method according to any one of claims 1 to 8.

Images