A Radar Anti-jamming Strategy Generation Method Based on the Fusion of MARL and Mind Chain Reasoning
By integrating multi-agent reinforcement learning with thought chain reasoning, this method solves the problems of policy non-stationarity and credit allocation in complex environments of traditional radar anti-jamming strategies. It improves radar detection performance and enhances the stability and adaptability of the strategy, thus meeting the dynamic countermeasure requirements of strong and weak interference scenarios.
Patent Information
- Application Number
- CN202610625016.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-26
AI Technical Summary
Traditional radar anti-jamming strategies are difficult to meet the dynamic countermeasure requirements in complex and ever-changing electronic warfare environments. They suffer from problems such as strategy non-stationarity, credit allocation dilemma, and weak strategy generalization ability. Furthermore, the traditional MARL algorithm is difficult to adapt to complex electromagnetic environments.
This paper adopts a method that integrates multi-agent reinforcement learning and thought chain reasoning. By constructing a multi-agent radar anti-jamming environment, local states are acquired and features are extracted. Anti-jamming strategies are generated using interference recognition subnetwork, inducement strategy generation subnetwork and action fusion subnetwork. The strategy training is optimized by combining an enhanced experience replay mechanism, and a comprehensive reward function is constructed to improve the stability and effectiveness of the strategy.
It achieves improved radar detection performance in complex electromagnetic environments, simultaneous improvement of signal-to-interference-plus-noise ratio and interference suppression ratio, ensures target signal fidelity, and accelerates model convergence through explicit decision logic and priority experience playback mechanism, adapting to the dynamic countermeasure requirements of strong and weak interference scenarios.
Smart Images

Figure CN122283620A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of radar electronic countermeasures and intelligent decision-making technology. Specifically, it provides a radar anti-jamming strategy generation method based on the fusion of MARL (Multi-Agent Reinforcement Learning) and thought chain reasoning, which is applicable to radar network collaborative anti-jamming decision-making in complex electromagnetic countermeasures environments. Background Technology
[0002] In modern electronic warfare environments, radar network systems often face complex and ever-changing jamming threats, and traditional anti-jamming strategies are insufficient to meet the dynamic countermeasures required in electronic warfare. Multi-Agent Reinforcement Learning (MARL) technology can study the reinforcement learning branch where multiple agents interact and learn optimal strategies in the same environment. This aligns with the need for autonomous decision-making and collaborative optimization among radar nodes in the network, making it theoretically feasible to use MARL to improve overall anti-jamming capabilities. However, when multiple agents (radar nodes) simultaneously optimize strategies based on local observations, the increased environmental dynamism leads to non-stationary challenges in the learning process for each agent, affecting strategy convergence and stability. Optimizing the algorithm model to address the non-stationarity, credit allocation, and communication cooperation issues arising from multi-agent collaboration, and better integrating MARL technology into radar network anti-jamming strategy generation research, is a problem that needs to be solved in the current environment.
[0003] Furthermore, current common MARL-based multi-agent reinforcement learning algorithms often suffer from several problems. First, anti-interference remains passive. When interference sources perform interference actions in complex environments, the reasoning logic becomes invisible. Traditional methods, through centralized training and decentralized execution, leave the decision-making logic hidden. This means that agents adjust their policies solely based on reward signals, without explaining why a particular action is chosen or how it adapts to changes in the environment. Second, problem attribution becomes ambiguous. In traditional MARL scenarios involving multi-agent collaboration and multiple action types, whether in active inducement or passive anti-interference, a common dilemma exists in credit allocation. Global reward improvements cannot be attributed to specific agents or actions, leading to ambiguity in subsequent optimization directions. Finally, policy generalization is weak. The core application scenarios of MARL algorithms are generally dynamic, with multiple interference types and constraints. MARL algorithms struggle to adapt to this complexity in electronic warfare environments.
[0004] Therefore, there is an urgent need for a radar anti-jamming strategy generation method that can integrate multi-agent collaborative decision-making, interference induction mechanism, thought chain reasoning, and multi-index comprehensive evaluation. Summary of the Invention
[0005] The purpose of this invention is to provide a radar anti-jamming strategy generation method based on multi-agent reinforcement learning and induction mechanism, so as to realize multi-radar node collaborative decision-making, active induction and response modeling of jamming behavior, improvement of strategy training stability and unified optimization of multiple indicators.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning, characterized by the following steps:
[0008] Step 1: Construct a multi-agent radar anti-jamming environment and acquire the local states of each agent to form the global state;
[0009] Get the An intelligent agent at time radar received signal Interference observation signals Radar operating parameters Interacting with historical information ;
[0010] right and Feature extraction was performed separately to obtain radar signal features. : and interference characteristics : ,in, , These represent feature extraction mapping functions;
[0011] based on , , and Construct the first Local state of an agent : , by all The local states of each agent constitute the global state. : ;
[0012] Step 2: Identify the local states of each agent through the interference identification sub-network and output the interference representation results;
[0013] For the An agent, its local state Input interference identification subnetwork, output interference type probability vector and interference situation vector , represented as: ,in, This represents the data processing mapping of the interference identification subnetwork; the global interference identification result set is formed by all agents: and ;
[0014] Step 3: Candidate actions are generated in parallel by the induction strategy generation subnetwork and the passive anti-interference action subnetwork;
[0015] For the Each agent generates an active induced action through a policy generation subnetwork. : ,in, This represents the data processing mapping of the induction strategy generation subnetwork; the passive anti-interference action subnetwork generates passive anti-interference actions. : ,in, This represents the data processing mapping of the passive anti-jamming action subnetwork; thereby generating a pair of induced and passive action combinations. A global set of candidate actions is formed by all agents. ;
[0016] Step 4: The distributed execution layer output is completed by the action fusion subnetwork;
[0017] For the Each agent generates fusion weights via an action fusion subnetwork. : ,in, This represents the data processing mapping of the action fusion subnetwork;
[0018] based on By fusing the candidate actions, we obtain the first... The final action of an agent at the current moment. : And then Normalization and boundary constraints are applied to obtain the execution actions. A coordinated action formed by all intelligent agents ;
[0019] Step 5: Apply the joint action to the multi-agent radar anti-jamming environment and generate the next state and the evaluation results of the inducement effect;
[0020] Joint action Input a multi-agent radar anti-jamming environment, and based on the current global state and joint actions Update the state to the next time step using the environment state transition function. After the state is updated, the local state of each agent is obtained again for the next time step. This, in turn, constitutes the next global state. ;
[0021] The induction effect evaluation module generates induction effect values. : ,in, This represents the data processing mapping of the induction effect evaluation module, which forms the induction effect set by all agents. ;
[0022] Simultaneously, the anti-interference effectiveness metrics are calculated, including: the signal-to-interference-to-noise ratio after anti-interference. Interference suppression ratio and signal fidelity ;
[0023] Step 6: Based on the global state Joint actions With the set of induction effects The value outputs of the global value branch, local value branch, and induced value branch are calculated respectively, and a comprehensive value is constructed.
[0024] Global value branch based on global state and joint actions Output global value Local value branches according to the first Local state of an agent and local actions Output local value Inducing value branches according to the first Local state of an agent Local actions and induction effect value Output Induced Value ;
[0025] This leads to a comprehensive value: ,in, , , Weighting for value integration;
[0026] Step 7: Based on the signal-to-interference-to-noise ratio after anti-interference... Interference suppression ratio Signal fidelity and comprehensive value Construct the comprehensive reward at the current moment. : ,in, These are the weighting coefficients;
[0027] Step 8: Construct an enhanced experience replay mechanism, consisting of an experience replay pool, a storage layer, a priority index layer, a sampling layer, a priority index table, a dynamic update module, and a sampling weight correction module;
[0028] Generate enhanced experience samples : ,in, This is a termination marker;
[0029] Based on comprehensive value Constructing timing difference error: ,in, Discount factor;
[0030] The priority index layer calculates empirical priority based on timing difference error: ,in, To avoid constants with zero priority, As a priority control index, The priority of the nth empirical sample;
[0031] The storage layer will write enhanced experience samples and their priorities together into the experience replay pool, forming... Storage format;
[0032] The priority index table is used to record the correspondence between experience samples and priorities, and uses the SumTree data structure to achieve efficient indexing;
[0033] Step 9: Sample training batches from the experience replay pool and generate the optimal anti-interference strategy based on the optimal target.
[0034] Furthermore, the interference recognition subnetwork uses the local state of the agent. As input, each element in the local state is flattened and dimensionally aligned using Flatten. The flattened radar signal features and interference features are then fed into a one-dimensional convolutional layer. The features extracted by the one-dimensional convolutional layer are then subjected to high-dimensional mapping and nonlinear transformation through three fully connected layers to obtain semantic features. These semantic features are then passed sequentially through a LayerNorm normalization layer, an Integration Layer fusion layer, and a Dropout deactivation layer to obtain interference characterization features. Finally, an output mapping layer generates an interference type probability vector. and interference situation vector .
[0035] Furthermore, in the induction strategy generation sub-network, for , and Flattening and dimension alignment are performed separately to obtain input features in a unified form. The three types of input features are then fed into their respective feature encoding layers to extract feature codes. The encoded results are then concatenated to form the induced decision fusion feature. This fusion feature is sequentially passed through a fully connected mapping layer, a LayerNorm normalization layer, an Integration Layer fusion layer, and a Dropout deactivation layer to obtain the induced decision feature. Finally, an active induced action is generated through an output mapping layer and a Tanh activation function. .
[0036] Furthermore, in the passive anti-interference action sub-network, for , and The input features are flattened and aligned in dimension to obtain a unified form. The three types of input features are then fed into their respective feature encoding layers to extract feature codes. The encoded results are then concatenated to form a fusion feature for generating passive anti-interference actions. This fusion feature is passed sequentially through multiple fully connected mapping layers, a ReLU activation function, a LayerNorm normalization layer, an Integration Layer, and a Dropout deactivation layer to obtain passive anti-interference decision features. Finally, passive anti-interference actions are generated using an output mapping layer and a Tanh activation function. .
[0037] Furthermore, in the action fusion sub-network, for , and The input features are flattened and aligned in dimension to obtain a unified form. These three types of input features are then fed into their respective feature encoding layers to obtain interference situation encoding, action encoding, and historical interaction encoding. The encoding results are then concatenated to form action fusion features. These action fusion features are sequentially passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, IntegrationLayer ensemble layers, and Dropout layers to obtain robust fusion decision features. Finally, fusion weights are generated through a Softmax weight output layer. .
[0038] Furthermore, in the induction effect evaluation module, for , and The input features are flattened and aligned in dimension to obtain a unified form. These three types of input features are then fed into their respective feature encoding layers to obtain pre-state encoding, action encoding, and post-state encoding. Next, state difference features are constructed based on the pre-state and post-state encodings, and interference response change features are constructed based on the interference frequency, interference power distribution, or false target offset before and after the action execution. Finally, the pre-state encoding, action encoding, post-state encoding, state difference features, and interference response change features are concatenated to form a fusion feature for evaluating the induction effect. This fusion feature is then passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layer fusion layers, and Dropout deactivation layers to obtain robust induction effect evaluation features. Finally, the fusion feature is generated through a Sigmoid output layer. Induction effect value of individual agents .
[0039] Furthermore, the induced value branch is represented as:
[0040] , ,in, This represents the eigenvector concatenation function. This represents the data processing mapping of the induced value branch network. This represents network parameters.
[0041] Furthermore, in the induced value branch, for , , and The input features are flattened and aligned in dimension to obtain a unified form. These four types of input features are then fed into their respective feature encoding layers to obtain state encoding, induced action encoding, interference posture encoding, and induced effect encoding. The encoding results are then concatenated to form a fused induced value feature. This fused feature is sequentially passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layer fusion layers, and Dropout deactivation layers to obtain the induced value feature. Finally, a linear output layer is used to generate the first... The inducible value of an agent .
[0042] Furthermore, in the enhanced experience replay mechanism, sampling is performed according to priority during training. ,in, Indicates the first The probability that an empirical sample is sampled. Indicates the first Priority of empirical samples This represents the priority adjustment coefficient. This represents the total number of experience samples in the experience replay pool.
[0043] Furthermore, the optimal objective is expressed as: ,in, This represents a joint strategy comprised of all agents. This represents the optimal anti-interference strategy. This represents the maximum number of decision steps within a single round.
[0044] Based on the above technical solution, the beneficial effects of the present invention are as follows:
[0045] This invention provides a radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning, which has the following advantages;
[0046] 1) This invention deeply integrates passive anti-jamming and active guidance control through heterogeneous action space and dynamic weighting mechanism, and can dynamically allocate action weights according to the type of interference. Compared with traditional single anti-jamming strategy, this invention can simultaneously improve signal-to-interference-plus-noise ratio (SINR) and interference rejection ratio (JSR) in scenarios of suppressing interference, deceiving interference, targeting interference and mixed interference, while ensuring the fidelity of target signal, and effectively solves the problem that radar detection performance is easily interfered with in complex electromagnetic environments.
[0047] 2) This invention decomposes the multi-agent reinforcement learning decision-making process into explicit thought chain reasoning, making the decision logic traceable and explainable, thus avoiding the unreliability problem of traditional black box models. At the same time, the priority experience replay mechanism based on TD error and interference response can prioritize the sampling of high-value samples (such as successful induction and significant interference suppression effects), and correct distribution bias through importance sampling, thereby accelerating model convergence, reducing blind trial and error during training, and improving the learning efficiency and stability of multi-agent collaborative decision-making.
[0048] 3) This invention uses multiple indicators as core indicators to construct a comprehensive reward function that takes into account effectiveness, fidelity and induction effect, and designs a dynamic evaluation system that can adaptively adjust indicator weights according to interference intensity. In strong interference scenarios, detection effectiveness is prioritized, while in weak interference scenarios, real-time performance and robustness are taken into account, making the generated anti-interference strategy more adaptable to the dynamic confrontation needs of real battlefield, while providing accurate and stable gradient signals for model training. Attached Figure Description
[0049] Figure 1 This is a schematic diagram of the algorithm network structure of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided by the present invention.
[0050] Figure 2 This is a schematic diagram of the induced generation subnetwork structure of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided by the present invention.
[0051] Figure 3 This is a schematic diagram of the dual-branch parallel network structure of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided by the present invention.
[0052] Figure 4 This is a schematic diagram of the enhanced experience pool replay structure of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided by the present invention.
[0053] Figure 5 This is a schematic diagram of the experience storage data structure for the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided by the present invention.
[0054] Figure 6 The figure shows the experimental performance curves of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning provided in this invention. Figure 6 (a) represents the reward value. Figure 6 In the middle (b), the signal-to-interference-plus-noise ratio is . Figure 6 (c) represents fidelity. Figure 6 In the middle (d), the interference suppression ratio is , Figure 6 In the middle (e), the detection probability is represented. Detailed Implementation
[0055] To make the objectives, technical solutions, and effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0056] This embodiment provides a radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning, such as... Figure 1 As shown, the main steps include:
[0057] Step 1: Construct a multi-agent radar anti-jamming environment and generate the input states for each agent;
[0058] By decomposing the signal of the original dataset, the first... Each radar node at time radar received signal Interference observation signals Radar operating parameters Interacting with historical information ;in, Used to characterize the The target echo currently received by the intelligent agent and the signal after interference; Used to characterize the current local interference power, bandwidth coverage, and interference pattern; Used to characterize the transmit power, waveform parameters, frequency parameters, and beam control parameters of the current radar node; Used to represent historical information such as actions, rewards, and induced effects in previous moments;
[0059] right and Feature extraction was performed separately to obtain radar signal features. : and interference characteristics : ,in, This represents a mapping that performs normalization, time-frequency analysis, and feature compression on radar received signals. This represents a mapping for extracting interference intensity, bandwidth, and type-related features from the observed interference signal;
[0060] based on , , and Construct the first Local state of an agent : ,
[0061] From all The local states of each agent constitute the global state. : , where local state Entering the A distributed execution layer for individual agents; global state It is used for global value evaluation in subsequent centralized training layers, that is, global information is used during training, and local observations of each agent are relied upon during execution.
[0062] Step 2: The interference identification subnetwork identifies the state of each agent and outputs the interference characterization results;
[0063] For any number An agent, its local state Input interference identification subnetwork, the interference identification subnetwork for and Conduct joint analysis, and combine it with the current radar operating parameters. Output the probability vector of the interference type corresponding to the agent. and interference situation vector , represented as: ,in, The data processing mapping representing the interference recognition subnetwork is essentially the process of feature encoding, fusion, and outputting the three types of inputs. Feature encoding refers to transforming data from different sources and dimensions into a unified feature vector that can be fused later using convolutional extraction layers and nonlinear mapping layers. The essence of fusion is to perform feature concatenation and use fully connected mapping layers for full mapping connections.
[0064] Specifically, the interference identification subnetwork is as follows: Figure 2 As shown, the local state of the agent As input, each element is flattened using Flatten to unify it into a one-dimensional feature representation, facilitating subsequent processing by convolutional and fully connected layers to achieve input dimension alignment and data format unification; the flattened radar signal features ( ) and characteristics of interference signals ( The corresponding data are fed into one-dimensional convolutional layers Conv1d1 and Conv1d2 to extract local correlation features from the state sequence, including: local change features of the signal waveform, interference frequency band distribution features, and time-related disturbance pattern features. After processing by the convolutional network layers, the neural network obtains a preliminary representation of local perceptual features.
[0065] The features extracted by the one-dimensional convolutional layers Conv1d1 and Conv1d2 are respectively input to three fully connected layers Linear 1 / 2 / 3. High-dimensional mapping and nonlinear transformation (the function of the ReLU function) are applied to the previously extracted local features, further transforming the original features into abstract semantic features. These semantic features are then input to the normalization module LayerNorm to normalize the feature distribution, reducing scale differences between different feature dimensions and improving the stability and convergence of the training process. The normalized features are further input to the Integration Layer, which fuses the feature information extracted by the convolutional and fully connected layers to form a unified interference representation feature. To prevent network overfitting and improve the model's generalization ability in complex electromagnetic environments, the fused features are input to the Dropout layer, randomly deactivating some neurons during the training phase to obtain robustly enhanced output features. Finally, the output features are processed by the output mapping layer to generate two results: one is the interference recognition result. The first is the discrimination result representing the current type of interference, and the second is the interference situation representation vector. These are used to characterize the comprehensive state information of the current interference environment, and both serve as the input basis for the subsequent induction strategy generation subnetwork and action fusion subnetwork.
[0066] right After all the agents execute in parallel, a global set of interference recognition results is formed: and ;
[0067] Step 3: The induced strategy generation subnetwork and the passive anti-interference action subnetwork are run in parallel to generate candidate actions;
[0068] like Figure 3 As shown, for the first A smart agent, inducing policy generation subnetwork receiver , and interference characteristics Generate active induced actions : ,in, The data processing mapping of the sub-network representing the induction strategy generation also uses feature encoding to obtain encoding vectors from the three types of inputs, and fuses the three encoding results to form the features of the induction decision and outputs the induction action parameters; It is mainly used to guide the behavior of the jammer to deviate. The specific actions include false target parameters, false frequency parameters, and decoy power allocation parameters. These actions are directly related to the jamming identification results and are therefore generated from step 2 and the jamming features.
[0069] Specifically, similar to the interference identification subnetwork, in the inducement strategy generation subnetwork, firstly, for , as well as Flattening and dimension alignment are performed separately to obtain input features in a unified form. Then, the three types of inputs are fed into their respective feature encoding layers, where fully connected mapping and non-linear activation functions are used to extract the three types of feature codes. The three encoding results are then concatenated to form the induced decision fusion feature. This fusion feature passes through a fully connected mapping layer, a LayerNorm normalization layer, an Integration Layer, and a Dropout deactivation layer to obtain the induced decision feature. Finally, an active induced action is generated through an output mapping layer and a Tanh activation function. ;
[0070] Meanwhile, the passive anti-jamming action sub-network receives... , Interacting with historical information Generate passive anti-interference actions according to the above steps. : ,in, The data processing mapping representing the passive anti-interference action sub-network, and its relationship with Similarly, the difference lies in the different functions of the three inputs; This is used to adjust the transmit power, waveform parameters, frequency agility mode, and beam control mode of the current radar node. Here, we use... This is because historical effects can influence whether current passive adjustment strategies continue, weaken, or are modified.
[0071] Specifically, in the passive anti-interference action sub-network, for , and The input vectors are flattened and aligned in dimension to obtain a uniform form. Then, they are fed into the corresponding feature encoding layers to extract the current state encoding. The encoding results are then concatenated to form a fusion feature for generating passive anti-interference actions. This fusion feature passes through multiple fully connected mapping layers, a ReLU activation function, a LayerNorm normalization layer, an Integration Layer, and a Dropout deactivation layer to obtain passive anti-interference decision features. Finally, passive anti-interference actions are generated through the output mapping layer and the Tanh activation function. It is used to adjust the transmit power, frequency agility parameters and beam control parameters of the current radar node;
[0072] Single agent A combination of induced and passive actions After parallel execution on all radar nodes, a global candidate action set is obtained. ;
[0073] Step 4: The action fusion subnetwork completes the output of the distributed execution layer;
[0074] For the Each agent receives the interference situation vector output in step 2 from the action fusion subnetwork. Candidate actions output in step 3 and historical interaction information Calculate the fusion weights : ,in, and The processing of input is similar; among which, This is used to measure the respective weights of active induction actions and passive anti-interference actions in the current scenario. In historical scenarios with good induction effects, the algorithm will... Increase the adjustment; if history shows that passive adjustment is more stable, decrease it. ;
[0075] Specifically, in the action fusion sub-network, the input is first flattened and dimensionally aligned to obtain input features in a unified form; subsequently, the interference situation is... , and The corresponding feature encoding layers are input separately to obtain interference situation encoding, action encoding, and historical interaction encoding. Then, the encoding results are concatenated to form action fusion features. The action fusion features are then passed through multiple fully connected mapping layers, ReLU activation function, LayerNorm normalization layer, Integration Layer, and Dropout layer to obtain robust fusion decision features. Finally, the fusion weights are generated through the Softmax weight output layer. ;
[0076] based on By fusing the candidate actions, we obtain the first... The final action of an agent at the current moment. :
[0077] ,
[0078] in, This represents element-wise multiplication, also known as the Hadamard product, which refers to multiplying elements one by one. Each component is multiplied by the corresponding action component at that position;
[0079] Again Normalization and boundary constraints are applied to obtain the final value. After performing this step on all agents, the input to the centralized training layer is formed, namely the set of joint actions: ,in, On one hand, it is input into the environment for execution; on the other hand, it enters the centralized training layer for value branch calculation.
[0080] Step 5: The joint action is applied to the multi-agent radar anti-jamming environment and the next state and the induction effect evaluation results are generated;
[0081] The combined action obtained in step 4 Input a multi-agent radar anti-jamming environment, the environment is determined based on the current global state. and joint actions Update the state to the next time step using the environment state transition function. After the state is updated, the local state of each agent is obtained again for the next time step. This, in turn, constitutes the next global state. ;
[0082] After the environment is executed, anti-interference performance metrics are calculated simultaneously, including the signal-to-interference-to-noise ratio (SNR) after anti-interference. Interference suppression ratio and signal fidelity ;
[0083] In addition, the induction effect evaluation module receives the current local state. Execution of actions and the local state at the next moment , for the The induction effectiveness of each agent is quantified to obtain the induction effect value. : , Essentially, it encodes the state before the action, the state after the action, and the state after the action separately, then constructs change features through state difference, and finally integrates and evaluates the induction effectiveness score.
[0084] Specifically, in the induction effect evaluation module, firstly, for , and The input vectors are flattened and aligned in dimension to obtain a uniform form. Then, the pre-action state, the action execution state, and the post-action state are input into the corresponding feature encoding layers to obtain pre-state encoding, action encoding, and post-state encoding, respectively. Next, state difference features are constructed based on the pre-state and post-state encodings, and interference response change features are constructed based on the interference frequency, interference power distribution, or false target offset before and after the action execution. Then, the pre-state encoding, action encoding, post-state encoding, state difference features, and interference response change features are concatenated to form a fusion feature for induction effect evaluation. This fusion feature passes through multiple fully connected mapping layers, ReLU activation function, LayerNorm normalization layer, Integration Layer fusion layer, and Dropout deactivation layer to obtain robust induction effect evaluation features. Finally, the fusion feature is generated through the Sigmoid output layer. Induction effect value of individual agents This is used to characterize the degree of influence of the current active induced action on the behavioral deviation of the interference source;
[0085] The set of induction effects is obtained for all agents. ;
[0086] Step 6: The centralized training layer receives the global state. Joint actions and induction effect set Calculate the output of each of the three value branches;
[0087] The global value branch outputs the global value based on the global state and the joint action. This is used to measure the overall contribution of the joint actions of all agents at the current moment to the overall anti-interference performance of the network; local value branches are based on the first... Local state of an agent and local actions Output local value This is used to measure the agent's independent contribution to overall performance; the induced value branch is based on the first... A single agent Local actions and induction effect value Output Induced Value This is used to measure the additional benefits brought by active induction actions compared to general anti-interference actions; all of the above are achieved by encoding features of different inputs and splicing and fusing them, and finally obtaining the final value output through the network value output layer;
[0088] Then construct the comprehensive value:
[0089] ,
[0090] in, , , The value fusion weights, whose sum is 1, are fixed by the algorithm through the optimal solution. The three Q-values generated in this step are not the final policy itself, but rather the basis for subsequent reward construction, credit allocation, and parameter updates. It should also be noted that the global value branch and local value branch are existing MARL algorithm models, which will not be elaborated here. This invention introduces an induced value branch, specifically:
[0091] ,
[0092] in, For the first An intelligent agent, inducing value branches to accept local states. Actively guiding actions Interference situation vector and induction effect value The constructed induced value input features This is a feature vector concatenation function used to join multiple input vectors from different sources end-to-end according to their feature dimensions, forming a unified network input vector. This operation does not change the internal values of each input vector; it is only used to achieve the joint representation of multi-source features. ,
[0093] To represent an induced value branching network, firstly, for the local state... Actively guiding actions Interference situation vector and induction effect value The inputs are flattened and aligned in dimension. Then, the four types of inputs are fed into their respective encoding layers to obtain state codes, induced action codes, interference posture codes, and induced effect codes. Next, the encoding results are concatenated to form an induced value fusion feature. This fusion feature is then passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layer fusion layers, and Dropout deactivation layers to obtain the induced value feature. Finally, the linear output layer generates the first... The inducible value of an agent , These represent network parameters, which are adaptively generated by the algorithm.
[0094] Step 7: Based on the environmental performance index signal-to-interference-plus-noise ratio obtained in Step 5. Interference suppression ratio and signal fidelity The comprehensive reward at the current moment is constructed using the combined value obtained in step 6. :
[0095] ,
[0096] in, The weighting coefficients are adaptively adjusted during the algorithm training process; the environmental effectiveness index is responsible for directly representing the quality of the current action in anti-interference effect, while the comprehensive value is responsible for representing the overall collaborative effect, individual contribution and active induction effectiveness, respectively.
[0097] Step 8: Based on the results obtained in the preceding steps, the network structure of the entire experience replay pool is as follows: Figure 4 As shown, the enhanced experience replay mechanism consists of an experience replay pool, a storage layer, a priority index layer, a sampling layer, a priority index table, a dynamic update module, and a sampling weight correction module. After the agent interacts with the radar anti-jamming environment, it first generates enhanced experience samples. :
[0098] ,
[0099] in, This represents the current global state. This is the probability set of all network interference types. This is a summary of the overall network interference situation. For a set of combined actions, For the set of inducing effects, For the current reward, This represents the global state at the next moment. Termination marker; based on comprehensive value Constructing timing difference error: ,in, As a discount factor, the priority index layer further calculates empirical priorities based on the time-series difference error and the induction effect evaluation value: ,in, To avoid constants with zero priority, As a priority control index, The priority of the nth empirical sample;
[0100] The storage layer will write enhanced experience samples and their priorities together into the experience replay pool, forming... Storage format;
[0101] The priority index table records the correspondence between experience samples and priorities, and uses a SumTree data structure to achieve efficient indexing, such as... Figure 5 As shown, SumTree is a binary tree accumulation structure, where leaf nodes are used to store the priority values corresponding to each experience sample. Each non-leaf node stores the sum of the priorities of its left and right child nodes, and the root node stores the sum of the priorities of all samples in the entire experience replay pool. When a new experience sample is written or the priority of an existing sample is updated, only the priority of the corresponding leaf node needs to be modified, and the accumulated value of each parent node is updated layer by layer along the path from that leaf node to the root node. Therefore, the time complexity of both write and update operations is O(n log n). During training sampling, sampling is performed according to priority. ,in, Indicates the first The probability that an empirical sample is sampled. Indicates the first Priority of empirical samples This represents the priority adjustment coefficient, used to control the degree of priority sampling. When, it degenerates into uniform random sampling; when As the value increases, higher-priority samples are more likely to be sampled. This represents the total number of experience samples in the experience replay pool;
[0102] The high-value experiences and their weights obtained from sampling are used for subsequent global value branch, local value branch, induced value branch and policy network parameter updates. The dynamic update module recalculates the sample priority based on the new temporal difference error after each round of training and updates the priority index table, so that the experience replay pool can continuously focus on more valuable experience samples in the current training stage.
[0103] Step 9: Sample training batches from the experience replay pool to update the network parameters of the centralized training layer and the distributed execution layer; during training, utilize the global state. Joint actions Induction effect set and rewards The parameters of the three value branches are updated, and the updated value gradients are then applied in reverse to the interference recognition subnetwork, the induced policy generation subnetwork, the passive anti-interference action subnetwork, and the action fusion subnetwork of each agent, so that each subnetwork can generate better actions in the next round of decision-making.
[0104] The optimal objective of the entire method is: ,in, This represents a joint policy comprised of a distributed execution layer consisting of all agents from 1 to N. This represents the final optimal anti-interference strategy; because The value already includes environmental performance rewards, global value, local value, and incentive value, therefore the final result is... It is not a strategy that simply pursues the optimal performance of a single metric, but rather a comprehensive optimal strategy that simultaneously considers the network-wide collaborative anti-interference performance, the independent contribution of each node, and the effectiveness of proactive induction; during the execution phase, agents 1 to N only need to base their decisions on their respective local states. Repeat steps 2 through 5 to output the local action. During the training phase, steps 6 through 9 are repeated until the convergence condition is met, at which point the final optimal anti-interference strategy is output. ;
[0105] In this embodiment, the model is trained using radar simulation signals and datasets of three different types of interference (narrowband interference, broadband interference, and mixed interference). On different algorithms of multi-agent reinforcement learning, the evaluation index system is used to evaluate and quantify the different algorithms, and the contribution ratio of each index is visualized through heatmaps to intuitively show the advantages and disadvantages of the strategy.
[0106] The following comparative experiments further illustrate the beneficial effects of the present invention.
[0107] like Figure 6 The figure shown is an experimental performance curve of the radar anti-jamming strategy generation method based on the fusion of multi-agent reinforcement learning and thought chain reasoning in this embodiment. Figure 6 (a) represents the reward value. Figure 6 In the middle (b), the signal-to-interference-plus-noise ratio is . Figure 6 (c) represents fidelity. Figure 6 In the middle (d), the interference suppression ratio is , Figure 6 In the middle (e), the detection probability gradually stabilizes as the number of training rounds increases, eventually settling at a relatively normal amplitude level, which verifies that the training model of the present invention is convergent and effective.
[0108] from Figure 6As can be seen in (a), as the number of training rounds increases, the reward curve of the model increases rapidly and gradually flattens out. Finally, the reward value stabilizes between 9 and 10. This indicates that the strategy is constantly improving during the initial reward increase period. When the strategy matures, the reward tends to stabilize. The reward peak represents the best performance that the algorithm can achieve, indicating that the model of the present invention has a relatively good comprehensive score.
[0109] from Figure 6 As can be seen in (b), before the anti-interference strategy was applied, the SINR value was stable at around 4, indicating that the signal was severely affected by interference. After the anti-interference strategy was applied, the SINR rose rapidly in the early stage of training rounds 0-50, and after 100 training rounds, it began to stabilize at around 14, which is 10dB higher than before the anti-interference strategy was applied, indicating that the signal quality was better at this time. This index curve intuitively measures the effect of the anti-interference algorithm.
[0110] Furthermore, signal fidelity is an important indicator used to measure the similarity between the signal after anti-interference processing and the original signal. It reflects whether the waveform after anti-interference retains the waveform characteristics of the original signal and assesses the degree of signal distortion; from Figure 6 As can be seen in (c), the fidelity curve fluctuates around 0.92. Using the fidelity curve can effectively avoid over-processing by the algorithm and ensure the accuracy of subsequent signal processing, such as target recognition. This indicator can form a balance with SINR, preventing the algorithm from sacrificing fidelity in pursuit of SINR.
[0111] In radar anti-jamming strategy model algorithms, the Jam Suppression Ratio (JSR) is a crucial indicator used to measure the degree of jamming suppression achieved by anti-jamming processing. In JSR, the vertical axis represents the degree of jamming suppression. By calculating the jamming power before and after suppression, the effectiveness of the anti-jamming algorithm is verified, and the differences in JSR for different jamming types are also reflected. Figure 6 As can be seen from (d), the JSR value eventually reaches 5~6dB with the number of training rounds, which indicates that the interference suppression effect is good;
[0112] In addition to the basic metrics mentioned above for measuring the effectiveness of anti-interference strategies, Figure 6 Figure (e) visually illustrates the probability curve of a radar network system successfully detecting a target signal. Regarding the target detection capability of anti-jamming algorithms, a higher detection probability (Pd) indicates fewer missed detections, directly impacting the reliability of the radar network system and determining its suitability for practical applications and engineering implementation. Figure 6As can be seen from (e), a detection probability of 0.9 or higher indicates that the current detection is good. This indicator is also a fallback indicator for radar anti-jamming signals. Even if other indicators are good, a low Pd indicates that many useful signals have been missed in signal detection.
[0113] To further evaluate the performance of different algorithm models, this embodiment selects other advanced versions of multi-agent reinforcement learning algorithms from recent years and conducts comparative experiments on the processed radar simulation signal and jamming signal data test sets, specifically MADDPG, QMIX, MAAC, and MAPPO. During the comparative experiments, this embodiment records the performance of different models. , (dB) , (Detection probability) (Policy generation delay, ms) (Fault tolerance) and The specific comparative experimental results for indicators such as (interference adaptability) are shown in Table 1. In the table, the "↑" sign after the evaluation index indicates that the larger the index, the better the algorithm performance, and the "↓" sign indicates that the smaller the index, the better the algorithm performance. The bold text in the table indicates the value that achieves the best effect in the entire column.
[0114] Table 1. Comparison of Radar Interference Signal Datasets: Experimental Results
[0115]
[0116] The following analysis compares the effectiveness and efficiency of the algorithm model's anti-jamming decision-making: In terms of effectiveness, the signal-to-interference-plus-noise ratio (SNR) and interference suppression ratio (ISR) of this invention remain at a relatively high level. Although this invention does not have the lowest average delay in the table, it is significantly higher than the MAAC algorithm and the baseline algorithm, which have better anti-jamming decision-making effectiveness. This indicates that this invention has good anti-jamming decision-making effectiveness while ensuring a high detection probability. Furthermore, due to its superior adaptability, it can easily achieve corresponding game theory strategies in complex electronic warfare environments. While maintaining high SNR and INR, it also maintains a certain advantage in preserving the fidelity of the original signal. In terms of efficiency, the strategy generation delay is low, resulting in faster decision generation in practical applications. Its convergence is also better than most algorithms in the table. This indicates that this invention has good anti-jamming decision-making efficiency and can quickly adapt to complex scenarios in radar electronic warfare environments.
[0117] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.
Claims
1. A radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning, characterized in that, Includes the following steps: Step 1: Construct a multi-agent radar anti-jamming environment and acquire the local states of each agent to form the global state; Get the An intelligent agent at time radar received signal Interference observation signals Radar operating parameters Interacting with historical information ; right and Feature extraction was performed separately to obtain radar signal features. : and interference characteristics : ,in, , These represent feature extraction mapping functions; based on , , and Construct the first Local state of an agent : , by all The local states of each agent constitute the global state. : ; Step 2: Identify the local states of each agent through the interference identification sub-network and output the interference representation results; For the An agent, its local state Input interference identification subnetwork, output interference type probability vector and interference situation vector , represented as: ,in, This represents the data processing mapping of the interference identification subnetwork; the global interference identification result set is formed by all agents: and ; Step 3: Candidate actions are generated in parallel by the induction strategy generation subnetwork and the passive anti-interference action subnetwork; For the Each agent generates an active induced action through a policy generation subnetwork. : ,in, This represents the data processing mapping of the induction strategy generation subnetwork; the passive anti-interference action subnetwork generates passive anti-interference actions. : ,in, This represents the data processing mapping of the passive anti-jamming action subnetwork; thereby generating a pair of induced and passive action combinations. A global set of candidate actions is formed by all agents. ; Step 4: The distributed execution layer output is completed by the action fusion subnetwork; For the Each agent generates fusion weights via an action fusion subnetwork. : ,in, This represents the data processing mapping of the action fusion subnetwork; based on By fusing the candidate actions, we obtain the first... The final action of an agent at the current moment. : And then Normalization and boundary constraints are applied to obtain the execution actions. A coordinated action formed by all intelligent agents ; Step 5: Apply the joint action to the multi-agent radar anti-jamming environment and generate the next state and the evaluation results of the inducement effect; Joint action Input a multi-agent radar anti-jamming environment, and based on the current global state and joint actions Update the state to the next time step using the environment state transition function. After the state is updated, the local state of each agent is obtained again for the next time step. This, in turn, constitutes the next global state. ; The induction effect evaluation module generates induction effect values. : ,in, This represents the data processing mapping of the induction effect evaluation module, which forms the induction effect set by all agents. ; Simultaneously, the anti-interference effectiveness metrics are calculated, including: the signal-to-interference-to-noise ratio after anti-interference. Interference suppression ratio and signal fidelity ; Step 6: Based on the global state Joint actions With the set of induction effects The value outputs of the global value branch, local value branch, and induced value branch are calculated respectively, and a comprehensive value is constructed. Global value branch based on global state and joint actions Output global value Local value branches according to the first Local state of an agent and local actions Output local value Inducing value branches according to the first Local state of an agent Local actions and induction effect value Output Induced Value ; This leads to a comprehensive value: ,in, , , Weighting for value integration; Step 7: Based on the signal-to-interference-to-noise ratio after anti-interference... Interference suppression ratio Signal fidelity and comprehensive value Construct the comprehensive reward at the current moment. : ,in, These are the weighting coefficients; Step 8: Construct an enhanced experience replay mechanism, consisting of an experience replay pool, a storage layer, a priority index layer, a sampling layer, a priority index table, a dynamic update module, and a sampling weight correction module; Generate enhanced experience samples : ,in, This is a termination marker; Based on comprehensive value Constructing timing difference error: ,in, Discount factor; The priority index layer calculates empirical priority based on timing difference error: ,in, To avoid constants with zero priority, As a priority control index, The priority of the nth empirical sample; The storage layer will write enhanced experience samples and their priorities together into the experience replay pool, forming... Storage format; The priority index table is used to record the correspondence between experience samples and priorities, and uses the SumTree data structure to achieve efficient indexing; Step 9: Sample training batches from the experience replay pool and generate the optimal anti-interference strategy based on the optimal target.
2. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, The interference recognition subnetwork is based on the local state of the agent. As input, each element in the local state is flattened and dimensionally aligned using Flatten. The flattened radar signal features and interference features are then fed into a one-dimensional convolutional layer. The features extracted by the one-dimensional convolutional layer are then subjected to high-dimensional mapping and nonlinear transformation through three fully connected layers to obtain semantic features. These semantic features are then passed sequentially through a LayerNorm normalization layer, an Integration Layer fusion layer, and a Dropout deactivation layer to obtain interference characterization features. Finally, an output mapping layer generates an interference type probability vector. and interference situation vector .
3. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, In the subnetwork generated by the induction strategy, for , and Flattening and dimension alignment are performed separately to obtain input features in a uniform form; the three types of input features are then fed into their respective feature encoding layers to extract feature codes. Then, the encoding results are concatenated to form the induced decision fusion feature. The fusion feature is then passed through a fully connected mapping layer, a LayerNorm normalization layer, an Integration Layer fusion layer, and a Dropout deactivation layer to obtain the induced decision feature. Finally, the active induced action is generated through the output mapping layer and the Tanh activation function. .
4. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, In the passive anti-interference action sub-network, for , and The input features are flattened and aligned in dimension to obtain a uniform form; the three types of input features are then fed into their respective feature encoding layers to extract feature codes. Then, the encoding results are concatenated to form the fusion feature for generating passive anti-interference actions. The fusion feature is then passed through multiple fully connected mapping layers, ReLU activation function, LayerNorm normalization layer, Integration Layer fusion layer, and Dropout deactivation layer to obtain passive anti-interference decision features. Finally, passive anti-interference actions are generated through the output mapping layer and Tanh activation function. .
5. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, In the action fusion subnetwork, for , and The input features are flattened and aligned in dimension to obtain a unified form. These three types of input features are then fed into their respective feature encoding layers to obtain interference situation encoding, action encoding, and historical interaction encoding. The encoding results are then concatenated to form action fusion features. These action fusion features are sequentially passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layers, and Dropout layers to obtain robust fusion decision features. Finally, fusion weights are generated through a Softmax weight output layer. .
6. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, In the induction effect evaluation module, for , and The input features are flattened and aligned in dimension to obtain a unified form. These three types of input features are then fed into their respective feature encoding layers to obtain pre-state encoding, action encoding, and post-state encoding. Next, state difference features are constructed based on the pre-state and post-state encodings, and interference response change features are constructed based on the interference frequency, interference power distribution, or false target offset before and after the action execution. Finally, the pre-state encoding, action encoding, post-state encoding, state difference features, and interference response change features are concatenated to form a fusion feature for evaluating the induction effect. This fusion feature is then passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layer fusion layers, and Dropout deactivation layers to obtain robust induction effect evaluation features. Finally, the fusion feature is generated through a Sigmoid output layer. Induction effect value of individual agents .
7. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, The induced value branch is represented as follows: , ,in, This represents the eigenvector concatenation function. This represents the data processing mapping of the induced value branch network. This represents network parameters.
8. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 7, characterized in that, In the aforementioned induced value branch, for , , and The input features are flattened and aligned in dimension to obtain a unified form. These four types of input features are then fed into their respective feature encoding layers to obtain state encoding, induced action encoding, interference posture encoding, and induced effect encoding. The encoding results are then concatenated to form a fused induced value feature. This fused feature is sequentially passed through multiple fully connected mapping layers, ReLU activation functions, LayerNorm normalization layers, Integration Layer fusion layers, and Dropout deactivation layers to obtain the induced value feature. Finally, a linear output layer is used to generate the first... The inducible value of an agent .
9. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, In the enhanced experience replay mechanism, sampling is performed according to priority during training. ,in, Indicates the first The probability that an empirical sample is sampled. Indicates the first Priority of empirical samples This represents the priority adjustment coefficient. This represents the total number of experience samples in the experience replay pool.
10. The radar anti-jamming strategy generation method based on the fusion of MARL and thought chain reasoning as described in claim 1, characterized in that, The optimal objective is expressed as: ,in, This represents a joint strategy comprised of all agents. This represents the optimal anti-interference strategy. This indicates the maximum number of decision steps.
