Unknown interference effectiveness evaluation method and system for irregular electronic countermeasure scene

Abstract

The invention discloses an unknown interference efficiency evaluation method and system for an irregular electronic countermeasure scene, in the electronic countermeasure scene, a deep reinforcement learning model based on a MuZero algorithm is introduced, and the greatest advantage of the MuZero algorithm is that decision optimization and efficiency evaluation in a complex environment can be completed without explicit rule modeling. By learning environment dynamics in a potential state space, the model can adapt to dynamic changes of an interference mode and a radar anti-interference strategy, and dependence of a traditional method on rule hypothesis is avoided. Besides, real-time strategy optimization of the model is realized in combination with Monte Carlo tree search (MCTS) and deep learning, the optimal interference strategy can be quickly generated in confrontation scenes with unknown rules and complex environments, and the intelligence and efficiency of the electronic confrontation system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of interference effectiveness evaluation, and in particular to a method and system for evaluating unknown interference effectiveness in irregular electronic countermeasure scenarios. Background Art

[0002] In electronic warfare scenarios, the game between radar and jammer often takes place in a complex environment characterized by high dynamics and uncertainty. Traditional jamming effectiveness assessment methods rely on assumptions about enemy jamming patterns, typically modeled through rule-based or expert knowledge. However, with the advancement of electronic warfare technology, enemy jamming methods have become increasingly complex and dynamic, and traditional rule-based assessment methods are no longer able to meet practical needs, resulting in the following problems:

[0003] (1) Environmental complexity: The interference patterns in real battlefield environments are often difficult to fully describe, and traditional evaluation methods lack the ability to deal with unknown interference behaviors.

[0004] (2) Lack of real-time performance: Traditional methods require preset models or a large amount of test data, which makes it difficult to quickly adapt to the dynamic changes in the battlefield environment.

[0005] (3) Low evaluation accuracy: Hypothetical rule models can easily lead to evaluation results that deviate from reality and are difficult to truly reflect the effectiveness of interference.

[0006] In view of this, the present invention is proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and system for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, and to construct an efficient interference effectiveness evaluation system based on the dynamic changes of interference patterns in irregular electronic countermeasure scenarios.

[0008] The purpose of the present invention is achieved through the following technical solutions:

[0009] A method for evaluating the effectiveness of unknown jamming in irregular electronic countermeasure scenarios includes:

[0010] The jammer is simulated and confronted with the radar to generate interaction data. During the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module are set up. Given an initial state, the deep reinforcement learning model maps it to the potential state space and serves as the root node of the Monte Carlo tree. It predicts the state and reward value after executing each action under the initial state, as well as the policy distribution of the predicted state value and action. The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to each action under each node in the Monte Carlo tree. It makes action decisions based on its own strategy and repeats until it expands to the leaf node to complete a search process. After the search is completed, the number of visits to each node in the search process is saved, the Monte Carlo tree search module calculates the improved policy distribution, and all data is saved as interaction data.

[0011] Using the data of the deep reinforcement learning model and the data of the Monte Carlo tree search module in the interaction data to construct a loss function to train the deep reinforcement learning model;

[0012] After training, the deep reinforcement learning model maps the input state to the latent state space, predicts the corresponding reward value, and uses it as the interference effectiveness evaluation result.

[0013] A system for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, used to implement the aforementioned method, comprises: a simulation unit, a training unit, and an interference effectiveness evaluation unit; wherein:

[0014] The simulation unit is used to simulate the jammer and confront the radar to generate interaction data. During the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module are set up. Given an initial state, the deep reinforcement learning model maps it to the potential state space and uses it as the root node of the Monte Carlo tree to predict the state and reward value after executing each action under the initial state, as well as the policy distribution of the predicted state value and action. The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to the nodes in the Monte Carlo tree to make action decisions based on its own strategy, and repeats until it expands to the leaf node, completing a search process and obtaining a search trajectory. The search process is repeated multiple times, and all data is saved as interaction data.

[0015] A training unit, configured to train the deep reinforcement learning model using the data of the deep reinforcement learning model in the interaction data and the loss function constructed by the Monte Carlo tree search module;

[0016] The interference effectiveness evaluation unit is used to map the input state to the potential state space through the deep reinforcement learning model after training, predict the corresponding reward value, and use it as the interference effectiveness evaluation result.

[0017] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0020] It can be seen from the technical solution provided by the above-mentioned present invention that the combination of Monte Carlo tree search (MCTS) and deep learning technology can perform efficient planning in an environment with unknown rules. By learning the dynamic characteristics of the potential state space and guiding planning decisions with reinforcement learning goals, the adaptability to complex dynamic environments is significantly improved; in addition, applying MuZero to electronic countermeasure scenarios can solve the shortcomings of existing methods in relying on rules, and realize efficient evaluation and dynamic optimization of unknown interference effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 The present invention provides a method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios.

[0023] Figure 2 A schematic diagram of the interactive data generation and model training process provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the working process of the three networks in the deep reinforcement learning model provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of an unknown interference effectiveness evaluation system for irregular electronic countermeasure scenarios provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] First, the following terms may be used in this article:

[0029] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0030] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0031] The following describes in detail a scheme for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, as provided by the present invention. Any information not described in detail in the embodiments of the present invention is prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, the procedures were conducted in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of the present invention, where the manufacturer is not specified, are all commercially available conventional products.

[0032] Example 1

[0033] The embodiment of the present invention provides a method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios. Figure 1 As shown, it mainly includes the following steps:

[0034] Step 1: Simulate the jammer and engage in confrontation with the radar to generate interaction data.

[0035] In an embodiment of the present invention, during the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module are set; given an initial state, the deep reinforcement learning model maps it to a potential state space and serves as the root node of the Monte Carlo tree, predicting the state and reward value after executing each action under the initial state, and predicting the policy distribution of the state value and the action; the Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to each action under each node in the Monte Carlo tree, makes action decisions according to its own strategy, and repeats it until it expands to the leaf node, completing a search process. After the search is completed, the number of visits to each node in the search process is saved, the Monte Carlo tree search module calculates the improved policy distribution, and all data is saved as interaction data.

[0036] In an embodiment of the present invention, the deep reinforcement learning model based on the MuZero algorithm includes: a representation network, a dynamic network and a prediction network; wherein: the representation network is used to map the state to the potential state space dynamic network; the dynamic network is used to predict the state and reward value after performing each action in a given state; the prediction network is used to predict the state value and the strategy distribution of the action.

[0037] In an embodiment of the present invention, the Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to the nodes in the Monte Carlo tree to make action decisions based on its own strategy, including:

[0038] The action with the highest upper confidence interval algorithm score is selected using the following formula:

[0039]

[0040] Among them, a k The action with the highest score of the upper bound of the confidence interval selected for time step k; (s,a) represents the selection of action a in state s, (s,b) represents the selection of action b in state s, N(s,a) is the number of visits to action a in the node corresponding to state s, Q(s,a) is the average value of action a in the node corresponding to state s, P is the policy distribution of each action output by the prediction network, and c1 and c2 are constants.

[0041] In the embodiment of the present invention, after completing a search process, a search trajectory is obtained, and the following update process is performed through the backtracking process:

[0042] The number of visits to each action under each node is incremented by one, and the corresponding average value is updated, which is expressed as:

[0043] N(s,a)←N(s,a)+1

[0044]

[0045] Among them, ← is the assignment symbol, N(s,a) is the number of visits to action a under the node corresponding to state s, Q(s,a) is the average value of action a under the node corresponding to state s, and G is the cumulative reward obtained by simulating from the current node downward.

[0046] Step 2: Use the data of the deep reinforcement learning model and the data of the Monte Carlo tree search module in the interaction data to construct a loss function to train the deep reinforcement learning model.

[0047] In an embodiment of the present invention, the method of constructing a loss function using the data of the deep reinforcement learning model and the Monte Carlo tree search module in the interaction data includes: constructing a loss function using the policy distribution of the action predicted by the deep reinforcement learning model in the interaction data, the predicted state value and reward value, and the improved policy distribution and obtained value of the Monte Carlo tree search module.

[0048] The process of constructing the loss function includes:

[0049] The policy distribution of the deep reinforcement learning model predicting the action and the improved policy distribution of the Monte Carlo tree search module are denoted as π t+k , calculate the error between the two Where T is the transpose symbol, t represents the current time step, and k represents the number of steps executed starting from the current time step t. To predict the policy distribution of the action at time step t+k of the network output; Improve the policy distribution π t+k Calculated by the following formula:

[0050]

[0051] In the above formula, π t+k (a) is the improved strategy distribution corresponding to action a, N(s t+k ,a) is the state s at time step t+k t+k The number of times action a is executed, that is, state s t+k The number of visits to the corresponding node action a, ∑ a′ N(s t+k ,a′) is the state s at time step t+k t+k The number of times all actions are executed, i.e. state s t+k The number of visits to all actions under the corresponding node, a′ refers to any action.

[0052] The state value predicted by the deep reinforcement learning model and the value obtained by the Monte Carlo tree search module are denoted as z t+k , calculate the error between the two in, is the state value predicted by the prediction network at time step t+k, z t+k is the predicted value calculated using the value obtained by the Monte Carlo Tree Search module, expressed as:

[0053] z t+k =u t+k+1 +γu t+k+2 +...+γ n-1 u t+k+n +γ n v t+k+n

[0054] In the above formula, u t+k+i is the observation reward, which is the reward obtained during the actual interaction between the jammer and the radar during the confrontation simulation, i = 1,…,n, γ is the discount factor, n is the number of steps of reward accumulation, v t+k+n The estimated value obtained by the Monte Carlo Tree Search module.

[0055] The reward value predicted by the deep reinforcement learning model and the reward value obtained by the Monte Carlo tree search module are denoted as u t+k , calculate the error between the two in, is the reward value predicted by the dynamic network at time step t+k, u t+k is the observation reward at time step t+k.

[0056] Combining the three errors, we get the loss function:

[0057]

[0058] Among them, l t (θ) is the loss function, c‖θ‖ 2 is the regularization term, c is a constant, θ is the reinforcement learning model parameter, and ‖.‖ is the L2 norm.

[0059] Step 3. After training, the deep reinforcement learning model maps the input state to the latent state space, predicts the corresponding reward value, and uses it as the interference effectiveness evaluation result.

[0060] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0061] 1. Overall overview of the plan.

[0062] The MuZero algorithm is a reinforcement learning method that does not require explicit rules. The present invention proposes a method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios based on the MuZero algorithm by combining Monte Carlo tree search (MCTS) and deep learning technology to cope with the dynamic changes of interference patterns in irregular electronic countermeasure scenarios. By building an efficient interference effectiveness evaluation system, efficient planning can be performed in an environment with unknown rules. In this method, the dynamic characteristics of the potential state space are learned, and planning decisions are guided by reinforcement learning goals, which significantly improves the adaptability to complex dynamic environments. Applying MuZero to electronic countermeasure scenarios can solve the shortcomings of existing methods in relying on rules and achieve efficient evaluation and dynamic optimization of unknown interference effectiveness.

[0063] The embodiment of the present invention combines the three modules of the MuZero algorithm (representation network, dynamic network and prediction network) to establish a jamming effectiveness evaluation model for irregular electronic countermeasure scenarios; the representation network is mainly responsible for encoding the observation data and extracting state representation as the input of the dynamic network and prediction network; the dynamic network is mainly responsible for state transition and reward value under specific actions; the prediction network is responsible for predicting strategy distribution and value for decision optimization.

[0064] The Monte Carlo Tree Search module is then combined with a simulated radar-jammer confrontation module to generate interaction data. This interaction data is then used to jointly train the three networks in the model. After training, the high-dimensional hidden state of the radar and the decision action are input to the dynamic network, and the reward value output is the current jamming effectiveness evaluation.

[0065] 2. Detailed introduction of the plan.

[0066] 1. Overall architecture.

[0067] The present invention mainly includes: a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module.

[0068] The deep reinforcement learning model mainly includes the representation network, dynamic network and prediction network. (1) Representation network h: The observation data of radar and jammer (o1,...,o t ) is encoded and high-dimensional potential state representation is extracted as the input of the subsequent dynamic network and prediction network, that is, s = h θ (o1,...,o t ). (2) Dynamic network g: simulates the interaction process between radar and jammer, and predicts the state transition and reward value under specific actions. That is, r k ,s k =g θ (s k-1 ,a k ), where s k-1 and sk Indicates radar status, a k Indicates interference action, r k Represents the immediate reward. (3) Prediction network f: Combines the abstract potential state, outputs the strategy distribution and value for decision optimization, i.e., p k ,v k =f θ (s k ), s k Indicates radar status, p k represents the policy distribution of interference actions, v k represents the state value; θ is the parameter of the entire deep reinforcement learning model.

[0069] In the embodiment of the present invention, the jammer is mainly simulated. Therefore, the predicted state and state value are both information on the radar side (i.e., the state and state value of the radar), and the predicted action is the jamming action of the jammer.

[0070] The Monte Carlo Tree Search module is mainly responsible for planning in the potential state space using Monte Carlo tree search to generate optimized interference strategies.

[0071] 2. State space and action space design.

[0072] The present invention mainly considers the confrontation process between radar and jammer, so the state space and action space of the two are introduced separately.

[0073] (1) State space.

[0074] (1.1) Radar state space.

[0075] The radar status mainly includes: the current working mode of the radar (such as search, tracking, confirmation, etc.), antenna parameters, current observation signals, historical observation records, etc.

[0076] Representation method: current working mode (search, tracking, confirmation, etc.), antenna parameters (such as frequency, direction angle, bandwidth), historical observation record: target motion parameters received at the last moment (such as speed, direction).

[0077] (1.2) Jammer state space.

[0078] The jammer status mainly includes: the jammer's current jamming pattern, historical observation records, movement information, etc.

[0079] Representation method: interference pattern (such as broadband noise, false target deception, FM interference, etc.); historical observation record: intercepted radar pulse descriptor word (PDW); motion information: current position, speed, direction, etc.

[0080] Abstract state space: Based on the above-mentioned real-world state, the present invention performs learning and search without knowing the specific adversarial rules. During the operation of the deep reinforcement learning model and Monte Carlo tree search module, the following adversarial rules are not used: the judgment of the final adversarial state and the final reward; and the rules for how to transition to the next state given a state and action.

[0081] (2) Action space.

[0082] (2.1) Radar action space.

[0083] The anti-interference actions that radar can choose in different scenarios include: frequency agility: quickly switching the transmission frequency to reduce the impact of interference; pulse repetition frequency (PRF) jitter: adjusting the time interval of the transmitted pulse to increase the robustness of target detection; radio frequency shielding: avoiding jammer lock by selectively shielding part of the echo signal; beam optimization: adjusting the direction and width of the antenna beam to enhance the target signal reception capability.

[0084] (2.2) Jammer action space.

[0085] The actions that the jammer can take include: jamming style switching: switching the jamming type (such as false target deception, jamming noise, etc.) according to the working mode of the target radar; modulation parameter adjustment: modifying the modulation parameters of the jamming signal (such as frequency modulation rate, bandwidth); motion maneuvering: adjusting its own motion parameters (such as speed, direction) to optimize the jamming effect.

[0086] 3. Interaction data generation.

[0087] The confrontation process between radar and jammer is simulated in the simulation platform to generate interaction data.

[0088] In each step of the confrontation, execute in sequence:

[0089] (3.1) Initial state sampling: First, the current state needs to be mapped to the potential state space through the representation network, and used as the root node of the tree in the MCTS module.

[0090] (3.2) Action selection: The interference action is obtained through the Monte Carlo tree search module. The following are the steps of the Monte Carlo tree search module:

[0091] (a) Select:

[0092] At each time step k = 1, ..., l in each simulation, the action with the highest UCB score is selected by combining the value function (dynamic network), the policy function (prediction network) and the number of node visits (the number of times an action is performed in a state). The UCB formula is as follows:

[0093]

[0094] The symbols here mean the number of visits N, the average value Q (the first one is the initialization value, and then it is updated using the formula introduced later), the policy distribution P (the policy distribution of each action output by the prediction network), the reward R and the state transition S. Constants c1 and c2 can be set. Represents the sum of the number of visits to all actions under the node corresponding to state s.

[0095] (b) Extension and Evaluation:

[0096] According to the action selected above, the new state node is expanded, and the dynamic network predicts the next state and reward. Until time l reaches the leaf node, the dynamic network predicts the MCTS search process to expand to the leaf node s l The reward value r l , save it to R(s l-1 ,a l ),S(s l-1 ,a l ) table, p is obtained through prediction network prediction l ,v l (The strategy distribution of the action and the state value corresponding to the node are in turn), thereby updating Q(s,a) and P(s,a), which are then used to calculate the UCB formula.

[0097] (c) Backtracking:

[0098] Update the number of visits and average value of all state-action pairs (s, a) in the search trajectory. Specifically, for each visited state s and its corresponding action a in the trajectory, update its number of visits (N(s, a)) and average value Q(s, a) respectively. During the backtracking process of the search trajectory, first increase the number of visits N(s, a) of each state-action pair (s, a) by one, and then use the current reward to update its average value Q(s, a) based on the updated N(s, a) value. The update process is as follows:

[0099] N(s,a)←N(s,a)+1

[0100]

[0101] Among them, ← is the assignment symbol, G is the cumulative reward obtained by simulating downward from the current node, which can be calculated by the following formula:

[0102]

[0103] Among them, γ τ represents the discount factor, i.e., the power of γ to the power of τ, which is used to measure the impact of future rewards on current decisions; r1+τ represents the immediate reward obtained from the 1+τ step along the simulation trajectory starting from the current time step; γ l Represents the cumulative discount from the current time step to the leaf node time step l, which is used to convert the predicted value v of the leaf node l Backtrack to the current time step.

[0104] (d) Execution: The above process is repeated multiple times. After the MCTS search is completed, a set {N(s,a)} is returned at the root node s0. This set represents the number of visits to each candidate action at the root node during the MCTS search process, which is used to construct the empirical policy distribution. After normalization, the elements of the set are obtained to obtain the improved policy distribution π(a|s)=N(s,a) / ∑ b N(s, b), the above ratios constitute the empirical strategy distribution, which can be regarded as the strategy obtained by MCTS search improvement, and the improved strategy distribution π defined in the subsequent goal of this invention. t+k ' are consistent; they are essentially descriptions of the same policy in different contexts. Interference actions that actually interact with the environment are then sampled from this policy distribution. As mentioned earlier, the action with the highest UCB score is used to expand nodes in the search path. The final action that interacts with the environment is sampled from the policy distribution constructed after the search, improving policy diversity and exploration capabilities.

[0105] (3.3) Environmental Feedback: In the electronic warfare scenario of the present invention, the immediate rewards during the confrontation are unknown; failure or success is known only after the confrontation ends. In this case, the present invention expands the final interference result into the immediate rewards for each step of the confrontation.

[0106] (3.4) Data storage: The trajectory data during the confrontation process is stored in the replay buffer. The trajectory data includes state, interference action, and reward.

[0107] 4. Model training.

[0108] In the embodiment of the present invention, the model parameters are optimized in combination with the data in the playback buffer.

[0109] All parameters of the model are trained jointly to accurately match the policy, value, and reward at each step k in the adversarial game with the corresponding target value observed k actual time steps later. However, unlike traditional model-based reinforcement learning methods, the latent state s in deep reinforcement learning models is k There is no semantics attached to the state of the environment, it is just the potential state of the model.

[0110] Since deep reinforcement learning models search in an abstract Markov decision process (MDP) space, to ensure that planning in the abstract MDP is equivalent to planning in the real environment, we ensure that value equivalence is met. That is, starting from the same real state, the cumulative return of the trajectory through the abstract MDP matches the cumulative return of the trajectory in the real environment.

[0111] In the embodiment of the present invention, there are mainly three optimization goals.

[0112] Objective 1: The target improvement strategy is generated through the Monte Carlo Tree Search module. In order to improve the strategy, it is necessary to minimize the strategy distribution of each action in the prediction network output in the deep reinforcement learning model. (i.e., the policy distribution of actions at time step t+k) and the improved policy distribution π obtained by the Monte Carlo tree search module t+k The error between Among them, t represents the current time step, that is, the starting position of the training sample in this round, k represents the number of steps executed from the current time step t, and the improved strategy distribution π t+k Calculated by the following formula:

[0113]

[0114] Among them, π t+k (a) is the improved strategy distribution corresponding to action a, n(s t+k ,a) is the state s at time step t+k t+k The number of times action a is executed, ∑ a′ N(s t+k ,a′) is the state s at time step t+k t+k The number of times all actions are executed, a′ refers to any action.

[0115] Objective 2: In the deep reinforcement learning model, integrate the intermediate rewards with decay factors and the estimated value obtained by the Monte Carlo tree search module to update the predicted value. t Defined as:

[0116] z t =u t+1 +γu t+2 +...+γ n-1 u t+n +γ n v t+n

[0117] Among them, u t+irepresents the observation reward, which is the reward obtained during the actual interaction between the jammer and the radar during the confrontation simulation. After the jamming action is performed in the confrontation trajectory, the immediate reward returned by the environment (i.e., the observed jamming effect, such as whether it successfully deceives the radar, the shielding time, etc.) can be given by a rule evaluation module or by posterior observation statistics in implementation; γ is a discount factor, which is usually set in (0,1) and is used to regulate the importance of current rewards and future rewards; γ n-1 , γ n Indicates the power of n-1, n steps, which is used to weight the future reward or value of each step; n is the number of steps in which the reward is accumulated; v t+n Represents the estimated value obtained by the Monte Carlo Tree Search module. It is the state value estimate output by the prediction network when the Monte Carlo Tree Search (MCTS) is expanded to the leaf node of the t+nth step.

[0118] Following the same approach, starting from time step t+k, we accumulate rewards for n steps backward to obtain the predicted value z t+k :

[0119] z t+k =u t+k+1 +γu t+k+2 +...+γ n-1 u t+k+n +γ n v t+k+n

[0120] Among them, u t+k+i is the observation reward, i=1,…,n,v t+k+n The estimated value obtained by the Monte Carlo Tree Search module.

[0121] Then minimize the prediction value z of the prediction network t+k The error from the target value is: Among them, the target value is the state value output by the prediction network at time step t+k.

[0122] Goal 3: Minimize the error between the predicted reward and the actual observed reward: in, is the reward value predicted by the dynamic network at time step t+k, u t+k is the observation reward at time step t+k.

[0123] Combining the above three optimization objectives, we get the loss function:

[0124]

[0125] Among them, K is the number of time steps for each training sample to be predicted forward.

[0126] like Figure 2 As shown in , this is the interactive data generation and model training process. Figure 3 As shown in Figure 1, this is a schematic diagram of the working process of the three networks in the deep reinforcement learning model.

[0127] After model training is complete, in actual adversarial decision-making, a Monte Carlo tree search module is used to obtain probabilistic sampling to identify interference actions. The action selection during each simulation uses the Unified Combination (UCB) algorithm for path expansion. However, after the search, rather than selecting the action with the highest UCB score as the final output, a probability distribution is constructed based on the number of visits N(s, a) for each action at the root node. Probabilistic sampling is then performed using the improved strategy distribution described above, or a greedy strategy is used to select the action with the most visits as the interference decision result. This approach balances exploration and robustness, effectively improving the expressiveness of the actual interference strategy.

[0128] At the same time, a dynamic network is also used to make real-time interference effectiveness evaluations. That is, the high-dimensional hidden state of the current radar state and the decision-making action are fed into the dynamic network, and the reward value in the output is the current interference effectiveness evaluation.

[0129] The above-mentioned solution provided by the embodiment of the present invention introduces a deep reinforcement learning model based on the MuZero algorithm in the electronic countermeasure scenario. The greatest advantage of the MuZero algorithm is that it can complete decision optimization and performance evaluation in complex environments without explicit rule modeling. By learning environmental dynamics in the latent state space, the model can adapt to the dynamic changes of interference patterns and radar anti-interference strategies, avoiding the reliance of traditional methods on rule assumptions. In addition, by combining Monte Carlo tree search (MCTS) with deep learning, the model realizes real-time strategy optimization, and can quickly generate the optimal interference strategy in confrontation scenarios with unknown rules and complex environments, significantly improving the intelligence and efficiency of electronic countermeasure systems.

[0130] 3. Example description

[0131] In the one-on-one electronic countermeasure process between radar and jammer, the present invention evaluates the jamming effectiveness through the following steps:

[0132] 1. Build environment initialization: Based on the existing radar knowledge base and actual electronic countermeasure data, define the state, action space, and reward function of the radar and jammer to provide an environmental foundation for the training process.

[0133] 2. Characterize the radar environment: Through the characterization network, the radar observation data is mapped into the potential state space to construct an abstract radar environment that adapts to dynamic changes.

[0134] 3. Jammer strategy exploration and adversarial simulation: Using the Monte Carlo tree search module and dynamic network, a jammer simulator is constructed in the latent state space to generate a strategy to counter the radar (i.e., the jamming strategy), and predict the next potential state and reward to form a complete adversarial trajectory.

[0135] 4. Self-game and data collection: The jammer engages in self-competition with the radar in a simulated environment, dynamically adjusting its strategy to maximize the jamming benefit, while storing the confrontation data generated during the interaction into the replay buffer.

[0136] 5. Jointly train neural networks: Sample adversarial data from the replay buffer and use the prediction errors of reward, policy, and value functions to optimize the representation network, dynamic network, and prediction network to improve the model's adversarial capabilities.

[0137] 6. Evaluation and Application: After training, the strategy and effectiveness evaluation values ​​output by the model are used as the core basis for jamming strategy decision-making and effectiveness analysis in actual confrontations to ensure that the jamming effectiveness is accurate and reliable.

[0138] The above solution provided by the present invention can not only achieve efficient jamming effectiveness evaluation, but also provide dynamic and intelligent electronic countermeasure jamming strategy guidance.

[0139] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0140] Example 2

[0141] The present invention also provides a system for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, which is mainly used to implement the method provided in the above embodiments, such as Figure 4 As shown, the system mainly includes: a simulation unit, a training unit and an interference effectiveness evaluation unit; wherein:

[0142] The simulation unit is used to simulate the jammer and confront the radar to generate interaction data. During the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module are set up. Given an initial state, the deep reinforcement learning model maps it to the potential state space and uses it as the root node of the Monte Carlo tree to predict the state and reward value after executing each action under the initial state, as well as the policy distribution of the predicted state value and action. The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to the nodes in the Monte Carlo tree to make action decisions based on its own strategy, and repeats until it expands to the leaf node, completing a search process and obtaining a search trajectory. The search process is repeated multiple times, and all data is saved as interaction data.

[0143] A training unit, configured to train the deep reinforcement learning model using the data of the deep reinforcement learning model in the interaction data and the loss function constructed by the Monte Carlo tree search module;

[0144] The interference effectiveness evaluation unit is used to map the input state to the potential state space through the deep reinforcement learning model after training, predict the corresponding reward value, and use it as the interference effectiveness evaluation result.

[0145] Considering that the various technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0146] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0147] Example 3

[0148] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0149] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0150] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0151] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0152] The output device may be a display terminal;

[0153] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0154] Example 4

[0155] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0156] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0157] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, characterized in that: include: Simulate jammers and engage in confrontation with radars to generate interactive data; During the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module were set up; Given an initial state, the deep reinforcement learning model maps it to a latent state space and uses it as the root node of a Monte Carlo tree. It predicts the state and reward value after executing each action under the initial state, as well as the policy distribution of the predicted state value and action. The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model with the number of visits to each action under each node in the Monte Carlo tree. It makes action decisions based on its own strategy and repeats the search process until it reaches a leaf node. After the search is completed, the number of visits to each node during the search process is saved, the Monte Carlo tree search module calculates the improved policy distribution, and all data is saved as interaction data. Using the data of the deep reinforcement learning model and the data of the Monte Carlo tree search module in the interaction data to construct a loss function to train the deep reinforcement learning model; After training, the deep reinforcement learning model maps the input state to the latent state space, predicts the corresponding reward value, and uses it as the interference effectiveness evaluation result.

2. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 1 is characterized in that: The deep reinforcement learning model based on the MuZero algorithm includes: a representation network, a dynamic network, and a prediction network; wherein: Representation network, used to map states to latent state space Dynamic networks, used to predict the state and reward value after performing each action in a given state; Prediction network, used to predict the policy distribution of state value and action.

3. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 1, characterized in that: The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to the nodes in the Monte Carlo tree to make action decisions based on its own strategy, including: The action with the highest upper confidence interval algorithm score is selected using the following formula: Among them, a k The action with the highest score of the upper bound of the confidence interval selected for time step k; (s,a) represents the selection of action a in state s, (s,b) represents the selection of action b in state s, N(s,a) is the number of visits to action a in the node corresponding to state s, Q(s,a) is the average value of action a in the node corresponding to state s, and the policy distribution of each action output by the prediction network is predicted. c1 and c2 are constants.

4. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 3 is characterized in that: After completing a search process, a search trajectory is obtained, and the following update process is performed through the backtracking process: The number of visits to each action under each node is incremented by one, and the corresponding average value is updated, which is expressed as: N(s,a)←N(s,a)+1 Among them, ← is the assignment symbol, N(s,a) is the number of visits to action a under the node corresponding to state s, Q(s,a) is the average value of action a under the node corresponding to state s, and G is the cumulative reward obtained by simulating from the current node downward.

5. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 1, characterized in that: The loss function of the deep reinforcement learning model and the Monte Carlo tree search module constructed using the interactive data includes: The loss function is constructed by using the deep reinforcement learning model to predict the action policy distribution, predicted state value and reward value in the interaction data, as well as the improved policy distribution and obtained value of the Monte Carlo tree search module.

6. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 5, characterized in that: The process of constructing the loss function includes: The policy distribution of the deep reinforcement learning model predicting the action and the improved policy distribution of the Monte Carlo tree search module are denoted as π t+k , calculate the error between the two Where T is the transpose symbol, t represents the current time step, and k represents the number of steps executed starting from the current time step t. To predict the policy distribution of the action at time step t+k of the network output; Improve the policy distribution π t+k Calculated by the following formula: In the above formula, π t+k (a) is the improved strategy distribution corresponding to action a, N(s t+k ,a) is the state s at time step t+k t+k The number of times action a is executed, that is, state s t+k The number of visits to action a under the corresponding node, ∑ a′ N(s t+k ,a′) is the state s at time step t+k t+k The number of times all actions are executed, i.e. state s t+k The number of visits to all actions under the corresponding node, a′ refers to any action; The state value predicted by the deep reinforcement learning model and the value obtained by the Monte Carlo tree search module are denoted as z t+k , calculate the error between the two in, is the state value predicted by the prediction network at time step t+k, z t+k is the predicted value calculated using the value obtained by the Monte Carlo Tree Search module, expressed as: z t+k =u t+k+1 +γu t+k+2 +...+γ n-1 u t+k+n +γ n v t+k+n In the above formula, u t+k+i is the observation reward, which is the reward obtained during the actual interaction between the jammer and the radar during the confrontation simulation, i = 1,…,n, γ is the discount factor, n is the number of steps of reward accumulation, v t+k+n Estimated value for the Monte Carlo Tree Search module The reward value predicted by the deep reinforcement learning model and the reward value obtained by the Monte Carlo tree search module are denoted as u t+k , calculate the error between the two in, is the reward value predicted by the dynamic network at time step t+k, u t+k is the observation reward at time step t+k. Combining the three errors, we can obtain the loss function.

7. The method for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios according to claim 6, characterized in that: The loss function is expressed as: Among them, l t (θ) is the loss function, c‖θ‖ 2 is the regularization term, c is a constant, θ is the reinforcement learning model parameter, and ‖.‖ is the L2 norm.

8. A system for evaluating the effectiveness of unknown interference in irregular electronic countermeasure scenarios, characterized in that: The method for implementing any one of claims 1 to 7 comprises: a simulation unit, a training unit and an interference effectiveness evaluation unit; wherein: The simulation unit is used to simulate the jammer and confront the radar to generate interaction data. During the simulation process, a deep reinforcement learning model based on the MuZero algorithm and a Monte Carlo tree search module are set up. Given an initial state, the deep reinforcement learning model maps it to the potential state space and uses it as the root node of the Monte Carlo tree to predict the state and reward value after executing each action under the initial state, as well as the policy distribution of the predicted state value and action. The Monte Carlo tree search module combines the prediction results of the deep reinforcement learning model and the number of visits to the nodes in the Monte Carlo tree to make action decisions based on its own strategy, and repeats until it expands to the leaf node, completing a search process and obtaining a search trajectory. The search process is repeated multiple times, and all data is saved as interaction data. A training unit, configured to train the deep reinforcement learning model using the data of the deep reinforcement learning model in the interaction data and the loss function constructed by the Monte Carlo tree search module; The interference effectiveness evaluation unit is used to map the input state to the potential state space through the deep reinforcement learning model after training, predict the corresponding reward value, and use it as the interference effectiveness evaluation result.

9. A processing device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images