MCTS and self-game-based confrontation effectiveness evaluation method, system and equipment and medium

Abstract

The invention discloses an MCTS and self-gaming-based confrontation efficiency evaluation method, system and device and a medium, a self-gaming mechanism is introduced innovatively by combining Monte Carlo tree search and a deep neural network (agent), and high-quality training data is generated by simulating a real confrontation environment in the self-gaming process, so that the self-gaming efficiency is evaluated. Continuously optimizing a strategy network and a value network of the intelligent agent; according to the method, effective search and evaluation are carried out in a large state space, rapid change and complex feedback of opponent strategies can be adapted in real time, and through multi-round self-game simulation, the precision and the real-time performance of efficiency evaluation are improved, and reliable technical support is provided for strategy optimization in a complex confrontation environment.

Description

Technical Field

[0001] The present invention relates to the technical field of adversarial effectiveness evaluation, and in particular to an adversarial effectiveness evaluation method, system, device and medium based on MCTS and self-game. Background Art

[0002] In recent years, the development of intelligent technology has provided new solutions for radar countermeasure effectiveness evaluation. For example, artificial intelligence has been introduced into electronic warfare systems to improve the accuracy and efficiency of evaluation. Intelligent radar countermeasure systems can effectively cope with the challenges of complex electromagnetic environments by performing real-time signal processing and decision optimization through deep learning.

[0003] Although some studies have achieved initial results, overall, the current interference effectiveness assessment technology still needs to be improved. It has not yet fully utilized intelligent technology, the core technology is not mature enough, and the research directions are relatively scattered.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, device and medium for evaluating the effectiveness of confrontation based on MCTS and self-game, which not only improves the accuracy and real-time performance of performance evaluation, but also provides reliable technical support for strategy optimization in complex confrontation environments.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A confrontation effectiveness evaluation method based on MCTS and self-game, including:

[0008] Step 1: Build a radar simulator environment and build the jammer agent corresponding to the jammer;

[0009] Step 2: Use the jammer agent to conduct a self-game with the radar simulator environment, including: the jammer agent determines the optimal action based on the state information obtained from the radar simulator environment and MCTS, and acts on the radar simulator environment, continues to obtain state information from the radar simulator environment and makes the optimal action decision; stores the data generated by the self-game; wherein MCTS is Monte Carlo tree search;

[0010] Step 3: Use the stored data to train the interfering agent;

[0011] Step 4: Alternately execute steps 2 and 3 until the set stop condition is met, obtain the trained jammer intelligence, and deploy the trained jammer intelligence in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

[0012] A confrontation effectiveness evaluation system based on MCTS and self-game, including:

[0013] The environment and agent construction unit is used to construct the radar simulator environment and the jammer agent corresponding to the jammer;

[0014] The sample data collection unit based on MCTS and self-game is used to use the interference party agent to conduct self-game with the radar simulator environment, including: the interference party agent makes an optimal action based on the state information obtained from the radar simulator environment and in combination with MCTS, and acts on the radar simulator environment, and continues to obtain state information from the radar simulator environment and make action decisions; the data generated by the self-game are stored, including: the optimal action decided and the actual result of the confrontation; wherein MCTS is Monte Carlo tree search;

[0015] An agent training unit, used for training the interfering party agent using the stored data;

[0016] The performance evaluation unit is used to control the sample data collection unit based on MCTS and self-game and the agent training unit to work alternately until the set stop condition is met, so as to obtain the trained interference party intelligence, and deploy the trained interference party intelligence in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

[0017] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0020] It can be seen from the technical solution provided by the present invention that, by combining Monte Carlo tree search and deep neural network (agent), a self-game mechanism is innovatively introduced. During the self-game, high-quality training data is generated by simulating a real adversarial environment, and the strategy network and value network of the agent are continuously optimized. The present invention effectively searches and evaluates in a large state space (mainly referring to the large number of radar states in an adversarial environment), and can adapt to the rapid changes and complex feedback of the opponent's strategy in real time. Through multiple rounds of self-game simulation, not only the accuracy and real-time performance evaluation are improved, but also reliable technical support is provided for strategy optimization in complex adversarial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 A flowchart of a method for evaluating the effectiveness of a confrontation based on MCTS and self-game provided in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a self-game process based on MCTS provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a Monte Carlo tree search process provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of model training based on Monte Carlo tree search and self-game provided in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a simulation experiment provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a confrontation effectiveness evaluation system based on MCTS and self-game provided in an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0030] First, the terms that may be used in this article are explained as follows:

[0031] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0032] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0033] The following is a detailed description of a method, system, device and medium for evaluating the effectiveness of confrontation based on MCTS and self-game provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out in accordance with the conventional conditions in the field or the conditions recommended by the manufacturer. The instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0034] Embodiment 1

[0035] like Figure 1 As shown in FIG. 1 , a confrontation effectiveness evaluation method based on MCTS and self-game mainly includes the following steps:

[0036] Step 1: Build a radar simulator environment and build the jammer agent corresponding to the jammer.

[0037] Step 2: Use the jammer agent to play self-game with the radar simulator environment and store the data generated by the self-game.

[0038] In an embodiment of the present invention, the self-game process based on MCTS includes: the interfering party intelligent agent decides the optimal action based on the state information obtained from the radar simulator environment in combination with MCTS (Monte Carlo Tree Search), and acts in the radar simulator environment, continues to obtain state information from the radar simulator environment and makes the optimal action decision.

[0039] In an embodiment of the present invention, obtaining status information from a radar simulator environment includes: receiving a radar pulse signal in the radar simulator environment and extracting pulse-related parameters, combining all extracted parameters to obtain a pulse description word vector, and using the pulse description word vector as status information.

[0040] Preferably, MCTS decides the optimal action including: 1) performing a Monte Carlo tree search based on the state information obtained from the radar simulator environment, recursively selecting the most valuable child node starting from the root node until reaching the leaf node; wherein, the Monte Carlo tree search is implemented based on a tree model, the nodes in the tree model are associated with state information, and the edges are associated with actions, the root node is the node corresponding to the state information obtained from the radar simulator environment, and the information stored in each edge includes: the number of visits, the probability of action, the average value of the action and the total state value; the most valuable child node is evaluated by the information stored in the corresponding edge; 2) determining the optional action in combination with the corresponding state of the leaf node, and expanding it to the child node of the leaf node; 3) evaluating the action probability and state value of each child node through the interfering agent; 4) reversely updating the information stored in the corresponding edge in the search path; 5) using the information stored in the updated edge to select the optimal action.

[0041] In the embodiment of the present invention, the data stored in the self-game at least includes: each game result (that is, the actual result of the confrontation), the optimal action decided, etc.

[0042] Step 3: Use the stored data to train the interfering agent.

[0043] In the embodiment of the present invention, the interference party agent is trained by using the data generated by the self-game in a reinforcement learning manner.

[0044] Step 4: Alternately execute steps 2 and 3 until the set stop condition is met, obtain the trained jammer agent, and deploy the trained jammer agent in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

[0045] In an embodiment of the present invention, a stop condition may be set according to actual conditions. For example, a maximum number of times step 2 and step 3 are alternately executed may be set as a stop condition, or convergence of a loss function may be set as a stop condition.

[0046] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0047] 1. Introduction to electronic countermeasure scenarios.

[0048] Consider the situation of dynamic game between the two opposing parties. At this time, the state of the radar (Radar, abbreviated as R) is no longer discretized. In the same working mode, there are often multiple sets of different waveform parameters, and the radar will select different radar waveform parameters according to the target and environmental characteristics. The jammer (Jammer, abbreviated as J) receives the pulse signal of the radar and extracts its parameters, mainly including pulse width (Pulse Width, PW), pulse amplitude (Pulse Amplitude, PA), pulse repetition interval (Pulse Repetition Interval, PRI) and carrier frequency (Radio Frequency, RF), and then combines all parameters to obtain a pulse description word (Pulse Description Word, PDW) vector. The jammer receives the PDW, decides the jamming action according to a pre-set algorithm, and applies this action to the radar in the form of a signal. The radar adjusts the working mode and parameters in time to dynamically play a game with the jammer. The scene of the present invention will be constructed by a real radar simulation platform, so the radar's own state switching, parameter adjustment, and jamming action impact will be realized by the platform itself, and the scene will be modeled as a dual-perspective MDP (Markov decision process) of the radar and the jammer.

[0049] In the above scenario, the interference party’s perspective is:

[0050] {s J (t)∈S J ,a J (t)∈A J ,p J (s J (t+1)|s J (t),a J (t),s R (t),a R (t)),r J (s J (t),a J (t),s R (t),a R (t))}

[0051] The radar perspective is:

[0052] {s R (t)∈S R ,a R (t)∈A R ,p R (s R (t+1)|s R (t),a R (t),s J (t),a J (t)),rR (s J (t),a J (t),s R (t),a R (t))}

[0053] Among them, s J is the PDW vector (representing state information) of the radar signal received by the jammer after sorting, S J For a series of s J A collection of J is the interference action of the interferer (i.e. the action decided), A J is the set of interference actions, p J is the state transition of the radar, r j is the interference effectiveness, t represents the t time step; similarly, s T is the track of the interferer detected by the radar, including distance, angle, radial velocity and other related information. R For a series of s R A collection of R A is the anti-interference action taken by the radar. R Anti-interference action set, p R is the state transition of the interferer, r R It is the anti-interference performance of the radar.

[0054] Taking the interference party’s perspective as an example, the goal is to ensure that the interference party has the best situation in the current confrontation situation. The present invention uses the state value function to describe this. J (that is, the effectiveness of the interference) can only show the quality of the interference action at the current time step, but cannot reflect the quality of the interference strategy. Therefore, it is necessary to use the state value function to reflect the long-term value of the strategy.

[0055]

[0056] in, is the long-term reward of the interfering agent under strategy π, γ k is the discount factor, ranging from 0<γ k ≤1, the discount factor is used to reduce the importance of future rewards. The larger the value (i.e., the closer to 1), the more attention is paid to long-term rewards; the smaller the value, the more attention is paid to short-term rewards. The parameter k represents the consideration of long-term discounted rewards. The farther the state is, the smaller the impact is. s represents the state, s(t) = s J Indicates that the state considered in this part is s J .

[0057] When interference is performed according to a certain strategy, the state value function is expressed as:

[0058]

[0059] Among them, p J (s J ′ ∣s J ,a J ,s R ,a R ) indicates that the interferer is in state s J Take interference action a J , and the radar is in state s R Take anti-interference action a R After the interferer receives the state s J ′ The probability of J (s J ,a J ,s R ,a R ) indicates that the interferer is in state s J Take interference action a J , and the radar is in state s R Take anti-interference action a R After that, the interference effectiveness (i.e., reward) of the interferer is determined, and γ is the discount factor.

[0060] Finally, the optimal strategy of an MDP can be intuitively expressed by the following formula: J Next, we look for a function that can convert the value Approximate maximization strategy

[0061]

[0062] Similarly, the radar's goal is to find a strategy that maximizes the radar's value function. Its state value function is:

[0063]

[0064] in, is the state value function of the radar with strategy π, which is used to measure the long-term return of the radar, p R (s′ R ∣s J ,a J ,s R ,a R ) indicates that the interferer is in state s J Take interference action a J , and the radar is in state s R Take anti-interference action a R After the radar receives the state s′ RThe probability of R (s J ,a J ,s R ,a R ) indicates that the interferer is in state s J Take interference action a J , and the radar is in state s R Take anti-interference action a R Finally, determine the radar's countermeasure effectiveness.

[0065] Considering that the solution target of the above radar party corresponds to the solution target of the above interference party, the meaning of the relevant symbols can be understood together, so they will not be elaborated.

[0066] 2. Self-game process based on MCTS (Monte Carlo Tree Search).

[0067] In real environment, the effectiveness of the confrontation J In order to solve this problem, the present invention uses a confrontation effectiveness evaluation method based on Monte Carlo tree search and self-game, which combines Monte Carlo tree search and deep neural network for self-learning, and each search includes a series of simulated self-games.

[0068] like Figure 2 As shown in the figure, in the self-game stage, the jammer and the simulated radar confront each other. In each state, the jammer selects the optimal action through Monte Carlo tree search, guided by the current strategy and state value. In each confrontation process, all states, selected actions and final confrontation results are recorded. These confrontation data are stored and used to train the deep neural network, and the network parameters are updated by minimizing the strategy loss and value loss functions, thereby improving its performance in future confrontations. Monte Carlo tree search is implemented based on the tree model. In the tree model, nodes are associated with states, edges are associated with actions, and each edge is associated with the node it points to. It can be understood that a state associated with a node executes an action associated with an edge and then transfers to a state associated with another node. The information stored on each edge is: N(s,a), P(s,a), Q(s,a), W(s,a), where the meanings of the symbols are: number of visits N, average value Q, total state value W, and action probability P. The search process is the growth process of the tree model: starting from the root node (the node corresponding to the current state), select the progressive state of the action, expand the tree node and evaluate the utility until the search end condition is reached. The final decision is generated from the formed tree model and is the action with the highest utility value under the root node. The core of MCTS is the combination of optimal finite search and Monte Carlo valuation. It builds a tree model based on random sampling. This process involves repeated iterative searches until a certain constraint (such as computing time, maximum number of iterations, etc.) is reached.

[0069] Figure 2 In it, s is the state, p is the state value distribution, t is the time step, Tmax is the maximum time step, and z is the final result of the confrontation, such as interference success or failure.

[0070] like Figure 3 The figure shows a schematic diagram of the Monte Carlo tree search process, which mainly includes four stages. The following are the descriptions of each stage:

[0071] (1) Selection

[0072] Starting from the root node, recursively select the most valuable child node until a leaf node (i.e. a node without child nodes) is found. The selection strategy is based on the balance between the value of the child node, the prior probability and the number of visits. Specifically, the score of the action associated with each edge is calculated through the UCB (Upper Confidence Bound) algorithm, and the child node pointed to by the edge with the highest score is selected as the most valuable child node; the relevant calculation formula is expressed as:

[0073]

[0074] Among them, a K is the action corresponding to the edge with the highest score, K is the serial number of the corresponding action, Q(s,a), P(s,a), and N(s,a) are the information stored in the edge, which are the average value, action probability, and number of visits of the action, respectively. (s,a) means executing action a in state s; b N(s,b) represents the total number of visits to all actions in state s, b is an index symbol, referring to any action among all actions; C is a hyperparameter that controls the intensity of exploration and determines the weight of the exploration item in action selection. A larger c tends to explore new actions more actively.

[0075] (2) Expansion

[0076] Search according to the selection method described above until a leaf node is encountered. The leaf node is added to the search tree, and the optional actions in the current state (i.e. the state corresponding to the leaf node) are obtained by the mechanism of the environment. These optional actions are used to expand the child nodes, and the information of its child nodes is initialized: {N(s,a)=0,Q(s,a)=0,P(s,a)}.

[0077] (3) Evaluation

[0078] The action probability of each child node is evaluated through the policy network in the interfering agent, and the state value of each child node is evaluated through the value network in the interfering agent, and the action probability of each child node is saved on the edge between each child node and the leaf node.

[0079] Figure 3 In, p l ,(v l )=f θ (.) indicates that the state of the leaf node is input to the interfering agent f θ , and obtain the corresponding action probability distribution p l and value v l , action probability distribution p l Contains the state value p of two child nodes l,0 With p l,1 .

[0080] (4) Backpropagation

[0081] At the end of this simulation, W, Q, and N on the search path are updated. The evaluation value is back-propagated back to all nodes in the search path. For each visited node, its visit count is increased by 1, and its value is updated to the average value of the k simulations that have been performed.

[0082] Specifically: the search path includes all nodes and corresponding edges involved from the root node to the child node; the information stored in all edges in the search path is updated using the following formula:

[0083] N(s,a)←N(s,a)+1

[0084] W(s,a)←W(s,a)+v L

[0085]

[0086] Among them, ← is the assignment symbol, N(s,a) is the number of visits, W(s,a) is the total state value, v L is the state value of the child node; Q(s,a) is the average value of the action, and W(s,a) and N(s,a) on the right side of the formula are the updated information.

[0087] The above process will be repeated many times. After the MCTS search is completed, the visit_counts set is returned at the root node. After that, an action is selected based on the number of visits to the edge of the root node. Specifically, it will select the one with the most visits or sample the distribution derived from visit_counts to obtain the final execution action.

[0088] All training samples of the present invention are generated by self-game, and the strategy function and value function are continuously optimized during the training process. These two functions guide the Monte Carlo tree search to search, and at the same time, the MCTS search results are used to interact with the environment to generate training samples, and then the objective function is optimized. Through the gradual improvement of this strategy, higher quality training data can be continuously obtained.

[0089] 3. Model training based on Monte Carlo tree search and self-game.

[0090] like Figure 4 As shown, it is an overview of the model training based on Monte Carlo tree search and self-game. The upper part is the training process of optimizing the policy network through cross entropy loss and optimizing the value network through mean square error loss. The lower part is the reasoning process of giving actions when interfering with the party during electronic confrontation.

[0091] In the training process, data samples are first obtained through self-game, which is the scheme introduced in the second part above. In the self-game, the root node of the Monte Carlo search tree is first constructed according to the current state, and the UCB score is calculated using the policy network and the value network, and the action with the highest score is selected until a leaf node is encountered. The leaf node is expanded, and the node information of the path is reversely updated according to the output of the value network and multiple simulations are performed. Execute actions in multiple states. Then, when a round of confrontation is over, the sample data is stored in the experience replay pool, and samples are sampled from it for training reinforcement learning cross entropy loss for optimization. With the continuous improvement of strategies and value functions, the training samples are also constantly improving, so that the ability of the trained intelligent agent (implemented using a deep neural network model) is gradually enhanced.

[0092] In the embodiment of the present invention, the following loss function loss is used to optimize the strategy network and value network in the agent:

[0093] loss = l v (z t ,v t )+l p (π t ,p t )+c||θ|| 2

[0094] Among them, c||θ|| 2 is the square norm (L2 norm) of the interference agent parameter θ, also known as the weight penalty. c is a hyperparameter that controls the strength of regularization. The larger the value, the more strictly the scale of parameter θ is restricted. v (z t ,v t )=(z t -v t ) 2 , l v (z t ,v t ) is the loss function of the value network (using the mean square error loss function), T is the transposed symbol, v t is the state value output by the value network, z t is the data saved in the self-game, which refers to the actual result of the confrontation (i.e. the game result), l p (π t ,p t ) is the loss function of the policy network (using the cross entropy loss function), π t is the strategy obtained by MCTS, that is, the optimal action decided from the game, p t is the action probability distribution output by the policy network.

[0095] The second and third parts are executed alternately, so that the interfering agent and the MCTS-based self-game process can be improved together.

[0096] The above method provided by the embodiment of the present invention can be applied to the complex environment of the dynamic game of strategies of the adversaries. By combining Monte Carlo tree search and deep neural network, the method can effectively search and evaluate the adversarial effectiveness in a large state space. In the self-game process, the interfering agent conducts multiple rounds of confrontation with the radar simulator environment to generate high-quality training data and continuously optimize the strategy network and value network. In the decision-making process, MCTS searches by selecting high-value, low-access actions and updates node information (nodes and edges are interconnected) through back propagation, thereby providing a dynamic performance evaluation and optimization solution. This method not only improves the accuracy and real-time performance evaluation, but also provides reliable technical support for strategy optimization in complex adversarial environments.

[0097] 4. Example explanation.

[0098] like Figure 5 As shown, an example of a simulation experiment is provided. During the simulation experiment, it is mainly used to provide a description of the radar status. In this example, the simulation platform is implemented based on the MATLAB language, and the entire method is implemented based on the Python language, and there are platform barriers in data interaction. During the training process, the platform acts as the environment, the interferer acts as the agent, and the data interaction between the two parties is transmitted through Redis. Redis is an open source in-memory database that stores data in key-value pairs. Because the data is stored in memory, Redis has very fast read and write speeds and can process millions of requests per second. Through Redis, the agent can interact with a large amount of data in the environment during training. The main process of this example is as follows:

[0099] (1) Before the training phase, a radar environment is constructed based on the existing radar knowledge base.

[0100] (2) During the training process, MCTS is used to construct an interference agent to simulate the confrontation with the radar.

[0101] (3) The jammer agent collects training data into the Buffer by playing self-game with the constructed radar simulator environment.

[0102] (4) Sample data from the buffer to train the interfering agent, and alternately execute steps (3) and (4) until the stopping condition is met.

[0103] (5) After the training, in the actual confrontation (applied in the real environment), the trained interfering agent makes the optimal action decision and obtains the state value corresponding to the optimal action at the same time. The state value is used as an immediate reward, which is the evaluation result of the confrontation effectiveness.

[0104] Those skilled in the art can understand that the interfering agent includes a strategy network and a value network, wherein the strategy network is responsible for deciding the optimal action, and the value network is responsible for outputting the corresponding state value.

[0105] In the above example, according to the MCTS-based self-game process and training, accurate confrontation effectiveness can be obtained and the interference process of electronic confrontation can be guided. Figure 5 The functions involved in multiple processing processes are listed in the figure. For example, collect() under the collector is a sample collection function, which means collecting sample data generated by the game process. Since the functions involved are all common functions in this field, their meanings can be understood by referring to conventional technologies, they will not be elaborated on.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0107] Embodiment 2

[0108] The present invention also provides a confrontation effectiveness evaluation system based on MCTS and self-game, which is mainly used to implement the method provided in the above embodiment, such as Figure 6 As shown, the system mainly includes:

[0109] The environment and agent construction unit is used to construct the radar simulator environment and the jammer agent corresponding to the jammer;

[0110] The sample data collection unit based on MCTS and self-game is used to use the interference party agent to conduct self-game with the radar simulator environment, including: the interference party agent makes an optimal action based on the state information obtained from the radar simulator environment and in combination with MCTS, and acts on the radar simulator environment, and continues to obtain state information from the radar simulator environment and make action decisions; the data generated by the self-game are stored, including: the optimal action decided and the actual result of the confrontation; wherein MCTS is Monte Carlo tree search;

[0111] An agent training unit, used for training the interfering party agent using the stored data;

[0112] The performance evaluation unit is used to control the sample data collection unit based on MCTS and self-game and the agent training unit to work alternately until the set stop condition is met, so as to obtain the trained interference party intelligence, and deploy the trained interference party intelligence in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

[0113] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0114] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0115] Embodiment 3

[0116] The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0117] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0118] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0119] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0120] The output device may be a display terminal;

[0121] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0122] Embodiment 4

[0123] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0124] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0125] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: include: Step 1: Build a radar simulator environment and build the jammer agent corresponding to the jammer; Step 2: Use the jammer agent to conduct a self-game with the radar simulator environment, including: the jammer agent determines the optimal action based on the state information obtained from the radar simulator environment and MCTS, and acts on the radar simulator environment, continues to obtain state information from the radar simulator environment and makes the optimal action decision; stores the data generated by the self-game; wherein MCTS is Monte Carlo tree search; Step 3: Use the stored data to train the interfering agent; Step 4: Alternately execute steps 2 and 3 until the set stop condition is met, obtain the trained jammer intelligence, and deploy the trained jammer intelligence in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

2. According to claim 1, a method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: The obtaining of status information from the radar simulator environment comprises: A radar pulse signal in a radar simulator environment is received and pulse-related parameters are extracted, all extracted parameters are combined to obtain a pulse description word vector, and the pulse description word vector is used as state information.

3. According to claim 1, a method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: The interference agent uses the state information obtained from the radar simulator environment and combines MCTS to decide the optimal action, including: A Monte Carlo tree search is performed based on the state information obtained from the radar simulator environment, and the most valuable child node is recursively selected from the root node until a leaf node is reached; wherein the Monte Carlo tree search is implemented based on a tree model, in which nodes in the tree model are associated with state information, and edges are associated with actions, the root node is the node corresponding to the state information obtained from the radar simulator environment, and the information stored in each edge includes: the number of visits, the probability of an action, the average value of the action, and the total state value; the most valuable child node is evaluated by the information stored in the corresponding edge; Combine the optional actions corresponding to the current state and expand them into child nodes of the leaf node; The interfering agent evaluates the action probability and state value of each child node; Reversely update the information stored in the corresponding edge in the search path; Use the information stored in the updated edges to select the best action.

4. According to claim 3, a method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: The recursive selection of the most valuable child node includes: calculating the score of the action associated with each edge by an upper confidence interval algorithm, and selecting the child node pointed to by the edge with the highest score as the most valuable child node; The relevant calculation formula is expressed as: Among them, a k is the action corresponding to the edge with the highest score, Q(s,a), P(s,a), and N(s,a) are the information stored in the edge, which are the average value, action probability, and number of visits of the action, respectively. (s,a) means executing action a in state s; ∑ b N(s,b) represents the total number of visits to all actions in state s, b is an index symbol, referring to any action among all actions; C is a hyperparameter.

5. According to claim 2, a method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: The evaluation of the action probability and state value of each child node by the interfering agent includes: The action probability of each child node is evaluated through the policy network in the interfering agent, and the state value of each child node is evaluated through the value network in the interfering agent; The action probability of each child node is saved on the edge between each child node and the leaf node.

6. According to claim 2, a method for evaluating the effectiveness of confrontation based on MCTS and self-game, characterized in that: The information stored in the corresponding edge in the reverse update search path includes: The search path includes all nodes and corresponding edges involved from the root node to the child node; The information stored in all edges in the update search path includes: update visit count, average value of actions and total state value; the update method is expressed as: N(s,a)←N(s,a)+1 W(s,a)←W(s,a)+v L Among them, ← is the assignment symbol, N(s,a) is the number of visits, W(s,a) is the total state value, v L is the state value of the child node, Q(s,a) is the average value of the action, (a,a) means executing action a in state s, and W(s,a) and N(s,a) on the right side of the formula are the updated information.

7. The method for evaluating the effectiveness of confrontation based on MCTS and self-game according to claim 1, characterized in that: The method of using the stored data to train the interfering party agent comprises: The following loss function loss is used to optimize the policy network and value network in the interfering agent: loss=l v (z t ,v t )+l p (π t ,p t )+c||θ|| 2 Among them, θ is the interference agent parameter, c is the hyperparameter, l v (z t ,v t )=(z t -v t ) 2 , l v (z t ,v t ) is the loss function of the value network, l p (π t ,p t ) is the loss function of the policy network, v t is the state value output by the value network, p t is the action probability distribution output by the policy network, z t and π t All of them are data generated from the game, which are the game results and the optimal actions decided in turn.

8. A confrontation effectiveness evaluation system based on MCTS and self-game, characterized in that: include: The environment and agent construction unit is used to construct the radar simulator environment and the jammer agent corresponding to the jammer; The sample data collection unit based on MCTS and self-game is used to use the interference party agent to conduct self-game with the radar simulator environment, including: the interference party agent makes an optimal action based on the state information obtained from the radar simulator environment and combines MCTS to act on the radar simulator environment, and continues to obtain state information from the radar simulator environment and make action decisions; the data generated by the self-game are stored, including: the optimal action decided and the actual result of the confrontation; wherein MCTS is Monte Carlo tree search; An agent training unit, used for training the interfering party agent using the stored data; The performance evaluation unit is used to control the sample data collection unit based on MCTS and self-game and the agent training unit to work alternately until the set stop condition is met, so as to obtain the trained interference party intelligence, and deploy the trained interference party intelligence in the actual confrontation scenario to output the optimal action and the corresponding state value, and use the state value as the evaluation result of the confrontation effectiveness.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images