A method and device for generating signal-level radar interference in an open environment

Abstract

The present invention discloses a signal-level radar interference generation method and device in an open environment, the method comprising: obtaining radar interference data information; the radar interference data information comprises radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space; the radar radiation source description word information comprises carrier frequency information, pulse interval information, pulse width information and radar state judgment information; processing the radar interference data information to obtain state information S, interference pattern action space A, interference pattern parameter space B; setting the interference effectiveness reward value and the termination state, using a preset radar interference generation model, processing the radar interference data information to be processed, and obtaining radar interference signal information. The method of the present invention uses a large amount of real-time online radar confrontation training, and the training algorithm is from the radar signal to the underlying logic of the output interference signal, so as to effectively deal with the radar target confrontation problem in an unknown and small sample environment.

Description

Technical Field

[0001] The present invention relates to the field of electromagnetic space security technology, and in particular to a method and device for generating signal-level radar interference in an open environment. Background Art

[0002] Radar jamming is a jamming method widely used in the field of modern confrontation. Traditional radar jamming strategies are based on preset databases, templates or a large number of pre-prepared training sample data to achieve target recognition and jamming. Traditional electromagnetic environment perception and threat identification methods rely on the collection and analysis of a large amount of known target data, obtain target features through analysis, form recognition templates, and identify targets based on template matching during wartime to guide the implementation of jamming.

[0003] In the open environment of actual confrontation, we are more likely to face the problem of "small sample" and "zero sample", which is mainly reflected in the following two aspects: a large amount of data collected in peacetime is unlabeled and incomplete, and a small amount of useful signals are submerged in a large amount of garbage and useless information, and high-quality data is characterized by small samples; wartime threats are mostly unknown targets, the system lacks prior knowledge, and only a very small number of signal samples can be obtained online; in actual confrontation, it is very likely that target signals that are not collected in peacetime will appear, and the pre-set sample library or the intelligent confrontation model obtained based on normal training cannot be used. For electronic warfare applications with small samples, the traditional electronic warfare database construction method based on data analysis, and the existing primary artificial intelligence algorithms cannot solve the real-time online recognition and confrontation problems well. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for generating signal-level radar interference in an open environment. Based on the idea of ​​reinforcement learning, a large number of real-time online confrontation games are conducted with radar signals to generate radar signal interference and corresponding parameters. The algorithm model learns the underlying logic from radar signals to interference signals through a large number of trainings. The trained generation model has a certain robustness in the game confrontation against unknown targets. The present invention first proposes a signal-level intelligent interference generation method in an open electromagnetic confrontation environment; and then proposes a training method for the signal-level intelligent electromagnetic spectrum interference generation method.

[0005] In order to solve the above technical problems, a first aspect of an embodiment of the present invention discloses a method for generating signal-level radar interference in an open environment, the method comprising:

[0006] S1, obtain radar interference data information;

[0007] The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space;

[0008] The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information;

[0009] S2, processing the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B;

[0010] S3, setting the interference effectiveness reward value and the termination state, and processing the radar interference data information to be processed using a preset radar interference generation model according to the state information S, the interference pattern action space A and the interference pattern parameter space B to obtain radar interference signal information.

[0011] As an optional implementation, in the first aspect of the embodiment of the present invention, the state information S includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information, s t ∈S,s t is the radar status at time t;

[0012] The interference pattern action space A includes noise information, false target information and movement information;

[0013] The interference pattern parameter space B includes starting distance information, stay time information and suppression distance information.

[0014] As an optional implementation, in the first aspect of the embodiment of the present invention, the radar interference generation model includes a Q function network and an interference pattern network;

[0015] The Q function network includes a first linear layer with a size of 82×64, a second linear layer with a size of 64×32, and a third linear layer with a size of 32×3, and the activation function is ReLU;

[0016] The first linear layer is used to receive the flattened 74-dimensional radar emitter description word information and 8-dimensional action parameter information;

[0017] The output of the first linear layer is input into the second linear layer;

[0018] The output of the second linear layer is input into a third linear layer;

[0019] The output of the third linear layer is the Q value of 4 discrete actions;

[0020] The interference pattern network includes a fourth linear layer of size 74×8, a fifth linear layer of size 74×64, a sixth linear layer of size 64×32, a seventh linear layer of size 32×8 and a tanh layer, and the activation function is ReLU;

[0021] The fourth linear layer and the fifth linear layer are used to input the flattened 74-dimensional radar radiation source description word information;

[0022] The output of the fifth linear layer is input into the sixth linear layer;

[0023] The output of the sixth linear layer is input into the seventh linear layer;

[0024] The output of the fourth linear layer and the output of the seventh linear layer are added to obtain 8-dimensional action parameter information;

[0025] The tanh layer is used to truncate the 8-dimensional action parameter information to obtain the action parameter information within (-1, 1).

[0026] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the setting of the interference effectiveness reward value and the termination state, using a preset radar interference generation model, processing the radar interference data information to be processed, and obtaining the radar interference signal information, includes:

[0027] S31, initialize the parameters θ of the interference style network μ, the network parameters ω of the Q function network, the exploration hyperparameter ε and the moving average hyperparameter τ;

[0028] S32, taking the exploration hyperparameter ε as the probability, randomly selects a interference pattern k, a total of K interference patterns, and randomly samples all parameters x corresponding to all interference patterns, and takes out the parameters x corresponding to interference pattern k k , x k ∈X(k), X(k) is the interference pattern space;

[0029] a t =(k,x k ), all t =(k,x)

[0030] In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern k k The action that constitutes t The action composed of the interference pattern k and all parameters x corresponding to all interference patterns at time t, t = 1, 2, ..., T, T is the termination state;

[0031] S33, in state s t Execute action a t , get the state s t ′ and reward value r t , the quaternion (s t ,all t ,s t ′,r t ) join the playback pool;

[0032] S34, randomly selecting b quadruplets from the playback pool, where b is a positive integer;

[0033] S35, using the b quadruplets to train the preset radar interference generation model to obtain a radar interference generation optimization model;

[0034] S36, using the radar interference generation optimization model to process the radar interference data information to be processed to obtain radar interference signal information.

[0035] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the using the b quadruplets to train the preset radar interference generation model to obtain the radar interference generation optimization model includes:

[0036] S351, using the value function y j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0037] The value function y j The expression is:

[0038] y j =r j +γm k QUR ω (s′ j ,k,μ θ (s j ))

[0039] In the formula, r j is the jth reward value, γ is the weight function, Q ω (s′ j ,k,μ θ (s j )) is the Q value expression of the Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0040] S352, using the parameter x′ jUpdating the network parameters of the Q function network to obtain an updated Q function network;

[0041] S353, using the value function y j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0042] S354, integrating the updated Q function network and the updated interference pattern network to obtain a radar interference generation optimization model.

[0043] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the radar interference generation optimization model is used to process the radar interference data information to be processed to obtain radar interference signal information, including:

[0044] S361, using a standardized processing model, performing standardized processing on the radar interference data information to be processed to obtain standardized radar interference data information;

[0045] The standardized processing model expression is:

[0046]

[0047] In the formula, x is the original input, x stand is the standardized radar interference data information, μ is the mean of x, σ is the standard deviation of x;

[0048] S362, using the radar interference generation optimization model, processing the standardized radar interference data information to obtain radar interference signal information; the radar interference signal information is the action that maximizes the Q value of the Q function network.

[0049] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the method of using the b quadruple to train the preset radar interference generation model to obtain the radar interference generation optimization model also includes:

[0050] Using the value function y 1j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0051] The value function y 1j The expression is:

[0052]

[0053] In the formula, r j is the jth reward value, γ is the weight function, Q ω1 (s′ j ,k,μ θ (s j )) is the first Q value expression of the first Q function network, Q ω2 (s′ j ,k,μ θ (s j )) is the second Q value expression of the second Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0054] Using the parameter x′ j Updating network parameters of the first Q function network and the second Q function network to obtain an updated Q function network; the Q function network envelops the first Q function network and the second Q function network;

[0055] Using the value function y 1j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0056] The updated Q-function network and the updated interference pattern network are integrated to obtain a radar interference generation optimization model.

[0057] A second aspect of an embodiment of the present invention discloses a signal-level radar interference generating device in an open environment, the device comprising:

[0058] An information acquisition module, used to obtain radar interference data information;

[0059] The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space;

[0060] The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information;

[0061] An information processing module, used to process the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B;

[0062] The radar jamming signal generation module is used to set the jamming effectiveness reward value and the termination state, and to process the radar jamming data information to be processed according to the state information S, the jamming pattern action space A and the jamming pattern parameter space B using a preset radar jamming generation model to obtain radar jamming signal information.

[0063] As an optional implementation, in the second aspect of the embodiment of the present invention, the state information S includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information, s t ∈S,s t is the radar status at time t;

[0064] The interference pattern action space A includes noise information, false target information and movement information;

[0065] The interference pattern parameter space B includes starting distance information, stay time information and suppression distance information.

[0066] As an optional implementation, in the second aspect of the embodiment of the present invention, the radar interference generation model includes a Q function network and an interference pattern network;

[0067] The Q function network includes a first linear layer with a size of 82×64, a second linear layer with a size of 64×32, and a third linear layer with a size of 32×3, and the activation function is ReLU;

[0068] The first linear layer is used to receive the flattened 74-dimensional radar emitter description word information and 8-dimensional action parameter information;

[0069] The output of the first linear layer is input into the second linear layer;

[0070] The output of the second linear layer is input into a third linear layer;

[0071] The output of the third linear layer is the Q value of 4 discrete actions;

[0072] The interference pattern network includes a fourth linear layer of size 74×8, a fifth linear layer of size 74×64, a sixth linear layer of size 64×32, a seventh linear layer of size 32×8 and a tanh layer, and the activation function is ReLU;

[0073] The fourth linear layer and the fifth linear layer are used to input the flattened 74-dimensional radar radiation source description word information;

[0074] The output of the fifth linear layer is input into the sixth linear layer;

[0075] The output of the sixth linear layer is input into the seventh linear layer;

[0076] The output of the fourth linear layer and the output of the seventh linear layer are added to obtain 8-dimensional action parameter information;

[0077] The tanh layer is used to truncate the 8-dimensional action parameter information to obtain the action parameter information within (-1, 1).

[0078] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the setting of the interference effectiveness reward value and the termination state, using a preset radar interference generation model, processing the radar interference data information to be processed, and obtaining the radar interference signal information, includes:

[0079] S31, initialize the parameters θ of the interference style network μ, the network parameters ω of the Q function network, the exploration hyperparameter ε and the moving average hyperparameter τ;

[0080] S32, taking the exploration hyperparameter ε as the probability, randomly selects a interference pattern k, a total of K interference patterns, and randomly samples all parameters x corresponding to all interference patterns, and takes out the parameters x corresponding to interference pattern k k , x k ∈X(k), X(k) is the interference pattern space;

[0081] a t =(k,x k ), all t =(k,x)

[0082] In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern k k The action that constitutes t The action composed of the interference pattern k and all parameters x corresponding to all interference patterns at time t, t = 1, 2, ..., T, T is the termination state;

[0083] S33, in state s t Execute action a t , get the state s t ′ and reward value r t , the quaternion (s t ,all t ,s t ′,r t ) join the playback pool;

[0084] S34, randomly selecting b quadruplets from the playback pool, where b is a positive integer;

[0085] S35, using the b quadruplets to train the preset radar interference generation model to obtain a radar interference generation optimization model;

[0086] S36, using the radar interference generation optimization model to process the radar interference data information to be processed to obtain radar interference signal information.

[0087] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the using the b quadruplets to train the preset radar interference generation model to obtain the radar interference generation optimization model includes:

[0088] S351, using the value function y j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0089] The value function y j The expression is:

[0090] y j =r j +γm k QUR ω (s′ j ,k,μ θ (s j ))

[0091] In the formula, r j is the jth reward value, γ is the weight function, Q ω (s′ j ,k,μ θ (s j )) is the Q value expression of the Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0092] S352, using the parameter x′ j Updating the network parameters of the Q function network to obtain an updated Q function network;

[0093] S353, using the value function y j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0094] S354, integrating the updated Q function network and the updated interference pattern network to obtain a radar interference generation optimization model.

[0095] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the radar interference generation optimization model is used to process the radar interference data information to be processed to obtain radar interference signal information, including:

[0096] S361, using a standardized processing model, performing standardized processing on the radar interference data information to be processed to obtain standardized radar interference data information;

[0097] The standardized processing model expression is:

[0098]

[0099] In the formula, x is the original input, x stand is the standardized radar interference data information, μ is the mean of x, σ is the standard deviation of x;

[0100] S362, using the radar interference generation optimization model, processing the standardized radar interference data information to obtain radar interference signal information; the radar interference signal information is the action that maximizes the Q value of the Q function network.

[0101] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the method of using the b quadruple to train the preset radar interference generation model to obtain the radar interference generation optimization model also includes:

[0102] Using the value function y 1j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0103] The value function y 1j The expression is:

[0104] y 1j =r j +γmin{m k QUR ω1 (s′ j ,k,μ θ (s j )),m k QUR ω2 (s′j ,k,μ θ (s j ))}

[0105] In the formula, r j is the jth reward value, γ is the weight function, Q ω1 (s′ j ,k,μ θ (s j )) is the first Q value expression of the first Q function network, Q ω2 (s′ j ,k,μ θ (s j )) is the second Q value expression of the second Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0106] Using the parameter x′ j Updating network parameters of the first Q function network and the second Q function network to obtain an updated Q function network; the Q function network envelops the first Q function network and the second Q function network;

[0107] Using the value function y 1j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0108] The updated Q-function network and the updated interference pattern network are integrated to obtain a radar interference generation optimization model.

[0109] A third aspect of the present invention discloses another device for generating signal-level radar interference in an open environment, the device comprising:

[0110] A memory storing executable program code;

[0111] a processor coupled to the memory;

[0112] The processor calls the executable program code stored in the memory to execute part or all of the steps in the method for generating signal-level radar interference in an open environment disclosed in the first aspect of an embodiment of the present invention.

[0113] The fourth aspect of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the signal-level radar interference generation method in an open environment disclosed in the first aspect of an embodiment of the present invention.

[0114] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0115] The advantage of the method of the present invention is that it adopts the method of reinforcement learning and hybrid action space, that is, through a large number of real-time online radar confrontation training, the training algorithm is trained from the underlying logic of radar signal to output interference signal, so as to effectively deal with the problem of radar target confrontation in unknown and small sample environments. In addition, the present invention provides an intelligent algorithm and training method for signal-level radar confrontation. The algorithm input is a radar signal description word, and supports signal-level parameter output, which ensures that the algorithm improves its exploration efficiency under the condition of fully exploring the space until the optimal interference is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0117] Figure 1 It is a flow chart of a method for generating signal-level radar interference in an open environment disclosed in an embodiment of the present invention;

[0118] Figure 2 is a schematic diagram of the structure of a radar interference generation model disclosed in an embodiment of the present invention;

[0119] Figure 3 is a cumulative reward curve diagram for every 20 steps disclosed in an embodiment of the present invention;

[0120] Figure 4 is a cumulative reward curve diagram for every 20 steps with and without duel network disclosed in the embodiment of the present invention;

[0121] Figure 5 It is a cumulative reward curve diagram of every 20 steps in the one-hot format disclosed in an embodiment of the present invention;

[0122] Figure 6 It is a cumulative reward curve chart of every 20 steps after standardizing the data disclosed in the embodiment of the present invention;

[0123] Figure 7 is a curve diagram of cumulative rewards for every 20 steps after using the terminal disclosed in an embodiment of the present invention;

[0124] Figure 8 It is a schematic diagram of the structure of a signal-level radar interference generating device in an open environment disclosed in an embodiment of the present invention;

[0125] Fig. 9 It is a schematic diagram of the structure of another signal-level radar interference generating device in an open environment disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0126] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0127] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0128] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0129] The present invention discloses a method and device for generating signal-level radar interference in an open environment, the method comprising: obtaining radar interference data information; the radar interference data information comprises radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space; the radar radiation source description word information comprises carrier frequency information, pulse interval information, pulse width information and radar state judgment information; processing the radar interference data information to obtain state information S, interference pattern action space A, interference pattern parameter space B; setting the interference effectiveness reward value and the termination state, and using the preset radar interference generation model to process the radar interference data information to be processed to obtain radar interference signal information. The method of the present invention uses a large amount of real-time online radar confrontation training, and the training algorithm is from the underlying logic of radar signal to output interference signal, so as to effectively deal with the problem of radar target confrontation in unknown and small sample environments. The following are detailed descriptions.

[0130] Embodiment 1

[0131] See also Figure 1 , Figure 1 1 is a flow chart of a method for generating signal-level radar interference in an open environment disclosed in an embodiment of the present invention. Figure 1 The signal-level radar interference generation method in an open environment described is used to improve its exploration efficiency under the condition of fully exploring the space until the optimal interference signal is generated, which is not limited in the embodiments of the present invention. Figure 1 As shown, the signal-level radar interference generation method may include the following operations:

[0132] S1, obtain radar interference data information;

[0133] The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space;

[0134] The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information;

[0135] S2, processing the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B;

[0136] S3, setting the interference effectiveness reward value and the termination state, and processing the radar interference data information to be processed using a preset radar interference generation model according to the state information S, the interference pattern action space A and the interference pattern parameter space B to obtain radar interference signal information.

[0137] Optionally, the state information S includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information, s t∈S,s t is the radar status at time t;

[0138] The interference pattern action space A includes noise information, false target information and movement information;

[0139] The interference pattern parameter space B includes starting distance information, stay time information and suppression distance information.

[0140] Optionally, the radar interference generation model includes a Q function network and an interference pattern network;

[0141] The Q function network includes a first linear layer with a size of 82×64, a second linear layer with a size of 64×32, and a third linear layer with a size of 32×3, and the activation function is ReLU;

[0142] The first linear layer is used to receive the flattened 74-dimensional radar emitter description word information and 8-dimensional action parameter information;

[0143] The output of the first linear layer is input into the second linear layer;

[0144] The output of the second linear layer is input into a third linear layer;

[0145] The output of the third linear layer is the Q value of 4 discrete actions;

[0146] The interference pattern network includes a fourth linear layer of size 74×8, a fifth linear layer of size 74×64, a sixth linear layer of size 64×32, a seventh linear layer of size 32×8 and a tanh layer, and the activation function is ReLU;

[0147] The fourth linear layer and the fifth linear layer are used to input the flattened 74-dimensional radar radiation source description word information;

[0148] The output of the fifth linear layer is input into the sixth linear layer;

[0149] The output of the sixth linear layer is input into the seventh linear layer;

[0150] The output of the fourth linear layer and the output of the seventh linear layer are added to obtain 8-dimensional action parameter information;

[0151] The tanh layer is used to truncate the 8-dimensional action parameter information to obtain the action parameter information within (-1, 1).

[0152] Optionally, the setting of the interference effectiveness reward value and the termination state, using a preset radar interference generation model, processing the radar interference data information to be processed, and obtaining radar interference signal information, includes:

[0153] S31, initialize the parameters θ of the interference style network μ, the network parameters ω of the Q function network, the exploration hyperparameter ε and the moving average hyperparameter τ;

[0154] S32, taking the exploration hyperparameter ε as the probability, randomly selects a interference pattern k, a total of K interference patterns, and randomly samples all parameters x corresponding to all interference patterns, and takes out the parameters x corresponding to interference pattern k k , x k ∈X(k), X(k) is the interference pattern space;

[0155] a t =(k,x k ), all t =(k,x)

[0156] In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern k k The action that constitutes t The action composed of the interference pattern k and all parameters x corresponding to all interference patterns at time t, t = 1, 2, ..., T, T is the termination state;

[0157] S33, in state s t Execute action a t , get the state s t ′ and reward value r t , the quaternion (s t ,all t ,s t ′,r t ) join the playback pool;

[0158] S34, randomly selecting b quadruplets from the playback pool, where b is a positive integer;

[0159] S35, using the b quadruplets to train the preset radar interference generation model to obtain a radar interference generation optimization model;

[0160] S36, using the radar interference generation optimization model to process the radar interference data information to be processed to obtain radar interference signal information.

[0161] Optionally, the using the b quadruplets to train the preset radar interference generation model to obtain the radar interference generation optimization model includes:

[0162] S351, using the value function y j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state sj The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0163] The value function y j The expression is:

[0164] y j =r j +γm k QUR ω (s′ j ,k,μ θ (s j ))

[0165] In the formula, r j is the jth reward value, γ is the weight function, Q ω (s′ j ,k,μ θ (s j )) is the Q value expression of the Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0166] S352, using the parameter x′ j Updating the network parameters of the Q function network to obtain an updated Q function network;

[0167] S353, using the value function y j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0168] S354, integrating the updated Q function network and the updated interference pattern network to obtain a radar interference generation optimization model.

[0169] Optionally, the using the radar interference to generate the optimization model to process the radar interference data information to be processed to obtain radar interference signal information includes:

[0170] S361, using a standardized processing model, performing standardized processing on the radar interference data information to be processed to obtain standardized radar interference data information;

[0171] The standardized processing model expression is:

[0172]

[0173] In the formula, x is the original input, x stand is the standardized radar interference data information, μ is the mean of x, σ is the standard deviation of x;

[0174] S362, using the radar interference generation optimization model, processing the standardized radar interference data information to obtain radar interference signal information; the radar interference signal information is the action that maximizes the Q value of the Q function network.

[0175] Optionally, the method of using the b quadruplets to train the preset radar interference generation model to obtain the radar interference generation optimization model also includes:

[0176] Using the value function y 1j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b;

[0177] The value function y 1j The expression is:

[0178] y 1j =r j +γmin{m k QUR ω1 (s′ j ,k,μ θ (s j )),m k QUR ω2 (s′ j ,k,μ θ (s j ))}

[0179] In the formula, r j is the jth reward value, γ is the weight function, Q ω1 (s′ j ,k,μ θ (s j )) is the first Q value expression of the first Q function network, Q ω2 (s′ j ,k,μ θ (s j )) is the second Q value expression of the second Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μθ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters;

[0180] Using the parameter x′ j Updating network parameters of the first Q function network and the second Q function network to obtain an updated Q function network; the Q function network envelops the first Q function network and the second Q function network;

[0181] Using the value function y 1j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network;

[0182] The updated Q-function network and the updated interference pattern network are integrated to obtain a radar interference generation optimization model.

[0183] It can be seen that the advantage of the method of the present invention lies in the use of reinforcement learning and hybrid action space, that is, through a large number of real-time online radar confrontation training, the training algorithm is trained from the underlying logic of radar signal to output interference signal, so as to effectively deal with the problem of radar target confrontation in unknown and small sample environments. In addition, the present invention provides an intelligent algorithm and training method for signal-level radar confrontation. The algorithm input is a radar signal description word, and supports signal-level parameter output, which ensures that the algorithm improves its exploration efficiency under the condition of fully exploring the space until the optimal interference is generated.

[0184] Embodiment 2

[0185] The process design of the signal-level radar interference generation method in an open environment disclosed in this embodiment is as follows:

[0186] (1) Input information: The received radar radiation source description word, i.e., EDW information, mainly includes information such as carrier frequency, pulse interval, pulse width, and radar status judgment, as output status information S.

[0187] (2) Output information: the action space A of the interference pattern: such as noise, false targets, movement, etc., and the interference pattern and its corresponding parameter space B: such as starting distance, dwell time, suppression distance, etc.

[0188] (3) Interference effectiveness reward value: that is, the reward space R generated by the interference strategy, which is divided into online and offline parts. The online data includes the single interference benefit and resource cost, and the offline data includes the average interference onset delay time and the total effective time of the interference in one round. The specific example is shown in Table 1:

[0189] Table 1 Rewards

[0190] Field Name Field meaning jameff float, gr validity, [0-1] rc_ocpy float, gr resource cost, [-1-0] jameff_delay float, gr onset delay time, ms jameff_overall float, total effective time of gr, s

[0191] (4) Training terminal state: In this method, the terminal state is set to T: the last state of a round or the previous state when the interference target is switched is the terminal state (terminal).

[0192] 2. Radar jammer generation method based on PDQN-TD3

[0193] In this embodiment, the electromagnetic countermeasure generation problem is set as a reinforcement learning problem in a hybrid action space, and the output action is a discrete action and corresponding parameters, that is, a certain interference pattern and its corresponding interference pattern parameters, and each interference pattern has different corresponding parameters. For this reason, this embodiment uses a radar countermeasure interference generation algorithm based on P-DQN (Parameterized DQN). At the same time, the TD3 (Twin Delayed Deep Deterministic Policy Gradient algorithm) paradigm is used to train the deterministic gradient descent (DDPG) part involved in the algorithm training, as follows:

[0194] (1) The radar countermeasures jamming generation algorithm based on P-DQN (Parameterized DQN) generates jamming actions and jamming action parameters:

[0195] In the present invention, a radar interference generation model based on P-DQN is used to generate interference patterns and corresponding interference pattern parameters. The specific algorithm flow is designed as follows:

[0196] The specific radar interference generation model is as follows: Figure 2 As shown:

[0197] The radar countermeasure jamming style generation hybrid action space A (Actor) is formally defined as:

[0198] A=(k,x k )|x k ∈χ k ,k∈[K]

[0199] where χ k That is, the k-th interference style corresponding to the style parameter x k In reinforcement learning, there are K interference patterns, and the optimal Bellman equation is:

[0200]

[0201] In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern kk The action that constitutes t is the state at time t, Q(s t ,a t ) is the value function at time t, E represents the mathematical expectation, s t+1 is the state at time t+1, Q(s t+1 ,a t+1 ) is the value function at time t+1, r t is the reward value at time t, and γ is the weight function.

[0202] In the mixed action space of interference style + style discrete parameters, it can be rewritten as:

[0203]

[0204] In the formula, sup represents the upper limit.

[0205] When the Q function is determined, for any given state S composed of radar signal description words EDW and any given interference pattern (discrete action k), it can be As a function

[0206]

[0207] If a strategy network (i.e., the interference pattern network in Example 1, with parameters represented by θ) is used to fit x k Q (s) This function can be seen as maximizing the Q function for all k for this policy network. Simply let the sum of the Q functions of all k be the maximum as the goal, so we can get an objective function for learning this policy network:

[0208]

[0209] As you can see, this is very close to the objective function of Deterministic Policy Gradient (DDPG), both of which optimize the Q function of the policy network, except that there is an additional summation. The same method as DDPG can be used to optimize this objective function. According to the Deterministic Policy Gradient Theorem, the gradient of the above objective function to the policy network parameter θ is:

[0210]

[0211] In addition, according to x k Q The optimal Bellman equation can be rewritten as:

[0212]

[0213] At this time, we can imitate the general operation of DQN and use a Q function network (the parameter is represented by ω) to fit the above Q function Q(s,k,x k ), use the above optimal Bellman equation and TD loss to update the Q network, the objective function is:

[0214]

[0215] In the formula is the value function of the Q function network, k t is the interference pattern at time t, is the time t and the interference pattern k t The corresponding parameters.

[0216] Minimize this objective function (mean square error), and directly calculate the gradient of the parameter ω of the Q network to update ω using the gradient descent method. t is the target value of TD loss, that is:

[0217]

[0218] It can also be changed to the target of n-step reward:

[0219]

[0220] In the above two equations, when a certain step state is the terminal state T, the terms of the following steps are all 0. At the same time, the Q network here can be used to obtain the Q function of the DDPG part of the optimization strategy network, and the parameters of the Q network can be fixed using the DDPG method to update the parameters of the strategy network.

[0221] In summary, in this embodiment, when training the algorithm, the policy network is first fixed to update the parameters of the Q network, and then the Q network is fixed to update the parameters of the policy network, and so on. When using the model for inference, it is still necessary to find the action output (k, x k ), that is, find:

[0222]

[0223] In practice, the network architecture of P-DQN is: For the policy network x k (s), since the number of parameters corresponding to each discrete interference pattern k may be different (the output is not uniform), in order to unify, let x k The output dimension of (s) is the sum of all parameters corresponding to all discrete actions. For a certain k, we can just take out those parameters corresponding to this k action. k (s)), parameter x k(s) becomes the input of the network. Since the number of parameters corresponding to each discrete interference pattern k may be different (the input is not uniform), in order to unify, all the parameters corresponding to all discrete interference pattern actions output by the policy network are directly flattened and used as the input of the Q network. The output dimension is k. Therefore, during inference, it is actually:

[0224] x=x k (s)

[0225]

[0226] x k =parameters in x with respect to action k

[0227] So we get the action (k,x k ).

[0228] The pseudo code for executing this embodiment is as follows:

[0229] Initialize the parameters θ of the interference pattern network μ and the network parameters ω of the Q function network;

[0230] Initialize the interference pattern parameter θ′=θ of the target interference pattern network target_u, and the parameter ω′=ω of the combined action network target_Q of the interference pattern and pattern parameter;

[0231] Initialize the exploration hyperparameter ε and the moving average hyperparameter τ;

[0232] For t=1,2,…,T do

[0233] Taking the exploration hyperparameter ε as the probability, randomly select a interference style k, and randomly sample all parameters x corresponding to all interference styles, and take out the interference style parameters x corresponding to interference style k k ; With a probability of 1-ε, the interference pattern k and its corresponding interference pattern parameter x are obtained through the following steps: k ;

[0234] a t =(k,x k ), all t =(k,x);

[0235] In status t Execute a t Get status s t ′ and reward value r t ;

[0236] The four-tuple (s t ,all t ,s t ′,rt ) join the playback pool;

[0237] Randomly pick b quadruplets from the replay pool (s j ,all j ,s′ j ,r j );

[0238] For j=1,2,…,b,do

[0239] (k j ,x j )=all j ;

[0240] y j =r j +γm k QUR ω (s j ,k,x′ j ), y j When it is maximum, the Q function network fits best;

[0241] x′ j =μ θ (s j ), μ θ (s j ) is the target interference pattern network; γ∈(0,1] is the discount factor, Q ω (s j ,k,x′ j ) is the value function;

[0242] Update the Q function network;

[0243] end for

[0244] Use all y j and Q ω (s j ,k,x j ) updates ω by the sum of the root mean square errors;

[0245] Use all Q j ′ and update θ;

[0246] ω′=ωτ+(1-τ)ω′

[0247] θ′=θτ+(1-τ)θ′

[0248] s t =s′

[0249] end for

[0250] (2) Using the TD3 paradigm for algorithm training

[0251] In order to improve the training method of the DDPG-like part in P-DQN, this embodiment adopts the TD3 paradigm for algorithm training. The improvement of TD3 on DDPG can be divided into three main points.

[0252] First, DDPG involves minimizing the mean square error of the target and value function, similar to DQN. Here, the Q function is easily overestimated when finding the target. The solution to this problem in DQN is double DQN, while TD3 uses two sets of networks to estimate the Q value in the target, and uses the smaller one as the result of finding the Q value in the target to solve the overestimation problem, namely twin. In P-DQN, the algorithm step (10) can be changed to find the minimum value of the two targets.

[0253] y j =r j +γmin{m k QUR w1′ (s′ j ,k,μ θ (s′ j )),m k QUR w′2 (s′ j ,k,μ θ (s′ j ))}

[0254] When training the Q network, the mean square error of the target and the two Q values ​​is minimized at the same time, so the objective function is changed to minimizing the sum of the two mean square errors.

[0255]

[0256] When the target is not sought but the Q network is generally used to output the Q value, the first Q value of the two is used as the output.

[0257] Second, when using TD loss to find the target to update the Q network, DDPG needs a next action μ θ (s′ j ), so as to obtain the next step of Q value estimation Q w′ (s′ j ,μ θ (s′ j ) to calculate the target. TD3 adds a noise disturbance to the next action here, making the next Q value estimate, that is, the target estimate, more stable and robust. In P-DQN, the algorithm only needs to be modified as follows:

[0258]

[0259] Here, ε′ is the added noise, which is sampled from a Gaussian distribution with a mean of 0 and a variance of 0.2 in the experiment, and the noise is truncated to (-0.5, 0.5) to avoid a large difference from the original strategy.

[0260] Third, the policy network is no longer updated every time, but is updated once every multiple Q network updates, i.e. 'delayed'. This can also make the learning of the policy network more stable. In P-DQN, just change the algorithm to execute the steps of the algorithm every m times.

[0261] (3) Environment interaction and training details

[0262] 1) Interaction and exploration

[0263] ① In the present invention, the ε-greedy exploration method is used, and its ε value is set to exponential decay, that is, a random strategy is used at the beginning, a discrete action is randomly selected, and each continuous parameter is randomly and uniformly sampled. As the model learns and updates, it gradually tends to use the strategy output by the model. In the interaction process, in order to encourage exploration, noise is added to all actions of continuous parameters, and the noise is sampled from a Gaussian distribution with a mean of 0 and a variance of 0.1. More specifically, since each interference style selection in P-DQN involves all parameters corresponding to all discrete actions, the selected discrete actions and their corresponding parameters are used for execution in the environment, and all parameters are added to the buffer together with the discrete actions for training, it is set that when adding noise to the continuous parameters during the exploration process, it is only added to the parameters corresponding to the selected discrete actions, that is, the parameter action added to the buffer only has noise on the parameters corresponding to the discrete actions. In addition, it is set that random strategies are used in all the first 100 interactions to strengthen exploration and obtain more diverse data at the beginning of the interaction.

[0264] ② When the model output is not used in the interaction, the expert strategy is used as the random strategy output in the exploration strategy. The reward statistics of each round are counted from the beginning of each round, and the current random discrete action is determined based on the historical reward statistics and domain knowledge. The random continuous parameters are still obtained by uniform sampling. This is equivalent to "teaching" the expert strategy to the model.

[0265] 2) Network design

[0266] ①Network structure:

[0267] The reinforcement learning model includes two networks, the Q network and the policy network, and all network layers in the network are linear layers. In the present invention, the input of the Q network is the 74-dimensional EDW after flattening and all 8-dimensional action parameters, a total of 82-dimensional input, and the output is 4-dimensional Q values ​​of 4 discrete actions. The Q network includes three linear layers of 82*64, 64*32, and 32*3, and the activation function is ReLU. The input of the policy network is the 74-dimensional edw after flattening, and the output is all 8-dimensional action parameters. The policy network includes three linear layers of 74*64, 64*32, and 32*8, and the activation function is ReLU. In addition, similar to the ResNet approach, the input EDW is directly passed through a 74*8 linear layer, and its result is added to the output result of the three linear layers to obtain the output of the policy network, and finally the output value of each parameter is truncated within (-1,1) through the tanh layer. A test was conducted on whether to use the passthrough layer, and the experimental results are as follows: Figure 3 , which is the cumulative reward curve for every 20 steps.

[0268] ② Initialize the network parameters. According to the general practice of reinforcement learning, use orthogonal initialization of network weights and initialize all biases to 0.

[0269] ③ Use a dueling network in the output layer of the Q network. The dueling network comes from the improvement of DQN: dueling DQN. In the output layer of the Q network, the state-action value function Q value is decomposed into a combination of the value function V value and the advantage function A value. The former is independent of the action. This separates the good or bad state from the good or bad action under the state, which can make training faster. In the network design, just change the last output layer of the Q network into two linear layers (outputting a 1-dimensional V value and an A value in the dimension of the number of discrete actions respectively), and then combine them. The previous layers remain unchanged and are shared by the two linear layers. Suppose the parameter of the layer before the output layer is w, the parameter of the linear layer that outputs the V value is a, and the parameter of the linear layer that outputs the A value is β. The actual combination formula is as follows:

[0270]

[0271] It can be seen that when combining the two to get the final Q function, the V value and the A value are not directly added to get the Q value. Instead, in order to reflect "identifiability", the A value is forced to be centralized. In each state, the A value under each action must subtract the mean of the A values ​​under these different actions, so that the mean of the A values ​​of different actions in this state is 0, which makes the final Q value for different actions The mean is the V value. A large number of experiments in reinforcement learning environments have proved that this combination has a better learning effect. Verified in our environment, tested with and without the duel network, and obtained the cumulative reward curve for every 20 steps as shown below Figure 4 .

[0272] 3) Input and Output

[0273] ①The input state is EDW data, which includes several discrete attributes of radar pulse signals, especially multi-valued discrete attributes with no size relationship, such as rf_typ (carrier frequency pulse width information), pw_type (pulse repetition interval type), pri_type (pulse repetition interval type), etc. In theory, these attributes should be converted into one-hot representation. If this attribute is x, the xth dimension is 1, and the others are 0. However, in reality, although there are 9 possible values ​​for the above three attributes, only two of them appear in the actual experimental data, only one appears in pw_type, and only two appear in pri_type. In this way, rewriting them into one-hot form as the features of the input network may not be good, and the experimental results also verify this point. Figure 5 The cumulative reward curve for every 20 steps is whether to change the multi-valued discrete attributes to one-hot form: The above results show that not using one-hot significantly works better. Therefore, instead of changing the multi-valued discrete attributes to one-hot form, the input data is flattened and directly input into the network.

[0274] ② During training and inference, the input edw data is first standardized. The standardization method is:

[0275]

[0276] In the formula, x is the original input, x stand To standardize radar interference data information, μ is the mean of x, and σ is the standard deviation of x; the mean of the standardized data is 0, the variance is 1, and it approximately obeys the standard normal distribution. μ and σ can be incrementally updated during the interaction process. 2 , the update method is as follows:

[0277]

[0278]

[0279] Where x is the new input and n is the total number of previous inputs. If one-hot is used for multi-valued discrete attributes, these dimensions are not standardized, and only the continuous attribute column data is standardized. We verified it in our environment and tested it with and without input EDW data standardization. The cumulative reward curve for every 20 steps is as follows: Figure 6 . It can be seen that standardization should be used for the input edw data.

[0280] ③All action parameters here are continuous parameters except for a binary discrete parameter Doppler frequency modulation point. For continuous parameters, each continuous parameter has a different value range. Our processing method is that all continuous parameters output by the network are between -1 and 1, and then scaled to the corresponding value range. For the only discrete parameter, because it is binary, we still regard it as a continuous parameter, and finally output a value between -1 and 1, and then take the final parameter value as 1 or 0 according to whether it is greater than 0. As for how to control the network output between -1 and 1, we can use the tanh function, a commonly used activation function in deep learning, and its output is (-1,1). At the final output of the policy network, a layer of tanh activation function is passed. In addition, as mentioned above, the continuous parameters output during interaction will be added with noise. If the network output result obtained by tanh is not within (-1,1) after adding noise, we will truncate it and truncate it to -1 or 1 if it exceeds the range. This is used as the result of adding buffer, and finally it is scaled to the corresponding value range of the parameter and executed in the environment.

[0281] 4) Algorithm training

[0282] ①Terminal

[0283] In the previous content, we have emphasized the application environment of the present invention to determine the Terminal state. In fact, the Terminal is very important in reinforcement learning training. We tested whether to use the Terminal and obtained the cumulative reward curve for every 20 steps as follows Figure 7 The results of using terminal are significantly better than those of ignoring it.

[0284] ② Use gradient norm clipping to avoid excessive gradients

[0285] Enter a cutoff value, and find the second norm of the gradient on the parameters of a layer of network. If the second norm is greater than the cutoff value, multiply the gradient by a factor: the ratio of the cutoff value to the second norm. Therefore, if the second norm of the original gradient is greater than the cutoff value, the second norm of the new gradient is equal to the cutoff value and is truncated. Avoiding excessively large gradients makes training more stable.

[0286] ③Use invert gradient

[0287] For continuous parameter actions, DDPG and TD3 tend to output "boundary values" close to the upper and lower bounds of the parameter. We don't know if this is beneficial, but we still need to alleviate this problem. This leads to the gradient reversal method, whose intuitive understanding is: suppose that during a certain model training, a certain parameter to be output is very close to the upper bound of its range, and the current gradient is to make the update result of this parameter larger, that is, to make it closer to the upper bound or even exceed the upper bound, which is not good, then we multiply this gradient by a factor to make it smaller, that is, to prevent the parameter from continuing to increase. If the current parameter has exceeded its upper bound and the gradient is still positive, then multiply the gradient by a negative factor, that is, reverse the gradient and force the update result of the parameter back to within the range. When the parameter is very close to the lower bound of its range or is already outside the lower bound and the gradient is negative, the same is true. Multiply the gradient by a factor to prevent the parameter from decreasing or bring it back to within the range. To do this, we can write the formula for gradient reversal:

[0288]

[0289] Where p is a parameter, p is the gradient at the current p. As you can see, multiplying the gradient by such a factor can achieve the effects mentioned above, effectively forcing the parameters to stay within their range.

[0290] Embodiment 3

[0291] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of a signal-level radar interference generating device in an open environment disclosed in an embodiment of the present invention. Figure 8 The signal-level radar interference generation device in the open environment described is used to improve its exploration efficiency under the condition of fully exploring the space until the optimal interference signal is generated, and the embodiment of the present invention is not limited. Figure 8 As shown, the signal-level radar interference generating device may include the following operations:

[0292] S301, an information acquisition module, used to acquire radar interference data information;

[0293] The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space;

[0294] The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information;

[0295] S302, an information processing module, used to process the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B;

[0296] S303, a radar jamming signal generating module is used to set a jamming effectiveness reward value and a termination state, and to process the radar jamming data information to be processed using a preset radar jamming generating model to obtain radar jamming signal information.

[0297] Embodiment 4

[0298] See also Fig. 9 , Fig. 9 1 is a schematic diagram of the structure of another signal-level radar interference generating device in an open environment disclosed in an embodiment of the present invention. Fig. 9 The signal-level radar interference generation device in the open environment described is used to improve its exploration efficiency under the condition of fully exploring the space until the optimal interference signal is generated, and the embodiment of the present invention is not limited. Fig. 9 As shown, the signal-level radar interference generating device may include the following operations:

[0299] A memory 401 storing executable program codes;

[0300] a processor 402 coupled to the memory 401;

[0301] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the signal-level radar interference generation method in an open environment described in the first and second embodiments.

[0302] Embodiment 5

[0303] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the signal-level radar interference generation method in an open environment described in Embodiment 1 and Embodiment 2.

[0304] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0305] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0306] Finally, it should be noted that the signal-level radar interference generation method and device in an open environment disclosed in the embodiment of the present invention discloses only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating signal-level radar interference in an open environment, characterized in that: The method comprises: S1, obtain radar interference data information; The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space; The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information; S2, processing the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B; S3, setting the jamming effectiveness reward value and the termination state, and processing the radar jamming data information to be processed by using a preset radar jamming generation model according to the state information S, the jamming pattern action space A, and the jamming pattern parameter space B, to obtain radar jamming signal information; The radar interference generation model includes a Q function network and an interference pattern network; The Q function network includes a first linear layer with a size of 82×64, a second linear layer with a size of 64×32, and a third linear layer with a size of 32×3, and the activation function is ReLU; The first linear layer is used to receive the flattened 74-dimensional radar emitter description word information and 8-dimensional action parameter information; The output of the first linear layer is input into the second linear layer; The output of the second linear layer is input into a third linear layer; The output of the third linear layer is the Q value of 4 discrete actions; The interference pattern network includes a fourth linear layer of size 74×8, a fifth linear layer of size 74×64, a sixth linear layer of size 64×32, a seventh linear layer of size 32×8 and a tanh layer, and the activation function is ReLU; The fourth linear layer and the fifth linear layer are used to input the flattened 74-dimensional radar radiation source description word information; The output of the fifth linear layer is input into the sixth linear layer; The output of the sixth linear layer is input into the seventh linear layer; The output of the fourth linear layer and the output of the seventh linear layer are added to obtain 8-dimensional action parameter information; The tanh layer is used to truncate the 8-dimensional action parameter information to obtain action parameter information within (-1, 1); The method of using a preset radar interference generation model to process the radar interference data information to be processed to obtain radar interference signal information includes: S31, initialize the parameters θ of the interference style network μ, the network parameters ω of the Q function network, the exploration hyperparameter ε and the moving average hyperparameter τ; S32, taking the exploration hyperparameter ε as the probability, randomly selects a interference pattern k, a total of K interference patterns, and randomly samples all parameters x corresponding to all interference patterns, and takes out the parameters x corresponding to interference pattern k k , x k ∈X(k), X(k) is the interference pattern space; a t =(k,x k ),all t =(k,x) In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern k k The action that constitutes t The action composed of the interference pattern k and all parameters x corresponding to all interference patterns at time t, t = 1, 2, ..., T, T is the termination state; S33, in state s t Execute action a t , get the state s t ′ and reward value r t , the quaternion (s t ,all t ,s t ′,r t ) join the playback pool; S34, randomly selecting b quadruplets from the playback pool, where b is a positive integer; S35, using the b quadruplets to train the preset radar interference generation model to obtain a radar interference generation optimization model, including: S351, using the value function y j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b; The value function y j The expression is: In the formula, r j is the jth reward value, γ is the weight function, Q ω (s′ j ,k,μ θ (s j )) is the Q value expression of the Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters; S352, using the parameter x′ j Updating the network parameters of the Q function network to obtain an updated Q function network; S353, using the value function y j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network; S354, integrating the updated Q function network and the updated interference pattern network to obtain a radar interference generation optimization model; S36, using the radar interference generation optimization model to process the radar interference data information to be processed to obtain radar interference signal information.

2. The method for generating signal-level radar interference in an open environment according to claim 1, characterized in that: The state information S includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information. t ∈S,s t is the radar status at time t; The interference pattern action space A includes noise information, false target information and movement information; The interference pattern parameter space B includes starting distance information, stay time information and suppression distance information.

3. The method for generating signal-level radar interference in an open environment according to claim 1, characterized in that: The radar interference generation optimization model is used to process the radar interference data information to be processed to obtain radar interference signal information, including: S361, using a standardized processing model, performing standardized processing on the radar interference data information to be processed to obtain standardized radar interference data information; The standardized processing model expression is: In the formula, x is the original input, x stand is the standardized radar interference data information, μ is the mean of x, σ is the standard deviation of x; S362, using the radar interference generation optimization model, processing the standardized radar interference data information to obtain radar interference signal information; the radar interference signal information is the action that maximizes the Q value of the Q function network.

4. The method for generating signal-level radar interference in an open environment according to claim 1, characterized in that: The method of using the b quadruplets to train the preset radar interference generation model to obtain the radar interference generation optimization model also includes: Using the value function y 1j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b; The value function y 1j The expression is: In the formula, r j is the jth reward value, γ is the weight function, is the first Q value expression of the first Q function network, is the second Q value expression of the second Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters; Using the parameter x′ j Updating network parameters of the first Q function network and the second Q function network to obtain an updated Q function network; the Q function network envelops the first Q function network and the second Q function network; Using the value function y 1j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network; The updated Q-function network and the updated interference pattern network are integrated to obtain a radar interference generation optimization model.

5. A signal-level radar jammer generating device in an open environment, characterized in that: The device comprises: An information acquisition module, used to obtain radar interference data information; The radar interference data information includes radar radiation source description word information, interference pattern action space, interference pattern and interference pattern parameter space; The radar radiation source description word information includes carrier frequency information, pulse interval information, pulse width information and radar state judgment information; An information processing module, used to process the radar interference data information to obtain state information S, interference pattern action space A, and interference pattern parameter space B; A radar jamming signal generation module is used to set a jamming effectiveness reward value and a termination state, and to process the radar jamming data information to be processed using a preset radar jamming generation model according to the state information S, the jamming pattern action space A, and the jamming pattern parameter space B, to obtain radar jamming signal information; The radar interference generation model includes a Q function network and an interference pattern network; The Q function network includes a first linear layer with a size of 82×64, a second linear layer with a size of 64×32, and a third linear layer with a size of 32×3, and the activation function is ReLU; The first linear layer is used to receive the flattened 74-dimensional radar emitter description word information and 8-dimensional action parameter information; The output of the first linear layer is input into the second linear layer; The output of the second linear layer is input into a third linear layer; The output of the third linear layer is the Q value of 4 discrete actions; The interference pattern network includes a fourth linear layer of size 74×8, a fifth linear layer of size 74×64, a sixth linear layer of size 64×32, a seventh linear layer of size 32×8 and a tanh layer, and the activation function is ReLU; The fourth linear layer and the fifth linear layer are used to input the flattened 74-dimensional radar radiation source description word information; The output of the fifth linear layer is input into the sixth linear layer; The output of the sixth linear layer is input into the seventh linear layer; The output of the fourth linear layer and the output of the seventh linear layer are added to obtain 8-dimensional action parameter information; The tanh layer is used to truncate the 8-dimensional action parameter information to obtain action parameter information within (-1, 1); The method of using a preset radar interference generation model to process the radar interference data information to be processed to obtain radar interference signal information includes: S31, initialize the parameters θ of the interference style network μ, the network parameters ω of the Q function network, the exploration hyperparameter ε and the moving average hyperparameter τ; S32, taking the exploration hyperparameter ε as the probability, randomly selects a interference pattern k, a total of K interference patterns, and randomly samples all parameters x corresponding to all interference patterns, and takes out the parameters x corresponding to interference pattern k k , x k ∈X(k), X(k) is the interference pattern space; a t =(k,x k ),all t =(k,x) In the formula, a t is the interference pattern k at time t and the parameter x of the interference pattern k k The action that constitutes t The action composed of the interference pattern k and all parameters x corresponding to all interference patterns at time t, t = 1, 2, ..., T, T is the termination state; S33, in state s t Execute action a t , get the state s t ′ and reward value r t , the quaternion (s t ,all t ,s t ′,r t ) join the playback pool; S34, randomly selecting b quadruplets from the playback pool, where b is a positive integer; S35, using the b quadruplets to train the preset radar interference generation model to obtain a radar interference generation optimization model, including: S351, using the value function y j For the four-tuple (s j ,all j ,s′ j ,r j ) to process and obtain the state s j The interference pattern k in the interference pattern network j The corresponding parameter x′ j , j = 1, 2, ..., b; The value function y j The expression is: In the formula, r j is the jth reward value, γ is the weight function, Q ω (s′ j ,k,μ θ (s j )) is the Q value expression of the Q function network, s′ j is the jth state, k represents the interference pattern k, x′ j =μ θ (s j ) is state s j The interference pattern k in the interference pattern network j Corresponding parameters; S352, using the parameter x′ j Updating the network parameters of the Q function network to obtain an updated Q function network; S353, using the value function y j and the updated Q function network, updating the parameters of the interference pattern network to obtain an updated interference pattern network; S354, integrating the updated Q function network and the updated interference pattern network to obtain a radar interference generation optimization model; S36, using the radar interference generation optimization model to process the radar interference data information to be processed to obtain radar interference signal information.

6. A signal-level radar jammer generating device in an open environment, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the signal-level radar interference generation method in an open environment as described in any one of claims 1 to 4.

7. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the signal-level radar interference generation method in an open environment as described in any one of claims 1 to 4.

Images