HPPO-based anti-interference measure decision and parameter optimization method

Abstract

The invention provides an anti-interference measure decision and parameter optimization method based on HPPO, and belongs to the technical field of radar anti-interference. Comprising the following steps: setting a radar anti-interference process in a radar and jammer confrontation scene, and defining an interference environment state, an execution action and an instant reward of a radar system in the confrontation process; initializing actor-commentator network parameters, constructing an HPPO network based on the actor-commentator network, and performing hyper-parameter setting; for each time step of each training screen, based on an HPPO actor network decision, outputting an anti-interference measure and a measure parameter combination as an execution action of the current time step, a reviewer network of the HPPO being responsible for estimating a state value function, obtaining an instant reward after the radar executes the action, and transferring to a next state; in each update cycle, calculating loss functions of actor and reviewer networks according to the execution track, and updating actor and reviewer network parameters based on the loss functions; and after the preset training screen number is reached, finishing the training.

Description

Technical Field

[0001] The invention belongs to the technical field of radar anti-interference, and in particular relates to an anti-interference measure decision and parameter optimization method based on HPPO. Background Art

[0002] Radar anti-interference decision-making is an important part of the electronic attack and defense system. Its core task is to select the best anti-interference measures in a complex electromagnetic environment with interference to minimize the adverse effects of interference. The selection of anti-interference measures and corresponding parameters is directly related to the survivability and detection effect of the radar system.

[0003] In recent years, with the development of deep reinforcement learning (DRL) and radar anti-interference, the radar's perception and adaptive decision-making capabilities have been further optimized. At present, there are many reinforcement learning algorithms that actively optimize waveform parameters such as transmission power, frequency, pulse width, etc. to avoid interference, or passively adjust the interference suppression method after receiving interference to achieve anti-interference effects. The selection method of anti-interference measures in passive anti-interference decision-making has been widely discussed. However, the discussion on how to select the optimal parameter of the anti-interference measures is still insufficient. Summary of the invention

[0004] In view of this, the present invention provides an anti-interference measure decision and parameter optimization method based on hybrid proximal policy optimization (HPPO), which can decide the anti-interference measures and their parameters at the same time, thereby improving the anti-interference performance of the radar.

[0005] The technical solution for implementing the present invention is as follows:

[0006] A method for anti-interference measure decision and parameter optimization based on HPPO, comprising the following steps:

[0007] Step S1, setting the radar anti-interference process in the radar and jammer confrontation scenario, and defining the interference environment state, execution action, and instant reward of the radar system during the confrontation process;

[0008] Step S2, initialize the parameters of the actor-critic network, build the HPPO network based on the actor-critic network and set the hyperparameters, and further set the initial interference environment state s 0 ;

[0009] Step S3: For each time step of each training scene, the actor network decision outputs the anti-interference measure and the combination of measure parameters based on HPPO as the execution action a of the current time step.t , the critic network of HPPO is responsible for estimating the state value function V, which is used to evaluate the quality of action selection. The radar performs action a t After that, you will get an immediate reward. t , and transfer to the next state s t+1 ;

[0010] Step S4, in each update cycle, according to the execution trajectory (s t ,a t ,r t ), calculate the loss function of the actor and critic network, and update the parameters of the actor and critic network respectively based on the loss function;

[0011] Step S5, after reaching the preset number of training scenes, the training ends.

[0012] Optionally, the interference environment state of the present invention is: the interference state faced by the radar system, which is a 1×7-dimensional feature set consisting of the interference type emitted by the jammer and its corresponding interference parameters, expressed as:

[0013] S={C,θ′,A,P}

[0014] Wherein: C is the interference type, including narrowband aiming frequency, broadband blocking, repeated forwarding, and dense false target interference; θ′ is the incident angle; A is the interference amplitude; P = {B, T, N, I} is the interference parameter: for narrowband aiming frequency interference and broadband blocking interference, the interference parameters include the interference bandwidth B; for intermittent sampling forwarding interference, the interference parameters include the slice width T and the number of forwarding times N; for dense false target interference, the interference parameters include the forwarding interval I.

[0015] Optionally, the execution action of the present invention is: the anti-interference action taken by the radar system at each time step, including anti-interference measures and corresponding parameters:

[0016] A=(M,P r )

[0017] Where M = {a 1 ,a 2 ,a 3 ,a 4} indicates anti-interference measures, a 1 to a 4 They represent four anti-interference measures: sidelobe blanking, mainlobe cancellation, sidelobe cancellation and false target elimination. Indicates the parameters corresponding to each anti-interference measure; for sidelobe blanking, its anti-interference parameters include blanking threshold v 1,1 And the hidden widening v 1,2 ; For main lobe cancellation and side lobe cancellation, the anti-interference parameters include the sampling window length factor v 2,1 and v3,1 , sliding window step factor v 2,2 and v 3,2 ; For false target removal, the anti-interference parameters include interference selection width v 4,1 and interference removal broadening v 4,2 .

[0018] Optionally, the instant reward in the present invention refers to: when the radar system performs an action in each time step state, the instant reward value obtained is:

[0019] r=r 1 +r 2 +r 3 +r 4 +r 5 +r 6

[0020] Where: r 1 Indicates the target detection result feedback; r 2 Indicates false alarm feedback; r 3 Indicates the feedback of radar working mode change; r 4 Represents track output feedback; r 5 is the target distance feedback; r 6 Indicates peak-to-average power ratio feedback.

[0021] Optionally, if the present invention detects a target, 1 is +4, otherwise r 1 is -4; if there is a false alarm, r 2 is -1, otherwise r 2 +1; tracking becomes searching r 3 If -1, the search becomes tracking r 3 +1, keep searching 3 -1, keep tracking r 3 +1; target r is detected in tracking mode 4 +3, target r detected in search mode 4 is 0; in tracking mode, the closer you are to the target, the more reward r 5 The higher the value, the range is 0 to +2. If the current PRT is disturbed, r 6 is -1, otherwise r 6 is +1.

[0022] Optionally, the HPPO-based actor network decision-making outputs anti-interference measures and measure parameter combinations described in the present invention, and its specific implementation method is: the HPPO actor network adopts two parallel output layers to output anti-interference measures and measure parameter values ​​respectively; wherein, for discrete anti-interference measures, the output layer adopts a Softmax function to obtain the selection probability of different measure types, and selects measures based on random sampling; for continuous measure parameters, the output layer uniformly outputs two parameters α and β, constructs a Beta distribution (Beta Distribution) and randomly samples values.

[0023] Optionally, the present invention constructs a Beta distribution and randomly samples values ​​as follows: for different parameters, the sampling results are linearly mapped to their respective actual numerical ranges, thereby determining the specific value of each continuous parameter; wherein, the Beta distribution is used to characterize the probability distribution, with a value range of [0,1], and the parameters α and β control the shape of the probability density function of the Beta distribution.

[0024] Optionally, the process of updating the actor and critic network parameters includes:

[0025] Select the trajectory formed by U time steps (s t ,a t ,r t ); calculating the advantage function, and based on the advantage function, respectively calculating the loss functions of the discrete measure action and the continuous parameter action as the total loss of the actor network, and updating the actor and critic network parameters respectively according to the loss functions of the actor and critic networks.

[0026] Optionally, the loss function formulas for the critic and actor networks are:

[0027]

[0028] Among them, θ v and θ are the parameters of the critic and actor networks respectively, L c (θ v ) is the time difference error or action advantage function, r t represents the reward at time t, γ represents the discount factor, Indicates state s t The state value function under p θ (a t |s t ) means that in strategy π θ Next input state s t , output action a t The probability of In the old strategy The probability of input state st and output action at, clip(x,xb ,x e ) is the clipping function, ε c ∈(0,1) is the cropping factor.

[0029] Optionally, the process of updating the critic and actor network parameters based on the loss function is as follows:

[0030] Calculate the time difference error function L c (θ v ), back propagation minimizes the error function and updates the critic network parameters θ based on gradient descent v ;

[0031] calculate Where α is the actor network learning rate, and the actor network parameters θ are updated based on gradient ascent.

[0032] Beneficial effects:

[0033] In the radar and jammer confrontation scenario, the present invention needs to simultaneously decide on a limited number of anti-interference measures and their corresponding continuous parameters. By modifying the single output layer of the PPO actor network into two independent parallel output layers to form HPPO, and drawing on the idea of ​​policy gradient optimization, the importance sampling values ​​of the measure type and measure parameters are respectively calculated and the advantage function value is shared to update the strategy, so that the algorithm can be applicable to the anti-interference decision-making of the radar in the multi-dimensional mixed action space, thereby improving the anti-interference performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 Schematic diagram of the confrontation scenario;

[0036] Figure 2 It is a schematic diagram of radar signal and data processing flow and working mode conversion;

[0037] Figure 3 It is a schematic diagram of the HPPO network structure;

[0038] Figure 4 Comparison chart of scores, tracking rate and root mean square error of distance among HPPO, PPO and expert experience decision method. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0041] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0042] First make the following assumptions:

[0043] First, there is a radar system and a jammer in the confrontation scenario, such as Figure 1 As shown. The jammer can launch typical suppression and deception jamming, including narrowband aiming frequency jamming (NBAFJ), wideband blocking jamming (WBBJ), interrupted sampling repeater jamming (ISRJ) and dense false target jamming (DFTJ). Radar anti-jamming measures include sidelobe blanking, main lobe cancellation, sidelobe cancellation and false target elimination.

[0044] Second, the radar has two working modes: search and tracking. The radar initially works in the search mode. In each pulse repetition period, the received echo signal is interfered with and the target signal is detected and the parameters are estimated to form a point track. If the radar detects the target for three consecutive PRTs and is within the relevant wave gate, an initial track is established, the radar switches to the tracking mode, and the subsequent point tracks are tracked and filtered. Otherwise, the search mode is maintained. In the tracking mode, if the radar fails to detect the target for three consecutive PRTs, the search mode is entered. Otherwise, the tracking mode is maintained. The interference randomly transmits narrowband aiming frequency interference and broadband blocking interference in the search mode, and randomly transmits intermittent sampling forwarding interference and dense false target interference in the tracking mode. The schematic diagram of the radar signal and data processing flow and the working mode conversion is shown in the figure. Figure 2shown.

[0045] Combine the following Figure 1 and Figure 2 The specific implementation scheme of the present invention is described as follows: This embodiment is an anti-interference measure decision and parameter optimization method based on HPPO, and the detailed process is as follows:

[0046] In step 1 (S1), the radar anti-jamming decision problem in the confrontation scenario between the radar system and the jammer is modeled as a Markov decision process problem. The confrontation scenario parameter settings are shown in Table 1.

[0047] Table 1

[0048]

[0049] The environmental state refers to the interference state faced by the radar system, which is a 1×7-dimensional feature set consisting of the interference type emitted by the jammer and its corresponding interference parameters. It is obtained through interference perception and other means, and is expressed as:

[0050] S={C,θ′,A,P}

[0051] Wherein: C is the interference type, including narrowband aiming frequency, broadband blocking, repeated forwarding, and dense false target interference; θ′ is the incident angle; A is the interference amplitude; P = {B, T, N, I} is the interference parameter: for narrowband aiming frequency interference and broadband blocking interference, the interference parameters include the interference bandwidth B; for intermittent sampling forwarding interference, the interference parameters include the slice width T and the number of forwarding times N; for dense false target interference, the interference parameters include the forwarding interval I.

[0052] The execution action refers to the anti-interference action taken by the radar system at each time step, including anti-interference measures and corresponding parameters:

[0053] A=(M,P r ) (2)

[0054] Where M = {a 1 ,a 2 ,a 3 ,a 4} indicates anti-interference measures, a 1 to a 4 They represent four anti-interference measures: sidelobe blanking, mainlobe cancellation, sidelobe cancellation and false target rejection. Indicates the parameters corresponding to each anti-interference measure. For sidelobe blanking, its anti-interference parameters include the blanking threshold v 1,1 And the hidden widening v 1,2 ; For main lobe cancellation and side lobe cancellation, the anti-interference parameters include the sampling window length factor v 2,1 and v 3,1 , sliding window step factor v2,2 and v 3,2 ; For false target removal, the anti-interference parameters include interference selection width v 4,1 and interference removal broadening v 4,2 .

[0055] The instant reward refers to the reward obtained when the radar system performs an action at each time step, which is defined as follows:

[0056] r=r 1 +r 2 +r 3 +r 4 +r 5 +r 6 (3)

[0057] : The reward value is associated with radar combat technical indicators, including target detection, false alarm, tracking mode maintenance time, etc., where: 1 Indicates the target detection result feedback, if the target is detected, it is +4, otherwise it is -4; r 2 Indicates false alarm feedback, if there is a false alarm, it is -1, otherwise it is +1; r 3 Indicates the feedback of radar working mode change. Tracking changes to searching, which is -1; searching changes to tracking, which is +1; keeping searching, which is -1; keeping tracking, which is +1. 4 Indicates track output feedback. In tracking mode, it is +3 when a target is detected, and in search mode, it is 0; r 5 It is the target distance feedback. In the tracking mode, the closer you are to the target, the higher the reward. The range is 0 to +2. 6 Indicates the peak-to-average power ratio feedback, which indicates the interference situation of the current PRT. If there is interference, it is -1, otherwise it is +1. 1 ~r 5 Calculated for each PRT, r 6 Calculated for each CPI.

[0058] Step 2 (S2) adjusts the actor network and critic network parameters θ and θ v Initialize and set the initial interference state s 0 , the network hyperparameters are set as shown in Table 2. Among them, the time step unit is PRT, and CPI is defined as 32 PRTs.

[0059] Table 2

[0060]

[0061] Step 3 (S3), at each time step of each training scene, output the anti-interference measures and measure parameter combination a based on the HPPO algorithm decision t =(m,p r), m is an element of M, p r P r elements.

[0062] HPPO network structure Figure 3 As shown in the figure. For discrete anti-interference measures, the output layer uses the Softmax function to obtain the selection probability of different types of measures, and selects measures based on random sampling. For continuous measure parameters, considering that their parameters may have different value ranges, the output layer uniformly outputs two parameters α and β, thereby constructing a Beta distribution (BetaDistribution) and randomly sampling values.

[0063] For different parameters, the sampling results are linearly mapped to their respective actual numerical ranges to determine the specific values ​​of each continuous parameter. Among them, the Beta distribution is used to characterize the probability distribution, and the value range is [0,1]. This distribution is regulated by two parameters α and β, whose sizes control the shape of the probability density function of the Beta distribution.

[0064] After that, the radar performs action a t , after the signal and data processing process, the reward r is obtained t , and by the current state s t Transition to next state s t+1 The signal and data processing flow has been described in the assumptions.

[0065] Step 4 (S4), after U time steps, the update period is reached and all trajectories in this period are selected (s t ,a t ,r t ), calculate the advantage function based on the temporal difference error, calculate the loss functions of discrete measure actions and continuous parameter actions respectively and add them up as the total loss of the actor network, and update the parameters of the actor and critic networks respectively according to their loss functions.

[0066] The loss function formulas for the critic and actor networks are:

[0067]

[0068] Among them, θ v and θ are the parameters of the critic and actor networks respectively, L c (θ v ) is also called the temporal difference error or action advantage function, r t represents the reward at time t, γ represents the discount factor, Indicates state s t The state value function under p θ (a t |s t ) means that in strategy πθ Next input state s t , output action a t The probability of In the old strategy The probability of input state st and output action at. clip(x,x b ,x e ) is the clipping function, which controls the output range of x to [x b ,x e ], ε c ∈(0,1) is the clipping coefficient. The purpose of the clipping function is to Constraints are imposed to avoid large differences between new and old strategies and improve stability.

[0069] Calculate the time difference error function L c (θ v ), back propagation minimizes the error function and updates the critic network parameters θ based on gradient descent v ;

[0070] calculate Where α is the actor network learning rate, and the actor network parameters θ are updated based on gradient ascent.

[0071] As the training process progresses, based on the idea of ​​policy gradient optimization, the actor and critic network parameters, including the parameters α and β that control the beta distribution, will be gradually optimized to form the optimal strategy.

[0072] Step 5 (S5), after reaching the designated number of training episodes, the training ends.

[0073] The beneficial effects of the embodiments of the present invention are further described below through simulation results:

[0074] In addition to the reward score of reinforcement learning training, the tracking rate P is introduced T And the root mean square error (RMSE) is used to evaluate the anti-interference decision performance:

[0075]

[0076] Among them, N T is the total number of tracking times for each training episode, n T is the number of tracking failures. Yi, f(x i ) are the target position information obtained by each CPI tracking filter and the real position information of the target, N S The number of CPIs in each training session.

[0077] Figure 4(a), (b), and (c) show the convergence process of the reward score, tracking rate, and root mean square error of the distance in the training process using the proposed algorithm, the PPO algorithm, and the expert experience decision method in the embodiment of the present invention. Among them, after PPO decides the type of anti-interference measure, it decides the anti-interference measure parameters from the discrete value set of each parameter value. The expert experience decision method makes a logical judgment on the anti-interference measure according to the interference incident angle and the interference type, and the parameters are the values ​​preset according to experience, as shown in Table 3. The number of training scenes is 100. The comparison of various indicators after training convergence is shown in Table 4.

[0078] Table 3

[0079]

[0080]

[0081] Table 4

[0082]

[0083] It can be seen that compared with the expert experience decision-making method, the proposed method and PPO can both optimize the anti-interference performance through strategy learning, and show higher scores, tracking rates and lower distance errors. Furthermore, compared with PPO, the proposed method has stronger anti-interference ability after convergence, with the average reward score increased by 38.4 to 117.4, the average tracking rate increased by 8.1% to 81.8%, and the average distance root mean square error reduced by 0.9m to 7.0m.

[0084] The improvement in anti-interference performance is mainly attributed to the HPPO framework's ability to handle multi-dimensional mixed action spaces, which enables the agent to explore more potential optimal strategies and thus explore better actions in continuous space relative to discrete space. Similarly, with the substantial expansion of the action space, the convergence speed and stability have been reduced.

[0085] In summary, the anti-interference measure decision-making and parameter optimization method based on HPPO proposed in the present invention can explore actions that are better than those in discrete space in a multi-dimensional mixed action space where discrete anti-interference measures and continuous measures parameters need to be decided, thereby improving the anti-interference performance of the radar system.

[0086] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for anti-interference measure decision and parameter optimization based on HPPO, characterized in that: The following steps are involved: Step S1, setting the radar anti-interference process in the radar and jammer confrontation scenario, and defining the interference environment state, execution action, and instant reward of the radar system during the confrontation process; Step S2, initializing the actor-critic network parameters, constructing the HPPO network based on the actor-critic network and setting the hyperparameters, and further setting the initial interference environment state s0; Step S3: For each time step of each training scene, the actor network decision outputs the anti-interference measure and the combination of measure parameters based on HPPO as the execution action a of the current time step. t , the critic network of HPPO is responsible for estimating the state value function V, and the radar performs action a t After that, you will get an immediate reward. t , and transfer to the next state s t+1 ; Step S4, in each update cycle, according to the execution trajectory (s t ,a t ,r t ), calculate the loss function of the actor and critic network, and update the parameters of the actor and critic network respectively based on the loss function; Step S5, after reaching the preset number of training scenes, the training ends.

2. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 1, characterized in that: The interference environment state is: the interference state faced by the radar system, which is a 1×7-dimensional feature set consisting of the interference type emitted by the jammer and its corresponding interference parameters, expressed as: S={C,θ′,A,P} Wherein: C is the interference type, including narrowband aiming frequency, broadband blocking, repeated forwarding, and dense false target interference; θ′ is the incident angle; A is the interference amplitude; P = {B, T, N, I} is the interference parameter: for narrowband aiming frequency interference and broadband blocking interference, the interference parameters include the interference bandwidth B; for intermittent sampling forwarding interference, the interference parameters include the slice width T and the number of forwarding times N; for dense false target interference, the interference parameters include the forwarding interval I.

3. The anti-interference measure decision and parameter optimization method based on HPPO according to claim 1, characterized in that: The execution action is: the anti-interference action taken by the radar system at each time step, including anti-interference measures and corresponding parameters: A=(M,P r ) Wherein, M = {a1, a2, a3, a4} represents anti-interference measures, a1 to a4 represent four anti-interference measures, namely, sidelobe blanking, mainlobe cancellation, sidelobe cancellation and false target rejection. Indicates the parameters corresponding to each anti-interference measure; for sidelobe blanking, its anti-interference parameters include blanking threshold v 1,1 And the hidden widening v 1,2 ; For main lobe cancellation and side lobe cancellation, the anti-interference parameters include the sampling window length factor v 2,1 and v 3,1 , sliding window step factor v 2,2 and v 3,2 ; For false target removal, the anti-interference parameters include interference selection width v 4,1 and interference removal broadening v 4,2 .

4. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 1, characterized in that: The instant reward refers to the instant reward value obtained when the radar system performs an action at each time step state: r＝r1+r2+r3+r4+r5+r6 Among them: r1 represents the target detection result feedback; r2 represents the false alarm feedback; r3 represents the radar working mode change feedback; r4 represents the track output feedback; r5 is the target distance feedback; r6 represents the peak-to-average power ratio feedback.

5. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 4, characterized in that: If the target is detected, r1 is +4, otherwise r1 is -4; if there is a false alarm, r2 is -1, otherwise r2 is +1; tracking changes to search r3 is -1, search changes to tracking r3 is +1, keep searching r3 is -1, keep tracking r3 is +1; r4 is +3 when the target is detected in tracking mode, and r4 is 0 when the target is detected in search mode; in tracking mode, the closer to the target, the higher the reward r5, and the range of variation is 0 to +2; if the current PRT is interfered with, r6 is -1, otherwise r6 is +1.

6. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 1, characterized in that: The HPPO-based actor network decision-making outputs anti-interference measures and measure parameter combinations, and its specific implementation method is: the HPPO actor network adopts two parallel output layers to output anti-interference measures and measure parameter values ​​respectively; wherein, for discrete anti-interference measures, the output layer adopts a Softmax function to obtain the selection probability of different measure types, and selects measures based on random sampling; for continuous measure parameters, the output layer uniformly outputs two parameters α and β, constructs a Beta distribution and randomly samples values.

7. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 1, characterized in that: The construction of the Beta distribution and random sampling are as follows: for different parameters, the sampling results are linearly mapped to their respective actual numerical ranges, so as to determine the specific value of each continuous parameter; wherein the Beta distribution is used to characterize the probability distribution, and the value range is [0,1], and the parameters α and β control the shape of the probability density function of the Beta distribution.

8. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 1, characterized in that: The process of updating the actor and critic network parameters includes: Select the trajectory formed by U time steps (s t ,a t ,r t ); calculating the advantage function, and based on the advantage function, respectively calculating the loss functions of the discrete measure action and the continuous parameter action as the total loss of the actor network, and updating the actor and critic network parameters respectively according to the loss functions of the actor and critic networks.

9. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 8, characterized in that: The loss function formulas for the critic and actor networks are: Among them, θ v and θ are the parameters of the critic and actor networks respectively, L c (θ v ) is the action advantage function, r t represents the reward at time t, γ represents the discount factor, Indicates state s t The state value function under p θ (a t |s t ) means that in strategy π θ Next input state s t , output action a t The probability of In the old strategy The probability of input state st and output action at, clip(x,x b ,x e ) is the clipping function, ε c ∈(0,1) is the cropping factor.

10. The HPPO-based anti-interference measure decision and parameter optimization method according to claim 9, characterized in that: The process of updating the critic and actor network parameters based on the loss function is: Calculate the time difference error function L c (θ v ), back propagation minimizes the error function and updates the critic network parameters θ based on gradient descent v ; calculate Where α is the actor network learning rate, and the actor network parameters θ are updated based on gradient ascent.

Images