A low-speed target adaptive MTI method in a reinforced learning ground clutter environment

Abstract

The application discloses a low-speed target adaptive MTI method in a ground object clutter environment based on reinforcement learning, and belongs to the field of radar signal processing. The application uses reinforcement learning to screen low-speed target MTI filter groups, and adaptively selects a filter with the optimal improvement factor for low-speed targets. The algorithm maintains left and right boundaries, corresponding to the filters with the lowest and highest stopband edge frequencies. In each cycle, the left and right split indexes are calculated according to the two boundaries, and the improvement factors of the filters corresponding to the two indexes are used as the current state input DQN for action prediction. The action space includes two actions: action one updates the left boundary to the left split index; and action two updates the right boundary to the right split index. The left and right boundaries are repeatedly contracted until the left and right boundaries coincide, and the algorithm returns the filter corresponding to the current index. The application uses reinforcement learning, reduces the filtering frequency, and has the ability to accurately screen the optimal filter.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of radar signal processing. BACKGROUND

[0002] In modern radar systems, moving target indication (MTI) is a key technology for detecting and tracking moving targets. Traditional MTI systems rely on fixed parameter filter designs, which perform well under specific environmental and target characteristics. However, as the complexity of the environment and the diversity of targets increase, the limitations of fixed parameter filters gradually emerge.

[0003] In low-speed target detection, the statistical characteristics of ground clutter vary over time in different spatial distributions, and traditional fixed parameter MTI filters are difficult to effectively suppress them throughout the entire process and time. A feasible technical approach is to pre-store a set of MTI filters with different parameters, and according to the changes in the statistical characteristics of clutter, to autonomously select the optimal MTI filter for clutter suppression. Usually, such methods rely on expert experience, and the filter generalization ability is insufficient. Moreover, when all filters need to participate in the operation for evaluation and optimization, the algorithm complexity will rise sharply, seriously affecting the real-time performance of the operation.

[0004] In recent years, machine learning techniques have gradually increased in radar signal processing. In particular, reinforcement learning techniques, which can optimize strategies through continuous experimentation and feedback in dynamic environments, have shown great potential in adaptive signal processing. Reinforcement learning-based adaptive filter optimization methods can automatically learn and adjust filter parameters to adapt to different environments and target characteristics, significantly improving the detection performance of MTI systems. SUMMARY

[0005] To improve the detection capability of radar systems in complex environments for low-speed targets, the present application proposes a reinforcement learning-based low-speed target adaptive MTI method in ground clutter environments. By introducing a reinforcement learning algorithm based on filter decision interval segmentation, the method can automatically select the filter with the optimal improvement factor for low-speed targets from the filter set, to achieve adaptive optimization of the filter, effectively suppress ground clutter, and improve the detection performance of low-speed targets, especially in the case of Doppler domain overlap between clutter and targets.

[0006] This invention utilizes a reinforcement learning algorithm based on filter decision interval segmentation. It maintains two boundaries: a left boundary and a right boundary, corresponding to the filters with the lowest and highest stopband edge frequencies. Based on these boundaries, filter decision interval segmentation is performed. The left and right segmentation indices, along with the improvement factors of the corresponding filters, are calculated as the current state and used as input to a DQN for action prediction. The action space contains two actions: action one is to shrink the left boundary, updating it with the left segmentation index; action two is to shrink the right boundary, updating it with the right segmentation index. The left and right boundaries are repeatedly shrunk until they coincide, at which point the algorithm returns the filter corresponding to the current index. During training, the actual action corresponding to the label is executed, and the DQN gradient is updated; during testing, the DQN-predicted action is executed. The technical solution for implementing this invention includes:

[0007] Step 1: Based on the clutter sequence set {C i |i=0,...,N-1}、FIR clutter suppression filter bank {w j |j=0,...,M-1} and the system's detection range for low-speed targets in the velocity dimension [-v0,v0], calculate the improvement factor sequence set {IF ij |i=0,...,N-1,j=0,...,M-1}, where N represents the number of ground clutter sequences, M represents the number of filters in the clutter suppression filter bank, and IF ij This indicates the use of the j-th clutter suppression filter w j For the i-th clutter sequence C i Improvement factors;

[0008] Step 2: For the i-th clutter sequence C i For a specific i, the factor sequence IF will be improved. ij The optimal improvement factor IF ij * index As clutter sequence C i Filtering decision label L i Construct a ground clutter suppression dataset {D} i}={(C i ,L i )|i=0,...,N-1}, and set the dataset {D i Divide into training sets according to a certain ratio. and test set Where P is the number of samples in the training set and Q is the number of samples in the test set;

[0009] Step 3: Reinforcement learning model training phase, based on the i-th sample in the training set from Step 2 Obtain the state space segmented by the filtering decision interval and filter decision label Li , utilizing L i training the reinforcement learning model;

[0010] Step 4: reinforcement learning model test phase, according to the i-th sample in step 2 test set get the filter decision interval segmentation state space and the filter decision label utilizing test the reinforcement learning model;

[0011] Step 5: repeat steps 3, 4 until the reinforcement learning model converges and has the required generalization ability.

[0012] Compared with the prior art, the present application has the following advantages:

[0013] (1) The traditional MTI filter depends on expert experience to optimize filter parameters according to clutter prior knowledge. The present application can realize adaptive optimization of the MTI filter based on the reinforcement learning algorithm of filter decision interval segmentation.

[0014] (2) The traditional fixed parameter MTI filter bank is difficult to effectively suppress ground clutter in complex environments, resulting in limited low-speed target detection performance. The present application trains in multiple environments and conditions, so that the reinforcement learning model can learn more robust and generalizable filter parameters, ensuring excellent detection performance in various complex environments.

[0015] (3) The present application uses a lightweight reinforcement learning network to optimize low-speed targets, which has the best improvement factor for low-speed targets. Especially in the case of target and clutter aliasing in the Doppler domain, without traversing the MTI filter bank, the low-speed target detection performance in clutter environment can be improved with low computational complexity.

[0016] The present application will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the preferred flowchart of the present application;

[0018] Figure 2 is the frequency response diagram of the equal-ripple filter bank;

[0019] Figure 3 is the DQN network structure diagram;

[0020] Figure 4 is the boundary and segmentation index relationship diagram;

[0021] Figure 5 is the filter stopband edge frequency and improvement factor relationship diagram;

[0022] Figure 6 is the decision interval equalization score K and model accuracy relationship diagram;

[0023] Figure 7 is the decision interval equalization score and the corresponding improvement factor of the model false prediction result and the average difference / standard deviation of the corresponding improvement factor of the true label relationship diagram;

[0024] Figure 8 is the decision interval equalization score K and the relationship diagram of the average number of steps and the filtering times of the model. DETAILED DESCRIPTION

[0025] The present application proposes a low-speed target adaptive MTI method in a reinforcement learning ground clutter environment. For low-speed targets, a reinforcement learning algorithm based on filtering decision interval segmentation is introduced to select the filter with the optimal improvement factor from the high-pass FIR filter set, realize adaptive optimization of the filter set, overcome many shortcomings in traditional methods, not only can effectively suppress ground clutter and other interference signals in complex environments, but also can improve the detection accuracy and reliability of low-speed targets, thereby improving the performance of the entire MTI system. At the same time, the present application uses a lightweight network to realize its function, which is suitable for real-time signal processing occasions.

[0026] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0027] As shown in the figure, the preferred process of the present application includes the following steps: Figure 1

[0028] Step 1: According to the clutter sequence set {C i |i=0,...,N-1}, the FIR clutter suppression filter set {w j |j=0,...,M-1} and the detection range of the system on the low-speed target in the speed dimension [-v0,v0], the improvement factor sequence set {IF ij |i=0,...,N-1,j=0,...,M-1} is calculated, wherein N represents the number of ground clutter sequences, M represents the number of filters in the clutter suppression filter set, IF ij represents the improvement factor of the ith clutter sequence C j using the jth clutter suppression filter w i .

[0029] Preferably, the FIR clutter suppression filter set in step 1 includes:

[0030] {w j |j=0,...,M-1} (1)

[0031] ​The filter bank is designed using the equiripple method, with the stopband edge frequencies uniformly distributed within [v]. start ,v end Within the range, where v start v is the minimum clutter half-width supported by the system. end The transition band width is fixed at Δv1, which is the maximum clutter half-width supported by the system. Δv1 is specifically described as follows:

[0032] Δv1=α1Δv D (2)

[0033] Where, Δv D For Doppler velocity resolution, α1 is an adjustable parameter. A smaller α1 results in a narrower transition band and stronger resolution against clutter and targets, but this increases the filter order. The filter order should be such that (order + N) c -1) Powers close to but not exceeding 2, where N c The length of the clutter sequence;

[0034] The interval between filters is fixed at Δv2:

[0035] Δv2=α2Δv D (3)

[0036] α2 is an adjustable parameter, and M represents the number of filters in the clutter suppression filter bank, including:

[0037]

[0038] In the formula, This is for rounding up.

[0039] Preferably, the improvement factor sequence set {IF} in step 1 ij The calculation methods for} include:

[0040] To find the autocorrelation matrix of a low-speed target, given the target's velocity range [-v0, v0], the autocorrelation function of the target signal is:

[0041]

[0042] In the formula, τ mn =t m -t n = (mn) / fr is the correlation time, λ is the wavelength, and fr is the pulse repetition frequency, thus obtaining the autocorrelation matrix R of the N-pulse signal. s for:

[0043] Based on the autocorrelation matrix R of the low-speed target sFind the FIR clutter suppression filter w j For the i-th clutter sequence C i Improvement factor IF ij :

[0044]

[0045] Step 2: For the i-th clutter sequence C i For a specific i, the factor sequence IF will be improved. ij The optimal improvement factor IF ij * index As clutter sequence C i Filtering decision label L i Construct a ground clutter suppression dataset {D} i}={(C i ,L i )|i=0,...,N-1}, and set the dataset {D i Divide into training sets according to a certain ratio. and test set Where P is the number of samples in the training set and Q is the number of samples in the test set;

[0046] Step 3: Reinforcement learning model training phase, based on the i-th sample in the training set from Step 2 Obtain the state space segmented by the filtering decision interval and filter decision label L i ,use L i Train the reinforcement learning model;

[0047] Step 4: Reinforcement learning model testing phase, based on the i-th sample in the test set from Step 2. Obtain the state space segmented by the filtering decision interval and filter decision labels use Test reinforcement learning models;

[0048] Step 5: Repeat steps 3 and 4 until a convergent reinforcement learning model with the required generalization ability is obtained.

[0049] Preferably, step 2 includes the following steps:

[0050] Step 2-1: For each clutter sequence C i Select the filter index with the optimal improvement factor.

[0051]

[0052] Step 2-2: Index the filters with the optimal improvement factor As clutter sequence C i Filtering decision label L i :

[0053]

[0054] Steps 2-3: Record each clutter sequence C i and its filtering decision label L i Construct dataset D = {D} i}:

[0055] {D i}={(C i ,L i ) |i=0,...,N-1} (10) Step 2-4: Divide the dataset D into training sets according to the proportion. and test set Where P is the number of samples in the training set and Q is the number of samples in the test set.

[0056] Preferably, step 3 includes the following steps:

[0057] Step 3-1: For the training set training samples Initialize left and right boundaries Corresponding to the filter banks {w j The first and last filters in};

[0058] Step 3-2: Based on the left and right boundaries Calculate the left partition index and right-split index

[0059]

[0060] Where K is the interval fraction for each decision.

[0061] Step 3-3: Calculate the filters corresponding to these two segmentation indices. and clutter sequences Improvement factor IF i L and IF i R The calculation method is specifically described as follows:

[0062]

[0063] Obtain training samples state space

[0064]

[0065] Steps 3-4: Use DQN (Deep Q-Network) to divide the state space Mapping to action space Represents a 1×2 vector.

[0066] Steps 3-5: Based on the training samples The label L i Calculate the current state space Corresponding real actions

[0067]

[0068] Steps 3-6: Execution At that time, the left boundary is updated to the left partition index:

[0069]

[0070] At that time, the right boundary is updated to the right partition index:

[0071]

[0072] Steps 3-7: Calculate the cross-entropy loss i :

[0073]

[0074] In the formula, N a Indicates the number of actions in the action space; y i One-hot encoding representing the actual action, i.e., if the a-th action is the correct action. So y a =1, otherwise equal to 0; Representing the action space The probability of predicting action a.

[0075] Steps 3-8: Calculate the loss i Perform backpropagation, update the gradients, and train DQN.

[0076] Steps 3-9: The current training sample is terminated when one of the following two conditions is met. Loop ends:

[0077] 1) That is, the left and right boundaries coincide, return.

[0078] 2) If the difference between the left and right boundaries is less than K, calculate the left and right boundaries of the improved factor Select the index corresponding to the optimal improved factor Return Ind * .

[0079] If the above two conditions are not met, go to step 3-2.

[0080] Step 3-10: Let i <- i+1, re-execute step 3-1~step 3-9 for the next training set sample training until the training set is traversed i.e. i = P.

[0081] Preferably, step 4 includes the following steps:

[0082] Step 4-1: For each training sample in the training set Initialize the left and right boundaries corresponding to the first and last filters in the filter set {w j} respectively;

[0083] Step 4-2: Calculate the left split index and the right split index according to the left and right boundaries

[0084]

[0085] Where K is the interval equalization of each decision.

[0086] Step 4-3: Calculate the filters and corresponding to the two split indexes Calculate the improved factor IF i L and IF i R The calculation method is described as follows:

[0087]

[0088] Get the state space of the test sample

[0089]

[0090] Step 4-4: Map the state space to the action space represents a 1x2 vector. action is obtained according to the action space i :

[0091]

[0092] action i ∈{0,1}. Perform action action i , action i = 0, the left boundary is updated as the left split index:

[0093]

[0094] action i = 1, the right boundary is updated as the right split index:

[0095]

[0096] Step 4-5: When one of the following two conditions occurs, the current test sample Loop ends:

[0097] 1) i.e. the left and right boundaries coincide, return

[0098] 2) i.e. the difference between the left and right boundaries is less than K, calculate the improvement factor of to select the index corresponding to the optimal improvement factor return Ind * .

[0099] If the above two conditions are not met, go to step 4-2.

[0100] Step 4-6: Let i <- i + 1, re-execute steps 4-1 to 4-5 for the next test set sample test until the test set is traversed i.e. i = Q.

[0101] Preferably, step 5 includes the following steps:

[0102] Step 5-1: Perform step 3 to obtain the trained model DQN train and the accuracy δ train of the training set D εtrain ;

[0103] Step 5-2: Perform step 4 using the model DQN train of step 5-1 to obtain the accuracy of the test set Dtest accuracy δ εtest ;

[0104] Step 5-3: Repeat steps 5-1 to 5-3. If the generalization condition is met:

[0105] |δ εtrain -δ εtest |<δ ε (twenty four)

[0106] Then stop training and save the model that has been trained at this point. In the formula, δ ε For the DQN network model during training train The threshold for whether the generalization requirement is met.

[0107] Example:

[0108] 1) Ground clutter suppression dataset parameters

[0109] v0=10m / s, N=44400, M=25, P=37000, Q=7400.

[0110] 2) Radar radiation source parameters

[0111] Pulse repetition count N pulse =4096, wavelength λ = 0.0375m, pulse repetition period 125us, Δf D =0.037m / s.

[0112] 3) Filter bank parameters

[0113] f start =0.1m / s, f end =2.5m / s, Δf1=0.3m / s, Δf2=0.1m / s, order=746. Figure 2 This is an example diagram showing the frequency response of three filters in a filter bank.

[0114] Figure 3 This is a DQN model. The model input is a 1×4 state space, which is passed through a fully connected layer to obtain a 1×256 state space. Then, it passes through a ReLU activation function and is mapped to a 1×2 action space through a second fully connected layer. The model uses the Adam optimizer with a learning rate of 1e-5 and employs the cross-entropy loss function with a delta coefficient. ε Set it to 0.025.

[0115] Figure 4 When K=3, the relationship between the left and right split indices and the left and right boundaries.

[0116] Figure 5The relationship curve between the stopband edge frequency of the filter bank and the improvement factor for a clutter sequence. The filter with a stopband edge frequency of 1.5 is the optimal filter in terms of the improvement factor.

[0117] Figure 6 The relationship curve between the equal interval number K of the decision interval and the model accuracy. When K increases from 2 to 3, the accuracy of the algorithm improves most obviously, increasing by 11.43%. When K increases from 3 to 4 and from 4 to 5, the accuracy increases by 1.35% and 0.33%, respectively.

[0118] Figure 7 The relationship curve between the average difference / standard deviation of the improvement factor corresponding to the false prediction result and the improvement factor corresponding to the true label. When K is 3, the average difference and the standard deviation between the false prediction result and the true label are both the lowest, being 0.04 dB and 0.08 dB, respectively.

[0119] Figure 8 The relationship curve between the equal interval number K of the decision interval and the number of steps and the number of filtering times. As can be seen from the curve, as K increases, the interval segmentation becomes more refined, and the average number of steps and the average number of filtering times both increase. In particular, when the number of segmented intervals increases from 3 to 4, the average number of steps increases by 0.61, and the average number of filtering times increases by 1.96.

Claims

1. A low-speed target adaptive MTI method in a reinforced learning ground clutter environment, characterized in that: Step 1: Calculate the clutter sequence set {C i | i = 0,..., N-1}, the FIR clutter suppression filter set {w j | j = 0,..., M-1}, and the system's detection range in the velocity dimension [-v0, v0] for low-speed targets, the improvement factor sequence set {IF ij | i = 0,..., N-1, j = 0,..., M-1} is calculated, where C i represents the complex sequence of the i-th clutter IQ data, and the sequence length is fixed as N pulse ; N represents the number of ground clutter sequences, M represents the number of filters in the clutter suppression filter set, IF ij represents the improvement factor of the i-th clutter sequence C j using the j-th clutter suppression filter w i ; Step 2: the i-th clutter sequence C i , the improved factor sequence IF ij , the optimal improved factor IF ij in the sequence * , the index of the optimal improved factor IF , the filtering decision label L i of the clutter sequence C i , the constructed terrain clutter suppression dataset {D i} = {(C i , L i ) | i = 0,..., N-1}, and the dataset {D i} is divided into a training set and a test set in a certain proportion, where P is the number of samples in the training set, and Q is the number of samples in the test set. Step 3: reinforcement learning model training phase, according to the i-th sample in the training set of step 2 Obtain the filter decision interval segmentation state space And filter decision label L i , using L i Train the reinforcement learning model; Step 4: Reinforcement learning model testing phase, according to the i-th sample in the test set in step 2 Obtaining a filtered decision interval segmentation state space And a filtering decision label Using Test the reinforcement learning model; Step 5: repeat steps 3 and 4 until a reinforced learning model that converges and has a generalization ability meeting requirements is obtained.

2. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 1, characterized in that: The FIR clutter suppression filter set in step 1 includes: {w j |j = 0,..., M - 1} (1) The filter bank is designed by using the equal-ripple method, and the edge frequencies of the stop band are uniformly distributed in the range of [v start , v end ], where v start is the minimum value of the half spectrum width of the noise supported by the system, and v end is the maximum value of the half spectrum width of the noise supported by the system, and the transition band width is fixed as Δv1, and Δv1 is specifically described as follows: Δvi = αiΔv D (2) where Δv D is the Doppler velocity resolution, and α1is an adjustable parameter. The smaller α1is, the narrower the transition band is, and the stronger the resolution of the clutter and target is, but the order of the filter increases. The order of the filter should be close to but not exceed the power of 2, where N c is the length of the clutter sequence. c is the length of the clutter sequence. The interval between filters is fixed as Δv2: Δv2= α2Δv D (3) α2 is an adjustable parameter, and M represents the number of filters in the clutter suppression filter set, including: In the formula, is a rounding up operation.

3. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 1, characterized in that: The calculation method of the improved factor sequence set {IF ij} in step 1 includes: The autocorrelation matrix of the low-speed target is calculated, the detection range of the low-speed target in the velocity dimension is [-v0, v0], and the autocorrelation function of the target signal is obtained as: where τ mn = t m -t n = (m - n) / fr is the correlation time, λ is the wavelength, and fr is the pulse repetition frequency, the autocorrelation matrix R s of the signal of N pulses is given by: Based on the autocorrelation matrix R of the low-speed target s Find the FIR clutter suppression filter w j For the i-th clutter sequence C i Improvement factor IF ij : .

4. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 2, characterized in that: The step 2 includes: Step 2-1: For each clutter sequence C i , select the filter index with the best improvement factor Step 2-2: Filter index with the best improvement factor As clutter sequence C i Filter decision tag L i : Step 2-3: Record each clutter sequence C i and its filtering decision label L i , construct dataset D = {D i} : {D i} = {(C i ,L i ) | i = 0,..., N - 1} (10) Step 2-4: Split dataset D into training set and test set proportionally where P is the number of samples in the training set and Q is the number of samples in the test set.

5. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 3, characterized in that: The step 3 includes: Step 3-1: For the training set Initialize the left and right boundaries Initialize the left and right boundaries corresponding to the first and last filters in the filter bank {w j}, respectively. Step 3-2: According to left and right boundaries Calculate left split index and right split index Wherein, K is the interval equal division number of each decision; Step 3-3: Calculate the filter corresponding to the two partition indices and Improvement factor IF for the clutter sequence i L and IF i R : Obtaining training samples State space of the training samples Step 3-4: Map the state space to the action space with DQN (Deep Q- Network) represents a 1x2 vector;​ Step 3-5: Compute the label L from the training sample i Compute the current state space Corresponding real action Step 3-6: Perform When the left boundary is updated to the left split index: At this time, the right boundary is updated to the right split index: Step 3-7: Compute cross-entropy loss loss i : where N a represents the number of actions in the action space; y a represents the one-hot encoding of the true action, i.e. if the a-th action is the correct action then y a = 1, otherwise equals 0; represents the action space predicted probability of action a; Step 3-8: loss i Backpropagation is performed, the gradient is updated, and the DQN is trained. Step 3-9: When one of the following two conditions occurs, the current training sample Loop end: 1) i.e. left and right borders coincide, return 2) i.e. the difference between the left and right boundaries is less than K, compute to the improvement factor select the index corresponding to the optimal improvement factor return Ind * ; if the above two conditions are not satisfied, go to step 3-2; Step 3-10: Let i <- i + 1, re-execute Step 3-1 to Step 3-9 for the next training set sample Training is performed until the training set is traversed i.e. i = P.

6. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 4, characterized in that: The step 4 includes: Step 4-1: For each training sample in the training set Initialize left and right boundaries Initialize left and right boundaries corresponding to the first and last filter in the filter set {w j}, respectively Step 4-2: According to left and right boundaries Calculate left split index and right split index Wherein, K is the interval equal division number of each decision; Step 4-3: Compute the filter corresponding to the two partition indices and to the clutter sequence Compute the improvement factor IF i L and IF i R : Obtaining a test sample State space of the test sample Step 4-4: Map the state space to the action space with DQN (Deep Q-Network) represents a 1x2 vector; action is obtained from the action space i :​ action i ∈{0,1};perform action action i , action i = 0, left boundary is updated to left split index: action i = 1, right boundary is updated to right split index: Step 4-5: The current test sample is considered as a failure when one of the following two conditions occurs Loop end: 1) i.e. left and right borders coincide, return 2) i.e. the difference between the left and right boundaries is less than K, compute to the improvement factor select the index corresponding to the optimal improvement factor return Ind * ; if the above two conditions are not satisfied, go to step 4-2; Step 4-6: Let i <- i + 1, re-execute Step 4-1 through Step 4-5 for the next test set sample Test until the test set is traversed i.e., i = Q.

7. The low-speed target adaptive MTI method in a reinforced learning ground clutter environment according to claim 5, characterized in that: The step 5 includes: Step 5-1: Perform Step 3 to obtain the trained model DQN train , and the accuracy rate δ train of the training set D εtrain ; Step 5-2: Perform Step 4 with the model DQN of Step 5-1 train to obtain the accuracy δ test on the test set D εtest ; Step 5-3: repeat steps 5-1 to 5-3, and if the generalization condition is met: Step 5-3: repeat steps 5-1 to 5-3, and if the generalization condition is met: | δ εtrain - δ εtest | < δ ε (24) Then stop training and save the model trained at this time where δ ε is the network model DQN train threshold value whether the generalization requirement is met.

Images