Doppler tolerant discrete waveform design method based on reinforcement learning

Abstract

The invention discloses a Doppler tolerant discrete waveform design method based on reinforcement learning, which takes CCS as a theoretical basis, and generates an anti-Doppler sequence by using good complementarity of the CCS. Meanwhile, a discrete sequence generation problem is processed by means of reinforcement learning, so that a discrete sequence which is relatively simple in hardware implementation and has the advantages of low cost and low power consumption is designed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of radar, and particularly relates to a Doppler-tolerant discrete waveform design method based on reinforcement learning. BACKGROUND

[0002] In the field of radar systems, designing a single-mode sequence set with excellent Auto-Ambiguity Function (AF) and Cross-Ambiguity Function (CAF) characteristics has been a key challenge for a long time. At present, sequence design methods are mainly divided into two categories. The first category is sequence design based on mathematical construction, typical of which are Barker code, M-sequence, Gold sequence, Golomb code, Zadoff-Chu code, Frank code, and Complete Complementary Sequences (CCS). This kind of sequence only considers correlation when constructing, but ignores the influence of Doppler effect. In high-speed scenarios, the rapid movement of targets will cause significant Doppler frequency shift, making these sequences unable to respond effectively, ultimately leading to a decline in radar performance, and making it difficult to meet the demand for high-precision detection. The other category is sequence design based on numerical optimization theory, such as Prouhet-Thue-Morse (PTM) sequence, generalized Prouhet-Thue-Morse (GPTM) sequence, etc. They take into account the Doppler effect and perform relatively well in high-speed scenarios. However, this kind of sequence generates continuous phase sequences. In terms of hardware implementation, continuous phase sequences require more complex circuit structures and higher precision signal processing, which not only greatly increases the hardware generation cost, but also leads to high power consumption, severely limiting its widespread application in industrial production. SUMMARY

[0003] In order to overcome the deficiencies of the prior art, the application provides a Doppler-tolerant discrete waveform design method based on reinforcement learning, which is based on CCS as a theoretical basis to generate anti-Doppler sequences using its good complementarity. At the same time, with the help of reinforcement learning to process discrete sequence generation problems, a discrete sequence with the advantages of simple hardware implementation, low cost and low power consumption is designed.

[0004] The technical scheme adopted by the application to solve its technical problems is as follows:

[0005] Step 1: Model the Doppler-tolerant sequence and impose discrete constraints on the sequence elements, and construct the optimization objective;

[0006] Step 2: Construct a reinforcement learning model based on deep neural network (DNN) and Monte Carlo tree search (MCTS), convert sequence design into sequence filling problem, and construct evaluation function, reward function and loss function in reinforcement learning based on optimization objective.

[0007] Preferably, the step 1 is specifically:

[0008] Step 1-1: Doppler tolerant sequence modeling:

[0009] Consider an MxK single-mode sequence of length L as follows:

[0010]

[0011] Wherein:

[0012]

[0013] Wherein, j represents an imaginary number, D represents the number of discrete phase values; define the matrix X = [x0, x1, …, x K-1 ] as an (M, K, L) sequence family; M is the number of sequences, K is the number of pulses, and L is the length of the sequence;

[0014] Define the aperiodic correlation function of any two sequences in X as:

[0015]

[0016] Wherein (·) * represents complex conjugate, τ is a delay variable; by defining the shift matrix as:

[0017]

[0018] The aperiodic correlation matrix of x k is written as:

[0019]

[0020] If X satisfies:

[0021]

[0022] Where a, b = 0, 1, …, M-1, δ(l) is the Kronecker function, then the (M, K, L) sequence family X is called (M, K, L)-ACCS.

[0023] Step 1-2: Construct optimization objective

[0024] Consider a multiple-input multiple-output radar system with M transmitters and M receivers, transmitting K pulses within a coherent processing interval (CPI), and the length of the transmission sequence is L.

[0025] with variable time delay l and normalized Doppler shift θ d The ambiguity function is:

[0026]

[0027] Normalized Doppler interval θ d is discretized into N intervals, where and d = 0, 1, …, N - 1, N represents the number of pulses; considering both the slow-time Doppler shift and the fast-time Doppler shift, the slow-time Doppler shift is denoted as θ d = 2πf d T PRI , and the fast-time Doppler shift is denoted as η d = 2πf d T c , where T PRI is the pulse repetition time, and T c is the chip length;

[0028] The peak side lobe levels APSL and CPSL of AF and CAF are defined based on the ambiguity function;

[0029] APSL is defined as:

[0030]

[0031] CPSL is defined as:

[0032]

[0033] where 1 M is an M x M identity matrix, and f D represents the Doppler shift;

[0034] The performance of AF and CAF is simultaneously optimized, and the objective function is defined as:

[0035]

[0036] that is, the maximum APSL or CPSL is minimized, so that the performance of all ambiguity functions of the designed sequence set is optimal.

[0037] Preferably, the step 2 is specifically:

[0038] Step 2-1: input and output of DNN;

[0039] The output of the DNN is denoted as Each time an intermediate state s i is input into the DNN ψ θ with parameters θ, it will output a reward estimate And a probabilistic action selection strategy P;

[0040] Step 2-2: Use MCTS for symbol filling;

[0041] Symbol filling is performed using MCTS, with symbol filling conducted each round under the guidance of MCTS; in state s i In the middle, the agent follows the probabilistic policy Π(s) given by MCTS. i )choose Use one symbol to fill the next position in the sequence set. At this position, this action will generate a new state s. i+1 ;

[0042] In state s i In MCTS, the agent first sets the root node v0 to S. i It initializes a "visited tree" and then performs look-ahead simulations along the visited tree, starting from the root node; each simulation proceeds along the path of the visited tree until an unvisited node v is encountered. L This unvisited node will be evaluated by the DNN and added to the visited tree; the newly added node v L To be given a metric In state s i In the final movement choice distribution It is based on MCTS in state s i The access counts of the child nodes of the root node are generated;

[0043] Final Process After a certain number of time steps, the agent obtains a complete sequence set X and calculates its metric. This results in a reward R(X); then, the agent assigns all states s in the set to this reward. i and its corresponding Π(s) i )and It is stored as experience. One round provides DNN with One experience;

[0044] Steps 2-3: DNN update;

[0045] Given experience and Actual reward Value function approximation for improving DNNs Meanwhile, MCTS provides the strategy Π(s) i ) used to improve policy estimation P(s) of DNN i Therefore, the training process is to make P and Closer to Π and

[0046] In one iteration, using the current DNNψ θ The guided MCTS is run for G rounds to generate experience; then a new DNNψ is trained using the experience accumulated over the most recent z×G rounds. θ′ ;

[0047] In the next iteration, use ψ θ′ The guided MCTS performs G-time sequence generation to generate further experience. This experience is then used to train another new DNN, and so on, through multiple iterations, to obtain a DNN with the maximum reward function R(X) by learning the latest experience accumulated during the search process, thus achieving sequence encoding with the best self-blurring and mutual-blurring performance;

[0048] Steps 2-4: Define the evaluation function;

[0049] M(X) = P X

[0050] X is a sequence family with M sequences, K pulses, and L sequence lengths;

[0051] The reward function is designed as a linear mapping from the range of values ​​of the evaluation function to the interval [-1, 1]. Specifically, the reward function is defined as follows:

[0052]

[0053] Where M u It is max(APSL,CPSL) obtained by randomly generating a family of sequences.

[0054] Preferably, the implementation principle of the MCTS module is as follows:

[0055] In each intermediate state s i Monte Carlo Tree Search (MCTS) determines the policy Π(s) through multi-step look-ahead simulation. i For each new node encountered in a simulation, MCST performs a random simulation from that node until a leaf node is reached to obtain a reward; then, the reward is randomly sampled in all simulations and used to evaluate the node.

[0056] This paper combines the Deep Q-Network (DQN) method and the deep neural network ψ0 to analyze nodes, and employs the Parallel Upper Confidence Interval Tree Search (P-UCT) algorithm to accelerate the search process. A search tree is constructed based on the root node v0; each node in this tree has... Edges, corresponding to each state Possible moves; each edge, denoted as (v i ,a j ),in, Store four statistics: visit count N(v i ,a j ), average reward Q(v i ,a j ), edge probability P(v i ,a j ) and unobserved sample count O(v i ,a j );

[0057] MCTS uses a deep neural network ψ0to evaluate each node; when v i is input into ψ0, it generates a policy estimate P and a reward estimate R′, i.e. (P,R′) = ψ0(v i ). MCTS performs forward simulation, starting from the root node v0, first creating an "access tree" to track all nodes accessed during MCTS; at initialization, there is only one root node in the access tree, and MCTS performs q simulations on this access tree;

[0058] Introduce P-UCT for parallel search; specifically, parallelize q simulations through multi-threading; set a main process that undertakes the tasks of selection, expansion and backpropagation, and allocate evaluation tasks to worker processes; at the same time, deploy W worker processes that use GPUs to evaluate nodes, thereby improving the overall efficiency of the search algorithm; each simulation follows the following steps:

[0059] (1) Selection: all simulations start from the root node v0 until the first encounter with an unvisited node. During simulation, always select the edge with the highest upper confidence interval; specifically, the simulation selects edge j i to visit at each node v * , and the calculation formula is:

[0060]

[0061] where c p is a constant used to control the trade-off between exploration and exploitation;

[0062] (2) Expansion and evaluation: when the simulation encounters an unvisited node v L , the main process sends an evaluation task to the worker process and performs an incomplete update:

[0063] O(v i ,a j ) = O(v i ,a j ) + 1

[0064] The evaluation task is to evaluate v Land get (P, R') = ψ0(v L ), where the policy distribution is If all the worker processes are occupied, the master process will wait for the evaluation task to complete; otherwise, if there are available worker processes, the master process will start the next simulation. Once the evaluation task is completed, this new node will be added to the visit tree, and the edge parameters of v L are initialized as N(v L ,a j ) = 0, Q(v L ,a j ) = 0, and P(v L ,a j ) = P L (j), where

[0065] (3) Backpropagation: After adding node v L to the visit tree, the simulation updates all nodes on the trajectory that encounters v L . Update each edge (v i ,a j ) on the trajectory respectively:

[0066] O(v i ,a j ) = O(v i ,a j ) - 1

[0067] N(v i ,a j ) = N(v i ,a j ) + 1

[0068]

[0069] After completing q simulations, MCTS gives the move selection probability of node v0 by the following formula:

[0070]

[0071] Parameter τ is a temperature parameter; in a training episode, set τ to 1 in the first third of time steps and set τ to 10 in the remaining time steps -4 , so that in the first third of time steps, the policy selection is more random, encouraging exploration; in the remaining time steps, the policy selection is more deterministic, biased towards exploitation;

[0072] In the process of training iteration, when generating a sequence set in each training episode to provide experience for the deep neural network, Dirichlet noise is used, that is:

[0073]

[0074] wherein, is a positive real parameter; in MCTS, Dirichlet noise is added to the prior probability of the root node; specifically, for each action a j , its prior probability P(a j ) is updated as:

[0075] P(a j ) = (1 - ∈) · P(a j ) + ∈ · η j

[0076] where ∈ is the weight of the noise, controlling the degree of influence of the noise; η j is a random value sampled from Dirichlet distribution, satisfying In this way, Dirichlet noise can effectively balance exploration and exploitation during training.

[0077] Preferably, the deep neural network DNN is implemented as follows:

[0078] The deep neural network DNN is a deep convolutional network ConvNet, which is composed of six convolutional layers and combines batch normalization and rectified linear unit ReLU activation function;

[0079] (1) Input: The convolutional network takes a stack of images of size K' × N' × D as input; for a state s i , i.e., a partially filled K' × N' × D sequence set pattern, it is first converted into an image of size K' × N' × 1, and then converted into a stack of images of size K' × N' × D through feature extraction; D represents the number of discrete phase values, and the range of sequence element values is

[0080] (2) Feature extraction: Feature extraction is the process of converting an image of size K' × N' × 1 into a stack of images of size K' × N' × D; this stack is composed of D feature planes, each of which corresponds to a symbolic value indicating the presence or absence of the symbolic value;

[0081] (3) Output: For each state s i , the deep neural network outputs a policy estimate, i.e., a probability distribution , as the prior probability of the i th edge of state s , and outputs a scalar estimate R' ∈ [-1, 1] as the expected reward of state s i ;

[0082] (4) Training: after G games are played, the experience accumulated in the last z x G games is used to update the deep neural network using the stochastic gradient descent method; the size of the mini-batch is set to batchSize, and the loss function is defined as the sum of the mean square error and the cross-entropy loss:

[0083]

[0084] The first term in the above formula measures the difference between the reward value R' predicted by the deep neural network and the actual reward value R, ensuring that the deep neural network can accurately predict the expected reward of each state; the second term measures the difference between the policy distribution P predicted by the deep neural network and the policy distribution Π generated by MCTS, by minimizing the second term, the deep neural network can better fit the policy distribution of MCTS, thereby improving the accuracy of action selection; the third term is an L2 regularization term, which is used to constrain the model parameters θ to prevent model overfitting and improve its generalization ability; during the entire training process, the learning rate is fixed at 10 -4 to stably optimize the model parameters.

[0085] An electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned Doppler tolerant discrete waveform design method.

[0086] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the above-mentioned Doppler tolerant discrete waveform design method.

[0087] A chip, comprising: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes the above-mentioned Doppler tolerant discrete waveform design method.

[0088] A computer program product, the computer program product comprising a computer storage medium storing a computer program, the computer program comprising instructions executable by at least one processor, when the instructions are executed by the at least one processor, the above-mentioned Doppler tolerant discrete waveform design method is implemented.

[0089] The beneficial effects of the present application are as follows:

[0090] ​​​1. Reinforcement learning-based innovative architecture: The invention builds a reinforcement learning model based on deep neural network (DNN) and Monte Carlo tree search (MCTS). Unlike traditional methods, this model cleverly transforms the sequence design into a sequence filling problem, through the iterative "sequence search using MCTS" and "DNN update" process, MCTS and DNN cooperate with each other and improve together. MCTS uses multi-step lookahead simulation to determine the strategy, providing training experience for DNN; DNN evaluates the nodes in MCTS, helping MCTS make better decisions. This collaborative mechanism is not available in traditional sequence design methods. In traditional methods, sequence design based on mathematical construction ignores the Doppler effect, and sequence design based on numerical optimization theory has hardware implementation problems, while the reinforcement learning model of the invention effectively overcomes these problems and can generate discrete sequences that are more suitable for complex scenarios.

[0091] 2. Innovative evaluation and reward function design: In terms of optimization objectives, appropriate evaluation and reward functions are defined. The evaluation function aims to minimize the maximum APSL or CPSL, optimizing the performance of the sequence set's ambiguity function; the reward function is designed as a linear mapping from the evaluation function value range to the interval [-1, 1], and the reward value is determined by comparing the max(APSL, CPSL) average value of the randomly generated sequence family. This design effectively guides the model to learn towards generating sequences with better AF and CAF performance, and compared to existing methods that do not fully consider the Doppler effect or only optimize a single indicator, the invention has innovation in the comprehensiveness and effectiveness of performance optimization.

[0092] 3. Optimization of MCTS module: In the MCTS module, the parallel upper confidence bound tree search algorithm (P-UCT) is introduced for parallel search. Traditional MCTS is executed sequentially, which is slow when dealing with large-scale search problems, while P-UCT greatly improves search efficiency through multi-thread parallel simulation. At the same time, in the node evaluation, the deep Q network (DQN) method and deep neural network are combined, abandoning the traditional random simulation evaluation method, making the node evaluation more accurate and the search process more efficient, and quickly finding more potential sequence combinations, which is an important improvement over traditional MCTS algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0093] Figure 1 For the application of CCS in MIMO radar;

[0094] Figure 2 For the reinforcement learning framework diagram;

[0095] Figure 3 For the MCTS symbol filling process;

[0096] Figure 4 For the DNN structure diagram;

[0097] Figure 5 AF map for smart home scenario;

[0098] Figure 6 AF map for vehicle radar scenario;

[0099] Figure 7 AF map for smart traffic monitoring scenario;

[0100] Figure 8 AF map for industrial detection scenario;

[0101] Figure 9 AF map for vehicle radar scenario. DETAILED DESCRIPTION

[0102] The application will be further described below in conjunction with the accompanying drawings and examples.

[0103] The method of the application comprises the following steps: 1) modeling the Doppler tolerant sequence and imposing discrete constraints on the sequence elements to build an optimization target; 2) building a reinforcement learning model based on deep neural networks (DNN) and Monte Carlo tree search (MCTS), converting the sequence design into a sequence filling problem, and building an evaluation function, a reward function and a loss function in reinforcement learning based on the optimization target;

[0104] The following is a specific technical solution:

[0105] 1) modeling the Doppler tolerant sequence and imposing discrete constraints on the sequence elements to build an optimization target:

[0106] Modeling of Doppler tolerant sequence:

[0107] Consider an MxK single-mode sequence of length L as follows:

[0108]

[0109] where

[0110]

[0111] j represents an imaginary number, and D represents the number of discrete phase values. Define the matrix X = [x0, x1, …, x K-1 ] as a (M, K, L) sequence family. Where M is the number of sequences, K is the number of pulses, and L is the sequence length.

[0112] Next, define the aperiodic correlation function of any two sequences in X as:

[0113]

[0114] where (·) * denotes complex conjugate, τ is a delay variable. By defining a shift matrix J as

[0115]

[0116] x k The aperiodic correlation matrix of X can be written as:

[0117]

[0118] If X satisfies where a, b = 0, 1, …, M-1, δ(l) is the Kronecker function, then the family of sequences X is called (M, K, L)-ACCS.

[0119] The optimization objective is constructed as:

[0120] Consider a multiple input multiple output (MIMO) radar system with M transmitters and M receivers. Within a coherent processing interval (CPI), K pulses are transmitted, and the length of the transmit sequence is L, as shown in Figure 1

[0121] The ambiguity function with variable time delay l and normalized Doppler shift θ d is:

[0122]

[0123] The normalized Doppler interval θ d is discretized into N intervals, where and d = 0, 1, …, N-1, N represents the number of pulses, and the present application simultaneously considers the slow-time Doppler shift and the fast-time Doppler shift, the slow-time Doppler shift is expressed in radians as θ d = 2πf d T PRI , and the fast-time Doppler shift is expressed in radians as η d = 2πf d T c , where T PRI is the pulse repetition time, and T c is the chip length.

[0124] The peak side lobe levels APSL and CPSL of AF and CAF are defined based on the ambiguity function;

[0125] APSL is defined as:

[0126] ​

[0127] CPSL is defined as:

[0128]

[0129] The objective function is defined as:

[0130]

[0131] i.e. minimizing the maximum of APSL or CPSL, so that all the ambiguity function performance of the designed sequence set is optimal. 2) Construct a reinforcement learning model based on DNN and MCTS, convert the sequence design into a sequence filling problem, and construct the evaluation function, reward function and loss function in reinforcement learning based on the optimization objective;

[0132] The present application proposes a reinforcement learning model based on DNN and MCTS, as shown in Figure 2 ;

[0133] This model can be summarized as an iterative process of "sequence search using MCTS" and "DNN update". On the one hand, "searching using MCTS" provides experience for training DNN, so that DNN can improve its evaluation of the goodness of the state in the search. On the other hand, the better evaluation of the state by DNN enables MCTS to make better decisions, thereby providing higher quality experience to train DNN in turn. Through the iterative process, MCTS and DNN promote each other in the underlying reinforcement learning process and gradually improve.

[0134] The specific training process is as follows:

[0135] A. Input and output of DNN;

[0136] DNN aims to estimate the value function and policy function of the intermediate state. The value function is the expected terminal reward estimate of a given intermediate state. Specifically, the output of DNN can be represented as Each time an intermediate state s i is input into DNN ψ θ with parameters θ, it will output a reward estimate (value function estimate) and a probabilistic action selection policy P (policy function estimate, policy P is a distribution over all possible next actions given the current state s i ).

[0137] B. Symbol filling using MCTS;

[0138] Figure 2 The left part of the algorithm iteration is symbol filling using MCTS. The specific process is as follows Figure 3Each round is shown to be played with symbol filling guided by MCTS. Figure 3 The upper half of FIG. 1 shows all states in one filling process, where a square represents a position in the sequence set: a gray square indicates that the position has been filled (with a value in the set); a white square indicates that the position is still empty (with a value of 0). The initial state of each set is the all-zero state s0. In state s i , the agent will choose i symbols to fill the next position in the sequence set according to the probability policy given by MCTS, which is a distribution over possible actions. This action will result in a new state s i+1 .

[0139] Figure 3 The lower half of FIG. 1 shows the MCTS process in each state s i , where each node represents a possible state in the forward search. In the MCTS of state s i , the agent first sets the root node v0 to S i , and initializes an “explored tree” (which is used to record all nodes visited in MCTS, initially only the root node). Then, starting from the root node, the agent performs forward simulation along the explored tree. Each simulation will follow a path in the explored tree until it encounters an unvisited node v L . This unvisited node will be evaluated by the DNN and added to the explored tree (i.e., the newly added node v L will be assigned a metric to help future simulations evaluate which next action to choose when visiting the same node v L again). As more and more simulations are performed, the size of the tree gradually increases. The metric used to choose the next action also changes as the nodes are visited in consecutive simulations. In short, nodes that are estimated to be good are visited frequently, while nodes that are estimated to be bad are visited rarely. In state s i , the final move selection distribution is generated according to the visit counts of the children of the root node in state s i . After time steps, the agent obtains a complete sequence set X and computes its metric , which gives the reward R(X). Then, the agent will fill in all states s i in the set with their corresponding i and As experience is stored. One episode provides DNN with one experience.

[0140] C. DNN is updated;

[0141] Figure 2 The right part is to train DNN based on accumulated experience. From the previous description, we know that MCTS is guided by DNN. The ability of DNN determines the performance of MCTS, because a better DNN can more accurately evaluate the nodes in MCTS. In the extreme case, if the DNN perfectly knows which sequence set pattern is good and which is bad, then MCTS will always move in the optimal direction, so the selected action is also optimal. However, the DNN is randomly initialized, and its evaluation of nodes is quite random and inaccurate at the beginning. Therefore, the goal of this framework is to improve this DNN using the experience generated by MCTS.

[0142] In the DNN updating process, DNN is updated by learning the latest experience accumulated during the game process. Given experience and actual reward can be used to improve the value function approximation of DNN At the same time, the strategy Π(s i ) given by MCTS can be used to improve the strategy estimation P(s i ) of DNN. Therefore, the training process is to make P and closer to Π and When using MCTS for sequence generation to generate experience, Dirichlet noise is added to the prior probability of the root node v0 to increase the exploration ability of MCTS.

[0143] Overall, in one iteration, G episodes of MCTS guided by the current DNN ψ θ are used to generate experience; then a new DNN ψ θ′ is trained using the experience accumulated in the last z x G episodes.

[0144] In the next iteration, MCTS guided by ψ θ′ is used to generate G sequences, generating further experience. Then these experiences will be used to train another new DNN, and so on. After several iterations, the DNN that maximizes the reward function R(X) is obtained by learning the latest experience accumulated in the search process, to obtain the sequence encoding with the best self-blurring and mutual-blurring performance.

[0145] According to the reinforcement learning model proposed above, the evaluation function is defined as:

[0146] M(X) = P XX is a sequence family with sequence number M, pulse number K, and sequence length L.

[0147] To map the found sequence family X to the reward R(X) and facilitate calculation, the reward function is designed as a linear mapping from the value range of the evaluation function to the interval [-1, 1], and the reward function is specifically defined as:

[0148]

[0149] where M u is the max(APSL, CPSL) obtained by randomly generating a sequence family, specifically, 100 random generation experiments are performed before the experiment starts, and finally the average value of the max(APSL, CPSL) of the 100 random experiments is obtained. If the generated sequence family is greater than M u , it means that the generation effect is not ideal, and the reward R is set to -1.

[0150] The following is an explanation of the specific implementation principles of the MCTS module and the DNN module.

[0151] A. MCTS module

[0152] At each intermediate state s i , the Monte Carlo tree search (MCTS) determines the strategy Π(s i ) through multi-step forward simulation. In the simulation process, nodes with more potential are accessed more frequently, while nodes with less potential are accessed less frequently. The problem is how to determine which nodes have more potential and which nodes have less potential in the simulation, or how to evaluate a node in MCTS. In the traditional MCTS algorithm, the evaluation of such nodes is completed through random simulation. That is, for each new node encountered in each simulation, MCST performs random simulation from the node until it reaches a leaf node to obtain the reward. Then, the rewards randomly sampled in all simulations are used to evaluate the nodes.

[0153] Unlike the random simulation method, the framework combines the deep Q network (DQN) method and the deep neural network ψ0 to analyze the nodes, and uses the parallel upper confidence bound tree search algorithm (P-UCT) to speed up the search process. Based on the root node v0, a search tree can be established. Each node in this tree has edges, corresponding to possible moves in each state. Each edge, denoted as (v i , a j ), has stores four statistical information: visit count N(v i , a j ), average reward Q(v i , aj ), the edge probability P(v i ,a j ) and the number of unobserved samples O(v i ,a j ). MCTS uses a deep neural network ψ0to evaluate each node. When v i is input into ψ0, it generates a policy estimate P and a reward estimate R', i.e., (P, R') = ψ0(v i ). MCTS performs forward simulation, starting from the root node v0, first creating an "access tree" to track all nodes visited during the MCTS process. At initialization, there is only one root node in the access tree, and MCTS performs q simulations on this access tree.

[0154] It is worth noting that traditional MCTS is executed sequentially. Therefore, when facing large-scale search problems, its processing speed is quite slow. To solve this problem, P-UCT is introduced to perform parallel search. Specifically, q simulations are parallelized through multi-threading, which greatly speeds up the search process. Its working principle is as follows: set a master process, which undertakes the tasks of selection, expansion, and back propagation, and assigns evaluation tasks to worker processes. At the same time, deploy W worker processes, which use GPU to evaluate nodes, thereby improving the overall efficiency of the search algorithm. Each simulation within this framework follows the following steps:

[0155] 1. Selection: All simulations start from the root node v0 until the first encounter with an unvisited node. During the simulation process, the edge with the highest upper confidence interval is always selected. Specifically, the simulation selects edge j i to visit at each node v * , and the calculation formula is:

[0156]

[0157] where c p is a constant used to control the trade-off between exploration and exploitation.

[0158] 2. Expansion and evaluation: When the simulation encounters an unvisited node v L , the master process sends an evaluation task to the worker process and performs an incomplete update:

[0159] O(v i ,a j ) = O(v i ,a j ) + 1

[0160] The evaluation task is to evaluate v L using a deep neural network and obtain (P, R') = ψ0(vL ), where the strategy distribution is If all worker processes are occupied, the main process will wait for the evaluation task to complete. Otherwise, if a worker process becomes available, the main process will start the next simulation. Once the evaluation task is complete, the new node will be added to the access tree, and v L The edge parameters are initialized to N(v L ,a j ) = 0, Q(v) L ,a j )=0, P(v L ,a j ) = P L (j), where

[0161] 3. Backpropagation: When node v L After being added to the access tree, the simulation will update the encounter with v L All nodes on the trajectory. Update each edge (v) on the trajectory respectively. i ,a j (including v) L ):

[0162] O(v i ,a j )=O(v i ,a j )-1

[0163] N(v i ,a j )=N(v i ,a j )+1

[0164]

[0165] After completing q simulations, MCTS gives the probability of node v0 choosing to move using the following formula:

[0166]

[0167] The parameter τ is a temperature parameter. In a training segment, τ is set to 1 for the first third of the time steps and to 10 for the remaining time steps. -4 This approach allows for more random strategy selection in the first third of the time steps, encouraging exploration; while in the remaining time steps, it makes strategy selection more deterministic, favoring exploitation.

[0168] During training iterations, Dirichlet noise is used when generating sequence sets in each training segment to provide experience for the deep neural network.

[0169]

[0170] where, is a positive real parameter. The introduction of this noise helps to perform additional exploration during training, avoiding the agent to converge prematurely to a local optimal solution. The Dirichlet noise encourages the agent to try more possible actions by introducing randomness, not just the currently estimated optimal action. It also helps to avoid overfitting, preventing the agent from relying too much on early experiences, thus maintaining a broad exploration of the environment. In MCTS, the Dirichlet noise is added to the prior probability of the root node. Specifically, for each action a j , its prior probability P(a j ) is updated as: P(a j ) = (1 - ∈) · P(a j ) + ∈ · η j

[0171] where ∈ is the weight of the noise, controlling the degree of influence of the noise; η j is a random value sampled from Dirichlet distribution, satisfying In this way, the Dirichlet noise can effectively balance exploration and exploitation during training.

[0172] B. Deep Neural Network (DNN)

[0173] The deep neural network (DNN) implemented in this invention is a deep convolutional network (ConvNets). This convolutional network consists of six convolutional layers, combined with batch normalization and rectified linear unit (ReLU) activation functions. The specific details are shown in Figure 4 .

[0174] 1. Input: The convolutional network takes an image stack of size K' x N' x D as input. For a state s i (i.e., a partially filled K' x N' x D sequence set pattern), it is first converted into an image of size K' x N' x 1 (usually set as K' = MK / L, N' = L), and then converted into an image stack of size K' x N' x D through feature extraction. Here, D represents the number of discrete phase values, and the range of sequence elements is

[0175] 2. Feature extraction: Feature extraction is the process of converting an image of size K' x N' x 1 into an image stack of size K' x N' x D. This stack consists of D feature planes, each corresponding to a symbolic value, indicating the presence or absence of that symbolic value. For example, the dth feature plane x d represents the presence of the symbolic value presence: if the value at position (i,j) in the image of size K' x N' x 1 is then X d (i,j) = 1, otherwise 0.

[0176] 3. Output: for each state S i , the deep neural network outputs a policy estimate (i.e., a probability distribution) as the prior probability of choosing an edge of the graph at state s i , and a scalar estimate R' e [-1, 1] as the expected reward at state S i .

[0177] 4. Training: after G games, the deep neural network is updated using the experience accumulated in the last z x G games (i.e., the experience) using stochastic gradient descent. The size of the mini-batch is set to batchSize, and mini-batches are randomly selected from the experience to train the convolutional network. For each mini-batch, the loss function is defined as the sum of the mean squared error and the cross-entropy loss:

[0178]

[0179] The first term measures the difference between the reward value R' predicted by the deep neural network and the actual reward value R, ensuring that the deep neural network accurately predicts the expected reward at each state. The second term measures the difference between the policy distribution P predicted by the deep neural network and the policy distribution Π generated by MCTS. By minimizing this term, the deep neural network can better fit the policy distribution of MCTS, improving the accuracy of action selection. The third term is an L2 regularization term that constrains the model parameters θ, preventing overfitting and improving the generalization ability of the model. Throughout the training process, the learning rate is fixed at 10 -4 to stably optimize the model parameters.

[0180] Embodiment:

[0181] The simulation results under different scenarios are shown below:

[0182] Table 1

[0183]

[0184] Table 2 Sequence Results (Binary Sequence)

[0185] Sequence Class APSL CPSL 【2、8、8】 -24.08 -24.08 【4，8，8】 -16.12 -16.10 【2、16、16】 -23.03 -22.97 【4、16、16】 -18.62 -19.22

[0186] Table 3

[0187] ​

[0188] Table 4 Sequence results (binary sequences)

[0189]

[0190]

[0191] Table 5

[0192]

[0193] Table 6 Sequence results (binary sequences)

[0194] Sequence Class APSL CPSL 【2、8、8】 -22.76 -20.56 【2、16、16】 -24.23 -21.56

[0195] Table 7

[0196]

[0197] Table 8 Sequence results (binary sequences)

[0198] Sequence Class APSL CPSL 【2、8、8】 -19.44 -19.63 【4，8，8】 -14.23 -14.41 【2、16、16】 -22.00 -20.56 【4、16、16】 -19.87 -18.62

[0199] Table 9

[0200]

[0201]

[0202] Table 10 Sequence results (binary sequences)

[0203] Sequence Class APSL CPSL 【2、8、8】 -23.70 -24.08 【4，8，8】 -17.77 -16.12 【2、16、16】 -22.14 -22.97 【4、16、16】 -19.22 -19.22

[0204] The AlphaCCS: Doppler-tolerant discrete waveform design method based on reinforcement learning provided by the application effectively solves the performance limitations of traditional sequence design under the Doppler effect and the hardware implementation difficulty of existing anti-Doppler sequences by combining reinforcement learning and discrete sequence optimization.

Claims

1. A Doppler-tolerant discrete waveform design method based on reinforcement learning, characterized in that, Comprising the following steps: Step 1: model the Doppler tolerant sequence and impose discrete constraints on the sequence elements, and build an optimization target; Step 2: build a reinforcement learning model based on a deep neural network DNN and a Monte Carlo tree search MCTS, convert the sequence design into a sequence filling problem, and build an evaluation function, a reward function and a loss function in reinforcement learning based on the optimization target.

2. The method of claim 1, wherein, Said step 1 is specifically: Step 1-1: Doppler tolerant sequence modeling: Consider an MxK single-mode sequence of length L as follows: Wherein: where j represents imaginary number, D represents the number of discrete phase values; define matrix X = [x0, x1, …, x K-1 ] as (M, K, L) sequence family; M is the number of sequences, K is the number of pulses, and L is the sequence length; Define the aperiodic correlation function of any two sequences in X as: where (·) * denotes complex conjugation, and τ is a delay variable; the shift matrix is defined by x k The non-periodic correlation matrix of is written as: If X satisfies: Where a, b = 0, 1, …, M-1, and δ(l) is the Kronecker function, then the (M, K, L) sequence family X is called (M, K, L)-ACCS; Step 1-2: build an optimization target; Consider a multiple-input multiple-output radar system with M transmitters and M receivers, transmit K pulses within a coherent processing interval CPI, and the length of the transmission sequence is L; Having a variable time delay l and a normalized Doppler shift θ d The ambiguity function is: Normalized Doppler interval θ d is discretized into N intervals, where and d = 0, 1,..., N - 1, N represents the number of pulses; considering both slow-time Doppler shift and fast-time Doppler shift, the slow-time Doppler shift is expressed in radian as θ d = 2πf d T PRI , and the fast-time Doppler shift is expressed in radian as η d = 2πf d T c , where T PRI is the pulse repetition time, and T c is the chip length; Define the peak side lobe level APSL and CPSL of AF and CAF based on the ambiguity function; APSL is defined as: CPSL is defined as: wherein 1 M denotes an identity matrix of size M x M, f D denotes the Doppler shift; Optimize the performance of AF and CAF at the same time, and the objective function is defined as: That is, minimize the maximum APSL or CPSL, so that the performance of all ambiguity functions of the designed sequence set is optimal.

3. The method of claim 2, wherein, Said step 2 is specifically: Step 2-1: input and output of DNN; The output of the DNN is denoted as Each time an intermediate state s i is input to the DNN ψ with parameters θ θ At time t, it will output a reward estimate and a probabilistic action selection policy P; Step 2-2: symbol filling using MCTS; Symbol filling is performed using MCTS, with each round of symbol filling guided by MCTS; in state s i , the agent selects i symbols to fill the next positions in the sequence set according to the probability policy Π(s ) given by MCTS, which action results in a new state s i+1 ; In a MCTS of state s i , the agent first sets a root node v0 to S i , and initializes an "explored tree", then, starting from the root node, performs a forward simulation along the explored tree; each simulation proceeds along the path of the explored tree until an unvisited node v L is encountered; this unvisited node is evaluated by the DNN and added to the explored tree, the newly added node v L is assigned a metric ; in state s i , the final move selection distribution is generated from the visit counts of the children of the root node in state s i ; Final Process After a certain number of time steps, the agent obtains a complete sequence set X and calculates its metric. This results in a reward R(X); then, the agent assigns all states s in the set to this reward. i and its corresponding Π(s) i )and It is stored as experience. One round provides DNN with One experience; Step 2-3: DNN update; Given experience And Actual reward Value function approximation for improving DNN The policy Π(s i ) given by MCTS for improving the policy estimate P(s i ) of the DNN; thus, the training process is to make P and closer to Π and In one iteration, the current DNN ψ is used θ G guided MCTSs are performed for G rounds to generate experiences; then a new DNN ψ is trained using the experiences accumulated in the last z x G rounds θ′ ; In the next round of iteration, using ψ θ′ The guided MCTS generates G sequences, generating further experience. Then, these experiences will be used to train another new DNN, and so on. After multiple iterations, the DNN that maximizes the reward function R(X) is obtained by learning the latest experience accumulated in the search process, so as to obtain the sequence code with the best self-blurring and mutual-blurring performance. Step 2-4: define the evaluation function; M(X) = P X X is a sequence family with M sequences, K pulses, and L sequence length; The reward function is designed as a linear mapping from the value range of the evaluation function to the interval [-1, 1], and the reward function is specifically defined as: where M u is max(APSL, CPSL) found from the family of randomly generated sequences.

4. The method of claim 3, wherein, The implementation principle of the MCTS module is as follows: At each intermediate state s i , the Monte Carlo tree search MCTS determines a policy Π(s i ) by multi-step lookahead simulation; for each new node encountered in the simulation, the MCST performs a random simulation from that node until a leaf node is reached to obtain a reward; then, the rewards are randomly sampled over all simulations and used to evaluate the nodes; The node is analyzed in combination with a deep Q network (DQN) method and a deep neural network ψ0, and a parallel upper confidence bound tree search algorithm (P-UCT) is used to speed up the search process; a search tree is established based on a root node v0; each node in the tree has a branch corresponding to each possible movement under each state ; each branch is denoted as (v i ,a j ), wherein four statistical information are stored: a visit number N(v i ,a j ), an average reward Q(v i ,a j ), an edge probability P(v i ,a j ) and an unobserved sample number O(v i ,a j ); MCTS utilizes a deep neural network ψ0to evaluate each node; when v i is input into ψ0, it generates a policy estimate P and a reward estimate R', i.e., (P, R') = ψ0(v i ). MCTS performs forward simulation, starting from the root node v0, first creating an "access tree" to keep track of all nodes visited during the MCTS process; at initialization, there is only one root node in the access tree, and MCTS performs q simulations on this access tree; Introduce P-UCT for parallel search; specifically, parallelize q simulations through multi-threading; set a main process that undertakes the tasks of selection, expansion, and back propagation, and allocate evaluation tasks to worker processes; at the same time, deploy W worker processes that use GPU to evaluate nodes, thereby improving the overall efficiency of the search algorithm; each simulation follows the following steps: (1) Selection: All simulations start from the root node v0 until the first unvisited node is encountered. During the simulation, the edge with the highest upper confidence interval is always selected; specifically, the simulation proceeds to each node v i Selecting an edge j * is visited, the formula for the calculation is: where c p is a constant used to control the trade-off between exploration and exploitation; (2) Expansion and evaluation: When the simulation encounters an unvisited node v L the master sends an evaluation task to the worker and performs an incomplete update: O(v i ,a j ) = O(v i ,a j ) + 1 The evaluation task is to evaluate v L and get (P, R') = ψ0(v L ), where the policy distribution is If all the worker processes are occupied, the master process will wait for the evaluation task to complete; otherwise, if there are available worker processes, the master process will start the next simulation. Once the evaluation task is completed, this new node will be added to the access tree, and the edge parameters of v L are initialized to N(v L ,a j ) = 0, Q(v L ,a j ) = 0, and P(v L ,a j ) = P L (j), where (3) Backpropagation: After adding node v L to the access tree, the simulation updates all nodes on the trajectories that encounter v L . Each edge (v i , a j ) on the trajectory is updated separately: O(v i ,a j ) = O(v i ,a j )-1 N(v i ,a j ) = N(v i ,a j ) + 1 After completing q simulations, MCTS gives the move selection probability of node v0 through the following formula: The parameter τ is a temperature parameter; in a training episode, τ is set to 1 for the first third of the time steps and set to 10 for the remaining time steps -4 This makes policy selection more random in the first third of the time steps, encouraging exploration; and makes policy selection more deterministic in the remaining time steps, favoring exploitation. In the training iteration process, Dirichlet noise is used when generating a sequence set in each training segment to provide experience for the deep neural network, that is: where, is a positive real parameter; in MCTS, Dirichlet noise is added to the prior probabilities of the root nodes; specifically, for each action a j , its prior probability P(a j ) is updated as: P(a j ) = (1 - ∈) · P(a j ) + ∈ · η j where ∈ is the weight of the noise, controlling the degree of influence of the noise; η j is a random value sampled from a Dirichlet distribution, satisfying In this way, the Dirichlet noise can effectively balance exploration and exploitation during training.

5. The method of claim 1, wherein, The implementation principle of the deep neural network DNN is as follows: The deep neural network DNN is a deep convolutional network ConvNet composed of six convolutional layers, combined with batch normalization and rectified linear unit ReLU activation function; (1) Input: The convolutional network takes as input a stack of images of size K' x N' x D; for a state s i i.e. a partially filled K' x N' x D set of sequence patterns, is first converted into an image of size K' x N' x 1 and then into a stack of images of size K' x N' x D through feature extraction; D denotes the number of discrete phase values, the range of values of the sequence elements is (2) Feature extraction: feature extraction is the process of converting an image of size K'xN'x1 into an image stack of size K'xN'xD; this stack consists of D feature planes, each corresponding to a symbol value indicating the presence or absence of that symbol value; (3) Output: For each state S i A deep neural network outputs a policy estimate, which is a probability distribution. As state s i of The prior probability of each edge is output, along with a scalar estimate R′∈[-1,1], which serves as the state S. i Expected rewards; (4) Training: After every G games, the accumulated experience in the last z x G games is used to update the deep neural network using the stochastic gradient descent method; the size of the mini-batch is set to batchSize, and the mini-batch is randomly selected from the z x G experiences . The convolutional network is trained using the mini-batch; for each mini-batch, the loss function is defined as the sum of the mean square error and the cross-entropy loss: .​ In the above formula, the first term measures the difference between the reward value R' predicted by the deep neural network and the actual reward value R, ensuring that the deep neural network can accurately predict the expected reward of each state; the second term measures the difference between the strategy distribution P predicted by the deep neural network and the strategy distribution Π generated by MCTS, by minimizing the second term, the deep neural network can better fit the strategy distribution of MCTS, thereby improving the accuracy of action selection; the third term is an L2 regularization term, which is used to constrain the model parameters θ to prevent model overfitting and improve its generalization ability; The learning rate is fixed at 10 throughout the training process -4 to steadily optimize the model parameters.

6. An electronic device, comprising: Comprising: a processor and a memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the method as claimed in any one of claims 1 to 5.

8. A chip, characterized by Comprising: a processor for calling and running a computer program from a memory, so that the device installed with the chip executes the method as claimed in any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program product comprises a computer storage medium, the computer storage medium stores a computer program, the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method as claimed in any one of claims 1 to 5 is realized.

Images