Synthetic broadband waveform optimization method based on reinforcement learning

Abstract

The invention relates to the technical field of radar waveform optimization, in particular to a synthetic broadband waveform optimization method based on reinforcement learning. According to the method, a deep reinforcement learning framework is adopted, and end-to-end optimization is carried out on the center frequency and amplitude parameters of multiple sub-pulses. The method comprises the following steps: constructing a synthetic broadband waveform signal model, and parameterizing sub-waveforms to determine a search space of an optimization problem; converting a waveform optimization problem into a reinforcement learning form, and defining a state space, an action space, a reward function and a state transition mechanism; based on a deterministic soft action value iteration strategy algorithm, point spread function features are extracted in combination with a Transform encoder, and sub-pulse parameters are dynamically adjusted. Through the method, the main lobe resolution can be effectively suppressed and improved, the target detection performance of a radar system is improved, and the method has good global optimization capability and environmental adaptability and is suitable for cognitive radar and adaptive waveform design scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar waveform optimization, and particularly relates to a synthetic wideband waveform (SWW) optimization method based on reinforcement learning. BACKGROUND

[0002] In modern radar systems, in order to achieve high range resolution, it is usually necessary to use a transmitted signal with a large bandwidth. However, due to the performance of the hardware platform, especially the maximum sampling rate of the analog-to-digital converter (ADC), the actual available signal bandwidth is often limited, making it difficult to meet the needs of high-resolution detection, which is particularly common in ground penetrating radar systems.

[0003] To overcome the above limitations, synthetic wideband waveform technology has gradually developed. This technology combines multiple narrow-band sub-pulses (sub-pulse or sub-chirp) with different center frequencies in the time or frequency domain, and performs synthesis processing at the receiving end, thereby obtaining an equivalent large bandwidth signal without increasing the ADC sampling rate, and thus improving the range resolution of the radar system. However, during the synthesis process, significant grating lobes can appear in the point spread function (PSF), leading to increased false alarm probability, target masking, and decreased signal-to-noise ratio, which seriously affects the target detection and recognition ability of the radar.

[0004] To address the above problems, existing technologies mainly rely on manual experience to adjust the parameters (such as center frequency, amplitude, phase, etc.) of each sub-pulse to suppress the grating lobe effect. However, such methods have limited design space and lack systematic and global optimization capabilities. From a mathematical modeling perspective, this problem is essentially a non-convex optimization problem, and traditional convex optimization algorithms are difficult to apply directly, limiting the optimization effect. To solve this type of non-convex optimization problem, some research has introduced swarm intelligence optimization algorithms, such as genetic algorithm (GA) and particle swarm optimization (PSO). Although these methods have certain global search capabilities, they still have problems such as slow convergence speed, easy to fall into local optimum, high computational complexity, etc. in practical applications, especially in the case of high parameter dimension or dynamic environment changes, the optimization efficiency is low.

[0005] In recent years, reinforcement learning (RL) as a sequence decision method based on the interaction between agent and environment has shown significant advantages in dealing with non-convex optimization problems. Reinforcement learning can dynamically adjust the sub-pulse parameters through continuous interaction with the radar environment, gradually optimize the waveform design strategy, and has good convergence and global optimization ability. At the same time, this method has strong expansibility and can flexibly adapt to different design requirements such as changes in the number of sub-pulses, adjustment of frequency interval, and changes in amplitude constraints, and is suitable for cognitive radar, adaptive waveform design and other frontier application scenarios.

[0006] In summary, the prior art still has many deficiencies in the design of synthetic wideband waveforms, and a new optimization method is needed to effectively suppress the grating lobe effect while ensuring waveform performance. Therefore, the present application proposes a synthetic wideband waveform optimization method based on reinforcement learning to solve the above problems and improve the detection performance of the radar system. SUMMARY

[0007] To solve the above problems, the present application proposes a synthetic wideband waveform optimization method based on reinforcement learning to realize the optimization of multi-sub-pulse waveform parameters, thereby effectively suppressing the grating lobe effect, improving the main lobe resolution, and improving the target detection performance of the radar system.

[0008] To achieve the above purpose, the present application provides the following technical solution: a synthetic wideband waveform optimization method based on reinforcement learning, characterized by the following steps:

[0009] Step S1: Construct a synthetic wideband waveform signal model.

[0010] The synthetic wideband waveform is composed of sub-waveforms, each of which is a linear frequency modulation signal, with the center frequency denoted as , the amplitude as , and the bandwidth and time length as fixed values; the sub-waveform set is defined as follows:

[0011]

[0012] where the sub-waveform center frequency satisfies the monotonic increasing relationship, i.e.

[0013] ;

[0014] Each sub-waveform is transmitted in a time-sharing manner;

[0015] Step S2: Model the synthetic wideband waveform optimization problem as a reinforcement learning problem;

[0016] S2.1 State space definition

[0017] The state space The PSF sequence is divided into sub-vectors, which constitute the state of the current time :

[0018]

[0019] where is the length of the complete PSF sequence, is the preset number of divisions, is the PSF sub-vector of the th segment, and R represents the set of real numbers;

[0020] S2.2 Action space definition

[0021] The action space A is defined as a continuous space , where each action is composed of the amplitude and center frequency parameters of sub-waveforms, and is specifically represented as:

[0022]

[0023] where , , represent the amplitude and center frequency of the sub-waveform at time , respectively; , are the minimum and maximum transmit amplitude limits of the radar system, respectively, are the minimum and maximum center frequency ranges supported by the radar transmitter, respectively;

[0024] S2.3 Definition of state transition function

[0025] The state transition function is specifically:

[0026]

[0027] where represents the environment simulation mapping function;

[0028] S2.4 Incremental reward function comparing the performance of the current action with the historical optimal performance ,

[0029] S3: Synthesis of wideband waveform optimization based on deterministic soft action value iteration policy

[0030] S3.1 Action network based on transformer encoder, denoted as policy function ;

[0031] The input state information is preprocessed and reorganized into a matrix form

[0032] where L represents the total length of the state vector, and M represents the set embedding sequence length; the matrix is subjected to an embedding operation to obtain an initial embedding representation where is the embedding dimension;

[0033] In the embedding representation , a position encoding is added, and the embedding representation after adding the position encoding is input into a Transformer encoder composed of layers to perform multi-layer feature extraction and context modeling, and output an enhanced embedding representation: ;

[0034] The embedding representation is added with a position encoding, and the embedding representation after adding the position encoding is input into a Transformer encoder composed of layers to output an enhanced embedding representation: ;

[0035] The enhanced embedding representation is subjected to a pooling operation to obtain a global feature vector ,

[0036] The global feature vector is input into a multi-layer fully connected neural network, and after being processed by a nonlinear activation function, a normalized action prototype vector is output , and its expression is:

[0037]

[0038] wherein, the dimension of is 2N, corresponding to the amplitude and frequency parameters of N action units respectively, and the value range of each element is , is a learnable scalar, W1 and W2 are weight matrices in the neural network, and σ is an activation function;

[0039] Linear mapping of the action prototype is denoted as , and a composite action instruction set composed of multiple action units is output:

[0040]

[0041] S3.2 State-Action Value Network Based on Multi-Layer Perceptron

[0042] The state-action value evaluation module adopts a deep neural network structure based on a multi-layer perception as a value function estimator, denoted as , for jointly evaluating the combination of the current state and action , and outputting the corresponding state-action value estimate;

[0043] S3.3 Environment interaction and reward mechanism: the deep reinforcement learning system completes policy training and evaluation through interaction with a simulated or actual radar environment;

[0044] S3.4 Parameter update method based on deterministic soft action value iteration policy.

[0045] According to the synthetic wideband waveform optimization method based on reinforcement learning, in step S1, the sub-waveform bandwidth is W, the maximum instantaneous bandwidth is , is the center frequency; the center frequency of the synthetic wideband waveform and the equivalent total bandwidth are:

[0046] .

[0047] According to the synthetic wideband waveform optimization method based on reinforcement learning, in step S2.3: at time , the agent outputs an action based on the current state , the environment configures a corresponding synthetic wideband waveform according to the action, and calculates its point spread function sequence; the PSF sequence is divided into sub-vectors to form the environment state at the next time.

[0048] According to the synthetic wideband waveform optimization method based on reinforcement learning, in step S2.4, the reward function is:

[0049]

[0050] wherein: : represents the -3dB main lobe width of the point spread function corresponding to the current action ;

[0051] : represents the peak sidelobe level of the point spread function corresponding to the current action ;

[0052] : is a preset weight coefficient; : Represents the maximum value of the weighted performance metrics obtained by all actions up to time t in the current round.

[0053] According to the reinforcement learning-based synthetic broadband waveform optimization method described above, the method further includes step S2.5, discount factor. Fixed discount factor .

[0054] According to the reinforcement learning-based synthetic broadband waveform optimization method described above, the characteristic is that: in step S3.1, the average pooling method is used to compress the sequence dimension into a global feature vector. .

[0055] According to the reinforcement learning-based synthetic broadband waveform optimization method described above, the method of linear mapping in step S3.1 is as follows: for the first... Individual action units:

[0056]

[0057] in, Indicates that at time t, the first The amplitude parameters of each action unit, This indicates the corresponding frequency parameter; and These are the allowable ranges for amplitude and frequency, respectively.

[0058] According to the reinforcement learning-based synthetic broadband waveform optimization method described above, the specific process of S3.2 is as follows:

[0059] First, the global state feature vector Action instructions at the current moment The features are concatenated to form a joint input feature vector:

[0060]

[0061] The joint feature vector is input into a multi-layer fully connected neural network, and processed through multiple hidden layers of nonlinear transformations and activation functions to finally output the corresponding state-action value estimate.

[0062]

[0063] The MLP contains several fully connected layers and activation functions.

[0064] According to the reinforcement learning-based synthetic broadband waveform optimization method described above, the specific interaction process of the environment interaction and reward mechanism in S3.3 is as follows:

[0065] Action instructions output by the action network It is input into the radar waveform simulation environment to control the amplitude and center frequency parameters of each sub-pulse in the current transmitted signal;

[0066] A corresponding broadband radar waveform is synthesized in the environment, and a point spread function is calculated based on the waveform. The instantaneous reward at time t is obtained according to the point spread function. ;

[0067] The state during this interaction ,action Rewards received And the state in the next moment. Form an experience transfer tuple And store it in the experience replay buffer;

[0068] The current round ends when the following termination conditions are met:

[0069]

[0070] in, This indicates the maximum acceptable threshold for the main lobe width. This indicates the highest acceptable threshold for the grid lobe level.

[0071] According to the above-described reinforcement learning-based synthetic broadband waveform optimization method, the feature is that: in S3.4, the action network and the state-action value network are updated using a deep reinforcement learning algorithm based on a deterministic soft action value iteration strategy, thereby achieving efficient learning and stable convergence of the radar waveform optimization strategy.

[0072] Compared with existing technologies, this invention has the following advantages: 1. It realizes a waveform design method that eliminates the need for manual setting of sub-waveform parameters, and can automatically generate synthetic broadband waveforms that simultaneously possess main lobe resolution enhancement and grating lobe suppression characteristics. 2. It introduces a feature extraction mechanism based on the Transformer architecture, realizing feature sharing between the Actor and Critic networks, thereby significantly reducing computational overhead and improving training efficiency. 3. It constructs a reward function design strategy based on performance increments, which significantly improves optimization convergence speed and waveform design quality by modeling the difference between the current action performance and the historical best performance. Attached Figure Description

[0073] Figure 1 The schematic diagram of the synthesized broadband waveform in the frequency domain provided as an example of the present invention illustrates the structure of a synthesized broadband signal composed of multiple sub-pulses with different center frequencies.

[0074] Figure 2This is a schematic diagram illustrating the interaction process between the agent and the environment in an example of the present invention, used to explain the dynamic relationship between state, action, and reward under the reinforcement learning framework.

[0075] Figure 3 The schematic diagram of the intelligent network architecture provided for the example of the present invention includes an action network (Actor) based on a Transformer encoder and a state-action value network (Critic) based on a multilayer perceptron.

[0076] Figure 4 The complete pseudocode of the synthetic broadband waveform optimization algorithm based on the deterministic soft action value iteration strategy (DSAC) provided for the example of this invention demonstrates the main steps of the training and optimization process.

[0077] Figure 5 The original point spread function (PSF) shows obvious grating lobes.

[0078] Figure 6 The optimized point spread function indicates that the method of the present invention can effectively suppress grating lobes and improve the main lobe resolution.

[0079] Figure 7 This is a flowchart of the present invention. Detailed Implementation

[0080] This invention provides a reinforcement learning-based method for optimizing Synthetic Wideband Waveform (SWW). It employs a deep reinforcement learning framework that combines a Transformer encoder with a Deterministic Soft Actor-Critic (DSAC) strategy to achieve adaptive optimization of multi-sub-pulse waveform parameters. This effectively reduces sidelobe peaks in the point spread function, improves main lobe resolution, and enhances the waveform performance and target detection capability of the radar system.

[0081] To achieve the above objectives, the present invention will be described in detail with reference to the accompanying drawings. As shown in Figures 1-7, the specific embodiments proposed by the present invention include the following steps:

[0082] Step S1: Construct a synthetic broadband waveform signal model.

[0083] Step S2: Model the synthetic broadband waveform optimization problem as a reinforcement learning problem.

[0084] Step S3: Synthetic broadband waveform optimization based on the deterministic soft actor-critic (DSAC) value iteration strategy.

[0085] S1: Construct a synthetic broadband waveform signal model.

[0086] As shown in Figure 1, the designed synthetic broadband waveform is composed of... It consists of 10 sub-waveforms, each of which is a linear frequency modulated signal, and its center frequency is denoted as . , amplitude The bandwidth and duration are fixed values, W and W respectively. The sub-waveform set is defined as follows:

[0087]

[0088] The center frequency of the sub-waveform satisfies a monotonically increasing relationship, that is:

[0089]

[0090] Each sub-waveform can be transmitted sequentially in a time-division manner to form a set of sub-pulses covering a wide frequency band.

[0091] In this embodiment of the invention, the bandwidth of all sub-waveforms is uniformly set to W to maximize the utilization of the available instantaneous bandwidth of the radar transmitter. Specifically, it is assumed that the maximum instantaneous bandwidth of the radar transmitter is W. Furthermore, its center frequency can be flexibly adjusted within a set range. Therefore, the [specific details / concepts] described in this article... That is, the corresponding radar transmitter at the time of transmission The center frequency setting value for each sub-waveform.

[0092] Center frequency of synthesized broadband waveform With equivalent total bandwidth They are defined as follows:

[0093]

[0094] Therefore, a complete synthesized broadband waveform can be represented by a set of sub-waveform parameters, namely: This parameter set serves as the decision variable in the subsequent optimization process, defining the action space in the reinforcement learning environment.

[0095] S2: Model the synthetic broadband waveform optimization problem as a reinforcement learning problem.

[0096] This invention transforms the problem of synthesizing wideband waveforms into a reinforcement learning problem, and constructs an interaction model between the agent and the environment (as shown in the appendix). Figure 2 As shown in the figure, the state space, action space, state transition function, reward function, and discount factor are defined in sequence as follows:

[0097] S2.1 State Space Definition

[0098] Reinforce the state space in the learning environment Defined as a sequence of point spread function (PSF) signals generated by the parameters of the previous set of sub-waveforms. To reduce the state dimension and enhance feature expressiveness, the PSF sequence is divided into... The sub-vectors constitute the current time step. Status:

[0099]

[0100] in, The length of the complete PSF sequence. To preset the number of segments, For the first A segment of PSF subvectors, where R represents the set of real numbers.

[0101] S2.2 Action Space Definition

[0102] Define the action space A as a continuous space. , where each action Depend on The sub-waveform is composed of the amplitude and center frequency parameters, specifically expressed as follows:

[0103]

[0104] in, , , , respectively representing the sub-waveform in time The amplitude and center frequency. , These are the minimum and maximum transmission amplitude limits for the radar system, respectively. These are the minimum and maximum center frequency ranges supported by the radar transmitter, respectively. The above ranges are all preset based on the actual performance of the radar hardware platform.

[0105] S2.3 Definition of State Transition Function

[0106] In this invention, the state transition function Determined by the environmental simulation process. Specifically:

[0107] At any moment The agent is based on the current state Output Action The environment generates the corresponding synthesized broadband waveform based on this action configuration and calculates its point spread function (PSF) sequence. Subsequently, this PSF sequence is divided into... These subvectors constitute the environment state at the next time step. .

[0108] The above process can be formally represented as:

[0109]

[0110] in, This represents the environment simulation mapping function.

[0111] Through this state transition mechanism, reinforcement learning agents can continuously optimize sub-waveform parameter configurations based on the impact of historical actions on the environmental state, thereby gradually improving the overall waveform performance.

[0112] S2.4 Reward Function Design

[0113] The deep reinforcement learning system employs an incremental reward function based on a comparison between the current action performance and the historical best performance. This is used to guide the action network (Actor) in generating better radar waveform parameter configurations. Specifically, the reward function is defined as follows:

[0114]

[0115] in:

[0116] : Indicates the current action The -3dB main lobe width of the corresponding point spread function (PSF) is used to measure the range resolution of the radar system;

[0117] : Indicates the current action The peak sidelobe level (PSL) of the corresponding point spread function (PSF) is used to evaluate the grating lobe level of the synthesized broadband wave.

[0118] : This is a preset weighting coefficient used to adjust the weight of different performance indicators on the reward value;

[0119] : This represents the maximum value of the weighted performance index obtained by all actions up to time t in the current round, which is the historical best performance score of the current round.

[0120] S2.5 Discount Factor

[0121] In this invention, a discount factor is set. This setting is based on the designed incremental reward mechanism: since the difference between the reward function and the historical best performance is introduced as the basis for the current reward, the agent always aims at the relative improvement during the policy update process, thus avoiding the problem of reward value divergence or numerical overflow caused by long-term reward accumulation in traditional reinforcement learning.

[0122] Using full discount factor The advantages are:

[0123] a) It can increase the agent's focus on long-term performance optimization;

[0124] b) Enhance the policy network's ability to approximate complex nonlinear targets;

[0125] c) Accelerate algorithm convergence speed and improve training stability.

[0126] Therefore, even without decaying future rewards, the system can still maintain good numerical stability and policy learning efficiency.

[0127] S3: Synthetic Broadband Waveform Optimization Based on Deterministic Soft Actor-Critic (DSAC) Value Iteration Strategy

[0128] S3.1 An action network (Actor) based on a transformer encoder, denoted as the policy function.

[0129] In a preferred embodiment of the present invention, as shown in the appendix Figure 3 As shown, the action generation module uses a deep neural network structure based on a Transformer encoder as the policy function. Used to determine the current state Output continuous control actions.

[0130] Specifically, firstly, the input status information Preprocessing is performed to reorganize it into a matrix form.

[0131] Where L represents the total length of the state vector, and M represents the set length of the embedding sequence. Subsequently, an embedding operation is performed on this matrix to obtain the initial embedding representation. ,in For the embedded dimension.

[0132] To further incorporate positional information of the sequence, in the embedded representation Add position encoding to obtain an embedded representation with position information.

[0133] Next, the position-encoded embedding representation will be input into the system. In the Transformer encoder, which consists of multiple layers, multi-layer feature extraction and context modeling are performed, and the output is an enhanced embedding representation: .

[0134] The action generation module uses a deep neural network structure based on a Transformer encoder as the policy function. Used to determine the current state Output continuous control actions.

[0135] To further incorporate positional information of the sequence, in the embedded representation Position encoding is added to the embedding to obtain an embedding representation with positional information. Then, the position-encoded embedding representation is input into the system... In the Transformer encoder, which consists of multiple layers, multi-layer feature extraction and context modeling are performed, and the output is an enhanced embedding representation: Furthermore, regarding the enhanced embedding representation... Perform pooling operations, preferably using average pooling, to compress the sequence dimension into a single global feature vector. It is used to represent the overall semantic information of the current state.

[0136] Next, the aforementioned global feature vector is input into a multi-layer fully connected neural network. After processing by a non-linear activation function, it outputs a normalized action prototype vector. Its expression is:

[0137]

[0138] in, The dimension is 2N, corresponding to the amplitude and frequency parameters of N action units, and the value range of each element is... , σ is a learnable scalar, W1 and W2 are the weight matrices in the neural network, and σ is the activation function.

[0139] To ensure that the output action parameters meet the amplitude and center frequency ranges of each sub-pulse in the actual radar system, each term in the action prototype vector is linearly mapped to fall within a preset effective range. For the first... One action unit ( ):

[0140]

[0141] in, Indicates that at time t, the first The amplitude parameters of each action unit, This indicates the corresponding frequency parameter; and These are the allowable ranges for amplitude and frequency, respectively.

[0142] The above linear mapping is denoted as Finally, the output is a set of compound action instructions consisting of multiple action units:

[0143]

[0144] S3.2 State-Action Value Network (Critic) Based on Multilayer Perceptron

[0145] In a preferred embodiment of the present invention, the state-action value assessment module employs a deep neural network structure based on a multilayer perceptron (MLP) as the value function estimator, denoted as... Used to describe the current state With action The combination of these factors is used for joint evaluation, and the corresponding state-action value estimate is output, as shown in the appendix. Figure 3 As shown.

[0146] Specifically, firstly, the global state feature vector output by the Actor network is... Action instructions at the current moment The features are concatenated to form a joint input feature vector:

[0147]

[0148] Subsequently, the joint feature vector is input into a multi-layer fully connected neural network, which undergoes nonlinear transformations and activation functions through multiple hidden layers, ultimately outputting the corresponding state-action value estimate.

[0149]

[0150] The MLP contains several fully connected layers and activation functions, such as ReLU, ELU, or LeakyReLU, to introduce nonlinear modeling capabilities. The output layer typically does not contain activation functions to ensure the numerical stability and interpretability of the value estimation results.

[0151] The policy function is evaluated through this Critic network. The generated actions are evaluated for value, which can effectively guide the direction of strategy updates, thereby achieving efficient optimization and adaptive adjustment of transmitted waveform parameters (such as subpulse amplitude, center frequency, etc.) in the radar system.

[0152] S3.3 Environmental Interaction and Reward Mechanism

[0153] In a preferred embodiment of the present invention, the deep reinforcement learning system completes policy training and evaluation through interaction with a simulated or actual radar environment. The specific interaction process is as follows:

[0154] First, the action instructions output by the action network (Actor). It is input into the radar waveform simulation environment to control the amplitude and center frequency parameters of each sub-pulse in the current transmitted signal.

[0155] Subsequently, a corresponding broadband radar waveform is synthesized in the environment, and a point spread function is calculated based on this waveform. The instantaneous reward at time t is then obtained according to the point spread function. .

[0156] Next, the status during this interaction process will be... ,action Rewards received And the state in the next moment. Form an experience transfer tuple And store it in the experience playback buffer.

[0157] The current episode ends when the following termination conditions are met:

[0158]

[0159] in, This indicates the highest acceptable threshold for the main lobe width (-3dB width). This indicates the highest acceptable threshold for the grating lobe level.

[0160] S3.4 Parameter Update Method Based on Deterministic Soft Actor-Critic (DSAC) Value Iteration Strategy

[0161] In a preferred embodiment of the present invention, the action network (Actor) and the state-action value network (Critic) are updated using a deep reinforcement learning algorithm based on the deterministic soft actor-critic (DSAC) strategy to achieve efficient learning and stable convergence of the radar waveform optimization strategy.

[0162] DSAC is an improved form of SAC, removing the stochastic policy modeling found in SAC while retaining the dual-Q network structure and target network mechanism. This improves policy update efficiency and stability while maintaining exploration capabilities. (Appendix) Figure 4The complete algorithm flow of DSAC is given.

[0163] I. Symbol Definition

[0164] For ease of description, the following variables and functions are defined:

[0165] : The current state information (a sequence of point spread function (PSF) vectors, divided into M sub-vectors);

[0166] : Action prototype vectors output by the Actor network;

[0167] :Will The actual actions after mapping to the physically executable region;

[0168] The comprehensive reward value calculated based on environmental feedback;

[0169] : Execute action Then, the environment returns the state of the next moment;

[0170] Discount factor, used to measure the importance of future rewards;

[0171] : These represent the parameters of the Critic and Actor networks, respectively;

[0172] The target Critic network, whose parameters are obtained from the main Critic network through a soft update mechanism;

[0173] Experience replay buffer, used to store historical interaction data.

[0174] II. Mini-Batch Sample Sampling

[0175] During each training iteration, a batch of data is randomly sampled from the experience replay buffer D:

[0176]

[0177] Where $B$ represents the batch size, which is used for subsequent network parameter updates.

[0178] III. Calculation of the Target Q Value

[0179] For each sample Perform the following operations in sequence:

[0180] 1. Calculate the action prototype corresponding to the next state using the current Actor network:

[0181]

[0182] 2. The action prototype Mapping to the physical action space, we get:

[0183]

[0184] 3. Utilizing the target Critic network Calculate the next state-action value estimate:

[0185]

[0186] 4. Take the minimum of the two Q values ​​as the final estimate of the target value to mitigate overestimation bias:

[0187]

[0188] 5. Calculate the target Q value:

[0189]

[0190] IV. Critic Network Update

[0191] Construct and update loss functions for the two Critic networks respectively:

[0192] For the A Critic network, with the loss function defined as:

[0193]

[0194] Subsequently, the Critic network parameters were analyzed. Perform gradient descent update:

[0195]

[0196] in For the first The learning rate of each Critic network.

[0197] V. Actor Network Updates

[0198] Furthermore, the Actor network is updated using the value assessment information provided by Critic to maximize expected returns:

[0199] 1. For each sample, calculate the physical action corresponding to the current state:

[0200]

[0201] 2. Calculate the Q value corresponding to Critic:

[0202]

[0203] 3. Construct the loss function for the Actor:

[0204]

[0205] 3. Perform gradient descent updates on the Actor network parameters:

[0206]

[0207] in is the learning rate of the Actor network.

[0208] VI. Target Critic Network Soft Update

[0209] To improve training stability, the target Critic network parameters are updated using a soft update method:

[0210]

[0211] in The soft update coefficient is 0.006 in a preferred embodiment of the present invention, and is used to control the smoothness of the target network parameter update.

[0212] VII. Determining Training Termination Conditions

[0213] Repeat Steps 2 through 6 until any of the following training termination conditions are met:

[0214] In the current round, the main lobe width is below a set threshold. And the grating lobes are below the set threshold. ;

[0215] Reach the preset maximum number of training rounds.

[0216] Technical effect description:

[0217] The parameter update mechanism based on the DSAC framework described above can effectively improve policy exploration capabilities and training stability while ensuring the policy update direction. This allows for the rapid generation of synthetic broadband radar waveforms with excellent range resolution and low grating lobe characteristics, such as... Figure 5 and Figure 6 The image shows a comparison of the waveform optimization results.

[0218] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for optimizing synthetic broadband waveforms based on reinforcement learning, characterized in that: Includes the following steps: Step S1: Construct a synthetic broadband waveform signal model: Synthesized broadband waveforms are made from It consists of 10 sub-waveforms, each of which is a linear frequency modulated signal, and its center frequency is denoted as . , amplitude The bandwidth and duration are fixed values; the sub-waveform set is defined as follows: The center frequency of the sub-waveform satisfies a monotonically increasing relationship, that is: ； Each sub-waveform is transmitted sequentially in a time-division manner; Step S2: Model the synthetic broadband waveform optimization problem as a reinforcement learning problem; S2.1 State Space Definition state space Defined as the point spread function sequence generated by the previous set of sub-waveform parameters, the PSF sequence is divided into... The subvectors constitute the current time step. Status: in, The length of the complete PSF sequence. To preset the number of segments, For the first A segment of PSF subvectors, where R represents the set of real numbers; S2.2 Action Space Definition Define the action space A as a continuous space. , where each action Depend on The sub-waveform is composed of the amplitude and center frequency parameters, specifically expressed as follows: in, , , , respectively representing the sub-waveform in time The amplitude and center frequency; , These are the minimum and maximum transmission amplitude limits for the radar system, respectively. These are the minimum and maximum center frequency ranges supported by the radar transmitter, respectively. S2.3 Definition of State Transition Function State transition function Specifically: in, Represents the environment simulation mapping function; S2.4 Incremental reward function based on comparison of current action performance with historical best performance. , S3: Synthetic Broadband Waveform Optimization Based on Deterministic Soft-Action Value Iteration Strategy S3.1 Action network based on transformer encoder, denoted as policy function. ; Input status information Preprocessing is performed, and the data is reorganized into a matrix form. Where L represents the total length of the state vector and M represents the set embedding sequence length; the initial embedding representation is obtained by performing an embedding operation on this matrix. ,in For the embedded dimension; In the embedded representation Add positional encoding to the above, and input the positionally encoded embedding representation into the input... In the Transformer encoder, which consists of multiple layers, multi-layer feature extraction and context modeling are performed, and the output is an enhanced embedding representation: ; The embedding representation Add positional encoding to the above, and input the positionally encoded embedding representation into the input... In a Transformer encoder composed of layers, the output is an enhanced embedding representation: ; For the enhanced embedding representation Perform pooling operations to obtain global feature vectors , The global feature vector is input into a multi-layer fully connected neural network, processed by a non-linear activation function, and outputs a normalized action prototype vector. Its expression is: in, The dimension is 2N, corresponding to the amplitude and frequency parameters of N action units, and the value range of each element is... , σ is a learnable scalar, W1 and W2 are the weight matrices in the neural network, and σ is the activation function; Linear mapping of the action prototype is denoted as It outputs a set of compound action instructions consisting of multiple action units: S3.2 State-Action Value Network Based on Multilayer Perceptron The state-action value assessment module uses a deep neural network structure based on a multilayer perceptron as the value function estimator, denoted as . Used to describe the current state With action The combination of these factors is used for joint evaluation, and the corresponding state-action value estimate is output. S3.3 Environment Interaction and Reward Mechanism: The deep reinforcement learning system completes policy training and evaluation through interaction with simulated or actual radar environments; S3.4 Parameter update method based on deterministic soft action value iteration strategy.

2. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: In step S1, the sub-waveform bandwidth is W, and the maximum instantaneous bandwidth is , The center frequency; the center frequency of the synthesized broadband waveform. With equivalent total bandwidth for: 。 3. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: In step S2.3: at time... The agent is based on the current state Output Action The environment generates the corresponding synthesized broadband waveform based on this action configuration and calculates its point spread function (PSF) sequence; the PSF sequence is then divided into... These subvectors constitute the environment state at the next time step. .

4. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: The reward function in step S2.4 is: in: : Indicates the current action The corresponding point spread function has a main lobe width of -3dB; : Indicates the current action The peak sidelobe level of the corresponding point spread function; : These are the preset weighting coefficients; : Represents the maximum value of the weighted performance metrics obtained by all actions up to time t in the current round.

5. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: It also includes step S2.5, the discount factor. Fixed discount factor .

6. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: In step S3.1, the average pooling method is used to compress the sequence dimension into a global feature vector. .

7. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: The method for linear mapping in step S3.1 is as follows: For the first... Individual action units: in, Indicates that at time t, the first The amplitude parameters of each action unit, This indicates the corresponding frequency parameter; and These are the allowable ranges for amplitude and frequency, respectively.

8. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: The specific process in S3.2 is as follows: First, the global state feature vector Action instructions at the current moment The features are concatenated to form a joint input feature vector: The joint feature vector is input into a multi-layer fully connected neural network, and processed through multiple hidden layers of nonlinear transformations and activation functions to finally output the corresponding state-action value estimate. The MLP contains several fully connected layers and activation functions.

9. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: The specific interaction flow of the environment interaction and reward mechanism in S3.3 is as follows: Action instructions output by the action network It is input into the radar waveform simulation environment to control the amplitude and center frequency parameters of each sub-pulse in the current transmitted signal; A corresponding broadband radar waveform is synthesized in the environment, and a point spread function is calculated based on the waveform. The instantaneous reward at time t is obtained according to the point spread function. ; The state during this interaction ,action Rewards received And the state in the next moment. Form an experience transfer tuple And store it in the experience replay buffer; The current round ends when the following termination conditions are met: in, This indicates the maximum acceptable threshold for the main lobe width. This indicates the highest acceptable threshold for the grid lobe level.

10. The method for optimizing synthetic broadband waveforms based on reinforcement learning according to claim 1, characterized in that: In S3.4, the action network and state-action value network use a deep reinforcement learning algorithm based on a deterministic soft action value iteration strategy for parameter updates, achieving efficient learning and stable convergence of the radar waveform optimization strategy.

Images