Adaptive radar signal detection method and system based on reinforcement learning

Abstract

The invention discloses an adaptive radar signal detection method and system based on reinforcement learning. The method comprises the following steps: S1, collecting pulse radar echo signals; s2, dividing a reference unit and a to-be-detected unit for the signal; s3, extracting basic features of the reference unit and the to-be-detected unit signals; s4, extracting deep features of the reference unit signals; s5, extracting deep features of the to-be-detected unit signal, and outputting a confidence coefficient; s6, inputting the joint deep features of the reference unit and the to-be-detected unit into a reinforcement learning agent as a state, and adjusting an action output detection threshold by the agent based on the current state; s7, performing target detection on the to-be-detected unit, and calculating a reward value; s8, repeating the steps S3-S7, constructing empirical data, training and updating the intelligent agent by adopting a reinforcement learning algorithm, and outputting a signal detection decision intelligent agent model; and S9, obtaining a confidence coefficient after the test set passes through the steps S3-S5, inputting the deep features into the trained intelligent agent to obtain a threshold, judging that the intelligent agent with the confidence coefficient exceeding the threshold value is a signal, and otherwise, judging that the intelligent agent is a noise.

Description

Technical Field

[0001] The present invention belongs to the technical field of radar signal processing, and in particular relates to an adaptive radar signal detection method and system based on reinforcement learning. Background Art

[0002] Radar signal detection technology is crucial for applications such as target detection, tracking, and identification. This is particularly true in environments with complex background clutter, multi-target interference, and non-Gaussian noise, which place higher demands on the adaptive capabilities of detection algorithms. Constant False Alarm Rate (CFAR), a classic method for radar target detection, is widely used in radar systems of various architectures. Its basic principle is to use the statistical background noise characteristics of reference cells surrounding the target cell to set a detection threshold based on these characteristics, thereby distinguishing the target signal from the background noise.

[0003] However, classic radar CFAR technology has the following shortcomings: reliance on fixed parameters leads to poor adaptability to complex scenarios (such as multiple targets and clutter edges); inaccurate background clutter modeling causes false alarms or missed detections; sliding window design is sensitive to size; and there is a strong reliance on manual experience and a lack of adaptability in parameter adjustment.

[0004] In recent years, with the rapid development of deep learning and reinforcement learning technologies, researchers have attempted to introduce models such as convolutional neural networks and recurrent neural networks into radar signal processing tasks to enhance the intelligence and adaptability of target detection. However, most current methods focus primarily on feature extraction or end-to-end classification and detection. They lack structural modeling of the statistical relationship between the "reference unit and the unit to be detected" in the traditional CFAR architecture and fail to fully utilize the statistical characteristics of different signal backgrounds for adaptive threshold adjustment.

[0005] Therefore, there is an urgent need for radar signal detection technology that combines deep statistical feature modeling with optimized decision-making mechanisms to enhance the adaptive adjustment capability of detection thresholds, thereby achieving intelligent and efficient radar detection in dynamic and non-uniform scenarios under complex backgrounds. Based on this, the present invention proposes an adaptive radar signal detection method and system based on reinforcement learning. Summary of the Invention

[0006] To address the shortcomings of the aforementioned existing technologies, the present invention proposes an adaptive radar signal detection method based on reinforcement learning. This method introduces a reinforcement learning mechanism to model the detection process as a decision-making problem involving the interaction between an agent and its environment. By sensing the characteristic states of reference cells and cells to be detected, the method dynamically optimizes the threshold setting strategy, thereby achieving adaptive detection in complex environments and improving the robustness and accuracy of target detection.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is as follows:

[0008] An adaptive radar signal detection method based on reinforcement learning comprises the following steps:

[0009] S1, collects pulse radar echo signals and provides data of target signals and noise signals;

[0010] S2. Divide the pulse radar echo signal into reference units and test units, and then divide the pulse radar echo signal data set into a training set and a test set;

[0011] S3, inputting the reference unit and the unit to be tested signals into the shared feature extraction network in parallel to extract basic features;

[0012] S4, input the basic features of the reference unit signal into the R network (i.e., the reference unit feature extraction network) to extract deep features;

[0013] S5. Input the basic features of the signal of the unit to be detected into the D network (i.e., the feature extraction network of the unit to be detected) to extract deep features and output the confidence level of the signal.

[0014] S6. Input the joint deep feature representation of the reference unit and the unit to be inspected into the reinforcement learning agent, and use the joint deep feature as the state. The reinforcement learning agent adjusts the action output detection threshold based on the current state to achieve adaptive setting of the threshold;

[0015] S7. Perform target detection on the unit to be inspected based on the detection threshold output by the reinforcement learning agent and the confidence level output by the D network, and calculate the reward value based on the detection result and the true label;

[0016] S8. Repeat steps S3-S7 until a set amount of state-action-reward data is collected for training the reinforcement learning agent. The current state, action, reward, and next state are used to construct experience data. The reinforcement learning agent is interactively trained and updated using a reinforcement learning algorithm. The signal detection decision agent model is output after the algorithm converges, that is, the average reward tends to be stable or the maximum number of training rounds is reached.

[0017] S9. After the test set data of step S2 passes through steps S3-S5, the signal existence confidence is obtained, and the joint deep features of the R network and the D network are input into the reinforcement learning agent that has converged after training in step S8 to obtain the threshold online. If the confidence exceeds the signal detection threshold value, it is judged as a signal. If it is lower than the signal detection threshold value, it is considered to be noise.

[0018] Furthermore, in step S2, the reference unit and the to-be-tested unit of the pulse radar echo signal are divided as follows:

[0019] After matched filtering and sampling, the radar echo signal is divided into multiple discrete units according to the range bin, forming a one-dimensional range image. Subsequently, a sliding window is moved sequentially on the range image to construct a detection unit and several reference units before and after each range unit to form a local detection window.

[0020] Each subsequent detection task is centered around a single range cell. To do this, all collected one-dimensional range profile signals must first be partitioned, and each range cell and its corresponding reference cell are combined to form a detection window, facilitating subsequent batch detection algorithm processing.

[0021] After the signal window and unit division, the processed signal needs to be divided into data sets. The division of the data set should take into account the generalization ability of the model to ensure that the data set can represent the real working environment and include different scene changes.

[0022] Furthermore, since the target detection process requires accurate modeling of the differences between the target unit and its surrounding reference units, it is necessary to ensure that they are well comparable and consistent at the feature level. Therefore, in step S3, the reference unit and target unit signals are input into the shared feature extraction network in parallel to extract their basic features. The specific steps are as follows:

[0023] S31. Initial local feature extraction: The input one-dimensional radar signal is recorded as:

[0024]

[0025] Where B represents the batch size and L represents the time dimension length of each range unit signal. A preliminary convolution process is performed on the one-dimensional radar signal to extract local information:

[0026]

[0027] Among them, σ(·) represents the ReLU activation function, BN(·) represents batch normalization, ConvlD k×1 Represents a one-dimensional convolution with a kernel width of k.

[0028] S32, Multi-scale Perception Parallel: In order to capture feature information under different receptive fields, a multi-scale convolution structure is introduced, using one-dimensional convolution with kernel sizes of 3×1, 5×1, and 7×1 respectively:

[0029]

[0030] The features at the above three scales are spliced ​​in the channel dimension to obtain multi-scale fusion features:

[0031]

[0032] S33, channel compression and fusion: The multi-scale features then enter the 1×1 convolution layer for channel fusion and dimensionality reduction:

[0033]

[0034] S34, Deep Convolution Residual Fusion: In order to further enhance the feature expression capability, a deep residual structure is introduced. This module consists of a main branch and a shortcut branch. The main branch contains two consecutive layers of convolution:

[0035] F main =BN(Convld 3×1 (σ(BN(Convld 3×1 (F red )))))

[0036] At the same time, a 1×1 convolution branch is used as a residual connection to align the channel dimensions:

[0037] F res =Convld 1×1 (F red )

[0038] Finally, the two are added together to form the basic features of the residual fusion output signal:

[0039]

[0040] The deep stacked main branch structure further improves the depth of feature expression and enhances the network's ability to model subtle differences between target echo details and background.

[0041] Furthermore, to more effectively utilize the discriminative representation features extracted by the shared feature extraction network, the present invention proposes to separately design a detection branch network (D network) for the target unit and a modeling branch network (R network) for the reference unit. Because the two types of signals have a consistent feature space representation after passing through the shared network, their domain-specific information can be mined in subsequent branches, enabling comparative learning of differential modeling.

[0042] Structurally, the D network focuses on extracting the fine structure and key details of potential targets from shared features; while the R network focuses on extracting the statistical characteristics and spatial structure of the background environment, building a stable contextual reference model, and providing a reliable reference for subsequent judgment.

[0043] Specifically, in step S4, the basic features of the reference unit signal are input into the R network to extract its deep features. The specific steps are as follows:

[0044] S41, residual: The basic characteristics of the reference unit signal are recorded as:

[0045]

[0046] Where L′ represents the total length of all reference units after concatenation. In order to enhance information flow and avoid gradient vanishing, a residual connection is performed to fuse the input signal and convolution features:

[0047] F r_main =σ(BN(Convld 3×1 (X ref )))

[0048]

[0049] Among them, F r_main is the result of main feature extraction, which represents the signal features after processing by convolution layer and batch normalization operation, F r_resout is the residual output, combined with F r_main and signal features after additional convolution operations.

[0050] S42, Maximum Pooling: Perform maximum pooling on the output of the residual module, extract important global features through pooling operations, and reduce the size of the feature map:

[0051]

[0052] Among them, L pool Indicates the length of the feature after pooling, which is usually smaller than L′ (i.e., downsampling).

[0053] S43, Full Connection: The features after maximum pooling are entered into the fully connected layer for mapping to obtain a 64-dimensional deep feature representation:

[0054]

[0055] Through the above steps, the R network can effectively extract the local features of the reference unit signal, focus on the global background information, provide support for subsequent reinforcement learning adaptive detection, and ensure the accuracy and robustness of background modeling and target detection.

[0056] Furthermore, in step S5, the basic features of the signal of the unit to be detected are input into the D network to extract its deep features, and the confidence level of the signal is output. The specific steps are as follows:

[0057] S51, convolution and channel compression: The basic characteristics of the signal of the unit to be detected are recorded as:

[0058]

[0059] Firstly, a series of convolution and activation processes are performed to extract local timing features F d_conv :

[0060]

[0061] S52, spatial attention mechanism: after the convolution operation and the activation function, the network strengthens the feature extraction of important timing positions by introducing a spatial attention mechanism. The spatial attention mechanism calculates the weight through a 3x1 convolution layer and weights the features:

[0062]

[0063] S53, convolution and channel restoration: the features processed by the spatial attention mechanism enter another 3x1 convolution layer for further feature extraction:

[0064]

[0065] S54, max pooling: the above output is subjected to max pooling to extract important global features and reduce the size of the feature map:

[0066]

[0067] S55, deep feature and confidence output: the features subjected to max pooling enter the first fully connected layer to extract 64-dimensional deep features of the unit under test:

[0068]

[0069] The deep features output by the first fully connected layer are processed by the second fully connected layer and output the confidence through the Sigmoid activation function:

[0070]

[0071] Through the above steps, the basic features of the unit under test signal are processed by the D network. The network first extracts local timing features, then focuses on important timing positions through the spatial attention mechanism to further enhance the sensitivity to target detection. The D network can extract deep features from the unit under test signal to provide a basis for subsequent reinforcement learning to output adaptive thresholds, and output the confidence to determine whether the target exists.

[0072] Further, in steps S6-S8, a reinforcement learning mechanism is introduced to model the detection process as a decision-making problem of an agent interacting with the environment. Through the deep features of the reference unit and the unit under test (steps S3-S5), the threshold setting strategy is dynamically optimized to achieve adaptive detection of complex environments. The specific steps are as follows:

[0073] In step S6, the joint deep feature representation of the reference unit and the unit to be inspected is input into the reinforcement learning agent as the state. The agent adjusts the action output detection threshold based on the current state to achieve adaptive threshold setting. Specifically:

[0074] State: This represents the environment in which the agent is located at a given moment, containing all the necessary information to describe the current environment, enabling the agent to make decisions based on the current state. Through the shared feature extraction network in steps S3-S5, as well as the D and R networks, 64-dimensional feature vectors for the reference and test cells are obtained as state inputs.

[0075] Action: This represents the behavior the agent takes based on its current state. The goal is to maximize long-term rewards by selecting appropriate actions. In step S6, the agent adjusts the target detection threshold based on the current state information. It completes the detection task by comparing it with the confidence score output by the D network.

[0076] Policy: It represents the way or rule by which the agent chooses actions in each state. It is a mapping from state to action.

[0077] Based on the current state, the agent selects and outputs an action, i.e., the threshold value for target detection, according to the strategy continuously optimized by the reinforcement learning algorithm during training. This strategy is used to guide the agent to adaptively adjust the detection threshold in different states, thereby optimizing detection performance.

[0078] Step S7: Target detection is performed on the unit to be inspected based on the detection threshold output by the agent and the confidence level output by the D network, and the reward value is calculated based on the detection result and the true label. Specifically:

[0079] S71, target signal detection, use the threshold λ output by the agent in step S6 to compare with the confidence γ of the signal of the unit to be detected to determine whether there is a target signal. If the confidence γ of the signal to be detected is greater than the threshold λ output by the agent at this time, ， It is considered that there is a target at this location. The specific expression is:

[0080]

[0081] Wherein, Detection(n) indicates whether the target signal is detected at the distance unit n.

[0082] Reward: The environment’s feedback signal to the agent’s actions in a certain state, guiding the agent to gradually learn the optimal behavior strategy through trial and error.

[0083] In step S7, as a preferred solution, the reward value is calculated based on the matching degree between the detection result output by the agent and the true label, thereby guiding the agent to adjust its detection strategy. Specifically:

[0084] S72. Result evaluation: By comparing the detection result Detection(n) and the true label Lable(n), four basic detection results are defined for structural evaluation: true positive, false positive, true negative, and false negative.

[0085] True positive: When the detection result and the true label are both targets, it means the detection is correct. The specific expression is:

[0086] TP={n|Detection(n)=1and Label(n)=1}

[0087] False positive: When the detection result is a target, but the true label is non-target, it means the detection is wrong. The specific expression is:

[0088] FP={n|Detection(n)=1and Label(n)=0}

[0089] True negative: When both the detection result and the true label are non-target, it means the detection is correct. The specific expression is:

[0090] TN={n|Detection(n)=0and Label(n)=0}

[0091] False negative: When the detection result is non-target, but the true label is target, it means missed detection. The specific expression is:

[0092] FN={n|Detection(n)=0and Label(n)=1}

[0093] S73. Design a composite reward function that takes into account both the accuracy and stability of target detection, allowing the reward function to better guide the agent to optimize the detection strategy. Specifically:

[0094] Reward based on target detection results: This part of the reward is based on the evaluation of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) in step S72;

[0095] Rewards based on threshold fluctuations: This part of the reward avoids the negative impact of excessive threshold adjustments on detection performance, while ensuring that the threshold can still be flexibly adjusted when the target signal is present. Specifically:

[0096] Threshold fluctuation penalty: In the absence of a target (false positive or true negative), excessive threshold fluctuations can lead to false alarms or missed detections, so the threshold fluctuation is penalized at this time:

[0097] R stability =-ω stability ×|T t -T t-1 |×(FP+TN)

[0098] Threshold Adaptive Reward: When the target signal is present and the judgment is correct (true positive), the agent is encouraged to flexibly adjust the threshold according to the changes in the signal, so a moderate reward is given at this time:

[0099] R adaptivity =ω adaptivity ×|T t -T t-1 |×TP

[0100] Among them, T t and T t-1 Represent the detection threshold values ​​at the current moment and the previous moment respectively, ω stability and ω adaptivity are the weight coefficients of threshold stability penalty and adaptability reward respectively;

[0101] Comprehensive reward function: The reward of comprehensive target detection results and threshold fluctuations is expressed as:

[0102] R=ω TP ×TP-ω FP ×FP-ω FN ×FN-ω stability ×|T t -T t-1 |×(FP+TN)+ω adaptivity ×|T t -T t-1 |×TP

[0103] Among them, ω TP ,ω FP ,ω FN ,ω TN is a weight coefficient used to control the influence of different detection results on the total reward, thereby optimizing the target detection performance.

[0104] Step S8: Repeat steps S3-S7, use the current state, action, reward, and next state to build experience data, and use the reinforcement learning algorithm to interactively train and update the agent. After the algorithm converges, the output is the signal detection decision agent model. Specifically:

[0105] The core goal of reinforcement learning algorithms is to learn an optimal decision-making strategy through the interaction of intelligent agents in the environment, that is, to select the optimal detection threshold action under a given signal state. Generally, reinforcement learning algorithms are divided into two categories: model-based and model-free. As the preferred option, non-model-based algorithms do not rely on the dynamic model of the environment (such as state transition probabilities and reward functions), but learn the optimal strategy directly from empirical data. Such methods are generally divided into two categories: value function (Value-Based Methods) and policy function (Policy-Based Methods). Specifically:

[0106] Value function methods: The core idea is to help the agent choose the optimal action by estimating the value of each state or state-action pair. Common value function methods include Q-learning and SARSA (State-Action-Reward-State-Action):

[0107] Q-learning guides decision-making by learning a Q-value function, the Q-value function Q(s t ,a t ) means that in a certain state s t Perform an action a t The goal of Q-learning is to find an optimal Q-value function Q*(s t ,a t ), which represents the Q value that can obtain the maximum reward among all possible strategies, so that the agent can choose the optimal action in each state. The core formula is:

[0108]

[0109] Among them, α is the learning rate, which controls the magnitude of each update, γ is the discount factor, which measures the relative importance of future rewards, and r t Represented as the immediate reward at the current moment, Represented as the next state s t+1 The maximum Q value selected under represents the maximum reward that may be obtained in the future.

[0110] SARSA is very similar to Q-learning. The main difference is that it updates the Q value by taking the action under the current policy instead of the maximum Q value. The formula is:

[0111] Q(s t ,a t )←Q(s t ,a t )+α(r t+γQ(s t+1 ,a t+1 )-Q(s t ,a t ))

[0112] Policy function method: The core idea is to directly learn the policy function, that is, given a state s t , select action a t Probability distribution, common policy function methods include policy gradient and PPO algorithm:

[0113] Policy Gradient Methods (PGMs) update policy parameters by maximizing the cumulative return using Monte Carlo or temporal difference methods, and use gradient ascent for optimization. Specifically, the objective function J(θ) of the policy is the expected return of the agent under the current policy, expressed as:

[0114]

[0115] Among them, R represents the return, π θ Represented as the agent's policy.

[0116] In order to optimize the policy parameter θ, it is necessary to calculate the gradient of the objective function relative to it, that is, the policy gradient. According to the policy gradient theorem, the gradient can be expressed as:

[0117]

[0118] Among them, θ logπ θ (a t |s t ) is expressed as the gradient of the policy with respect to the parameter, R t It is expressed as the reward starting from time step t. Using this gradient, the agent can update the policy parameters θ so that the probability of choosing the best action in each state maximizes the reward.

[0119] The PPO (Proximal Policy Optimization) algorithm is an enhanced version of the policy gradient method that aims to improve the stability and efficiency of training by reducing excessive changes during policy updates. The amplitude of policy updates is limited by maximizing an objective function that includes clipping. The objective function is expressed as:

[0120]

[0121] in, Expressed as the ratio of the new and old strategies, π θ (a t |s t )and Represent the new strategy and the old strategy respectively, It is expressed as an advantage function, that is, the quality of the current action relative to other actions, and ε is expressed as a hyperparameter that controls the range of policy changes.

[0122] Furthermore, in step S9, the test set data is processed through steps S3-S5 to obtain the signal presence confidence, and the deep features are input into the intelligent agent that has converged through training in step S8 to obtain the threshold online. If the confidence exceeds the signal detection threshold value, it is judged as a signal, and if it is lower than the signal detection threshold value, it is considered as noise.

[0123] The present invention also discloses an adaptive radar signal detection system based on reinforcement learning, which is used to execute the above method and includes the following modules:

[0124] Pulse radar echo signal acquisition module: collects radar echo signals in complex environments and provides target signal and noise signal data for subsequent processing;

[0125] Unit division and data set division module: divides the pulse radar echo signal into reference units and units to be tested, and divides the radar echo signal data into training sets and test sets. The training set is used for agent training, and the test set is used to evaluate the model effect;

[0126] Shared feature extraction network module: The reference unit and the unit to be tested signals are input into the shared feature extraction network in parallel to extract their basic features;

[0127] D network and R network modules: input the basic features of the reference unit signal into the R network to extract its deep features; input the basic features of the unit signal to be tested into the D network to extract its deep features and output the signal existence confidence;

[0128] Target detection and reward calculation module: The joint deep feature representation of the reference unit and the unit to be detected is input into the reinforcement learning agent, and the joint deep feature is used as the state. The reinforcement learning agent adjusts the action output detection threshold based on the current state. Then, based on the threshold value output by the reinforcement learning agent and the confidence level output by the D network, the signal to be detected is detected. The reward value is calculated based on the difference between the detection result and the actual label, and fed back to the agent for training;

[0129] Experience data construction and reinforcement learning training module: This module constructs experience data based on the current state, agent action, reward, and next state. It uses reinforcement learning algorithms to interactively train and update the reinforcement learning agent, optimizes the detection threshold output strategy, and outputs a signal detection decision agent model.

[0130] Online testing and result output module: After feature extraction, the test set data obtains the signal confidence level, and the deep features are input into the reinforcement learning agent. The agent outputs the detection threshold in real time. If the confidence level exceeds the signal detection threshold, it is judged as a signal. If it is lower than the signal detection threshold, it is considered as noise.

[0131] Compared with the prior art, the present invention has the following significant advantages:

[0132] 1. This invention implements adaptive detection of radar signals. By introducing a reinforcement learning mechanism, it can adaptively adjust the detection threshold based on the statistical characteristics of the current radar signal. This enables the system to automatically optimize the threshold setting under different environmental conditions, improving detection accuracy and robustness.

[0133] 2. The present invention implements intelligent decision-making driven by reinforcement learning, models the radar signal detection problem as an interactive decision-making process between the agent and the environment, and continuously optimizes the decision-making strategy during the training process through reinforcement learning. This avoids the limitations brought by manual adjustment and fixed parameters, and realizes more flexible and intelligent signal detection.

[0134] 3. This invention combines the feature extraction networks of the reference unit and the unit under test to more comprehensively model the statistical characteristics of the signal through joint feature representation. This enables the agent to fully understand the changes in different signal backgrounds, thereby improving the accuracy and reliability of detection.

[0135] 4. The present invention has high robustness and accuracy, can effectively cope with multi-target interference, complex clutter and non-Gaussian noise environment, automatically adjust the detection strategy, greatly reduce the occurrence of false alarms and missed detections, and thus improve the accurate recognition rate of target signals.

[0136] In summary, this invention implements an adaptive radar signal detection method by introducing a reinforcement learning mechanism. By dividing the pulse radar echo signal into reference cells and target cells, and inputting these into a feature extraction network, the combined features of the reference and target cells are fed into a reinforcement learning agent. The agent then adjusts the detection threshold based on real-time conditions, dynamically optimizing the target detection strategy. This method can automatically adjust the threshold in complex environments, improving the accuracy and robustness of target detection, and effectively enhancing the intelligence and automation level of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 This is a flow chart of an adaptive radar signal detection method based on reinforcement learning according to a preferred embodiment of the present invention;

[0138] Figure 2 This is the flowchart of the classic method CFAR in radar target detection;

[0139] Figure 3 This is a structural diagram of a shared feature extraction network module involved in a preferred embodiment of the present invention;

[0140] Figure 4 This is a structural diagram of the R network module involved in the preferred embodiment of the present invention;

[0141] Figure 5 This is a structural diagram of the D network module involved in the preferred embodiment of the present invention;

[0142] Figure 6 This is an overall framework diagram of the reinforcement learning algorithm involved in the preferred embodiment of the present invention;

[0143] Figure 7 1 is a block diagram of the PPO algorithm in a preferred embodiment of the present invention;

[0144] Figure 8 This is a block diagram of an adaptive radar signal detection system based on reinforcement learning according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0145] To more clearly illustrate the embodiments of the present invention, specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive efforts.

[0146] See also Figure 1 This embodiment provides an adaptive radar signal detection method based on reinforcement learning, which specifically includes the following steps:

[0147] S1. Collect pulse radar echo signals in complex environments and provide data on target signals and noise signals. The complex environment described here refers to the typical scenario encountered by radar when detecting targets. Its core characteristics include:

[0148] 1. Multi-target masking: Multiple flying targets are densely distributed in space, and the echo signals overlap in the time and frequency domains, making it difficult to separate target features;

[0149] 2. Clutter edge effect: The clutter power distribution caused by non-target reflections such as ground objects and weather in the radar beam area suddenly increases, causing a sharp increase in the false alarm rate.

[0150] 3. Strong clutter obscures weak targets: If there are signals with a very large dynamic range in the scene (such as strong ground clutter and weak targets coexisting), the clutter may not be effectively suppressed, causing the weak targets to be submerged.

[0151] See also Figure 2CFAR is a classic method in radar target detection, used to maintain a constant false alarm rate in dynamic environments. By setting adaptive thresholds for the signal, the target signal is automatically identified and noise is suppressed. The core problem of the traditional CFAR method is that it relies on static models and fixed parameters, making it difficult to adapt to the above-mentioned dynamic and non-uniform actual environment. Therefore, the present invention proposes an adaptive radar signal detection method based on reinforcement learning, which can improve the adaptive adjustment capability of the detection threshold, thereby realizing intelligent and efficient radar detection in dynamic and non-uniform scenes under complex backgrounds.

[0152] S2. Divide the pulse radar echo signal into reference units and units to be tested, and then divide the pulse radar echo signal dataset into a training set and a test set. The division of the reference unit and the units to be tested of the pulse radar echo signal is as follows:

[0153] Detection unit: A signal unit that determines whether a target exists. It is usually located in the center of the sliding window. If the signal strength of the detection unit exceeds the threshold set by the noise background estimated by the reference unit, it is considered that there is a target at that location.

[0154] Reference cells: These are signal cells used to estimate the background noise level, usually located around the detection cells. By averaging or performing other statistical processing on the signal strength of the reference cells, the noise level of the current environment can be estimated, and the detection threshold of the corresponding detection cell can be further determined.

[0155] Protection unit: used to protect the detection unit from the influence of the reference unit. It is usually located between the reference unit and the detection unit to prevent the target signal from mixing into the reference unit and affecting the estimation of the noise background.

[0156] After matched filtering and sampling, the radar echo signal is divided into multiple discrete units according to the range bin, forming a one-dimensional range image. Subsequently, a sliding window is moved sequentially on the range image to construct a detection unit and several reference units before and after each range unit to form a local detection window.

[0157] Each subsequent detection task is centered around a single range cell. To do this, all collected one-dimensional range profile signals must first be partitioned, and each range cell and its corresponding reference cell are combined to form a detection window, facilitating subsequent batch detection algorithm processing.

[0158] After the signal window and unit division, the processed signal needs to be divided into data sets. The division of the data set should take into account the generalization ability of the model to ensure that the data set can represent the real working environment and include different scene changes.

[0159] Furthermore, since the target detection process requires accurate modeling of the differences between the unit to be detected and its surrounding reference units, it is necessary to ensure that they have good comparability and consistency at the feature level. Therefore, in step S3, the reference unit and the unit to be detected signals are input to the shared feature extraction network in parallel to extract their basic features, such as Figure 3 As shown, S3, the reference unit and the unit to be tested signals are input in parallel into the shared feature extraction network to extract basic features. The specific steps are as follows:

[0160] S31. Initial local feature extraction: The input one-dimensional radar signal is recorded as:

[0161]

[0162] Where B represents the batch size and L represents the time dimension length of each distance unit signal. A preliminary convolution process is performed to extract local information:

[0163]

[0164] Among them, σ(·) represents the ReLU activation function, BN(·) represents batch normalization (BatchNormalization), ConvlD k×1 It is represented as a one-dimensional convolution with a kernel width of k.

[0165] S32, multi-scale perception parallel: In order to capture the feature information under different receptive fields, a multi-scale convolution structure is introduced (see Figure 3 ), using one-dimensional convolution with kernel sizes of 3×1, 5×1, and 7×1 respectively:

[0166]

[0167] The features at the above three scales are spliced ​​in the channel dimension to obtain multi-scale fusion features:

[0168]

[0169] Through a multi-scale parallel structure, the same network can simultaneously perceive texture details and statistical patterns at different time scales. Smaller convolution kernels (such as 3×1) focus on capturing local changes and faint target features, while larger convolution kernels (such as 7×1) are more adept at extracting stable structures and global contour features in background areas. This structure enhances the network's ability to jointly model background patterns and target signals at multiple scales, allowing the background patterns reflected in the reference unit and the potential target features in the unit to be detected to be effectively integrated in a unified structure, providing a more stable and discriminative feature foundation for subsequent anomaly detection and target recognition.

[0170] S33, channel compression and fusion: The multi-scale features then enter the 1×1 convolution layer for channel fusion and dimensionality reduction:

[0171]

[0172] S34, Deep Convolution Residual Fusion: In order to further enhance the feature expression capability, a deep residual structure is introduced. This module consists of a main branch and a shortcut branch. The main branch contains two consecutive layers of convolution:

[0173] F main =BN(Convld 3×1 (σ(BN(Convld 3×1 (F red )))))

[0174] At the same time, a 1×1 convolution branch is used as a residual connection to align the channel dimensions:

[0175] F res =Convld 1×1 (F red )

[0176] Finally, the two are added together to form the basic features of the residual fusion output signal:

[0177]

[0178] The deep stacked main branch structure further enhances the depth of feature expression and strengthens the network's ability to model subtle differences between target echo details and background. Residual connections introduce shortcut paths, helping to alleviate the vanishing gradient problem in deep networks and improving training convergence speed and feature transfer efficiency.

[0179] Furthermore, to more effectively utilize the discriminative representation features extracted by the shared feature extraction network, the present invention proposes to separately design a detection branch network (D network) for the target unit and a modeling branch network (R network) for the reference unit. Because the two types of signals have a consistent feature space representation after passing through the shared network, their domain-specific information can be mined in subsequent branches, enabling comparative learning of differential modeling.

[0180] Structurally, the D network focuses on extracting the fine structure and key details of potential targets from shared features; while the R network focuses on extracting the statistical characteristics and spatial structure of the background environment, building a stable contextual reference model, and providing a reliable reference for subsequent judgment.

[0181] S4, input the basic features of the reference unit signal into the R network to extract its deep features, such as Figure 4 As shown, the specific steps are as follows:

[0182] S41, residual: The basic characteristics of the reference unit signal are recorded as:

[0183]

[0184] Where L′ represents the total length of all reference units after concatenation. In order to enhance information flow and avoid gradient vanishing, a residual connection is performed to fuse the input signal and convolution features:

[0185] F r_main =σ(BN(Convld 3×1 (X ref )))

[0186]

[0187] Among them, F r_main is the result of main feature extraction, which represents the signal features after processing by convolution layer and batch normalization operation, F r resout is the residual output, combined with F r main and signal features after additional convolution operations.

[0188] S42, Maximum Pooling: Perform maximum pooling on the output of the residual module, extract important global features through pooling operations, and reduce the size of the feature map:

[0189]

[0190] Among them, L pool Indicates the length of the feature after pooling, which is usually smaller than L′ (i.e., downsampling).

[0191] S43, Full Connection: The features after maximum pooling are entered into the fully connected layer for mapping to obtain a 64-dimensional deep feature representation:

[0192]

[0193] Through the above steps, the R network can effectively extract the local features of the reference unit signal, focus on the global background information, provide support for subsequent reinforcement learning adaptive detection, and ensure the accuracy and robustness of background modeling and target detection.

[0194] S5. Input the basic features of the signal of the unit to be detected into the D network to extract its deep features, and output the confidence level of the signal, such as Figure 5 As shown, the specific steps are as follows:

[0195] S51, convolution and channel compression: The basic characteristics of the signal of the unit to be detected are recorded as:

[0196]

[0197] First, a series of convolution and activation processes are performed to extract the local temporal features F d_conv :

[0198]

[0199] S52, Spatial Attention Mechanism: After the convolution operation and activation function, the network introduces a spatial attention mechanism to strengthen the feature extraction of important temporal positions. The spatial attention mechanism calculates weights through a 3×1 convolution layer and weights the features:

[0200]

[0201] This operation assigns different weights to each temporal position in the signal, increasing the network's focus on key time steps. This mechanism helps improve the network's object detection capabilities, especially when the temporal signal is complex, by focusing on more critical time points.

[0202] S53, Convolution and Channel Restoration: The features processed by the spatial attention mechanism will enter another 3×1 convolution layer for further feature extraction:

[0203]

[0204] This processing helps the network provide sufficient dimensions for subsequent feature fusion, thereby further mining high-level features in the signal and enabling the network to learn more complex patterns.

[0205] S54, maximum pooling: Perform maximum pooling on the above output, extract important global features through pooling operation, and reduce the size of the feature map:

[0206]

[0207] S55, deep features and confidence output: The features after maximum pooling are fed into the first fully connected layer to extract the 64-dimensional deep features of the unit to be inspected:

[0208]

[0209] The deep features output by the first fully connected layer are processed by the second fully connected layer and the confidence is output through the Sigmoid activation function:

[0210]

[0211] Through these steps, the basic features of the signal from the unit under inspection are processed by the D network. The network first extracts local temporal features and then uses a spatial attention mechanism to focus on important temporal locations, further enhancing sensitivity for target detection. The D network extracts deep features from the signal from the unit under inspection, providing the basis for the subsequent reinforcement learning output of the adaptive threshold, and determines the presence of the target through the output of the confidence level.

[0212] Furthermore, in steps S6-S8, a reinforcement learning mechanism is introduced to model the detection process as a decision-making problem of the interaction between the intelligent agent and the environment. By perceiving the deep features of the reference unit and the unit to be detected (steps S3-S5), the threshold setting strategy is dynamically optimized to achieve adaptive detection in complex environments. The specific steps are as follows:

[0213] S6. The joint deep feature representation of the reference unit and the unit to be inspected is input to the reinforcement learning agent, and the joint deep feature is used as the state. The agent adjusts the action output detection threshold based on the current state to achieve adaptive threshold setting. Specifically:

[0214] State: This represents the environment in which the agent is located at a given moment and contains all the necessary information to describe the current environment, enabling the agent to make decisions based on the current state. Through the shared feature extraction network in steps S3-S5, as well as the D network and R network, 64-dimensional feature vectors for the reference unit and the unit to be detected are obtained, respectively. These feature vectors contain relevant characteristics such as the current background noise level, target signal characteristics, and environmental conditions (such as signal-to-noise ratio and number of targets) for target detection.

[0215] Action: This represents the behavior the agent takes based on its current state. The goal is to maximize long-term rewards by selecting appropriate actions. In step S6, the agent adjusts the target detection threshold based on the current state information. It completes the detection task by comparing it with the confidence score output by the D network.

[0216] Policy: This represents the method or rules by which an agent chooses actions in each state, representing a mapping from state to action. The goal of reinforcement learning is to learn an optimal policy that maximizes the agent's reward over time. Through reinforcement learning training, the agent gradually learns how to adjust its detection threshold based on environmental changes, achieving adaptive detection.

[0217] Based on the current state, the agent selects and outputs an action, i.e., the threshold value for target detection, according to the strategy continuously optimized by the reinforcement learning algorithm during training. This strategy is used to guide the agent to adaptively adjust the detection threshold in different states, thereby optimizing detection performance.

[0218] S7, according to the detection threshold output by the reinforcement learning agent and the confidence output by the D network, target detection is performed on the unit to be detected, and a reward value is calculated according to the detection result and the true label, specifically:

[0219] S71, target signal detection, the threshold λ output by the agent in step S6 is compared with the confidence γ of the signal of the unit to be detected to determine whether there is a target signal. If the confidence γ of the detected signal is greater than the threshold λ output by the agent at this time ， it is considered that there is a target at this position. The specific expression is:

[0220]

[0221] wherein Detection(n) represents whether a target signal is detected at a distance of unit n.

[0222] Reward: the feedback signal of the environment to the action taken by the agent in a certain state, guiding the agent to gradually learn the optimal behavior strategy through the trial and error process. In step S7, as a preferred solution, the reward value is calculated according to the matching degree of the detection result output by the agent and the true label, so as to guide the agent to adjust its detection strategy, specifically:

[0223] S72, result evaluation, four basic detection results are defined by comparing the detection result Detection(n) and the true label Lable(n):

[0224] True positive (TP): when the detection result and the true label are both targets, it indicates that the detection is correct, and the specific expression is:

[0225] TP={n|Detection(n)=1and Lable(n)=1}

[0226] False positive (FP): when the detection result is a target, but the true label is a non-target, it indicates that the detection is wrong, and the specific expression is:

[0227] FP={n|Detection(n)=1and Lable(n)=0}

[0228] True negative (TN): when the detection result and the true label are both non-targets, it indicates that the detection is correct, and the specific expression is:

[0229] TN={n|Detection(n)=0and Lable(n)=0}

[0230] False Negative (FN): When the detection result is non-target, but the true label is target, it means missed detection. The specific expression is:

[0231] FN={n|Detection(n)=0and Label(n)=1}

[0232] S73. Design a composite reward function that takes into account both the accuracy and stability of target detection, allowing the reward function to better guide the agent to optimize the detection strategy. Specifically:

[0233] Rewards based on target detection results: This part of the reward is based on the evaluation of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) in S72. The specific expression is:

[0234] True Positive (TP): If an object is detected and the true label is also an object, a positive reward is given. The size of the reward is proportional to the quality of the object detection, and the expression is:

[0235] R TP =+ω TP ×TP

[0236] False Positive (FP): If an object is detected but the true label is non-object, a negative reward is given to penalize the wrong detection. The expression is:

[0237] R FP =-ω FP ×FP

[0238] False negative (FN): If the target is not detected but the true label is the target, a negative reward is given to penalize missed detection. The expression is:

[0239] R FN =-ω FN ×FN

[0240] True Negative (TN): If the target is not detected and the true label is non-target, give zero reward and keep it neutral. The expression is:

[0241] R TN =0

[0242] Among them, ω TP ,ω FP ,ω FN ,ω TN is a weight coefficient used to control the influence of different detection results on the total reward, thereby optimizing the target detection performance.

[0243] Rewards based on threshold fluctuations: This part of the reward avoids the negative impact of excessive threshold adjustments on detection performance, while ensuring that the threshold can still be flexibly adjusted when the target signal is present. Specifically:

[0244] Threshold fluctuation penalty: In the absence of a target (false positive or true negative), excessive threshold fluctuations can lead to false alarms or missed detections, so the threshold fluctuation is penalized at this time:

[0245] R stability =-ω stability ×|T t -T t-1 |×(FP+TN)

[0246] Threshold Adaptive Reward: When the target signal is present and the judgment is correct (true positive), the agent is encouraged to flexibly adjust the threshold according to the changes in the signal, so a moderate reward is given at this time:

[0247] R adaptivity =ω adaptivity ×|T t -T t-1 |×TP

[0248] Among them, T t and T t-1 Represent the detection threshold values ​​at the current moment and the previous moment respectively, ω stability and ω adaptivity are the weight coefficients of threshold stability penalty and adaptability reward respectively.

[0249] Comprehensive reward function: Finally, the reward function of the integrated target detection results and threshold fluctuations can be expressed as:

[0250] R=R TP +R FP +R FN +R TN +R stability +R adaptivity

[0251] The specific formula is:

[0252] R=ω TP ×TP-ω FP ×FP-ω FN ×FN-ω stability ×|T t -T t-1 |×(FP+TN)+ω adaptivity ×|T t -T t-1 |×TP

[0253] Among them, ω TP ,ωFP ,ω FN ,ω TN is a weight coefficient used to control the influence of different detection results on the total reward, thereby optimizing the target detection performance.

[0254] S8. Repeat steps S3-S7 until a sufficient amount of state-action-reward data is collected to train the reinforcement learning agent. The current state, action, reward, and next state are used to construct experience data. The reinforcement learning algorithm is used to interactively train and update the agent. After the algorithm converges, that is, the average reward tends to be stable or the maximum number of training rounds is reached, the signal detection decision agent model is output. Specifically:

[0255] See also Figure 6 In reinforcement learning, experience data is a crucial part of the agent’s learning process. It records each round of interaction that the agent experiences in the environment, including the current state, the action taken, the reward obtained, and the next state. It is usually stored in the form of a four-tuple, expressed as:

[0256] e t =(s t ,a t ,r t ,s t+1 )

[0257] Among them, s t Indicates the current state (state), a t Indicates the action taken in the current state, r t Indicates that by taking action a t The reward obtained after s t+1 Indicates taking action a t The next state obtained.

[0258] Typically, these experience data are stored in the Experience Replay Buffer, and small batches of data are randomly extracted for training, thereby avoiding over-reliance on adjacent data in the time series, reducing autocorrelation during training, and improving training stability.

[0259] See also Figure 7 As a preferred solution, the Proximal Policy Optimization (PPO) algorithm is an enhanced version of the policy gradient method. It aims to improve the stability and efficiency of training by reducing excessive changes during policy updates. The amplitude of policy updates is limited by maximizing an objective function that includes clipping. The specific training and update process is as follows:

[0260] 1. Initialization:

[0261] Policy Network (Actor Network): The goal is to output the current policy π θ (a t |s t ), that is, according to the current state s t Output: Take each possible action a t probability.

[0262] Value Network (Critic Network): The goal is to estimate the current state s t The state value V φ (s t ), helping to evaluate the quality of the current strategy.

[0263] Hyperparameters: Choose appropriate hyperparameters, such as learning rate α, clipping threshold ε, discount factor γ, GAE (generalized advantage estimation) parameter λ, etc.

[0264] 2. Data Collection

[0265] During training, the agent uses the current strategy to interact with the environment and collect a batch of sample data.

[0266] 3. Calculate the advantage function:

[0267] Advantage function It measures the performance of the current strategy compared to the baseline strategy (state value function) in a certain state. It is usually calculated using the generalized advantage estimate (GAE):

[0268]

[0269] Among them, δ t is the TD (temporal difference) residual at time step t:

[0270] δ t =r t +γV θ (s t+1 )-V θ (s t )

[0271] Where γ is the discount factor, V θ (s t ) represents the state value estimated by the value network.

[0272] 4. Calculate the objective function:

[0273] The PPO algorithm updates the strategy by optimizing the objective function. In the PP0-clip version, the amplitude of the strategy update is limited by maximizing an objective function that includes clipping. The objective function is expressed as:

[0274]

[0275] in, Expressed as the ratio of the new and old strategies, It is expressed as an advantage function, that is, the quality of the current action relative to other actions, and ε is expressed as a hyperparameter that controls the range of policy changes.

[0276] 5. Policy Network Update:

[0277] Use gradient descent to optimize the objective function L CLIP (θ), update the parameters θ of the policy network:

[0278] θ←θ+α▽ θ L CLIP (θ)

[0279] Where α is the learning rate, ▽ θ L CLIP (θ) is the gradient of the objective function with respect to the policy parameters θ.

[0280] 6. Value Network Update:

[0281] While updating the strategy, PPO also updates the value network and optimizes the loss function of the value network:

[0282]

[0283] Among them, R t is the cumulative return, usually calculated via Monte Carlo returns or time-difference (TD) methods.

[0284] 7. Repeat training:

[0285] Repeat steps 2-6 until the maximum number of training times is reached or the loss function converges.

[0286] S9. After the test set data passes through steps S3-S5, the signal presence confidence is obtained, and the joint deep features of the R network and the D network are input into the online acquisition threshold of the intelligent agent trained and converged in step S8. If the confidence exceeds the signal detection threshold, it is judged as a signal. If it is lower than the signal detection threshold, it is considered as noise. The specific evaluation indicators are as follows:

[0287] Detection Rate (DR): The ratio of signals correctly identified by the model as target signals. It measures the model's ability to detect true target signals:

[0288]

[0289] False Negative Rate (FNR): indicates the proportion of samples that are not correctly detected as target signals by the model among all samples that are actually target signals:

[0290]

[0291] False Alarm Rate (FAR): Indicates the proportion of samples that are mistakenly detected as target signals among all samples that are actually noise or non-target signals:

[0292]

[0293] Precision: Indicates the proportion of samples predicted by the model as target signals that are actually target signals. It measures how many of the predicted results detected as target signals are actually target signals:

[0294]

[0295] Recall: Indicates the proportion of target signals that the model can identify to the total target signals, which measures the completeness of the model in detecting target signals:

[0296]

[0297] F1 Score: The harmonic mean of precision and recall, an indicator that comprehensively considers precision and recall:

[0298]

[0299] like Figure 8 As shown, the present invention also discloses an adaptive radar signal detection system based on reinforcement learning, which is used to execute the above method, and includes the following modules:

[0300] Pulse radar echo signal acquisition module: collects radar echo signals in complex environments and provides target signal and noise signal data for subsequent processing;

[0301] Unit division and data set division module: divides the pulse radar echo signal into reference units and units to be tested, and divides the radar echo signal data into training sets and test sets. The training set is used for agent training, and the test set is used to evaluate the model effect;

[0302] Shared feature extraction network module: The reference unit and the unit to be tested signals are input into the shared feature extraction network in parallel to extract their basic features;

[0303] D network and R network modules: input the basic features of the reference unit signal into the R network to extract its deep features; input the basic features of the unit signal to be tested into the D network to extract its deep features and output the signal existence confidence;

[0304] Target detection and reward calculation module: The joint deep feature representation of the reference unit and the unit to be detected is input into the reinforcement learning agent, and the joint deep feature is used as the state. The reinforcement learning agent adjusts the action output detection threshold based on the current state. Then, based on the threshold value output by the reinforcement learning agent and the confidence level output by the D network, the signal to be detected is detected. The reward value is calculated based on the difference between the detection result and the actual label, and fed back to the agent for training;

[0305] Experience data construction and reinforcement learning training module: This module constructs experience data based on the current state, agent action, reward, and next state. It uses reinforcement learning algorithms to interactively train and update the reinforcement learning agent, optimizes the detection threshold output strategy, and outputs a signal detection decision agent model.

[0306] Online testing and result output module: After feature extraction of the test data, the signal presence confidence level is obtained. This deep feature is then fed into the reinforcement learning agent, which then outputs a detection threshold in real time. If the confidence level exceeds the signal detection threshold, the signal is considered a signal; if it is below the threshold, the signal is considered noise. This invention enables adaptive detection of radar signals in complex environments, improving the robustness and accuracy of target detection.

[0307] For other contents of this embodiment, please refer to the above method embodiment.

[0308] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive radar signal detection method based on reinforcement learning, characterized in that: The following steps are involved: S1, collects pulse radar echo signals and provides data of target signals and noise signals; S2. Divide the pulse radar echo signal into reference units and test units, and then divide the pulse radar echo signal data set into a training set and a test set; S3, inputting the reference unit and the unit to be tested signals into the shared feature extraction network in parallel to extract basic features; S4, input the basic features of the reference unit signal into the reference unit feature extraction network, i.e., the R network, to extract deep features; S5. Input the basic features of the signal of the unit to be inspected into the feature extraction network of the unit to be inspected, i.e., the D network, to extract deep features and output the confidence level of the signal. S6. Input the joint deep feature representation of the reference unit and the unit to be inspected into the reinforcement learning agent, and use the joint deep feature as the state. The reinforcement learning agent adjusts the action output detection threshold based on the current state; S7. Perform target detection on the unit to be inspected based on the detection threshold output by the reinforcement learning agent and the confidence level output by the D network, and calculate the reward value based on the detection result and the true label; S8. Repeat steps S3-S7 until a set amount of state-action-reward data is collected for training the reinforcement learning agent. The current state, action, reward, and next state are used to construct experience data. The reinforcement learning agent is interactively trained and updated using a reinforcement learning algorithm. After the algorithm converges, a signal detection decision agent model is output. S9. After the test set data of step S2 passes through steps S3-S5, the signal existence confidence is obtained, and the joint deep features of the R network and the D network are input into the reinforcement learning agent that has converged after training in step S8 to obtain the threshold online. If the confidence exceeds the signal detection threshold value, it is judged as a signal. If it is lower than the signal detection threshold value, it is judged as noise.

2. The adaptive radar signal detection method based on reinforcement learning according to claim 1, characterized in that: Step S3 is as follows: S31. Initial local feature extraction: The input one-dimensional radar signal is recorded as: Among them, B represents the batch size, L represents the time dimension length of each range unit signal, and preliminary convolution processing is performed on the one-dimensional radar signal to extract local feature information: Among them, σ(·) represents the ReLU activation function, BN(·) represents batch normalization, and ConvlD k×1 Represents one-dimensional convolution, k is the kernel width; S32, Multi-scale Perception Parallel: Introducing a multi-scale convolution structure, using one-dimensional convolution with kernel sizes of 3×1, 5×1, and 7×1 respectively: The features at the above three scales are spliced ​​in the channel dimension to obtain multi-scale fusion features: S33, channel compression and fusion: S34, deep convolution residual fusion: F main =BN(Convld 3×1 (σ(BN(Convld 3×1 (F red ))))) Use a 1×1 convolution branch as the residual connection to align the channel dimensions: F res =Convld 1×1 (F red ) Finally, the two are added together to form the basic features of the residual fusion output signal:

3. The adaptive radar signal detection method based on reinforcement learning according to claim 2, characterized in that: Step S4 is specifically as follows: S41, residual: The basic characteristics of the reference unit signal are recorded as: Among them, L′ represents the total length of all reference units after splicing; residual connection is performed to fuse the input signal and convolution features: F r_main =σ(BN(Convld 3×1 (X ref ))) Among them, F r_main represents the signal features after processing by convolution layer and batch normalization operation, F r_resout is the residual output; S42, maximum pooling: Among them, L pool Indicates the length of the feature after pooling; S43, Full Connection: The features after maximum pooling are entered into the fully connected layer for mapping to obtain a 64-dimensional deep feature representation:

4. The adaptive radar signal detection method based on reinforcement learning according to claim 3, characterized in that: Step S5 is specifically as follows: S51, convolution and channel compression: The basic characteristics of the signal of the unit to be detected are recorded as: After convolution and activation processing, local temporal features F are extracted d_conv : S52, spatial attention mechanism: calculate weights through a 3×1 convolution layer and weight the features: S53, convolution and channel recovery: S54, maximum pooling: S55, deep features and confidence output: The features after maximum pooling are fed into the first fully connected layer to extract the 64-dimensional deep features of the unit to be inspected: Output confidence through Sigmoid activation function:

5. The adaptive radar signal detection method based on reinforcement learning according to claim 4, characterized in that: In step S6: Status: Through the shared feature extraction network of steps S3-S5, as well as the D network and the R network, 64-dimensional feature vectors of the reference unit and the unit to be tested are obtained respectively; Action: In step S6, the agent adjusts the target detection threshold based on the current state information and completes the detection task by comparing it with the confidence level output by the D network; Strategy: Through reinforcement learning training, the agent learns to adjust the detection threshold according to environmental changes.

6. The adaptive radar signal detection method based on reinforcement learning according to claim 5, characterized in that: Step S7 is specifically as follows: The threshold λ output by the agent in step S6 is compared with the confidence γ of the signal of the unit to be detected to determine whether the target signal exists. The specific expression is: Where Detection(n) indicates whether the target signal is detected at range unit n; By comparing the detection result Detection(n) and the true label Lable(n), four basic detection results are defined for structural evaluation: True positive: When the detection result and the true label are both targets, it means the detection is correct. The specific expression is: TP={n|Detection(n)=1and Label(n)=1} False positive: When the detection result is a target, but the true label is non-target, it means the detection is wrong. The specific expression is: FP={n|Detection(n)=1and Label(n)=0} True negative: When both the detection result and the true label are non-target, it means the detection is correct. The specific expression is: TN={n|Detection(n)=0and Label(n)=0} False negative: When the detection result is non-target, but the true label is target, it means missed detection. The specific expression is: FN={n|Detection(n)=0and Label(n)=1} Design a composite reward function as follows: Rewards based on target detection results: evaluation based on true positive TP, false positive FP, false negative FN, and true negative TN; Rewards based on threshold fluctuations: including threshold fluctuation penalties and threshold adaptive rewards, specifically: Threshold Fluctuation Penalty: Penalize threshold fluctuations: R stability =-ω stability ×|T t -T t-1 |×(FP+TN) Threshold Adaptability Reward: Gives rewards for: R adaptivity =ω adaptivity ×|T t -T t-1 |×TP Among them, T t and T t-1 Represent the detection threshold values ​​at the current moment and the previous moment respectively, ω stability and ω adaptivity are the weight coefficients of threshold stability penalty and adaptability reward respectively; Comprehensive reward function: The reward of comprehensive target detection results and threshold fluctuations is expressed as: R=ω TP ×TP-ω FP ×FP-ω FN ×FN-ω stability ×|T t -T t-1 |×(FP+TN)+ω adaptivity ×|T t -T t-1 |×TP Among them, ω TP ,ω FP ,ω FN ,ω TN is the weight coefficient.

7. The adaptive radar signal detection method based on reinforcement learning according to claim 6, characterized in that: In step S8, steps S3-S7 are repeated with the training set data, and experience data is constructed using the current state, action, reward and next state. The agent is interactively trained and updated using a reinforcement learning algorithm. After the algorithm converges, a signal detection decision agent model is output. The algorithm converges when the average reward tends to be stable or reaches the maximum number of training rounds. The reinforcement learning algorithm is non-model-based, including two types: value function and policy function.

8. The adaptive radar signal detection method based on reinforcement learning according to claim 7, characterized in that: In step S8, the value function helps the agent select the optimal action by estimating the value of each state or state-action pair, including Q-learning and SARSA; Among them, Q-learning guides decision-making by learning a Q value function, the Q value function Q(s t ,a t ) means that in a certain state s t Perform an action a t The goal of Q-learning is to find an optimal Q-value function Q*(s t ,a t ), so that the agent can choose the optimal action in each state, the formula is: Among them, α represents the learning rate, γ represents the discount factor, and r t Indicates the instant reward at the current moment, Indicates the next state s t+1 The maximum Q value selected next represents the maximum reward that may be obtained in the future; Among them, SARSA adopts the action under the current strategy when updating its Q value. The formula is: Q(s t ,a t )←Q(s t ,a t )+α(r t +γQ(s t+1 ,a t+1 )-Q(s t ,a t ))。 9. The adaptive radar signal detection method based on reinforcement learning according to claim 7, characterized in that: In step S8, the policy function is a direct learning policy function, that is, given the state s t , select action a t The probability distribution of , including policy gradient and PPO algorithms; Among them, the policy gradient method updates the policy parameters by maximizing the cumulative return using the Monte Carlo method or the temporal difference method, and uses the gradient ascent method for optimization. The objective function J(θ) of the policy is the expected return of the agent under the current policy, which is expressed as: Among them, R represents the return, π θ Represented as the agent's policy; In order to optimize the policy parameter θ, it is necessary to calculate the gradient of the objective function with respect to it. The gradient is expressed as: in, represents the gradient of the policy with respect to the parameters, R t represents the return starting from time step t; Among them, PPO limits the amplitude of policy update by maximizing an objective function including clipping. The objective function is expressed as: in, represents the ratio of the new and old strategies, π θ (a t |s t )and Represent the new strategy and the old strategy respectively, represents the advantage function and ε represents the hyperparameter.

10. An adaptive radar signal detection system based on reinforcement learning, configured to execute the method according to any one of claims 1 to 9, characterized in that: Includes the following modules: Pulse radar echo signal acquisition module: collects radar echo signals and provides data on target signals and noise signals; Unit division and data set division module: divides the pulse radar echo signal into reference units and units to be tested, and divides the radar echo signal data into training sets and test sets; Shared feature extraction network module: The reference unit and the unit to be tested signals are input into the shared feature extraction network in parallel to extract basic features; D network and R network modules: input the basic features of the reference unit signal into the R network to extract deep features; input the basic features of the unit signal to be tested into the D network to extract deep features and output the signal existence confidence; Target detection and reward calculation module: The joint deep feature representation of the reference unit and the unit to be detected is input into the reinforcement learning agent, and the joint deep feature is used as the state. The reinforcement learning agent adjusts the action output detection threshold based on the current state. Then, based on the threshold value output by the reinforcement learning agent and the confidence level output by the D network, the signal to be detected is detected. The reward value is calculated based on the difference between the detection result and the actual label, and fed back to the agent for training; Experience data construction and reinforcement learning training module: This module constructs experience data based on the current state, agent action, reward, and next state, uses reinforcement learning algorithms to interactively train and update the reinforcement learning agent, and outputs a signal detection decision agent model. Online testing and result output module: After feature extraction, the test set data obtains the signal confidence level, and the deep features are input into the reinforcement learning agent. The agent outputs the detection threshold in real time. If the confidence level exceeds the signal detection threshold, it is judged as a signal. If it is lower than the signal detection threshold, it is considered as noise.

Images