An autonomous start-stop control decision method for unmanned radar

Abstract

The application discloses an unmanned radar autonomous start-stop control decision method and system and a medium, and belongs to the field of radar system comprehensive guarantee. The method comprises the following steps: step S1, establishing an unmanned radar operation control state machine; step S2, designing a state-based reinforcement learning decision model; step S3, model training and optimization evaluation based on multiple scenes; and step S4, instruction execution and real-time feedback. The application combines the state machine and the reinforcement learning, overcomes the problems of single decision capability and insufficient environmental adaptability under the control of traditional preset rules, and realizes autonomous and intelligent start-stop control of the unmanned radar under complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of integrated support technology for radar systems, and specifically to an autonomous start-stop control decision-making method for unmanned radar. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, the Internet of Things, and big data, unmanned systems have become important equipment in civilian, scientific research, and other fields. As the "eyes" of these unmanned systems, radar sensors play a crucial role. However, traditional radar operating modes can no longer fully meet the new requirements of unmanned and intelligent systems, and "autonomous decision-making" has become a key bottleneck for unmanned systems to move from "automation" to "intelligence."

[0003] Compared to traditional manned radar, the biggest advantage of unmanned radar is its high degree of automation and intelligence, enabling it to operate, monitor, and manage without human intervention. Autonomous start-up and shutdown control of unmanned radar is a crucial aspect of radar health management and a key foundation for its autonomous operation. By quantitatively analyzing and evaluating the radar's status information and self-test data, a comprehensive assessment of its operational readiness is achieved. This allows remote users to monitor the radar's overall status in real time, improving its autonomous operation and maintenance capabilities in new system environments.

[0004] Existing unmanned radar systems rely on preset rules for equipment control, making it difficult to make optimal decisions in complex and ever-changing environments. This invention proposes an autonomous start-stop control design method based on state machines and reinforcement learning. By preprocessing the acquired system state information, combining state machine processing with reinforcement analysis under complex operating conditions, a complete autonomous start-stop control solution for radar in unmanned environments is formed, providing a reference for the operational control design of other unmanned equipment.

[0005] In June 2023, Hong Chang, Zhai Dongliang, Ding Zhihui, and Shen Yang proposed "A Design Method and Implementation of a Shipborne Radar Health Management System," published in Volume 43, Issue 2 of *Radar and Countermeasures*. This article primarily proposes a shipborne health management system capable of adapting to different radar characteristics and being rapidly constructed and deployed; it completes the overall system architecture and core function design, including status monitoring, fault diagnosis, fault prediction, status assessment, and maintenance support decision-making; and it implements and verifies the system based on a certain type of shipborne radar. However, the article does not address the autonomous start-stop control of the radar, nor does it address the autonomous control design for unmanned radars.

[0006] In July 2023, Han Zhonghua proposed a "Research on Intelligent Decision-Making Model Based on Reinforcement Learning DQN Algorithm," published in *Modern Computer*, Vol. 29, No. 14. This article primarily focuses on whether there are mutually reinforcing or inhibiting relationships among the three optimization factors (Dueling, Double-Q, and Prioritized-replay) of the reinforcement learning DQN algorithm. It investigates the arbitrary combination of these three optimization factors as a trading strategy, using the closing prices of HDFC Bank stock on Yahoo Finance from September 2, 2020, to September 2, 2022, as the research object. However, the article does not elaborate on the design of reward mechanisms and loss assessments related to DQN model training, which differs significantly from the design of autonomous start-stop control for unmanned radar. Summary of the Invention

[0007] To overcome the shortcomings of traditional pre-defined rules in terms of limited processing capabilities and insufficient environmental adaptability, and to achieve highly reliable and autonomous operation of unmanned radar under complex conditions, this invention provides an autonomous start-stop control decision-making method for unmanned radar, comprising the following steps:

[0008] Step S1: Establish the unmanned radar operation control state machine;

[0009] Step S2: Design a state-based reinforcement learning decision model;

[0010] Step S3: Model training and optimization evaluation based on multiple scenarios;

[0011] Step S4: Instruction execution and real-time feedback.

[0012] Preferably, step S1 includes the following steps:

[0013] Step S1-1: Design the state machine: The states include power-on, self-test, working, standby, and power-off.

[0014] Step S1-2: Design the state transitions and input / output of the state machine:

[0015] The state machine inputs include unmanned radar periodic status monitoring data, self-test data, external trigger switch control commands, and target detection data.

[0016] The outputs of a state machine include control output instructions, state transition instructions, and state control result feedback.

[0017] Preferably, step S2 includes the following steps:

[0018] Step S2-1: Design the reinforcement learning decision model:

[0019] Based on DQN as the decision-making model, the control state input information is designed. ,in Characterizing individual state monitoring parameters of a radar system A set of parameters characterizing the overall system condition monitoring;

[0020] Design a convolutional neural network with two hidden layers; apply activation functions... Functions to obtain the state output layer ;

[0021] Step S2-2: Design the reward function, target Q value, and loss function: the root mean square error is selected as the loss function.

[0022] Preferably, step S3 includes the following steps:

[0023] Step S3-1: Design training samples for multi-scenario simulation;

[0024] Step S3-2: Design the training process for the DQN model and train the state network using the memory replay mode;

[0025] Step S3-3: Evaluation of DQN model training performance.

[0026] Preferably, step S4 includes the following steps:

[0027] Real-time state data acquisition; processing of the acquired state data and updating of the state machine; DQN agent makes decisions, and the agent outputs radar start / stop control commands based on the decision results; real-time monitoring of radar operating status and control command changes, providing feedback information for updating the reinforcement learning model.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] (1) Improve decision-making adaptability in complex environments: This invention establishes an operation control state machine that includes five core states: power-on, self-test, working, standby, and power-off, as well as their state transition conditions. This enables the unmanned radar to make reasonable start-up and shutdown decisions based on preset transition logic under different working conditions, overcoming the problem of insufficient decision-making ability of traditional preset rule control methods in complex and ever-changing environments, and effectively improving the unmanned radar's ability to handle abnormal states.

[0030] (2) Achieving autonomous optimization of control strategy: This invention combines the DQN reinforcement learning model with the state machine, and uses the DQN model to output the optimal action under complex working conditions, so that the start and stop strategy of the unmanned radar can be autonomously optimized according to the changes in the environment during operation, avoiding the single dependence on the manual preset rules, and improving the intelligent control level of the unmanned radar in uncertain environments.

[0031] (3) Improve the accuracy of decision-making: This invention comprehensively considers the evaluation of the switching capabilities of the three states of power-on, self-test and operation, and constructs a multi-dimensional reward function to provide accurate feedback signals for the DQN agent, ensuring that the agent can learn the optimal decision-making strategy under different abnormal scenarios.

[0032] (4) Possesses online continuous optimization capability: This invention combines multi-scenario simulation training with real-time control feedback. During the model training stage, the model performance is evaluated and optimized through multi-dimensional indicators such as cumulative reward, loss function value, and success rate. During the real-time operation stage, the model is continuously updated through feedback information, enabling the unmanned radar to quickly adapt to environmental changes in actual operation and has good engineering application prospects. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the unmanned radar operation control state machine in an embodiment of the present invention.

[0034] Figure 2 This is the DQN reinforcement learning decision model architecture in this embodiment of the invention.

[0035] Figure 3 This is the training process based on the DQN model in this embodiment of the invention.

[0036] Figure 4 This is a flowchart of step S4 in an embodiment of the present invention. Detailed Implementation

[0037] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0038] This invention provides an autonomous start-stop control decision-making method for unmanned radar, comprising the following steps:

[0039] S1: Establish the unmanned radar operation control state machine;

[0040] S2: Design a state-based reinforcement learning decision model;

[0041] S3: Model training and optimization evaluation based on multiple scenarios;

[0042] S4: Instruction execution and real-time feedback.

[0043] Specifically, step S1 establishes the unmanned radar operation control state machine, including the following steps:

[0044] Step S1-1, design the states of the state machine:

[0045] The unmanned radar start / stop control state machine includes the following states: power on, self-test, working, standby, and power off. The definitions of each state are as follows:

[0046] Power-on status: After the radar is powered on from the off mode, it switches from the off state to the power-on mode for each device. During the automatic power-on process, the radar system collects and provides feedback status information in real time to determine whether the power-on was successful.

[0047] Self-test status: A state that the radar system enters during startup or periodic maintenance. Its main purpose is to perform a comprehensive self-test of the system's hardware and software and system status to ensure that the system functions normally.

[0048] Operating status: The radar system is in normal working condition, completing modes including target search and target tracking, and locking onto and continuously tracking the target;

[0049] Standby mode: The default state of the radar system when there are no mission commands or external trigger signals. In this state, the system maintains low power consumption and waits for new mission commands or trigger signals;

[0050] Power off state: The state of the radar system when it is not in operation. In this state, the system stops the operation of all hardware and software modules and enters a completely shut-down mode.

[0051] Step S1-2: Design the state transitions and input / output of the state machine:

[0052] State transition conditions determine the requirements for the unmanned radar to jump from one state to another, and are the key control logic of the state machine. Specific transition judgment conditions include:

[0053] Transition from power-off state to power-on state: The radar system determines whether the conditions for power-on are met based on the power supply status of the unmanned radar. If the conditions are met, the power-on operation is performed.

[0054] Power-on state transition to self-test state: The radar system monitors the power-on status information of the unmanned radar and determines whether the radar RF front-end and integrated back-end have the conditions for self-test. If they do, it enters the self-test state.

[0055] Self-test state transition to standby or power-off state: The radar system determines whether it has working conditions (such as normal uplink and downlink, and normal tracking of simulated targets) based on the self-test results of the radar front-end and back-end, and enters standby or power-off state respectively.

[0056] The radar transitions between standby and operational states: When in standby mode, it starts operating upon receiving a command, performing functions such as transmission, target search, and target tracking, until it receives a stop command and then re-enters standby mode.

[0057] Transition from standby to shutdown: After receiving a shutdown command, the radar system will execute commands such as power-off to put the radar into a shutdown state.

[0058] The state machine input data includes unmanned radar periodic status monitoring data, self-test data, external trigger switch control commands, and target detection data.

[0059] The output actions of a state machine include control output commands (such as initiating self-test control, power supply switch control, etc.), state transition commands, and state control result feedback.

[0060] Specifically, step S2 designs a state-based reinforcement learning decision model, including the following steps:

[0061] Step S2-1: Design the reinforcement learning decision model:

[0062] This embodiment uses Deep Q-Network (DQN) as the basis for the decision model, and its model architecture is referenced. Figure 2 Specifically, it includes control state input information, deep convolutional neural network training, and the output agent's possible actions and their corresponding Q-values ​​(i.e., the value of the actions). Its components are defined as follows:

[0063] Control status input information:

[0064] ,in Characterizing individual state monitoring parameters of a radar system A set of parameters characterizing the overall system condition monitoring;

[0065] A deep convolutional neural network includes a first hidden layer and a second hidden layer, wherein...

[0066] First hidden layer: ,in Given the input state matrix, To apply the activation function, This is the weight matrix. The value of the bias variable;

[0067] Second hidden layer: ,in This is the output matrix of the first hidden layer. To apply the activation function, This is the weight matrix. The value of the bias variable;

[0068] In the first and second hidden layers, activation functions are applied. use The function normalizes device status monitoring information to a range of -1 to 1 for easier, unified processing. Response function formula: .

[0069] State output layer: The final output is obtained through linear transformation. ,in This is the output matrix of the second hidden layer. This is the weight matrix. This is the value of the bias variable.

[0070] Step S2-2: Design the reward function, target Q-value, and loss function:

[0071] reward function Reflecting the actions performed by the intelligent agent In the present The reward or penalty obtained in a given state. Design a reward function. The specific form is as follows:

[0072] If the intelligent agent successfully guides the completion of the power-on process, a positive reward is given, such as adding 10 points.

[0073] If the agent successfully completes the self-check operation, it will be given a positive reward, such as adding 10 points.

[0074] If the intelligent agent successfully guides the completion of the work state operation, it is given a positive reward, such as adding 10 points;

[0075] If the intelligent agent successfully guides the device to fail the self-test, a negative reward is given, such as deducting 5 points.

[0076] If the intelligent agent fails to successfully guide the device to work, a negative reward is given, such as deducting 5 points.

[0077] The reward function comprehensively considers the evaluation of power-on status, self-test status, and operational status switching capability. The calculation formula is as follows:

[0078]

[0079] in, To complete the reward upon startup, A reward will be given for completing the self-inspection. Rewards for completing tasks. The reward weighting coefficient can be adjusted according to the actual situation.

[0080] The objective Q-value is the expected cumulative reward of the agent after choosing an action in the current state, and the specific calculation method is as follows:

[0081]

[0082] in For instant reward function, The discount factor is between 0 and 1. For the next state The maximum possible action value. Used to balance the distribution of immediate and future rewards.

[0083] Loss function: used to calculate and evaluate the model's predictions. Values ​​and Objectives The difference in values ​​is used for model parameter updates. Specifically, the root mean square error is chosen as the loss function L, and the specific formula is as follows:

[0084]

[0085] in It is the current prediction of the model. Actions in a state of value, For the goal The value is N, where N is the number of samples used in the loss calculation.

[0086] Update the weight matrix using the results of backpropagation and gradient descent. , , and bias variable values , and .

[0087] Specifically, step S3, based on multi-scenario model training and optimization evaluation, includes the following steps:

[0088] Step S3-1, Design training samples for multi-scenario simulation:

[0089] To enable the DQN model to adapt to complex and varied state conditions, this invention designs multi-scenario simulation training samples for start-stop control under different conditions, thereby improving the model's adaptability. The designed multi-scenario simulation training samples include:

[0090] The startup process, including scenarios of different subsystems failing to start, is used to train the model's ability to handle anomalies during startup, including periodic status monitoring data and external switch control command data. The self-test process, including scenarios of different subsystems failing to self-test, trains the model's ability to handle self-test anomalies, including self-test data from each subsystem. The radar's operation startup process, including scenarios of different system failures, trains the model's ability to handle system malfunctions, including periodic status data and target detection data. The shutdown process, including scenarios of subsystems failing to shut down, trains the model's ability to handle shutdown anomalies, including periodic status monitoring data.

[0091] Step S3-2, Training process based on the DQN model:

[0092] The DQN model uses a memory-replay mode to train the state network, making the neural network training more convergent and stable, and improving learning and training efficiency. See the steps below. Figure 3 The training steps based on the DQN model are as follows:

[0093] Step S3-2-1: Initialize the network and training parameters, including... Network parameters Action set Memory playback unit Number of iterations T, discount factor Number of gradient descent samples State feature dimension wait;

[0094] Step S3-2-2: Begin iterative looping from 0 to T:

[0095] (1) Initialize the first state sequence vector As Network input yields the output of all actions. Value, based on the reward function Get the corresponding action ;

[0096] (2) In Perform work in state To obtain the next state vector and instant reward function Determine whether it is a terminated state;

[0097] (3) Transfer the parameter vector Store in memory playback unit ;

[0098] (4) Take Assign to and retrieve from the memory playback unit 1 sample, calculate the current value ,in ,and:

[0099] ;

[0100] (5) Based on the root mean square error of the loss function Updated through backpropagation and gradient descent results Network weight and bias variable values ;in, It is the current prediction of the model. Actions in a state of value, For the goal The value is N, where N is the number of samples involved in the loss calculation, N=m.

[0101] (6)Judgment Is it a terminated state? If it is, the iteration ends; otherwise, jump to step (1).

[0102] Step S3-2-3, output the iteration ends. Network parameters.

[0103] Step S3-3, Evaluation of DQN model training performance:

[0104] During model parameter training, the model's performance is evaluated periodically, including metrics such as cumulative reward, loss function value, and success rate. Based on the evaluation results, the reward mechanism and training parameters are adjusted to optimize model performance.

[0105] Cumulative reward: Calculate the total reward obtained by the agent in each training cycle and perform trend analysis on the changes in cumulative reward with the number of training steps: If the cumulative reward continues to increase, it indicates that the agent is learning an effective strategy; if the increase is slow or fluctuates greatly, the parameters need to be adjusted or the training strategy optimized.

[0106] Loss function value: Analyze the trend of the loss function value after each parameter update: if the loss value gradually decreases and tends to stabilize, it indicates that the model is converging; if the loss value fluctuates greatly or decreases slowly, the learning rate needs to be adjusted or the network structure optimized.

[0107] Success rate: Set success conditions, such as terminating training if the target state is reached; if the target state is not reached, parameters need to be adjusted or strategies optimized.

[0108] Specifically, step S4 involves instruction execution and real-time feedback, see [link to relevant documentation]. Figure 4 Step S4 mainly includes the following steps:

[0109] Real-time status data acquisition includes radar push power-on, periodic BIT data (such as power supply voltage, current and on / off status, etc.), target detection data (such as target azimuth, speed, etc.), and control information from the upper-level system.

[0110] State machine update: Preprocess the periodically collected state information and update the radar's state machine based on the radar's current state (e.g., from power-on state to self-test state).

[0111] Reinforcement learning decision-making: The DQN agent makes decisions based on the current radar state and control information, that is, outputs the optimal action (such as maintaining the current state, switching to the working state, etc.).

[0112] Output command execution: Based on the decision results of the intelligent agent, the output is converted into radar start-up and stop control commands (such as starting operation, switching to standby mode, etc.).

[0113] Real-time feedback: Monitor the radar's operating status and changes in control commands in real time, and provide feedback information to facilitate the updating of the reinforcement learning model.

[0114] Stop working.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An autonomous start / stop control decision-making method for an unmanned radar, comprising the following steps: Step S1: Establish the unmanned radar operation control state machine; Step S2: Design a state-based reinforcement learning decision model; Step S3: Model training and optimization evaluation based on multiple scenarios; Step S4: Instruction execution and real-time feedback.

2. The method according to claim 1, characterized in that, Step S1 includes the following steps: Step S1-1: Design the state machine: The states include power-on, self-test, working, standby, and power-off. Step S1-2: Design the state transitions and input / output of the state machine: The state machine inputs include unmanned radar periodic status monitoring data, self-test data, external trigger switch control commands, and target detection data. The outputs of a state machine include control output instructions, state transition instructions, and state control result feedback.

3. The method according to claim 2, characterized in that, Step S2 includes the following steps: Step S2-1: Design the reinforcement learning decision model: Based on DQN as the decision-making model, the control state input information is designed. ,in Characterizing individual state monitoring parameters of a radar system A set of parameters characterizing the overall system condition monitoring; Design a convolutional neural network with two hidden layers; apply activation functions... Functions to obtain the state output layer ; Step S2-2: Design the reward function, target Q value, and loss function: the root mean square error is selected as the loss function.

4. The method according to claim 3, characterized in that, Step S3 includes the following steps: Step S3-1: Design training samples for multi-scenario simulation; Step S3-2: Design the training process for the DQN model and train the state network using the memory replay mode; Step S3-3: Evaluation of DQN model training performance.

5. The method according to claim 4, characterized in that, Step S4 includes the following steps: Real-time state data acquisition; processing of the acquired state data and updating of the state machine; DQN agent makes decisions, and the agent outputs radar start / stop control commands based on the decision results; real-time monitoring of radar operating status and control command changes, providing feedback information for updating the reinforcement learning model.

6. The method according to claim 3, characterized in that, The target Q value is the expected cumulative reward of the agent after choosing an action in the current state, and is calculated using the following formula: ,in, For instant reward function, As a discount factor, For the next state The maximum possible action value.

7. The method according to claim 6, characterized in that, The formula for calculating the loss function is as follows: ,in, It is the current prediction of the model. Actions in a state of value, For the goal The value is N, where N is the number of samples used in the loss calculation.

8. The method according to claim 4, characterized in that, The training process of the DQN model includes the following steps: Step S3-2-1: Initialize the network and training parameters, including... Network parameters Action set Memory playback unit Number of iterations T, discount factor Number of gradient descent samples State feature dimension ; Step S3-2-2, begin iterative looping from 0 to T, including the following sub-steps: Step a, initialize the first state sequence vector As Network input yields the output of all actions. Value, based on the reward function Get the corresponding action ; Step b, in Perform work in state To obtain the next state vector and instant reward function Determine whether it is a terminated state; Step c, convert the parameter vector Store in memory playback unit ; Step d, take Assign to and retrieve from the memory playback unit 1 sample, calculate the current value ,in ,and: ; Step e, based on the root mean square error of the loss function Updated through backpropagation and gradient descent results Network weight and bias variable values ;in, It is the current prediction of the model. Actions in a state of value, For the goal The value is N, where N is the number of samples involved in the loss calculation, N=m; Step f, determine Is it a terminated state? If it is, the iteration ends; otherwise, jump to step a. Step S3-2-3, output the iteration ends. Network parameters.

Images