Unmanned aerial vehicle detection linkage system based on detection peak value following method

Abstract

The invention discloses an unmanned aerial vehicle detection linkage system based on a detection peak value following method, and the system comprises a complex environment dynamic adaptation module, a target tracking precision module, a big data processing and analysis module, and an autonomous decision optimization module. The system not only can perform efficient and accurate investigation in a complex environment, but also has strong autonomous operation capability, so that the unmanned aerial vehicle can accurately grasp environment information and provide more accurate information when tracking a target, thereby further improving the investigation effect of the unmanned aerial vehicle in the complex environment.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned aerial vehicles, and in particular to an unmanned aerial vehicle detection and intervention linkage system based on a detection peak following method. Background Art

[0002] The current design of UAV reconnaissance systems is mainly based on some standard or average environmental parameters. However, in actual application scenarios, the complexity and dynamics of the environment often exceed the design expectations, which makes the UAV's operating performance and reconnaissance effect fail to meet expectations; for example, UAVs need to respond and adjust quickly in weather changes, terrain fluctuations, and emergency situations;

[0003] Target tracking accuracy: In current drone applications, how to accurately focus and track targets is a major problem. Generally speaking, the real-time acquisition of data such as the moving speed and direction of remote targets and the adjustment of their own flight status is a very complex dynamic optimization problem, which requires accurate judgment and decision-making in a very short time.

[0004] Data processing and analysis issues: How to accurately find useful information from the large amount of data collected by sensors and conduct real-time processing and decision-making is a major technical challenge at present. Especially when dealing with complex backgrounds or weak signals, how to improve the efficiency of data processing and reduce the risk of misjudgment are important technical issues;

[0005] System autonomy: Currently, most UAV reconnaissance systems rely on remote manual control. In some extreme application environments or tasks, how to improve the autonomy of the system, reduce manual intervention, and improve the performance and stability of the system is an important technical challenge; Summary of the invention

[0006] In view of this, the present invention proposes a UAV detection and intervention linkage system based on the detection peak following method, which can better solve the above-mentioned emerging problems; firstly, the detection peak following method can effectively improve the accuracy of target tracking, reduce interference, and reduce misjudgment; secondly, the system combines multiple sensors and uses advanced data processing technology to achieve accurate target detection and intervention in complex environments; thirdly, by optimizing the flight strategy of the UAV and improving the autonomy of the system, the UAV can perform well in challenging tasks; therefore, the system will bring important improvements in the quality and stability of UAV detection and intervention technology;

[0007] It includes a complex environment dynamic adaptation module, a target tracking accuracy module, a big data processing and analysis module, and an autonomous decision-making module;

[0008] Complex environment dynamic adaptation module

[0009] Our goal is to enable the drone to dynamically adapt its hardware system in various environments according to environmental changes. We have added multimodal sensors to the drone, such as infrared sensors, pulse radars, and high-precision positioning devices. At the same time, we have also developed a new type of 3D mapping device that can perform 3D scanning and generate 3D environmental models during the flight of the drone, helping the drone to better perceive the environment.

[0010] Through deep convolutional neural networks (DCNN) and reinforcement learning (RL) algorithms, a large amount of historical environmental data is autonomously learned to train drones to identify the environment and make the best adaptive decisions. In addition, we also introduced the drone's flight control system and airborne neural computing unit to provide the drone with real-time, autonomous decision-making capabilities;

[0011] 1. Environmental perception module:

[0012] The environmental perception module uses the sensors on board the drone to obtain environmental information around the drone, including but not limited to weather conditions, topography, air turbulence intensity, and electromagnetic signal intensity;

[0013] In order to obtain more comprehensive and higher-quality environmental information, we have equipped the drone with the following types of sensors: infrared sensors, radars, gravity sensors, meteorological sensors, electromagnetic signal sensors, etc. For example, infrared sensors can obtain surrounding thermal radiation information and estimate the ambient temperature; gravity sensors can sense the flight altitude; radars can obtain terrain information in real time and detect obstacles; meteorological sensors can detect wind direction, wind speed and other weather conditions in real time; electromagnetic signal sensors can collect surrounding electromagnetic signals;

[0014] 2. Data preprocessing module:

[0015] Data preprocessing is used to filter out noise, remove outliers, and remove redundant data. In actual operation, we use several methods to preprocess data:

[0016] Filtering: Use some special filters, including but not limited to low-pass filters, high-pass filters, and band-pass filters to remove unnecessary noise;

[0017] Standardization: Standardize the data, eliminate the dimension, make the data at the same order of magnitude, reduce the impact of outliers, and make it easier to observe the laws of the data;

[0018] Gap data filling: For data with gaps, such as those caused by sensor failure or data loss, gaps can be filled by interpolation, averaging, maximum probability value, etc.

[0019] 3. Feature extraction module:

[0020] The main task of the feature extraction module is to find features that have an impact on the target task from the preprocessed data, that is, those that characterize environmental changes and help the drone make decisions;

[0021] In specific operations, we adopt the following methods:

[0022] Parametric statistics: For example, calculating basic statistics of various data, such as mean, variance, skewness, kurtosis, etc. These statistics can often reflect the overall trend and distribution characteristics of the data;

[0023] Feature selection method: For high-dimensional data, we can also use some feature selection methods, such as univariate feature selection, recursive feature elimination, model-based feature selection, etc., to select the most valuable features;

[0024] 4. Adaptive control module:

[0025] The adaptive control module is the core of the entire complex environment dynamic adaptation module. Its main responsibility is to enable the UAV to actively adapt to environmental changes and make optimized decisions through learning and optimization. Here, we use deep reinforcement learning to achieve adaptive control of the UAV. Deep reinforcement learning combines the representation learning ability of deep learning and the decision optimization ability of reinforcement learning. It usually uses neural networks as function approximators to estimate environmental models or value functions, and continuously optimizes the model through feedback information in the interaction process to guide the UAV to make dynamic adaptive decisions.

[0026] The above four steps form an interlocking data flow. First, the environmental perception module collects environmental information around the drone in real time. Then, the data preprocessing module preprocesses the collected raw data to improve the data quality. Then, the feature extraction module extracts useful features from the preprocessed data. Finally, the adaptive control module uses deep reinforcement learning to enable the drone to learn and make decisions about environmental adaptation. These four links run simultaneously, allowing the drone to achieve continuous dynamic adaptation to complex environments during flight.

[0027] Directly learn the optimal policy in a model-free environment; when we introduce a neural network (NN) to approximate the Q function (denoted as Q(s, a; θ)), this forms a deep Q network (DQN); in DQN, the parameters θ of the neural network are iteratively updated through the learning process;

[0028] During each iteration (time step = t), the network performs the following steps:

[0029] According to the current state s_t, select an action a_t; generally adopt the ε-greedy strategy, that is, randomly select an action with a probability of ε, and select the action with the largest Q value with a probability of (1-ε);

[0030] Execute action a_t and obtain the next state s_(t+1) and reward r_t from the environment;

[0031] Update Q value; in traditional Q-learning, the Q value update formula is: Q(s,a)←r+γmax_aQ(s',a), where γ is a discount factor used to control the impact of future rewards;

[0032] For DQN, we transform the Q value update formula into the form of a loss function, namely: L(θ) = E[(r+γmax_aQ(s',a;θ-)-Q(s,a;θ))^2], where E[] represents the expectation, θ- represents the parameters of the target network, and θ represents the parameters of the current network;

[0033] Optimize the loss function by stochastic gradient descent (SGD) or other optimization algorithms, and gradually adjust the value of θ;

[0034] in:

[0035] s represents the current state, such as the drone’s geographic location, altitude, speed, etc.;

[0036] a represents an action, such as the flight direction and speed change of the drone;

[0037] θ represents the weight parameter of the neural network;

[0038] s' represents the new state after executing action a;

[0039] r represents the immediate reward obtained after executing action a;

[0040] Obtain initial state input s_1 from the drone’s posture sensor, image sensor, radar and other modalities to obtain preliminary information about the environment;

[0041] For each step t, in state s_t, an action a_t is selected according to the ε-greedy strategy to execute; as learning proceeds, ε may gradually decrease, making future action decisions more dependent on the output of the neural network;

[0042] For executing action a_t, the drone will receive a reward r_t from the environment and move to a new state s_(t+1); the reward r can be designed as a function related to the mission goal and the drone's flight performance;

[0043] Store the transfer tuple (s_t, a_t, r_t, s_(t+1)) in the experience replay memory; this memory is used to store past transfer tuples, which helps to break the association between data and make training more stable;

[0044] A batch of samples are randomly drawn from the experience replay memory and used to update the parameters θ of the neural network; the update process is carried out by optimizing the objective function (i.e., minimizing the mean square error between the predicted Q value and the target Q value); the target Q value is obtained by the target network;

[0045] Every certain steps, the parameters θ of the current network are used to update the parameters θ' of the target network;

[0046] If the task is successfully completed or the maximum number of steps is reached, the current round ends and then returns to step 2 to start a new round;

[0047] After many rounds of learning, the drone’s flight strategy will continue to improve, and the drone will be able to make more intelligent action decisions based on the state of the environment;

[0048] In summary, the drone adjusts its action strategy, i.e. the parameter θ of the neural network, by learning the combination of a series of past states, actions, and rewards and their results, so as to achieve better performance in future environments. This is the application of deep reinforcement learning in drone decision-making systems.

[0049] 2. Target tracking accuracy module optimization:

[0050] In order to improve the detection accuracy of the UAV, we added a target tracking system based on detection peak following to the UAV; this system uses an advanced Kalman filter to track the target and adaptively changes the flight parameters of the UAV (such as speed and direction) to attract the target; in addition, we also added an adaptive noise suppression module to reduce the impact of environmental noise on the detection results, ensuring that the UAV can track the target stably and accurately;

[0051] The working principle of the detection peak following system is to detect the peak value in the signal (i.e. the point with the largest signal strength), and use it as the detection target, and then track this peak point; this process usually involves filtering algorithms and optimization methods; this system uses the application of Kalman filter in this, and introduces its workflow:

[0052] In general, the standard form of the Kalman filter can be expressed as:

[0053] 1) Prediction step:

[0054] x^-(k)=Ax(k-1)+Bu(k-1)

[0055] P^-(k)=AP(k-1)A^T+Q

[0056] in:

[0057] x is the system state vector, which may include position, velocity, and other information in drone tracking, for example;

[0058] A is the transfer matrix, which describes how the system moves from the previous state to the next state;

[0059] B is the control matrix, which describes how the control input affects the system state;

[0060] u is the control input;

[0061] P is the estimation error covariance matrix;

[0062] Q is the process noise covariance matrix;

[0063] '^-' indicates the predicted next state;

[0064] 2) Update steps:

[0065] K(k)=P^-(k)H^T(HP^-(k)H^T+R)^-1

[0066] x(k)=x^-(k)+K(k)(z(k)-Hx^-(k))

[0067] P(k)=(IK(k)H)P^-(k)

[0068] in:

[0069] K is the Kalman gain, which is used to measure the contribution of prediction and measurement results to the final state estimation;

[0070] z is the actual measured value;

[0071] H is the observation matrix, which describes how to obtain the predicted observation value from the system state vector;

[0072] R is the measurement noise covariance matrix;

[0073] I is the unit matrix;

[0074] For a drone, if it wants to track the movement of a target, it needs to change its flight parameters such as speed and direction in real time. This is usually achieved by introducing a PID controller, which processes the gap between the target and the current one:

[0075] The basic formula for a PID controller is as follows:

[0076] u(t)=Kpe(t)+Ki∫e(t)dt+Kd*de(t) / dt

[0077] in:

[0078] u(t) is the control quantity, i.e. the action that the drone needs to perform;

[0079] e(t) is the error, usually obtained by the difference between the target state and the current state;

[0080] Kp, Ki, Kd are proportional, integral, and differential coefficients, which can be adjusted through experiments;

[0081] ∫e(t)dt is the integral of the error, which can be understood as the accumulated error in the past;

[0082] de(t) / dt is the differential of the error, which can be understood as the expected future error;

[0083] In addition, in order to use the PID controller, we need to set limits on the flight environment and various states of the target (such as maximum and minimum speed, acceleration, angular velocity, etc.); a reasonable limit can effectively prevent the drone from making excessive movements during flight;

[0084] This module can significantly improve the performance of drones in reconnaissance missions by accurately finding and tracking targets in complex environments.

[0085] In practical applications, these parameters need to be set according to the characteristics of the actual system and environmental conditions. In a drone reconnaissance system, for example, the system state vector x may include the position, velocity, acceleration, etc. of the drone; the control input u may include acceleration instructions in the vertical and horizontal directions, etc.; the observation value z may be the actual position of the drone;

[0086] 3. Big data processing and analysis module optimization:

[0087] We have comprehensively upgraded the traditional data processing architecture and adopted big data processing methods, including distributed storage and parallel computing, to improve the system's data processing capabilities. At the same time, we have also introduced deep learning algorithms:

[0088] 1. Data Preprocessing

[0089] Before entering the model training phase, the first thing to do is data preprocessing. Image data usually contains a lot of redundant and irrelevant information. We need to use a series of preprocessing methods to clean and optimize the image. The preprocessing steps usually include:

[0090] Resizing: Since the size of the original images may vary greatly, we need to resize all images to the same size to facilitate model processing;

[0091] Color space conversion: Although RGB color space is the most common, not every model needs to use RGB color space. Some models may perform better in HSV or grayscale color space.

[0092] Data normalization: We scale the numerical range of image data, usually to between 0-1 or between -1 and 1; this can prevent numerical instability in the model during training;

[0093] Data enhancement: We use transformations such as translation, rotation, flipping, and scaling to generate more training samples to enhance the generalization ability of the model;

[0094] These preprocessing operations greatly reduce the redundancy of the original data and ensure that the model can be trained efficiently under optimal conditions;

[0095] 2. Building a DCNN Model

[0096] The task of building a DCNN model is to determine the number of layers, the type and parameters of each layer of the model; a typical DCNN model consists of multiple convolutional layers, activation layers, pooling layers and fully connected layers;

[0097] Convolutional layer: The convolutional layer performs sliding window convolution operations on the input image through convolution kernels, which can extract local spatial features of the image while keeping the image size unchanged. We can adjust the model's ability to learn features by controlling the number and size of convolution kernels.

[0098] Activation layer: The activation layer performs nonlinear mapping on the output of the convolutional layer, introducing the model's nonlinear transformation capability, which is the key to the model's learning of complex patterns. The ReLU activation function is widely used in deep learning models because of its simple calculation and its ability to effectively alleviate the gradient vanishing problem.

[0099] Pooling layer: The pooling layer downsamples the input image, reduces the spatial dimension of the data, reduces model parameters and computation, prevents overfitting, and improves computational efficiency. Common pooling methods include maximum pooling, average pooling, etc.

[0100] Fully connected layer: The fully connected layer converts the two-dimensional feature map of the previous layer into a one-dimensional form to achieve global connection of features; the essence of the fully connected layer is a linear transformation, which corresponds to the last few layers in the neural network model. The purpose is to map the learned "distributed feature representation" to the label space of the sample;

[0101] 3. Training and Optimizing DCNN Models

[0102] After the model is built, the next step is to start model training. The training process is to continuously adjust the weight parameters in the model through optimization algorithms such as back propagation and gradient descent to minimize the error of the model on the training data. Common optimizers include SGD, Adam, Adagrad, etc.

[0103] During the training process, we input the training data into the model in batches. For each batch of data, the model will perform one forward propagation and one back propagation. Forward propagation means that the model calculates the output of each layer based on the current parameters and input data, and finally obtains the prediction result. Back propagation means that the model calculates the prediction error based on the prediction result of forward propagation and the true label, and then propagates this error back to each layer through the chain rule to update the parameters of each layer.

[0104] During the training process, a common technique is to use a validation set to validate the model. We will divide the original training data into two parts, one as the training set and the other as the validation set. After each training cycle (epoch), we will use the performance of the model on the validation set to adjust the model's hyperparameters. Here, we will detail the complete process of how to use DCNN for image recognition in practical applications.

[0105] 1. Data preprocessing

[0106] Before we start training a deep neural network, we usually need to perform a series of preprocessing steps on the raw image data to filter out irrelevant information, enhance the expression of target information, and improve the efficiency of model training.

[0107] First, the original images often have different sizes. In order to ensure the consistency of input when training the network model, we need to adjust all images to a uniform size. Then, we may need to convert the image to a color space, such as converting the RGB color space to the Lab color space that is more in line with human eye perception, or converting the color image to a grayscale image, etc., to reduce computational complexity. Next, we usually normalize the image, which can limit the pixel value of the image to a certain range and reduce the numerical instability during model training. Finally, we may perform some data enhancement operations, such as rotation, flipping, scaling, etc., to enrich data diversity and improve the generalization ability of the model.

[0108] 2. Build DCNN model

[0109] A typical DCNN model consists of multiple layers of convolutional layers, activation layers, pooling layers, and fully connected layers arranged alternately. The basic computing unit of DCNN is a neuron. Each neuron receives multiple inputs, then weights and sums these inputs and passes them through an activation function to get the output.

[0110] Convolutional layer: The convolutional layer is the core of DCNN. Its main function is to effectively extract local image features. Through the convolution operation, we can analyze the entire image with a small number of parameters (convolution kernels), and no matter where the target is in the image, as long as the same features appear, they can be captured by the same convolution kernel.

[0111] Activation layer: The main function of the activation layer is to introduce nonlinear factors, because only by adding nonlinear factors can deep neural networks build complex models; if there is no nonlinearity, no matter how many layers the neural network has, its final effect is no different from that of a single-layer neural network; the most commonly used activation function is the ReLU function;

[0112] Pooling layer: The pooling layer is also called the downsampling layer or subsampling layer. Its main function is to reduce the number of model parameters by reducing the dimension of the data to prevent overfitting. At the same time, it can also retain important features, making the model stable to small displacements, rotations, and scaling.

[0113] Fully connected layer: A fully connected layer is usually added at the end of the convolutional neural network to map the learned distributed feature representation to the sample label, perform large-scale feature integration and information extraction, and achieve the final classification of the sample;

[0114] 3. Training the DCNN model

[0115] The training process usually uses gradient descent or stochastic gradient descent to minimize the network loss function by optimizing the network strength (weights and biases) to achieve the purpose of network training. There are many types of loss functions, the most common ones are mean squared error (MSE) and cross entropy. In fact, the training process is essentially a continuous optimization process, constantly adjusting the weight parameters to minimize the value of the loss function.

[0116] 4. Apply DCNN model for image recognition

[0117] After the DCNN model is trained, we can use it in actual image recognition tasks. By inputting the image to be recognized into the model and performing forward propagation calculations, the model will give the recognition result of the input image based on the trained parameters. Generally speaking, the softmax function is used in the output layer so that the output value is between 0 and 1, which is equivalent to the classification probability. The classification result with the highest probability is taken as the final result.

[0118] The above is the application process of deep convolutional neural network in image recognition. Not only image recognition, DCNN is also widely used in many fields such as speech recognition and natural language processing. Its powerful feature learning and representation capabilities provide a powerful tool for the development of deep learning.

[0119] The beneficial effects of the present invention include:

[0120] We have built a new type of drone reconnaissance and joint system. By effectively integrating hardware equipment, algorithm models and control strategies, the system can not only perform efficient and accurate reconnaissance in complex environments, but also has strong autonomous operation capabilities, allowing drones to accurately grasp environmental information when tracking targets and provide more accurate intelligence, thereby further improving the reconnaissance effect of drones in complex environments. DETAILED DESCRIPTION

[0121] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0122] The present invention relates to a UAV detection and intervention linkage system based on a detection peak following method, which specifically includes a complex environment dynamic adaptation module, a target tracking accuracy module, a big data processing and analysis module, and an autonomous decision-making module;

[0123] Complex environment dynamic adaptation module

[0124] Our goal is to enable the drone to dynamically adapt its hardware system in various environments according to environmental changes. We have added multimodal sensors to the drone, such as infrared sensors, pulse radars, and high-precision positioning devices. At the same time, we have also developed a new type of 3D mapping device that can perform 3D scanning and generate 3D environmental models during the flight of the drone, helping the drone to better perceive the environment.

[0125] Through deep convolutional neural networks (DCNN) and reinforcement learning (RL) algorithms, a large amount of historical environmental data is autonomously learned to train drones to identify the environment and make the best adaptive decisions. In addition, we also introduced the drone's flight control system and airborne neural computing unit to provide the drone with real-time, autonomous decision-making capabilities;

[0126] 1. Environmental perception module:

[0127] The environmental perception module uses the sensors on board the drone to obtain environmental information around the drone, including but not limited to weather conditions, topography, air turbulence intensity, and electromagnetic signal intensity;

[0128] In order to obtain more comprehensive and higher-quality environmental information, we have equipped the drone with the following types of sensors: infrared sensors, radars, gravity sensors, meteorological sensors, electromagnetic signal sensors, etc. For example, infrared sensors can obtain surrounding thermal radiation information and estimate the ambient temperature; gravity sensors can sense the flight altitude; radars can obtain terrain information in real time and detect obstacles; meteorological sensors can detect wind direction, wind speed and other weather conditions in real time; electromagnetic signal sensors can collect surrounding electromagnetic signals;

[0129] 2. Data preprocessing module:

[0130] Data preprocessing is used to filter out noise, remove outliers, and remove redundant data. In actual operation, we use several methods to preprocess data:

[0131] Filtering: Use some special filters, including but not limited to low-pass filters, high-pass filters, and band-pass filters to remove unnecessary noise;

[0132] Standardization: Standardize the data, eliminate the dimension, make the data at the same order of magnitude, reduce the impact of outliers, and make it easier to observe the laws of the data;

[0133] Gap data filling: For data with gaps, such as those caused by sensor failure or data loss, gaps can be filled by interpolation, averaging, maximum probability value, etc.

[0134] 3. Feature extraction module:

[0135] The main task of the feature extraction module is to find features that have an impact on the target task from the preprocessed data, that is, those that characterize environmental changes and help the drone make decisions;

[0136] In specific operations, we adopt the following methods:

[0137] Parametric statistics: For example, calculating basic statistics of various data, such as mean, variance, skewness, kurtosis, etc. These statistics can often reflect the overall trend and distribution characteristics of the data;

[0138] Feature selection method: For high-dimensional data, we can also use some feature selection methods, such as univariate feature selection, recursive feature elimination, model-based feature selection, etc., to select the most valuable features;

[0139] 4. Adaptive control module:

[0140] The adaptive control module is the core of the entire complex environment dynamic adaptation module. Its main responsibility is to enable the UAV to actively adapt to environmental changes and make optimized decisions through learning and optimization. Here, we use deep reinforcement learning to achieve adaptive control of the UAV. Deep reinforcement learning combines the representation learning ability of deep learning and the decision optimization ability of reinforcement learning. It usually uses neural networks as function approximators to estimate environmental models or value functions, and continuously optimizes the model through feedback information in the interaction process to guide the UAV to make dynamic adaptive decisions.

[0141] The above four steps form an interlocking data flow. First, the environmental perception module collects environmental information around the drone in real time. Then, the data preprocessing module preprocesses the collected raw data to improve the data quality. Then, the feature extraction module extracts useful features from the preprocessed data. Finally, the adaptive control module uses deep reinforcement learning to enable the drone to learn and make decisions about environmental adaptation. These four links run simultaneously, allowing the drone to achieve continuous dynamic adaptation to complex environments during flight.

[0142] Directly learn the optimal policy in a model-free environment; when we introduce a neural network (NN) to approximate the Q function (denoted as Q(s, a; θ)), this forms a deep Q network (DQN); in DQN, the parameters θ of the neural network are iteratively updated through the learning process;

[0143] During each iteration (time step = t), the network performs the following steps:

[0144] According to the current state s_t, select an action a_t; generally adopt the ε-greedy strategy, that is, randomly select an action with a probability of ε, and select the action with the largest Q value with a probability of (1-ε);

[0145] Execute action a_t and obtain the next state s_(t+1) and reward r_t from the environment;

[0146] Update Q value; in traditional Q-learning, the Q value update formula is: Q(s,a)←r+γmax_aQ(s',a), where γ is a discount factor used to control the impact of future rewards;

[0147] For DQN, we transform the Q value update formula into the form of a loss function, namely: L(θ) = E[(r+γmax_aQ(s',a;θ-)-Q(s,a;θ))^2], where E[] represents the expectation, θ- represents the parameters of the target network, and θ represents the parameters of the current network;

[0148] Optimize the loss function by stochastic gradient descent (SGD) or other optimization algorithms, and gradually adjust the value of θ;

[0149] in:

[0150] s represents the current state, such as the drone’s geographic location, altitude, speed, etc.;

[0151] a represents an action, such as the flight direction and speed change of the drone;

[0152] θ represents the weight parameter of the neural network;

[0153] s' represents the new state after executing action a;

[0154] r represents the immediate reward obtained after executing action a;

[0155] Obtain initial state input s_1 from the drone’s posture sensor, image sensor, radar and other modalities to obtain preliminary information about the environment;

[0156] For each step t, in state s_t, an action a_t is selected according to the ε-greedy strategy to execute; as learning proceeds, ε may gradually decrease, making future action decisions more dependent on the output of the neural network;

[0157] For executing action a_t, the drone will receive a reward r_t from the environment and move to a new state s_(t+1); the reward r can be designed as a function related to the mission goal and the drone's flight performance;

[0158] Store the transfer tuple (s_t, a_t, r_t, s_(t+1)) in the experience replay memory; this memory is used to store past transfer tuples, which helps to break the association between data and make training more stable;

[0159] A batch of samples are randomly drawn from the experience replay memory and used to update the parameters θ of the neural network; the update process is carried out by optimizing the objective function (i.e., minimizing the mean square error between the predicted Q value and the target Q value); the target Q value is obtained by the target network;

[0160] Every certain steps, the parameters θ of the current network are used to update the parameters θ' of the target network;

[0161] If the task is successfully completed or the maximum number of steps is reached, the current round ends and then returns to step 2 to start a new round;

[0162] After many rounds of learning, the drone’s flight strategy will continue to improve, and the drone will be able to make more intelligent action decisions based on the state of the environment;

[0163] In summary, the drone adjusts its action strategy, i.e. the parameter θ of the neural network, by learning the combination of a series of past states, actions, and rewards and their results, so as to achieve better performance in future environments. This is the application of deep reinforcement learning in drone decision-making systems.

[0164] 3. Target tracking accuracy module optimization:

[0165] In order to improve the detection accuracy of the UAV, we added a target tracking system based on detection peak following to the UAV; this system uses an advanced Kalman filter to track the target and adaptively changes the flight parameters of the UAV (such as speed and direction) to attract the target; in addition, we also added an adaptive noise suppression module to reduce the impact of environmental noise on the detection results, ensuring that the UAV can track the target stably and accurately;

[0166] The working principle of the detection peak following system is to detect the peak value in the signal (i.e. the point with the largest signal strength), and use it as the detection target, and then track this peak point; this process usually involves filtering algorithms and optimization methods; this system uses the application of Kalman filter in this, and introduces its workflow:

[0167] In general, the standard form of the Kalman filter can be expressed as:

[0168] 1) Prediction step:

[0169] x^-(k)=Ax(k-1)+Bu(k-1)

[0170] P^-(k)=AP(k-1)A^T+Q

[0171] in:

[0172] x is the system state vector, which may include position, velocity, and other information in drone tracking, for example;

[0173] A is the transfer matrix, which describes how the system moves from the previous state to the next state;

[0174] B is the control matrix, which describes how the control input affects the system state;

[0175] u is the control input;

[0176] P is the estimation error covariance matrix;

[0177] Q is the process noise covariance matrix;

[0178] '^-' indicates the predicted next state;

[0179] 2) Update steps:

[0180] K(k)=P^-(k)H^T(HP^-(k)H^T+R)^-1

[0181] x(k)=x^-(k)+K(k)(z(k)-Hx^-(k))

[0182] P(k)=(IK(k)H)P^-(k)

[0183] in:

[0184] K is the Kalman gain, which is used to measure the contribution of prediction and measurement results to the final state estimation;

[0185] z is the actual measured value;

[0186] H is the observation matrix, which describes how to obtain the predicted observation value from the system state vector;

[0187] R is the measurement noise covariance matrix;

[0188] I is the unit matrix;

[0189] For a drone, if it wants to track the movement of a target, it needs to change its flight parameters such as speed and direction in real time. This is usually achieved by introducing a PID controller, which processes the gap between the target and the current one:

[0190] The basic formula for a PID controller is as follows:

[0191] u(t)=Kpe(t)+Ki∫e(t)dt+Kd*de(t) / dt

[0192] in:

[0193] u(t) is the control quantity, i.e. the action that the drone needs to perform;

[0194] e(t) is the error, usually obtained by the difference between the target state and the current state;

[0195] Kp, Ki, Kd are proportional, integral, and differential coefficients, which can be adjusted through experiments;

[0196] ∫e(t)dt is the integral of the error, which can be understood as the accumulated error in the past;

[0197] de(t) / dt is the differential of the error, which can be understood as the expected future error;

[0198] In addition, in order to use the PID controller, we need to set limits on the flight environment and various states of the target (such as maximum and minimum speed, acceleration, angular velocity, etc.); a reasonable limit can effectively prevent the drone from making excessive movements during flight;

[0199] This module can significantly improve the performance of drones in reconnaissance missions by accurately finding and tracking targets in complex environments.

[0200] In practical applications, these parameters need to be set according to the characteristics of the actual system and environmental conditions. In a drone reconnaissance system, for example, the system state vector x may include the position, velocity, acceleration, etc. of the drone; the control input u may include acceleration instructions in the vertical and horizontal directions, etc.; the observation value z may be the actual position of the drone;

[0201] 3. Big data processing and analysis module optimization:

[0202] We have comprehensively upgraded the traditional data processing architecture and adopted big data processing methods, including distributed storage and parallel computing, to improve the system's data processing capabilities. At the same time, we have also introduced deep learning algorithms:

[0203] 1. Data Preprocessing

[0204] Before entering the model training phase, the first thing to do is data preprocessing. Image data usually contains a lot of redundant and irrelevant information. We need to use a series of preprocessing methods to clean and optimize the image. The preprocessing steps usually include:

[0205] Resizing: Since the size of the original images may vary greatly, we need to resize all images to the same size to facilitate model processing;

[0206] Color space conversion: Although RGB color space is the most common, not every model needs to use RGB color space. Some models may perform better in HSV or grayscale color space.

[0207] Data normalization: We scale the numerical range of image data, usually to between 0-1 or between -1 and 1; this can prevent numerical instability in the model during training;

[0208] Data enhancement: We use transformations such as translation, rotation, flipping, and scaling to generate more training samples to enhance the generalization ability of the model;

[0209] These preprocessing operations greatly reduce the redundancy of the original data and ensure that the model can be trained efficiently under optimal conditions;

[0210] 2. Building a DCNN Model

[0211] The task of building a DCNN model is to determine the number of layers, the type and parameters of each layer of the model; a typical DCNN model consists of multiple convolutional layers, activation layers, pooling layers and fully connected layers;

[0212] Convolutional layer: The convolutional layer performs sliding window convolution operations on the input image through convolution kernels, which can extract local spatial features of the image while keeping the image size unchanged. We can adjust the model's ability to learn features by controlling the number and size of convolution kernels.

[0213] Activation layer: The activation layer performs nonlinear mapping on the output of the convolutional layer, introducing the model's nonlinear transformation capability, which is the key to the model's learning of complex patterns. The ReLU activation function is widely used in deep learning models because of its simple calculation and its ability to effectively alleviate the gradient vanishing problem.

[0214] Pooling layer: The pooling layer downsamples the input image, reduces the spatial dimension of the data, reduces model parameters and computation, prevents overfitting, and improves computational efficiency. Common pooling methods include maximum pooling, average pooling, etc.

[0215] Fully connected layer: The fully connected layer converts the two-dimensional feature map of the previous layer into a one-dimensional form to achieve global connection of features; the essence of the fully connected layer is a linear transformation, which corresponds to the last few layers in the neural network model. The purpose is to map the learned "distributed feature representation" to the label space of the sample;

[0216] 3. Training and Optimizing DCNN Models

[0217] After the model is built, the next step is to start model training. The training process is to continuously adjust the weight parameters in the model through optimization algorithms such as back propagation and gradient descent to minimize the error of the model on the training data. Common optimizers include SGD, Adam, Adagrad, etc.

[0218] During the training process, we input the training data into the model in batches. For each batch of data, the model will perform one forward propagation and one back propagation. Forward propagation means that the model calculates the output of each layer based on the current parameters and input data, and finally obtains the prediction result. Back propagation means that the model calculates the prediction error based on the prediction result of forward propagation and the true label, and then propagates this error back to each layer through the chain rule to update the parameters of each layer.

[0219] During the training process, a common technique is to use a validation set to validate the model. We will divide the original training data into two parts, one as the training set and the other as the validation set. After each training cycle (epoch), we will use the performance of the model on the validation set to adjust the model's hyperparameters. Here, we will detail the complete process of how to use DCNN for image recognition in practical applications.

[0220] 1. Data preprocessing

[0221] Before we start training a deep neural network, we usually need to perform a series of preprocessing steps on the raw image data to filter out irrelevant information, enhance the expression of target information, and improve the efficiency of model training.

[0222] First, the original images often have different sizes. In order to ensure the consistency of input when training the network model, we need to adjust all images to a uniform size. Then, we may need to convert the image to a color space, such as converting the RGB color space to the Lab color space that is more in line with human eye perception, or converting the color image to a grayscale image, etc., to reduce computational complexity. Next, we usually normalize the image, which can limit the pixel value of the image to a certain range and reduce the numerical instability during model training. Finally, we may perform some data enhancement operations, such as rotation, flipping, scaling, etc., to enrich data diversity and improve the generalization ability of the model.

[0223] 2. Build DCNN model

[0224] A typical DCNN model consists of multiple layers of convolutional layers, activation layers, pooling layers, and fully connected layers arranged alternately. The basic computing unit of DCNN is a neuron. Each neuron receives multiple inputs, then weights and sums these inputs and passes them through an activation function to get the output.

[0225] Convolutional layer: The convolutional layer is the core of DCNN. Its main function is to effectively extract local image features. Through the convolution operation, we can analyze the entire image with a small number of parameters (convolution kernels), and no matter where the target is in the image, as long as the same features appear, they can be captured by the same convolution kernel.

[0226] Activation layer: The main function of the activation layer is to introduce nonlinear factors, because only by adding nonlinear factors can deep neural networks build complex models; if there is no nonlinearity, no matter how many layers the neural network has, its final effect is no different from that of a single-layer neural network; the most commonly used activation function is the ReLU function;

[0227] Pooling layer: The pooling layer is also called the downsampling layer or subsampling layer. Its main function is to reduce the number of model parameters by reducing the dimension of the data to prevent overfitting. At the same time, it can also retain important features, making the model stable to small displacements, rotations, and scaling.

[0228] Fully connected layer: A fully connected layer is usually added at the end of the convolutional neural network to map the learned distributed feature representation to the sample label, perform large-scale feature integration and information extraction, and achieve the final classification of the sample;

[0229] 3. Training the DCNN model

[0230] The training process usually uses gradient descent or stochastic gradient descent to minimize the network loss function by optimizing the network strength (weights and biases) to achieve the purpose of network training. There are many types of loss functions, the most common ones are mean squared error (MSE) and cross entropy. In fact, the training process is essentially a continuous optimization process, constantly adjusting the weight parameters to minimize the value of the loss function.

[0231] 4. Apply DCNN model for image recognition

[0232] After the DCNN model is trained, we can use it in actual image recognition tasks. By inputting the image to be recognized into the model and performing forward propagation calculations, the model will give the recognition result of the input image based on the trained parameters. Generally speaking, the softmax function is used in the output layer so that the output value is between 0 and 1, which is equivalent to the classification probability. The classification result with the highest probability is taken as the final result.

[0233] The above is the application process of deep convolutional neural network in image recognition. Not only image recognition, DCNN is also widely used in many fields such as speech recognition and natural language processing. Its powerful feature learning and representation capabilities provide a powerful tool for the development of deep learning.

[0234] The beneficial effects of the present invention include:

[0235] We have built a new type of drone reconnaissance and joint system. By effectively integrating hardware equipment, algorithm models and control strategies, the system can not only perform efficient and accurate reconnaissance in complex environments, but also has strong autonomous operation capabilities, allowing drones to accurately grasp environmental information when tracking targets and provide more accurate intelligence, thereby further improving the reconnaissance effect of drones in complex environments.

[0236] 1. Environmental simulation

[0237] In actual operation, we first need to build a simulation environment, which should reflect the real situation that the drone may encounter when flying as much as possible. For example, this environment may contain various terrains, weather conditions, objects (such as buildings, other drones and birds, etc.), and these factors may have an impact on the flight of the drone.

[0238] 2. Define states, actions, and rewards

[0239] Next, we need to define the state, action, and reward of the drone. These three factors determine the basic framework of the reinforcement learning process. In the scenario of drone decision-making, the state may include the drone's current position, speed, acceleration, fuel level, as well as information such as the target position and various environmental factors; the action includes various control operations of the drone, such as ascending, descending, turning, accelerating, decelerating, and may even include special actions such as taking pictures and recording videos; the reward is what we use to measure the effectiveness of the drone after performing a certain action. For example, if the drone successfully reaches the target position or takes high-quality photos, it should be given a positive reward, and if the drone hits an obstacle or moves farther and farther away from the target position, it should be given a negative reward.

[0240] 3. Choose the appropriate neural network structure and training algorithm

[0241] The key to DRL lies in how to effectively combine deep learning and reinforcement learning. A common choice is to use a deep neural network (DNN) as an approximator of the value function or policy function in reinforcement learning.

[0242] In the Q-learning algorithm, we can use DNN as an approximator of the Q function. For each state-action pair (s, a), DNN can give a Q value, which represents the long-term reward that can be obtained if the drone performs action a in state s. The input of this DNN is state s and action a (or only state s, and action a corresponds to a neuron in the output layer of DNN. In this case, DNN is called Dueling DNN-this is commonly used in control problems, that is, the output is action, so it is often used in guidance, robot motion, etc.), and the output is the corresponding Q value. Training this DNN is to find a set of parameters so that the output of DNN is as close to the actual Q value as possible, and the actual Q value is obtained through the Bellman equation. This process can be achieved through various gradient descent algorithms.

[0243] The choices of DNN include fully connected network (FCN), convolutional neural network (CNN), recurrent neural network (RNN) and their various variants. The specific choice mainly depends on the characteristics of the input state information.

[0244] 4. Model Training

[0245] In DRL, the drone will choose an action based on its current state and environment. This action will make the drone get a reward and enter a new state, which constitutes an experience (s, a, r, s'). The drone will continue to accumulate such experience during the flight and use this experience to update the parameters of the DNN, thereby improving its decision-making strategy.

[0246] The training process usually uses optimization algorithms such as RMSProp and Adam, and uses techniques such as Experience Replay (experience replay to break the correlation between samples) and Fixed Q-target (fixed Q target to stabilize the training process) to improve training efficiency and stability.

[0247] The Adam (Adaptive Moment Estimation) algorithm is an adaptive learning rate optimization method, commonly used in the training of deep learning models. The Adam algorithm combines the advantages of the momentum and RMSProp (Root Mean Square Propagation) optimization algorithms.

[0248] The specific mathematical expression is as follows:

[0249] First calculate the first and second moments of the gradient:

[0250] m_t=β1*m_{t-1}+(1-β1)*g_t

[0251] v_t=β2v_{t-1}+(1-β2)(g_t)^2

[0252] Among them, m_t and v_t represent the first-order moment (mean value) and second-order moment (unbiased variance) of the gradient respectively; β1 and β2 are hyperparameters used to control the update speed of the first-order moment and the second-order moment; g_t is the gradient at time t.

[0253] Apply bias correction to the first and second moments:

[0254] m_t^=m_t / (1-(β1)^t)

[0255] v_t^=v_t / (1-(β2)^t)

[0256] The bias correction here is used to prevent the first and second moments from being underestimated in the early stages of training.

[0257] Update parameters:

[0258] θ_t=θ_{t-1}-α*m_t^ / (sqrt(v_t^)+ε)

[0259] Among them, θ_t represents the parameter value at time t; α is the learning rate; ε is a small constant added to prevent division by zero errors (usually set to 1e-8).

[0260] The specific application steps are usually:

[0261] Initialize the parameter θ, select an appropriate learning rate α, and set the values ​​of β1, β2, and ε (generally set β1 to 0.9, β2 to 0.999, and ε to 1e-8).

[0262] At each training step, the gradient g is calculated with respect to the current parameters θ.

[0263] The gradient g is used to update the first-order moment and the second-order moment m and v.

[0264] Perform deviation correction on m and v to obtain m^ and v^.

[0265] Update the parameter θ using the above formula according to m^ and v^.

[0266] Repeat steps 2-5 until the model converges or reaches the preset maximum number of training steps.

[0267] In general, applying deep reinforcement learning to optimize the autonomous decision-making of drones involves many aspects, from the preparation of simulation environments and the definition of state-action rewards based on reinforcement learning to the selection of neural network structures and training strategies based on deep learning. All of these need to be considered comprehensively based on the specific application scenarios and characteristics of drones. Only by improving the accuracy and stability of autonomous decision-making of drones as much as possible can drones serve us better.

[0268] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A UAV detection and intervention linkage system based on the detection peak following method, characterized in that: It includes a complex environment dynamic adaptation module, a target tracking accuracy module, a big data processing and analysis module, and an autonomous decision-making optimization module; The complex environment dynamic adaptation module includes: (1) Environmental perception module: The environmental perception module uses the sensors on board the drone to obtain environmental information around the drone, including but not limited to weather conditions, topography, air turbulence intensity, and electromagnetic signal intensity; (2) Data preprocessing module: Data preprocessing is used to filter out noise, remove outliers, and remove redundant data; (3) Feature extraction module: The feature extraction module is used to find the features specified for the target task from the preprocessed data, including time, location information, and air index; (4) Adaptive control module: The adaptive control module uses the neural network in deep reinforcement learning as a function approximator to estimate the environmental model or value function, and continuously optimizes the model through feedback information during the interaction process to guide the drone to make dynamic adaptive decisions; The target tracking accuracy module includes: The system uses an advanced Kalman filter to track the target and adaptively changes the flight parameters of the drone (such as speed and direction) to attract the target. In addition, we have also added an adaptive noise suppression module. The detection peak following system is used to detect the peak value in the signal and use it as the detection target, and then track this peak point; Big data processing and analysis modules include: 1>, Data preprocessing Before entering the model training phase, the first thing to do is data preprocessing; image data needs to go through a series of preprocessing methods, including: resizing, color space conversion, data normalization, and data enhancement; 2>, build DCNN model The task of building a DCNN model is to determine the number of layers, the type and parameters of each layer of the model; a typical DCNN model consists of multiple convolutional layers, activation layers, pooling layers and fully connected layers; 3>, training and optimizing the DCNN model The training data is input into the model in batches. For each batch of data, the model will perform one forward propagation and one back propagation. Forward propagation means that the model calculates the output of each layer based on the current parameters and input data, and finally obtains the prediction result. Back propagation means that the model calculates the prediction error based on the prediction result of forward propagation and the true label, and then uses the chain rule to propagate this error back to each layer to update the parameters of each layer. During the training process, we use the validation set to verify the model. We divide the original training data into two parts, one as the training set and the other as the validation set. After each training cycle (epoch), we use the performance of the model on the validation set to adjust the model's hyperparameters. 4>, Apply DCNN model for image recognition After the DCNN model training is completed, we use it in actual image recognition tasks. By inputting the image to be recognized into the model and performing forward propagation calculations, the model will give the recognition result of the input image based on the trained parameters. The softmax function will be used in the output layer to make the output value between 0 and 1, which is equivalent to the classification probability. The classification result with the highest probability is taken as the final result. The self-service decision optimization module includes:

1. Environmental simulation We first need to build a simulation environment, which should reflect all the situations that the drone will encounter when flying; 2. Define states, actions, and rewards We need to define the state, action, and reward of the drone. The state includes the current position, speed, acceleration, fuel level, and target position of the drone. The action includes the operation of the drone: ascending, descending, turning, accelerating, and decelerating. The reward is used to measure the effectiveness of the drone after performing a certain action.

3. Choose the appropriate neural network structure and training algorithm By combining deep learning and reinforcement learning, deep neural networks are used as approximators of value functions or policy functions in reinforcement learning. In the Q-learning algorithm, we use DNN as an approximator of the Q function. For each state-action pair (s, a), DNN can give a Q value, which represents the long-term reward that can be obtained if the drone performs action a in state s. The input of this DNN is state s and action a (or only state s, and action a corresponds to a neuron in the output layer of DNN. In this case, DNN is called Dueling DNN-this is commonly used in control problems, that is, the output is action, so it is often used in guidance, robot motion, etc.), and the output is the corresponding Q value. Training this DNN is to find a set of parameters so that the output of DNN is as close as possible to the actual Q value, and the actual Q value is obtained through the Bellman equation.

4. Model Training In DRL, the drone will choose an action based on its current state and environment. This action will make the drone get a reward and enter a new state, which constitutes an experience (s, a, r, s'). The drone will continue to accumulate such experience during the flight and use this experience to update the parameters of the DNN, thereby improving its decision-making strategy; 5. Model testing and further optimization The model training in step 4 requires at least 100 test trainings in the environment and relatively stable record management.

2. A UAV detection and intervention linkage system based on the detection peak following method according to claim 1, characterized in that: The detection peak following system involves the application of a Kalman filter in the process of operation, wherein the standard form of the Kalman filter can be expressed as: 1) Prediction step: x^-(k)=Ax(k-1)+Bu(k-1) P^-(k)=AP(k-1)A^T+Q in: x is the system state vector, which may include position, velocity, and other information in drone tracking, for example; A is the transfer matrix, which describes how the system moves from the previous state to the next state; B is the control matrix, which describes how the control input affects the system state; u is the control input; P is the estimation error covariance matrix; Q is the process noise covariance matrix; '^-' indicates the predicted next state; 2) Update steps: K(k)=P^-(k)H^T(HP^-(k)H^T+R)^-1 x(k)=x^-(k)+K(k)(z(k)-Hx^-(k)) P(k)=(IK(k)H)P^-(k) in: K is the Kalman gain, which is used to measure the contribution of prediction and measurement results to the final state estimation; z is the actual measured value; H is the observation matrix, which describes how to obtain the predicted observation value from the system state vector; R is the measurement noise covariance matrix; I is the unit matrix; For a drone, if it wants to track the movement of a target, it needs to change its flight parameters such as speed and direction in real time. The implementation method is to introduce a PID controller, which processes the gap between the target and the current one: The basic formula for a PID controller is as follows: u(t)=Kpe(t)+Ki∫e(t)dt+Kd*de(t) / dt in: u(t) is the control quantity, i.e. the action that the drone needs to perform; e(t) is the error, usually obtained by the difference between the target state and the current state; Kp, Ki, Kd are proportional, integral, and differential coefficients, which can be adjusted through experiments; ∫e(t)dt is the integral of the error, representing the accumulated error in the past; de(t) / dt is the differential of the error and is expressed as the expected future error.