A collaborative intelligent detection system and method based on unmanned aerial vehicle (UAV) rescue

Abstract

This invention relates to the field of drone rescue technology, specifically a collaborative intelligent detection system and method for drone-based rescue. The system includes a detection module, a lightweight detection module, an unloading module, and a communication module. The method involves: the detection module capturing an image of the current area; the lightweight detection module, based on a lightweight detection model and the captured image, determining whether there is a trapped object in the current area, and obtaining a local detection result; the unloading module, based on an unloading model, determining the accuracy of the local detection result. If accurate, the local detection result is sent to the rescue command center via the communication module; otherwise, the current area image is sent to the rescue command center via the communication module for manual identification, and the unloading model is continuously updated based on the manual identification result and the local detection result. This addresses the technical problems of existing drone-based rescue systems, such as the high computational pressure on the drone side and the limited detection performance due to reliance on a single detection method.

Description

Technical Field

[0001] This invention relates to the field of drone rescue technology, specifically to a collaborative intelligent detection system and method based on drone rescue. Background Technology

[0002] Natural disasters often strike suddenly, catching people off guard. Therefore, the effectiveness of disaster emergency response hinges on the swiftness of the rescue efforts after a disaster occurs. Drones possess the direct capability to rapidly provide situational awareness over large areas. They reduce the time and number of search and rescue personnel required in emergencies, and significantly lower the cost and risk of search and rescue operations.

[0003] Lü Ming et al. introduced an FPGA-based UAV disaster relief image acquisition and analysis system. By using FPGA for image analysis and processing and a microcontroller for system control, the system can effectively realize image acquisition and basic stitching functions, and can basically meet the image acquisition and analysis functions for disaster areas (Lü Ming, Zheng Jingchen, Chen Jinhong, et al. Design of disaster relief UAV image acquisition and analysis system based on FPGA technology [J]. Medical and Health Equipment, 2016, 37(8):6-9.DOI:10.7687 / J.ISSN1003-8868.2016.08.006.). Wu Chunxue constructed a novel network architecture based on ResNet50 and CPM to solve the problem of human pose estimation in complex earthquake rescue environments. This is of great significance for improving the efficiency of rescue by promoting the behavior recognition and state analysis of disaster victims in earthquake rescue (Wu Chunxue, He Xinxin. Research on human pose estimation in earthquake rescue based on ResNet50 [J]. Information Technology and Network Security, 2022, 41(3):50-58,70.DOI:10.19358 / j.issn.2096-5133.2022.03.009.).

[0004] Chinese patent CN111498111A discloses a search and rescue drone with life detection function. The invention incorporates a built-in radio wave detector and a sound wave detector, which work together to more efficiently and accurately identify the location of survivors. It also incorporates built-in smoke paintballs, allowing search and rescue personnel at a distance to see the survivor's location. Furthermore, fluorescent agents can be sprayed at the survivor's location, making it easier to identify survivors even in low-light conditions, thus facilitating search and rescue operations (Qingdao Didu Life Support and Rescue Equipment Technology Co., Ltd. A Search and Rescue Drone with Life Detection Function: CN202010383832.1 [P]. 2020-08-07.).

[0005] While the aforementioned existing technologies can detect disaster victims at rescue sites, they do not take into account the computing power of the drones. The monitoring models used are all relatively large, resulting in a high computing power burden on the drones. Furthermore, they cannot evaluate or assist in the detection performance, thus limiting their detection capabilities. Summary of the Invention

[0006] The purpose of this invention is to provide a collaborative intelligent detection system and method based on drone rescue, which solves the technical problems of high computing power pressure on the drone side and limited detection performance due to reliance on a single detection method in existing drone rescue systems.

[0007] The technical solution of the present invention is: a collaborative intelligent detection system based on drone rescue, which is characterized by: including a detection module, a lightweight detection module, an unloading module and a communication module placed on the drone end;

[0008] The detection module is used to capture images of the current area;

[0009] The lightweight detection module is used to determine whether there is a trapped object in the current area based on the current area image captured by the detection module, and obtain local detection results;

[0010] The unloading module is used to determine whether the local detection results are accurate based on the unloading model. If they are accurate, the local detection results are sent to the rescue command center via the communication module. Otherwise, the current area image is sent to the rescue command center via the communication module for manual identification, and the unloading model is continuously updated based on the manual identification results and the local detection results.

[0011] To further reduce the computational burden on the UAV side, the MobileNetv3 small model was selected as a lightweight detection model.

[0012] Furthermore, the unloading model is a DQN network model, which includes a value network and a target network connected in sequence; both the value network and the target network are fully connected networks; the input of the value network is connected to the "Avg_pool" layer of the MobileNetv3 small model.

[0013] Furthermore, the local detection results are judged based on the unloading model, specifically as follows:

[0014] The unloading model receives the output of the "Avg_pool" layer in the MobileNetv3 small model and determines the local detection results based on the output of the "Avg_pool" layer.

[0015] Furthermore, the unloading model is continuously updated based on the results of manual identification and local detection, specifically as follows:

[0016] When the unloading model is in state S, it selects action a, receives reward r, and transitions to the next state S′; a large amount of experience (S, a, r, S′) is stored in the experience pool for continuous updating of the unloading model;

[0017] Wherein, state S is the output of the "Avg_pool" layer for the current region image, and state S′ is the output of the "Avg_pool" layer for the next region image; a∈{0,1}, a=0 indicates that the local detection result is sent to the rescue command center, a=1 indicates that the current region image is sent to the rescue command center for manual identification, and the identification result is then transmitted to the UAV; reward r is a reward parameter determined based on the relationship between action a, local detection result, and manual identification result.

[0018] Furthermore, the reward r is obtained based on the following formula:

[0019]

[0020] In the formula, r t For the reward at time t, s t Let a be the state at time t. t Represents the action at time t; z1 is negative, z2 and z3 are positive, and z2 is less than z3; ∧ is the union symbol, x t G(x) represents the current region image captured by the detection module at time t. t ) for the rescue command center according to x t The given manual recognition result, g(x) t ) = 0 indicates that the manual identification result is that there is no trapped object in the current area, g(x) t f(x) = 1 indicates that the manual identification result shows that there is a trapped object in the current area. t |μ) represents the local detection result on the UAV side, μ is the network parameter of the lightweight detection model, and f(x) t |μ)=0 indicates that the local detection result on the UAV side is that there is no trapped object in the current area, f(x) t |μ)=1 indicates that the local detection result on the UAV side is that there is a trapped object in the current area.

[0021] Furthermore, the unloading model is updated by minimizing the loss function using a gradient descent strategy, where the loss function Loss(θ) is defined as:

[0022] Loss(θ)=E(y t -Q(s t ,a t ;θ)) 2

[0023] In the formula, E represents the expectation, and Q(s) represents the expectation. t ,at y is the state-action value function of the network at time t. t Let θ be the target value of the value network at time t, and let θ be the network parameters of the value network.

[0024] The state-action value function Q(s,a) of the value network is obtained by the following formula:

[0025]

[0026] In the formula, k is a parameter related to the number of rewards, and γ represents the discount factor.

[0027] This invention also provides a collaborative intelligent detection method based on drone rescue, characterized in that it is implemented based on the aforementioned collaborative intelligent detection system for drone rescue, and includes the following steps:

[0028] Step 1: Capture the image of the current area based on the detection module;

[0029] Step 2: Detect whether there are trapped objects in the current area image based on the lightweight detection model and obtain local detection results; determine whether the local detection results are accurate based on the unloading model. If accurate, send the local detection results to the rescue command center through the communication module; otherwise, send the current area image to the rescue command center through the communication module for manual identification, and continuously update the unloading model based on the manual identification results and the local detection results.

[0030] Furthermore, in step 2, the local detection result is determined based on the unloading model, specifically as follows:

[0031] The unloaded model receives the output of the "Avg_pool" layer in the lightweight detection model MobileNetv3 small, and determines the local detection result based on the output of the "Avg_pool" layer.

[0032] Furthermore, in step 2, the unloading model is continuously updated based on the results of manual identification and local detection, specifically as follows:

[0033] When the unloading model is in state S, it selects action a, receives reward r, and transitions to the next state S′; a large amount of experience (S, a, r, S′) is stored in the experience pool for continuous updating of the unloading model;

[0034] Wherein, state S is the output of the "Avg_pool" layer for the current region image, and state S′ is the output of the "Avg_pool" layer for the next region image; a∈{0,1}, a=0 indicates that the local detection result is sent to the rescue command center, a=1 indicates that the current region image is sent to the rescue command center for manual identification, and the identification result is then transmitted to the UAV; reward r is a reward parameter determined based on the relationship between action a, local detection result, and manual identification result.

[0035] Furthermore, the aforementioned reward r is obtained based on the following formula:

[0036]

[0037] In the formula, r t For the reward at time t, s t Let a be the state at time t. t Represents the action at time t; z1 is negative, z2 and z3 are positive, and z2 is less than z3; ∧ is the union symbol, x t G(x) represents the current region image captured by the detection module at time t. t ) for the rescue command center according to x t The given manual recognition result, g(x) t ) = 0 indicates that the manual identification result is that there is no trapped object in the current area, g(x) t f(x) = 1 indicates that the manual identification result shows that there is a trapped object in the current area. t |μ) represents the local detection result on the UAV side, μ is the network parameter of the lightweight detection model, and f(x) t |μ)=0 indicates that the local detection result on the UAV side is that there is no trapped object in the current area, f(x) t |μ)=1 indicates that the local detection result on the UAV side is that there is a trapped object in the current area.

[0038] Furthermore, the unloading model is updated by minimizing the loss function using a gradient descent strategy, where the loss function Loss(θ) can be defined as:

[0039] Loss(θ)=E(y t -Q(s t ,a t ;θ)) 2

[0040] In the formula, E represents the expectation, and Q(s) represents the expectation. t ,a t y is the state-action value function of the network at time t. t Let θ be the target value of the value network at time t, and let θ be the network parameters of the value network.

[0041] The state-action value function Q(s,a) of the value network can be obtained by the following formula:

[0042]

[0043] In the formula, k is a parameter related to the number of rewards, and γ represents the discount factor.

[0044] The beneficial effects of this invention are:

[0045] 1. Considering the computing power limitations of drones, this invention employs a lightweight detection model to detect whether there are trapped objects in the current area, reducing computing power pressure. Furthermore, an offloading model is added to the lightweight detection model on the drone side. The offloading model is used to evaluate the lightweight detection model's understanding of the image. Data that the lightweight detection model is unsure about is sent to the rescue command center for manual identification, thereby improving detection performance. The offloading model is continuously updated based on the manually identified data, further improving detection accuracy, reducing the number of manual identifications, and lowering detection costs.

[0046] 2. The lightweight detection model of this invention is a lightweight Mobilenetv3 small model, with a size of only 10.3M. The offloading model is simply Mobilenetv3 small with two layers of fully connected networks added. Such a lightweight model is more convincing when deployed on drones where computing power and storage energy are limited.

[0047] 3. This invention utilizes an offloading model to determine when to offload perception tasks, thereby minimizing cloud communication costs. Attached Figure Description

[0048] Figure 1 Here is a flowchart of a collaborative intelligent detection method for drone-based rescue, as shown in the example.

[0049] Figure 2 This is a structural diagram of the MobileNetv3 small model in the embodiment;

[0050] Figure 3 This is a diagram of the bneck structure in the MobileNetv3 small model in the embodiment;

[0051] Figure 4 This is a diagram of the DQN architecture in the embodiment. Detailed Implementation

[0052] This invention considers a scenario where drones assist in rescue and search operations. The scenario mainly consists of two parts: a drone and a rescue command center. The drone is equipped with a detection module (which can be a camera), limited computing power, a navigation module, and a communication module. The drone flies along a planned route according to navigation, capturing images of the current area through its camera. A visual detection model is deployed on the drone to assist in detecting whether there are any trapped objects in the current area. If information about a trapped object is sampled, the drone's location is sent to the rescue command center, which then dispatches manpower to carry out rescue operations. Considering the computing power limitations of drones in real-world situations, a lightweight detection model can be used as the visual detection model on the drone side. While lightweight detection models are friendly to storage and computing power, their generalization performance is not ideal. Disaster relief cannot be taken lightly; retraining a cloud model for constantly enriched and new data places a burden and challenge on network transmission, cloud storage computing, and manual annotation. In this situation, to improve the performance of search and rescue detection, this invention adopts an interactive method with a human system, sending data that the lightweight detection model is unsure about to the rescue command center for human verification. There is already a great deal of academic research on UAV path planning, wireless channel estimation, and resource allocation. We assume that path planning and wireless channel estimation can be solved using existing methods. The focus of this invention is to improve the search and rescue performance of UAVs in a collaborative manner, thereby better assisting humans.

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] This embodiment is based on a collaborative intelligent detection system for drone rescue, including a camera mounted on the drone, a lightweight detection module, an offloading module, and a wireless communication module. The camera captures images of the current area. The lightweight detection module uses a lightweight detection model to determine whether there are trapped objects in the current area based on the images, obtaining local detection results. The offloading module interacts with a human system to send data that the lightweight detection model is unsure about to the rescue command center for verification. Specifically, the offloading model can determine the accuracy of the local detection results. If accurate, the local detection results are sent to the rescue command center via the communication module; otherwise, the current area image is sent to the rescue command center for manual identification, and the offloading model is continuously updated based on the manual identification results and the local detection results.

[0055] like Figure 1As shown, the detection process in this embodiment may include the following steps: First, capture an image of the current area based on the camera, and detect whether there is a trapped object in the current area image based on the lightweight detection model to obtain a local detection result; determine whether the local detection result is accurate based on the unloading model. If it is accurate (judged as 0 in the figure), the local detection result is sent to the rescue command center based on the communication module; otherwise, the current area image is sent to the rescue command center based on the communication module (judged as 1 in the figure) for manual identification, and the unloading model is continuously updated based on the manual identification result and the local detection result.

[0056] This embodiment selects the MobileNetv3 small model as a lightweight detection model that can be deployed on drones. Other embodiments may also use other types of lightweight detection models. The MobileNetv3 small model was proposed by the Google AI team, such as... Figure 2 As shown, Conv is the convolution operation, BatchNorm is the batch normalization operation, Avg_pool is the average pooling operation, and Bneck is a unique structure of MobileNetv3, as shown in the diagram. Figure 3 As shown, it combines the following four features: 1. An inverse residual structure with a linear bottleneck, first using 1x1 convolutions to increase dimensionality before proceeding with subsequent operations, and featuring residual edges. 2. Depthwise separable convolution. After increasing the dimensionality of the input 1x1 convolution, a 3x3 depthwise separable convolution is performed. 3. A lightweight attention model, where the attention mechanism adjusts the weights of each channel. 4. Using h-swish instead of the swish function. The structure uses the h-swishj activation function instead of the swish function, reducing computation and improving performance.

[0057] The output of the MobileNetv3 small model is a two-dimensional vector {0,1}. An output of 1 indicates that a trapped object has been detected in the current area, while an output of 0 indicates that no trapped object has been detected. MobileNetv3 small is a lightweight model, well-suited for deployment on low-computing-power terminals such as drones, but its detection performance is slightly inferior. Therefore, this embodiment adds an offloading model based on the MobileNetv3 small model. The offloading model's role is to upload inaccurate data detected by the MobileNetv3 small model to the rescue center for manual identification. The results of the manual identification are then returned to the drone to optimize the offloading model. Considering the interaction between the drone and the rescue command center, this embodiment uses a reinforcement learning DQN network model as the offloading model.

[0058] DQN network model architecture as follows: Figure 4 As shown, it includes a value network and a target network with the same structure but different parameters, connected sequentially; both the value network and the target network are fully connected networks; the input of the value network is connected to the "Avg_pool" layer of the MobileNetv3 small model. The output of the "Avg_pool" layer in the MobileNetV3 small model is 576-dimensional. The DQN network model consists of two fully connected layers. The first fully connected layer has a 128-dimensional output followed by an h-swish function, and the second fully connected layer has a 2-dimensional output. The activation function used is the LogSoftmax function.

[0059]

[0060]

[0061] In the formula, x is the input variable of the h-swish function, ReLU is the linear rectified function, C = max(Z), Z is the vector composed of all the input variables of the LogSoftmax function, and z i z j All are variables in Z;

[0062] The unloading model is continuously updated based on the results of manual identification. Specifically, when the unloading model is in state S, it selects action a, receives reward r, and then transitions to the next state S′. A large amount of experience (S,a,r,S′) is stored in the experience pool for continuous updating of the unloading model.

[0063] Where state S is the output of the "Avg_pool" layer in the MobileNetV3 small model, a∈{0,1}, a=0 means sending the local detection result to the rescue command center, a=1 means sending the current area image to the rescue command center for manual identification, and then downloading the identification result to the drone; the reward r is related to the local lightweight detection result and the identification result of the rescue command center:

[0064]

[0065] In the formula, r t For the reward at time t, s t Let a be the state at time t. t This represents the action at time t; z1 is negative, z2 and z3 are positive, and z2 is less than z3; in this embodiment, z1 = -2, z2 = 8, and z3 = 10, but in other embodiments, these can be adjusted according to the actual application scenario. ∧ is the union symbol, x t G(x) represents the current region image captured by the detection module at time t. t ) for the rescue command center according to x tThe given manual recognition result, g(x) t ) = 0 indicates that the manual identification result is that there is no trapped object in the current area, g(x) t f(x) = 1 indicates that the manual identification result shows that there is a trapped object in the current area. t |μ) represents the local detection result on the UAV side, μ is the network parameter of the lightweight detection model, and f(x) t |μ)=0 indicates that the local detection result on the UAV side is that there is no trapped object in the current area, f(x) t |μ)=1 indicates that the local detection result on the UAV side is that there is a trapped object in the current area.

[0066] The reward function is designed to guide drones to upload data incorrectly identified by the lightweight detection model to the rescue guidance center, thereby maximizing overall detection performance. If the drone cannot make a judgment without uploading the data, there is no reward or penalty. If the drone can predict the same result as the rescue guidance center, it is penalized for consuming its own power and wireless bandwidth; however, the penalty value is lower than the reward value for correctly uploading data incorrectly predicted by the lightweight detection model. This is to prevent drones from using their own lightweight detection model exclusively to avoid penalties. The reward for the fourth scenario is higher than that for the third scenario because the value of correctly detecting a distressed object is higher than eliminating a false alarm.

[0067] During the update process, the target network is typically kept fixed, and only the value network model is updated during training. After the value network model has been updated a certain number of times, the network parameters of the target network are then updated to match the network parameters of the value network. The network parameters θ of the value network are optimized by minimizing the loss function using a gradient descent strategy.

[0068] According to Bayes' theorem:

[0069]

[0070] In the formula, y t Let γ be the target value of the value network, and let γ be the discount factor, satisfying 0 < γ < 1. Let Q' be the state-action value function of the target network, and θ be the value of the target network. - For the network parameters of the target network, s t+1 Let a be the state at time t+1. t+1 This represents the action at time t+1;

[0071] Long-term rewards are defined as:

[0072]

[0073] In the formula, k is a parameter related to the number of rewards;

[0074] The state-action value function of the value network is:

[0075]

[0076] The loss function is defined as:

[0077] Loss(θ)=E(y t -Q(s t ,a t ;θ)) 2

[0078] In the formula, Q(s) t ,a t y is the state-action value function of the network at time t. t Let θ be the target value of the value network at time t, and let θ be the network parameters of the value network.

Claims

1. A collaborative intelligent detection system based on unmanned aerial vehicle (UAV) rescue, characterized in that: It includes a detection module, a lightweight detection module, an unloading module, and a communication module mounted on the drone. The detection module is used to capture images of the current area; The lightweight detection module is used to determine whether there is a trapped object in the current area based on the current area image captured by the detection module, and obtain local detection results; The unloading module is used to determine whether the local detection results are accurate based on the unloading model. If they are accurate, the local detection results are sent to the rescue command center via the communication module. Otherwise, the current area image is sent to the rescue command center via the communication module for manual identification. The unloading model is continuously updated based on the manual identification results and the local detection results. The lightweight detection model is the MobileNetv3 small model; The unloading model is a DQN network model, which includes a value network and a target network connected in sequence; both the value network and the target network are fully connected networks; the input of the value network is connected to the "Avg_pool" layer of the MobileNetv3 small model. The unloading model is continuously updated based on both manual identification results and local detection results, specifically as follows: In state S, the unloading model selects action a, receives reward r, and transitions to the next state S'. The experience (S, a, r, S') is stored in the experience pool for continuous updates to the unloading model. State S is the output of the "Avg_pool" layer for the current region image, and state S' is the output of the "Avg_pool" layer for the next region image. a∈{0,1}, a=0 indicates sending the local detection result to the rescue command center, and a=1 indicates sending the current region image to the rescue command center for manual identification, with the identification result then being transmitted to the drone. Reward r is a reward parameter determined based on the relationship between action a, the local detection result, and the manual identification result. The reward r is obtained based on the following formula: ； In the formula, The reward at time t, Let t be the state at time t. This represents the action at time t; z1 is negative, z2 and z3 are positive, and z2 is less than z3; This represents the current region image captured by the detection module at time t. For the rescue command center according to The results of manual identification are given. This indicates that the manual identification result shows there are no trapped individuals in the current area. This indicates that the manual identification result shows that there are trapped individuals in the current area. This indicates the local detection results from the drone side. For the network parameters of the lightweight detection model, This indicates that the drone's local detection results show that there are no trapped objects in the current area. This indicates that the drone's local detection results show that there is a trapped object in the current area.

2. The collaborative intelligent detection system based on UAV rescue according to claim 1, characterized in that, The local detection results are determined based on the unloading model, specifically as follows: The unloaded model receives the output of the "Avg_pool" layer in the MobileNetv3 small model and determines the local detection result based on the output of the "Avg_pool" layer.

3. The collaborative intelligent detection system based on UAV rescue according to claim 2, characterized in that, The unloading model is continuously updated by minimizing the loss function using a gradient descent strategy, where the loss function Loss(θ) of the unloading model is defined as: ； In the formula, E represents the expectation, and Q(s) represents the expectation. t ,a t ) is the state-action value function of the network at time t. Let θ be the target value of the value network at time t, and let θ be the network parameters of the value network. State-action value function of a value network Obtained through the following formula: ； In the formula, k is a parameter related to the number of rewards. This is the discount factor.

4. A collaborative intelligent detection method based on unmanned aerial vehicle (UAV) rescue, characterized in that: This is implemented based on the collaborative intelligent detection system for drone-based rescue as described in any one of claims 1-3.

Images