An unmanned aerial vehicle-based field personnel automatic searching method

Abstract

The application discloses a kind of based on unmanned plane's field personnel automatic searching method.Unmanned plane receives the flight instruction of route sent by handheld terminal equipment, and carries out route flight in search area;Unmanned plane obtains the infrared image of search area by camera in the process of route flight, carries out image processing to infrared image, identifies the result whether personnel exists in the image by thermal imaging threshold analysis and trapped personnel movement analysis;According to image recognition result, the flight attitude of unmanned plane is controlled to track and capture personnel, and the position information, environmental information of personnel are acquired and sent to handheld terminal set and cloud server.The application can accurately identify trapped personnel in field under the condition that there are more sheltering objects in field, and personnel identification is carried out automatically, and the environmental information of surrounding is acquired, accurate rescue is carried out to trapped personnel, which not only reduces cost, but also reduces human resources.

Description

Technical Field

[0001] This invention relates to a drone application method in the field of drone search and rescue technology research, and particularly to a drone-based automatic search method for personnel in the field. Background Technology

[0002] In recent years, there have been frequent incidents of people going missing during wilderness exploration, tourism, and natural disasters. Traditional manual search and rescue methods have revealed problems such as high costs and slow search speeds. In particular, for some complex terrains inaccessible to vehicles and search equipment, the application of drone-based intelligent search and rescue systems in wilderness rescue has been promoted. However, at present, drone-based wilderness search still relies solely on human operation. Conducting a comprehensive search still requires search and rescue personnel at the base station to visually assess images transmitted by the drone, which consumes a lot of manpower and resources. Moreover, ordinary cameras cannot accurately distinguish the background and the trapped people in the wilderness. Summary of the Invention

[0003] To address the problems in the background technology, the purpose of this invention is to provide an automatic search method for personnel in the wild based on unmanned aerial vehicles (UAVs). This method can accurately identify trapped personnel in the wild even when there are many obstructions. By automatically identifying personnel and acquiring surrounding environmental information, it can accurately rescue trapped personnel, thereby reducing costs and manpower and material resources.

[0004] The technical solution adopted in this invention is as follows:

[0005] The drone receives flight instructions from a handheld terminal device and flies along the search area.

[0006] During its flight path, the drone acquires infrared images of the search area using its own camera, processes the infrared images, and identifies whether there are people in the images through thermal imaging threshold analysis and trapped personnel movement analysis.

[0007] When it is determined that there are people in the image, the drone is controlled to track and capture the people based on the image recognition results, and the location information and environmental information of the people are obtained and sent to the handheld terminal device.

[0008] The handheld terminal device sends the location and environmental information to the cloud server of the remote monitoring center via the 5G network.

[0009] The flight path command is generated after the GPS coordinates of the search area are determined. The handheld terminal device sends the flight path command to the drone through the data transmission module.

[0010] Preferably, the flight route command is generated by customizing the GPS coordinates of the search area through an APP installed on a handheld terminal. The waypoints are displayed on the Gaode Map. Then, the drone's data transmission module sends the flight route command to the drone, and the drone begins to execute the flight route task and acquire infrared images.

[0011] The infrared image video of the search area acquired by the UAV is processed by frame extraction, and the extracted video is transcoded into H.246 format. The transcoded video is then re-filed and saved.

[0012] Histogram equalization and background denoising are performed on the infrared image of the search area in real time. Then, thermal imaging threshold analysis and personnel motion analysis are performed to obtain the possible ROI pixel regions of personnel, where:

[0013] The region of human ROI pixels obtained through thermal imaging threshold analysis is denoted as region Rt.

[0014] The region of ROI pixels obtained from the motion analysis of trapped personnel is denoted as the Rm region.

[0015] The Rt and Rm regions are merged to obtain the union region. The union region is then denoised, the target contour of the union region is extracted, and the smallest solid line rectangle is drawn.

[0016] Input the smallest solid-line rectangle into the pre-trained person recognition model and output the recognition result.

[0017] The personnel identification model can specifically be a convolutional neural network model, and the identification result refers to whether there is a person in the smallest solid-line rectangle.

[0018] The thermal imaging threshold analysis involves performing binary segmentation on the infrared image using a preset threshold, and then performing opening and closing operations on the binarized infrared image before performing connected component operations to obtain each connected component.

[0019] Only connected components with an area greater than the reference area S calculated by the following formula are retained as connected component pixel ROI regions, and these connected component pixel ROI regions are defined as Rt regions, where:

[0020] S = 0.0025 × r × c

[0021] Where r is the width of the connected component with the largest area, and c is the height of the connected component with the largest area.

[0022] The motion analysis of the trapped personnel is performed by comparing the pixel values ​​of each pixel at the same location in the infrared image between two consecutive frames. Pixels with a difference Q greater than a preset pixel difference threshold are designated as hotspot pixels. The preset pixel difference threshold is set to 16.

[0023] |I1-I2|>Q

[0024] Where I1 is the pixel value of the same pixel in the current frame, and I2 is the pixel value of the same pixel in the previous frame.

[0025] Multiple hotspot pixel ROI regions are formed by connecting numerous hotspot pixels in a coherent manner according to connected components. Then, the average pixel value M of all pixels in all connected component pixel ROI regions obtained from thermal imaging threshold analysis is calculated, and the following judgments are made:

[0026] When the number of pixels in the hot spot pixel ROI region is greater than 5% of the number of pixels in all connected component pixel ROI regions obtained from the thermal imaging threshold analysis, and the average pixel value of the hot spot pixel ROI region is greater than the average pixel value M, the hot spot pixel ROI region is regarded as the Rm region.

[0027] When it is determined that there is a person in the image, the center pixel coordinates of the smallest solid line rectangle are extracted in real time as the center point of the person (x1, y1), and the center pixel coordinates of the infrared image are obtained as the center point of the image (x0, y0), that is, the center pixel coordinates of the infrared camera on the image display interface of the terminal device.

[0028] Based on the positional deviation of the personnel center point (x1, y1) and the image center point (x0, y0) in the vertical direction of the infrared image, calculate the camera pitch angle deviation θ during UAV tracking. y ,in:

[0029]

[0030] In the formula, β is the rated pitch angle of the drone camera;

[0031] Then, the corrected yaw angle δ(t) in the horizontal yaw direction of the UAV is obtained by the tracking algorithm according to the following formula;

[0032]

[0033] In the formula, δ(t) represents the corrected yaw angle at time t, and θ x (t) represents the yaw angle deviation of the UAV at time t, where yaw angle deviation θ x The calculation method of (t) and pitch angle deviation θ y The calculation method is the same, where λ is the adjustment parameter, d(t) is the aiming distance deviation at time t, and V(t) is the current speed of the UAV.

[0034] The aiming distance deviation d(t) is calculated according to the following formula:

[0035]

[0036] In the formula, a max V(t) represents the maximum braking speed of the drone, and K represents the current speed of the drone. e For parameters regarding the travel distance of a drone in case of abnormal reactions, K is used in specific implementation. e Set to 0.1, R min This represents the minimum turning radius of the drone.

[0037] Through pitch angle deviation θ y The corrected yaw angle δ(t) is input into the pitch control channel and yaw control channel of the UAV in real time, and added to the original pitch control and yaw control values ​​to achieve accurate tracking and capture control.

[0038] In the above process, by adjusting the pre-aiming distance deviation d(t), the UAV can track and avoid obstacles more smoothly in conjunction with the obstacle avoidance module, and the target personnel can be accurately locked on the image interface displayed on the terminal device.

[0039] At a safe altitude between the drone and the ground, and at a safe horizontal distance between the drone and the target personnel, the drone tracks and captures images of the target personnel in real time, then displays infrared and visible light monitoring images. The drone's throttle and flight attitude are controlled in real time to hover directly above the target personnel with the camera pointing vertically downwards. At this point, the drone takes a picture and records the target personnel's GPS coordinates. After completion, the drone returns to its flight path to continue the search.

[0040] The drone in question is the DJI Mavic 2 Enterprise Edition drone, equipped with dual-light cameras, namely an infrared camera and a visible light camera. From the moment the drone locks onto the target person in the wild, it transmits multiple captured environmental infrared photos, visible light photos, and the GPS coordinates of the person to a handheld terminal device in real time.

[0041] The handheld terminal device transmits captured environmental infrared photos, visible light photos, and GPS coordinates of personnel to a cloud server in a remote monitoring center via a 5G network for real-time monitoring.

[0042] This invention can accurately identify trapped personnel in the wild when there are many obstacles. By automatically identifying personnel and obtaining information about the surrounding environment, it can accurately rescue trapped personnel, which not only reduces costs but also reduces human resources.

[0043] Compared with existing technologies, the drone-based field personnel search method provided in this invention has the following advantages:

[0044] This invention enables users to customize flight path settings on handheld terminals and repeatedly search areas in the wild where people may be present.

[0045] This invention utilizes a dual-light camera to accurately and automatically identify stranded individuals in the wild, even when there are many obstructions. It can also track and locate individuals in real time and obtain information about the surrounding environment, enabling accurate rescue of stranded individuals. This not only reduces costs but also saves manpower and resources.

[0046] The handheld terminal device of this invention uploads the information of personnel in the field to the database of a remote monitoring center via a 5G network. Remote monitoring personnel can then further confirm the personnel in the field and take effective next steps in the rescue plan. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a structural block diagram of a specific system of the present invention;

[0049] Figure 3 This is a control flowchart of the method of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments, so as to clearly and completely describe the technical solutions in the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0051] like Figure 2 As shown, the system structure of this invention includes: a drone, a remote controller, a handheld terminal device, and a cloud server in the monitoring center.

[0052] The working principle of the above scheme is as follows: The drone receives flight instructions from a handheld terminal device via a remote controller and controls the drone to fly along a flight path within the search area; the drone acquires infrared images of the search area and sends them to the handheld terminal device via the data transmission module of the remote controller; the infrared images are processed to identify whether there are people in the images; when it is determined that there are people in the images, the remote controller controls the drone's flight attitude to track and capture images of the environment in which the people are located; the drone acquires the location information and environmental information of the people and sends it to the handheld terminal device via the data transmission module of the remote controller; the handheld terminal device sends the location information and environmental information to the cloud server of the remote monitoring center via a 5G network, and the personnel at the monitoring center conduct remote monitoring.

[0053] The following is for reference. Figures 1 to 3The following describes the embodiments of the present invention and their implementation process:

[0054] like Figure 1 As shown, an automated search method for personnel in the field based on unmanned aerial vehicles includes:

[0055] S1: The UAV receives flight instructions from the handheld terminal device and flies a flight path within the search area;

[0056] S2: Acquire an infrared image of the search area, perform image processing on the infrared image, and identify whether there are people in the image through thermal imaging threshold analysis and trapped personnel motion analysis;

[0057] S3: When it is determined that there are people in the image, control the flight attitude of the drone to track and capture the people, and obtain the location information and environmental information of the people;

[0058] S4: The handheld terminal device sends the location information and environmental information to the cloud server of the remote monitoring center via the 5G network.

[0059] like Figure 2 As shown, the structure of an unmanned aerial vehicle (UAV)-based automated search system for personnel in the field includes: a UAV, a remote controller, a handheld terminal device, and a cloud server in the monitoring center.

[0060] The working principle of the above scheme is as follows: The drone receives flight instructions from a handheld terminal device via a remote controller and controls the drone to fly along a flight path within the search area; the drone acquires infrared images of the search area and sends them to the handheld terminal device via the data transmission module of the remote controller; the infrared images are processed to identify whether there are people in the images; when it is determined that there are people in the images, the remote controller controls the drone's flight attitude to track and capture images of the environment in which the people are located; the drone acquires the location information and environmental information of the people and sends it to the handheld terminal device via the data transmission module of the remote controller; the handheld terminal device sends the location information and environmental information to the cloud server of the remote monitoring center via a 5G network, and the personnel at the monitoring center conduct remote monitoring.

[0061] like Figure 3 As shown, the specific software flow of an unmanned aerial vehicle (UAV)-based automated personnel search system in the field is as follows:

[0062] Preferably, the flight route command is generated by customizing the GPS coordinates of the search area through an APP installed on a handheld terminal. The waypoints are displayed on the Gaode Map. Then, the drone's data transmission module sends the flight route command to the drone, and the drone begins to execute the flight route task and acquire infrared images.

[0063] The infrared image video of the search area acquired by the UAV is processed by frame extraction. The extracted video is then transcoded to H.246 format. The transcoded video is re-filed and saved. Conventional image preprocessing such as histogram equalization and background denoising is then performed. Further thermal imaging threshold segmentation analysis and trapped personnel motion analysis are conducted to identify potential ROI pixel regions for the trapped personnel.

[0064] The region of human ROI pixels in thermal imaging threshold analysis is denoted as region Rt.

[0065] The region of ROI pixels for the motion analysis of trapped personnel is denoted as Rm region;

[0066] The union of the Rt and Rm regions is merged, then denoised, the target contour of the Rf region is extracted, and the smallest solid line rectangle is drawn for the region in real time.

[0067] The solid-line rectangular region Rf to be identified is input into a pre-trained personnel recognition model, and the recognition result is output.

[0068] The thermal imaging threshold analysis involves binary segmenting the image using a threshold, performing opening and closing operations on the binarized image, and then performing connected component operations. After calculation, only the connected component pixel ROI regions with an area greater than S are retained. These ROI regions are then designated as Rt regions, where:

[0069] S = 0.0025 × r × c

[0070] The motion analysis of the trapped personnel is performed by comparing the pixel values ​​of each pixel at the same location in the infrared image between two consecutive frames. Pixels with a difference Q greater than a preset pixel difference threshold are designated as hotspot pixels. The preset pixel difference threshold is set to 16.

[0071] |I1-I2|>Q

[0072] Where I1 is the current frame threshold for the same pixel position, and I2 is the previous frame threshold for the same pixel position;

[0073] Calculate the average pixel value M of all pixels in the connected region ROI region retained by the thermal imaging threshold analysis. When the number of pixels in the hot spot pixel ROI region is greater than 5% of the number of pixels in the retained connected region ROI region, and the average pixel value of the hot spot pixel ROI region is greater than the average pixel value M, the hot spot pixel ROI region is regarded as the Rm region.

[0074] When it is determined that there are people in the image, the flight path task is interrupted, and the center point pixel coordinate position of the solid line rectangle is calculated in real time and recorded as (x1, y1). The center point pixel coordinate position of the infrared image is obtained as the image center point (x0, y0), that is, the center point pixel coordinate position of the infrared camera on the terminal device display image interface.

[0075] Based on the positional deviation of the personnel center point (x1, y1) and the image center point (x0, y0) in the vertical direction of the infrared image, calculate the camera pitch angle deviation θ during UAV tracking. y ,in:

[0076]

[0077] In the formula, β is the rated pitch angle of the drone camera gimbal;

[0078] Then, the corrected yaw angle δ(t) in the horizontal yaw direction of the UAV is obtained by the tracking algorithm according to the following formula;

[0079]

[0080] In the formula, δ(t) represents the corrected yaw angle at time t, and θ x (t) represents the yaw angle deviation of the UAV at time t, where yaw angle deviation θ x The calculation method of (t) and pitch angle deviation θ y The calculation method is the same, where λ is the adjustment parameter, d(t) is the aiming distance deviation at time t, and V(t) is the current speed of the UAV.

[0081] The aiming distance deviation d(t) is calculated according to the following formula:

[0082]

[0083] In the formula, a max V(t) represents the maximum braking speed of the drone, and K represents the current speed of the drone. e For parameters regarding the travel distance of a drone in case of abnormal reactions, K is used in specific implementation. e Set to 0.1, R min This represents the minimum turning radius of the drone.

[0084] Through pitch angle deviation θ y The corrected yaw angle δ(t) is input into the pitch control channel and yaw control channel of the UAV in real time, and added to the original pitch control and yaw control values ​​to achieve accurate tracking and capture control.

[0085] In the above process, by adjusting the pre-aiming distance deviation d(t), the UAV can track and avoid obstacles more smoothly in conjunction with the obstacle avoidance module, and the target personnel can be accurately locked on the image interface displayed on the terminal device.

[0086] At a safe altitude between the drone and the ground, and at a safe horizontal distance between the drone and the target personnel, the drone tracks and captures images of the target personnel in real time, then displays infrared and visible light monitoring images. The drone's throttle and flight attitude are controlled in real time to hover directly above the target personnel with the camera pointing vertically downwards. At this point, the drone takes a picture and records the target personnel's GPS coordinates. After completion, the drone returns to its flight path to continue the search.

[0087] The drone used was a DJI Mavic 2 Enterprise Edition drone, equipped with dual-light cameras, namely an infrared camera and a visible light camera. From the moment the drone locked onto the target person in the wild, it transmitted multiple captured environmental infrared photos, visible light photos, and the GPS coordinates of the person to the handheld terminal device in real time.

[0088] The handheld terminal device transmits captured environmental infrared photos, visible light photos, and GPS coordinates of personnel to a cloud server in a remote monitoring center via a 5G network for real-time monitoring.

[0089] Furthermore, after completing the above operations, the drone returns to the flight path to continue executing the remaining waypoints and continues to identify the target area. Finally, it determines whether the entire flight path mission is completed. If it is completed, the mission ends.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the content described in the above specific embodiments. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for automatic search of personnel in the field based on unmanned aerial vehicles (UAVs), characterized in that the method... include: The drone receives flight instructions from a handheld terminal device and flies along the search area. During its flight path, the drone acquires infrared images of the search area using its own camera, processes the infrared images, and identifies whether there are people in the images through thermal imaging threshold analysis and trapped personnel movement analysis. When it is determined that there are people in the image, the drone is controlled to track and capture the people based on the image recognition results, and the location information and environmental information of the people are obtained and sent to the handheld terminal device. The handheld terminal device sends the location and environmental information to the cloud server of the remote monitoring center via a 5G network. Histogram equalization and background denoising are performed on the infrared image of the search area in real time. Then, thermal imaging threshold analysis and personnel motion analysis are performed to obtain the personnel ROI pixel area, where: The region of human ROI pixels obtained through thermal imaging threshold analysis is denoted as region Rt. The region of ROI pixels obtained from the motion analysis of trapped personnel is denoted as the Rm region. The Rt and Rm regions are merged to obtain the union region. The union region is then denoised, the target contour of the union region is extracted, and the smallest solid line rectangle is drawn. Input the smallest solid-line rectangle into a pre-trained person recognition model and output the recognition result. The motion analysis of the trapped personnel is achieved by comparing the pixel values ​​of each pixel at the same location in the infrared image between two consecutive frames. Pixels with a difference Q greater than a preset pixel difference threshold are designated as hotspot pixels. |I1-I2|>Q Where I1 is the pixel value of the same pixel in the current frame, and I2 is the pixel value of the same pixel in the previous frame. Multiple hotspot pixel ROI regions are formed by connecting numerous hotspot pixels in a coherent manner according to connected components. Then, the average pixel value M of all pixels in all connected component pixel ROI regions obtained from thermal imaging threshold analysis is calculated, and the following judgment is made: when the number of pixels in the hotspot pixel ROI region is greater than 5% of the number of pixels in all connected component pixel ROI regions obtained from thermal imaging threshold analysis, and the average pixel value of the hotspot pixel ROI region is greater than the average pixel value M, the hotspot pixel ROI region is regarded as Rm region.

2. The method for automatic search of personnel in the field based on unmanned aerial vehicles according to claim 1, characterized in that: The flight route command is generated after the GPS coordinates of the search area are determined.

3. The method for automatic search of personnel in the field based on unmanned aerial vehicles according to claim 1, characterized in that: The thermal imaging threshold analysis involves performing binary segmentation on the infrared image using a preset threshold, and then performing opening and closing operations on the binarized infrared image before performing connected component operations to obtain each connected component. Only connected components with an area greater than the reference area S calculated by the following formula are retained as connected component pixel ROI regions, and these connected component pixel ROI regions are defined as Rt regions, where: S = 0.0025 × r × c Where r is the width of the connected component with the largest area, and c is the height of the connected component with the largest area.

4. The method for automatic search of personnel in the field based on unmanned aerial vehicles according to claim 1, characterized in that: When it is determined that there are people in the image, the center pixel coordinates of the smallest solid line rectangle are extracted in real time as the center point of the person (x1, y1), and the center pixel coordinates of the infrared image are obtained as the center point of the image (x0, y0). Based on the positional deviation of the personnel center point (x1, y1) and the image center point (x0, y0) in the vertical direction of the infrared image, calculate the camera pitch angle deviation θ during UAV tracking. y ,in: In the formula, β is the rated pitch angle of the drone camera; Then, the corrected yaw angle δ(t) in the horizontal yaw direction of the UAV is obtained by the tracking algorithm according to the following formula; In the formula, δ(t) represents the corrected yaw angle at time t, and θ x (t) represents the yaw angle deviation of the UAV at time t, where yaw angle deviation θ x The calculation method of (t) and pitch angle deviation θ y The calculation method is the same, where λ is the adjustment parameter, d(t) is the aiming distance deviation at time t, and V(t) is the current speed of the UAV. The aiming distance deviation d(t) is calculated according to the following formula: In the formula, a max V(t) represents the maximum braking speed of the drone, and K represents the current speed of the drone. e R is the distance parameter for the drone to travel when encountering abnormal reactions. min This represents the minimum turning radius of the drone. Through pitch angle deviation θ y The corrected yaw angle δ(t) is input into the pitch control channel and yaw control channel of the UAV in real time and added to the original pitch control and yaw control values.

5. The method for automatic search of personnel in the field based on unmanned aerial vehicles according to claim 1, characterized in that: The handheld terminal device transmits captured environmental infrared photos, visible light photos, and GPS coordinates of personnel to a cloud server in a remote monitoring center via a 5G network for real-time monitoring.

Images