Method, system, medium and equipment for identifying and early warning personnel below low-altitude unmanned aerial vehicle

Abstract

The invention discloses a method, a system, a medium and equipment for identifying and early warning personnel below a low-altitude unmanned aerial vehicle. The method comprises the following steps: S1, data acquisition: acquiring multi-modal data of a lower area; s2, establishing image recognition: establishing a mapping relation between an image coordinate system and a ground projection area according to the real-time height of the unmanned aerial vehicle and IMU attitude data; s3, outputting an identification result: performing target identification on the dynamic region of interest by adopting a visual detection model and a thermal imaging segmentation model which are connected in parallel; s4, feature extraction: performing time sequence attitude feature extraction on the identified personnel target, classifying personnel motion states by using an LSTM network, and predicting a target motion track in combination with Kalman filtering; and S5, completing early warning: constructing a dynamic risk assessment matrix according to the flight height, the personnel density and the movement speed of the unmanned aerial vehicle, and triggering a graded early warning signal when a risk value exceeds an adaptive threshold value. The method has the advantage that the control safety of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and specifically to a method, system, medium, and device for identifying and warning people under low-altitude unmanned aerial vehicles (UAVs). Background Art

[0002] Low altitude refers to the space close to the ground, with a flight altitude of 100 - 1000 meters above ground level. The sale of small UAVs is not regulated, and the flight of small UAVs in places such as parks is not restricted. Moreover, UAV pilots include children, youths, middle-aged people, and the elderly. Some pilots have poor safety awareness during the flight, which causes UAVs to fly over pedestrians' heads. Once dropped or out of control, it is easy for the propellers of the UAV to cause harm to pedestrians.

[0003] In view of this, it is necessary to provide a method, system, medium, and device for identifying and warning people under low-altitude UAVs. Summary of the Invention

[0004] The method, system, medium, and device for identifying and warning people under low-altitude UAVs provided by the present invention effectively solve the problem that the existing flight monitoring of low-altitude UAVs is not in place, resulting in the possibility that low-altitude UAVs may fall on people.

[0005] The technical solution adopted by the present invention is as follows:

[0006] The method for identifying and warning people under low-altitude UAVs includes the following steps:

[0007] S1. Data acquisition: Synchronously collect multi-modal data of the area below through a visible light camera, an infrared thermal imager, and a lidar mounted on the UAV;

[0008] S2. Establish image recognition: Calculate the dynamic region of interest (ROI) through affine transformation according to the real-time height of the UAV and the IMU attitude data, and establish the mapping relationship between the image coordinate system and the ground projection area;

[0009] S3. Output recognition result: Use a parallel visual detection model and a thermal imaging segmentation model to perform target recognition on the dynamic ROI. Among them, the visual detection model uses an improved lightweight convolutional neural network, the thermal imaging segmentation model is based on the U-Net architecture, and the personnel recognition result is output through a confidence weighted fusion strategy;

[0010] S4. Feature extraction: Extract the temporal pose features of the identified personnel targets, classify the personnel movement states using an LSTM network, and simultaneously predict the target movement trajectory in combination with Kalman filtering;

[0011] S5. Complete warning: Construct a dynamic risk assessment matrix based on the UAV flight altitude, personnel density, and movement speed, and trigger a graded warning signal when the risk value exceeds the adaptive threshold.

[0012] Furthermore, the calculation method of the dynamic region of interest in S2 is as follows: the ground width = 2 × flight height × tan(camera field of view angle / 2), and an affine transformation is performed in combination with the drone's elevation angle and roll angle. The transformation formula is:

[0013]

[0016] where θ is the attitude angle compensation amount, and dx, dy are the translation compensation amounts, realizing the real-time mapping from the image coordinate system to the actual ground coordinates.

[0017] Furthermore, the fusion strategy in S3 includes the following steps:

[0018] S301. Perform temperature feature verification on the confidence of the human body bounding box output by the visual detection model. When the overlapping rate between the 35 - 42°C feature region detected by thermal imaging segmentation and the visual detection box is greater than 70%, increase the comprehensive confidence by 0.15;

[0019] S302. Start the thermal imaging dominant mode under low light conditions. When the average brightness of the visible light image is lower than 50 lux, increase the weight of the thermal imaging segmentation result to 0.7;

[0020] 303. Use the D - S evidence theory to perform uncertainty reasoning on the multi - modal detection results and eliminate sensor conflicts through the basic probability assignment function.

[0021] Furthermore, the trajectory prediction method in S4 is to establish a Kalman filter state equation:

[0022] X_k = F·X_{k - 1}+W_k

[0023] Z_k = H·X_k+V_k

[0024] where the state vector X = [x, y, v_x, v_y]^T contains the target position and velocity, the observation matrix H fuses the lidar ranging data and the image pixel displacement amount, and the process noise W_k and the observation noise V_k dynamically adjust the covariance parameters according to the vibration amplitude of the drone.

[0025] Furthermore, the adaptive threshold calculation method in S5 is: T = T_base × (1 + α·Δbattery) × (1 + β·wind_speed), where T_base is the reference threshold, α is the battery power attenuation coefficient (Δbattery = 1 - current power / full power), β is the wind speed influence coefficient. When the GPS signal strength is lower than - 110 dBm, trigger a conservative strategy to automatically reduce T by 20%.

[0026] Furthermore, the low-altitude unmanned aerial vehicle is a multi-rotor unmanned aerial vehicle.

[0027] Furthermore, the warning signals of the S5 include voice prompts issued at the operation end, automatic avoidance, and automatic return.

[0028] The personnel recognition system under the low-altitude unmanned aerial vehicle includes

[0029] a data acquisition module for acquiring image data under the unmanned aerial vehicle;

[0030] a real-time processing module for performing target recognition on the ROI using the real-time altitude of the unmanned aerial vehicle and IMU attitude data;

[0031] a warning and communication module for sending warning messages.

[0032] A computer-readable storage medium stores a computer program, and when the computer program is processed and executed, it implements the steps of the personnel recognition and warning method under the low-altitude unmanned aerial vehicle. In addition, the computer-readable storage medium of this embodiment can adopt any combination of one or more readable storage media, where the readable storage medium includes systems, devices or components of electricity, light, electromagnetism, infrared rays or semiconductors, or any combination of the above.

[0033] A computer device includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus: where

[0034] the memory is used to store a computer program;

[0035] the processor is used to execute the steps of the personnel recognition and warning method under the low-altitude unmanned aerial vehicle by running the program stored on the memory. As an implementation manner of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0036] As an implementation manner of the present invention, the communication interface is used for communication between the above terminal and other devices.

[0037] As an implementation manner of the present invention, the memory may include a Random Access Memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0038] As an implementation manner of the present invention, the aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0039] Advantages of the invention:

[0040] 1. Solve the reliability problem of the traditional UAV vision system in a dynamic environment, meet the real-time requirements of the low-altitude security scenario, and improve the safety of UAV operation.

[0041] 2. Can automatically predict the walking trajectory of pedestrians, make corresponding predictions and issue appropriate warnings to the operator.

[0042] 3. Use a 3×3 matrix method to represent affine transformation in a two-dimensional space, realize the processing of translation and rotation, and innovatively decompose the UAV attitude angle into explicit expressions of θ, tx, and ty, solving the problem of projection distortion caused by attitude changes in the traditional UAV vision system.

[0043] 4. By setting different warning methods such as voice prompts, automatic avoidance, and automatic return, the safety can be maximally improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of the method for identifying and warning people below a low-altitude UAV provided by the embodiment of the present application. DETAILED IMPLEMENTATION MANNER

[0045] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific implementation manner of the present invention will be given in conjunction with the accompanying drawings.

[0046] As Figure 1 shown, the first embodiment provided by this application is a method for identifying and warning people under a low-altitude drone, including the following steps:

[0047] S1. Data acquisition: Synchronously acquire multi-modal data of the area below through a visible light camera, an infrared thermal imager, and a lidar mounted on the drone;

[0048] S2. Establish image recognition: According to the real-time altitude of the drone and the IMU attitude data, calculate the dynamic region of interest through affine transformation, and establish the mapping relationship between the image coordinate system and the ground projection area;

[0049] S3. Output recognition result: Use a parallel visual detection model and a thermal imaging segmentation model to perform target recognition on the dynamic region of interest. Among them, the visual detection model uses an improved lightweight convolutional neural network, and the thermal imaging segmentation model is based on the U-Net architecture, and outputs the personnel recognition result through a confidence weighted fusion strategy;

[0050] S4. Feature extraction: Extract the temporal pose features of the recognized personnel targets, use the LSTM network to classify the personnel movement states, and at the same time combine the Kalman filter to predict the target movement trajectory;

[0051] S5. Complete warning: Construct a dynamic risk assessment matrix according to the drone flight altitude, personnel density, and movement speed, and trigger a hierarchical warning signal when the risk value exceeds the adaptive threshold.

[0052] In the above design, it is possible to automatically identify the people below the flight route during the flight of the drone, provide a warning for the operator to avoid crowds, and improve flight safety.

[0053] Specifically: The calculation method of the dynamic region of interest in S2 is: Ground width = 2 × flight altitude × tan(camera field of view angle / 2), and perform affine transformation in combination with the drone elevation angle and roll angle. The transformation formula is:

[0054]

[0057] where θ is the attitude angle compensation amount, dx, dy are the translation compensation amounts, to realize the real-time mapping from the image coordinate system to the actual ground coordinates. θ = arctan(tan(roll) / tan(pitch)) is the attitude compensation angle; tx = Δh × (sin(pitch) - cos(pitch) × tan(roll / 2)); ty = Δh × (sin(roll) - cos(roll) × tan(pitch / 2)).

[0058] In the above design, an affine transformation is represented in a two-dimensional space using a 3×3 matrix method to achieve the processing of translation and rotation. The attitude angles of the UAV are innovatively decomposed into explicit expressions of θ, tx, and ty, solving the problem of projection distortion caused by attitude changes in traditional UAV vision systems.

[0059] Specifically: The fusion strategy in S3 includes the following steps:

[0060] S301. Verify the temperature characteristics of the confidence of the human body bounding box output by the visual detection model. When the overlap rate between the 35 - 42°C characteristic region detected by thermal imaging segmentation and the visual detection frame is greater than 70%, increase the comprehensive confidence by 0.15.

[0061] S302. Activate the thermal imaging dominant mode under low light conditions. When the average brightness of the visible light image is lower than 50 lux, increase the weight of the thermal imaging segmentation result to 0.7.

[0062] S303. Use the D - S evidence theory to perform uncertainty reasoning on the multi - modal detection results and eliminate sensor conflicts through the basic probability assignment function.

[0063] Specifically: The trajectory prediction method in S4 is to establish a Kalman filter state equation:

[0064] X_k = F·X_{k - 1}+W_k

[0065] Z_k = H·X_k+V_k

[0066] Where the state vector X = [x, y, v_x, v_y]^T contains the target position and velocity, the observation matrix H fuses the lidar ranging data and the image pixel displacement, and the process noise W_k and the observation noise V_k dynamically adjust the covariance parameters according to the vibration amplitude of the UAV.

[0067] In the above design, it is possible to automatically predict the trajectory of pedestrians.

[0068] Specifically: The adaptive threshold calculation method in S5 is: T = T_base×(1 + α·Δbattery)×(1 + β·wind_speed), where T_base is the reference threshold, α is the battery power attenuation coefficient (Δbattery = 1 - current power / full power), β is the wind speed influence coefficient. When the GPS signal strength is lower than - 110 dBm, trigger a conservative strategy to automatically reduce T by 20%.

[0069] In the above design, it is possible to achieve hierarchical early warnings for different situations, enabling operators to make different responses.

[0070] Specifically: The low - altitude UAV mentioned above is a multi - axis rotor UAV.

[0071] Specifically, the warning signals of S5 include voice prompts, automatic avoidance, and automatic return at the operation end.

[0072] In the above design, different warning levels can be achieved, and corresponding prompts can be made to facilitate the operator's further response.

[0073] The second embodiment provided by the present application is a personnel recognition system under a low-altitude unmanned aerial vehicle, including

[0074] A data acquisition module for acquiring image data below the unmanned aerial vehicle;

[0075] A real-time processing module for performing target recognition on the ROI using the real-time altitude of the unmanned aerial vehicle and IMU attitude data;

[0076] A warning and communication module for sending warning messages.

[0077] The third embodiment provided by the present application is a computer-readable storage medium storing a computer program, and when the computer program is processed and executed, it implements the steps of the personnel recognition and warning method below the low-altitude unmanned aerial vehicle. In addition, the computer-readable storage medium of this embodiment can adopt any combination of one or more readable storage media, where the readable storage medium includes systems, devices, or components of electricity, light, electromagnetism, infrared, or semiconductors, or any combination of the above.

[0078] The fourth embodiment provided by the present application is a computer device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete mutual communication through the communication bus: where

[0079] The memory is used to store a computer program;

[0080] The processor is used to execute the steps of the personnel recognition and warning method below the low-altitude unmanned aerial vehicle by running the program stored on the memory. As an implementation manner of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0081] As an implementation manner of the present invention, the communication interface is used for communication between the above terminal and other devices.

[0082] As an embodiment of the present invention, the memory may include a Random Access Memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0083] As an embodiment of the present invention, the aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0084] The fifth embodiment provided by this application is a method for identifying and warning people below a low-altitude unmanned aerial vehicle, including the following steps: S1. Data collection: Synchronously collect multi-modal data of the area below through a visible light camera, an infrared thermal imager, and a lidar carried on the unmanned aerial vehicle; S2. According to the real-time altitude of the unmanned aerial vehicle and the IMU attitude data, calculate the dynamic region of interest through affine transformation, and establish the mapping relationship between the image coordinate system and the ground projection area; S3. Output the recognition result: Use a parallel visual detection model and a thermal imaging segmentation model to perform target recognition on the dynamic region of interest, where the visual detection model uses an improved lightweight convolutional neural network, the thermal imaging segmentation model is based on the U-Net architecture, and the person recognition result is output through a confidence weighted fusion strategy; S4. Feature extraction: Extract the temporal pose features of the recognized person target, use the LSTM network to classify the person's movement state, and at the same time combine the Kalman filter to predict the target movement trajectory; S5. Complete the warning: Construct a dynamic risk assessment matrix according to the flight altitude of the unmanned aerial vehicle, the person density, and the movement speed, and trigger a hierarchical warning signal when the risk value exceeds the adaptive threshold. The calculation method of the dynamic region of interest in S2 is: ground width = 2 × flight altitude × tan(camera field of view angle / 2), and affine transformation is performed in combination with the elevation angle and roll angle of the unmanned aerial vehicle. The transformation formula is:

[0085]

[0088] Where θ is the attitude angle compensation amount, and dx and dy are the translation compensation amounts, which realize the real-time mapping from the image coordinate system to the actual ground coordinates. The fusion strategy in S3 includes the following steps: S301: Verify the temperature characteristics of the confidence of the human body bounding box output by the visual detection model. When the overlap rate between the 35 - 42°C feature region detected by thermal imaging segmentation and the visual detection box is greater than 70%, increase the comprehensive confidence by 0.15; S302: Start the thermal imaging dominant mode under low light conditions. When the average brightness of the visible light image is lower than 50 lux, increase the weight of the thermal imaging segmentation result to 0.7; 303: Use the D-S evidence theory to perform uncertainty reasoning on the multi-modal detection results and eliminate sensor conflicts through the basic probability assignment function. The trajectory prediction method in S4 is to establish a Kalman filter state equation:

[0089] X_k = F·X_{k - 1}+W_k

[0090] Z_k = H·X_k+V_k

[0091] Where the state vector X = [x, y, v_x, v_y]^T includes the target position and velocity, the observation matrix H fuses the lidar ranging data and the image pixel displacement amount, and the process noise W_k and the observation noise V_k dynamically adjust the covariance parameters according to the vibration amplitude of the drone. The adaptive threshold calculation method in S5 is: T = T_base×(1 + α·Δbattery)×(1 + β·wind_speed), where T_base is the reference threshold, α is the battery power attenuation coefficient (Δbattery = 1 - current power / full power), β is the wind speed influence coefficient. When the GPS signal strength is lower than -110 dBm, trigger the conservative strategy to automatically reduce T by 20%. The low-altitude drone mentioned above is a multi-rotor drone. The warning signals in S5 include issuing a voice prompt, automatic avoidance, and automatic return on the operation terminal.

[0092] In the above design, the reliability problem of the traditional drone vision system in a dynamic environment is solved, and the real-time requirements of the low-altitude security scenario are met, with significant technological progress.

[0093] For further detailed description, it should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying and warning people under a low-altitude drone, characterized by: The steps include: S1. Data collection: The visible light camera, infrared thermal imager and lidar mounted on the drone are used to synchronously collect multimodal data of the area below; S2. Establish image recognition: According to the real-time altitude and IMU attitude data of the drone, the dynamic region of interest is calculated through affine transformation, and the mapping relationship between the image coordinate system and the ground projection area is established; S3. Output recognition results: A parallel visual detection model and thermal imaging segmentation model are used to identify targets in dynamic regions of interest. The visual detection model uses an improved lightweight convolutional neural network, and the thermal imaging segmentation model is based on the U-Net architecture. The personnel recognition results are output through a confidence-weighted fusion strategy. S4, feature extraction: extract the temporal posture features of the identified personnel targets, use the LSTM network to classify the movement status of the personnel, and combine the Kalman filter to predict the target movement trajectory; S5. Complete warning: Build a dynamic risk assessment matrix based on the drone’s flight altitude, personnel density, and movement speed. When the risk value exceeds the adaptive threshold, a graded warning signal is triggered.

2. The method for identifying and warning people under a low-altitude drone according to claim 1 is characterized in that: The calculation method of the dynamic region of interest in S2 is: ground width = 2 × flight height × tan (camera field of view / 2), and affine transformation is performed in combination with the elevation angle and roll angle of the drone. The transformation formula is: [x'][cosθ-sinθtx][x] [y']=[sinθcosθty][y] [1][0 0 1][1] Where θ is the attitude angle compensation, and dx,dy is the translation compensation, which realizes the real-time mapping from the image coordinate system to the actual ground coordinates.

3. The method for identifying and warning people under a low-altitude drone according to claim 1 is characterized in that: The fusion strategy in S3 includes the following steps: S301, verify the temperature feature of the confidence of the human body bounding box output by the visual detection model. When the overlap rate between the 35-42°C feature area detected by the thermal imaging segmentation and the visual detection box is greater than 70%, the comprehensive confidence is increased by 0.15; S302: Start the thermal imaging dominant mode under low light conditions. When the average brightness of the visible light image is lower than 50 lux, the weight of the thermal imaging segmentation result is increased to 0.

7. S303, using DS evidence theory to perform uncertainty reasoning on multimodal detection results, and eliminating sensor conflicts through basic probability distribution functions.

4. The method for identifying and warning people under a low-altitude UAV according to claim 1 is characterized in that: The trajectory prediction method in S4 is to establish the Kalman filter state equation: X_k=F·X_{k-1}+W_k Z_k=H·X_k+V_k The state vector X = [x, y, v_x, v_y]^T contains the target position and velocity, the observation matrix H fuses the lidar ranging data and the image pixel displacement, and the process noise W_k and observation noise V_k dynamically adjust the covariance parameters according to the vibration amplitude of the drone.

5. The method for identifying and warning people under a low-altitude UAV according to claim 1 is characterized in that: The adaptive threshold calculation method in S5 is: T=T_base×(1+α·Δbattery)×(1+β·wind_speed), where T_base is the reference threshold, α is the battery power attenuation coefficient (Δbattery=1-current power / full power), and β is the wind speed influence coefficient. When the GPS signal strength is lower than -110dBm, the conservative strategy is triggered to automatically reduce T by 20%.

6. The method for identifying and warning people under a low-altitude UAV according to claim 1 is characterized in that: The low-altitude UAV is a multi-rotor UAV.

7. The method for identifying and warning people under a low-altitude UAV according to claim 1 is characterized in that: The warning signal of S5 includes voice prompt, automatic avoidance and automatic return at the operation end.

8. Low-altitude drone personnel identification system, characterized by: include, Data acquisition module, used to collect image data from below the drone; Real-time processing module, used to identify the target of ROI using the real-time altitude and IMU attitude data of the drone; The early warning and communication module is used to issue early warning information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is processed and executed, the steps of the method for identifying and warning people under a low-altitude unmanned aerial vehicle according to any one of claims 1 to 7 are implemented.

10. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus: The memory is used to store computer programs; The processor is used to execute the steps of the method for identifying and warning persons under a low-altitude unmanned aerial vehicle according to any one of claims 1 to 7 by running the program stored in the memory.

Images