Low-altitude flight obstacle detection method and device based on thermal imaging perception
By using thermal imaging perception and deep neural network detection models in low-altitude flight, combined with laser rangefinder, the problem of low-altitude flight obstacle detection accuracy is solved, achieving higher detection accuracy and flight safety.
Patent Information
- Application Number
- CN202411763518.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-15
AI Technical Summary
In the prior art, the detection of low-altitude flight obstacles is low, and it is difficult to effectively identify obstacles such as wires and cables in severe weather and unknown areas.
Using a low-altitude flight obstacle detection method based on thermal imaging perception, thermal images of scenes around the aircraft are collected through a thermal imaging camera and input them into an obstacle detection model based on a deep neural network, and the distance between the obstacle and the aircraft is obtained by combining a laser rangefinder.
It improves the accuracy of obstacle detection and flight safety, can effectively identify and track obstacles in bad weather, and provides real-time distance detection and alarm information.
Smart Images
Figure CN119832450A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target detection technology, and in particular to a method and device for detecting low-altitude flying obstacles based on thermal imaging perception. Background Art
[0002] Helicopters need to fly close to the ground under different visibility conditions. It is difficult for pilots to accurately identify obstacles such as wires and cables in the travel area, which may cause accidents and casualties. Relevant detection equipment is urgently needed to assist.
[0003] The Flight Alarm System (FLARM) consists of a global positioning system and a flight alarm device. It can effectively provide alarms in all weather conditions and at night, but the system requires information on the location of cables and towers in the operating area and cannot be used in unknown areas. The Wire Strike Protection System (WSPS) includes multiple cutters and is easy to install, but it cannot cut when it hits the wire at an angle less than 60 degrees, and the system only works when the speed exceeds 30 knots. The obstacle sensing radar system performs well in rain and fog, but cannot detect objects behind or directly above the aircraft. The laser obstacle sensing system consists of a scanning head mounted on the fuselage and an electronic control box in the cockpit. It can point in the direction the pilot wants to detect. It is effective in adverse weather conditions, but it is expensive and heavy.
[0004] At present, the ultra-low-altitude operation scenarios of helicopters are mostly emergency scenarios, such as heavy rain and flood disaster relief, forest fire detection, etc., and it is impossible to obtain the location of power towers and related cables in the operation area in advance. The scene is also prone to bad weather such as rain and fog, the flight speed is limited, the cutter is difficult to achieve a good effect, and the detection method needs to have good adaptability. In short, the detection accuracy of the current low-altitude flight obstacle detection method needs to be improved. Summary of the invention
[0005] The present invention provides a low-altitude flying obstacle detection method and device based on thermal imaging perception, which are used to solve the technical problem of low obstacle detection accuracy in the prior art.
[0006] In a first aspect, the present invention provides a method for detecting low-altitude flying obstacles based on thermal imaging perception, comprising the following steps.
[0007] Using a thermal imaging camera to collect thermal images of the scene around the aircraft; Inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; A laser rangefinder is used to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0008] In some embodiments, the method further comprises: A color RGB image of the scene around the aircraft is collected by using a visible light camera; the visible light camera is mounted in the optoelectronic pod; The RGB image is input into a second obstacle detection model to obtain an obstacle detection result; the second obstacle detection model is obtained by training a deep neural network model based on RGB image sample data.
[0009] In some embodiments, the method further comprises: The electronic pan-tilt platform is used to track the obstacles in the obstacle detection result and continuously detect the distance; the electronic pan-tilt platform is mounted in the optoelectronic pod.
[0010] In some embodiments, the method further comprises: Based on the internal parameters of the thermal imaging camera in the optoelectronic pod and the position of the thermal imaging camera in the optoelectronic pod, calibrating the external parameters of the thermal imaging camera; and / or Based on the intrinsic parameters of the visible light camera in the optoelectronic pod and the position of the visible light camera in the optoelectronic pod, the visible light camera is calibrated with extrinsic parameters.
[0011] In some embodiments, the first obstacle detection model is trained by the following steps: Acquire thermal imaging sample data; Performing data annotation on the thermal imaging sample data; the data annotation content includes obstacle category and obstacle detection frame position; The deep neural network model is trained based on the labeled thermal imaging sample data to obtain a trained first obstacle detection model.
[0012] In some embodiments, the method further comprises: The warning information of paying attention to obstacles is displayed on the display screen; The detected distance between the obstacle and the aircraft is displayed on a display screen.
[0013] In a second aspect, the present invention provides a low-altitude flying obstacle detection device based on thermal imaging perception, comprising the following modules.
[0014] A first acquisition module, used to acquire a thermal image of the scene around the aircraft using a thermal imaging camera; A first acquisition module, used for inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; The second acquisition module is used to use a laser rangefinder to acquire the distance between the obstacle and the aircraft in the obstacle detection result; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0015] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting low-altitude flying obstacles based on thermal imaging perception as described above is implemented.
[0016] In a fourth aspect, a non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-described methods for detecting low-altitude flying obstacles based on thermal imaging perception.
[0017] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the low-altitude flying obstacle detection method based on thermal imaging perception as described in the first aspect above.
[0018] The low-altitude flight obstacle detection method and device based on thermal imaging perception provided by the present invention uses a thermal imaging camera to collect thermal images of the scene around the aircraft, and the thermal imaging camera is mounted in the optoelectronic pod of the aircraft; the thermal image is input into a first obstacle detection model to obtain an obstacle detection result, and the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; then a laser rangefinder is used to obtain the distance between the obstacle and the aircraft in the obstacle detection result. Obstacle detection based on thermal imaging perception combined with a deep neural network detection model improves detection accuracy and flight safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 It is one of the flow charts of the low-altitude flying obstacle detection method based on thermal imaging perception provided by the present invention.
[0021] Figure 2 This is the second flow chart of the low-altitude flying obstacle detection method based on thermal imaging perception provided by the present invention.
[0022] Figure 3 It is a schematic diagram of the overall architecture of low-altitude flying obstacle detection based on thermal imaging perception provided by the present invention.
[0023] Figure 4 It is a structural schematic diagram of a low-altitude flying obstacle detection device based on thermal imaging perception provided by the present invention.
[0024] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Figure 1 This is one of the flow charts of the low-altitude flight obstacle detection method based on thermal imaging perception provided by the present invention, such as Figure 1 As shown, the present invention provides a low-altitude flying obstacle detection method based on thermal imaging perception. The method comprises: Step 101: Use a thermal imaging camera to collect a thermal image of the scene around the aircraft.
[0027] Specifically, thermal imaging infrared cameras mainly use the thermal radiation characteristics of obstacles to collect thermal images of the scene in front of or around the aircraft. In rainy, snowy and foggy weather with limited visibility, thermal radiation can have better penetration ability. In the high-voltage cable scene, which is the most risky in the aircraft operation scene, its high voltage and high temperature characteristics are better displayed in the thermal image of the thermal imaging camera, which is convenient for the subsequent detection of obstacles such as cables.
[0028] Step 102: input the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data.
[0029] Specifically, due to the changeable posture of the aircraft in the air, the cable shapes in the collected high-voltage cable data are also different, and it is difficult to use a unified set of simple features for detection. Therefore, a detection method based on a deep neural network model is adopted.
[0030] The first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data, and is used to detect obstacles in thermal images. By training the model with a large amount of thermal imaging sample data, the recognition accuracy of various obstacles such as various high-voltage cables can be improved. The deep neural network model can be a YOLO (You Only Look Once) model.
[0031] The thermal image captured by the thermal imaging camera is input into the trained first obstacle detection model, obstacles in the thermal image are detected, and obstacle detection results are output, which can be the pixel positions of each obstacle detected in the thermal image or the target detection frame.
[0032] Step 103: using a laser rangefinder to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in an optoelectronic pod of the aircraft.
[0033] Specifically, the laser rangefinder uses a high-penetration laser transmitter to detect the distance of obstacles in front of the aircraft and obtain the distance between the obstacle and the aircraft, which makes it easier for the pilot to choose a more appropriate avoidance strategy.
[0034] The thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0035] The low-altitude flying obstacle detection method based on thermal imaging perception provided in the embodiment of the present application has high distinguishability in an environment with air as the background, because high-voltage cables and the like have strong thermal sensitivity and their temperature can reach 70-90°C. Since the thermal radiation characteristics have strong penetrating power, the thermal imaging camera can be used to collect data on high-voltage cables at a long distance, thereby achieving accurate obstacle detection. Moreover, by adopting a detection method based on deep learning, the accuracy of obstacle detection is improved, and the processing speed is fast, which can achieve the effect of real-time early warning prompts and realize real-time perception of high-voltage cables and the like.
[0036] In some embodiments, the first obstacle detection model is trained by the following steps: Acquire thermal imaging sample data; Performing data annotation on the thermal imaging sample data; the data annotation content includes obstacle category and obstacle detection frame position; The deep neural network model is trained based on the labeled thermal imaging sample data to obtain a trained first obstacle detection model.
[0037] Specifically, in the training phase of the first obstacle detection model, a deep learning model based on a YOLO model such as a YOLO-v8 model architecture can be used to detect obstacles such as cables in thermal images and frame them using a multi-scale target frame. The specific process is as follows: First, a large amount of thermal imaging sample data is collected and annotated using annotation tools. The annotation content includes obstacle categories (such as cables) and detection frame locations.
[0038] Then, the labeled data is divided into training set and validation set according to a certain ratio, trained using the YOLO-v8 deep neural network model, and verified using the validation set to avoid overfitting.
[0039] Finally, a trained model is obtained, namely the first obstacle detection model.
[0040] The low-altitude flight obstacle detection method based on thermal imaging perception provided in the embodiment of the present application improves the recognition accuracy of obstacles such as various high-voltage cables by training the model with thermal imaging sample data, thereby reminding flight personnel to accurately avoid obstacles and improving the safety of aircraft flight.
[0041] In some embodiments, the method further comprises: A color RGB image of the scene around the aircraft is collected by using a visible light camera; the visible light camera is mounted in the optoelectronic pod; The RGB image is input into a second obstacle detection model to obtain an obstacle detection result; the second obstacle detection model is obtained by training a deep neural network model based on RGB image sample data.
[0042] Specifically, the aerial environment of an aircraft is relatively complex, and it is difficult to adapt to perception tasks in different weather scenarios through a single sensor. Therefore, an optoelectronic pod with multi-modal acquisition functions can be used, mainly including visible light cameras, thermal imaging infrared cameras, and laser rangefinders.
[0043] Among them, the visible light camera realizes obstacle detection in scenes with good visibility. It can collect RGB images in front of or around the aircraft to collect image data of conventional obstacles such as power towers, mountains, etc.
[0044] The visible light camera is also mounted in the optoelectronic pod. The optoelectronic pod is equipped with a variety of sensors to achieve multi-modal data collection, so that the aircraft can complete the data collection task in a more complex real environment, laying a good foundation for the subsequent detection algorithm.
[0045] The collected RGB image is input into the trained second obstacle detection model, which is used to identify obstacles in the RGB image and output obstacle detection results, i.e., the pixel position of each obstacle in the RGB image or the target detection frame.
[0046] In the embodiment of the present application, in the training stage of the second obstacle detection model, a deep learning model based on a YOLO model such as a YOLO-v8 model architecture can be used to detect obstacles such as electric towers in thermal images, and a multi-scale target frame is used to frame them. The specific process is as follows: First, a large amount of RGB image sample data is collected and annotated using annotation tools. The annotation content includes the obstacle category (such as power towers) and the detection frame position of the obstacle.
[0047] Then, the labeled data is divided into training set and validation set according to a certain ratio, trained using the YOLO-v8 deep neural network model, and verified using the validation set to avoid overfitting.
[0048] Finally, a trained model is obtained, namely the second obstacle detection model.
[0049] The low-altitude flight obstacle detection method based on thermal imaging perception provided in the embodiment of the present application detects obstacles such as power towers based on a visible light camera, and combines obstacle detection based on an infrared camera to improve the overall flight safety of the aircraft.
[0050] In some embodiments, the method further comprises: The electronic pan-tilt platform is used to track the obstacles in the obstacle detection result and continuously detect the distance; the electronic pan-tilt platform is mounted in the optoelectronic pod.
[0051] Specifically, after detecting an obstacle, the electronic gimbal of the optoelectronic pod can track and detect the obstacle, and continuously output the situation and distance of the obstacle.
[0052] The low-altitude flight obstacle detection method based on thermal imaging perception provided in the embodiment of the present application facilitates the pilot's grasp of the obstacle situation and improves the usability of the system by tracking and updating the distance of the detected obstacles.
[0053] In some embodiments, the method further comprises: Based on the internal parameters of the thermal imaging camera in the optoelectronic pod and the position of the thermal imaging camera in the optoelectronic pod, calibrating the external parameters of the thermal imaging camera; and / or Based on the intrinsic parameters of the visible light camera in the optoelectronic pod and the position of the visible light camera in the optoelectronic pod, the visible light camera is calibrated with extrinsic parameters.
[0054] Specifically, before using the laser rangefinder to obtain the distance between the detected obstacle and the aircraft, it is necessary to calibrate the external parameters of each sensor in the optoelectronic pod. For example, based on the internal parameters of the thermal imaging camera in the optoelectronic pod and the position of the thermal imaging camera in the optoelectronic pod, the external parameters of the thermal imaging camera are calibrated; based on the internal parameters of the visible light camera in the optoelectronic pod and the position of the visible light camera in the optoelectronic pod, the external parameters of the visible light camera are calibrated.
[0055] The low-altitude flying obstacle detection method based on thermal imaging perception provided in the embodiment of the present application realizes the unification of optical axes between the sensors by calibrating the external parameters of the various sensors in the optoelectronic pod, achieves data synchronization of multiple sensors, and facilitates the laser rangefinder to detect the distance to the obstacle.
[0056] In some embodiments, the method further comprises: The warning information of paying attention to obstacles is displayed on the display screen; The detected distance between the obstacle and the aircraft is displayed on a display screen.
[0057] Specifically, the display screen may be a display mounted on an aircraft, for displaying various information to the pilot, including warning information about obstacles, the detected distance between the obstacles and the aircraft, etc.
[0058] The display screen may also be a display screen on a ground controller that controls the unmanned aerial vehicle, which is used to display various information to ground operators, including warning information about obstacles, the detected distance between the obstacles and the aircraft, etc.
[0059] The low-altitude flight obstacle detection method based on thermal imaging perception provided in the embodiment of the present application displays the obstacle detection status in real time through a display screen, reminds the driver or operator to avoid the obstacle, and improves the safety of flight.
[0060] In addition to supporting real-time detection on helicopter equipment, the embodiments of the present application can also be extended to drones for offline applications. In some emergency scenarios, multiple drones will be used to scan the environment and build maps of the work area. Through the color images and thermal imaging image data collected by the drones, this technology can detect electric towers and cables offline. Combined with the drone's own positioning, it can be annotated into the electronic map constructed by the drone, providing a data basis for the helicopter's path planning and improving overall safety.
[0061] Figure 2 This is the second flow chart of the low-altitude flight obstacle detection method based on thermal imaging perception provided by the present invention, such as Figure 2As shown, thermal imaging infrared cameras and visible light cameras can be used to detect obstacles together. The color camera (visible light camera) is used to obtain the image data of the electric tower (i.e., RGB image), and the image data of the electric tower is input into the trained electric tower detection model (i.e., the second obstacle detection model) to detect the electric tower at a longer distance; at the same time, the thermal imaging camera is used to obtain the thermal imaging data of the cable (i.e., the thermosensitive image), and the thermal imaging data of the cable is input into the trained cable detection model (i.e., the first obstacle detection model) to detect the high-voltage cables that appear within the dangerous distance. In good weather conditions, the electric tower 2 kilometers away can be identified, and in harsh environments, the cables 500 meters away can be accurately detected. The optoelectronic pod rotates the viewing angle to the direction of the high-voltage cable according to the three-dimensional map, and continuously detects and tracks the cable area synchronously through the perception algorithm. Combined with the obstacle distance information obtained by the laser rangefinder, the onboard display screen is used to issue an audible and visual warning, and the high-voltage cables that appear within the dangerous distance are continuously warned to remind the driver to avoid them, thereby improving flight safety.
[0062] Figure 3 is a schematic diagram of the overall architecture of low-altitude flight obstacle detection based on thermal imaging perception provided by the present invention, such as Figure 3 As shown, at the perception end (i.e., in the optoelectronic pod), a color camera, a thermal imaging camera, and a laser rangefinder are used to form a multimodal acquisition module. At the detection end, the detection function is realized through an edge computing module. At the alarm display end, display and alarm are performed through a portable display screen.
[0063] The low-altitude flight obstacle detection method based on thermal imaging perception provided in the embodiment of the present application, the low-altitude flight obstacle detection technology based on thermal imaging perception, through the color image and thermal imaging image data of the helicopter optoelectronic pod, combined with the deep neural network detection model and the tracking logic of the optoelectronic pod, realizes the detection of high-voltage towers and high-voltage cables in the flight area, has high accuracy in obstacle detection, and has generalization ability, and the equipment weight is low, and the modification difficulty for existing helicopters is relatively small, and it has high promotion. Moreover, the embodiment of the present application has stronger pertinence and scalability, and can realize the detection of towers 2 kilometers away and cables 500 meters away, and can be applied in drone scenarios.
[0064] Figure 4 is a schematic diagram of the structure of a low-altitude flight obstacle detection device based on thermal imaging perception provided by the present invention, such as Figure 4 As shown, the present invention provides a low-altitude flying obstacle detection device based on thermal imaging perception, including a first acquisition module 401, a first acquisition module 402 and a second acquisition module 403.
[0065] The first acquisition module 401 is used to acquire a thermal image of the scene around the aircraft using a thermal imaging camera.
[0066] The first acquisition module 402 is used to input the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data.
[0067] The second acquisition module 403 is used to obtain the distance between the obstacle and the aircraft in the obstacle detection result by using a laser rangefinder; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0068] In some embodiments, it also includes: A second acquisition module is used to acquire a color RGB image of the scene around the aircraft using a visible light camera; the visible light camera is mounted in the optoelectronic pod; The third acquisition module is used to input the RGB image into a second obstacle detection model to obtain an obstacle detection result; the second obstacle detection model is obtained by training a deep neural network model based on RGB image sample data.
[0069] In some embodiments, it also includes: The tracking and detection module is used to track the obstacles in the obstacle detection result and continuously detect the distance by using the electronic pan-tilt platform; the electronic pan-tilt platform is mounted in the optoelectronic pod.
[0070] In some embodiments, it also includes: A first calibration module, configured to perform external parameter calibration on the thermal imaging camera in the optoelectronic pod based on the intrinsic parameters of the thermal imaging camera and the position of the thermal imaging camera in the optoelectronic pod; and / or The second calibration module is used to perform external parameter calibration on the visible light camera in the optoelectronic pod based on the internal parameters of the visible light camera and the position of the visible light camera in the optoelectronic pod.
[0071] In some embodiments, the first obstacle detection model is trained by the following steps: Acquire thermal imaging sample data; Performing data annotation on the thermal imaging sample data; the data annotation content includes obstacle category and obstacle detection frame position; The deep neural network model is trained based on the labeled thermal imaging sample data to obtain a trained first obstacle detection model.
[0072] In some embodiments, it also includes: A first display module is used to display warning information of obstacles through a display screen; The second display module is used to display the detected distance between the obstacle and the aircraft through a display screen.
[0073] Specifically, the above-mentioned low-altitude flying obstacle detection device based on thermal imaging perception provided by the present invention can implement all the method steps implemented by the above-mentioned low-altitude flying obstacle detection method embodiment based on thermal imaging perception, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.
[0074] It should be noted that the division of units / modules in the above-mentioned embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.
[0075] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communications interface 502 and the memory 503 communicate with each other through the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute the low-altitude flight obstacle detection method based on thermal imaging perception, and the method includes: Using a thermal imaging camera to collect thermal images of the scene around the aircraft; Inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; A laser rangefinder is used to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0076] In some embodiments, the method further comprises: A color RGB image of the scene around the aircraft is collected by using a visible light camera; the visible light camera is mounted in the optoelectronic pod; The RGB image is input into a second obstacle detection model to obtain an obstacle detection result; the second obstacle detection model is obtained by training a deep neural network model based on RGB image sample data.
[0077] In some embodiments, the method further comprises: The electronic pan-tilt platform is used to track the obstacles in the obstacle detection result and continuously detect the distance; the electronic pan-tilt platform is mounted in the optoelectronic pod.
[0078] In some embodiments, the method further comprises: Based on the internal parameters of the thermal imaging camera in the optoelectronic pod and the position of the thermal imaging camera in the optoelectronic pod, calibrating the external parameters of the thermal imaging camera; and / or Based on the intrinsic parameters of the visible light camera in the optoelectronic pod and the position of the visible light camera in the optoelectronic pod, the visible light camera is calibrated with extrinsic parameters.
[0079] In some embodiments, the first obstacle detection model is trained by the following steps: Acquire thermal imaging sample data; Performing data annotation on the thermal imaging sample data; the data annotation content includes obstacle category and obstacle detection frame position; The deep neural network model is trained based on the labeled thermal imaging sample data to obtain a trained first obstacle detection model.
[0080] In some embodiments, the method further comprises: The warning information of paying attention to obstacles is displayed on the display screen; The detected distance between the obstacle and the aircraft is displayed on a display screen.
[0081] Specifically, the processor 501 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD), and the processor may also adopt a multi-core architecture.
[0082] The logic instructions in the memory 503 can be implemented in the form of software functional units and can be stored in a processor-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0083] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the low-altitude flight obstacle detection method based on thermal imaging perception provided by the above-mentioned method embodiments, the method comprising: Using a thermal imaging camera to collect thermal images of the scene around the aircraft; Inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; A laser rangefinder is used to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0084] Specifically, the above-mentioned computer program product provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0085] In some embodiments, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to enable a computer to execute the low-altitude flight obstacle detection method based on thermal imaging perception provided by the above-mentioned method embodiments, the method comprising: Using a thermal imaging camera to collect thermal images of the scene around the aircraft; Inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; A laser rangefinder is used to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
[0086] Specifically, the above-mentioned computer-readable storage medium provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effects. The parts and beneficial effects that are the same as the method embodiments in this embodiment will not be described in detail here.
[0087] It should be noted that the computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.
[0088] It should also be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more.
[0089] In the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0090] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.
[0091] In the present invention, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for another example, "when A meets the second condition, determine B", etc.; for another example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.
[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.
[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0094] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0095] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0096] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A low-altitude flying obstacle detection method based on thermal imaging perception, characterized in that: include: Using a thermal imaging camera to collect thermal images of the scene around the aircraft; Inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; The first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; A laser rangefinder is used to obtain the distance between the obstacle in the obstacle detection result and the aircraft; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
2. The low-altitude flying obstacle detection method based on thermal imaging perception according to claim 1 is characterized in that: The method further comprises: A color RGB image of the scene around the aircraft is collected by using a visible light camera; the visible light camera is mounted in the optoelectronic pod; The RGB image is input into a second obstacle detection model to obtain an obstacle detection result; the second obstacle detection model is obtained by training a deep neural network model based on RGB image sample data.
3. The low-altitude flying obstacle detection method based on thermal imaging perception according to claim 1 or 2 is characterized in that: The method further comprises: The electronic pan-tilt platform is used to track the obstacles in the obstacle detection result and continuously detect the distance; the electronic pan-tilt platform is mounted in the optoelectronic pod.
4. The low-altitude flying obstacle detection method based on thermal imaging perception according to claim 1 is characterized in that: The method further comprises: Based on the internal parameters of the thermal imaging camera in the optoelectronic pod and the position of the thermal imaging camera in the optoelectronic pod, calibrating the external parameters of the thermal imaging camera; and / or Based on the intrinsic parameters of the visible light camera in the optoelectronic pod and the position of the visible light camera in the optoelectronic pod, the visible light camera is calibrated with extrinsic parameters.
5. The low-altitude flying obstacle detection method based on thermal imaging perception according to claim 1 is characterized in that: The first obstacle detection model is trained by the following steps: Acquire thermal imaging sample data; Performing data annotation on the thermal imaging sample data; the data annotation content includes obstacle category and obstacle detection frame position; The deep neural network model is trained based on the labeled thermal imaging sample data to obtain a trained first obstacle detection model.
6. The low-altitude flying obstacle detection method based on thermal imaging perception according to claim 1 is characterized in that: The method further comprises: The warning information of paying attention to obstacles is displayed on the display screen; The detected distance between the obstacle and the aircraft is displayed on a display screen.
7. A low-altitude flying obstacle detection device based on thermal imaging perception, characterized in that: include: A first acquisition module, used to acquire a thermal image of the scene around the aircraft using a thermal imaging camera; A first acquisition module, used for inputting the thermal image into a first obstacle detection model to obtain an obstacle detection result; the first obstacle detection model is obtained by training a deep neural network model based on thermal imaging sample data; The second acquisition module is used to use a laser rangefinder to acquire the distance between the obstacle and the aircraft in the obstacle detection result; the thermal imaging camera and the laser rangefinder are both mounted in the optoelectronic pod of the aircraft.
8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting low-altitude flying obstacles based on thermal imaging perception as claimed in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for detecting low-altitude flying obstacles based on thermal imaging perception as described in any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the low-altitude flying obstacle detection method based on thermal imaging perception as described in any one of claims 1 to 6.