A data fusion-based inspection and adaptive landing drone and its usage method
By combining multi-sensor data fusion and deep learning algorithms with adaptive landing technology, the accuracy problem of detecting stray sheep and newborn lambs by drones in grassland pastures has been solved, achieving smooth landing and efficient detection on complex terrain.
Patent Information
- Application Number
- CN202410991868.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Traditional drones are ineffective at detecting stray sheep and newborn lambs in grasslands and pastures, and they are difficult to land smoothly on complex terrain, affecting detection accuracy and equipment stability.
Employing multi-sensor data fusion and deep learning neural network technology, combined with RGB cameras, multi-view thermal infrared cameras, and multispectral cameras, adaptive landing is achieved through a bionic claw and a multi-level telescopic support. Target detection and instance segmentation are performed using Faster R-CNN and Mask R-CNN models to optimize the landing process.
It improves the landing stability of UAVs on uneven ground and the accuracy of target detection, extends the service life of equipment, and enhances the robustness and adaptability of the model.
Smart Images

Figure CN118811149B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of multi-sensor data fusion, deep learning neural network algorithm recognition and analysis, and UAV buffer landing technology, and particularly to a data fusion-based inspection and adaptive landing UAV and its usage method. Background Technology
[0002] With the rapid development of drone technology, its application in agriculture is becoming increasingly widespread. Especially in grassland and pasture management, drones have become a highly promising tool due to their flexibility and efficiency. However, traditional drone inspection systems are often affected by the terrain and pasture when facing complex terrain and changing environments, making it difficult to achieve a smooth landing and maintain aircraft stability, which limits their effectiveness in grassland and pasture applications. Meanwhile, the combination of drone technology and artificial intelligence technology has become a key force driving progress in multiple industries. Drones equipped with thermal infrared and multispectral sensors and high-performance host computers enable real-time data acquisition and analysis, providing unprecedented possibilities for precision agriculture and monitoring.
[0003] In traditional grassland pasture management, sheep grazing is a process reliant on the natural environment and the experience of the herders. However, this process often results in sheep getting lost or newborn lambs not being found in time for various reasons. Lost sheep may be unable to return to the flock due to getting lost, illness, or other factors, while newborn lambs may face survival threats due to neglect by their mothers or severe weather. These situations not only pose risks to the sheep themselves but also impact the economic benefits and ecological balance of the pasture.
[0004] Therefore, in the face of complex and ever-changing natural environments and unpredictable threats to life, how to effectively integrate multi-source sensor data and use deep learning neural network technology for efficient and accurate analysis, so as to improve the accuracy and robustness of UAVs in detecting and identifying stray sheep and newborn lambs in grasslands and pastures, and the stability of adaptive and smooth landing in complex environments, remains an urgent problem to be solved in the current technological development. Summary of the Invention
[0005] The purpose of this invention is to provide a data-fusion-based inspection and adaptive landing drone and its usage method to solve various problems faced in the above-mentioned operational scenarios.
[0006] To achieve the above objectives, this application provides the following technical solution: A data-fusion-based inspection and adaptive landing drone includes: The device comprises a fuselage, landing support frames, RGB cameras, and a buffer bionic claw. The fuselage is connected to four landing support frames via rotating shafts. Inside the fuselage are a lithium battery, a Jetson Orin NX industrial computer, and an STM32F427IIH6 core control board. RGB cameras, a multi-view thermal infrared camera, and a multispectral camera are mounted at the front, middle, and rear of the fuselage bottom, respectively. The bottom of the fuselage has large heat dissipation holes, side heat dissipation holes, and circular heat dissipation holes. The landing support frame consists of a high-speed motor, a drive gear, a transmission gear, ultrasonic sensors, a multi-stage telescopic support, and a strong magnet. The high-speed motor is connected to the STM32F427IIH6 core control board and a lithium battery via internal data cables, enabling power and control connections. The high-speed motor's rotation transmits power to the transmission gears, thus rotating the UAV's rotor. Four ultrasonic sensors detect the drone's altitude from the ground in real time and return the data to the STM32F427IIH6 core control board, which then transmits it to a Jetson Orin NX industrial computer for analysis. The state of the multi-stage telescopic support is controlled by internal springs. The compression and extension of these springs are controlled by servo motors, which are controlled by the STM32F427IIH6 core control board via data cables. The servo motors have a wound housing at their front end; when the servo motor rotates, it pulls the springs via a cable, compressing them and causing the multi-stage telescopic support to retract, thus achieving lifting and cushioning control of the landing support frame.
[0007] The buffer bionic claw consists of a omnidirectional ball, a support base, and four support grippers. The omnidirectional ball can rotate at multiple angles, and the support grippers can open and close. Together, they can provide more stable support points for the drone landing.
[0008] The RGB camera is fixed to the lower front of the body via a rotating shaft and can capture RGB images at a resolution of 5280×3956. The multi-view thermal infrared camera is fixed to the middle of the bottom of the body and protected by a transparent protective cover. It has three independent detection lenses that can work simultaneously, providing a wider field of view and more detailed target analysis. It reconstructs the three-dimensional shape of the target by analyzing infrared images from different perspectives, and the three detection lenses can cross-verify the data, improving the stability and reliability of thermal infrared images.
[0009] The multispectral camera is attached to the bottom rear of the body via a magnetic chuck. It has five lenses: a red band lens, a green band lens, a red edge band lens, a blue band lens, and a near-infrared band lens. These lenses can distinguish and analyze the reflection characteristics of objects at different wavelengths, acquire images in five specific spectral ranges, and quickly transmit the data to a Jetson Orin NX industrial computer for analysis via a data cable.
[0010] The rotation of the servo motors is controlled by the STM32F427IIH6 core control board. When the drone lands, the four support grippers of the bionic claw first make contact with the ground. The support grippers slowly open to provide the first level of buffer. Then, the ultrasonic sensor monitors the altitude above the ground in real time and transmits the data to the STM32F427IIH6 core control board. The board calculates the length of the four multi-stage telescopic supports that keep the drone absolutely level. This data is then converted into the rotation angle of the servo motors after calculation, providing the second level of buffer when the drone lands on uneven ground. After landing, the drone can still maintain absolute levelness and protect its internal control modules.
[0011] A method for using a drone based on data fusion-based inspection and adaptive landing, including The Jetson Orin NX industrial computer is equipped with a real-time fusion analysis algorithm for multi-sensor data. It fuses the semantic segmentation algorithm Mask R-CNN and the object detection algorithm Faster R-CNN to form the MF R-CNN algorithm model. It integrates data from RGB cameras, multi-view thermal infrared cameras and multispectral cameras for real-time monitoring and analysis, determines the accurate geographical location of stray sheep and newborn lambs, segments the shape of sheep on pasture, and constructs their 3D model based on depth information.
[0012] The fused MF R-CNN algorithm model uses Faster R-CNN for object detection during feature extraction, which includes a Region Proposal Network (RPN) and a Classification and Bounding Box Regression Network. It uses Mask R-CNN for instance segmentation, which adds an extra branch to Faster R-CNN to predict the pixel-level mask for each proposed region. The fused MF R-CNN algorithm model uses a shared ResNet feature extractor to extract features from multi-source input images during data fusion, and is shared by the Faster R-CNN and Mask R-CNN models to reduce computational redundancy. Candidate regions are generated using the RPN of Faster R-CNN and used for classification and bounding box regression in Faster R-CNN, and also as input to Mask R-CNN; During model training, Faster R-CNN is trained using labeled bounding box data to identify bounding boxes of sheep and lambs, and Mask R-CNN is trained using labeled bounding box and segmentation mask data to generate more accurate instance segmentation. The loss functions for classification, bounding box regression and segmentation mask are optimized during training. When fusing outputs, a fusion strategy is designed by combining the classification confidence of Faster R-CNN and the segmentation quality of Mask R-CNN. A weighted average is used to comprehensively consider the output proportions of the two models, and finally, the IoU (Intersection over Union) is used to select the optimal bounding box. In post-processing, NMS is used to remove overlapping detection results to ensure the alignment of each target detection box, and an appropriate IoU threshold is set to control the execution of NMS. At the same time, the segmentation mask generated by Mask R-CNN is processed to remove small noise regions and fill holes to ensure the accuracy and integrity of the segmentation mask.
[0013] During real-time processing optimization, the MFR-CNN model is first pruned and quantized to reduce model size and inference time. A dedicated hardware accelerator GPU is used to improve real-time performance. A parallel processing strategy is designed to utilize a multi-core CPU to process multiple candidate regions in parallel, optimize data flow and computation process, reduce latency, and finally achieve high-precision bounding box selection and localization, and transmit the image to the user's PC interface.
[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. When the drone lands, the four support grips of the bionic claw first contact the ground and provide the first level of cushioning. Then, the automatic adjustment of the multi-level telescopic brackets provides the second level of cushioning. This multi-level cushioning mechanism significantly improves the stability of the drone when landing on uneven ground. It ensures that the drone can keep its fuselage level even when encountering bumpy or uneven ground during landing, thereby effectively protecting the sensitive components of the internal lithium battery, industrial control computer and core control board, and extending the service life of the equipment.
[0015] 2. Faster R-CNN excels at generating object bounding boxes quickly and accurately, while Mask R-CNN provides fine-grained pixel-level instance segmentation. The segmentation results from Mask R-CNN can provide Faster R-CNN with richer contextual information. By fusing these two models, their strengths can be fully utilized to improve the overall accuracy of object detection and instance segmentation. The fused model can perform object detection and instance segmentation simultaneously within the same framework, helping the model learn richer feature representations and better understand the shape and structure of objects, further improving detection accuracy.
[0016] 3. Fusion models can utilize diverse data information. More data information can provide higher-resolution bounding boxes and segmentation masks, which helps the model maintain robustness when facing complex scenarios and diverse targets. Through continuous data collection and model iteration, fusion models can continuously improve their generalization ability, enabling them to better adapt to new scenarios and task requirements.
[0017] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0019] Figure 1 This is a schematic diagram of the overall structure of a foldable quadcopter drone according to one embodiment of the present invention; Figure 2 This is a schematic diagram showing the positional relationship of the core components inside the fuselage in one embodiment of the present invention; Figure 3 This is a schematic diagram of the landing support frame in one embodiment of the present invention; Figure 4 This is a supplementary schematic diagram of the landing support frame structure in one embodiment of the present invention; Figure 5 This is a schematic diagram of the sensor distribution at the bottom of the fuselage in one embodiment of the present invention; Figure 6 This is a schematic diagram of a multispectral sensor structure in one embodiment of the present invention; Figure 7 This is a schematic diagram of a buffer bionic claw structure in one embodiment of the present invention; Figure 8 This is a schematic diagram of a multi-stage telescopic support structure in one embodiment of the present invention; Figure 9 This is a technical roadmap for drone inspection operations in one embodiment of the present invention; Figure 10 This is a flowchart of the data processing of the MF R-CNN fusion model in one embodiment of the present invention; Figure 11 This is a flowchart of an adaptive buffer landing process for a drone in one embodiment of the present invention.
[0020] The meanings of the labels in the attached diagram are as follows: 1-Fuse, 2-Descent support frame, 3-RGB camera, 4-Bionic buffer claw; 1-1 Rotating shaft, 1-2 Lithium battery, 1-3 Jetson Orin NX industrial computer, 1-4 STM32F427IIH6 core control board, 1-5 Multispectral camera, 1-6 Large heat dissipation hole, 1-7 Multi-view thermal infrared camera, 1-8 Circular heat dissipation hole, 1-9 Side heat dissipation hole; 1-5-1 Red band lens, 1-5-2 Near-infrared band lens, 1-5-3 Red edge band lens, 1-5-4 Magnetic chuck, 1-5-5 Green band lens, 1-5-6 Blue band lens; 2-1 Rotating hole; 2-2 Drive gear; 2-3 Transmission gear; 2-4 Ultrasonic sensor; 2-5 Multi-stage telescopic bracket; 2-6 Strong magnet; 2-7 High-speed motor; 2-8 Servo motor; 2-9 Winding housing; 2-10 Pull cord; 2-11 Spring; 2-12 Data cable; 4-1 Omnidirectional ball, 4-2 Support base, 4-3 Support grip; Detailed Implementation
[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0025] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.
[0026] A data fusion-based inspection and adaptive landing drone includes: a fuselage 1, landing support frames 2, RGB cameras 3, and a buffer bionic claw 4. The fuselage 1 is connected to four landing support frames 2 via rotating shafts 1-1. The fuselage 1 contains a lithium battery 1-2, a Jetson Orin NX industrial computer 1-3, and an STM32F427IIH6 core control board 1-4. The bottom of the fuselage 1 is equipped with an RGB camera 3 at the front, middle, and rear, a multi-view thermal infrared camera 1-7, and a multispectral camera 1-5, respectively. The bottom of the fuselage 1 has a large heat dissipation hole 1-6, a side heat dissipation hole 1-9, and a circular heat dissipation hole 1-8.
[0027] The landing support frame 2 consists of a high-speed motor 2-7, a drive gear 2-2, a transmission gear 2-3, ultrasonic sensors 2-4, a multi-stage telescopic support 2-5, and a strong magnet 2-6. The high-speed motor 2-7 is connected to the STM32F427IIH6 core control board 1-4 and the lithium battery 1-2 via internal data cables, establishing power and control connections. After the high-speed motor 2-7 rotates, it transmits power to the transmission gear 2-3 through gear meshing, thereby rotating the UAV rotor. The four ultrasonic sensors 2-4 can detect the drone's altitude from the ground in real time and return the data to the STM32F427IIH6 core control board 1-4, which then transmits it to the Jetson Orin. The NX industrial computer 1-3 performs data analysis; the state of the multi-stage telescopic support 2-5 is controlled by an internal spring, and the compression and extension of the spring 2-11 are controlled by the servo motor 2-8. The STM32F427IIH6 core control board 1-4 controls the rotation of the servo motor 2-8 through the data cable 2-7; the front end of the servo motor 2-8 has a winding shell 2-9. When the servo motor 2-8 rotates, it pulls the spring 2-11 through the pull rope 2-10, causing the spring 2-11 to compress, which in turn causes the multi-stage telescopic support 2-5 to retract, thereby realizing the lifting and buffering control of the landing support frame 2.
[0028] The buffer bionic claw 4 consists of a omnidirectional ball 4-1, a support base 4-2, and four support grippers 4-3. The omnidirectional ball 4-1 can rotate at multiple angles, and the support grippers 4-3 can open and close. Together, they can provide more stable support points for the drone landing.
[0029] The RGB camera 3 is fixed to the lower front of the body 1 via a rotating shaft and can capture RGB images with a resolution of 5280×3956. The multi-view thermal infrared cameras 1-7 are fixed to the middle of the bottom of the body 1 and protected by a transparent protective cover. They have three independent detection lenses that can work simultaneously, providing a wider field of view and more detailed target analysis. They reconstruct the three-dimensional shape of the target by analyzing infrared images from different perspectives, and the three detection lenses can cross-check the data to improve the stability and reliability of thermal infrared images.
[0030] The multispectral camera 1-5 is attached to the bottom rear of the body 1 via a magnetic chuck 1-5-4. It has five lenses: a red band lens 1-5-1, a green band lens 1-5-5, a red edge band lens 1-5-3, a blue band lens 1-5-6, and a near-infrared band lens 1-5-2. These lenses can distinguish and analyze the reflection characteristics of objects at different wavelengths, acquire images of five specific spectral ranges, and quickly transmit the data to the Jetson Orin NX industrial computer 1-3 for analysis via a data cable.
[0031] The rotation of servo motor 2-8 is controlled by STM32F427IIH6 core control board 1-4. When the UAV lands, the four support grips 4-3 of the bionic claw 4 first contact the ground. The support grips 4-3 slowly open to provide the first level of buffer. Then, the ultrasonic sensor 2-4 monitors the altitude above the ground in real time and transmits the data to the STM32F427IIH6 core control board 1-4. The board calculates the length of the four multi-stage telescopic brackets 2-5 that keep the fuselage 1 absolutely level. This data is then converted into the rotation angle of servo motor 2-8 after calculation, providing the second level of buffer when the UAV lands on uneven ground. After landing, the UAV can still maintain the absolute level of the fuselage 1, protecting its internal control module.
[0032] A method for using a drone based on data fusion-based inspection and adaptive landing includes: The Jetson Orin NX industrial PC 1-3 is equipped with a real-time fusion analysis algorithm for multi-sensor data. It fuses the semantic segmentation algorithm Mask R-CNN and the object detection algorithm Faster R-CNN to form the MF R-CNN algorithm model. It also fuses data from RGB camera 3, multi-view thermal infrared camera 1-7 and multispectral camera 1-5 for real-time monitoring and analysis. This allows it to determine the accurate geographical location of stray sheep and newborn lambs, segment the shape of sheep on the pasture, and construct their 3D model based on depth information.
[0033] The fused MF R-CNN algorithm model inputs weighted thermal infrared camera data, multispectral camera data, and high-resolution RGB image data into the network model during feature extraction. It uses Faster R-CNN for object detection, which includes a Region Proposal Network (RPN) and a classification and bounding box regression network. It uses Mask R-CNN for instance segmentation, which adds an extra branch to Faster R-CNN to predict the pixel-level mask for each proposed region.
[0034] This invention discloses a method for using a drone based on data fusion-based inspection and adaptive landing, which further includes the following steps: S1, the thermal infrared camera data, multispectral camera data and high-definition RGB image data of grassland pastures taken under different times and weather conditions are weighted and input into the network model. During data fusion, the MFR-CNN algorithm model uses a shared ResNet feature extractor to extract features from the multi-source input images and shares them with the Faster R-CNN and Mask R-CNN models to reduce computational redundancy. S2 uses Faster R-CNN's RPN to generate candidate regions, which are then used for classification and bounding box regression in Faster R-CNN, and also serve as input to Mask R-CNN. S3, during model training, Faster R-CNN is trained using labeled bounding box data to identify bounding boxes of sheep and lambs, and Mask R-CNN is trained using labeled bounding box and segmentation mask data to generate more accurate instance segmentation, and the loss functions of classification, bounding box regression and segmentation mask are optimized during training. S4. When fusion outputs, a fusion strategy is designed by combining the classification confidence of Faster R-CNN and the segmentation quality of Mask R-CNN. A weighted average is used to comprehensively consider the output proportions of the two models, and finally, the IoU (Intersection over Union) ratio is used to select the optimal bounding box. S5. In post-processing, NMS is used to remove overlapping detection results to ensure the alignment of each target detection box. An appropriate IoU (Intersection over Union) threshold is set to control the execution of NMS. At the same time, the segmentation mask generated by Mask R-CNN is processed to remove small noise regions and fill holes to ensure the accuracy and integrity of the segmentation mask.
[0035] In real-time processing optimization, the S6 first prunes and quantizes the MF R-CNN model to reduce model size and inference time, and uses a dedicated hardware accelerator GPU to improve real-time performance. It designs a parallel processing strategy, uses a multi-core CPU to process multiple candidate regions in parallel, optimizes data flow and computation process, reduces latency, and finally achieves high-precision bounding box selection and localization. The image is then transmitted to the user's PC interface to assist farmers in searching for and quickly rescuing them.
Claims
1. A data-fusion-based inspection and adaptive landing UAV, characterized in that, include: The device consists of a fuselage (1), a landing support frame (2), an RGB camera (3), and a buffer bionic claw (4). The fuselage (1) is connected to four landing support frames (2) via a rotating shaft (1-1). The fuselage (1) contains a lithium battery (1-2), a Jetson Orin NX industrial computer (1-3), and an STM32F427IIH6 core control board (1-4). The fuselage (1) has an RGB camera (3), a multi-view thermal infrared camera (1-7), and a multispectral camera (1-5) mounted at the front, middle, and rear of its bottom. The fuselage (1) has a large heat dissipation hole (1-6), a side heat dissipation hole (1-9), and a circular heat dissipation hole (1-8) at its bottom. The landing support frame (2) consists of a high-speed motor (2-7), a drive gear (2-2), a transmission gear (2-3), an ultrasonic sensor (2-4), a multi-stage telescopic support (2-5), and a strong magnet (2-6). The high-speed motor (2-7) is connected to the STM32F427IIH6 core control board (1-4) and the lithium battery (1-2) via internal data lines to achieve power and control connection. After the high-speed motor (2-7) rotates, it is transmitted to the transmission gear (2-3) through the drive gear, thereby realizing the rotation of the UAV rotor. The four ultrasonic sensors (2-4) detect their height above the ground in real time and return the data to the STM32F427IIH6 core control board (1-4), and further transmit it to the Jetson Orin NX industrial computer (1-3) for data analysis. The Jetson Orin NX industrial computer (1-3) is equipped with a real-time fusion analysis algorithm for multi-sensor data, which fuses the semantic segmentation algorithm Mask R-CNN and the target detection algorithm Faster R-CNN to form MF. The R-CNN algorithm model integrates data from RGB camera (3), multi-view thermal infrared camera (1-7), and multispectral camera (1-5) for real-time monitoring and analysis to determine the accurate geographical locations of stray sheep and newborn lambs, segment the shape of sheep on the pasture, and construct their three-dimensional models based on depth information; the state of the multi-level telescopic support (2-5) is controlled by an internal spring, and the compression and extension of the spring (2-11) are controlled by a servo motor (2-8). The STM32F427IIH6 core control board (1-4) controls the rotation of the servo motor (2-8) through a data cable; the servo motor (2-8) has a winding shell (2-9) at the front end, and when the servo motor (2-8) rotates, it will pull the wire (2-9) through the wire. -10) Pull the spring (2-11) to compress the spring (2-11), which in turn causes the multi-stage telescopic support (2-5) to retract, thereby realizing the lifting and buffering control of the landing support frame (2); The multispectral camera (1-5) is attached to the bottom rear of the body (1) by a magnetic chuck (1-5-4), and has five lenses, namely a red band lens (1-5-1), a green band lens (1-5-5), a red edge band lens (1-5-3), a blue band lens (1-5-6), and a near-infrared band lens (1-5-2). Through multiple lenses, it distinguishes and analyzes the reflection characteristics of objects at different wavelengths, acquires images of five specific spectral ranges, and quickly transmits the data to Jetson via a data cable. The Orin NX industrial computer (1-3) is used for analysis; the buffer bionic claw (4) is composed of a universal ball (4-1), a support base (4-2) and four support grippers (4-3). The universal ball (4-1) can rotate at multiple angles, and the support grippers (4-3) can open and close, providing more stable support points for the drone landing;When the UAV lands, the four support grips (4-3) of the buffer bionic claw (4) first contact the ground. The support grips (4-3) slowly open, providing the first level of buffer. Then, the ultrasonic sensor (2-4) monitors the altitude above the ground in real time and transmits the data to the STM32F427IIH6 core control board (1-4). The board calculates the length of the four multi-stage telescopic supports (2-5) that keep the fuselage (1) absolutely level. This data is then converted into the rotation angle of the servo motor (2-8) after calculation, providing the second level of buffer when the UAV lands on uneven ground. After landing, the UAV can still maintain the absolute level of the fuselage (1) and protect its internal control module.
2. The UAV with data fusion-based inspection and adaptive landing as described in claim 1, characterized in that, The RGB camera (3) is fixed to the lower front of the body (1) via a rotating shaft and can capture RGB images with a resolution of 5280×3956. The multi-view thermal infrared camera (1-7) is fixed to the middle of the bottom of the body (1) and protected by a transparent protective cover. It has three independent detection lenses, providing a wider field of view and more detailed target analysis. It reconstructs the three-dimensional shape of the target by analyzing infrared images from different perspectives. The three detection lenses can also perform data verification, improving the stability and reliability of thermal infrared images.
3. A method for using a UAV based on data fusion-based inspection and adaptive landing as described in claim 1, characterized in that, Includes the following steps: S1, the MFR-CNN algorithm model uses a shared ResNet feature extractor to extract features from multi-source input images during data fusion, and this feature is shared by the Faster R-CNN and Mask R-CNN models to reduce computational redundancy; S2 uses Faster R-CNN's RPN to generate candidate regions, which are then used for classification and bounding box regression in Faster R-CNN, and also serve as input to Mask R-CNN. S3. During model training, Faster R-CNN is trained using labeled bounding box data to identify the bounding boxes of stray sheep and newborn lambs. Mask R-CNN is trained using labeled bounding box and segmentation mask data to generate more accurate instance segmentation. The loss functions for classification, bounding box regression and segmentation mask are optimized during training. S4. When fusion outputs, a fusion strategy is designed by combining the classification confidence of Faster R-CNN and the segmentation quality of Mask R-CNN. A weighted average is used to comprehensively consider the output proportions of the two models, and finally, IoU is used to select the optimal bounding box. S5. In post-processing, NMS is used to remove overlapping detection results to ensure the alignment of each target detection box. An appropriate IoU threshold is set to control the execution of NMS. At the same time, the segmentation mask generated by Mask R-CNN is processed to remove small noise regions and fill holes to ensure the accuracy and integrity of the segmentation mask. In real-time processing optimization, the S6 first prunes and quantizes the MF R-CNN model to reduce model size and inference time, and uses a dedicated hardware accelerator GPU to improve real-time performance. It designs a parallel processing strategy, uses a multi-core CPU to process multiple candidate regions in parallel, optimizes data flow and computation process, reduces latency, and finally achieves high-precision bounding box selection and localization. The image is then transmitted to the user's PC interface for farmers to search and rescue quickly.
4. The method according to claim 3, characterized in that, During feature extraction, the MF R-CNN algorithm model uses weighted inputs of thermal infrared camera data, multispectral camera data, and high-resolution RGB image data of grassland pastures taken under different times and weather conditions into the network model. It uses Faster R-CNN for object detection, which includes a region proposal network and a classification and bounding box regression network. It uses Mask R-CNN for instance segmentation, which adds an extra branch to Faster R-CNN to predict the pixel-level mask for each proposed region.
Citation Information
Patent Citations
Hip and knee drivable bionic landing leg type six-rotor unmanned aerial vehicle and control method thereof
CN116161250A
Device and method for automatically monitoring quantity of grassland livestock based on aerial photography of unmanned aerial vehicle
CN116389693A
Small sample target detection method and system based on multi-modal fusion, and medium
CN116416503A
Danet-based drone patrol and inspection system for coastline floating garbage
US20210224512A1
