Unmanned aerial vehicle intelligent wild animal monitoring method based on FPGA and improved YOLO

Abstract

The invention relates to a method for intelligently monitoring wild animals through an unmanned aerial vehicle based on an FPGA and an improved YOLO, and belongs to the technical field of unmanned aerial vehicle monitoring. The method comprises the following steps: firstly, acquiring infrared and visible light images of wild animals by using an unmanned aerial vehicle, constructing a wild animal image data set, training a generative adversarial network (GAN) by using the wild animal image data set to perform visible light and infrared light image fusion, and taking a visible light and infrared light fusion image generated by the generative adversarial network as a data set and labeling; generating a bounding box and category information; sending the fused image into an improved YOLO v9 model for training, evaluating the performance of the improved YOLO v9 model on a test set, and adjusting a model structure or a training strategy according to an evaluation result; then, the trained improved YOLO v9 model is deployed in an FPGA (Field Programmable Gate Array); and finally, an infrared integrated camera on the unmanned aerial vehicle is used for monitoring wild animals, a specific target can be tracked, and a result is transmitted to terminal equipment.

Description

Technical Field

[0001] The invention relates to a method for intelligently monitoring wild animals by a drone based on FPGA and improved YOLO, and belongs to the technical field of drone monitoring. Background Art

[0002] With the increasing awareness of ecological and environmental protection, the monitoring and management of wildlife populations has become increasingly important. The traditional method of personnel going deep into the environment for monitoring is time-consuming and labor-intensive. The use of drones reduces the safety risks of personnel going deep into dangerous environments for field inspections, while also reducing interference with wildlife. FPGA can process image data in real time and has parallel data processing capabilities. It can quickly process large amounts of data collected by the cameras on drones. Deploying the improved YOLO v9 model in FPGA can be used to implement complex image recognition algorithms, automatically identify wildlife, and improve monitoring accuracy.

[0003] Wildlife monitoring requires the collection and analysis of large amounts of data in real time. Compared with general-purpose processors, the improved YOLO v9 model deployed in FPGAs has significant advantages in processing speed, energy consumption, and customization, and is particularly suitable for real-time image processing and signal processing tasks.

[0004] Wild animals vary greatly in size. In order to improve the detection capability of small wild animals or animals at a distance, it is proposed to add a feature pyramid structure and a multi-scale attention mechanism to YOLO v9.

[0005] The living environment of wild animals is complex. FPGA allows users to customize hardware logic according to specific monitoring needs, which enables the drone monitoring system to be optimized for different wild animals and environments. At the same time, FPGA also has parallel data processing capabilities, which can quickly process large amounts of data collected by the cameras on the drone to achieve real-time monitoring and tracking. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for intelligently monitoring wild animals by using a drone based on FPGA and improved YOLO, so as to realize the monitoring of wild animal populations.

[0007] The technical solution of the present invention is: a method for intelligently monitoring wild animals by using a drone based on FPGA and improved YOLO, comprising:

[0008] Step 1: Collect infrared and visible light images of wild animals through drones to build a wild animal image dataset. Use the wild animal image dataset to train a generative adversarial network (GAN) to fuse visible light images and infrared images to obtain visible light infrared fusion images. Use the visible light infrared fusion images as a dataset and annotate them to generate bounding boxes and category information.

[0009] Collecting large data sets using drones can provide more data variations, thereby reducing overfitting and improving generalization capabilities to ensure the monitoring accuracy of the improved YOLOv9 model.

[0010] Step 2: Send the annotated visible light and infrared light fusion images to the improved YOLOv9 model for training, and evaluate the performance of the improved YOLOv9 model on the test set, and adjust the model structure or training strategy according to the evaluation results;

[0011] Step 3: Deploy the trained improved YOLOv9 model to FPGA;

[0012] Step 4: Use the infrared integrated camera on the drone to start monitoring wild animals, track specific targets, and finally transmit the results to the terminal device.

[0013] The infrared integrated camera is used to acquire images. It can automatically switch between day and night. It uses visible light for imaging during the day and infrared light for illumination at night or when the light is poor, thus achieving night vision monitoring and having good waterproof and dustproof properties, and is suitable for use in outdoor environments.

[0014] The improved YOLO v9 model refers to adding a feature pyramid structure and a multi-scale attention mechanism to the YOLO v9 model.

[0015] The pyramid structure can effectively solve the problem of inconsistent scales of targets in the image, improving the robustness and accuracy of target detection. Secondly, through the feature pyramid structure, the algorithm can fuse features of different scales, thereby improving the detection ability of small wild animals or distant animals.

[0016] The multi-scale attention mechanism can enhance the features of small wild animals and improve the model's ability to recognize them. For large wild animals, it can help the model locate objects more accurately at these scales.

[0017] The wildlife image dataset covers wildlife activity scenes during the day and at night, including environment, lighting conditions and angles. The dataset is divided into 10 parts, 9 of which are used as training sets and 1 as test sets. The cycle is repeated 10 times, and the model evaluation results are obtained after calculating the mean.

[0018] The loss function selected by the generative adversarial network is used to evaluate the performance of the fused image, and the loss function includes adversarial loss and content loss.

[0019] The generative adversarial network is equipped with two discriminators, one for distinguishing the fused image from the infrared image, and the other for distinguishing the fused image from the visible light image. These two discriminators make the fused image have the high-resolution details of the visible light image and the highlighted areas of the infrared image. Since visible light and infrared images are captured by different sensors, the important features they focus on are also different. Infrared images can pay more attention to the intensity distribution of the input image, while visible light images focus on the texture information of the input image. The generator uses the texture information of the visible light image and the pixel intensity distribution of the infrared image to generate the fusion result through continuous adversarial learning.

[0020] The training process of the improved YOLOv9 model includes: firstly, loading images and annotation information from a training data set, and performing forward propagation through the YOLOv9 model to obtain predicted bounding boxes and category probabilities, then calculating losses based on the predicted results and true annotations, and then backpropagating model parameters based on the losses, and finally calculating gradients and using an optimizer to update model parameters.

[0021] The improved YOLOv9 model avoids the use of time-consuming sliding windows or complex neural network structures, and has a very fast detection speed, which is particularly suitable for real-time target monitoring tasks. Not only can the detection effect of small targets be improved through the feature pyramid structure and multi-scale attention mechanism, which is beneficial for monitoring small wild animals, but also by analyzing images at different scales, it can reduce the problems of false detection and missed detection caused by scale differences.

[0022] The generative adversarial network comprises:

[0023] An image processing module, used for preprocessing the collected images;

[0024] CLK clock module, used to generate clock signal for FPGA;

[0025] A power module, used to provide power to the FPGA;

[0026] SDRAM storage module, used to store the acquired images.

[0027] The preprocessing includes median filtering, homomorphic filtering and bilateral filtering. Median filtering is first used to remove noise, then bilateral filtering is used to retain details, and finally homomorphic filtering is used to improve contrast.

[0028] Median filtering is an image processing algorithm used to remove noise from images. It adapts to different image regions and noise intensities by dynamically adjusting the size of the filter based on the neighborhood information around the pixel.

[0029] Bilateral filtering is a nonlinear filtering technique that can remove noise while retaining edge information. Its advantage is that it can distinguish between high-frequency signals and noise, thereby avoiding blurring the edges of the image during the denoising process.

[0030] Homomorphic filtering uses a nonlinear mapping of linear filtering in different domains to improve the contrast of the image and reduce the impact of uneven illumination.

[0031] The beneficial effects of the present invention are as follows: the present invention can not only improve the efficiency and accuracy of wildlife monitoring, but also promote the understanding and protection of ecosystems. The use of drones reduces the safety risks of personnel going deep into dangerous environments for field inspections, while reducing interference with wildlife. In addition, drone monitoring helps to identify and protect endangered species, and can also track some protected animals and grasp their dynamics in real time. This can not only maintain the diversity of ecosystems, but also provide important data on animal behavior, ecological habits and interactions between species, and promote research in biology and ecology. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a system block diagram of the present invention;

[0033] Figure 2 It is a schematic diagram of the generative adversarial network fusion image of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0035] Example 1: Figure 1 As shown in the figure, a method for intelligently monitoring wild animals by using a drone based on FPGA and improved YOLO is first installed on the drone, and the image collected by the camera is transmitted to the image acquisition module. The image acquisition module transmits the data to the SDRAM storage module, and then reads the data from the SDRAM storage module to the image processing module. The image processing module includes median filtering, homomorphic filtering and bilateral filtering. Then the processed image data is transmitted to the trained improved YOLO v9 model, and the monitoring of wild animals is started and the results are sent to the terminal device through the transmission module.

[0036] A large number of data sets were collected using drones. The collected data sets included a total of 26,280 pairs of visible light and infrared light images, covering various wildlife activity scenes during the day and at night, including environment, lighting conditions and angles. The data sets were divided into 10 parts, 9 of which were used as training sets and 1 as a test set. The cycle was repeated 10 times, and the model evaluation results were obtained after calculating the mean. The formula is as follows:

[0037]

[0038] In the formula, S i is the test score of the ith cross-validation.

[0039] like Figure 2 As shown, a wildlife image dataset is used to train a generative adversarial network (GAN) to perform visible light and infrared light image fusion, where the generative adversarial network (GAN) requires the selection of a loss function.

[0040] The loss function is used to evaluate the performance of the fused image. The better the loss function, the better the performance of the fusion method. The loss function is composed of adversarial loss and content loss.

[0041] Adversarial loss is the core of the GAN framework. It promotes the generator to produce more realistic data by minimizing the distribution difference between the fake data generated by the generator and the real data. The formula is as follows:

[0042]

[0043] Where, L G is the loss of the generator, z is the value obtained from the prior noise distribution P Z The noise vector sampled in , G(z) is the image generated by the generator, and D(G(z)) is the judgment of the discriminator on the generated image.

[0044]

[0045] Where, L D is the loss of the discriminator, x is from the real data distribution P data The real image sampled from P data is the distribution of the real data.

[0046] Content loss ensures that the generated image or data is consistent with the original input at the pixel or feature level, thereby retaining important visual information. By using structural similarity loss (SSIM) as part of the content loss, the structural similarity between the generated image and the original image can be maintained. The formula is as follows:

[0047]

[0048]

[0049]

[0050] SSIM(x,y)=l(x,y)·c(x,y)·s(x,y)

[0051] In the formula, μ x and μ yare the mean brightness values ​​of the two images in a certain local area, C1 is a very small constant used to ensure that the denominator is not zero; σ x and σ y are the standard deviations of the two images in the corresponding areas, indicating the contrast, C2 is a constant; σ xy is the covariance of the pixel brightness in the corresponding area of ​​the two images, and C3 is a constant. l(x,y) represents brightness contrast, c(x,y) represents contrast comparison, and s(x,y) represents structure comparison.

[0052] The visible light infrared light fusion images generated by the Generative Adversarial Network (GAN) are used as datasets and annotated to generate bounding boxes and category information.

[0053] The trained improved YOLOv9 model is deployed to the FPGA, and the images collected by the camera are transmitted to the image acquisition module. The image acquisition module transmits the data to the SDRAM storage module, and then reads the data from the SDRAM storage module and transmits it to the image processing module.

[0054] The image processing module is used to pre-process the collected image, including median filtering, bilateral filtering and homomorphic filtering.

[0055] In the image preprocessing, median filtering is first used to remove noise, then bilateral filtering is used to retain details, and finally homomorphic filtering is used to improve contrast.

[0056] The median filter is an image processing algorithm used to remove noise from an image. It dynamically adjusts the size of the filter according to the area information around the pixel to adapt to different image areas and noise intensities.

[0057] The bilateral filtering is a nonlinear filtering technique that can retain edge information while removing noise. Its advantage is that it can distinguish high-frequency signals from noise, thereby avoiding blurring the edge of the image during the denoising process.

[0058] The homomorphic filtering uses a nonlinear mapping of linear filtering in different domains to improve the contrast of the image and reduce the impact of uneven illumination.

[0059] The processed image data is passed to the trained improved YOLOv9 model to start monitoring wild animals and send the results to the terminal device through the transmission module.

[0060] The improved YOLOv9 model first loads images and annotation information from the training data set, and performs forward propagation through the model to obtain the predicted bounding box and category probability, then calculates the loss based on the prediction result and the true annotation, and then backpropagates the model parameters based on the loss, and finally calculates the gradient and uses the optimizer to update the model parameters, where the loss function usually includes: object existence loss, position loss and category loss.

[0061] The position loss is used to measure the difference between the predicted bounding box and the actual bounding box, and its formula is as follows:

[0062]

[0063] Where N is the number of prediction boxes, and are the x coordinates of the upper left corner of the predicted box and the actual box, respectively. and are the y coordinates of the upper left corner of the predicted box and the actual box, respectively. and are the x coordinates of the lower right corner of the predicted box and the actual box, respectively. and They are the y coordinates of the lower right corner of the predicted box and the actual box, respectively.

[0064] The object existence loss is specifically used to deal with the problem of whether the predicted boxes should contain objects. The purpose of this loss function is to penalize those predicted boxes whose intersection with the actual box is lower than a certain threshold. The formula is as follows:

[0065]

[0066] Where, IoU i It is the intersection-over-union ratio of the i-th predicted box and the actual box.

[0067] Category loss is used to penalize the difference between the predicted category and the actual category. It is usually calculated using cross entropy loss. The formula is as follows:

[0068]

[0069] Where CE is the cross entropy loss function.

[0070] The total loss is the weighted sum of position loss, object existence loss, and category loss, and the formula is as follows:

[0071] L total =L coord +L obj +λ noobj ·L noobj +λ class ·L class

[0072] In the formula, λ noobj and λ class is the weight used to balance different loss terms.

[0073] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for intelligent wildlife monitoring by drone based on FPGA and improved YOLO, characterized in that: The method comprises: Step 1: Collect infrared and visible light images of wild animals through drones to build a wild animal image dataset. Use the wild animal image dataset to train a generative adversarial network to fuse visible light images and infrared images to obtain visible light infrared fusion images. Use the visible light infrared fusion images as a dataset and annotate them to generate bounding boxes and category information. Step 2: Send the annotated visible light and infrared light fusion images to the improved YOLOv9 model for training, and evaluate the performance of the improved YOLOv9 model on the test set, and adjust the model structure or training strategy according to the evaluation results; Step 3: Deploy the trained improved YOLOv9 model to FPGA; Step 4: Use the infrared integrated camera on the drone to start monitoring wild animals, track specific targets, and finally transmit the results to the terminal device.

2. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1 is characterized in that: The improved YOLO v9 model refers to adding a feature pyramid structure and a multi-scale attention mechanism to the YOLO v9 model.

3. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1, characterized in that: The wildlife image dataset covers wildlife activity scenes during the day and at night, including environment, lighting conditions and angles.

4. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1, characterized in that: The loss function selected by the generative adversarial network is used to evaluate the performance of the fused image, and the loss function includes adversarial loss and content loss.

5. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1, characterized in that: Two discriminators are set in the generative adversarial network, one is used to distinguish the fused image from the infrared image, and the other is used to distinguish the fused image from the visible light image. The two discriminators enable the fused image to have high-resolution details of the visible light image and highlighted areas of the infrared image.

6. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1, characterized in that: The training process of the improved YOLOv9 model includes: firstly, loading images and annotation information from a training data set, and performing forward propagation through the YOLOv9 model to obtain predicted bounding boxes and category probabilities, then calculating losses based on the predicted results and true annotations, and then backpropagating model parameters based on the losses, and finally calculating gradients and using an optimizer to update model parameters.

7. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 1, characterized in that: The generative adversarial network comprises: An image processing module, used for preprocessing the collected images; CLK clock module, used to generate clock signal for FPGA; A power module, used to provide power to the FPGA; SDRAM storage module, used to store the acquired images.

8. The method for intelligently monitoring wild animals by unmanned aerial vehicles based on FPGA and improved YOLO according to claim 7, characterized in that: The preprocessing includes median filtering, homomorphic filtering and bilateral filtering. Median filtering is first used to remove noise, then bilateral filtering is used to retain details, and finally homomorphic filtering is used to improve contrast.

Images