Wide-area bird flock detection and counting method based on multi-dimensional information fusion

Abstract

The present application belongs to the technical field of wetland bird detection, and particularly relates to a wide-area bird flock detection and counting method based on multi-dimensional information fusion. The method comprises the following steps: S1: mounting visible cameras, infrared cameras and polarization cameras with the same field of view on an unmanned aerial vehicle to perform multi-dimensional image acquisition; S2: using the multi-dimensional image acquisition results obtained in step S1 to prepare a flying bird detection data set; S3: constructing a flying bird detection deep neural network, training the flying bird detection deep neural network using the flying bird detection data set, and obtaining a trained flying bird detection deep neural network; S4: performing multi-dimensional image acquisition on flying birds in a to-be-detected area, and inputting the multi-dimensional image acquisition results into the trained flying bird detection deep neural network to perform flying bird type detection and flying bird pixel position detection; and S5: counting the types and quantities of flying birds in the to-be-detected area according to the detection results of step S4. The present application has the advantages of automation, high precision and rapidity.

Description

Technical Field

[0001] This invention belongs to the field of wetland bird community technology, and particularly relates to a wide-area bird flock detection and statistical method based on multi-dimensional information fusion. Background Technology

[0002] Bird communities are an important component of wetland resources. Regular dynamic monitoring of bird populations and distribution in wetland reserves is a crucial means of protecting wetland bird communities. Currently, bird population statistics and community distribution monitoring in wetland reserves mainly rely on manual methods, such as tag-and-recapture, but these methods are inefficient and labor-intensive. With advancements in photoelectric detection technology, researchers have begun using photoelectric detection equipment to record and count bird populations and community distribution, and to analyze bird numbers. However, the lack of automated analysis methods means that counting and analyzing bird populations and community distribution still requires significant manpower and time. Summary of the Invention

[0003] In view of this, the present invention aims to provide a wide-area bird flock detection and statistical method based on multi-dimensional information fusion, so as to solve the shortcomings of existing methods for statistically analyzing the number of bird flocks and the distribution of communities, which require a lot of manpower and time.

[0004] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0005] A wide-area bird flock detection and statistical method based on multidimensional information fusion includes the following steps:

[0006] S1: Mount a visible camera, an infrared camera, and a polarization camera with the same field of view on the unmanned aerial vehicle to collect multi-dimensional images of birds in a selected area;

[0007] S2: Use the multi-dimensional image acquisition results obtained in step S1 to create a bird detection dataset;

[0008] S3: Construct a deep neural network for bird detection, train the deep neural network for bird detection using the bird detection dataset, and obtain a trained deep neural network for bird detection.

[0009] S4: Perform multi-dimensional image acquisition of birds in the test area, and input the multi-dimensional image acquisition results into the trained bird detection deep neural network to detect the bird type and the bird pixel position;

[0010] S5: Based on the detection results of step S4, count the types and numbers of birds in the area to be tested.

[0011] Furthermore, in step S1, the multidimensional image includes an infrared image, a visible image, and a polarization image.

[0012] Furthermore, step S2 specifically includes the following steps:

[0013] S21: Super-resolution operation is performed on infrared images using bilinear interpolation to ensure that each infrared image with the same field of view has the same resolution as the visible image and the polarized image.

[0014] I high (2x, 2y) = 1 / 4[I] low (x,y)+I low (x+1,y)+I low (x,y+1)+I low (x+1,y+1)];

[0015] Among them, I low (x,y) represents the infrared image, I high (x,y) represents the visible image and the polarization image;

[0016] S22: Label the infrared, visible and polarized images with the same field of view in each group, including the types of birds and the pixel positions of the birds contained therein;

[0017] S23: Take the union of the bounding boxes of the bird pixel positions contained in the infrared image, visible image and polarization image with the same field of view in each group;

[0018] S24: Divide the infrared images, visible images, and polarized images with the same field of view into training and validation sets after taking the union of the sets, so as to realize the creation of the bird detection dataset.

[0019] Furthermore, in step S3, the bird detection deep neural network includes a multi-dimensional information fusion unit and a scale-aware bidirectional feature pyramid network unit. The infrared images, visible images, and polarization images with the same field of view are stacked by channel to obtain a stacked image. The stacked image is then input to the multi-dimensional information fusion unit for image fusion to obtain a multi-dimensional fused image. The multi-dimensional fused image is then processed by the scale-aware bidirectional feature pyramid network unit for feature extraction to obtain the bird type and bird pixel position on the multi-dimensional images with the same field of view.

[0020] Furthermore, the multi-dimensional information fusion unit includes a first 1×1 convolution module, a second 1×1 convolution module, a third 1×1 convolution module, a fourth 1×1 convolution module, a fifth 1×1 convolution module, a softmax module, a cascaded channel average module, and a channel maximum module. The stacked image is processed sequentially by the cascaded channel average module, channel maximum module, first 1×1 convolution module, second 1×1 convolution module, and softmax module to obtain a first feature. The stacked image is then convolved by the third 1×1 convolution module to obtain a second feature. The first feature is multiplied by the second feature to obtain a third feature. The third feature is input into the fourth 1×1 convolution module for convolution processing to obtain a fourth feature. The fourth feature is input into the fifth 1×1 convolution module for convolution processing to obtain a fifth feature. The fifth feature is multiplied by the stacked image to obtain the output feature.

[0021] Furthermore, the polarization images specifically include 0° polarization images, 45° polarization images, 90° polarization images, 135° polarization images, polarization degree images, and polarization phase angle images.

[0022] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0023] The present invention provides a wide-area bird flock detection and statistics method based on multi-dimensional information fusion. It trains a bird detection deep neural network based on infrared images, polarization images, and visible images, and uses the trained bird detection deep neural network to realize the detection and statistics of wide-area bird flocks, thereby achieving automated identification of bird communities. It has the advantages of automation, high accuracy, and speed.

[0024] The fusion of polarized and visible images makes it easy to quickly and accurately identify bird species in complex backgrounds (water, grass, etc.). At the same time, based on polarization characteristics, it is easy to distinguish birds of similar size. The integrated infrared image utilizes the birds' self-radiation to achieve all-weather detection of birds. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0026] Figure 1 A flowchart illustrating the wide-area bird flock detection and statistics method based on multi-dimensional information fusion as described in the embodiments of the present invention;

[0027] Figure 2 A schematic diagram of the structure of the bird detection deep neural network described in the embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the network structure of the multi-dimensional information fusion unit described in the embodiment of the present invention.

[0029] Figure descriptions: 1. Infrared image; 2. Visible image; 3. Polarized image; 4. Multidimensional information fusion unit; 5. Scale-aware bidirectional feature pyramid network unit; 6. Output; 61. Bird type; 62. Bird pixel position; 7. Stacked image; 8. Channel average value module; 9. Channel maximum value module; 10. First 1×1 convolution module; 11. Second 1×1 convolution module; 12. Softmax module; 13. First feature; 14. Third 1×1 convolution module; 15. Second feature; 16. Third feature; 17. Fourth 1×1 convolution module; 18. Fifth 1×1 convolution module; 19. Output feature. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0032] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0033] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0034] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] like Figure 1 As shown, the wide-area bird flock detection and statistics method based on multi-dimensional information fusion provided by this invention specifically includes the following steps:

[0036] S1: Mount a visible camera, an infrared camera, and a polarization camera with the same field of view on the unmanned aerial vehicle to collect multi-dimensional images of birds in a selected area.

[0037] In step S1, the multidimensional image includes infrared image 1, visible image 2, and polarized image 3. The optical axes of infrared image 1, visible image 2, and polarized image 3 are all adjusted to the center of the image plane, and then a mechanical device is used to make the fields of view of the multiple cameras consistent.

[0038] The polarization images specifically include 0° polarization images, 45° polarization images, 90° polarization images, 135° polarization images, polarization degree images, and polarization phase angle images.

[0039] S2: Use the multidimensional image acquisition results obtained in step S1 to create a bird detection dataset.

[0040] The bird detection dataset conforms to the VOC2007 format.

[0041] Step S2 specifically includes the following steps:

[0042] S21: Perform super-resolution operation on infrared image 1 using bilinear interpolation to make the resolution of infrared image 1, which has the same field of view as visible image 2 and polarized image 3, the same.

[0043] I high (2x, 2y) = 1 / 4[I] low (x,y)+I low (x+1,y)+I low (x,y+1)+I low (x+1,y+1)];

[0044] Among them, I low (x,y) represents infrared image 1, I high(x,y) represents visible image 2 and polarization image 3, which have the same resolution.

[0045] S22: Label each group of infrared image 1, visible image 2 and polarized image 3 with the same field of view, and the bird type 61 and bird pixel position 62 contained in each group;

[0046] S23: Take the union of the bounding boxes of the bird pixel positions 62 contained in each group of infrared image 1, visible image 2 and polarization image 3 with the same field of view;

[0047] S24: Divide the infrared image 1, visible image 2 and polarized image 3 with the same field of view after taking the union into training set and validation set in a ratio of 9:1 to realize the creation of bird detection dataset.

[0048] S3: Construct a deep neural network for bird detection, train the deep neural network for bird detection using the bird detection dataset, and obtain a trained deep neural network for bird detection.

[0049] In step S3, the loss function used to train the bird detection deep neural network is:

[0050]

[0051] Where Loss is the loss function, L cls Let L be the Sigmoid loss function. reg Let N be the L1 loss function, i be the total number of samples in the bird detection dataset, j be the positive samples in the bird detection dataset, and N be the total number of samples in the dataset. pos This represents the number of positive samples.

[0052] S4: Perform multi-dimensional image acquisition of birds in the area to be tested, and input the multi-dimensional image acquisition results into the trained bird detection deep neural network to detect bird type 61 and bird pixel position 62;

[0053] The detection of bird pixel position 62 is used to identify the specific area where birds inhabit.

[0054] S5: Based on the detection results of step S4, count the types and numbers of birds in the area to be tested.

[0055] By using different colored boxes to label different types of birds and counting the number of boxes of each color, the species and number of birds in the area to be measured can be counted.

[0056] The unmanned aerial vehicle is also equipped with a GPS positioning unit, which is used to calibrate the spatial position of birds captured by infrared image 1, visible image 2 and polarization image 3 with the same field of view.

[0057] The drone flight operation specifications comply with the "Interim Regulations on the Management of Unmanned Aerial Vehicle Flights," and the flight altitude is higher than 50m to ensure the shooting range of a single image.

[0058] like Figure 2 As shown, the bird detection deep neural network includes a multidimensional information fusion unit 4 and a scale-aware bidirectional feature pyramid network unit 5. The infrared image 1, visible image 2 and polarization image 3 with the same field of view are stacked by channel to obtain a stacked image 7. The stacked image 7 is then input to the multidimensional information fusion unit 4 for image fusion to obtain a multidimensional fused image. The multidimensional fused image is then processed by the scale-aware bidirectional feature pyramid network unit 5 for feature extraction to obtain the bird type 61 and bird pixel position 62 on the multidimensional images with the same field of view.

[0059] The scale-aware bidirectional feature pyramid network unit 5 used in this embodiment of the invention is derived from the scale-aware bidirectional feature pyramid network (including bidirectional feature pyramids and SSD units) in the paper "Scale-aware Bidirectional Feature Pyramid Network for Traffic Sign Detection". The scale-aware bidirectional feature pyramid network unit 5 used in this embodiment of the invention improves the detection accuracy and robustness for multi-scale targets by introducing a bidirectional pyramid structure and optimizing feature fusion.

[0060] like Figure 3 As shown, the multidimensional information fusion unit 4 includes a first 1×1 convolution module 10, a second 1×1 convolution module 11, a third 1×1 convolution module 14, a fourth 1×1 convolution module 17, a fifth 1×1 convolution module 18, a softmax module 12, a cascaded channel average module 8, and a channel maximum module 9. The stacked image 7 is processed sequentially by the cascaded channel average module 8 and channel maximum module 9, the first 1×1 convolution module 10, the second 1×1 convolution module 11, and the softmax module 12 to obtain a first feature 13. The stacked image 7 is further processed by the third 1×1 convolution module 14 to obtain a second feature 15. The first feature 13 and the second feature 15 are multiplied to obtain a third feature 16. The third feature 16 is input to the fourth 1×1 convolution module 17 for convolution processing to obtain a fourth feature. The fourth feature is input to the fifth 1×1 convolution module 18 for convolution processing to obtain a fifth feature. The fifth feature is multiplied by the stacked image 7 to obtain an output feature 19.

[0061] That is, the dimensions of the stacked image 7 are [c, w, h], where c is the number of features, w is the feature width, and h is the feature length. After the stacked image 7 is processed by the cascaded channel averaging module 8 and channel maximizing module 9, the dimensions of the obtained features are [2, w, h]. After being processed by the first 1×1 convolution module 10, the dimensions of the obtained features are [r, w, h]. After being processed by the second 1×1 convolution module 11, the dimensions of the obtained features are [1, w, h]. After being flattened by the softmax module 12, the dimensions of the obtained first feature 13 are [wh, 1]. When the stacked image 7... After processing by the third 1×1 convolution module 14, the second feature 15 is obtained with dimensions [c, wh, 1]. The first feature 13 is multiplied by the second feature 15 to obtain the third feature 16 with dimensions [c, 1, 1]. The third feature 16 is input to the fourth 1×1 convolution module 17 for convolution processing to obtain the fourth feature with dimensions [c / r, 1, 1]. The fourth feature is input to the fifth 1×1 convolution module 18 for convolution processing to obtain the fifth feature with dimensions [c, 1, 1]. The fifth feature is multiplied by the stacked image 7 to obtain the output feature 19.

[0062] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0063] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A wide-area bird flock detection and statistical method based on multi-dimensional information fusion, characterized in that: Specifically comprising the following steps: S1: mounting the same field of view visible camera, infrared camera and polarization camera on the unmanned aerial vehicle to collect multi-dimensional images of the flying birds in the selected area; In step S1, the multi-dimensional images include infrared images, visible images and polarization images; S2: using the multi-dimensional image collection results obtained in step S1 to make a flying bird detection data set; Step S2 specifically comprises the following steps: S21: using the bilinear interpolation method to perform super-resolution operation on the infrared images, so that the resolution of each group of infrared images, visible images and polarization images with the same field of view is the same: ； wherein, is an infrared image, is a visible image and a polarized image; S22: labeling each group of infrared images, visible images and polarization images with the same field of view, each containing the type of flying bird and the pixel position of the flying bird; S23: taking the union of the labeled boxes of the pixel positions of the flying birds contained in each group of infrared images, visible images and polarization images with the same field of view; S24: dividing each group of infrared images, visible images and polarization images with the same field of view after taking the union into a training set and a validation set to realize the making of the flying bird detection data set; S3: constructing a flying bird detection deep neural network, training the flying bird detection deep neural network using the flying bird detection data set, and obtaining a trained flying bird detection deep neural network; In step S3, the flying bird detection deep neural network includes a multi-dimensional information fusion unit and a scale perception based bidirectional feature pyramid network unit, each group of infrared images, visible images and polarization images with the same field of view are stacked by channel to obtain a stacked image, and the stacked image is input into the multi-dimensional information fusion unit for image fusion to obtain a multi-dimensional fusion image, and the multi-dimensional fusion image is input into the scale perception based bidirectional feature pyramid network unit for feature extraction to obtain the type of flying bird and the pixel position of the flying bird on each group of multi-dimensional images with the same field of view; S4: collecting multi-dimensional images of the flying birds in the to-be-tested area and inputting the multi-dimensional image collection results into the trained flying bird detection deep neural network for detection of the type of flying bird and the pixel position of the flying bird; S5: according to the detection results of step S4, counting the types and quantities of the flying birds in the to-be-tested area.

2. The wide-area bird flock detection and counting method based on multi-dimensional information fusion according to claim 1, characterized in that: The multi-dimensional information fusion unit comprises a first 1*1 convolution module, a second 1*1 convolution module, a third 1*1 convolution module, a fourth 1*1 convolution module, a fifth 1*1 convolution module, a softmax module, a cascaded channel average value module and a channel maximum value module, wherein the stacked images are sequentially processed by the cascaded channel average value module and the channel maximum value module, the first 1*1 convolution module, the second 1*1 convolution module and the softmax module to obtain first features, the stacked images are further subjected to convolution processing by the third 1*1 convolution module to obtain second features, the first features and the second features are multiplied to obtain third features, the third features are input into the fourth 1*1 convolution module for convolution processing to obtain fourth features, the fourth features are input into the fifth 1*1 convolution module for convolution processing to obtain fifth features, and the fifth features are multiplied with the stacked images to obtain output features.

3. The wide-area bird flock detection and counting method based on multi-dimensional information fusion according to claim 2, characterized in that: The polarized images are specifically 0° polarized images, 45° polarized images, 90° polarized images, 135° polarized images, polarized degree images and polarized phase angle images.

Images