Wide-area bird flock detection and statistics method based on multi-dimensional information fusion

By carrying a multi-dimensional camera on the unmanned aerial vehicle and using deep neural networks, the problem of high manpower and time costs in statistics and monitoring of wetland bird communities is solved, and automated, high-precision and rapid bird community detection are achieved.

CN119091330AActive Publication Date: 2024-12-06CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411207973.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The statistics of bird populations and monitoring of community distribution in existing wetland bird communities rely on artificial methods, are inefficient and require a lot of manpower and time costs.

Method used

Using wide-area bird flock detection and statistics methods based on multi-dimensional information fusion, we can use visible cameras, infrared cameras and polarization cameras to collect multi-dimensional images and use deep neural networks to detect and count birds to achieve automated recognition. .

Benefits of technology

It realizes automated identification of bird communities, improves detection accuracy and speed, reduces manpower and time costs, and can quickly and accurately identify bird species and conducts 24/7 inspections in complex contexts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091330A_ABST
    Figure CN119091330A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of wetland bird detection, and particularly relates to a wide-area bird flock detection and statistical method based on multi-dimensional information fusion. Comprising the following steps: S1, carrying a visible camera, an infrared camera and a polarization camera with the same view field on an unmanned aerial vehicle, and carrying out multi-dimensional image acquisition; s2, making a bird detection data set by using the multi-dimensional image acquisition result obtained in the step S1; s3, constructing a flying bird detection deep neural network, and training the flying bird detection deep neural network by using the flying bird detection data set to obtain a trained flying bird detection deep neural network; s4, performing multi-dimensional image acquisition on birds in the to-be-detected area, and inputting a multi-dimensional image acquisition result into the trained bird detection deep neural network to perform bird type detection and bird pixel position detection; and S5, according to the detection result of the step S4, counting the types and the number of the birds in the to-be-detected area. The method has the advantages of automation, high precision and rapidness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of wetland bird colonies, and in particular relates to a wide-area bird flock detection and statistical method based on multi-dimensional information fusion. Background Art

[0002] Bird colonies are an important part of wetland resources. Regular dynamic monitoring of bird populations and colony distribution in wetland reserves is an important means of protecting wetland bird colonies. At present, bird population statistics and colony distribution monitoring in wetland reserves mainly rely on manual methods, such as the mark-recapture method, but such methods are inefficient and labor-intensive. With the advancement of photoelectric detection technology, researchers have begun to use photoelectric detection equipment to record and count the number of bird flocks and the distribution of colonies, and to count and analyze the number of bird flocks. However, there is currently a lack of automated analysis methods, which results in a lot of manpower and time costs for counting and analyzing the number of bird flocks and the distribution of colonies. Summary of the invention

[0003] In view of this, the present invention aims to provide a wide-area bird flock detection and statistics method based on multi-dimensional information fusion, so as to solve the shortcomings of existing methods for counting and analyzing the number of bird flocks and the distribution of flocks, which require a lot of manpower and time costs.

[0004] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0005] A wide-area bird flock detection and statistical method based on multi-dimensional information fusion specifically comprises the following steps:

[0006] S1: A visible camera, an infrared camera, and a polarization camera with the same field of view are mounted on an unmanned aerial vehicle to collect multi-dimensional images of birds in a selected area;

[0007] S2: Using the multi-dimensional image acquisition results obtained in step S1 to create a bird detection data set;

[0008] S3: Construct a deep neural network for bird detection, and use the bird detection dataset to train the deep neural network for bird detection to obtain a trained deep neural network for bird detection.

[0009] S4: performing multi-dimensional image acquisition of the flying birds in the detection area, and inputting the multi-dimensional image acquisition results into the trained bird detection deep neural network to detect the flying bird type and the flying bird pixel position;

[0010] S5: Count the types and numbers of flying birds in the detected area according to the detection results of step S4.

[0011] Further, in step S1, the multi-dimensional image includes an infrared image, a visible image and a polarization image.

[0012] Furthermore, step S2 specifically includes the following steps:

[0013] S21: Use bilinear interpolation to perform super-resolution operation on infrared images, so that the resolution of each set of infrared images with the same field of view is the same as that of visible images and polarization images:

[0014] I high (2x,2y)=1 / 4[I low (x,y)+I low (x+1,y)+I low (x,y+1)+I low (x+1,y+1)];

[0015] Among them, I low (x, y) is the infrared image, I high (x, y) is the visible image and polarization image;

[0016] S22: annotate each group of infrared images, visible images and polarization images with the same field of view, the type of flying birds contained in each group and the pixel position of the flying birds;

[0017] S23: taking a union of the annotation boxes of the pixel positions of the flying birds contained in each set of infrared images, visible images and polarization images with the same field of view;

[0018] S24: Divide the groups of infrared images, visible images and polarization images with the same field of view after taking the union into a training set and a verification set to achieve the production of a bird detection data set.

[0019] Furthermore, in step S3, the bird detection deep neural network includes a multidimensional information fusion unit and a scale-aware bidirectional feature pyramid network unit, which stacks each group of infrared images, visible images and polarization images with the same field of view by channel to obtain a stacked image, and inputs the stacked image into the multidimensional information fusion unit for image fusion to obtain a multidimensional fused image, and extracts features from the multidimensional fused image through a scale-aware bidirectional feature pyramid network unit to obtain the bird type and bird pixel position on each group of multidimensional images with the same field of view.

[0020] Furthermore, the multidimensional information fusion unit includes a first 1×1 convolution module, a second 1×1 convolution module, a third 1×1 convolution module, a fourth 1×1 convolution module, a fifth 1×1 convolution module, a softmax module, a cascaded channel average module and a channel maximum module, wherein the stacked image is sequentially processed by the cascaded channel average module and the channel maximum module, the first 1×1 convolution module, the second 1×1 convolution module and the softmax module to obtain a first feature, the stacked image is further convolved by the third 1×1 convolution module to obtain a second feature, the first feature is multiplied by the second feature to obtain a third feature, the third feature is input into the fourth 1×1 convolution module for convolution processing to obtain a fourth feature, the fourth feature is input into the fifth 1×1 convolution module for convolution processing to obtain a fifth feature, and the fifth feature is multiplied by the stacked image to obtain an output feature.

[0021] Furthermore, the polarization image specifically includes a 0° polarization image, a 45° polarization image, a 90° polarization image, a 135° polarization image, a polarization degree image, and a polarization phase angle image.

[0022] Compared with the prior art, the invention can achieve the following beneficial effects:

[0023] The present invention creates the wide-area bird flock detection and statistics method based on multi-dimensional information fusion, trains a flying bird detection deep neural network based on infrared images, polarized images and visible images, and uses the trained flying bird detection deep neural network to realize the detection and statistics of wide-area bird flocks, thereby realizing the automatic identification of bird colonies, and has the advantages of automation, high precision and rapidity.

[0024] The fusion of polarized images and visible images makes it easy to quickly and accurately identify the species of birds in complex backgrounds (water surface, grass, etc.). At the same time, based on the polarization characteristics, it is easy to distinguish birds of similar size. The integrated infrared image uses the self-radiation of birds to achieve all-weather detection of birds. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0026] Figure 1 A flow chart of a wide-area bird flock detection and statistical method based on multi-dimensional information fusion according to an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the structure of a deep neural network for bird detection according to an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the network structure of the multidimensional information fusion unit described in an embodiment of the present invention.

[0029] Description of the drawings: 1. Infrared image; 2. Visible image; 3. Polarized image; 4. Multidimensional information fusion unit; 5. Scale-aware bidirectional feature pyramid network unit; 6. Output; 61. Bird type; 62. Bird pixel position; 7. Stacked image; 8. Channel average module; 9. Channel maximum module; 10. First 1×1 convolution module; 11. Second 1×1 convolution module; 12. Softmax module; 13. First feature; 14. Third 1×1 convolution module; 15. Second feature; 16. Third feature; 17. Fourth 1×1 convolution module; 18. Fifth 1×1 convolution module; 19. Output feature. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0031] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0032] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0033] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0034] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0035] like Figure 1 As shown, the wide-area bird flock detection and statistical method based on multi-dimensional information fusion provided by the present invention specifically includes the following steps:

[0036] S1: A visible camera, an infrared camera, and a polarization camera with the same field of view are mounted on an unmanned aerial vehicle to collect multi-dimensional images of birds in a selected area.

[0037] In step S1, the multi-dimensional image includes an infrared image 1, a visible image 2, and a polarization image 3. The optical axes of the infrared image 1, the visible image 2, and the polarization image 3 are all adjusted to the center of the image plane, and then the fields of view of the multiple cameras are made consistent by using a mechanical device.

[0038] The polarization images specifically include a 0° polarization image, a 45° polarization image, a 90° polarization image, a 135° polarization image, a polarization degree image, and a polarization phase angle image.

[0039] S2: Use the multi-dimensional image acquisition results obtained in step S1 to create a bird detection dataset.

[0040] The bird detection dataset conforms to the VOC2007 format.

[0041] Step S2 specifically includes the following steps:

[0042] S21: Perform super-resolution operation on the infrared image 1 using bilinear interpolation method, so that the resolution of each group of infrared image 1, visible image 2 and polarization image 3 with the same field of view is the same:

[0043] I high (2x,2y)=1 / 4[I low (x,y)+I low (x+1,y)+I low (x,y+1)+I low (x+1,y+1)];

[0044] Among them, I low (x, y) is the infrared image 1, I high(x, y) is the visible image 2 and the polarization image 3, and the resolution of the visible image 2 and the polarization image 3 is the same;

[0045] S22: marking each group of infrared images 1, visible images 2 and polarization images 3 with the same field of view, and the bird type 61 and bird pixel position 62 contained in each group;

[0046] S23: taking a union of the annotation boxes of the flying bird pixel positions 62 contained in each set of infrared image 1, visible image 2 and polarization image 3 with the same field of view;

[0047] S24: Divide the infrared image 1, the visible image 2, and the polarization image 3 with the same field of view after the union into a training set and a verification set in a ratio of 9:1, so as to realize the production of a flying bird detection data set.

[0048] S3: Construct a deep neural network for bird detection, and use the bird detection dataset to train the deep neural network for bird detection to obtain a trained deep neural network for bird detection.

[0049] In step S3, the loss function used to train the bird detection deep neural network is:

[0050]

[0051] Among them, Loss is the loss function, L cls is the Sigmoid loss function, L reg is the L1 loss function, i is the total samples of the bird detection dataset, j is the positive sample of the bird detection dataset, N pos is the number of positive samples.

[0052] S4: performing multi-dimensional image acquisition on the flying birds in the detection area, and inputting the multi-dimensional image acquisition results into the trained bird detection deep neural network to detect the flying bird type 61 and the flying bird pixel position 62;

[0053] The detection of flying bird pixel positions 62 is used to identify specific areas where birds inhabit.

[0054] S5: Count the types and numbers of flying birds in the detected area according to the detection results of step S4.

[0055] Different types of birds are marked with boxes of different colors, and the number of boxes of various colors is counted to realize the statistics of the types and numbers of flying birds in the area to be tested.

[0056] The unmanned aerial vehicle is also equipped with a GPS positioning unit, which is used to calibrate the spatial position of the flying bird captured by the same set of infrared images 1, visible images 2 and polarization images 3 with the same field of view.

[0057] The flight operation specifications of drones comply with the "Interim Regulations on Unmanned Aerial Vehicle Flight Management". The flight altitude is higher than 50m to ensure the shooting range of a single picture.

[0058] like Figure 2 As shown, the bird detection deep neural network includes a multidimensional information fusion unit 4 and a scale-aware bidirectional feature pyramid network unit 5. Each group of infrared images 1, visible images 2 and polarization images 3 with the same field of view are stacked by channel to obtain a stacked image 7, and the stacked image 7 is input into the multidimensional information fusion unit 4 for image fusion to obtain a multidimensional fused image. The multidimensional fused image is subjected to feature extraction by the scale-aware bidirectional feature pyramid network unit 5 to obtain the bird type 61 and the bird pixel position 62 on each group of multidimensional images with the same field of view.

[0059] The network structure of the scale-aware bidirectional feature pyramid network unit 5 used in the embodiment of the present invention comes from the scale-aware bidirectional feature pyramid network (including bidirectional feature pyramid and SSD unit) in the paper "Scale-aware bidirectional feature pyramid network for traffic sign detection". The scale-aware bidirectional feature pyramid network unit 5 used in the embodiment of the present invention improves the detection accuracy and robustness of multi-scale targets by introducing a bidirectional pyramid structure and optimizing feature fusion.

[0060] like Figure 3 As shown, the multidimensional information fusion unit 4 includes a first 1×1 convolution module 10, a second 1×1 convolution module 11, a third 1×1 convolution module 14, a fourth 1×1 convolution module 17, a fifth 1×1 convolution module 18, a softmax module 12, a cascaded channel average module 8 and a channel maximum module 9, wherein the stacked image 7 is sequentially processed by the cascaded channel average module 8 and the channel maximum module 9, the first 1×1 convolution module 10, the second 1×1 convolution module 11 and the softmax module 12 to obtain a first feature 13, the stacked image 7 is further convolved by the third 1×1 convolution module 14 to obtain a second feature 15, the first feature 13 is multiplied by the second feature 15 to obtain a third feature 16, the third feature 16 is input into the fourth 1×1 convolution module 17 for convolution processing to obtain a fourth feature, the fourth feature is input into the fifth 1×1 convolution module 18 for convolution processing to obtain a fifth feature, the fifth feature is multiplied by the stacked image 7 to obtain an output feature 19.

[0061] That is, the dimension of the stacked image 7 is [c, w, h], c is the number of features, w is the feature width, and h is the feature length. When the stacked image 7 is processed by the cascaded channel average module 8 and the channel maximum module 9, the dimension of the feature obtained is [2, w, h]. After being processed by the first 1×1 convolution module 10, the dimension of the feature obtained is [r, w, h]. After being processed by the second 1×1 convolution module 11, the dimension of the feature obtained is [1, w, h]. After being flattened by the softmax module 12, the dimension of the first feature 13 obtained is [wh, 1]. When the stacked image 7 is After being processed by the third 1×1 convolution module 14, the dimension of the second feature 15 obtained is [c, wh, 1]. The first feature 13 is multiplied by the second feature 15, and the dimension of the third feature 16 obtained is [c, 1, 1]. The third feature 16 is input into the fourth 1×1 convolution module 17 for convolution processing, and the dimension of the fourth feature obtained is [c / r, 1, 1]. The fourth feature is input into the fifth 1×1 convolution module 18 for convolution processing, and the dimension of the fifth feature obtained is [c, 1, 1]. The fifth feature is multiplied by the stacked image 7 to obtain the output feature 19.

[0062] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0063] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A wide-area bird flock detection and statistics method based on multi-dimensional information fusion, characterized by: The specific steps include: S1: A visible camera, an infrared camera, and a polarization camera with the same field of view are mounted on an unmanned aerial vehicle to collect multi-dimensional images of birds in a selected area; S2: Using the multi-dimensional image acquisition results obtained in step S1 to create a bird detection data set; S3: constructing a bird detection deep neural network, and using the bird detection data set to train the bird detection deep neural network to obtain a trained bird detection deep neural network; S4: performing multi-dimensional image acquisition of the flying birds in the detection area, and inputting the multi-dimensional image acquisition results into the trained bird detection deep neural network to detect the flying bird type and the flying bird pixel position; S5: Count the types and numbers of flying birds in the detected area according to the detection results of step S4.

2. The wide-area bird flock detection and statistics method based on multi-dimensional information fusion according to claim 1, characterized in that: In step S1 , the multi-dimensional image includes an infrared image, a visible image, and a polarization image.

3. The wide-area bird flock detection and statistics method based on multi-dimensional information fusion according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21: Use bilinear interpolation to perform super-resolution operation on infrared images, so that the resolution of each set of infrared images with the same field of view is the same as that of visible images and polarization images: I high (2x,2y)=1 / 4[I low (x,y)+I low (x+1,y)+I low (x,y+1)+I low (x+1,y+1)]; Among them, I low (x, y) is the infrared image, I high (x, y) is the visible image and polarization image; S22: annotate each group of infrared images, visible images and polarization images with the same field of view, the type of flying birds contained in each group and the pixel position of the flying birds; S23: taking a union of the annotation boxes of the pixel positions of the flying birds contained in each set of infrared images, visible images and polarization images with the same field of view; S24: Divide the groups of infrared images, visible images and polarization images with the same field of view after taking the union into a training set and a verification set, so as to realize the preparation of the flying bird detection data set.

4. The wide-area bird flock detection and statistics method based on multi-dimensional information fusion according to claim 3, characterized in that: In step S3, the bird detection deep neural network includes a multidimensional information fusion unit and a scale-aware bidirectional feature pyramid network unit. Each group of infrared images, visible images and polarization images with the same field of view are stacked by channel to obtain a stacked image, and the stacked image is input into the multidimensional information fusion unit for image fusion to obtain a multidimensional fused image. The multidimensional fused image is subjected to feature extraction by the scale-aware bidirectional feature pyramid network unit to obtain the bird type and bird pixel position on each group of multidimensional images with the same field of view.

5. The wide-area bird flock detection and statistics method based on multi-dimensional information fusion according to claim 4, characterized in that: The multidimensional information fusion unit includes a first 1×1 convolution module, a second 1×1 convolution module, a third 1×1 convolution module, a fourth 1×1 convolution module, a fifth 1×1 convolution module, a softmax module, a cascaded channel average module and a channel maximum module, wherein the stacked image is sequentially processed by the cascaded channel average module and the channel maximum module, the first 1×1 convolution module, the second 1×1 convolution module and the softmax module to obtain a first feature, the stacked image is further convolved by the third 1×1 convolution module to obtain a second feature, the first feature is multiplied by the second feature to obtain a third feature, the third feature is input into the fourth 1×1 convolution module for convolution processing to obtain a fourth feature, the fourth feature is input into the fifth 1×1 convolution module for convolution processing to obtain a fifth feature, and the fifth feature is multiplied by the stacked image to obtain an output feature.

6. The wide-area bird flock detection and statistics method based on multi-dimensional information fusion according to claim 5, characterized in that: The polarization images specifically include a 0° polarization image, a 45° polarization image, a 90° polarization image, a 135° polarization image, a polarization degree image, and a polarization phase angle image.

Citation Information

Patent Citations

  • All-weather portable multifunctional bionic positioning, posture-determining and observing-aiming system and method

    CN111652276A

  • Visual perception device and method for ship navigation environment

    CN113705375A

  • Infrared, visible light and polarization image fusion method based on deep learning

    CN116128780A

  • Flying bird detection method based on improved YOLO-v5 algorithm

    CN117496350A

  • Unmanned aerial vehicle flying bird detection method, system, equipment and medium

    CN117576724A