Bird detection and species determination

Abstract

A method for determining a bird species in flight is provided along with a corresponding system. The method may include capturing a video stream of a bird in flight (402) using at least one camera (104), generating a first species probability estimate (405) by delivering the images to a neural network (304) trained to recognize bird species from the images, obtaining additional parameters (401) from the video stream or from additional data, generating a second species probability estimate (407) by delivering the additional parameters as input to a domain knowledge module (306) having a domain knowledge statistical model, and generating a final species probability estimate (408) by combining the first and second species probability estimates. The additional parameters may include geometric features related to the bird's movement in flight, or parameters related to the environment.

Description

Technical Field

[0001] The present invention relates to the automatic detection and species determination of birds, and particularly to the detection and species identification using image recognition and machine learning.

Background Art

[0002] As the world moves towards reducing its dependence on fossil energy sources, wind power is becoming one of the important alternative technologies. However, there are certain drawbacks associated with wind power. Among these are the impacts on biodiversity, and particularly dangerous wind farms correspond to bird populations. The wind power industry may have to conduct surveys of bird populations before installing wind farms in order to estimate how the wind farms will impact, and may also need to monitor the bird population patterns and their occurrence around installed wind farms.

[0003] In most cases, current systems for automatic monitoring do not provide sufficiently high-quality data and are expensive in terms of the necessary equipment and processing power, so bird observations have to be done manually. For obvious reasons, manual observations are unrealistic and cannot be part of the continuous monitoring of bird populations over time. Therefore, better systems based on computer vision, image processing, as well as statistical processing and modeling are needed.

[0004] Among the more specific needs of the industry are systems for the detection, tracking and classification of bird species near wind farms. The availability of such systems can greatly improve the effectiveness and efficiency of conducting environmental impact assessments (EIAs) before and after the construction of wind farms. Furthermore, such systems can assist the operators of wind farms in adjusting the layout of the farms, controlling operations based on the bird species and behavior currently or recently observed, and implementing mitigation measures to prevent birds from colliding with wind turbines during operation or before construction by taking other mitigation measures.

[0005] To develop such a system, several technical challenges must be overcome. These challenges may relate to the requirement for high-quality data input, including determining what data to acquire and how accurate or detailed the data needs to be. Other challenges concern the methods needed to process the input data to extract features indicating the presence and behavior of birds, statistical models for correctly interpreting the extracted features, and initiating appropriate mitigation measures. [Overview of the project]

[0006] This specification discloses methods, apparatus, and systems that address many of the above-mentioned requirements in order to facilitate better mitigation of risks that wind turbines generally pose to birds, particularly vulnerable or endangered bird species. In particular, the present invention addresses the problem of observing birds, detecting their presence, determining individual bird species, and developing statistics. Such results can be stored, displayed, or delivered as input to control processes in order to better plan the construction of wind power plants, control their operation, and initiate deterrence and reduction.

[0007] According to a first aspect of the present invention, a method for determining the species of a bird in flight is provided. The method includes capturing at least one video stream of a bird in flight using at least one camera; generating a first species probability estimate by delivering images from the at least one video stream as input to an artificial neural network trained to recognize bird species from images; generating a second species probability estimate by obtaining additional parameters from at least one video stream or at least one additional data source; delivering the obtained additional parameters as input to a domain knowledge module having a domain knowledge statistical model; and generating a final species probability estimate by combining the first and second species probability estimates. The domain knowledge statistical model may be an influence diagram, for example, a Bayesian belief network.

[0008] Additional parameters may be derived from the video stream, or obtained from other data sources such as additional sensors, or from services accessible via a network such as the Internet.

[0009] Embodiments of the present invention may further include extracting geometric features relating to a bird in flight by delivering images from at least one video stream as input to a geometric feature extraction module. The output from this process may be used in one or both of the following ways: Extracted geometric features that may relate to how the bird moves in flight may be used to contribute to the generation of a first species probability estimate by being delivered together with features extracted from an artificial neural network as input to a shallow neural network trained to generate bird species probabilities based on features extracted by the artificial neural network in combination with observed geometric features. The first species probability estimate may be the probability of one species or several species, determined to be the most likely.

[0010] Alternatively, or in addition, geometric features extracted from the geometric extraction module can contribute to the generation of a second species probability estimate by delivering input to a domain knowledge statistical model. In other words, the extracted geometric features can be part of the additional parameters obtained.

[0011] In some embodiments of the present invention, only one camera may be present, or several cameras may have a field of view that is at least partially not covered by the other cameras, such that only one video stream is available. Extracted geometric features may then be obtained based on the identification of the same bird in a sequence of images from one video stream and the estimation of motion based on the change in the position of the identified bird between images in the sequence of images. This may be combined with an estimate of the distance from the camera and other methods, including several cameras.

[0012] In embodiments where at least one camera is two or more cameras and at least one video stream is two or more video streams, extracted geometric features may be obtained using multiview geometric analysis to determine the known position of each camera, the identification of the same bird in two or more sequences of images from two or more simultaneous video streams, the position of the identified bird in each image of each video stream, and 3D coordinates representing the position of the identified bird relative to the camera position from the determined position in each image of each video stream. The determined 3D coordinates can then be used to extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight path, and flapping frequency.

[0013] In some embodiments, one of the extracted geometric features is a flapping frequency determined by performing a Fourier analysis (e.g., using a Fast Fourier Transform, FFT) on a sequence of images from at least one video stream and identifying the dominant frequency component within a frequency interval that matches the flapping frequency of a bird.

[0014] Artificial neural networks can be trained by delivering a dataset containing labeled images of relevant bird species as input to the artificial neural network.

[0015] To detect and track birds in video images, object detection on the images may be performed from at least one video stream, and the images may be annotated with bounding boxes drawn around each object identified as a bird. The annotations may also include identifications representing the identification of individual birds from one frame to the next. These annotations may be used by artificial neural networks and geometric feature extraction. This object detection is performed using a second artificial neural network.

[0016] The species determined to have the highest determined final species probability may be delivered as an output, which may be used to control deterrent or mitigation measures to reduce the risk of birds of the determined species being injured by wind power plant equipment. Alternatively or additionally, the output may be stored or displayed.

[0017] A second aspect of the present invention provides a system for determining the species of a bird in flight. Such a system may include at least one video camera, an artificial neural network configured to receive video images from at least one of the cameras (104) and to be trained to recognize bird species from the images, a domain knowledge module having a domain knowledge statistical model configured to receive observations of additional parameters and to generate probabilities of observing each of the bird species given the observations of the additional parameters, and a species determination module configured to receive a first species probability estimate based on the output from the artificial neural network and a second species probability estimate based on the output from the domain knowledge module and to generate a final species probability estimate. The domain knowledge statistical model may be an influence diagram, for example, a Bayesian belief network.

[0018] Some embodiments of such a system may also have a geometric feature extraction module configured to receive at least one video stream from at least one camera and extract geometric features related to birds captured in flight within at least one video stream. Depending on further use of the extracted geometric features, the system may also include a shallow neural network configured to receive features extracted from an artificial neural network and geometric features extracted from the geometric feature extraction module and to generate a first species probability estimate. The system may also include a configuration of a domain knowledge module that allows receiving geometric features extracted from the geometric feature extraction module as additional parameters. In other words, the extracted geometric features may be used in combination with features extracted by the artificial neural network to generate a first species probability estimate, features extracted by the domain knowledge module to generate a second species probability estimate, or both.

[0019] Some embodiments of the system may also include a multi-view geometric analysis module configured to receive known positions of each camera, receive data related to at least two simultaneous video streams, determine the positions of identified birds in each image of each video stream, and use multi-view geometric analysis to determine 3D coordinates representing the positions of identified birds relative to the camera positions from the determined positions in each image of each video stream. Such a multi-view geometric analysis module may be part of a geometric feature module or may act as a preprocessor for feature extraction to make the estimated positions available for feature extraction. The geometric feature extraction module may then receive the 3D coordinates determined from the multi-view geometric analysis module and, based on the received 3D coordinates, extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency. Additional parameters or variables are possible within the scope of the invention.

[0020] A bird detection and tracking module may be included to facilitate the detection and tracking of birds in video images. The bird detection and tracking module may be configured to receive input from at least one video camera, perform object detection, and annotate the image by drawing a bounding box around each object identified as a bird. Each camera may have its own detection and tracking device, or a single computer may perform detection and tracking on video images from several cameras. The provided annotations, including bounding boxes and identification from one frame to the next, are typically included in the video data before feature extraction is performed. The bird detection and tracking module may include a second artificial neural network.

[0021] In all aspects and embodiments, one or more of the artificial neural networks may be convolutional neural networks.

[0022] The system according to the present invention may deliver a final species probability estimate as an output used to control processes of memory, display, or deterrence or reduction in order to reduce the risk that a determined species of bird will be damaged by wind power plant facilities. [Brief explanation of the drawing]

[0023] [Figure 1] This diagram shows a wind power generation base utilizing the system according to the present invention. [Figure 2] This illustrates the limitations of conventional systems that rely on only one method for determining bird species. [Figure 3] This is a block diagram showing modules included in an exemplary embodiment of the present invention. [Figure 4] This is a signal flow diagram showing how information flows through the system according to an embodiment of the present invention. [Figure 5] This is a diagram of a bird detected within an image and enclosed in a bounding box. [Figure 6] It is a diagram of positioning based on multi-view geometry. [Figure 7] It is an example of a Bayesian belief network that can be used in an embodiment of the present invention.

Best Mode for Carrying Out the Invention

[0024] In the following description of embodiments, reference is made to the drawings, and like reference numerals indicate the same or corresponding elements. If the drawings include multiple elements that are essentially multiple examples of the same element, they are not all necessarily labeled with reference numerals in order to avoid cluttering the drawings. The drawings are not necessarily to scale. Instead, certain features may be shown exaggerated, either to scale, or in a somewhat simplified or schematic manner, and certain conventional elements may be excluded for the purpose of illustrating the principles of the present invention rather than cluttering the drawings with details that do not contribute to an understanding of these principles.

[0025] Note that, unless otherwise specified, different features or elements can be combined with each other, regardless of whether they are described together as part of the same embodiment below. Combinations of features or elements in the exemplary embodiments are not intended to limit the scope of the present invention to a limited set of embodiments, but are made for the purpose of facilitating an understanding of the present invention, and are intended to be interchangeable as long as alternative elements having substantially the same function are shown in each embodiment, but no attempt has been made to disclose a complete description of all possible substitutions of features for the sake of brevity.

[0026] Furthermore, those skilled in the art will understand that the present invention can be implemented without many of the details included in this detailed description. Conversely, some well-known structures or functions may not be illustrated or described in detail in order to avoid unnecessarily obscuring the associated description of various implementations. The terms used in the description presented below are intended to be interpreted in the broadest reasonable manner, even if used in conjunction with a detailed description of a particular implementation of the present invention.

[0027] Referring first to Figure 1, which shows a representative wind power base 100 having several wind turbines 102, some of which are equipped with cameras 104. At least some of the cameras 104 may instead be mounted on a building, radio tower, or other man-made or natural object. The cameras 104 may be connected to a local computer system 106 capable of performing some of the operations according to the present invention. The local computer system 106 may comprise one or more computers. In some embodiments, the local computer system 106 simply collects data from the cameras 104 and transfers the collected data to a remote location such as one or more servers 108. The data received at the remote location may be stored in a database 110 for later processing, or delivered as input to one or more processing services that may be executed by the servers 108 and accessible as cloud services. Processing, storing, and providing the collected information and processing results is collectively referred to as cloud services in this disclosure.

[0028] After the data is processed by the server 108, the results may be stored in the database 110, from which the computer 112 may access them for display or use to control the process or to make decisions.

[0029] Various components included in system 100 can be connected to a computer network 114, such as the Internet.

[0030] In addition to camera 104, other sensors or data sources may be included in the system or provide data to the system. Such additional sensors or data sources, not shown in the drawings, may include sources that provide meteorological data such as temperature, precipitation, wind, atmospheric pressure, and humidity. These sources may be sensors that are part of the system, or the system may be configured to acquire data from external sources or services, or both, accessible via network 114.

[0031] In some embodiments, the local computer system 106 may be configured to perform operations on images collected by the camera 104 before transferring the results to a cloud service. Such edge computing may include filtering, denoising, normalization, and other preprocessing, as well as other forms of object detection and tracking, stereophotogrammetry, or 3D reconstruction and feature extraction, as will be described in more detail below. In some embodiments, even species identification may be performed as edge computing, i.e., all processing may be performed at the edge, but the results and statistics may be stored centrally or otherwise distributed. Other embodiments may not include any edge computing at all. Instead, all processing may be performed as a cloud service or in a dedicated data center.

[0032] Camera 104 is connected to a local computer system 106 using wired or wireless streaming of data. In some embodiments, the local computer system 106 comprises one or more computers that receive video streams from multiple cameras. In other embodiments, each camera 104 has a dedicated computer module that receives data only from that camera and transfers that data as a separate stream to a cloud service. Such dedicated computer modules may be further integrated into each camera 104.

[0033] The connection from the local computer system 106 to the cloud service may be wired or wireless. Fiber optic cables may be used to enable the provision of large amounts of video data from an offshore wind power plant to an onshore server 108.

[0034] As described above, processing can be distributed such that some processing is performed as edge computing on one or more local computers 106, while additional processing is performed as a cloud-based service by servers 108 connected to the network 114. The servers themselves may be one or more computers, and they may be located in a single location or distributed across several locations. It should be understood that the system itself may be configured to utilize certain external computing resources, such as cloud-based machine learning services (machine learning as a service, i.e., MLaaS). The present invention is not limited to any particular distribution of functionality, except that the cameras and sensors must, of course, be located where they can acquire the necessary information. All other functionality may be located in a single data center or distributed across several machines and / or locations in all possible permutations.

[0035] In recent years, there has been much progress in the field of computer vision (the research field that enables computers to interpret and understand the visual world). This progress has been made primarily by the use of deep learning, a type of machine learning algorithm based on artificial neural networks that works particularly well with "unstructured data" such as images and text. However, current solutions using computer vision to detect and identify birds require relatively high-resolution images. To acquire such images, expensive, high-quality cameras are required, and the birds must be relatively close to the cameras when those images are captured. This means that a great many cameras would be needed to cover an entire wind farm.

[0036] Figure 2 shows an example of the capabilities of a typical state-of-the-art system. This example is only an approximation and assumes that 20 pixels are needed to detect a bird and 2000 pixels are needed to determine the bird's species. For bird 201, which has a wingspan of approximately 1 meter and a camera with 4k pixel image resolution, a 48mm lens, and a 4 / 3-inch sensor, this means that the bird can be detected at a distance of approximately 520 meters, but for the species to be determined, the bird must come within 5 meters of the camera. The actual specifications of existing systems may differ from this example, but this illustrates the point that the ability to identify species decreases dramatically with distance, and the potential benefits of having alternative or additional strategies, such as simply adding more cameras with higher resolution and more expensive optics.

[0037] Therefore, it will be understood that the capabilities of a bird detection and species determination system are a trade-off between camera resolution, focal length, sensor size, and the number of cameras installed, as well as the available bandwidth and computing power.

[0038] This invention is based on the recognition that computer vision and deep learning can be combined with additional strategies. In particular, this invention leverages knowledge and information in a way that is based on how a skilled ornithologist recognizes birds, using domain knowledge statistical models, such as influence diagrams like Bayesian belief networks (BNNs). The exemplary embodiments described below utilize BNNs as the domain knowledge statistical model, but this invention is not limited to this particular type of statistical modeling, and other statistical models known in the art can be used as the domain knowledge statistical model. Similarly, the exemplary embodiments primarily utilize convolutional neural networks. However, this invention is not limited to convolutional neural networks, particularly when detection and tracking, as well as species recognition, are performed by a single neural network. Therefore, those skilled in the art will understand that this invention can be implemented using other types of artificial neural networks.

[0039] Figure 3 shows a modular structure of an exemplary system according to the present invention. The system includes a plurality of video cameras 104. These cameras deliver video input to a bird detection and tracking module 301. In some embodiments, these modules are implemented as edge computing services by having a dedicated computer connected to one or a group of the cameras 104 to detect objects that are likely to be birds and to annotate the video data before passing the video data on. The annotations include drawing bounding boxes around at least identified birds and may also include identification tags that can help identify the same bird frame by frame to track individual birds and enable motion analysis. The edge computer may also be configured to upload video data only when a potential bird is detected, so that bandwidth is not used to transmit unused video images.

[0040] The bird detection and tracking module 301, described in more detail below, may be connected to a multi-view geometric analysis module 302. The multi-view geometric analysis module 302 may be configured to estimate the three-dimensional (3D) coordinates of the bird being detected and tracked in the video input data received from the bird detection and tracking module 301, using stereophotogrammetry. Related techniques such as stereophotogrammetry, epipolar geometry, and 3D pose estimation are well known in the art and will not be described in detail herein, but a brief explanation is given below with reference to Figure 6. The output from the geometric analysis module is provided to a geometric feature extraction module 303. This module extracts features such as the bird's altitude, speed, flight pattern, flapping frequency, and size. Some of these features, such as altitude and flight pattern, may be derived directly from the 3D coordinates determined by the geometric analysis module 302. Other features may need to be combined with other information. The size may be determined, for example, based on the distance from the camera (e.g., based on the camera's 3D coordinates and known position) and the size of the bounding box generated by the detection and tracking module 301. The generation of the bounding box is described in more detail below. In some embodiments, the flapping frequency can be determined solely from the variation of the 3D coordinates. In other embodiments, image analysis of the data within the bounding box may be performed, for example, by using Fourier analysis. The present invention is not limited to the features mentioned in this example, and other features may be considered in some embodiments.

[0041] Annotated video data from the bird detection and tracking module 301 is also provided to a convolutional neural network (CNN) 304. This CNN 304 can be trained on video data of birds of known species. The CNN 304 analyzes each bounding box (i.e., each bird in the video image that may contain several detected birds) as a separate image. During training, the CNN 304 learns to identify features. These features are delivered as output from the CNN 304.

[0042] To combine the features detected by the geometric feature extraction module 303 with those detected by the CNN 304, these are joined in a combined model neural network 305. This may be a shallow neural network (SNN) 305 that takes the outputs from the geometric feature extraction module 303 and the CNN 304 as input and generates an output that is an estimate of the probability that the observed bird belongs to each species.

[0043] In addition to feature detection based on convolutional neural networks and engineering features based on geometric analysis, a third contribution to species determination is provided by domain knowledge module 306. This module includes statistical models, which may take the form of influence diagrams, particularly in Bayesian belief networks (BBNs). This module, described in more detail below, is based on the type of knowledge that ornithologists may rely on when determining the species of observed birds. The exact type of information to be considered in knowledge base module 306 may vary depending on the geographical location, the bird species typical of that location, etc. For example, if related bird species behave differently in response to wind, information about the current wind conditions is important. However, if all related bird species behave in the same way (or not at all) as the wind conditions change, the current wind conditions do not improve the prediction.

[0044] Inputs to the knowledge base module 306 may be collected from additional sensors 302. These sensors may be directly connected to the system, as already mentioned, or data from these sensors may be available from online services such as weather services. Other potentially relevant data may include time of day, season, birds observed in the relatively recent past (i.e., a history of recent species determinations made by the system), and features extracted by the geometric feature extraction module, such as flap frequency, speed, altitude, and flight pattern. The output from the knowledge base module 305 is a set of probabilities that the observed bird species is one of each species configured to be recognized by the system.

[0045] Therefore, the outputs from the domain knowledge module 306 and the coupling module SNN305 can be combined by the final species determination module 307. Several methods are possible to determine the combined probability distribution. One possibility is to simply calculate the mean or weighted mean of the given probabilities. Another possibility is to use a Bayesian method. How much weight each probability should be given can be determined on a conditional basis. For example, if the quality of the video input is poor (e.g., the bounding box has fewer pixels because the bird is far from the camera), the relative weights of the probabilities from the neural network may be reduced, and if some of the values ​​used as input to the network are estimates or default values, the relative weights of the BBN may be reduced because accurate measurements are not available.

[0046] Modifications to the exemplary embodiments shown in Figure 3 are possible without departing from the scope and spirit of the invention. For example, some embodiments may include only a single video camera 104, or several cameras may have fields of view that do not completely overlap with those of other cameras in the system, so that detection and species determination of birds captured by only one of multiple cameras can be performed. Similarly, some embodiments may use CNN 304 to perform species determination directly without input of geometric features from geometric feature extraction module 303. In such embodiments, the coupled model neural network 305 may be omitted, and the extracted geometric features are delivered only to the domain knowledge module 306. In other words, geometric features may be omitted entirely from some embodiments, in some embodiments they are used only by the domain knowledge module 306, in some embodiments they are used only by the coupled model neural network 305, and in some embodiments geometric features are used by both.

[0047] Here, we refer to Figure 4, which shows an embodiment of the present invention, for example, the flow of information and related data processing in the system of Figure 3. The columns in this figure roughly correspond to the modules in Figure 3. The description of this flow will be given sequentially as needed, but it should be noted that the flow of information may be continuous and the processing may be parallel. Processing may be performed in a different manner, in parallel, asynchronously, or in a different order than described below, except that the output from one module must be available as needed before it can be processed by the following modules.

[0048] In the first column of the diagram, data is collected. Process 401 obtains environmental parameters from, for example, an additional sensor module 302, online services, and an internal clock and table. Process 402 captures video from camera 104. The captured video is then subjected to object detection and tracking in process 403. This process may also be performed by the edge computer 106, which is described in more detail below. The output from this process is an annotated video image. In particular, detected objects (birds) may be enclosed by bounding boxes. The output from the object detection and tracking process 403 is transferred to two different processes that may operate in parallel. In process 404, geometric features are extracted. This process represents multi-view geometric analysis performed by module 302 and geometric feature extraction performed by module 303. The output is design parameters such as altitude, velocity, flapping frequency, and flight pattern, as described above.

[0049] Another branch that receives input from the object detection and tracking process 403 is the feature extraction process 405, which is performed by the convolutional neural network 304.

[0050] The results of geometric feature extraction in process 404 and the output from the CNN feature extraction process 405 are supplied as input to process 406 for generating a combined model based on the outputs from both. It should be noted that the input to this process does not necessarily have to include all the features extracted by process 404. In embodiments of the present invention, process 406 utilizes a shallow neural network 305 that receives probabilities as input from the CNN and geometric features and delivers a modified probability distribution as output. As already mentioned, in some embodiments, geometric features may be delivered only to the knowledge base module 306, in which case it is not necessary to combine the learned features from CNN 304 with the artificial features from the geometric feature extraction module 303, and process 406 can be omitted.

[0051] Environmental parameters from process 401, and optionally some geometric features extracted by process 404, are used as input to a knowledge-based process 407, which may be executed by the domain knowledge module 306. Using some, all, or none of the geometric features from process 303 in process 407 is consistent with the principles of the present invention. The geometry-based features used by processes 406 and 407 may be subsets of the output from process 404, and these subsets may be identical, overlapping, or different.

[0052] As mentioned above with reference to Figure 3, process 407 can be performed by a module that utilizes a Bayesian belief network or some similar representation of conditional probability. This will be described in more detail below.

[0053] The output from knowledge-based process 407 is a probability distribution representing the probability that the observed bird belongs to each species, taking into account the parameters delivered as input. In other words, at this stage, two probability distributions are generated: one generated by the neural network from the video input and geometric features, and the other generated by the Bayesian belief network based on environmental parameters and geometric features. The species can then be determined by combining the two distributions in process 408 and generating the final output in process 409. The final output may be the species with the highest probability, or it may include a confidence index based on the probabilities determined for the other species. For example, if the most likely (and therefore determined) species has a 55% probability and the second most likely species has a 40% probability, the confidence may be lower than if the probabilities of the two most likely species were 80% and 7%, respectively. The confidence can also be adjusted depending on whether the input to the BBN is an accurate and recent measurement, or an uncertain estimate or default value. If the probability distributions generated by two processes are similar, this can similarly enhance confidence; however, very different probabilities from the two processes can decrease confidence.

[0054] Referring here to Figure 5, the bird detection and tracking performed by the bird detection and tracking module 301 is described in more detail. As already described, object detection and tracking may also be performed by edge computing on one or more computers 106 connected to the camera 104, in which case the bird detection and tracking module 301 would be a combination of appropriate software installed on one or more computers 106 in combination with the computer's processing hardware. However, performing object detection and tracking remotely using the bird detection and tracking module 301 implemented on one or more computers on the network 114, for example as a cloud service, is consistent with the principles of the present invention. However, there may be advantages to performing these processes as edge computing, i.e., near the camera that captures the video image. For example, if a frame represents the size of the entire photograph 501 captured by the video camera 104, this image 501 contains a lot of background without relevant information. All background may be compressed by setting it to a uniform color, and the background may be excluded as a whole, with only the portion containing the relevant object, i.e., the bird, being transmitted. This can save considerable bandwidth.

[0055] The captured video images 501 are annotated by having bounding boxes inserted around each detected bird 201. Each bounding box can be associated with an identification criterion. In Figure 5, these are shown as reference numerals 201.01, 201.02, etc. The process for detecting and tracking objects can be implemented using a neural network (not necessarily the same neural network as the CNN304 and SNN305 described above, but this is also possible). Training neural networks for object detection and tracking is well known in the art. Available solutions include YOLOv3, an algorithm described in the DarkNet framework. DarkNet is an open-source neural network framework described in C and CUDA. YOLOv3 can be used to implement bird detection and tracking in the system according to the present invention. As described above, in some embodiments, object detection and tracking may be performed by the same neural network 304 trained to perform species determination, meaning that the neural network 304 encompasses one large model that also embodies the bird detection and tracking module 301. In these embodiments, the object detection and tracking module 301 is represented as part of the capabilities of the neural network 304, and the detection and tracking information provided to the multi-view geometric analysis module 302 is provided as output from the neural network 304. In these embodiments, the neural network may have a more complex architecture than a CNN and may be more precisely referred to as an artificial neural network, or ANN 304. The exemplary embodiments described herein include a separate detection and tracking module 301 and CNN 304, but this can be generalized to include embodiments having an ANN 304 that incorporates the functionality of the detection and tracking module 301. This is not explicitly shown in the drawings but will be readily apparent to those skilled in the art.

[0056] Each bounding box has a position within the image. This information, as well as the known positions and fields of view of each camera 104, defines a line from camera 104 in a particular direction. As shown in Figure 6, the multi-view geometric analysis module 302 can use input from several cameras 104A, 104B to identify objects present in several images as intersections of such lines. In Figure 6, bird 201 is located at position 606 in image 501A captured by camera 104A and at position 607 in image 501B captured by camera 104B. The three-dimensional coordinates of bird 201's position can be calculated based on the known positions of cameras 104A and 104B and bird 201's positions 606 and 607 in the images. If cameras 104A and 104B are within each other's fields of view, they are present in each other's images at positions 603 and 604 respectively, and this can be used to further calibrate the position determination.

[0057] The determined position of the observed bird can be used to provide variables or parameters such as flight altitude. By tracking the same bird over multiple frames over a period of time, it becomes possible to determine additional variables such as velocity, acceleration, vertical motion, and combinations thereof. More advanced geometric features from the flight path, such as curvature, may be parameterized and used as variables.

[0058] In embodiments having only one camera 104, or embodiments having the capability to extract geometric features related to birds detected in images from only one camera for non-overlapping fields of view, it is not possible to use multi-view geometric analysis such as stereophotogrammetry or epipolar geometry. Thus, the position of the detected bird 201 relative to camera 104 may be based on the position of the bounding box in only one image, but may be combined with other methods for distance estimation or distance search. Such methods are known from photography, computer vision and other arts and may include the use of additional neural networks. Detection of flapping frequency may be performed in the same manner as with several cameras.

[0059] Geometric feature extraction was described as being performed by two modules: a multi-view geometric analysis module 302 (not multi-view in the case of only one camera) and a geometric feature extraction module 303. This is because the first of these modules primarily uses methods such as triangulation (or direction and distance estimation) to determine the 3D position, while the latter primarily performs feature extraction based on the determined position. This means that the two modules may be implemented as separate software modules or different software applications, and they may run on different computers. However, implementing the two modules on a single computer is consistent with the principles of the present invention, even if the functionality is included in the same software package, software library, or software application.

[0060] Referring here to Figure 7, the domain knowledge module 306 is described in more detail. The domain knowledge module 306 takes as input various parameters on which a trained ornithologist may rely when determining what he or she is observing. As already mentioned above, what these parameters are may differ from system to system, depending on which bird species are dominant in a given area, which environmental parameters affect different bird species in different ways, and which parameters can be reliably obtained. For example, in a system designed primarily to distinguish between eagles and gulls, the parameters that make these species adapt their behavior in the same way are of less interest than parameters that cause different behavior in the two species. For example, if a particular type of weather (e.g., based on temperature or precipitation) keeps one species nesting and the other active, the relevant weather parameters will affect the probability of observing each type of bird. Conversely, if both species are equally active throughout the year, season is not a particularly valuable parameter for determining the probability.

[0061] A skilled ornithologist with access to relevant statistics can illustrate how various parameters relate in an influence diagram or with respect to some other statistical model. A type of influence diagram that may be particularly suitable in this context, and used in this exemplary embodiment, is a Bayesian belief network (or Bayesian network). A Bayesian belief network is a graphical model that represents a set of variables and their conditional dependencies. The network is a directed acyclic graph, where nodes represent variables and arrows (edges) represent conditional dependencies. Inputs to a node are sets of values ​​given as outputs from the node's parent, and the outputs are the probabilities or probability distributions of the variables represented by the node.

[0062] The design of Bayesian belief networks is based on knowledge of relations and conditional probabilities, and Figure 7 is intended to illustrate the principles behind such networks. It should be noted that this example is for illustrative purposes only and does not necessarily represent a network that would be useful in a real-world environment. For example, color is included as a variable due to its dependency, but observing color in a work environment may not be practical for various reasons.

[0063] Some of the variables are independent and can only be observed directly. These variables have no parents and, in the example shown in Figure 7, include season 701, temperature 702, and time 703. Although temperature 702 can actually conditionally depend on both season 701 and time 703, the fact that temperature is directly observable and there is no need to estimate the probability of a particular season or time considering the observed temperature, nor to estimate the probability of temperature considering a known season or time, means that these dependencies do not need to be included in the model.

[0064] The probability that an observed bird is a particular species 704 may depend on the season 701, temperature 702, and time of day 703. Thus, a conditional probability table for species 704 can be established. This is simplest when all variables are discrete (for example, in the example in Figure 7, true and false, which are values ​​such as summer (winter), above freezing (below freezing), and daytime (night), or can take only a limited number of discrete values ​​such as summer, autumn, winter, and spring), but it should be noted that Bayesian belief networks can also be constructed for continuous variables. This is well known in the art and will not be explained in further detail.

[0065] Several additional variables may depend on a given species 704. In the example in Figure 7, these include the variables color 705, flight altitude 706, flight pattern 707, and flapping frequency 708. Furthermore, color 705 is also seasonal. The dependence between color and season is not necessarily useful for offshore wind farms, but it may be potentially useful in mountainous areas where some birds change their plumage color seasonally.

[0066] Based on statistics on common bird species in the relevant domain, their migration patterns, behavioral responses to various weather conditions, and knowledge of flight patterns, altitude, and flap frequency for various species, a table of conditional probabilities can be constructed. Which variables to include and what probabilities to assign will likely depend on a combination of statistics and other knowledge that ornithologists can provide. The same applies to whether variables are discrete, continuous but can be discretized (e.g., above or below a threshold), or should be continuous within the model.

[0067] The following table is included for illustrative purposes only, but it shows what the conditions table might look like. [Table 1] [Table 2]

[0068] This example includes probabilities invented for illustrative purposes only and does not represent any specific real-world environment. The first table illustrates that, given various possible combinations of season, temperature, and time of day, the probability that an observed bird is either an eagle or a seagull can be as shown. The second table gives the probability that, if the observed bird is a seagull or an eagle, it is flying above or below 200 meters. Similar tables can be established for all variables. Based on the graph and associated conditional probability tables, it becomes possible to establish probabilities of unobserved variables based on observed variables. Note that such conditional probabilities can be established in both directions. They are not limited to establishing probabilities of child nodes given observations of parent variables. For example, in the above example, if the bird is observed flying above 200 meters, the probability that the bird is an eagle and not a seagull is much higher than if the bird were observed below 200 meters. How likely this is depends on the probability that the observed bird is an eagle in the first place, which also depends on other variables in the network.

[0069] Therefore, by providing observations of as many variables as possible, the probabilities of unobserved variables can be changed accordingly. The updated probabilities given for observed variables are sometimes called posterior probabilities.

[0070] The Bayesian belief network is programmed into the domain knowledge module 306 and can automatically generate estimated probabilities based on observed variables received from the geometric feature extraction module 303 (e.g., altitude, flapping frequency, flight pattern) and additional sensor modules 302 (e.g., temperature, season, time of day). The domain knowledge module 306 may be implemented as a combination of software and hardware on a single computer, on several computers, or on a cloud service.

[0071] As already mentioned, the exact variables or parameters to include may depend on the conditions within the area where the particular system is installed.

[0072] After species detection determines that a given observation is a bird belonging to a specific species, possibly combined with additional data such as confidence and some of the variables used by the domain knowledge module 306, the output can be stored in the database 110. Aggregated data can be used to create statistics, and all current, historical, and statistical information can be made accessible to user computers 112 connected to the network 114 and authorized to access the database 110.

[0073] The data may be used to plan future operations and to initiate suppression and reduction, for example, by activating sound suppression or temporary shutdown of wind power plants. Such means may be activated manually by a person accessing the information available in the database, or computer 112 may be configured to continuously monitor the data available in database 110 or receive it directly from the system and automatically activate suppression or reduction means based on predetermined conditions.

Claims

1. A method for determining the species of a bird in flight, Capturing at least one video stream of a bird in flight using at least one camera (104) (402), A first species probability estimate (405) is generated by delivering images from at least one video stream as input to an artificial neural network (304) trained to recognize bird species from images, Obtaining additional parameters from at least one video stream or from at least one additional data source (401), The process involves generating a second species probability estimate (407) by distributing the acquired additional parameters as input to a domain knowledge module (306) having a domain knowledge statistical model, The first seed probability estimate and the second seed probability estimate are combined to generate the final seed probability estimate (408), This includes extracting geometric features related to the flying bird (404) by delivering images from the at least one video stream as input to a geometric feature extraction module (303), and further -Generating the first species probability estimate (406) by delivering the features extracted from the artificial neural network (304) and the geometric features extracted from the geometric feature extraction module (303) as input to a shallow neural network (305) trained to generate the probability of a bird species based on features extracted by an artificial neural network combined with observed geometric features, - The second species probability estimate is generated by distributing the geometric features extracted from the geometric extraction module (303) as additional parameters obtained and input to the domain knowledge statistical model, A method that includes performing at least one of the following.

2. The method according to claim 1, wherein the extracted geometric features are obtained based on the identification of the same bird in an image sequence from the at least one video stream, and the motion is estimated based on the change in the position of the identified bird between images in the image sequence.

3. The at least one camera is two or more cameras, and the at least one video stream is two or more video streams, The extracted geometric features are used to determine the 3D coordinates representing the position of the identified bird relative to the position of the camera (104) from the determined position in each image of each video stream, using the known position of each camera (104), the identification of the same bird in two or more sequences of images from two or more simultaneous video streams, the determination of the position of the identified bird in each image of each video stream, and multi-view geometric analysis. The method according to claim 1 or 2, wherein the determined 3D coordinates are used to extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency.

4. The method according to claim 1 or 2, wherein one extracted geometric feature is a flapping frequency determined by performing a Fourier analysis on a series of images from at least one video stream and identifying a dominant frequency component within a frequency interval that matches the flapping frequency of a bird.

5. The method according to claim 1 or 2, further comprising training the artificial neural network (304) by delivering a dataset containing labeled images of relevant bird species as input to the artificial neural network (304).

6. The method according to claim 1 or 2, further comprising performing object detection on images from at least one video stream (403) and annotating the images with bounding boxes drawn around each object identified as a bird.

7. The method according to claim 6, wherein object detection is performed using a second artificial neural network (301).

8. The method according to claim 1 or 2, further comprising providing as output the species having the highest determined final species probability (409).

9. The method of claim 8, further comprising using the output to control deterrent or reduction measures in order to reduce the risk of the birds of the determined species being injured by the wind power plant equipment.

10. The method according to claim 1 or 2, wherein the domain knowledge statistical model is a Bayesian belief network, and / or one or more artificial neural networks are convolutional neural networks.

11. A system for determining the species of a bird in flight, At least one video camera (104) and An artificial neural network (304) configured to receive video images from at least one of the cameras (104) and to recognize bird species from the images, A domain knowledge module (306) having a domain knowledge statistical model configured to receive observations of additional parameters and generate probabilities of observing each species of bird given the observations of the additional parameters, A species determination module (307) is configured to receive a first species probability estimate based on the output from the artificial neural network (304) and a second species probability estimate based on the output from the domain knowledge module (306), and to generate a final species probability estimate. The system further comprises a geometric feature extraction module (303) configured to receive at least one video stream from the at least one camera (104) and to extract geometric features relating to a bird captured in flight within the at least one video stream, and - A shallow neural network (305) configured to receive features extracted from the artificial neural network (304) and geometric features extracted from the geometric feature extraction module (303), and to generate the first seed probability estimate, - The configuration of the domain knowledge module (306) that enables the reception of geometric features extracted from the geometric feature extraction module (303) as additional parameters, A system comprising at least one of the following:

12. The system according to claim 11, wherein the geometric feature extraction module is configured to receive data relating to at least one video stream, extract geometric features from the at least one video stream based on the identification of the same bird in an image sequence, and estimate motion based on the change in the position of the identified bird between images in the image sequence.

13. The at least one camera is two or more cameras, and the at least one video stream is two or more video streams, The system further comprises a multi-view geometric analysis module (302) configured to receive known positions of each camera (104), receive data relating to at least two simultaneous video streams, determine the position of identified birds in each image of each video stream, determine the position of the identified birds in each image of each video stream, and use multi-view geometric analysis to determine 3D coordinates representing the position of the identified birds relative to the position of the camera (104) from the determined positions in each image of each video stream, The system according to claim 11 or 12, wherein the geometric feature extraction module (303) is further configured to receive the determined 3D coordinates from the multi-view geometric analysis module (302) and to extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency based on the received 3D coordinates.

14. The system according to claim 11 or 12, wherein the geometric feature extraction module (303) is further configured to determine a flapping frequency by performing a Fourier analysis on a series of images from at least one video stream and identifying dominant frequency components within a frequency interval that matches the flapping frequency of a bird.

15. The system according to claim 11 or 12, further comprising a bird detection and tracking module (301) configured to receive input from at least one video camera (104), perform object detection, and annotate the image by drawing a bounding box around each object identified as a bird.

16. The system according to claim 15, wherein the bird detection and tracking module (301) includes a second artificial neural network.

17. The system according to claim 11 or 12, wherein the species determination module (307) is further configured to deliver the final species probability estimate as an output used to store, display, or use to control a deterrence or reduction process in order to reduce the risk that birds of the determined species will be damaged by wind power plant facilities.

18. The system according to claim 11 or 12, wherein the domain knowledge statistical model is a Bayesian belief network, and / or one or more artificial neural networks are convolutional neural networks.

Images