Bird detection and species determination

Abstract

A method for determining a bird species in flight is provided along with a corresponding system. The method may include capturing a video stream of a bird in flight (402) using at least one camera (104), generating a first species probability estimate (405) by delivering the images to a neural network (304) trained to recognize bird species from the images, obtaining additional parameters (401) from the video stream or from additional data, generating a second species probability estimate (407) by delivering the additional parameters as input to a domain knowledge module (306) having a domain knowledge statistical model, and generating a final species probability estimate (408) by combining the first and second species probability estimates. The additional parameters may include geometric features related to the bird's movement in flight, or parameters related to the environment.

Description

[Technical field]

[0001] The present invention relates to automated bird detection and species determination, and in particular to detection and species identification using image recognition and machine learning. [Background technology]

[0002] As the world moves towards reducing its reliance on fossil energy sources, wind power is one of the technologies that is becoming an important alternative. However, there are certain drawbacks associated with wind power. Among these are its impacts on biodiversity, particularly the risk of wind farms affecting bird populations. The wind industry may need to carry out surveys of bird populations before setting up wind farms to estimate how the farms will affect them, and may need to monitor bird population patterns and their occurrence around the installed wind farms.

[0003] In most cases, bird observations must be done manually, as current systems for automated monitoring do not provide sufficiently high quality data and are expensive in terms of the equipment and processing power required. For obvious reasons, manual observations are impractical and cannot be part of the continuous monitoring of bird populations over time. Therefore, there is a need for better systems based on computer vision, image processing, and statistical processing and modeling.

[0004] Among the more specific needs of the industry are systems for the detection, tracking and classification of bird species near wind farms. The availability of such systems would significantly improve the effectiveness and efficiency of performing environmental impact assessments (EIAs) pre- and post-construction of wind farms. Furthermore, such systems could assist wind farm operators in implementing mitigation measures to prevent birds from striking wind turbines during operation or before construction by adjusting the layout of the farm, controlling operations based on current or recently observed bird species and behaviors, and taking other mitigation measures.

[0005] To develop such a system, several technical challenges must be overcome. Such challenges may relate to requirements for high quality data input, including determining what data to acquire as well as how accurate or detailed the data needs to be. Other challenges relate to the methods needed to process the input data to extract features indicative of bird presence and behavior, statistical models to correctly interpret the extracted features, initiating appropriate mitigation measures, etc. Summary of the Invention

[0006] This specification discloses methods, apparatus, and systems that address many of the above-mentioned requirements to facilitate better mitigation of the risks that wind turbines pose to birds in general, and vulnerable or endangered bird species in particular. In particular, the invention addresses the problem of observing birds, detecting their presence, determining individual bird species, and developing statistics. Such results can be stored, displayed, or distributed as inputs to a control process to better plan the construction of wind farms, control their operation, and initiate suppression and curtailment.

[0007] According to a first aspect of the present invention, there is provided a method for determining the species of a bird in flight. The method includes: capturing at least one video stream of a bird in flight using at least one camera; generating a first species probability estimate by delivering images from the at least one video stream as input to an artificial neural network trained to recognize bird species from images; acquiring additional parameters from the at least one video stream or at least one additional data source; generating a second species probability estimate by delivering the acquired additional parameters as input to a domain knowledge module having a domain knowledge statistical model; and generating a final species probability estimate by combining the first and second species probability estimates. The domain knowledge statistical model can be an influence diagram, e.g., a Bayesian belief network.

[0008] The additional parameters may be derived from the video stream or may be obtained from other data sources, such as additional sensors or services accessible over a network, such as the Internet.

[0009] An embodiment of the present invention may further include extracting geometric features related to the bird in flight by delivering images from at least one video stream as input to a geometric feature extraction module. Output from this process may be used in one or both of the following ways: The extracted geometric features, which may be related to how the bird moves in flight, may be used to contribute to the generation of a first species probability estimate by being delivered along with the extracted features from the artificial neural network as input to a shallow neural network trained to generate bird species probabilities based on the features extracted by the artificial neural network combined with the observed geometric features. The first species probability estimate may be the probability of one species, or each of several species, determined to be most likely.

[0010] Alternatively or in addition, the geometric features extracted from the geometric extraction module may contribute to the generation of the second seed probability estimate by delivering inputs to the domain knowledge statistical model. In other words, the extracted geometric features may be part of the obtained additional parameters.

[0011] In some embodiments of the invention, there may be only one camera, or some cameras have a field of view that is at least partially not covered by other cameras, such that only one video stream is available. Extracted geometric features may then be obtained based on identification of the same bird in a sequence of images from one video stream and an estimation of the movement based on the change in the position of the identified bird between images in the sequence of images. This may be combined with estimates of the distance from the camera and also with other methods involving several cameras.

[0012] In embodiments where the at least one camera is two or more cameras and the at least one video stream is two or more video streams, the extracted geometric features may be obtained using a known position of each camera, identification of the same bird in two or more sequences of images from the two or more simultaneous video streams, determining the position of the identified bird in each image of each video stream, and multi-view geometric analysis to determine 3D coordinates representing the position of the identified bird relative to the position of the camera from the determined position in each image of each video stream. The determined 3D coordinates may then be used to extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency.

[0013] In some embodiments, one of the extracted geometric features is a flapping frequency, determined by performing a Fourier analysis (e.g., using a Fast Fourier Transform, FFT) on a sequence of images from at least one video stream and identifying dominant frequency components that lie within a frequency interval consistent with the bird's flapping frequency.

[0014] The artificial neural network may be trained by delivering a dataset containing labeled images of relevant bird species as input to the artificial neural network.

[0015] To detect and track birds in the video images, object detection on images from at least one video stream may be performed and the images may be annotated with bounding boxes drawn around each object identified as a bird. The annotation may also include an identification representing the identity of the individual bird from one frame to the next. This annotation may be used by the artificial neural network and geometric feature extraction. This object detection is performed using a second artificial neural network.

[0016] The determined species having the highest determined final species probability may be delivered as an output, which may be used to control deterrence or reduction measures to reduce the risk of birds of the determined species being injured by the wind farm installation. Alternatively or additionally, the output may be stored or displayed.

[0017] In a second aspect of the invention, a system for determining the species of a bird in flight is provided. Such a system may include at least one video camera, an artificial neural network configured to receive video images from at least one of the cameras (104) and be trained to recognize bird species from the images, a domain knowledge module having a domain knowledge statistical model configured to receive observed values ​​of an additional parameter and generate a probability of observing each species of bird given the observed values ​​of the additional parameter, and a species determination module configured to receive a first species probability estimate based on an output from the artificial neural network, receive a second species probability estimate based on an output from the domain knowledge module, and generate a final species probability estimate. The domain knowledge statistical model may be an influence diagram, e.g. a Bayesian belief network.

[0018] Some embodiments of such a system may also have a geometric feature extraction module configured to receive at least one video stream from the at least one camera and extract geometric features associated with birds captured in flight within the at least one video stream. Depending on further use of the extracted geometric features, the system may also include a shallow neural network configured to receive the features extracted from the artificial neural network and the geometric features extracted from the geometric feature extraction module and generate a first species probability estimate. The system may also include a configuration of a domain knowledge module that allows it to receive the geometric features extracted from the geometric feature extraction module as additional parameters. In other words, the extracted geometric features may be used in combination with the features extracted by the artificial neural network to generate a first species probability estimate, the features extracted by the domain knowledge module to generate a second species probability estimate, or both.

[0019] Some embodiments of the system may also include a multi-view geometric analysis module configured to receive the known positions of each camera, receive data associated with at least two simultaneous video streams, determine the positions of identified birds in each image of the respective video streams, and use multi-view geometric analysis to determine 3D coordinates representing the positions of the identified birds relative to the positions of the cameras from the determined positions in each image of the respective video streams. Such a multi-view geometric analysis module may be part of the geometric feature module or may operate as a pre-processor to the feature extraction to make the estimated positions available for feature extraction. The geometric feature extraction module may then be configured to receive the determined 3D coordinates from the multi-view geometric analysis module and extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency based on the received 3D coordinates. Additional parameters or variables are possible within the scope of the invention.

[0020] A bird detection and tracking module may be included to facilitate detection and tracking of birds in the video images. The bird detection and tracking module may be configured to receive input from at least one video camera, perform object detection, and annotate the images by drawing a bounding box around each object identified as a bird. Each camera may have its own detection and tracking device, or one computer may perform detection and tracking on video images from several cameras. The provided annotations, including the bounding boxes and identification from one frame to the next, are typically included in the video data before feature extraction is performed. The bird detection and tracking module may include a second artificial neural network.

[0021] For all aspects and embodiments, one or more of the artificial neural networks may be a convolutional neural network.

[0022] The system of the present invention may deliver the final species probability estimate as an output that may be stored, displayed, or used to control a deterrence or reduction process to reduce the risk that birds of the determined species will be damaged by the wind farm installation. [Brief description of the drawings]

[0023] [Figure 1] 1 shows a diagram of a wind farm utilizing a system according to the invention; [Diagram 2] This illustrates the limitations of prior art systems that rely on only one method of bird species determination. [Diagram 3] FIG. 2 is a block diagram illustrating modules included in an exemplary embodiment of the present invention. [Figure 4] FIG. 4 is a signal flow diagram illustrating how information flows through a system according to an embodiment of the present invention. [Diagram 5] A diagram of a bird detected in an image and surrounded by a bounding box. [Figure 6] FIG. 1 is an illustration of position determination based on multi-view geometry. [Figure 7] 1 is an example of a Bayesian belief network that may be used in embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the following description of the embodiments, reference is made to the drawings, in which like reference numbers indicate the same or corresponding elements. Where the drawings include multiple elements that are multiple instances of essentially the same element, they are not all referenced to avoid cluttering the drawings. The drawings are not necessarily to scale. Instead, certain features may be shown to scale or exaggerated in a somewhat simplified or schematic manner, and certain conventional elements may be omitted for the purpose of illustrating the principles of the invention, rather than cluttering the drawings with details that do not contribute to an understanding of these principles.

[0025] It should be noted that, unless otherwise stated, different features or elements may be combined with each other, regardless of whether they are described together as part of the same embodiment below. The combination of features or elements in the exemplary embodiments is made to facilitate understanding of the invention, rather than to limit the scope of the invention to a limited set of embodiments, and to the extent that alternative elements having substantially the same functionality are shown in each embodiment, they are intended to be interchangeable, although for the sake of brevity, no attempt has been made to disclose a complete description of all possible permutations of features.

[0026] Furthermore, those skilled in the art will understand that the present invention may be practiced without many of the details contained in this detailed description. Conversely, some well-known structures or functions may not be shown or described in detail to avoid unnecessarily obscuring the relevant description of various implementations. The terms used in the description presented below are intended to be interpreted in their broadest reasonable manner, even when used in conjunction with detailed descriptions of certain specific implementations of the present invention.

[0027] Referring initially to FIG. 1 , which shows a representation of a wind farm 100 having several wind turbines 102, some of which are equipped with cameras 104. At least some of the cameras 104 may alternatively be mounted on buildings, radio towers, or other man-made or natural objects. The cameras 104 may be connected to a local computer system 106 that may perform several operations according to the present invention. The local computer system 106 may comprise one or more computers. In some embodiments, the local computer system 106 simply collects data from the cameras 104 and forwards the collected data to a remote location, such as one or more servers 108. The data received at the remote location may be stored in a database 110 for later processing or delivered as input to one or more processing services that may be performed by the servers 108 and accessible as cloud services. The processing, storage, and provision of the collected information and processing results are collectively referred to in this disclosure as cloud services.

[0028] After the data is processed by the server 108, the results may be stored in a database 110 from which they may be accessed by a computer 112 for display or use to control a process or make decisions.

[0029] The various components included in the system 100 may be connected to a computer network 114, such as the Internet.

[0030] In addition to the camera 104, other sensors or data sources may also be included in the system or provide data to the system. Such additional sensors or data sources, while not shown in the drawings, may include sources providing weather data such as temperature, precipitation, wind, pressure, and humidity. These sources may be sensors that are part of the system, or the system may be configured to obtain data from external sources and / or services accessible via the network 114.

[0031] The local computer system 106 may, in some embodiments, be configured to perform operations on images collected by the camera 104 before transferring the results to a cloud service. Such edge computing may include filtering, denoising, normalization, and other pre-processing, as well as object detection and tracking, stereo photogrammetry, or other forms of 3D reconstruction and feature extraction, as described in more detail below. In some embodiments, even species identification may be performed as edge computing, i.e., all processing may be performed at the edge, but results and statistics may be stored centrally or otherwise distributed. Other embodiments may not include any edge computing. Instead, all processing may be performed as a cloud service or in a dedicated data center.

[0032] The cameras 104 are connected to a local computer system 106 using wired or wireless streaming of data. In some embodiments, the local computer system 106 comprises one or more computers that receive the video streams from the multiple cameras. In other embodiments, each camera 104 has a dedicated computer module that receives data from only that camera and forwards the data to a cloud service as a separate stream. Such dedicated computer modules may further be integrated into each camera 104.

[0033] The connection from the local computer system 106 to the cloud service may be wired or wireless. Fiber optic cables may be used to enable large amounts of video data to be provided from the offshore wind farm to the onshore server 108.

[0034] As mentioned above, the processing may be distributed such that some processing is performed as edge computing on one or more local computers 106, while additional processing is performed as a cloud-based service executed by a server 108 connected to the network 114. The server itself may be one or more computers, which may be located in a single location or distributed across several locations. It should be understood that the system itself may be configured to take advantage of certain external computational resources, such as cloud-based machine learning services (machine learning as a service, i.e., MLaaS). The invention is of course not limited to any particular distribution of functions, except that the cameras and sensors must be located in locations where the necessary information can be obtained. All other functions may be located in one single data center or distributed across several machines and / or locations in all possible permutations.

[0035] In recent years, there has been a lot of progress in the field of computer vision, the research field that focuses on enabling computers to interpret and understand the visual world. This progress has been made primarily through the use of deep learning, a type of machine learning algorithm based on artificial neural networks that works particularly well with "unstructured data" such as images and text. However, current solutions that use computer vision to detect and identify birds require relatively high-resolution images. To obtain such images, expensive high-quality cameras are needed, and the birds must be relatively close to the cameras when their images are captured. This means that a very large number of cameras are needed to cover an entire wind farm.

[0036] An example of the capabilities of a typical state-of-the-art system is shown in Figure 2. This example is only an approximation and assumes that 20 pixels are needed to detect the bird and 2000 pixels are needed to determine the bird's species. For a bird 201 with a wingspan of about 1 meter and a camera with 4k pixel image resolution, a 48mm lens, and a 4 / 3 inch sensor, this means that the bird can be detected at a distance of about 520 meters, but for the species to be determined, the bird must reach a distance of within 5 meters of the camera. While the actual specifications of existing systems may differ from this example, it illustrates the point at which the ability to identify species drops off dramatically with distance, and the potential benefits of having an alternative or additional strategy to simply adding more cameras with higher resolution and more expensive optics.

[0037] It will therefore be appreciated that the capability of a bird detection and species determination system is a trade-off between camera resolution, focal length, sensor size and number of cameras installed, as well as available bandwidth and computational power.

[0038] The present invention is based on the recognition that computer vision and deep learning can be combined with additional strategies. In particular, the present invention uses domain knowledge statistical models, such as influence diagrams, such as Bayesian belief networks (BNNs), to leverage knowledge and information in a manner based on how a skilled ornithologist would recognize a bird. The exemplary embodiment described below utilizes a BNN as the domain knowledge statistical model, but the present invention is not limited to this particular type of statistical modeling, and other statistical models known in the art can be used as the domain knowledge statistical model. Similarly, the exemplary embodiment primarily utilizes convolutional neural networks. However, the present invention is not limited to convolutional neural networks, especially in the case where detection and tracking as well as species recognition are performed by a single neural network. Thus, one skilled in the art will understand that the present invention can be implemented using other types of artificial neural networks.

[0039] FIG. 3 illustrates a modular structure of an exemplary system according to the present invention. The system includes multiple video cameras 104. These cameras deliver video input to a bird detection and tracking module 301. In some embodiments, these modules are implemented as an edge computing service, for example, by having one dedicated computer connected to one or a group of cameras 104 to detect objects that are likely to be birds and annotate the video data before passing it on. The annotation includes at least drawing a bounding box around identified birds and may also include an identification tag that may help identify the same bird from frame to frame to track individual birds and enable movement analysis. The edge computer may also be configured to upload video data only when a possible bird is detected so that bandwidth is not used to transmit unused video images.

[0040] The bird detection and tracking module 301, which will be described in more detail below, may be connected to the multi-view geometric analysis module 302. The multi-view geometric analysis module 302 may be configured to use stereo photogrammetry to estimate three-dimensional (3D) coordinates of birds detected and tracked in the video input data received from the bird detection and tracking module 301. Stereo photogrammetry and related techniques such as epipolar geometry and 3D pose estimation are well known in the art and will not be described in detail herein, but will be briefly described below with reference to FIG. 6. The output from the geometric analysis module is provided to a geometric feature extraction module 303. This module extracts features such as the bird's altitude, speed, flight pattern, flapping frequency, and size. Some of these features may be derived directly from the 3D coordinates determined by the geometric analysis module 302, such as altitude and flight pattern. Other features may have to combine this with other information. The size may be determined, for example, based on the distance from the camera (e.g., based on the camera's 3D coordinates and known position) and the size of the bounding box generated by the detection and tracking module 301. The generation of the bounding box is described in more detail below. In some embodiments, the flapping frequency may be determined solely from the variation of the 3D coordinates. In other embodiments, image analysis of the data within the bounding box may be performed, for example by using Fourier analysis. The invention is not limited to the features mentioned in this example, and other features may be considered in some embodiments.

[0041] The annotated video data from the bird detection and tracking module 301 is also provided to a convolutional neural network (CNN) 304. This CNN 304 may be trained on video data of birds of known species. The CNN 304 analyzes each bounding box (i.e., each bird in a video image, which may contain several detected birds) as a separate image. During training, the CNN 304 learns to identify features. These features are delivered as output from the CNN 304.

[0042] To combine the features detected by the geometric feature extraction module 303 with the features detected by the CNN 304, they are combined in a combined model neural network 305. This may be a shallow neural network (SNN) 305 that takes the outputs from the geometric feature extraction module 303 and the CNN 304 as inputs and produces outputs that are estimates of the probability that the observed bird is of the respective species.

[0043] In addition to feature detection based on convolutional neural networks and engineering features based on geometric analysis, a third contribution to species determination is provided by the domain knowledge module 306. This module includes statistical models, which may be in the form of influence diagrams, especially in Bayesian belief networks (BBNs). This module, which will be explained in more detail below, is based on the type of knowledge that an ornithologist may rely on in determining the species of an observed bird. The exact type of information to be considered in the knowledge base module 306 may vary depending on the geographic location, the bird species typical for that location, etc. For example, information about the current wind conditions is important if birds of related species behave differently in response to wind. However, if all birds of related species exhibit behaviors that change in the same way (or do not change at all) as wind conditions change, the current wind conditions will not improve the prediction.

[0044] Inputs to the knowledge base module 306 may be collected from additional sensors 302. These sensors may be directly connected to the system as already mentioned, or data from these sensors may be available from online services such as weather services. Other data that may be relevant include time of day, season, birds observed in the relatively recent past (i.e., the history of recent species determinations made by the system), and features extracted by the geometric feature extraction module such as flapping frequency, speed, altitude, flight pattern, etc. The output from the knowledge base module 305 is a set of probabilities that the observed bird species is any of the respective species that the system is configured to recognize.

[0045] Thus, the outputs from the domain knowledge module 306 and the combination module SNN 305 can be combined by the final species determination module 307. To determine the joint probability distribution, several methods are possible. One possibility is to simply calculate the average or weighted average of the provided probabilities. Another possibility is to use a Bayesian method. How much weight should be given to each probability can be determined depending on the conditions. For example, if the quality of the video input is low (e.g., the bird is far away from the camera, so the bounding box has few pixels), the relative weight of the probabilities from the neural network can be reduced, and if some of the values ​​used as inputs to the network are estimates or default values, the relative weight of the BBN can be reduced because accurate measurements are not available.

[0046] Variations to the exemplary embodiment shown in FIG. 3 are possible without departing from the scope and spirit of the present invention. For example, some embodiments may include only a single video camera 104, or may perform bird detection and species determination for birds captured by only one of multiple cameras, as some cameras have fields of view that are not completely overlapped by other cameras in the system. Similarly, some embodiments may use the CNN 304 to perform species determination directly without input of geometric features from the geometric feature extraction module 303. In such embodiments, there may be no combined model neural network 305, and the extracted geometric features are delivered only to the domain knowledge module 306. In other words, the geometric features may be omitted entirely from some embodiments, and in some embodiments, they are used only by the domain knowledge module 306, in some embodiments, they are used only by the combined model neural network 305, and in some embodiments, the geometric features are used by both.

[0047] Reference is now made to Figure 4, which illustrates the flow of information and associated data processing in an embodiment of the invention, e.g., the system of Figure 3. The columns in this diagram correspond roughly to the modules of Figure 3. Note that while the description of this flow is given sequentially where necessary, the flow of information may be sequential and the processing may be parallel. Processing may be performed differently, in parallel, asynchronously, or in a different order than described below, except for the fact that the output from one module must be available as necessary before it can be processed by the following module.

[0048] In the first column of the diagram, data is collected. Process 401 obtains environmental parameters, for example from additional sensor modules 302, online services, and internal clocks and tables. Process 402 captures video from camera 104. The captured video is then subject to object detection and tracking in process 403. This processing may be performed by edge computer 106 and is described in more detail below. The output from this process is annotated video images. In particular, detected objects (birds) may be surrounded by bounding boxes. The output from object detection and tracking process 403 is forwarded to two different processes that may run in parallel. In process 404, geometric features are extracted. This process represents the multi-view geometric analysis performed by module 302 and the geometric feature extraction performed by module 303. The output is design parameters such as altitude, speed, flapping frequency, flight pattern, etc., as described above.

[0049] The other branch that receives input from the object detection and tracking process 403 is the feature extraction process 405, which is performed by the convolutional neural network 304.

[0050] The results of the extraction of geometric features in process 404 and the output from CNN feature extraction process 405 are provided as inputs to process 406 for generating a combined model based on the outputs from both. Note that the input to this process does not necessarily have to include all the features extracted by process 404. In embodiments of the invention, process 406 utilizes a shallow neural network 305 that receives probabilities from the CNN and the geometric features as inputs and delivers modified probability distributions as outputs. As already mentioned, some embodiments may deliver the geometric features only to the knowledge base module 306, in which case there is no need to combine the learned features from the CNN 304 with the artificial features from the geometric feature extraction module 303 and process 406 can be omitted.

[0051] The environmental parameters from process 401, and optionally some of the geometric features extracted by process 404, are used as inputs to a knowledge-based process 407, which may be executed by domain knowledge module 306. Using none, some, or all of the geometric features from process 303 in process 407 is consistent with the principles of the invention. The geometry-based features used by process 406 and process 407 may be subsets of the output from process 404, and these subsets may be identical, overlapping, or different.

[0052] 3, process 407 may be performed by a module that utilizes a Bayesian belief network or some similar representation of conditional probabilities, as described in more detail below.

[0053] The output from the knowledge base process 407 is a probability distribution that represents the probability that the observed bird belongs to the respective species given the parameters delivered as input. In other words, at this stage, two probability distributions have been generated: one generated by a neural network from the video input and geometric features, and the other generated by a Bayesian belief network based on the environmental parameters and geometric features. Now, a species determination can be made by combining the two distributions in process 408 and generating a final output by process 409. The final output may be the species with the highest probability and may include an indication of confidence based on the determined probabilities for the other species. For example, if the most likely (and thus determined species) has a probability of 55% and the second most likely species has a probability of 40%, the confidence may be lower than if the respective probabilities of the two most likely species were 80% and 7%. The confidence may also be adjusted depending on whether the inputs to the BBN are accurate and recent measurements or uncertain estimates or default values. If the probability distributions produced by the two processes are similar, this may similarly strengthen the confidence, but very different probabilities from the two processes may weaken the confidence.

[0054] With reference now to FIG. 5, the bird detection and tracking performed by the bird detection and tracking module 301 will be explained in more detail. As already explained, the object detection and tracking may be performed by edge computing by one or more computers 106 connected to the camera 104, in which case the bird detection and tracking module 301 will be a combination of suitable software installed on one or more computers 106 in combination with the processing hardware of the computer. However, it is consistent with the principles of the present invention to perform the object detection and tracking remotely, for example as a cloud service, with the bird detection and tracking module 301 implemented on one or more computers on the network 114. However, there may be advantages to performing these processes as edge computing, i.e. close to the camera capturing the video image. For example, if a frame represents the size of the entire photo 501 captured by the video camera 104, this image 501 contains a lot of background without any relevant information. All the background may be compressed by setting it to be a uniform color, or the background may be totally excluded and only the part containing the relevant object, i.e. the bird, transmitted. This may save considerable bandwidth.

[0055] The captured video images 501 are annotated in that a bounding box is inserted around each detected bird 201. Each bounding box may be associated with an identification criterion. In FIG. 5, this is shown as references 201.01, 201.02, etc. The process for detecting and tracking objects may be implemented using a neural network (usually not the same neural network as the CNN 304 and SNN 305 described above, although this is also possible). Training neural networks for object detection and tracking is well known in the art. Available solutions include YOLOv3, an algorithm written in the DarkNet framework. DarkNet is an open source neural network framework written in C and CUDA. YOLOv3 may be used to implement bird detection and tracking in the system according to the invention. As mentioned above, in some embodiments, object detection and tracking may be performed by the same neural network 304 trained to perform species determination, meaning that the neural network 304 encompasses one large model that also embodies the bird detection and tracking module 301. In these embodiments, the object detection and tracking module 301 is represented as part of the capabilities of the neural network 304, and the detection and tracking information provided to the multi-view geometric analysis module 302 is provided as an output from the neural network 304. In these embodiments, the neural network may have a more complex architecture than a CNN and may more accurately be referred to as an artificial neural network, or ANN 304. While the exemplary embodiments described herein include a separate detection and tracking module 301 and CNN 304, this may be generalized to include embodiments having an ANN 304 that includes the functionality of the detection and tracking module 301. This is not explicitly shown in the drawings, but will be readily understood by one of ordinary skill in the art.

[0056] Each bounding box has a location in the image. This information, as well as the known positions and fields of view of each camera 104, defines a line from the camera 104 in a particular direction. As shown in FIG. 6, the multi-view geometric analysis module 302 may use input from several cameras 104A, 104B to identify objects present in several images as the intersection of such lines. In FIG. 6, a bird 201 is located at 606 in an image 501A captured by camera 104A and at a location 607 in an image 501B captured by camera 104B. The three-dimensional coordinates of the bird 201's location can be calculated based on the known positions of the cameras 104A, 104B and the bird 201's locations 606, 607 in the images. If the cameras 104A, 104B are within each other's fields of view, they will be present in each other's images at locations 603 and 604, respectively, which can be used to further calibrate the location determination.

[0057] The determined position of an observed bird can be used to provide variables or parameters such as flight altitude. Tracking the same bird over multiple frames over a period of time allows for the determination of additional variables such as speed, acceleration, vertical motion, and combinations of these. More advanced geometric features from the flight trajectory such as curvature may be parameterized and used as variables.

[0058] In embodiments with only one camera 104 or with the ability to extract geometric features related to birds detected in images from only one camera due to non-overlapping fields of view, it is not possible to use multi-view geometric analysis such as stereo photogrammetry or epipolar geometry. The position of the detected bird 201 relative to the camera 104 may therefore be based on the position of a bounding box in only one image, but may be combined with other methods for distance estimation or distance searching. Such methods are known from photography, computer vision and other technical fields and may include the use of additional neural networks. The detection of the flapping frequency may be performed in the same way as with several cameras.

[0059] The geometric feature extraction has been described as being performed by two modules: a multi-view geometric analysis module 302 (not multi-view in the case of only one camera) and a geometric feature extraction module 303. This is because the first of these modules mainly uses methods such as triangulation (or orientation and distance estimation) to determine the 3D position, while the latter mainly performs feature extraction based on the determined position. This means that the two modules may be implemented as separate software modules or different software applications, and they may run on different computers. However, implementing the two modules on a single computer, even as functions contained in the same software package, software library, or software application, is consistent with the principles of the present invention.

[0060] Now referring to FIG. 7, the domain knowledge module 306 is described in more detail. The domain knowledge module 306 takes as input various parameters on which a trained ornithologist may rely when determining what he or she is observing. As already mentioned above, which parameters this is may vary from system to system, depending on which bird species are dominant in an area, which environmental parameters affect different bird species in different ways, and which parameters can be reliably obtained. For example, in a system designed primarily to distinguish between eagles and gulls, parameters that make these species adapt their behavior in the same way are of less interest than parameters that cause different behavior in the two species. For example, if a particular type of weather (e.g., based on temperature or precipitation) causes one species to remain nesting and the other to be active, the relevant weather parameters will affect the probability of observing each type of bird. Conversely, if both species are equally active throughout the year, the season is not a particularly valuable parameter for determining the probability.

[0061] A skilled ornithologist with access to the relevant statistics can represent how the various parameters are related in an influence diagram or with respect to some other statistical model. A type of influence diagram that may be particularly suited to this context and that is used in this exemplary embodiment is a Bayesian Belief Network (or Bayesian Network). A Bayesian Belief Network is a graphical model that represents a set of variables and their conditional dependencies. The network is a directed acyclic graph, where the nodes represent the variables and the arrows (edges) represent the conditional dependencies. The inputs to a node are sets of values ​​given as outputs from the node's parents, and the outputs are probabilities or probability distributions for the variables represented by the node.

[0062] The design of Bayesian belief networks is based on knowledge about relations and conditional probabilities, and Figure 7 is intended to explain the principles behind such networks. Note that this example has been developed for illustrative purposes and does not necessarily represent a network that would be useful in a real environment. For example, color is included as a variable because of its dependency, but observing color in a work facility might be considered impractical for a variety of reasons.

[0063] Some of the variables are independent and can only be directly observed. These variables have no parents and in the example shown in Figure 7 include season 701, temperature 702, and time of day 703. Temperature 702 may in fact be conditionally dependent on both season 701 and time of day 703, but the fact that temperature is directly observable and there is no need to estimate the probability of a particular season or time of day given the observed temperature, nor to estimate the probability of temperature given a known season or time of day, means that these dependencies do not have to be included in the model.

[0064] The probability that an observed bird is of a particular species 704 may depend on the season 701, temperature 702, and time of day 703. Thus, a conditional probability table of species 704 can be established. This is simplest when all variables are discrete (e.g., in the example of FIG. 7, they can only take on a limited number of discrete values, such as true and false, e.g. values ​​summer (winter), above freezing (below freezing), and day (night), or summer, autumn, winter, spring), but it should be noted that Bayesian belief networks can also be created for continuous variables. This is well known in the art and will not be described in further detail.

[0065] Several additional variables may depend on a given species 704. In the example of Figure 7, the variables color 705, flight altitude 706, flight pattern 707, and flapping frequency 708 are included. Furthermore, color 705 also depends on season. The dependency between color and season is not necessarily useful for offshore wind farms, but could potentially be useful in mountainous areas where some birds change plumage color over the season.

[0066] Based on statistics about common species of birds in the relevant area, their migration patterns, behavioral responses to various weather conditions, and knowledge about flight patterns, altitudes and flapping frequencies for various species, tables of conditional probabilities can be constructed. Which variables to include and what probabilities to give will likely be based on a combination of statistics and other knowledge that the ornithologist can provide. The same applies to whether variables are discrete, continuous but can be discretized (e.g., above or below a threshold), or should be continuous in the model.

[0067] The following table, included for illustrative purposes only, illustrates what a term sheet might look like: [Table 1] [Table 2]

[0068] This example contains only invented probabilities for illustrative purposes and does not represent any particular real environment. What the first table illustrates is that, given the various possible combinations of season, temperature, and time of day, the probability that the observed bird is either an eagle or a seagull may be as shown. The second table gives the probability that the observed bird is flying above or below an altitude of 200 meters, respectively, if it is a seagull or an eagle. Similar tables can be established for all variables. Based on the graph and the associated conditional probability tables, it becomes possible to establish the probability of the unobserved variables based on the observed variables. Note that such conditional probabilities can be established in both directions. They are not limited to establishing the probability of a child node given the observation of the parent variable. For example, in the above example, if the bird is observed flying above 200 meters, there is a much higher probability that the bird is an eagle and not a seagull than if the bird was observed below 200 meters. How likely it is depends on the probability that the observed bird is an eagle in the first place, which again depends on the other variables in the network.

[0069] Thus, given observations of as many variables as possible, the probabilities of unobserved variables can be modified accordingly. The updated probabilities given the observed variables are sometimes called posterior probabilities.

[0070] The Bayesian belief network can be programmed into the domain knowledge module 306 and automatically generate estimated probabilities based on observed variables received from the geometric feature extraction module 303 (e.g., altitude, flapping frequency, flight patterns) and the additional sensor module 302 (e.g., temperature, season, time of day). The domain knowledge module 306 may be implemented as a combination of software and hardware on a single computer, on several computers, or on a cloud service.

[0071] As previously mentioned, the exact variables or parameters included may depend on the conditions within the area in which a particular system is installed.

[0072] After the species detection, possibly combined with additional data such as confidence levels and some of the variables used by the domain knowledge module 306, has determined that a given observation is a bird belonging to a particular species, the output may be stored in the database 110. The aggregate data may be used to create statistics, and all of the current, historical, and statistical information may be made accessible to a user computer 112 that is connected to the network 114 and authorized to access the database 110.

[0073] The data can be used to plan future operations and to initiate suppression and curtailment, for example by initiating voice suppression or temporary shutdown of the wind farm. Such measures may be initiated manually by a person accessing the information available in the database, or the computer 112 can be configured to continuously monitor the data available in the database 110 or receive it directly from the system and automatically initiate suppression or curtailment measures based on predefined conditions.

Claims

1. 1. A method for determining the species of a bird in flight, comprising the steps of: Capturing (402) at least one video stream of a bird in flight using at least one camera (104); generating (405) a first species probability estimate by delivering images from the at least one video stream as input to an artificial neural network (304) trained to recognize bird species from images; Obtaining (401) additional parameters from said at least one video stream or from at least one additional data source; generating (407) a second seed probability estimate by delivering the obtained additional parameters as input to a domain knowledge module (306) having a domain knowledge statistical model; generating a final seed probability estimate by combining the first seed probability estimate and the second seed probability estimate (408); A method comprising:

2. Extracting (404) geometric features associated with the bird in flight by delivering images from the at least one video stream as input to a geometric feature extraction module (303); - generating (406) said first species probability estimate by delivering the extracted features from said artificial neural network (304) and the extracted geometric features from said geometric feature extraction module (303) as inputs to a shallow neural network (305) trained to generate a bird species probability based on the features extracted by said artificial neural network combined with the observed geometric features; - generating said second seed probability estimates by delivering the geometric features extracted from said geometric extraction module (303) as obtained additional parameters input to said domain knowledge statistical model; and performing at least one of The method of claim 1 further comprising:

3. 3. The method of claim 2, wherein the extracted geometric features are obtained based on identification of the same bird in an image sequence from the at least one video stream, and motion is estimated based on changes in position of the identified bird between images in the image sequence.

4. the at least one camera is two or more cameras, the at least one video stream is two or more video streams, the extracted geometric features include: a known position of each camera (104); identifying the same bird in two or more sequences of images from two or more simultaneous video streams; determining a position of the identified bird in each image of each video stream; and determining 3D coordinates representing a position of the identified bird relative to the position of the camera (104) from the determined position in each image of each video stream using multi-view geometric analysis; The method according to claim 2 or 3, wherein the determined 3D coordinates are used to extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory and flapping frequency.

5. 4. The method of claim 2 or 3, wherein one extracted geometric feature is a flapping frequency determined by performing a Fourier analysis on a sequence of images from the at least one video stream and identifying dominant frequency components that lie within a frequency interval that coincides with a bird's flapping frequency.

6. 4. The method of claim 1, further comprising training the artificial neural network by delivering a dataset comprising labelled images of relevant bird species as input to the artificial neural network.

7. 4. The method of claim 1, further comprising: performing object detection on images from the at least one video stream (403); and annotating the images with a bounding box drawn around each object identified as a bird.

8. The method of claim 7, wherein object detection is performed using a second artificial neural network (301).

9. The method of claim 1 , further comprising providing (409) as an output the seed having the highest determined final seed probability.

10. 10. The method of claim 9, further comprising using the output to control a deterrent or abatement measure to reduce a risk that the birds of the determined species will be injured by wind farm equipment.

11. 4. The method according to claim 1, wherein the domain knowledge statistical model is a Bayesian belief network and / or the one or more artificial neural networks are convolutional neural networks.

12. 1. A system for determining a species of bird in flight, comprising: At least one video camera (104); an artificial neural network (304) configured to receive video images from at least one of the cameras (104) and to be trained to recognize bird species from the images; a domain knowledge module (306) having a domain knowledge statistical model configured to receive observed values ​​of an additional parameter and generate a probability of observing each species of bird given the observed values ​​of the additional parameter; a species determination module (307) configured to receive a first species probability estimate based on output from the artificial neural network (304) and a second species probability estimate based on output from the domain knowledge module (306) and generate a final species probability estimate; A system comprising:

13. a geometric feature extraction module (303) configured to receive at least one video stream from the at least one camera (104) and extract geometric features related to birds captured in flight in the at least one video stream; a shallow neural network (305) configured to receive the extracted features from said artificial neural network (304) and the extracted geometric features from said geometric feature extraction module (303) and to generate said first seed probability estimates; - configuration of a domain knowledge module (306) enabling it to receive, as additional parameters, the geometric features extracted from said geometric feature extraction module (303); At least one of The system of claim 12 further comprising:

14. 14. The system of claim 13, wherein the geometric feature extraction module is configured to receive data regarding at least one video stream, extract geometric features from the at least one video stream based on identification of the same bird in an image sequence, and estimate movement based on changes in position of the identified bird between images in the image sequence.

15. the at least one camera is two or more cameras, the at least one video stream is two or more video streams, a multi-view geometric analysis module (302) configured to receive known positions of each camera (104), receive data relating to at least two simultaneous video streams, determine a position of an identified bird in each image of each video stream, determine a position of the identified bird in each image of each video stream, and determine 3D coordinates representing a position of the identified bird relative to the position of the camera (104) from the determined positions in each image of each video stream, using multi-view geometric analysis; The system of claim 13 or 14, wherein the geometric feature extraction module (303) is further configured to receive the determined 3D coordinates from the multi-view geometric analysis module (302) and extract features selected from the group consisting of position, velocity, acceleration, vertical motion, flight trajectory, and flapping frequency based on the received 3D coordinates.

16. 15. The system of claim 13 or 14, wherein the geometric feature extraction module (303) is further configured to determine a flapping frequency by performing a Fourier analysis on the sequence of images from the at least one video stream and identifying dominant frequency components that lie within a frequency interval that coincides with a bird's flapping frequency.

17. The system of any one of claims 12 to 14, further comprising a bird detection and tracking module (301) configured to receive input from at least one video camera (104), perform object detection, and annotate images by drawing a bounding box around each object identified as a bird.

18. 20. The system of claim 17, wherein the bird detection and tracking module (301) includes a second artificial neural network.

19. 15. The system of any one of claims 12 to 14, wherein the species determination module (307) is further configured to deliver the final species probability estimate as an output that is stored, displayed, or used to control a deterrence or curtailment process to reduce a risk of the determined species of bird being damaged by a wind farm installation.

20. 15. The system of claim 12, wherein the domain knowledge statistical model is a Bayesian belief network and / or the one or more artificial neural networks are convolutional neural networks.

Images