Estimation system and estimation method

Abstract

An estimation system according to the present invention comprises: sensors 5 that are installed on flying bodies 2–4 and capture images that include whales; and an estimation device 1. The estimation device 1 comprises: a feature extraction unit 13 that extracts feature quantities that include the spouting of the whales from the images captured by the sensors 5; and an estimation unit 15 that inputs the feature quantities into an identification model 17 for individual whale identification, uses identification results from the identification model 17 to calculate a number of individually identified whales, and estimates a total number of whales on the basis of the number of whales and the area covered by the images.

Description

Estimation system and estimation method

[0001] The present disclosure relates to an estimation system and an estimation method.

[0002] Whales play an important role in combating climate change, contributing to the carbon cycle and said to absorb 340 tons of CO2 per whale. Whales absorb carbon by consuming plankton and other organisms, and the feces they excrete contain nutrients (Fe, P, N, etc.) obtained from the seafloor, which promote the growth of plankton and the absorption of CO2 through photosynthesis (Non-Patent Documents 1, 2, 11).

[0003] Whale populations are generally estimated using visual surveys (Non-Patent Documents 3 and 4). Techniques for identifying individual whales using AI image processing have also been proposed (Non-Patent Documents 5, 6, and 7). Research into whale monitoring has also been conducted using acoustics and neural networks (Non-Patent Document 8) and biologging (Non-Patent Documents 9 and 10).

[0004] "Whales play key role in climate change countermeasures, storing carbon: US study", [online], Internet <https: / / www.cnn.co.jp / fringe / 35197517.html> "Environmental issue: 'Whales are the hope for decarbonization!' It turns out that a single whale removes the carbon equivalent of 'thousands of trees!'" Shocking”, [online], Internet <https: / / wild-scene.com / %E7%92%B0%E5%A2%83%E5%95%8F%E9%A1%8C / 17637 / > “Current status and issues of visual survey methods for cetaceans”, [online], Internet <https: / / www.jstage.jst.go.jp / article / mammalianscience / 44 / 1 / 44_1_97 / _pdf / -char / ja> “Methods for estimating the population of marine mammals – current status and future issues”, [online], Internet <https: / / www.jstage.jst.go.jp / article / mammalianscience / 58 / 1 / 58_83 / _pdf / -char / ja> “AI-based identification of individual whales based on tail fin patterns and jagged edges” Utilizing 10,000 Photos,” [online], Internet <https: / / www.asahi.com / articles / ASQ2L5T0YQ2LDIFI00C.html#:~:text=%E8%B2%A1%E5%9B%A3%E3%81%8C30%E5%B9%B4%E4%BB%A5%E4%B8%8A,%E3%81%99%E3%82%8B%E3%81%93%E3%81%A8%E3%8 1%AB%E6%88%90%E5%8A%9F%E3%81%97%E3%81%9F%E3%80%82>“Developing an AI automatic identification system utilizing expert knowledge, we succeeded in identifying thousands of humpback whales by the shape of their tail fins,” [online], Internet <https: / / resou.osaka-u.ac.jp / ja / research / 2022 / 20220204_2> “Challenging Kaggle: Identifying whale and dolphin individuals from images,” [online], Internet <https: / / www.nri-digital.jp / tech / 20230418-13269>“Acoustic Detection of Humpback Whales Using a Convolutional Neural Network,” [online], Internet <https: / / blog.research.google / 2018 / 10 / acoustic-detection-of-humpback-whales.html>, “Capturing the Surprising Lifestyle of Whales,” [online], Internet <https: / / www.u-tokyo.ac.jp / focus / ja / features / z1304_00133.html>, “The Use and Development of Bio-Logging in Marine Mammalogy,” [online], Internet <https: / / www.jstage.jst.go.jp / article / mammalianscience / 60 / 2 / 60_281 / _pdf / -char / ja>, “Nature Offers Solutions to Climate Change,” [online], Internet <https: / / www.imf.org / external / japanese / pubs / ft / fandd / 2019 / 12 / pdf / Chami.pdf>.

[0005] Sighting surveys estimate the number of whales sighted, but it is difficult to improve the accuracy of individual estimates using sightings. Biologging and acoustic sensing are aimed at tracking a small number of individuals and do not take into account the estimation of whale populations.

[0006] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology for estimating the total number of whales using images.

[0007] In order to achieve the above-mentioned object, one aspect of the present disclosure is an estimation system comprising a sensor mounted on an aircraft that captures images including whales, and an estimation device, wherein the estimation device comprises: a feature extraction unit that extracts features including whale spouts from the images captured by the sensor; and an estimation unit that inputs the features into an identification model for individual identification of whales, counts the number of individually identified whales using the identification results of the identification model, and estimates the total number of whales based on the number of whales and the area covered by the image.

[0008] One aspect of the present disclosure is an estimation method performed by an estimation device that estimates the number of whales, which extracts features including whale spouts from an image containing whales captured by a sensor mounted on an aircraft, inputs the features into an identification model for identifying individual whales, counts the number of individually identified whales using the identification results of the identification model, and estimates the total number of whales based on the number of whales and the area covered by the image.

[0009] According to the present disclosure, it is possible to provide a technique for estimating the total number of whales using images.

[0010] Fig. 1 is an overall configuration diagram showing an example of an estimation system according to this embodiment. Fig. 2 is a block diagram showing an example of the configuration of an estimation device according to this embodiment. Fig. 3 is a flowchart showing an example of a model generation process. Fig. 4 is a flowchart showing an example of an estimation process. Fig. 5 is an example of a hardware configuration.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] 1 is a diagram showing the overall configuration of an example of an estimation system according to this embodiment. In this embodiment, images (image data) of whales are collected using remote sensing technology, and an identification model (learning model) for identifying individual whales is generated using machine learning based on the collected data. The identification model is then used to estimate the total number of whales.

[0013] The illustrated estimation system comprises a sensor 5 mounted on each of aircraft 2 to 4, which captures images including whales, and an estimation device 1. The estimation device 1 collects sensor result images (optical images, SAR images, thermal infrared images, LiDAR images, camera images, etc.) acquired by the sensor 5, and generates an identification model using the sensor result images. The aircraft 2 to 4 fly over the sea, and the sensor 5 captures images of whales appearing on the surface of the water.

[0014] In this embodiment, remote sensing is performed to observe whales, which are marine organisms, by mounting sensors 5 (measuring instruments) on aircraft 2 to 4. The aircraft 2 to 4 include at least one of an artificial satellite 2, a High Altitude Platform Station (HAPS) 3, and a drone 4 (UAV: Unmanned Aerial Vehicle).

[0015] The sensors 5 mounted on the flying bodies 2 to 4 include, for example, visible / reflected infrared sensors, SAR (Synthetic Aperture Radar) sensors, and thermal infrared sensors.

[0016] Visible and reflected infrared sensors capture high-resolution optical images (optical imaging). These optical images are useful for directly observing whale spouting. Although optical images cannot be taken when there are clouds overhead, they can capture the ocean surface with a resolution of several meters, making it possible to observe whale spouting and the whale itself.

[0017] SAR sensors overcome the limitations of optical imaging because they can penetrate clouds and darkness to capture SAR images. SAR sensors can indirectly indicate the presence of waterspouts by detecting changes in the roughness of the ocean surface.

[0018] Thermal infrared sensors use the temperature difference between the sea surface and the air to capture sensor images that can detect whale blows. Because whale blows are warmer than the surrounding seawater, this temperature difference can be used to identify them.

[0019] The HAPS 3 and the drone 4 may be equipped with sensors 5, such as a LiDAR sensor or a camera. A LiDAR sensor, such as an airborne LiDAR, can directly detect whale spouts with high spatial resolution. LiDAR sensors use light to measure the distance and shape of objects, allowing for detailed mapping of the height and distribution of spouts.

[0020] The images, such as photographs or videos, taken by the camera are very effective for direct observation of whale spouting in a specific area and the whales themselves. The camera mounted on the drone 4 provides high-resolution images from a low altitude, allowing detailed observation of whale behavior.

[0021] In this way, in this embodiment, by adopting a composite REET sensing technology, it is possible to detect and identify individual whales in remote locations that were previously difficult to access, using aircraft such as the satellite 2, HAPS 3, and drone 4. In other words, this embodiment improves the accuracy of whale detection, enables the identification of individual whales around the world, and makes it possible to estimate the number of identified whales and track (monitor) them.

[0022] 2 is a block diagram showing an example of the configuration of the estimation device 1. The estimation device 1 of this embodiment includes an input unit 11, a preprocessing unit 12, a feature extraction unit 13, a learning unit 14, an estimation unit 15, an observation unit 16, and an identification model 17.

[0023] The input unit 11 collects images (training images) containing whales captured by the various sensors 5. Specifically, the input unit 11 inputs a plurality of images containing whales in the ocean captured by the various sensors 5 as training data, and stores the images in a storage unit (not shown). Note that the images also include moving images (videos) consisting of a plurality of frames. It is assumed that the images contain at least one whale.

[0024] The preprocessing unit 12 may perform preprocessing on each image (each frame) collected by the input unit 11 to facilitate feature extraction. Specifically, the preprocessing unit 12 uses image analysis technology to perform preprocessing such as noise removal to reduce image noise caused by sea conditions and lighting conditions, contrast enhancement to enhance image contrast to make features such as a whale's spouting more clear, and background subtraction to remove the background to highlight the target whale's characteristic features (such as a whale's spouting, the shape of its dorsal fin, scars, spot patterns, and color patterns on its body surface). Note that the preprocessing may be omitted.

[0025] The feature extraction unit 13 extracts features including whale spouts from images including whales captured by the sensor 5. For example, the feature extraction unit 13 performs feature extraction, edge detection, and the like on each preprocessed image. Specifically, the feature extraction unit 13 uses image analysis technology to extract training features of whale characteristics (such as whale spouts, dorsal fin shape, scars on the body surface, spot patterns, and color patterns) used for individual whale identification from the image for each individual whale. The feature extraction unit 13 may also use edge detection technology to detect edges such as the shape of whale spouts and the outline of the whale as features.

[0026] The feature extraction unit 13 extracts features (training features) from the training images collected by the input unit 11, and also extracts features from images (estimation images) used by the estimation unit 15 to estimate the number of whales.

[0027] The learning unit 14 performs machine learning using the features (training features) as training data to generate an identification model 17 for identifying individual whales. In this embodiment, a convolutional neural network (CNN), which is a type of deep learning, is used for the identification model 17, but is not limited to this. CNN uses convolution and pooling. CNN is a neural network with multiple layers that can automatically learn features from input features and generate an identification model capable of identifying individual whales.

[0028] Individual whales can be identified using their features (for example, the shape of their spouts, the shape and scars on their dorsal fins, the shape and notches of their tail fins, the patterns and color of their bodies, scars and parasite marks on their skin, facial features, etc.). Therefore, by extracting these features and performing machine learning, it is possible to build an identification model that can identify individual whales.

[0029] The learning unit 14 may use a CNN previously trained on another task and perform transfer learning to enable efficient learning even with a small amount of training data (training features). The learning unit 14 may perform cross-validation to evaluate the versatility and accuracy of the discriminative model 17. The learning unit 14 may perform optimization of the discriminative model 17, such as tuning hyperparameters or introducing additional layers, to improve the performance of the discriminative model 17.

[0030] By using the image analysis techniques described above, it is possible to generate an identification model 17 that can identify individual whales from images that include whales.

[0031] The estimation unit 15 estimates the total number of whales using the identification model 17 generated by the learning unit 14. That is, the estimation unit 15 inputs the feature amounts extracted by the feature extraction unit 13 from the image (image for estimation) captured by the sensor 5 into the identification model 17 for identifying individual whales, counts the number of whales individually identified using the identification result of the identification model 17, and estimates the total number of whales based on the number of whales and the area covered by the image.

[0032] For example, the estimation unit 15 inputs multiple estimation images captured by a sensor 5 mounted on an aircraft such as a satellite 2 and covering a wide range of ocean areas into the identification model 17. The multiple estimation images are assumed to be captured at different times (time periods) and cover the same ocean area. The estimation unit 15 then counts the number of whales identified by the identification model 17 for each estimation image. The estimation unit 15 then eliminates whales that are individually identified in multiple estimation images, thereby counting the number of whales present in the ocean area covered by the multiple estimation images. The estimation unit 15 then estimates the total number of whales based on the counted number of whales and the area (range) covered by the estimation images. Examples of methods that can be used to estimate the total number include Markov Chain Monte Carlo (MCMC), other estimation methods, and statistics.

[0033] The estimation unit 15 also inputs a first estimation image captured by a sensor 5 mounted on a first aircraft and covering a wide ocean area to the identification model 17, and counts the number of whales in the first estimation image identified by the identification model 17. Similarly, the estimation unit 15 inputs a second estimation image captured by a sensor 5 mounted on a second aircraft and covering a wide ocean area different from that of the first aircraft to the identification model 17. The estimation unit 15 then counts the number of whales in the second estimation image identified by the identification model 17. Since individual whales are identified by the identification model 17, the estimation unit 15 counts the number of whales in the first estimation image and the second estimation image by, for example, eliminating duplicate whales. The estimation unit 15 then estimates the total number of whales based on the counted number of whales and the area (range) covered by the first estimation image and the second estimation image. The total number can be estimated using the aforementioned Markov Chain Monte Carlo (MCMC) method.

[0034] The observation unit 16 inputs images captured by the sensor 5 over a long period of time into the identification model 17 to identify individual whales, thereby monitoring the behavioral patterns of each individually identified whale.

[0035] FIG. 3 is a flowchart showing an example of the model generation process of the estimation device 1. The input unit 11 of the estimation device 1 collects images (training images) containing whales captured by the various sensors 5 mounted on the aircraft 2-4 (S11). The preprocessing unit 12 performs preprocessing, such as noise removal, contrast enhancement, and background subtraction, on each image collected by the input unit 11 (S12). Note that preprocessing may be omitted. The feature extraction unit 13 extracts whale features (training features), including spouting, from each preprocessed image (S13). The learning unit 14 performs deep learning such as CNN using the features of each image as training data to generate an identification model 17 for identifying individual whales (S14), and then evaluates and optimizes the generated identification model 17 (S15). In this manner, the identification model 17 is constructed.

[0036] 4 is a flowchart showing an example of the process for estimating the total number of whales performed by the estimation device 1. The input unit 11 of the estimation device 1 inputs at least one estimation image including a whale captured by the sensor 5 (S21). The preprocessing unit 12 performs preprocessing such as noise removal, contrast enhancement, and background subtraction on each image input by the input unit 11 (S22). Note that preprocessing may be omitted. The feature extraction unit 13 extracts whale feature amounts (estimation feature amounts) including spouting from each preprocessed image (S23).

[0037] The estimation unit 15 inputs the feature amounts into the identification model 17 for identifying individual whales, and counts the number of whales individually identified for each image using the identification results of the identification model 17 (S24).The estimation unit 15 then eliminates duplicates of whales identified for each image, counts the number of whales in the images input in S21, and estimates the total number of whales based on the number of whales and the area covered by the images input in S21 (S25).

[0038] The estimation system of this embodiment described above comprises a sensor 5 mounted on each of the aircraft 2 to 4 for capturing images including whales, and an estimation device 1. The estimation device 1 comprises a feature extraction unit 13 that extracts features including whale spouts from the images captured by the sensor 5, and an estimation unit 15 that inputs the features into an identification model 17 for identifying individual whales, counts the number of individually identified whales using the identification results of the identification model 17, and estimates the total number of whales based on the number of whales and the area covered by the image.

[0039] Furthermore, the estimation method performed by estimation device 1 for estimating the number of whales in this embodiment involves extracting features including whale spouts from images containing whales captured by sensors 5 mounted on aircraft 2 to 4, inputting these features into identification model 17 for identifying individual whales, counting the number of individually identified whales using the identification results of the identification model, and estimating the total number of whales based on the number of whales and the area covered by the image.

[0040] In this way, this embodiment focuses on the spouting behavior of whales, captures whales that come to the surface to spout, and adopts a combined remote sensing approach, which makes it possible to identify individual whales in remote locations that were previously difficult to access, thereby improving the accuracy of identifying individual whales and improving the accuracy of estimating the total number of whales around the world.

[0041] In this embodiment, by using multiple remote sensing technologies such as artificial satellites 2, HAPS 3, and drones 4, as well as image analysis technology, it is possible to identify individual whales and grasp the total number of whales around the world.

[0042] In addition, in this embodiment, by generating an identification model 17 for identifying individual whales, it becomes possible to identify and track individual whales, and to understand the behavioral patterns of whales.

[0043] The above-described estimation device 1 can use, for example, a general-purpose computer system as shown in Fig. 5. The illustrated computer system includes a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. The memory 902 and the storage 903 are storage devices. In this computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of the estimation device 1.

[0044] The estimation device 1 may be implemented on one computer or on multiple computers. The estimation device 1 may also be a virtual machine implemented on a computer. The program for the estimation device 1 may be stored on a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or may be distributed via a network. The computer-readable recording medium may be, for example, a non-transitory recording medium.

[0045] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

[0046] 1: Estimation device 11: Input unit 12: Preprocessing unit 13: Feature extraction unit 14: Learning unit 15: Estimation unit 16: Observation unit 17: Discrimination model 2: Satellite 3: HAPS 4: Drone 5: Sensor

Claims

1. An estimation system comprising: a sensor mounted on an aircraft that captures images including whales; and an estimation device, wherein the estimation device comprises: a feature extraction unit that extracts features including whale spouts from the images captured by the sensor; and an estimation unit that inputs the features into an identification model for identifying individual whales, counts the number of individually identified whales using the identification results of the identification model, and estimates the total number of whales based on the number of whales and the area covered by the images.

2. The estimation system according to claim 1, wherein the feature extraction unit extracts training features including whale spouts from training images including whales captured by the sensor, and the estimation device includes a learning unit that performs machine learning using the training features as training data to generate the identification model for identifying individual whales.

3. The estimation system according to claim 1, wherein the air vehicle includes at least one of a satellite, a HAPS, and a drone.

4. An estimation method performed by an estimation device that estimates the number of whales, which extracts features including whale spouts from images of whales captured by a sensor mounted on an aircraft, inputs the features into an identification model for identifying individual whales, counts the number of individually identified whales using the identification results of the identification model, and estimates the total number of whales based on the number of whales and the area covered by the image.

Images