Environment change device, environment detection device, and environment change detection method

Abstract

To provide an environment change device, an environment detection device, and an environment change detection method that accurately detect spatiotemporal phenomena occurring in the regional climate and environment.SOLUTION: The environment change detection method includes the steps of: acquiring multiple pieces of captured image data containing coordinate data and time data; setting a grid at predetermined intervals on a map of an object area; associating the captured image data with the coordinates on the map based on the coordinates indicated by the coordinate data; calculating the probability that a predetermined feature is included within an image of the captured image data; dividing the multiple pieces of captured image data associated with the coordinates within the grid into predetermined time units based on their respective time data, and calculating the average value of the calculated probability for the captured image data divided by the time unit; and creating a graph for each grid showing a change in the average probability value for the time indicated by the time data; thereby detecting or estimating changes in events from the graph.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a device for detecting or estimating changes in an environment. [Background technology]

[0002] Floral phenology, or the timing of flowering, is recognized by the scientific community as a valuable indicator of climate change (Non-Patent Document 1). However, estimating spatial flowering events is challenging because flowering timing is controlled by regional and temporal environmental variables such as temperature and precipitation.

[0003] To date, fixed-point observations are the most common method used to record floral phenology. The Japan Meteorological Agency has been conducting fixed-point observations of plants and animals every year since 1953. However, data collection is no longer being carried out at 40% of the original observation sites, and 94% of phenological events are no longer observed.

[0004] To address the lack of spatial observation sites, non-expert observations are useful for estimating phenological events, but non-expert observations have been reported to be inaccurate in plant species identification and reported locations.

[0005] Non-Patent Document 1 describes the study of flower phenology using satellite-based remote sensing technology. Specifically, Non-Patent Document 1 discloses a technology for monitoring the floral activity of almond orchards based on time-series multispectral remote sensing images.

[0006] Meanwhile, Non-Patent Document 2 discloses a deep learning model that accurately classifies various plant species and generates a species occurrence map from a dataset of over 23,000 social sensing photos.

[0007] Non-Patent Document 3 discloses a technology that uses social sensing photos with geotags and text tags to extract cherry-related images from the attached labels.

[0008] Non-Patent Document 4 discloses a participatory sensing system that automatically extracts cherry blossoms from videos taken with an in-vehicle smartphone and shares them among users in near real time. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] B. Chen et al., "An enhanced bloom index for quantifying floral phenology using multi-scale remote sensing observations", ISPRS P & RS (2019) [Non-patent document 2] DJ Dixon et al., "Satellite prediction of forest flowering phenology", Remote Sens. Environ. (2021) [Non-patent document 3] MM ElQadi et al. "The spatiotemporal signature of cherry blossom flowering across Japan revealed via analysis of social network site images", Flora (2023) [Non-patent document 4] Morishita, S., et al. “SakuraSensor: quasi-realtime cherry-lined roads detection through participatory video sensing by cars”, in: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp '15. (2015) Summary of the Invention [Problem to be solved by the invention]

[0010] To record flower phenology, multiple data sources have been proposed, including fixed-point observations as well as remote sensing, social media data, and street-level imagery. However, these techniques have not been able to accurately detect cherry blossom phenology at the local level.

[0011] An object of the present invention is to accurately detect or estimate spatiotemporal phenomena occurring in the climate, environment, etc. of a region. [Means for solving the problem]

[0012] To achieve the above object, the present invention provides an environmental change device having an environmental detection device and an environmental change analysis unit. The environmental detection device includes an image acquisition unit that acquires multiple captured image data including coordinate data and time data, a grid setting unit that sets a grid at predetermined intervals on a map of a desired target range, a coordinate processing unit that associates the captured image data with coordinates on the map using coordinates indicated by the coordinate data, a probability calculation unit that calculates a probability that a predetermined feature is included in an image of the captured image data based on the captured image data, and an average calculation unit that calculates, for each time, an average value of the probabilities calculated by the probability calculation unit for multiple captured image data associated with coordinates within the same area divided by the grid. The environmental change analysis unit includes a graph creation unit that creates a graph showing changes in the average value of the probability with respect to time indicated by the time data for each area divided by the grid.

[0013] According to the present invention, spatiotemporal phenomena occurring in the climate, environment, etc. of a region can be detected or estimated with high accuracy using image data. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram showing the configuration of an environment change device 3 according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing in detail the configuration of the environment change device 3 according to the embodiment. [Figure 3] 2 is an explanatory diagram showing a grid 21 set in a detection target range 20 by the environment detection device 1 of the embodiment, and captured image data associated with the divided areas by coordinate data. FIG. [Figure 4] FIG. 2 is a diagram showing a learning model 31 used as a probability calculation unit 14 of the environment detection device 1 according to the embodiment. [Figure 5] 10 is a graph showing the probability of cherry blossoms in bloom being included in an image for each date, created by the environment detection device 1 of the embodiment. [Figure 6] 4 is a flowchart showing the operation of the environment change device 3 according to the embodiment. [Figure 7] 4 is a flowchart showing the operation of the environment change device 3 of the embodiment. [Figure 8] 6 is a histogram showing the number of images of image data used by the environment detection device 1 of the embodiment for each shooting date. [Figure 9] 10 is a map showing the average value of the probability for each area detected by the environment detection device 1 of the embodiment. [Figure 10] 10 is a map showing the flowering start date, full bloom date, and flower fall date for each region, as determined by the environment change device 3 of the embodiment. [Figure 11] 10 is a histogram of the number of true samples and the probability of detecting false samples in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described.

[0016] The environment change device 3 of this embodiment detects environmental changes over time using image data captured from street photographs, etc. Furthermore, it estimates the occurrence of a predetermined event based on the detected environmental changes. <Summary> First, an overview of the environment detection device of this embodiment will be described with reference to Figures 1 to 6. Figures 1 and 2 are diagrams showing the overall configuration of an environment change device 3 of this embodiment. Figure 3 is a diagram showing a desired detection target range 20 in which it is desired to detect an environmental change.

[0017] As shown in FIG. 1 , the environmental change device 3 of this embodiment includes an environment detection device 1 and an environmental change analysis unit 2. The environment detection device 1 sets a grid at a predetermined interval on a map of a target range where environmental changes are to be detected or estimated, and calculates the probability that a predetermined event (characteristic) is included in one or more pieces of image data captured within the area partitioned by the grid. The environmental change analysis unit 2 estimates the occurrence of a predetermined event based on the environmental changes detected by the environment detection device 1. Specifically, the environmental change analysis unit 2 creates a graph showing the change over time in the probability of the predetermined event detected by the environment detection device 1, and detects or estimates the occurrence of the predetermined event based on the shape of the graph.

[0018] 2, the environment detection device 1 includes an image acquisition unit 11, a grid setting unit 12, a coordinate processing unit 13, a probability calculation unit 14, and an average calculation unit 15. The environment change analysis unit 2 includes a graph creation unit 16.

[0019] The image acquisition unit 11 acquires multiple captured image data of a point within the desired detection target range 20 where environmental changes are to be detected. The captured image data includes an image of the captured point, coordinate data of the captured point (position data (x, y), where x = longitude, y = latitude), and time data indicating the time of capture.

[0020] The captured image data may be an image captured by receiving visible light (RGB image), or an image captured by receiving light of a specific wavelength (for example, a thermal infrared camera image).

[0021] The photographer of the captured image data may be anyone, and the image may be taken by a specific person or an unspecified person. The captured image data may also be taken by a camera installed on the street, such as a surveillance camera, or an image may be taken by a camera installed in a vehicle, such as a drive recorder.

[0022] The image acquisition unit 11 acquires captured image data whose coordinate data is included within the detection target range. Any acquisition method may be used, and captured image data taken by a specific person may be imported via a storage medium or a network. When capturing image data via a network, the image acquisition unit 11 may also acquire the captured image data in real time. As another acquisition method, the image acquisition unit 11 may download captured image data whose coordinate data is included within the desired target range from cloud storage where photographs (captured image data) taken on the street by an unspecified number of photographers are uploaded.

[0023] As shown in FIG. 3, the grid setting unit 12 sets grids 21 at predetermined intervals on a map of the detection target range 20. The areas separated by the grids 21 become units for detecting environmental changes, as will be described later. The intervals between the grids 21 may be a specific interval that has been determined in advance, or may be a desired interval set by the user. As an example, the intervals between the grids 21 may be set to 10 m.

[0024] It is preferable to set the interval of the grid 21 to a value larger than the position error of the coordinate data attached to the captured image data in order to increase reliability.

[0025] The coordinate processing unit 13 associates the captured image data with coordinates on the map using the coordinates indicated by the coordinate data, as shown in Fig. 3. In this way, the coordinate processing unit 13 identifies which captured image data is included in which area divided into the grid 21.

[0026] The probability calculation unit 14 calculates the probability that a predetermined feature is included in the image of the captured image data, based on the image of the captured image data.

[0027] The features referred to here include, for example, the following (1) to (6), but are not limited to these, and may be any phenomenon that can be grasped by a change in shape or color that can be grasped from an image. (1) The blooming of designated flowers (2) Detection of disasters caused by rainfall, flooding, and fires (3)Identification of disaster damage caused by collapsed houses and structures (4) Land cover changes due to farm abandonment, deforestation, and land development (5) Changes in landscape (including deterioration of public safety and, conversely, beautification of the landscape through urban planning) (6) Changes in the thermal environment, such as heat islands (possible using thermal infrared camera images)

[0028] For example, in the case of (1) above, the probability calculation unit 14 calculates the probability that an image of "a predetermined flower in bloom" is included in the image. The closer the probability calculated by the probability calculation unit 14 is to 100%, the higher the probability that an image of a predetermined flower in bloom is included in the image. Also, in the case of calculating "flooding" as a feature in (2) above, the closer the probability calculated by the probability calculation unit 14 is to 100%, the higher the probability that an image of the ground surface being flooded is included in the image.

[0029] Any method for calculating the probability may be used by the probability calculation unit 14. For example, as shown in FIG. 4, a learning model 31 may be used as the probability calculation unit 14. This learning model is a trained model that is trained using captured image data as input data and features in the image of the captured image data that have been obtained in advance by another method as ground truth data. Therefore, when the captured image data acquired by the image acquisition unit 11 is input as input data to the probability calculation unit 14, the learning model 31 outputs the probability that the feature is included in the image of the captured image data.

[0030] The average calculation unit 15 divides multiple pieces of captured image data associated with coordinates within the same area divided by the grid into predetermined time units based on the time data, and calculates the average value of the probabilities calculated for the image data captured in the same area at the same time by the probability calculation unit 14. The predetermined time unit is set in advance depending on the feature (event) to be detected, such as one day, one hour, or ten minutes.

[0031] The graph creation unit 16 of the environmental change analysis unit 2 creates a graph showing the change in the average value of the probability with respect to the time indicated by the time data for each area separated by the grid 21. For example, in one area 20-A separated by the grid 21 in the detection target range 20, the change from day to day in the probability that an image showing cherry blossoms in bloom will be included is as shown in FIG.

[0032] The shape of this graph contains various information such as the rate of increase (slope) of the probability of the occurrence of the feature (event), the peak position, the rate of decline after reaching the peak, etc. Therefore, using this graph, it is possible to estimate the time (date) when the feature (event) occurred, the time (date) when the peak occurred, or the time (date) when the feature (event) will end.

[0033] Next, the operation of each part of the environment change device 3 will be explained using the flow chart of FIG.

[0034] The functions of each unit (11-16) of the environment detection device 1 and the environment change analysis unit 2 of the environment change device 3 can be realized by software. In this case, the environment detection device 1 is configured by a computer or the like equipped with a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) and memory, and the CPU reads and executes programs stored in the memory to realize the functions of each unit (11-16). It is also possible to configure part or all of the environment change device 3 by hardware. For example, a circuit can be designed to realize the functions of each unit using a custom IC such as an ASIC (Application Specific Integrated Circuit) or a programmable IC such as an FPGA (Field-Programmable Gate Array).

[0035] <Step S101> The image acquisition section 11 of the environment detection device 1 constituting the environment change device 3 acquires a plurality of captured image data obtained by photographing any point within the detection target range 20 .

[0036] <Step S102> The grid setting unit 12 sets grids 21 at predetermined intervals on a map of the detection target range 20 (see FIG. 3).

[0037] <Step S103> The coordinate processing unit 13 associates the captured image data with coordinates on a map using the coordinates indicated by the coordinate data (see FIG. 3).

[0038] <Step S104> The probability calculation unit 14 calculates the probability that a predetermined feature is included in the image of the captured image data, based on the image of the captured image data.

[0039] <Step S105> The average calculation unit 15 calculates the average value of the probability values ​​obtained for each of a plurality of pieces of captured image data associated with the same area divided by a grid, for each predetermined unit of time based on the time data of the captured image data.

[0040] <Step S106> The graph creation unit 16 of the environmental change analysis unit 2 that constitutes the environmental change device 3 creates a graph showing the change in the average value of the probability with respect to the time (unit: predetermined time) indicated by the time data for each area separated by the grid 21.

[0041] The environmental change analysis unit 2 detects or estimates the occurrence of a predetermined event based on the shape of the graph. [Example]

[0042] The present invention will be explained in more detail below using examples. In the embodiment, an environment change device 3 that detects cherry blossom blooming using captured image data of street photographs will be described.

[0043] The environment detection device 1 of the environment change device 3 of the embodiment acquires captured image data with coordinate data (hereinafter also referred to as geotags) and analyzes it using a deep learning model. The environment change analysis unit 2 detects the blooming of Somei-Yoshino cherry trees, as they are an important indicator of climate change.

[0044] The configuration of the environment detection device 1 is the same as that described in the embodiment, and therefore a description thereof will be omitted.

[0045] Fig. 7 is a flowchart showing the operation of each part of the environment change device 3 of the embodiment. In Fig. 7, steps corresponding to those in Fig. 6 are given the same reference numerals, and their explanation will be omitted.

[0046] In this embodiment, the image acquisition unit 11 acquires captured image data via a network from cloud storage 71 of a service (here, Mapillary by Meta Inc.) that allows geotagged captured image data (photo data) to be shared among an unspecified number of photographers.

[0047] In this way, the image acquisition unit 11 acquires photographic data as captured image data from the storage of a service that shares photographic data among a large number of photographers, thereby making it possible to acquire more captured image data than when acquiring photographic data taken by a specific one or more people as captured image data.

[0048] The detection target range 20 in which the image acquisition unit 11 detects the blooming of Somei-Yoshino cherry trees was set to the main campus of Saitama University in Japan.

[0049] In this example, the inventor took a total of 7,168 photos (captured image data) at multiple points in the detection target range 20 during the period from March 15 to April 10, 2022, excluding rainy days, and uploaded them to Mapillary. The image acquisition unit 11 acquired these captured image data from Mapillary.

[0050] The shooting dates of the 7,168 pieces of image data taken by the inventor were distributed as shown in Figure 8. Each piece of image data included coordinate information (specifically, latitude and longitude), the angle at which the image was taken, and the shooting date.

[0051] The grid setting unit 12 divided the detection target range 20 into two-dimensional grids 21 with 10 m intervals.

[0052] The probability calculation unit 14 used the YOLOv4 model as the learning model 31. The learning model 31 calculated the probability that each piece of photo data contained a blooming Somei-Yoshino cherry tree.

[0053] The YOLOv4 model was used to identify cherry blossoms from images of photographic data. YOLOv4 is a CNN-based object detection model that estimates and draws bounding boxes around target classes and their detection probabilities. YOLOv4 was chosen as the training model31 because it can run on a single standard graphics processing unit and can be trained using large datasets, as well as perform post-training detection.

[0054] To train the YOLOv4 model, 698 openly licensed images of Somei-Yoshino cherry blossoms in bloom were obtained from web-based repositories (e.g., Flickr (https: / / www.flickr.com / )), annotated, and used to train the YOLOv4 model.

[0055] The coordinate processing unit 13 associates the photograph data containing the coordinate data with the areas divided by the grid 21.

[0056] The average calculation unit 15 calculated the probability calculated by the probability calculation unit 14 for each area divided by the grid 21 and for each day of the time data of the photograph data.

[0057] In this way, by averaging the probability for each area divided by grid 21, the error in the distance between the observation point and the position of the cherry tree was reduced.

[0058] The graph creation unit 16 created a graph showing the change in the average probability for each region and each day, as shown in FIG.

[0059] Furthermore, the graph creation unit 16 created a map, as shown in FIG. 9, in which the average value of the probability of each area is represented in gray scale for each area and each day.

[0060] The graph creation unit 16 of the environmental change analysis unit 2 used a threshold approach to determine the first and last days on which the average probability exceeded 0.25 from the change in the average probability of the graph in Figure 5 as the flowering date and the flower abscission date. The peak of the graph was determined to be the peak of flowering. A map was created, as shown in Figure 10, in which the average values ​​of the probabilities of each region for the days of flowering, full bloom, and flower abscission are displayed in grayscale.

[0061] Analysis of such graphs provided reliable information regarding the blooming of cherry blossoms during the example period, confirming the effectiveness of the method of the example.

[0062] The daily probability map shown in Figure 9 was useful for recording spatiotemporal flowering events. According to the map in Figure 9, Somei-Yoshino cherry trees flowered around March 21, 2022, with peak flowering observed around March 31, 2022. Flower abscission occurred around April 9, 2022.

[0063] We observed slight variation in flowering timing among areas with different spatial locations, which we attribute to local environmental factors such as soil composition and photoperiod.

[0064] Furthermore, as shown in Figure 9, most of the observation points were limited to sidewalks and main roads that were easily accessible to photographers, and the observation frequency was unevenly distributed among the observation points that were set up.

[0065] The flowering, full bloom, and flower abscission dates could be estimated in a limited number of areas, as shown in the map in Figure 10. This estimation was possible in areas where a sufficient number of cherry blossom photographs had been taken.

[0066] Figure 10 shows that the dates of flowering, full bloom, and flower shedding in the study area varied by region. This is due to the variability in the flowering period of the trees. Flowering dates were between March 24th and 26th, full bloom dates were between March 29th and April 3rd, and flower shedding dates were between April 5th and 6th.

[0067] In the case of the area shown in the graph in Figure 5, a total of 66 photos were associated with the area. The first time the average probability exceeded 0.25 was on March 21, 2022, which was the day the flowers bloomed. After that, the peak (full bloom) was confirmed from March 28 to 31, 2022. On April 7, 2022, the average probability dropped to 0.25, and the flowers were observed falling as the leaves grew, confirming flower abscission.

[0068] Two different types of evaluation were used to verify the effectiveness of the methodology. First, test images at the street level that were not used for model training were prepared. The test images were sampled to remove biases arising from autocorrelation. A total of 568 photos were selected and analyzed, and the accuracy of cherry blossom detection was evaluated by visual interpretation.

[0069] Figure 11 shows a histogram of the detection probability for true samples (images with cherry blossoms in bloom) and false samples (images with cherry blossoms not in bloom). A threshold probability value of 0.25 was deemed appropriate for determining the accuracy of cherry blossom detection. Nevertheless, some true samples were identified with low probability scores below this threshold, suggesting the presence of false positives. On the other hand, visualization of the distribution of false samples shows that almost all samples (over 95%) were assigned a probability below 0.25, indicating good discrimination.

[0070] Therefore, it was found that the detection of cherry blossom blooming can be performed well by using a value of 0.25 as the detection probability threshold. The results of this evaluation, as shown in Table 1, showed an overall accuracy of 86.7%, a recall of 70.3%, and a precision of 90.1%. [Table 1]

[0071] As described above, the environment change device 3 of this embodiment can easily detect events such as flowering, full bloom, and fallen flowers using geotagged street-level images across both time and space, without the need to manually input data such as the flowering degree, location, and date information.

[0072] The environmental change device 3 of this embodiment does not require experts in ecology, remote sensing, or the like, and can detect events using photographs (captured image data) taken by a wide range of volunteers, including car drivers, hikers, cyclists, and pedestrians. The volunteers do not need to be identified; they simply take geotagged street-level photographs and upload them to a storage 71 such as Mapillary for sharing photos.

[0073] The learning model 31 used as the probability calculation unit 14 in the embodiment can be replaced with a different object detection model. In recent years, various object detection models based on deep learning have become known, and these object detection models can be used as the probability calculation unit 14.

[0074] The effectiveness of the method of this embodiment depends heavily on the detection accuracy of the learning model 31 employed. When the learning model 31 detects cherry blossom blooms in street-level image data, an overall accuracy of 86.7% was achieved. However, the recall rate was relatively low at 70.3%, which may result in insufficient detection accuracy of cherry blossoms in the images. In this case, detection accuracy can be stabilized by downloading a series of chronologically captured image data taken in time-lapse mode from the storage 71 and using it for detection. [Explanation of symbols]

[0075] 1. Environmental detection device 2. Environmental Change Analysis Department 3. Environmental change device 11 Image acquisition unit 12 Grid setting section 13 Coordinate processing section 14 Probability calculation section 15 Average calculation section 16 Graph Creation Section 20 Detection range 21 Grid 31 Learning Model 71 Cloud Storage

Claims

1. An environment detection device and an environment change analysis unit, The environment detection device is an image acquisition unit that acquires a plurality of captured image data including coordinate data and time data; a grid setting unit that sets grids at predetermined intervals on a map of a desired target range; a coordinate processing unit that associates the captured image data with coordinates on the map using the coordinates indicated by the coordinate data; a probability calculation unit that calculates a probability that a predetermined feature is included in an image of the captured image data based on the captured image data; an average calculation unit that calculates, for each time, an average value of the probabilities calculated by the probability calculation unit for a plurality of captured image data associated with coordinates within the same area divided by the grid, The environmental change analysis unit includes a graph creation unit that creates a graph showing a change in the average value of the probability with respect to the time indicated by the time data for each area divided by the grid. An environment change device characterized by:

2. 2. The environmental change device according to claim 1, wherein the probability calculation unit is a learning model that is pre-trained using the captured image data as input data and the probability that the feature is included in the image of the captured image data as correct answer data.

3. 2. The environmental change device according to claim 1, wherein the feature is one or more of an image of a predetermined flower in bloom, an image indicating rainfall or flooding, an image indicating the occurrence of a fire or the collapse of a house, an image indicating the occurrence of deforestation, an image indicating land use, and an image indicating a predetermined security situation.

4. The computer acquiring a plurality of pieces of captured image data including captured image data, coordinate data, and time data; establishing a grid of predetermined intervals on a map of the desired area of ​​interest; a step of associating the captured image data with coordinates on the map using the coordinates indicated by the coordinate data; calculating a probability that a predetermined feature is included in an image of the captured image data based on the captured image data; a step of dividing a plurality of pieces of captured image data associated with coordinates within the same area divided by the grid into predetermined time units based on time data of the plurality of pieces of captured image data, and calculating an average value of the calculated probabilities for the captured image data for each divided time unit; creating a graph showing the change in the average value of the probability with respect to the time indicated by the time data for each area divided by the grid; Detecting or estimating changes in said characteristics from said graph. An environmental change detection method comprising:

5. an image acquisition unit that acquires a plurality of captured image data including coordinate data and time data; a grid setting unit that sets grids at predetermined intervals on a map of a desired target range; a coordinate processing unit that associates the captured image data with coordinates on the map using the coordinates indicated by the coordinate data; a probability calculation unit that calculates a probability that a predetermined feature is included in an image of the captured image data based on the captured image data; an average calculation unit that calculates, for each time, an average value of the probabilities calculated by the probability calculation unit for a plurality of pieces of captured image data associated with coordinates within the same area divided by the grid. An environment detection device characterized by:

Images