Image generation model generating device and image estimation system

Abstract

An image generation model generating device and an image estimation system are provided that are capable of obtaining data relating to an observation target that is difficult to obtain. [Solution] The image generation model 12 of the image estimation system 20 accepts inputs of a pair including first satellite image data and second satellite image data, each of which has an observation date within a specified period before a specified date and whose observation range includes a specified area, and first satellite image data, the observation date of which is a specified date and whose observation range includes a specified area, and outputs estimated optical satellite image data having a second spatial resolution corresponding to the specified area and the specified date.

Description

[Technical Field]

[0001] The present disclosure relates to an image generation model generation device and an image estimation system for satellite images. [Background technology]

[0002] 2. Description of the Related Art Satellite remote sensing has been known for some time, in which an object to be observed is photographed from an artificial satellite and the state of the object to be observed is grasped from the photographed image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7643349 Summary of the Invention [Problem to be solved by the invention]

[0004] Optical satellites are sometimes used to identify observation targets. The higher the spatial resolution of an optical satellite, the more detailed the observation target can be. However, because optical satellites orbit the Earth to capture images of the observation target, the frequency with which a specific area including the observation target is captured may be inversely proportional to the spatial resolution. For example, optical satellites with high spatial resolution capture images of a specific area including the observation target infrequently, and may not be able to obtain satellite images on the desired day. Furthermore, because optical satellites capture satellite images by receiving sunlight reflected off the ground, if the ground is covered by clouds, the observation target is hidden, and the details of the observation target cannot be obtained from the obtained images. Therefore, optical satellites with high spatial resolution capture images infrequently and are difficult to obtain, and even if images are available on the desired day, the observation target may not be identified.

[0005] The embodiments of the present invention have been made to solve the above-mentioned problems, and aim to provide an image generation model generation device and an image estimation system that can obtain data about an observation target that is difficult to obtain. [Means for solving the problem]

[0006] An image generation model generation device according to one embodiment of the present invention is an image generation model generation device that generates an image generation model by machine learning using a plurality of satellite image data sets as training data, wherein the satellite training data sets include a first pair including first satellite image data and second satellite image data, each of which has an observation date within a first predetermined period and includes a predetermined area in its observation range, and a second pair including the first satellite image data and second satellite image data, each of which has an observation date within a second predetermined period and includes a predetermined area in its observation range, wherein the first satellite image data is optical satellite image data having a first spatial resolution, and the second satellite image data has a second spatial resolution that is higher than the first spatial resolution. the observation date of the first satellite image data and the second satellite image data of the first pair is earlier than the observation date of the first satellite image data and the second satellite image data of the second pair, and the machine learning is performed by using the first satellite image data and the second satellite image data of the first pair as input data and the second satellite image data of the second pair as training data to train the plurality of satellite image datasets, thereby estimating optical satellite image data that corresponds to the specified region and a specified day within the second specified period and has the second spatial resolution, and generating the image generation model that outputs the estimated optical satellite image data.

[0007] An image estimation system according to one embodiment of the present invention is an image estimation system equipped with the image generation model, which receives input of a pair including the first satellite image data and the second satellite image data, each of which has an observation date within the first predetermined period before the predetermined date and whose observation range includes a predetermined area, and the first satellite image data, the observation date of which is the predetermined date, and outputs estimated optical satellite image data having the second spatial resolution corresponding to the predetermined area and the predetermined date. [Effects of the Invention]

[0008] According to an embodiment of the present invention, it is possible to obtain data on an observation target that is difficult to obtain. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a hardware configuration of an image generation model generation device and an image estimation system according to an embodiment. [Figure 2] FIG. 1 is a diagram for explaining an image generation model generating device according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of an image estimation system according to an embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of an estimation operation of the image estimation system according to the embodiment. [Figure 5] 1 is an example of an operational flowchart of an image generation model generation device according to an embodiment. [Figure 6] 1 is an example of an operation flowchart of the image estimation system according to the embodiment. [Figure 7] FIG. 1 is a diagram for explaining an example using an image estimation system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image generation model generation device and an image estimation system according to an embodiment of the present invention will be described with reference to the drawings. In this specification, for the sake of convenience, unnecessary detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. Furthermore, in this specification, an artificial satellite may be simply referred to as a "satellite."

[0011] [composition] FIG. 1 is a diagram showing the hardware configuration of an image generation model generation device and an image estimation system according to this embodiment.

[0012] The image generation model generation device 10 and the image estimation system 20 may include a processor 1a, a storage device 1b, an input device 1c, an output device 1d, and a communication device 1e. The components 1a to 1e are connected by a bus 1f. Note that an interface may be interposed between the bus 1f and the components 1a to 1e as needed.

[0013] The image generation model generation device 10 and the image estimation system 20 may each include the components 1a to 1f. Furthermore, the image generation model generation device 10 and the image estimation system 20 may share some or all of the components 1a to 1f of these devices 10 and systems 20. In other words, some or all of the components 1a to 1f of the image estimation system 20 may also serve as some or all of the components 1a to 1f of the image generation model generation device 10.

[0014] The image generation model generation device 10 and the image estimation system 20 can be configured to include computers such as desktop computers, tablet computers, and laptop computers. That is, the image generation model generation device 10 and the image estimation system 20 may be configured as a single physical device or multiple physical devices. For example, the image generation model generation device 10 and the image estimation system 20 may be configured by cloud computing, which uses a computer located on a server on the Internet via the Internet.

[0015] The processor 1a controls the overall operation of the image generation model generation device 10 and / or the image estimation system 20. The processor 1a is, for example, an electronic circuit such as a CPU, MPU, or GPU. The processor 1a performs various processes by reading and executing programs and data stored in the storage device 1b. The processor 1a may be composed of multiple processors.

[0016] The storage device 1b includes RAM 1b-1, which is a volatile memory, and ROM 1b-2, which is a nonvolatile memory. The storage device 1b may also include an external memory 1b-3. RAM 1b-1 functions as the main memory and / or work area of ​​the processor 1a. The processor 1a loads programs and other data required for processing from ROM 1b-2 or the external memory 1b-3 into RAM 1b-1 and executes the loaded programs to perform various operations. The ROM 1b-2 and the external memory 1b-3 store the BIOS and OS, which are the control programs of the processor 1a, as well as various programs, data, tables, and other data required to perform functions executed by the computer. The external memory 1b-3 may include, for example, a flash memory, a hard disk, a DVD-RAM, a USB memory, an SSD, etc. The storage device 1b can store the results of calculations performed by the processor 1a.

[0017] The input device 1c accepts operation instructions and inputs from a user, etc. The input device 1c is a user interface such as an input button, a keyboard, a mouse, a touch panel, a touch pad, a wireless remote control, a microphone, a camera, etc. Note that the touch panel functions as both the input device 1c and the output device 1d.

[0018] The output device 1d outputs data processed by the processor 1a and data that is stored and / or has been stored in the storage device 1b. Examples of the output device 1d include display devices such as CRT displays, liquid crystal displays, organic EL displays, and plasma displays, audio devices such as speakers that emit sound, and printing devices such as printers.

[0019] The communication device 1e is an interface that connects to and communicates with external devices via a network or directly. The communication device 1e can be, for example, a serial interface, a LAN interface, or the like. The communication device 1e can communicate with artificial satellites and external storage.

[0020] The image generation model generation device 10 and the image estimation system 20 can realize the functions of various programs stored in the ROM 1b-2 and the external memory 1b-3 by using the components 1a to 1f as resources. Also, the image generation model generation device 10 and the image estimation system 20 may realize the functions of various programs by using the common components 1a to 1f.

[0021] (Image generation model generator) 2 is a diagram illustrating the image generation model generation device 10. The image generation model generation device 10 generates an image generation model 12 by machine learning using a plurality of satellite image data sets 11 as training data.

[0022] Each satellite image dataset of the multiple satellite image datasets 11 is training data used for machine learning and includes two pairs of first satellite image data and second satellite image data. The first satellite image data and second satellite image data of the first pair are satellite image data whose observation date falls within a first predetermined period and whose observation range includes a predetermined area. The first satellite image data and second satellite image data of the second pair are satellite image data whose observation date falls within a second predetermined period and whose observation range includes a predetermined area. The predetermined areas covered by the satellite image data of the first pair and the satellite image data of the second pair are the same. The observation date of the first satellite image data and second satellite image data of the first pair may be earlier than the observation date of the first satellite image data and second satellite image data of the second pair. The first and second predetermined periods may be on the order of days or weeks. The first and second predetermined periods may be, for example, one day, three days, or one week.

[0023] The first and second predetermined periods can be referred to as time widths, and can be common (constant) among each satellite image data set. For example, if the first and second predetermined periods (time widths) are three days, the observation date of each of the first and second satellite image data sets of the second pair is either day t-1, day t, or day t+1. The observation date of each of the first and second satellite image data sets of the first pair is either day (tn-1), day (tn), or day (t-n+1) (n is a natural number). The first and second predetermined periods can be common among each satellite image data set, but can have different time widths depending on the rate of change of the observation target. For example, if the rate of change of the observation target is fast, the predetermined period can be set short (e.g., one day), and if the rate of change of the observation target is slow, the predetermined period can be set long (e.g., one week). The observation target is not particularly limited, but can be, for example, the natural environment, a city, or a primary industry facility. The predetermined area to be the observation range is not particularly limited, but can be, for example, any area on the Earth.

[0024] The first and second predetermined periods may not overlap each other on the timeline. For example, if the time span is three days, the first pair of satellite image data may be data from the period from May 1st to 3rd, and the second pair of satellite image data may be data from the period from May 4th to 6th, etc., so that the time span is constant and the pairs arranged on the timeline do not overlap each other.

[0025] In this embodiment, the first predetermined period and the second predetermined period are one day. The first satellite image data and the second satellite image data of the first pair may be observed on the same day, and / or the first satellite image data and the second satellite image data of the second pair may be observed on the same day. Note that the first predetermined period (time span) and the second predetermined period (time span) may be different.

[0026] The first and second satellite image data of the second pair can be used as the first and second satellite image data of the first pair, and the new pair of first and second satellite image data can be used as the first and second satellite image data of the second pair. In other words, the second pair of satellite image data in the first satellite image dataset can be used as the first pair of satellite image data in the second satellite image dataset. Also, the first pair of satellite image data in the third satellite image dataset can be used as the second pair of satellite image data in the second satellite image dataset.

[0027] The first satellite image data is optical satellite image data having a first spatial resolution. The first spatial resolution can be, for example, several hundred to several km GDS, and in one example, 300 m GDS. The satellite that acquires the first satellite image data is an optical satellite with low resolution and wide-area observation capability that observes light (including visible light and infrared light) reflected from sunlight on the Earth's surface, and can be, for example, Sentinel-3, a satellite equipped with MODIS, or GCOM-C. The optical satellite can simultaneously observe various wavelengths in addition to RGB. For example, Sentinel-3 can observe 21 bands (colors). The recurrence period of the satellite that acquires the first satellite image data can be one to several days (e.g., one to two days). The satellite that acquires the first satellite image data in this embodiment is Sentinel-3.

[0028] The second satellite image data is training data and is optical satellite image data having a second spatial resolution higher than the first spatial resolution. The second spatial resolution can be, for example, 0.3 to 30 mGDS, and in one example, 10 mGDS. The satellite that acquires the second satellite image data is a medium- or high-resolution optical satellite that observes light (including visible light and infrared light) reflected from sunlight on the Earth's surface, such as Sentinel-2, Landsat, ALOS, a Planet satellite, a Maxar satellite, or an Airbus satellite. The optical satellite can simultaneously observe various wavelengths in addition to RGB. For example, Sentinel-2 can observe 13 bands (colors). The recurrence period of the satellite that acquires the second satellite image data can be several days to several weeks (for example, 5 days). The satellite that acquires the second satellite image data in this embodiment is Sentinel-2.

[0029] In this specification, the spatial resolution is defined as the same if the order of magnitude is the same after rounding, and different orders of magnitude are defined as different spatial resolutions. However, the definition of spatial resolution is not limited to this and can be based on common technical knowledge.

[0030] The first satellite image data and the second satellite image data may be data based on satellite data obtained from the various satellites, and may be, for example, image data obtained as a result of analyzing the satellite data. The satellite image data is data representing an image. The satellite image data may be data of one or more pixels. The data associated with a pixel is not particularly limited and may be, for example, data representing the reflection intensity of electromagnetic waves including visible light, color information (RGB, infrared, ultraviolet, etc.), which is the satellite data of the satellite, or data representing a predetermined index calculated from the satellite data of the satellite. If the satellite image data is index image data representing a predetermined index, the index image data may be data representing the distribution of the index at each point in a predetermined area. Furthermore, the index image data may be data for one pixel. In this case, an image representing the distribution of the index in a predetermined area can be obtained using multiple index image data for different pixels. Examples of the index include the Normalized Difference Vegetation Index (NDVI), the Normalized Difference Moisture Index (NDMI), the Normalized Water Index (NDWI), the Normalized Snow Index (NDSI), and the Normalized Soil Index (NDSI). The Normalized Difference Vegetation Index (NDVI) indicates the amount and activity of vegetation. The Normalized Wetness Index (NDMI) indicates the presence and amount of water in plants. The Normalized Water Index (NDWI) indicates the presence of water bodies (including snow) on the ground surface and the amount of water contained in vegetation. The Normalized Snow Cover Index (NDSI) indicates the area of ​​snow cover on the ground surface. The Normalized Soil Index (NDSI) indicates the distribution of sand, concrete, etc. on the ground surface. The indices are based on the reflectance of light reflected from the ground, and each index reflects light at a different wavelength. The indices exemplified above are publicly known, so details will be omitted.

[0031] The image generation model generation device 10 trains a machine learning model through machine learning using multiple satellite image data sets as training data, to generate an image generation model 12. That is, the image generation model generation device 10 estimates optical satellite image data having a second spatial resolution corresponding to a predetermined region and a predetermined day within a second predetermined period through machine learning that trains multiple satellite image data sets using a first pair of first satellite image data and second satellite image data, and the second pair of first satellite image data as training data, and generates an image generation model 12 that outputs the estimated optical satellite image data.

[0032] For example, the image generation model generation device 10 generates an image generation model by updating the machine learning model so as to reduce the error between the output result of the machine learning model and the second satellite image data of the second pair, which serves as training data, for each satellite image data set. In one example, the image generation model generation device 10 reads a program for implementing the machine learning model stored in the storage device 1b (e.g., ROM 1b-2, external memory 1b-3) into RAM 1b-1, and causes the processor 1a to execute the program, thereby operating the machine learning model and outputting the calculation results. The processor 1a also evaluates the error between the output result of the machine learning model and the training data, and updates the machine learning model by adjusting various parameters included in the machine learning model so as to reduce the error.

[0033] Examples of machine learning models that can be used include Unet, OneFormer, SegNet, Swin Transformer, Mask2Former, Pix2Pix, Cycle GAN, diffusion model, base model, multiple regression, decision tree regression, SVR (Support Vector Regression), Random Forest, kNN, gradient boosting (xgboost, lightgbm), Bayesian gradient boosting (ngboost), MLP (Multilayer Perceptron), and finetuning of pre-trained large-scale models. The machine learning model in this embodiment is LightGBM (decision tree based).

[0034] The number of training data sets is preferably large, but is not particularly limited. The number of training data sets can be, for example, on the order of 100. In one example, the number of training data sets can be 100, with each set being a satellite image dataset including a first pair and a second pair. As described above, a pair of first satellite image data and second satellite image data, each of whose observation dates falls within a predetermined period and whose observation range includes a predetermined area, can be used as both the first pair and the second pair. Furthermore, one pair may be used once or multiple times as the first pair or the second pair of different satellite image datasets. In other words, any two pairs from multiple pairs can be used as one satellite image dataset. For example, if there are three pairs A to C, three satellite image datasets ((1) pair A and pair B, (2) pair A and pair C, and (3) pair B and pair C) can be created. In this way, even if the number of pairs is limited, more satellite image datasets can be obtained.

[0035] The multiple satellite image datasets 11 used as training data may be stored in the storage device 1b of the image generation model generation device 10, or may be stored in external storage such as an external server connected online. The satellite image datasets may be newly stored in the storage device 1b or external storage and used for training the machine learning model. The image generation model generation device 10 can read or acquire the training data (multiple satellite image datasets 11) stored in the storage device 1b or external storage to perform machine learning.

[0036] The first satellite image data and the second satellite image data can be preprocessed data. That is, the satellites that acquire these satellite image data have different spatial resolutions, which may result in different observation ranges. Furthermore, the position of the satellite image may be shifted due to a shift in the position of the satellite that captures the image. Therefore, the first satellite image data and the second satellite image data can be satellite image data acquired from each satellite that has been subjected to subpixel accuracy correction (registration) to align the positions of the images, coordinate alignment, cropping to a predetermined size to make it easier to use in machine learning, and adding metadata including location information.

[0037] (Image estimation system) 3 is a diagram illustrating the configuration of an image estimation system according to this embodiment. The image estimation system 20 can include an input device 1c, an image generation model 12, and an output device 1d. The image generation model 12 can be a model generated by the image generation model generation device 10. A program and various parameters related to the image generation model 12 are stored in the storage device 1b of the image estimation system 20, and the image generation model 12 can be realized by reading the program and various parameters into RAM 1b-1 and executing the program with the processor 1a of the image estimation system 20.

[0038] The image generation model 12 can receive input of a pair of first and second satellite image data, each of which has an observation date within a predetermined period before a predetermined date and whose observation range includes a predetermined area, and the first satellite image data observed on the predetermined date, and output estimated optical satellite image data having a second spatial resolution corresponding to the predetermined area and the predetermined date. The predetermined period here can be the same as the first predetermined period in the learning stage. The first satellite image data observed on the predetermined date here can be data different from the first satellite image data of the learning data used in the image generation model generation device 10. Meanwhile, the first and second satellite image data of the input pair can be different from or the same as the learning data used in the image generation model generation device 10. The first and second satellite image data of the pair can be observed on the same date.

[0039] Furthermore, a pair of first satellite image data and second satellite image data can be used multiple times as part of input data for estimation by the image generation model 12. For example, a pair of first satellite image data and second satellite image data, each observed on January 1, can be used to estimate optical satellite image data for a predetermined date of January 5 and optical satellite image data for a predetermined date of January 7. After estimation by the image generation model 12, the first satellite image data observed on a predetermined date and the estimated optical satellite image data can be used as a pair as training data for machine learning in the image generation model generation device 10. If the satellite image data in the training dataset are each data for multiple pixels, the estimated optical satellite image data can be image data including data for multiple pixels. If the satellite image data in the training dataset are each data for one pixel, the estimated optical satellite image data can be image data for one pixel.

[0040] The image estimation system 20 receives an input of a predetermined date and a predetermined area, and based on the input, acquires a pair of first and second satellite image data, each of which has an observation date that falls within a predetermined period and is before the predetermined date, and whose observation range includes the predetermined area, to be input to the image generation model 12, as well as the first satellite image data whose observation date is the predetermined date and includes the predetermined area. For example, the input of the predetermined date and the predetermined area from the user can be received via the input device 1c. Based on the input, the image estimation system 20 searches for a group of first and second satellite image data stored in the storage device 1b of the image estimation system 20 or an external storage connected via the communication device 1e, and acquires from the group of data a pair of first and second satellite image data, each of which has an observation date that falls within a predetermined period and is before the predetermined date specified by the user, and whose observation range includes the predetermined area specified by the user, as well as the first satellite image data whose observation range on the predetermined date includes the predetermined area.

[0041] The period between the predetermined date and the observation date of each of the paired satellite image data is not particularly limited, but it is preferable that the period between the predetermined date and each of the observation dates is short. For example, if the observation dates are the same, the period between the predetermined date and the observation date of the paired satellite image data can be one day, two days, three days, one week, one month, etc.

[0042] The predetermined day can be a day outside the recurrent period of the optical satellite that acquires the second satellite image data, or a day on which the optical satellite acquires an image containing clouds in the second satellite image data. A day outside the recurrent period is a day on which the optical satellite does not observe the predetermined area. Even if the optical satellite observes the predetermined area, if the image shown by the second satellite image data contains clouds, the image may not be suitable for the user's intended use. Thus, the predetermined day can be a day on which the optical satellite acquires an image containing clouds in the second satellite image data.

[0043] The output device 1d outputs the estimated optical satellite data generated by the image generation model 12. For example, a display device serving as the output device 1d displays an image based on the estimated optical satellite data. The output device 1d or the image generation model 12 can output the estimated optical satellite data to an external device outside the image estimation system 20. The output destination can be, for example, a device including an application that uses the estimated optical satellite data, an external server, or the like.

[0044] Fig. 4 is a diagram for explaining an example of the estimation operation of the image estimation system 20. Fig. 4 shows an image 31 represented by first satellite image data in which a predetermined area is photographed on the (tn)th day and the tth day by an optical satellite having a first spatial resolution, and an image 32 represented by second satellite image data in which the predetermined area is photographed on the (tn)th day by an optical satellite having a second spatial resolution. The first satellite image data has a high temporal resolution (high observation frequency), but a lower spatial resolution than the optical satellite that photographs the second satellite image data.

[0045] Reference numeral 33 in Fig. 4 denotes an image represented by optical satellite image data having a second spatial resolution, including a predetermined region on day t, estimated by the image generation model 12, and is referred to here as an estimated satellite image 33. The estimated satellite image 33 is an image having colors such as RGB, similar to the image 32 represented by the second satellite image data. Note that S2 and S3 in the figure represent Sentinel-2 and Sentinel-3, respectively.

[0046] When a pair of first satellite image data and second generated image data on the (tn)th day and the first satellite image data on the tth day are input, the image generation model 12 of the image estimation system 20 outputs estimated satellite image data on the tth day. The estimated satellite image data is an estimate of the second satellite image data and is a substitute for the second satellite image data.

[0047] The second satellite image data is taken by an optical satellite with a relatively high spatial resolution, either medium or high, but the satellite has a low temporal resolution (i.e., observation frequency is low), and the image data becomes unavailable when the ground is covered with clouds. Therefore, the second satellite image data is difficult to obtain.

[0048] On the other hand, the first satellite image data image 31 is taken by an optical satellite with low spatial resolution and is affected by clouds, but since the satellite has high temporal resolution (i.e., frequent observation), the first satellite image data image 31 is highly available.

[0049] In this way, by inputting a first satellite image data that is highly available on a specified date and a pair of the first satellite image data and second satellite image data from before the specified date, the image generation model 12 can obtain estimated satellite image data that estimates the second satellite image data that is less available.

[0050] [Operation] 5 is an example of an operational flowchart of the image generation model generation device 10. Here, it is assumed that a plurality of satellite image data sets 11 are stored in the storage device 1b or an external storage connected via the Internet.

[0051] The image generation model generation device 10 reads or acquires multiple satellite image datasets 11 from the storage device 1b or external storage (S11: Read or acquire multiple satellite image datasets). For each satellite image dataset, the image generation model generation device 10 performs machine learning model training using satellite image data of the first pair of the set and the first satellite image data of the second pair as input and the second satellite image data of the second pair of the set as output (S12: Train machine learning model), thereby generating an image generation model 12 (S13: Generate image generation model). Specifically, for example, a program for the machine learning model is read from ROM 1b-1 or external memory 1b-3 to RAM 1b-1, and the processor 1a executes the program. At this time, for each satellite image dataset, the satellite image data of the first pair of the set and the first satellite image data of the second pair are input, and parameters related to the machine learning model are adjusted so that the output result of the machine learning model becomes the second satellite image data of the second pair of the set. These operations are performed for all of the multiple satellite image datasets 11 to generate the image generation model 12.

[0052] 6 is an example of an operation flowchart of the image estimation system 20. The image generation model 12 generated by the image generation model generation device 10 is used in the image estimation system 20.

[0053] First, the image estimation system 20 accepts input of a predetermined date and a predetermined area (S21: Accept input of predetermined date and predetermined area). In one example, the image estimation model 12 accepts input of the predetermined date and predetermined area from a user via the input device 1c. In another example, the image estimation model 12 accepts input of the predetermined date and predetermined area from an external application of the image estimation model 12.

[0054] Based on the received predetermined date and predetermined area, the image estimation system 20 acquires from the storage device 1b or an external storage a pair of first and second satellite image data whose observation date falls within a predetermined period before the predetermined date and whose observation range includes the predetermined area, as well as first satellite image data whose observation date falls on the predetermined date and whose observation range includes the predetermined area (S22: Acquisition of a pair of first and second satellite image data within a predetermined period, and first satellite image data for a predetermined date). For example, if the predetermined period is one day, the image estimation system 20 acquires first satellite image data whose observation date is the predetermined date, and a pair of first and second satellite image data whose observation date is before but on the same day as the predetermined date. The pair may be the first and second satellite image data for the observation date closest to the predetermined date, or the first and second satellite image data for any observation date before the predetermined date.

[0055] The image estimation system 20 inputs the acquired pair of first satellite image data and second satellite image data, and the first satellite image data observed on a predetermined date, into the image generation model 12 (S23: Input of pair of first satellite image data and second satellite image data within a predetermined period, and first satellite image data on a predetermined date into the image generation model). As a result, the image generation model 12 generates and outputs estimated satellite image data (S24: Generation of estimated satellite image data). This estimated satellite image data can be output to the output device 1d of the image estimation system 20 and / or an application external to the image estimation system 20 and used.

[0056] [Example] An example using the image estimation system 20 will be described with reference to FIG. 7. In response to inputting a pair of first satellite image data of image 31 taken on August 10, 2024 and second satellite image data of image 32 shown in FIG. 7, and the first satellite image data of image 31 taken on August 12, 2024, into the image generation model 12, the image generation model 12 outputs estimated satellite image data. The image represented by the estimated satellite image data is image 33 shown in FIG. 7. The first satellite image data is image data obtained by Sentinel-3 photographing an area in eastern Paraguay. The second satellite image data is image data obtained by Sentinel-2 photographing the above area.

[0057] On the other hand, image 34 shown in Fig. 7 is an image based on actual satellite data obtained by Sentinel-2 photographing the above area on August 12, 2024. Note that images 31 to 34 are all vegetation index images, which are images obtained by converting real images into grayscale.

[0058] As shown in Figure 7, it can be confirmed that the estimated satellite image data image 33 and the actual satellite image data image 34 are similar and have the same spatial resolution. Therefore, the estimated satellite image data can be used as a substitute for satellite image data with a second spatial resolution.

[0059] The image generation model 12 used in this embodiment was trained under the following conditions.

[0060] (conditions) Machine learning model: LightGBM (decision tree based) Number of training data sets: 100 The satellite that acquired the first satellite image data included in each training data set: Sentinel-2 The satellite that acquired the second satellite image data included in each training data set: Sentinel-3 Preprocessing of each satellite image data included in each set of training data: adding location information (Potential Encoding), aligning each image to sub-pixel accuracy, and adjusting image size The LightGBM code development (up to checking the learning operation) was done on a local Mac PC. The learning was performed and the learning results were easily checked on Google Colaboratory (abbreviated as Google Colab). The learning results dashboard was created on the local Mac PC. All Google Colab operations were performed on the browser on the local Mac PC.

[0061] [Actions and Effects] (1) An image generation model generation device 10 of this embodiment is an image generation model generation device that generates an image generation model 12 by machine learning using a plurality of satellite image datasets 11 as training data. The satellite training datasets include a first pair including first satellite image data and second satellite image data, each of which has an observation date within a first predetermined period and includes a predetermined area in its observation range, and a second pair including the first satellite image data and second satellite image data, each of which has an observation date within a second predetermined period and includes a predetermined area in its observation range, where the first satellite image data is optical satellite image data having a first spatial resolution and the second satellite image data has a spatial resolution lower than the first spatial resolution. the first pair of first satellite image data and the second satellite image data are observed on a date prior to the date of observation of the first satellite image data and the second satellite image data of the second pair; and machine learning is performed to train a plurality of satellite image datasets 11 using the first satellite image data and the second satellite image data of the first pair and the first satellite image data of the second pair as input data and the second satellite image data of the second pair as training data, thereby estimating optical satellite image data having the second spatial resolution corresponding to a predetermined region and a predetermined day within a second predetermined period, and generating an image generation model 12 that outputs the estimated optical satellite image data.

[0062] This makes it possible to obtain data (here, image data) about observation targets that are difficult to obtain. For example, even if high-spatial-resolution optical satellite images cannot be obtained by an optical satellite on a given day, a machine learning model that estimates the optical satellite images can be obtained, thereby complementing the high-spatial-resolution optical satellite images on the given day. As a result, changes in the observation targets can be captured. As described above, according to this embodiment, satellite image data that is highly available (e.g., high temporal resolution and low cost) but low spatial resolution and satellite image data that is high spatial resolution but less available (e.g., low temporal resolution and high cost) are combined into a set, and machine learning is performed using each satellite image data set for multiple periods. This makes it possible to obtain a machine learning model that can achieve super-resolution in both spatial and temporal resolution.

[0063] (2) The first satellite image data and the second satellite image data of the first pair are observed on the same day, or the first satellite image data and the second satellite image data of the second pair are observed on the same day. This improves the accuracy of the image data output by the image generation model 12 (spatial resolution and accuracy of the observation values ​​(data values) of each pixel of the image)) compared to when the first satellite image data and the second satellite image data are observed on different days.

[0064] (3) The first and second satellite image data of the second pair are used as the first and second satellite image data of the first pair, and the first and second satellite image data of another pair are used as the first and second satellite image data of the second pair. This makes it possible to obtain a larger number of satellite image data sets even with a small number of pairs.

[0065] (4) The image generation model 12 of the image estimation system 20 of this embodiment accepts input of a pair including first satellite image data and second satellite image data, each of which has an observation date within a specified period before a specified date and whose observation range includes a specified area, and first satellite image data, the observation date of which is a specified date and whose observation range includes a specified area, and outputs estimated optical satellite image data having a second spatial resolution corresponding to the specified area and the specified date.

[0066] This makes it possible to obtain data (here, image data) about observation targets that are difficult to obtain. For example, even if high-spatial-resolution optical satellite images cannot be obtained from an optical satellite on a given day, the optical satellite images can be estimated, making it possible to understand the status of the observation target on a given day. This makes it possible to capture changes in the observation target, leading to appropriate actions in response to these changes.

[0067] (5) The image estimation system 20 receives input of a predetermined date and a predetermined area, and based on the input, acquires a pair of first satellite image data and second satellite image data, each of which has an observation date that is before the predetermined date and within a predetermined period and whose observation range includes the predetermined area, and the first satellite image data, the observation date of which is the predetermined date and whose observation range includes the predetermined area, to be input to the image generation model 12. This makes it possible to automatically obtain estimated satellite image data by specifying a predetermined date and a predetermined area.

[0068] (6) The first and second satellite image data of a pair are observed on the same day. This improves the accuracy of the image data output by the image generation model 12 (spatial resolution and accuracy of the observed values ​​(data) of each pixel of the image)) compared to when the first and second satellite image data are observed on different days.

[0069] (7) The specified day is set to a day outside the orbital period of the optical satellite that acquires the second satellite image data or a day on which the optical satellite acquires an image containing clouds in the second satellite image data. This makes it possible to grasp the state of the specified area on days when there is no second satellite image data.

[0070] [Other embodiments] The image estimation system 20 may include the image generation model generation device 10. In this case, when a new satellite image data set is generated, the image estimation system 20 can update the image generation model 12 used in the image estimation system 20 using the image generation model generation device 10.

[0071] The image generation model is not limited to the generation method described in the above embodiment, and is not particularly limited as long as it accepts input of a pair of first satellite image data and second satellite image data, each of which has an observation date within a predetermined period before a predetermined date and whose observation range includes a predetermined area, and the first satellite image data, whose observation date is the predetermined date and whose observation range includes the predetermined area, and outputs estimated optical satellite image data having the second spatial resolution corresponding to the predetermined area and the predetermined date. The image generation model of image estimation system 20 may be, for example, a generation AI.

[0072] In the above embodiment, super-resolution has been described in which machine learning is performed using satellite image data and satellite image data with low spatial or temporal resolution is combined to improve both spatial and temporal resolution. However, the satellite image data may include index data calculated from the satellite data (e.g., numerical data indicated by an index of an area specified by latitude and longitude). The index indicated by the index data may be the index exemplified in the above embodiment. In this way, when satellite image data is used as index data, the image generation model generation device 10 may be replaced with an index generation model generation device, and the image estimation system 20 may be replaced with an index estimation system. Furthermore, the index estimation system may estimate index data corresponding to each pixel and generate an image showing the distribution of index data based on the estimated index data for multiple different pixels.

[0073] Furthermore, in the above embodiment, it is assumed that the satellite orbits the Earth, but the satellite may orbit a satellite other than the Earth. In other words, the present invention can be applied to cases where a satellite other than the Earth is the observation target.

[0074] In other embodiments of the present invention, the present invention may be a program that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts, or a computer-readable storage medium that stores the program.In still other embodiments, the present invention may be a method that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts.In still other embodiments, the present invention may be a server that can supply a program that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts to a computer.In still other embodiments, the present invention may be a virtual machine that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts.

[0075] In the processes or operations described above, the processes or operations can be freely changed as long as no inconsistencies in the processes or operations occur, such as the use of data that should not yet be available in a certain step. Furthermore, the embodiments described above are merely examples for explaining the present invention, and the present invention is not limited to these embodiments. The present invention can be embodied in various forms without departing from the spirit of the invention. [Explanation of symbols]

[0076] 1a processor 1b Storage device 1b-1 RAM 1b-2 ROM 1b-3 External memory 1c Input device 1d output device 1e Communication equipment 1F Bus 10 Image generation model generator 11 Multiple satellite imagery datasets 12 Image generation model 20 Image Estimation System 31 Image shown by the first satellite image data 32 Image shown by the second satellite image data 33 Image shown by estimated satellite image data 34 Actual satellite imagery

Claims

1. An image generation model generation device that generates an image generation model by machine learning using a plurality of satellite image datasets as training data, the satellite learning dataset includes a first pair including first satellite image data and second satellite image data, each of which has an observation date within a first predetermined period and an observation range that includes a predetermined area, and a second pair including first satellite image data and second satellite image data, each of which has an observation date within a second predetermined period and an observation range that includes a predetermined area; the first satellite image data is optical satellite image data having a first spatial resolution; the second satellite image data is optical satellite image data having a second spatial resolution higher than the first spatial resolution; an observation date of the first satellite image data and the second satellite image data of the first pair is earlier than an observation date of the first satellite image data and the second satellite image data of the second pair; using the first satellite image data and the second satellite image data of the first pair and the first satellite image data of the second pair as input data and the second satellite image data of the second pair as training data to train a plurality of satellite image data sets, estimating optical satellite image data corresponding to the specified area and a specified day within the second specified period and having the second spatial resolution, and generating the image generation model that outputs the estimated optical satellite image data. Image generation model generator.

2. the first satellite image data and the second satellite image data of the first pair are observed on the same day, or the first satellite image data and the second satellite image data of the second pair are observed on the same day; The image generation model generating device according to claim 1 .

3. the first satellite image data and the second satellite image data of the second pair are used as the first satellite image data and the second satellite image data of the first pair, and the first satellite image data and the second satellite image data of another pair are used as the first satellite image data and the second satellite image data of the second pair; The image generation model generating device according to claim 1 .

4. An image estimation system comprising the image generation model according to any one of claims 1 to 3, the image generation model receives input of a pair including the first satellite image data and the second satellite image data, each of which has an observation date within a predetermined period before the predetermined date and includes a predetermined area in its observation range, and the first satellite image data, the observation date of which is the predetermined date and includes the predetermined area in its observation range, and outputs estimated optical satellite image data having the second spatial resolution corresponding to the predetermined area and the predetermined date. Image estimation system.

5. Accepting input of the predetermined date and the predetermined area; Based on the input, a pair including the first satellite image data and the second satellite image data, each of which has an observation date that is before the specified date and within the specified period and whose observation range includes the specified area, is acquired, and the first satellite image data, the observation date of which is the specified date and whose observation range includes the specified area, is acquired. The image estimation system according to claim 4 .

6. the first satellite image data and the second satellite image data of the pair are observed on the same day; The image estimation system according to claim 4 .

7. the predetermined day is a day outside the recurrent period of an optical satellite that acquires the second satellite image data, or a day on which an image containing clouds is acquired from the second satellite image data acquired by the optical satellite. The image estimation system according to claim 4 .

Images