Image generation model generating device and image estimation system

Abstract

An image generation model generating device and an image estimation system are provided that are capable of obtaining data relating to an observation target that is difficult to obtain. [Solution] The image generation model 12 of the image estimation system 20 accepts input of first satellite image data and second satellite image data whose observation date is within a specified period including a specified date and whose observation range includes a specified area, and outputs estimated optical satellite image data having a third spatial resolution corresponding to the specified area and the specified date.

Description

[Technical Field]

[0001] The present disclosure relates to an image generation model generation device and an image estimation system for satellite images. [Background technology]

[0002] 2. Description of the Related Art Satellite remote sensing has been known for some time, in which an object to be observed is photographed from an artificial satellite and the state of the object to be observed is grasped from the photographed image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7643349 Summary of the Invention [Problem to be solved by the invention]

[0004] Optical satellites are sometimes used to identify observation targets. The higher the spatial resolution of an optical satellite, the more detailed the observation target can be. However, because optical satellites orbit the Earth to capture images of the observation target, the frequency with which a specific area including the observation target is captured may be inversely proportional to the spatial resolution. For example, optical satellites with high spatial resolution capture images of a specific area including the observation target infrequently, and may not be able to obtain satellite images on the desired day. Furthermore, because optical satellites capture satellite images by receiving sunlight reflected off the ground, if the ground is covered by clouds, the observation target is hidden, and the details of the observation target cannot be obtained from the obtained images. Therefore, optical satellites with high spatial resolution capture images infrequently and are difficult to obtain, and even if images are available on the desired day, the observation target may not be identified.

[0005] The embodiments of the present invention have been made to solve the above-mentioned problems, and aim to provide an image generation model generation device and an image estimation system that can obtain data about an observation target that is difficult to obtain. [Means for solving the problem]

[0006] An image generation model generation device according to one embodiment of the present invention is an image generation model generation device that generates an image generation model through machine learning using satellite image data sets for multiple time periods as training data, wherein the satellite image data sets include a set of first satellite image data, second satellite image data, and third satellite image data, the observation dates of which fall within a predetermined time period and the observation ranges of which include a predetermined area, the first satellite image data being radar satellite image data having a first spatial resolution, the second satellite image data being optical satellite image data having a second spatial resolution lower than the first spatial resolution, and the third satellite image data being training data that is optical satellite image data having a third spatial resolution higher than the second spatial resolution, and the image generation model generation device estimates optical satellite image data having the third spatial resolution that corresponds to the predetermined area and a predetermined day within the predetermined time period and that has the third spatial resolution, by using the machine learning to train the multiple satellite image data sets using the first satellite image data and the second satellite image data as input data and the third satellite image data as training data, and generates the image generation model that outputs the estimated optical satellite image data.

[0007] An image estimation system according to one embodiment of the present invention is an image estimation system equipped with the image generation model, wherein the image generation model receives input of the first satellite image data and the second satellite image data, the observation date of which is within a predetermined period including a predetermined date and the observation range of which includes a predetermined area, and outputs estimated optical satellite image data having the third spatial resolution corresponding to the predetermined area and the predetermined date. [Effects of the Invention]

[0008] According to an embodiment of the present invention, it is possible to obtain data on an observation target that is difficult to obtain. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a hardware configuration of an image generation model generation device and an image estimation system according to an embodiment. [Figure 2] FIG. 1 is a diagram for explaining an image generation model generating device according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of an image estimation system according to an embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of an estimation operation of the image estimation system according to the embodiment. [Figure 5] 1 is an example of an operational flowchart of an image generation model generation device according to an embodiment. [Figure 6] 1 is an example of an operation flowchart of the image estimation system according to the embodiment. [Figure 7] FIG. 1 is a diagram for explaining an example using an image estimation system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image generation model generation device and an image estimation system according to an embodiment of the present invention will be described with reference to the drawings. In this specification, for the sake of convenience, unnecessary detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. Furthermore, in this specification, an artificial satellite may be simply referred to as a "satellite."

[0011] [composition] FIG. 1 is a diagram showing the hardware configuration of an image generation model generation device and an image estimation system according to this embodiment.

[0012] The image generation model generation device 10 and the image estimation system 20 may include a processor 1a, a storage device 1b, an input device 1c, an output device 1d, and a communication device 1e. The components 1a to 1e are connected by a bus 1f. Note that an interface may be interposed between the bus 1f and the components 1a to 1e as needed.

[0013] The image generation model generation device 10 and the image estimation system 20 may each include the components 1a to 1f. Furthermore, the image generation model generation device 10 and the image estimation system 20 may share some or all of the components 1a to 1f of these devices 10 and systems 20. In other words, some or all of the components 1a to 1f of the image estimation system 20 may also serve as some or all of the components 1a to 1f of the image generation model generation device 10.

[0014] The image generation model generation device 10 and the image estimation system 20 can be configured to include computers such as desktop computers, tablet computers, and laptop computers. That is, the image generation model generation device 10 and the image estimation system 20 may be configured as a single physical device or multiple physical devices. For example, the image generation model generation device 10 and the image estimation system 20 may be configured by cloud computing, which uses a computer located on a server on the Internet via the Internet.

[0015] The processor 1a controls the overall operation of the image generation model generation device 10 and / or the image estimation system 20. The processor 1a is, for example, an electronic circuit such as a CPU, MPU, or GPU. The processor 1a performs various processes by reading and executing programs and data stored in the storage device 1b. The processor 1a may be composed of multiple processors.

[0016] The storage device 1b includes RAM 1b-1, which is a volatile memory, and ROM 1b-2, which is a nonvolatile memory. The storage device 1b may also include an external memory 1b-3. RAM 1b-1 functions as the main memory and / or work area of ​​the processor 1a. The processor 1a loads programs and other data required for processing from ROM 1b-2 or the external memory 1b-3 into RAM 1b-1 and executes the loaded programs to perform various operations. The ROM 1b-2 and the external memory 1b-3 store the BIOS and OS, which are the control programs of the processor 1a, as well as various programs, data, tables, and other data required to perform functions executed by the computer. The external memory 1b-3 may include, for example, a flash memory, a hard disk, a DVD-RAM, a USB memory, an SSD, etc. The storage device 1b can store the results of calculations performed by the processor 1a.

[0017] The input device 1c accepts operation instructions and inputs from a user, etc. The input device 1c is a user interface such as an input button, a keyboard, a mouse, a touch panel, a touch pad, a wireless remote control, a microphone, a camera, etc. Note that the touch panel functions as both the input device 1c and the output device 1d.

[0018] The output device 1d outputs data processed by the processor 1a and data that is stored and / or has been stored in the storage device 1b. Examples of the output device 1d include display devices such as CRT displays, liquid crystal displays, organic EL displays, and plasma displays, audio devices such as speakers that emit sound, and printing devices such as printers.

[0019] The communication device 1e is an interface that connects to and communicates with external devices via a network or directly. The communication device 1e can be, for example, a serial interface, a LAN interface, or the like. The communication device 1e can communicate with artificial satellites and external storage.

[0020] The image generation model generation device 10 and the image estimation system 20 can realize the functions of various programs stored in the ROM 1b-2 and the external memory 1b-3 by using the components 1a to 1f as resources. Also, the image generation model generation device 10 and the image estimation system 20 may realize the functions of various programs by using the common components 1a to 1f.

[0021] (Image generation model generator) 2 is a diagram illustrating the image generation model generation device 10. The image generation model generation device 10 generates an image generation model 12 by machine learning using satellite image data sets (plurality of satellite image data sets 11) for multiple time periods as training data.

[0022] Each satellite image dataset of the plurality of satellite image datasets 11 is training data used for machine learning and includes a set of first satellite image data, second satellite image data, and third satellite image data. The first satellite image data, second satellite image data, and third satellite image data are satellite image data whose observation dates fall within a predetermined period and whose observation range includes a predetermined area. The predetermined period for the satellite image datasets used as training data can be on the order of days or weeks. The predetermined period can be, for example, one day, three days, or one week.

[0023] The predetermined period can be referred to as a time width, and can be common (constant) among each satellite image data set. If the predetermined period (time width) is, for example, three days, the observation date of each of the first satellite image data, the second satellite image data, and the third satellite image data is either day t-1, day t, or day t+1. The predetermined period can be common among each satellite image data set, and can have different time widths depending on the rate of change of the observation object. For example, if the rate of change of the observation object is fast, the predetermined period can be set short (e.g., one day), and if the rate of change of the observation object is slow, the predetermined period can be set long (e.g., one week). The observation object is not particularly limited, and can be, for example, a natural environment, a city, a primary industry facility, etc. The predetermined area as the observation range is not particularly limited, and can be, for example, any area on the Earth.

[0024] The predetermined periods covered by the satellite image data sets may not overlap with each other in time series. For example, if the predetermined period is three days, the first satellite image data set may be a data set covering the period from May 1 to 3, the second satellite image data set may be a data set covering the period from May 4 to 6, etc., and the predetermined periods (time width) may be constant, and the predetermined periods of the sets arranged in time series may not overlap with each other. As another example, the predetermined periods covered by the satellite image data sets may at least partially overlap with each other in time series. For example, the first satellite image data set may be a data set covering the period from May 1 to 3, the second satellite image data set may be a data set covering the period from May 2 to 5, and the third satellite image data set may be a data set covering the period from May 8 to 10, etc., and the predetermined periods (time width) may be constant, and the predetermined periods of the sets arranged in time series may partially overlap with each other.

[0025] The first satellite image data is radar satellite image data having a first spatial resolution. The first spatial resolution can be, for example, 0.3 to 100 m GSD, preferably 0.3 to 30 m GSD, and in one example, 10 m GSD. The spatial resolution is the ground sampling distance (GSD). A 10 m GSD indicates that one pixel in an image captures a range of 10 m on the ground. The satellite that acquires the first satellite image data is a medium- or high-resolution radar satellite (SAR satellite) that emits radar light onto the ground and observes its reflection. Examples of medium-resolution radar satellites include Sentinel-1 (spatial resolution 0.3 to 30 m GSD, recurrence period 12 or 6 days), ALOS-2 (spatial resolution 1 to 100 m GSD, recurrence period 14 days), and ALOS-4 (spatial resolution 1 to 100 m GSD, recurrence period 14 days). Examples of high-resolution radar satellites include ICEYE, Capella Space, StriX, QPS-SAR, and Umbra (all with a spatial resolution of several tens of centimeters GDS). Because radar can penetrate clouds, it is possible to observe the target without being affected by clouds. The recurrence period of the satellite, i.e., the number of days between observations (photographs), can be, for example, 6 to 14 days. There are two Sentine-1 satellites, each with a recurrence period of 12 days, so the effective recurrence period from the perspective of data acquisition can be 6 days. The satellite that acquires the first satellite image data in this embodiment is Sentinel-1.

[0026] The second satellite image data is optical satellite image data having a second spatial resolution lower than the first spatial resolution. The second spatial resolution can be, for example, several hundred to several km GDS, and in one example, 300 m GDS. The satellite that acquires the second satellite image data is an optical satellite with low resolution and wide-area observation capability that observes light (including visible light and infrared light) reflected from sunlight on the Earth's surface, and can be, for example, Sentinel-3, a satellite equipped with MODIS, or GCOM-C. The optical satellite can simultaneously observe various wavelengths in addition to RGB. For example, Sentinel-3 can observe 21 bands (colors). The recurrence period of the satellite that acquires the second satellite image data can be one to several days (e.g., one to two days). The satellite that acquires the second satellite image data in this embodiment is Sentinel-3.

[0027] The third satellite image data is training data and is optical satellite image data having a third spatial resolution higher than the second spatial resolution. The third spatial resolution can be, for example, 0.3 to 30 m GDS, and in one example, 10 m GDS. The third spatial resolution can be the same as the first spatial resolution. The satellite that acquires the third satellite image data is a medium- or high-resolution optical satellite that observes light (including visible light and infrared light) reflected from sunlight on the Earth's surface, such as Sentinel-2, Landsat, ALOS, a Planet satellite, a Maxar satellite, or an Airbus satellite. The optical satellite can simultaneously observe various wavelengths in addition to RGB. For example, Sentinel-2 can observe 13 bands (colors). The recurrence period of the satellite that acquires the third satellite image data can be several days to several weeks (e.g., 5 days). The satellite that acquires the third satellite image data in this embodiment is Sentinel-2.

[0028] In this specification, the spatial resolution is defined as the same if the order of magnitude is the same after rounding, and different orders of magnitude are defined as different spatial resolutions. However, the definition of spatial resolution is not limited to this and can be based on common technical knowledge.

[0029] The first satellite image data, the second satellite image data, and the third satellite image data may be data based on satellite data obtained from the various satellites, and may be, for example, image data obtained as a result of analyzing the satellite data. The satellite image data is data representing an image. The satellite image data may be data of one or more pixels. The data associated with a pixel is not particularly limited and may be, for example, data representing the reflection intensity of electromagnetic waves including visible light, color information (RGB, infrared, ultraviolet, etc.), which is the satellite data of the satellite, or data representing a predetermined index calculated from the satellite data of the satellite. If the satellite image data is index image data representing a predetermined index, the index image data may be data representing the distribution of the index at each point in a predetermined area. Furthermore, the index image data may be data for one pixel. In this case, an image representing the distribution of the index in a predetermined area can be obtained using multiple index image data for different pixels. Examples of the index include the Normalized Difference Vegetation Index (NDVI), the Normalized Difference Moisture Index (NDMI), the Normalized Water Index (NDWI), the Normalized Snow Index (NDSI), and the Normalized Soil Index (NDSI). The Normalized Difference Vegetation Index (NDVI) indicates the amount and activity of vegetation. The Normalized Wetness Index (NDMI) indicates the presence and amount of water in plants. The Normalized Water Index (NDWI) indicates the presence of water bodies (including snow) on the ground surface and the amount of water contained in vegetation. The Normalized Snow Cover Index (NDSI) indicates the area of ​​snow cover on the ground surface. The Normalized Soil Index (NDSI) indicates the distribution of sand, concrete, etc. on the ground surface. The indices are based on the reflectance of light reflected from the ground, and each index reflects light at a different wavelength. The indices exemplified above are publicly known, so details will be omitted.

[0030] The image generation model generation device 10 uses satellite image data sets for multiple time periods as training data to train a machine learning model through machine learning, thereby generating an image generation model 12. That is, the image generation model generation device 10 uses first satellite image data and second satellite image data as input data and third satellite image data as training data to train multiple satellite image data sets, thereby estimating optical satellite image data that includes a specified region, corresponds to a specified day within a specified time period, and has a third spatial resolution, and generates an image generation model 12 that outputs the estimated optical satellite image data.

[0031] For example, the image generation model generation device 10 generates an image generation model by updating the machine learning model so as to reduce the error between the output result of the machine learning model, which uses first satellite image data and second satellite image data as input data, and third satellite image data that serves as training data, for each satellite image data set. In one example, the image generation model generation device 10 reads a program for implementing the machine learning model stored in the storage device 1b (e.g., ROM 1b-2, external memory 1b-3) into RAM 1b-1, and causes the processor 1a to execute the program, thereby operating the machine learning model and outputting the calculation results. The processor 1a also evaluates the error between the output result of the machine learning model and the training data, and updates the machine learning model by adjusting various parameters included in the machine learning model so as to reduce the error.

[0032] Examples of machine learning models that can be used include Unet, OneFormer, SegNet, Swin Transformer, Mask2Former, Pix2Pix, Cycle GAN, diffusion model, base model, multiple regression, decision tree regression, SVR (Support Vector Regression), Random Forest, kNN, gradient boosting (xgboost, lightgbm), Bayesian gradient boosting (ngboost), MLP (Multilayer Perceptron), and finetuning of pre-trained large-scale models. The machine learning model in this embodiment is Unet.

[0033] The number of training data sets is preferably large, but is not particularly limited. The number of training data sets can be, for example, on the order of 100. In one example, the number of training data sets can be 100, with each set consisting of first satellite image data, second satellite image data, and third satellite image data whose observation dates fall within a predetermined period. The predetermined periods of the sets do not need to overlap, or the sets may at least partially overlap. For example, if there are 100 sets, the predetermined period of each set is three days, and the predetermined periods of the sets do not overlap, the training data may include, for example, 300 pieces of data spanning a 300-day period. If the sets at least partially overlap, at least one of the first satellite image data, second satellite image data, and third satellite image data included in one set may also be used as data in another set. For example, the third satellite image data whose observation date is May 5 and which is included in the first set for the period from May 3 to 5 may be used as the third satellite image data in the second set for the period from May 4 to 6.

[0034] The multiple satellite image datasets 11 used as training data may be stored in the storage device 1b of the image generation model generation device 10, or may be stored in external storage such as an external server connected online. The satellite image datasets may be newly stored in the storage device 1b or external storage and used for training the machine learning model. The image generation model generation device 10 can read or acquire the training data (multiple satellite image datasets 11) stored in the storage device 1b or external storage to perform machine learning.

[0035] The first satellite image data, the second satellite image data, and the third satellite image data can be preprocessed data. That is, the satellites that acquire these satellite image data have different spatial resolutions, which may result in different observation ranges. Furthermore, the position of the satellite image may be shifted due to a shift in the position of the satellite that acquires the image. Therefore, the first satellite image data, the second satellite image data, and the third satellite image data can be satellite image data acquired from each satellite that has been subjected to subpixel accuracy correction (registration) to align the positions of the images, coordinate alignment, cropping to a predetermined size to make it easier to use in machine learning, and adding metadata including location information.

[0036] (Image estimation system) 3 is a diagram illustrating the configuration of an image estimation system according to this embodiment. The image estimation system 20 can include an input device 1c, an image generation model 12, and an output device 1d. The image generation model 12 can be a model generated by the image generation model generation device 10. A program and various parameters related to the image generation model 12 are stored in the storage device 1b of the image estimation system 20, and the image generation model 12 can be realized by reading the program and various parameters into RAM 1b-1 and executing the program with the processor 1a of the image estimation system 20.

[0037] The image generation model 12 can accept input of first satellite image data and second satellite image data, the observation date of which is within a predetermined period including a predetermined date and the observation range of which includes a predetermined area, and output estimated optical satellite image data that includes the predetermined area and has a third spatial resolution corresponding to the predetermined date. The first satellite image data and second satellite image data here can be data different from the training data used in the image generation model generation device 10. On the other hand, the input first satellite image data and second satellite image data can be either different from or the same as the training data used in the image generation model generation device 10. After estimation by the image generation model 12, the first satellite image data, the second satellite image data, and the estimated optical satellite image data can be used as a single satellite image dataset for machine learning in the image generation model generation device 10. The estimated optical satellite image data can be image data including data for multiple pixels if each of the satellite image data in the training dataset is data for multiple pixels, or can be single-pixel image data if each of the satellite image data in the training dataset is data for one pixel.

[0038] The image estimation system 20 receives input of a predetermined date and a predetermined area, and based on the input, acquires first satellite image data and second satellite image data whose observation date falls within a predetermined period including the predetermined date and whose observation range includes the predetermined area, to be input to the image generation model 12. For example, the input of the predetermined date and the predetermined area from the user can be received via the input device 1c. Based on the input, the image estimation system 20 searches for a group of first satellite image data and second satellite image data stored in the storage device 1b of the image estimation system 20 or an external storage connected via the communication device 1e, and acquires, from the group of data, first satellite image data and second satellite image data whose observation date falls within a predetermined period including the predetermined date specified by the user and whose observation range includes the predetermined area specified by the user.

[0039] The predetermined day can be a day outside the recurrent period of the optical satellite that acquires the third satellite image data, or a day on which the optical satellite acquires an image containing clouds in the third satellite image data. A day outside the recurrent period is a day on which the optical satellite does not observe the predetermined area. Even if the optical satellite observes the predetermined area, if the image shown by the third satellite image data contains clouds, the image may not be suitable for the user's intended use. Thus, the predetermined day can be a day on which the optical satellite acquires an image containing clouds in the third satellite image data.

[0040] The output device 1d outputs the estimated optical satellite data generated by the image generation model 12. For example, a display device serving as the output device 1d displays an image based on the estimated optical satellite data. The output device 1d or the image generation model 12 can output the estimated optical satellite data to an external device outside the image estimation system 20. The output destination can be, for example, a device including an application that uses the estimated optical satellite data, an external server, or the like.

[0041] FIG. 4 is a diagram illustrating an example of the estimation operation of the image estimation system 20. FIG. 4 shows an image 31 represented by first satellite image data, in which a predetermined area is photographed by a radar satellite having a first spatial resolution on the (td)th day, and an image 32 represented by second satellite image data, in which the predetermined area is photographed by an optical satellite having a second spatial resolution on the (t+d)th day. d can be, for example, 1, but is not limited to this. The first satellite image data is radar image data that is not affected by clouds and is black and white image data. However, the spatial resolution of the first satellite image data is equivalent to or the same as that of the third satellite image data and is higher than that of the second satellite image data. The second satellite image data has a high temporal resolution (high observation frequency) but a lower spatial resolution than that of the optical satellite that photographs the third satellite image data. However, the image 32 represented by the second satellite image data is an image having colors such as RGB. The symbol 32' in Figure 4 is an image represented by second satellite image data in which the specified area was photographed by an optical satellite having a second spatial resolution on the (td)th day and the tth day, respectively. However, the image represented by the data contains clouds, and therefore cannot be used for image estimation.

[0042] Reference numeral 33 in Fig. 4 denotes an image represented by optical satellite image data having a third spatial resolution, including a predetermined region on day t, estimated by the image generation model 12, and is referred to here as an estimated satellite image 33. The estimated satellite image 33 is an image having colors such as RGB, similar to the image 32 represented by the second satellite image data. Note that S1, S2, and S3 in the figure represent Sentinel-1, 3, and 2, respectively.

[0043] When the first satellite image data and the second generated image data are input, the image generation model 12 of the image estimation system 20 outputs estimated satellite image data. The estimated satellite image data is an estimate of the third satellite image data and is a substitute for the third satellite image data.

[0044] The third satellite image data is taken by a medium-resolution optical satellite with a relatively high spatial resolution, but the satellite has a low temporal resolution (i.e., observation frequency is low), and the image data becomes unavailable when the ground is covered with clouds. Therefore, the third satellite image data is difficult to obtain.

[0045] On the other hand, the first satellite image data image 31 was taken by a radar satellite with a medium resolution, which has a relatively high spatial resolution, and is therefore easily available because the radar can penetrate clouds and is not affected by clouds. The second satellite image data image 32 was taken by an optical satellite with a low spatial resolution, and although it is affected by clouds, the satellite has a high temporal resolution (i.e., observation frequency is high), so the second satellite image data image 32 is easily available.

[0046] In this way, by inputting the first satellite image data, which is highly available and has the same spatial resolution as the third satellite image data, and the second satellite image data, which is highly available and has low resolution but has color, the image generation model 12 can obtain estimated satellite image data that estimates the third satellite image data, which is less available.

[0047] [Operation] 5 is an example of an operational flowchart of the image generation model generation device 10. Here, it is assumed that each satellite image data set includes first satellite image data, second satellite image data, and third satellite image data within three days, and that multiple satellite image data sets 11 are stored in the storage device 1b or an external storage connected via the Internet.

[0048] The image generation model generation device 10 reads or acquires multiple satellite image datasets 11 from the storage device 1b or external storage (S11: Read or acquire multiple satellite image datasets). For each satellite image dataset, the image generation model generation device 10 performs machine learning model training using the first satellite image data and second satellite image data of the set as input and the third satellite image data as output (S12: Train machine learning model), thereby generating an image generation model 12 (S13: Generate image generation model). Specifically, for example, a program for the machine learning model is read from ROM 1b-1 or external memory 1b-3 to RAM 1b-1, and the processor 1a executes the program. At that time, for each satellite image dataset, the first satellite image data and second satellite image data of the set are input, and parameters related to the machine learning model are adjusted so that the output result of the machine learning model becomes the third satellite image data of the set. These operations are performed for all of the multiple satellite image datasets 11 to generate an image generation model 12.

[0049] 6 is an example of an operation flowchart of the image estimation system 20. The image generation model 12 generated by the image generation model generation device 10 is used in the image estimation system 20.

[0050] First, the image estimation system 20 accepts input of a predetermined date and a predetermined area (S21: Accept input of predetermined date and predetermined area). In one example, the image estimation model 12 accepts input of the predetermined date and predetermined area from a user via the input device 1c. In another example, the image estimation model 12 accepts input of the predetermined date and predetermined area from an external application of the image estimation model 12.

[0051] Based on the received predetermined date and predetermined area, the image estimation system 20 acquires from the storage device 1b or external storage first satellite image data and second satellite image data whose observation date falls within a predetermined period including the predetermined date and whose observation range includes the predetermined area (S22: Acquisition of first satellite image data and second satellite image data). For example, if the predetermined period is three days, the first satellite image data and second satellite image data may be data for one day before and after the predetermined date, and data for the predetermined date.

[0052] The image estimation system 20 inputs the acquired first satellite image data and second satellite image data to the image generation model 12 (S23: Input of first satellite image data and second satellite image data to image generation model). As a result, the image generation model 12 generates and outputs estimated satellite image data (S24: Generation of estimated satellite image data). This estimated satellite image data can be output to the output device 1d of the image estimation system 20 and / or an application external to the image estimation system 20 for use.

[0053] [Example] An example using the image estimation system 20 will be described with reference to FIG. 7. In response to inputting the first satellite image data of image 31 and the second satellite image data of image 32 shown in FIG. 7 into the image generation model 12, the image generation model 12 outputs estimated satellite image data. The image represented by the estimated satellite image data is image 33 shown in FIG. 7. The first satellite image data is image data obtained by photographing an area in eastern Paraguay by Sentinel-1 on August 13, 2024. The second satellite image data is image data obtained by photographing the above area by Sentinel-3 on August 15, 2024. The data associated with each pixel of the first satellite image data and the second satellite image data here is a vegetation index. Therefore, image 33 is an image indicating a vegetation index.

[0054] On the other hand, image 34 shown in Figure 7 is an image based on actual satellite data obtained by Sentinel-2 photographing the above area on August 15, 2024, the same time as images 31 to 33. Images 31 to 34 are all vegetation index images, and are images obtained by converting real images into grayscale.

[0055] As shown in Figure 7, it can be confirmed that the estimated satellite image data image 33 and the actual satellite image data image 34 are similar and have the same spatial resolution. Therefore, the estimated satellite image data can be used as a substitute for satellite image data with a third spatial resolution.

[0056] The image generation model 12 used in this embodiment was trained under the following conditions.

[0057] (conditions) Machine learning model: Unet Number of training data sets: 100 The satellite that acquired the first satellite image data included in each training data set: Sentinel-1 The satellite that acquired the second satellite image data included in each training data set: Sentinel-2 The satellite that acquired the third satellite image data included in each training data set: Sentinel-3 Preprocessing of each satellite image data included in each set of training data: adding location information (Potential Encoding), aligning each image to sub-pixel accuracy, and adjusting image size The Unet code development (confirmation of learning operation) was performed on a local Mac PC. Learning was performed and simple confirmation of learning results was performed on Google Colaboratory (abbreviated as Google Colab). A dashboard of learning results was created on the local Mac PC. All Google Colab operations were performed on the browser on the local Mac PC.

[0058] [Actions and Effects] (1) The image generation model generation device 10 of this embodiment is an image generation model generation device that generates an image generation model 12 through machine learning using satellite image data sets for multiple time periods as training data. The satellite image data sets include a set of first satellite image data, second satellite image data, and third satellite image data, the observation dates of which are within a specified time period and the observation ranges of which include a specified area. The first satellite image data is radar satellite image data having a first spatial resolution, the second satellite image data is optical satellite image data having a second spatial resolution lower than the first spatial resolution, and the third satellite image data is training data, which is optical satellite image data having a third spatial resolution higher than the second spatial resolution. The image generation model generation device 10 uses the first satellite image data and the second satellite image data as input data and the third satellite image data as training data to train multiple satellite image data sets, thereby estimating optical satellite image data having the third spatial resolution that corresponds to a specified area and a specified day within the specified time period, and generates an image generation model 12 that outputs the estimated optical satellite image data.

[0059] This makes it possible to obtain data (here, image data) about observation targets that are difficult to obtain. For example, even if high-spatial-resolution optical satellite images cannot be obtained by an optical satellite on a given day, a machine learning model that estimates the optical satellite images can be obtained, thereby complementing the high-spatial-resolution optical satellite images on the given day. As a result, changes in the observation targets can be captured. As described above, according to this embodiment, satellite image data from a type of satellite (radar satellite) different from optical satellites that is highly available because it is not affected by clouds, satellite image data that is highly available (e.g., high temporal resolution and low price) but low spatial resolution, and satellite image data that is high spatial resolution but less available (e.g., low temporal resolution and high price) are combined into a set, and machine learning is performed using each satellite image data set for a plurality of time periods. This makes it possible to obtain a machine learning model that can achieve super-resolution in both spatial and temporal resolution.

[0060] (2) The first spatial resolution is set to be the same as the third spatial resolution, which improves the accuracy of the image data output by the image generation model 12 (spatial resolution and accuracy of the observed values ​​(data values) of each pixel of the image)).

[0061] (3) The predetermined period of the satellite image dataset used as training data is set to a period on the order of days or weeks, which allows us to obtain an image generation model 12 that outputs estimated satellite image data that matches the rate of change of the observed object.

[0062] (4) The image generation model 12 of the image estimation system 20 of this embodiment accepts input of first satellite image data and second satellite image data whose observation date is within a specified period including a specified date and whose observation range includes a specified area, and outputs estimated optical satellite image data having a third spatial resolution corresponding to the specified area and the specified date.

[0063] This makes it possible to obtain data (here, image data) about observation targets that are difficult to obtain. For example, even if high-spatial-resolution optical satellite images cannot be obtained from an optical satellite on a given day, the optical satellite images can be estimated, making it possible to understand the status of the observation target on a given day. This makes it possible to capture changes in the observation target, leading to appropriate actions in response to these changes.

[0064] (5) The image estimation system 20 receives input of a predetermined date and a predetermined area, and based on the input, acquires first satellite image data and second satellite image data, which are input to the image generation model and whose observation date is within a predetermined period including the predetermined date and whose observation range includes the predetermined area. This makes it possible to automatically obtain estimated satellite image data by specifying a predetermined date and a predetermined area.

[0065] (6) The specified day is set to be a day outside the orbital period of the optical satellite that acquires the third satellite image data, or a day on which the optical satellite acquires an image containing clouds in the third satellite image data. This makes it possible to grasp the state of the specified area on days when there is no third satellite image data.

[0066] [Other embodiments] The image estimation system 20 may include the image generation model generation device 10. In this case, when a new satellite image data set is generated, the image estimation system 20 can update the image generation model 12 used in the image estimation system 20 using the image generation model generation device 10.

[0067] The image generation model is not limited to the generation method described in the above embodiment, and is not particularly limited as long as it is an image generation model that receives input of first satellite image data and second satellite image data whose observation date is within a predetermined period including a predetermined date and whose observation range includes a predetermined area, and outputs estimated optical satellite image data that includes the predetermined area and has a third spatial resolution corresponding to the predetermined date. The image generation model of image estimation system 20 may be, for example, a generation AI.

[0068] In the above embodiment, super-resolution is described in which machine learning is performed using satellite image data and satellite image data with low spatial or temporal resolution is combined to improve both spatial and temporal resolution. However, the satellite image data may include index data calculated from the satellite data (e.g., numerical data indicated by an index of an area specified by latitude and longitude). The index indicated by the index data may be the index exemplified in the above embodiment. In this way, when satellite image data is used as index data, the image generation model generation device 10 may be replaced with an index generation model generation device, and the image estimation system 20 may be replaced with an index estimation system. Furthermore, the index estimation system may estimate index data corresponding to each pixel and generate an image showing the distribution of index data based on the estimated index data for multiple different pixels.

[0069] In addition, although the above embodiment is based on the premise that the satellite orbits the Earth, the satellite may orbit a satellite other than the Earth. In other words, the present invention can be applied to cases where a satellite other than the Earth is the observation target.

[0070] In other embodiments of the present invention, the present invention may be a program that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts, or a computer-readable storage medium that stores the program.In still other embodiments, the present invention may be a method that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts.In still other embodiments, the present invention may be a server that can supply a program that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts to a computer.In still other embodiments, the present invention may be a virtual machine that realizes the functions of the embodiments of the present invention described above and the information processing shown in the flowcharts.

[0071] In the processes or operations described above, the processes or operations can be freely changed as long as no inconsistencies in the processes or operations occur, such as the use of data that should not yet be available in a certain step. Furthermore, the embodiments described above are merely examples for explaining the present invention, and the present invention is not limited to these embodiments. The present invention can be embodied in various forms without departing from the spirit of the invention. [Explanation of symbols]

[0072] 1a processor 1b Storage device 1b-1 RAM 1b-2 ROM 1b-3 External memory 1c Input device 1d output device 1e Communication equipment 1F Bus 10 Image generation model generator 11 Multiple satellite imagery datasets 12 Image generation model 20 Image Estimation System 31 Image shown by the first satellite image data 32 Image shown by the second satellite image data 32' Image unavailable for image estimation 33 Image shown by estimated satellite image data 34 Actual satellite imagery

Claims

1. An image generation model generation device that generates an image generation model by machine learning using satellite image data sets for multiple periods as training data, the satellite image data set includes a set of first satellite image data, second satellite image data, and third satellite image data, the observation dates of which are within a predetermined period and the observation ranges of which include a predetermined area; the first satellite image data is radar satellite image data having a first spatial resolution; the second satellite image data is optical satellite image data having a second spatial resolution lower than the first spatial resolution; the third satellite image data is training data and is optical satellite image data having a third spatial resolution higher than the second spatial resolution; using the first satellite image data and the second satellite image data as input data and the third satellite image data as training data to learn a plurality of satellite image data sets, the machine learning estimates optical satellite image data corresponding to the specified region and a specified day within a specified period and having the third spatial resolution, and generates the image generation model that outputs the estimated optical satellite image data. Image generation model generator.

2. the first spatial resolution is the same as the third spatial resolution; The image generation model generating device according to claim 1 .

3. the predetermined period of the satellite image dataset used as the training data is a period on the order of days or weeks; The image generation model generating device according to claim 1 .

4. An image estimation system including the image generation model according to any one of claims 1 to 3, the image generation model receives input of the first satellite image data and the second satellite image data, the observation date of which is within a predetermined period including a predetermined date and the observation range of which includes a predetermined area, and outputs estimated optical satellite image data having the third spatial resolution corresponding to the predetermined area and the predetermined date. Image estimation system.

5. Accepting input of the predetermined date and the predetermined area; based on the input, acquiring the first satellite image data and the second satellite image data, which are input to the image generation model and whose observation date is within a predetermined period including the predetermined date and whose observation range includes the predetermined area; The image estimation system according to claim 4 .

6. the predetermined day is a day outside the recurrent period of an optical satellite that acquires the third satellite image data, or a day on which an image containing clouds is acquired from the third satellite image data acquired by the optical satellite. The image estimation system according to claim 4 .

Images