Learning device, utilization device, land coverage classification system, learning method, utilization method, learning program, and utilization program

Abstract

To provide land coverage classification information with relatively high accuracy by combining the analysis using band information of an optical image and an AI base analysis.SOLUTION: A learning device 20 comprises a model generation unit 202 to generate the learned model for inferring the land coverage classification using the data for learning consisting of a reference land coverage map D101 and a simulated satellite image D102: the reference land coverage map is generated based on an image obtained by extracting the variable disturbance location with respect to a time axis from a reference image by analyzing the wavelength information of the reference image captured by a remote sensing method; the simulated satellite image is the image generated based on deterioration processing and the reference image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, an application device, a land cover classification system, a learning method, an application method, a learning program, and an application program.The present disclosure particularly relates to a land cover classification process using remote sensing data. [Background technology]

[0002] Remote sensing is a method for observing land areas over a wide area. One way in which remote sensing data can be used is for land cover classification. Remote sensing data can be used, for example, to check the state of vegetation over a wide area or the progress of urban construction. Examples of remote sensing include methods using aircraft or UAVs (drones), as well as optical satellites or synthetic aperture radar (SAR) satellites.

[0003] Non-Patent Document 1 discloses a technology for detecting vegetation areas using the Normalized Difference Vegetation Index (NDVI) by utilizing wavelength information from optical images. This technology can calculate various indices by combining wavelengths. Therefore, this technology can, as a specific example, detect water areas using the Normalized Difference Water Index (NDWI) and detect urban areas using the Normalized Difference Built-up Index (NDBI).

[0004] Patent Document 1 discloses a technology for generating a land cover situation assessment model by preparing data representing an arbitrary remote sensing image as an input image, generating and preparing data representing a labeled image in which features on the ground surface that appear in the remote sensing image are labeled as a teacher image, and performing machine learning using the prepared data pair. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-046416 [Non-patent literature]

[0006] [Non-Patent Document 1] Masaya Ichihara, "Land cover classification using MODIS satellite images", Kochi University of Technology, 2015 Summary of the Invention [Problem to be solved by the invention]

[0007] Wavelength analysis of optical images, such as the technology disclosed in Non-Patent Document 1, can perform land cover classification with relatively high accuracy when the optical image contains useful band information such as near-infrared light in addition to RGB (Red, Green, Blue). However, if the optical image does not contain such band information, there is a limit to the classification accuracy. Furthermore, even when the optical image contains multiple types of band information, there is a problem in that it is difficult to increase the accuracy of identification indicators for some classification items, such as buildings and roads. Furthermore, there are many land cover classification AI (Artificial Intelligence) processes that use optical satellite images as input, such as the technology disclosed in Patent Document 1. However, this technology has the problem that external disturbance factors such as clouds, shadows, and snow affect the land cover status as noise when capturing it. Furthermore, there are many general learning devices for land cover classification that use RGB images as input, and there are relatively few dedicated devices that can utilize the useful band information described above.

[0008] Therefore, one or more aspects of the present disclosure aim to provide relatively accurate land cover classification information by combining analysis utilizing band information from optical images with AI-based analysis. [Means for solving the problem]

[0009] The learning device according to the present disclosure includes: an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; a model generation unit that generates a trained model for inferring land cover classification of the earth's surface shown in a target satellite image by inputting a target satellite image captured by the target satellite using training data configured from the Equipped with. [Effects of the Invention]

[0010] According to the present disclosure, a model generation unit generates a trained model for inferring land cover classification using training data consisting of a reference land cover map and a simulated satellite image. Here, the reference land cover map is generated based on an image in which disturbance areas that vary over time are extracted from the reference image and the result of inferring land cover classification on the earth's surface shown in the reference image based on a pre-trained model that infers land cover classification. The simulated satellite image is an image generated based on degradation processing and is generated based on the reference image. Furthermore, the trained model is a model that takes disturbance areas into account. Therefore, by using the trained model, analysis is performed using band information from the optical image, and AI-based analysis is performed. Therefore, according to one or more aspects of the present disclosure, when optical satellite images are input, it is possible to provide results that classify the land cover status of the earth's surface from the satellite images with relatively high accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of a land cover classification system 1 according to a first embodiment. [Figure 2] 1 is a block diagram schematically showing the configuration of an input image generating device 10 according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing a schematic configuration of a degradation processing unit 103 according to the first embodiment. [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of a learning device 20 according to a first embodiment. [Figure 5] A diagram explaining a neural network. [Figure 6] FIG. 1 is a block diagram showing a schematic configuration of an inference device 30 according to a first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of the hardware configuration of a learning device 20 according to the first embodiment. [Figure 8] 4 is a flowchart showing the operation of the learning device 20 according to the first embodiment. [Figure 9] 4 is a flowchart showing the operation of the inference device 30 according to the first embodiment. [Figure 10] FIG. 10 is a block diagram schematically showing the configuration of an input image generating device 10b according to a modification of the first embodiment. [Figure 11] FIG. 10 is a block diagram schematically showing the configuration of an input image generating device 10c according to a modification of the first embodiment. [Figure 12] FIG. 10 is a block diagram showing a schematic configuration of an inference device 30b according to a modification of the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of the hardware configuration of a learning device 20 according to a modification of the first embodiment. [Figure 14] FIG. 10 is a block diagram schematically showing the configuration of a land cover classification system 1b according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing a process for generating an estimated land cover map DOUT according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the description of the embodiments and the drawings, the same elements and corresponding elements are given the same reference numerals. The description of elements given the same reference numerals will be omitted or simplified as appropriate. Arrows in the drawings mainly indicate the flow of data or the flow of processing. Furthermore, "unit" may be read as "circuit," "step," "procedure," "process," or "circuitry" as appropriate.

[0013] Embodiment 1 Hereinafter, this embodiment will be described in detail with reference to the drawings.

[0014] ***Configuration Description*** FIG. 1 is a block diagram showing an example of the configuration of a land cover classification system 1 according to the first embodiment. The land cover classification system 1 includes an input image generation device 10 that generates input images for learning, a learning device 20 that functions as a model generation device, and an inference device 30 that functions as an utilization device. Two or more devices included in the land cover classification system 1 may be configured integrally as appropriate. The processing method performed by the inference device 30 is called a utilization method. In the land cover classification system 1, land cover classification is performed on the target satellite image DIN3 using a trained model trained in the learning device 20. Specific examples of land cover labels related to land cover classification include "building," "road," "water body," "forest," "bare land," and "field." A specific example of the trained model is an inference model generated using an open-source pre-trained model. The pre-trained model is a model that infers land cover classification using satellite images and the like as input.

[0015] 2 is a block diagram showing an example of the configuration of the input image generating device 10. The input image generating device 10 includes a wavelength information analyzing unit 101, an inferring unit 102, and a degradation processing unit 103.

[0016] The wavelength information analysis unit 101 analyzes the wavelength information of the multiband optical image to mask disturbance areas that vary over time, such as clouds, shadows, and snow, in the reference image DIN1, thereby generating a disturbance area classification image D103. A specific example of a disturbance area is when there are no clouds in a certain area during a certain time period, and clouds are present in the same area during another time period. The disturbance area classification image D103 is also called a "variable disturbance area classification image."

[0017] The inference unit 102 acquires a pre-trained model from the trained model storage unit 203, and generates a land cover inference result D104 by inputting the reference image DIN1 to the acquired pre-trained model. The input image generating device 10 outputs a reference land cover map D101 that combines the disturbance location classification image D103 and the land cover inference result D104.

[0018] The reference land cover map D101 is generated based on an image extracted from the reference image DIN1 by analyzing the wavelength information of the reference image DIN1 to identify disturbance areas that vary over time, and the results of inferring the land cover classification of the earth's surface shown in the reference image DIN1 based on the reference image DIN1 and a pre-trained model. The reference land cover map D101 is also called an "estimated high-precision land cover map."

[0019] The degradation processing unit 103 receives the reference image DIN1 and the target satellite information DIN2 as inputs and applies degradation processing to the reference image DIN1 to simulate a satellite image, thereby generating a simulated satellite image D102. The simulated satellite image D102 is an image generated based on degradation processing, and is an image generated based on the reference image DIN1.

[0020] The reference image DIN1 is an image captured by a remote sensing technique that captures an image with a higher resolution than the image of the target satellite to which the inference process is applied. Specific examples of the remote sensing technique include aerial photography using a helicopter, a UAV (Unmanned Aerial Vehicle), or an aircraft, as well as satellite imaging that can capture an image with a higher resolution than the image of the target satellite.

[0021] The degradation process is a process for generating an image corresponding to an image captured by a target satellite based on information about the target satellite. Specific examples of the degradation process include degradation processes based on any one of the sensor arrangement of the target satellite, the point spread function (PSF) characteristics of the target satellite, the ground sampling distance (GSD) of the target satellite, the sensitivity performance of the target satellite, the noise characteristics of the target satellite, data compression, and quantization, or a combination thereof.

[0022] An example of degradation processing based on the sensor placement of a target satellite is explained below. Optical satellites are often equipped with multiple sensors that receive different wavelength bands, and composite images are often generated by combining multi-band images obtained from the multiple sensors. In this case, color shifts can occur in the composite image due to the influence of the amount of misalignment in the physical placement of the sensors. Here, color shifts may occur in sub-pixels. When downsampling from the reference image DIN1, it is possible to generate a simulated satellite image D102 that simulates sub-pixel color shifts by shifting the downsampling offset (binning start position) for each band. In other words, in the degradation process, downsampling may be performed on the reference image DIN1 with a different offset for each band.

[0023] An example of degradation processing based on the PSF of a target satellite is described below. In optical satellite images, a PSF is determined as an index of resolution that combines multiple degradation models, taking into account the influence of at least one of the sensor, optical performance, and aperture shape, as well as turbulence and image streaming. Similarly, the reference image DIN1 also has a PSF as an index of resolution that takes into account the influence of at least one of the sensor, optical performance, and turbulence. By using the PSF of the target satellite and the PSF of the reference image and applying the target satellite PSF to the reference image DIN1, degradation processing can be performed to convert the resolution of the reference image DIN1 to a resolution equivalent to that of the target satellite image DIN3. Specific examples of PSF-based degradation processing include multiplying the PSF by the PSF in frequency space or convolving the PSF with the reference image DIN1.

[0024] Furthermore, when checking the PSF for each band, the PSF may differ for each band. For example, aliasing may occur in the multispectral image because the MTF (Modulation Transfer Function) of the multispectral band is higher than the PSF of the panchromatic band. By incorporating such PSF differences into the degradation process, it is possible to simulate aliasing in the simulated satellite image D102.

[0025] An example of degradation processing based on the GSD of a target satellite will be described. The degradation processing unit 103 compares the GSD of the reference image DIN1 with the GSD of the target satellite, and simulates the GSD using a downsampling method such as bilinear interpolation or bicubic interpolation.

[0026] When degradation processing using GSD is performed, it is desirable that the ratio of the GSD of the reference image DIN1 to the GSD of the target satellite be equal to or greater than the high resolution magnification of the land cover classification system 1.

[0027] An example of degradation processing based on the sensitivity performance of the target satellite will be explained. The signal-to-noise ratio (SN ratio) of the target satellite image DIN3 can be estimated from the sensitivity performance of the target satellite and the amount of light from the assumed subject. If we assume that the noise is additive white Gaussian noise, the SN ratio is related to the standard deviation of the Gaussian noise. Therefore, degradation processing is possible by applying noise equivalent to the noise of the target satellite image DIN3 based on the SN ratio.

[0028] Furthermore, when noise characteristics other than additive white Gaussian noise are obtained from the target satellite information DIN2, for example, when information on striped noise and FPN (Fixed Pattern Noise) is obtained, the degradation processing unit 103 may perform degradation processing by adding these to the reference image DIN1.

[0029] Another type of noise caused by other noise factors is noise due to atmospheric influences. An example of degradation processing that simulates the atmosphere will be described. Here, if the remote sensing method is a method of capturing images from within the atmosphere, the reference image DIN1 does not contain noise due to the atmosphere. On the other hand, the satellite image contains noise such as atmospheric-caused image contrast reduction, occlusion, and turbulence. The degradation processing unit 103 may generate image degradation due to the atmosphere based on a publicly available atmospheric model, and generate an image that simulates a satellite image by adding the generated image degradation to the image.

[0030] An example of degradation processing based on data compression of a target satellite will be described. Data compression is performed when downlinking a target satellite image DIN3 to a ground station. If the data compression is lossy, image quality degradation due to data compression occurs. The degradation processing unit 103 can reproduce the degradation due to data compression by applying the same data compression as that of the target satellite to the reference image DIN1.

[0031] 3 is a block diagram showing an example of the configuration of the degradation processing unit 103. The degradation processing unit 103 according to this example performs any one of color shift degradation processing, resolution degradation processing, noise degradation processing, and compression degradation processing, or a combination of these.

[0032] 4 is a block diagram showing an example of the configuration of the learning device 20. The learning device 20 includes a data acquisition unit 201, a model generation unit 202, and a trained model storage unit 203.

[0033] The data acquisition unit 201 acquires learning data. The training data is teacher data composed of a reference land cover map D101 and a simulated satellite image D102. The acquired training data is provided to the model generation unit 202. In the training data, the reference land cover map D101 corresponds to the correct answer to be inferred from the simulated satellite image D102.

[0034] The model generation unit 202 learns a reference land cover map D101 corresponding to the simulated satellite image D102 based on the learning data provided by the data acquisition unit 201. In other words, the model generation unit 202 learns a combination of the simulated satellite image D102 and the reference land cover map D101 shown in the learning data, thereby generating a learned model for inferring an optimal estimated land cover map DOUT corresponding to the target satellite image DIN3. The estimated land cover map DOUT corresponds to an estimated land cover classification map. Thereafter, the model generation unit 202 stores the generated trained model in the trained model storage unit 203. The trained model is a model trained using training data, and is an inference model for inferring the land cover classification of the earth's surface shown by the target satellite image DIN3 using the target satellite image DIN3 as input.

[0035] The learning algorithm used by the model generation unit 202 may be a known algorithm such as supervised learning, unsupervised learning, or reinforcement learning. As an example, a case where a neural network is used as the learning algorithm will be described here.

[0036] When the learning algorithm is supervised learning, the reference land cover map D101 and the simulated satellite image D102 shown in the learning data must be a pair of data containing the same subject. Here, supervised learning refers to a technique in which a learning device learns the features of the learning data by providing a set of input and result data to the learning device as learning data, and then infers a result from the input using the learning result. When the learning algorithm is unsupervised learning, the simulated satellite image D102 and the reference land cover map D101 do not need to include the same subject.

[0037] As a specific example, the model generation unit 202 learns the reference land cover map D101 corresponding to the simulated satellite image D102 by so-called supervised learning according to a neural network model.

[0038] A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer or two or more layers.

[0039] FIG. 5 is a schematic diagram showing an example of a three-layer neural network. As shown in FIG. 5, in a three-layer neural network, when multiple input values ​​are input to input layer X1 through input layer X3, each input value is multiplied by a first weight w11 through w16 (hereinafter also referred to as a first weight W1). Calculated values ​​obtained by multiplying each input value by the first weight w11 through w16 are input to intermediate layer Y1 or intermediate layer Y2. Each calculated value is multiplied by a second weight w21 through w26 (hereinafter also referred to as a second weight W2), and output values ​​obtained by multiplying each calculated value by the second weight w21 through w26 are output from output layer Z1 through output layer Z3. Each output value is determined based on the value of the first weight W1 and the value of the second weight W2.

[0040] In this embodiment, the model generation unit 202 learns a trained model for inferring an optimal reference land cover map D101 corresponding to the simulated satellite image D102 by so-called supervised learning in accordance with training data created based on a combination of the reference land cover map D101 and the simulated satellite image D102 shown in the training data acquired by the data acquisition unit 201. That is, the model generation unit 202 learns the trained model by inputting the simulated satellite image D102 into the input layer and appropriately adjusting the first weight W1 and the second weight W2 so that the result output from the output layer approaches the reference land cover map D101 as the correct answer.

[0041] Returning to FIG. 4, the trained model storage unit 203 stores the trained model provided by the model generation unit 202.

[0042] 6 is a block diagram showing an example of the configuration of the inference device 30. The inference device 30 includes a data acquisition unit 301 and an inference unit 302. The inference device 30 uses the trained model generated by the learning device 20 to infer an estimated land cover map DOUT from the target satellite image DIN3.

[0043] The data acquisition unit 301 has a function of acquiring a target satellite image DIN3. The acquired target satellite image DIN3 is provided to the inference unit 302.

[0044] The target satellite image DIN3 is an image acquired from the target satellite. Note that the target satellite image DIN3 may include images captured by exposing to light rays having wavelengths in the visible light band, as well as images captured by exposing to light rays having wavelengths in the near-infrared light band and far-infrared light band (thermal infrared).

[0045] The inference unit 302 obtains an estimated land cover map DOUT that indicates the land cover classification of the earth's surface shown in the input image based on the learned model and the input image captured by the target satellite. As a specific example, the inference unit 302 infers an estimated land cover map DOUT corresponding to the target subject from the target satellite image DIN3 using the learned model stored in the learned model storage unit 203. In other words, the inference unit 302 inputs the target satellite image DIN3 to the learned model, thereby acquiring an estimated land cover map DOUT corresponding to the disturbance location classification image D301 that is inferred from the disturbance location classification image D301 corresponding to the target satellite image DIN3.

[0046] 7 shows an example of the hardware configuration of learning device 20 according to this embodiment. Learning device 20 is made up of a computer. Learning device 20 may also be made up of multiple computers.

[0047] As shown in the figure, the learning device 20 is a computer equipped with hardware such as a processor 91, a memory 92, an auxiliary storage device 93, an input / output IF (Interface) 94, and a communication device 95. These pieces of hardware are appropriately connected via signal lines 99.

[0048] The processor 91 is an integrated circuit (IC) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 91 include a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). The learning device 20 may include multiple processors that replace the processor 91. The multiple processors share the role of the processor 91.

[0049] The memory 92 is typically a volatile storage device, specifically a random access memory (RAM). The memory 92 is also called a primary storage device or a main memory. Data stored in the memory 92 is saved in the secondary storage device 93 as needed.

[0050] The auxiliary storage device 93 is typically a non-volatile storage device, and specific examples thereof include a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. Data stored in the auxiliary storage device 93 is loaded into the memory 92 as needed. The memory 92 and the auxiliary storage device 93 may be integrated into one unit.

[0051] The input / output IF 94 is a port to which an input device and an output device are connected. A specific example of the input / output IF 94 is a USB (Universal Serial Bus) terminal. Specific examples of the input device are a keyboard and a mouse. A specific example of the output device is a display.

[0052] The communication device 95 is a receiver and a transmitter, and is specifically a communication chip or a network interface card (NIC).

[0053] Each part of the learning device 20 may use the input / output IF 94 and the communication device 95 as appropriate when communicating with other devices.

[0054] The auxiliary storage device 93 stores a learning program. The learning program is a program that causes a computer to realize the functions of each part of the learning device 20. The learning program is loaded into the memory 92 and executed by the processor 91. The functions of each part of the learning device 20 are realized by software.

[0055] Data used when executing the learning program and data obtained by executing the learning program are stored in a storage device as appropriate. Each part of the learning device 20 uses a storage device as appropriate. Specific examples of the storage device include at least one of the memory 92, auxiliary storage device 93, registers in the processor 91, and cache memory in the processor 91. Note that the terms "data" and "information" may have the same meaning. The storage device may be independent of the computer. The functions of the memory 92 and the auxiliary storage device 93 may be realized by other storage devices.

[0056] Any of the programs described herein may be recorded on a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk and a flash memory. Any of the programs described herein may be provided as a program product. The hardware configuration of each of the other devices included in the land cover classification system 1 is similar to the hardware configuration of the learning device 20.

[0057] ***Explanation of Operation*** The operation procedure of the learning device 20 corresponds to a learning method, and the program that realizes the operation of the learning device 20 corresponds to a learning program. The operating procedure of the inference device 30 corresponds to an inference method, and the program that realizes the operation of the inference device 30 corresponds to an inference program.

[0058] 8 is a flowchart showing an example of a learning process performed by the learning device 20. This process will be described with reference to FIG.

[0059] (Step S11) The data acquisition unit 201 acquires learning data via the input image generation device 10. The acquired learning data is provided to the model generation unit 202.

[0060] (Step S12) The model generation unit 202 generates a trained model by learning the reference land cover map D101, which is an output corresponding to the simulated satellite image D102, through so-called supervised learning based on a combination of the reference land cover map D101 and the simulated satellite image D102 shown in the training data.

[0061] (Step S13) The trained model storage unit 203 stores the trained model generated by the model generation unit 202.

[0062] 9 is a flowchart showing an example of processing performed by the inference device 30 to infer an estimated land cover map DOUT corresponding to the target satellite image DIN3. This processing will be explained using FIG.

[0063] (Step S21) The data acquisition unit 301 acquires a target satellite image DIN3. The acquired target satellite image DIN3 is provided to the inference unit 302.

[0064] (Step S22) The inference unit 302 inputs the target satellite image DIN3 into the learned model stored in the learned model storage unit 203, and generates an estimated land cover map DOUT corresponding to the target satellite image DIN3.

[0065] (Step S23) The inference unit 302 outputs the generated estimated land cover map DOUT.

[0066] ***Explanation of the effect of the first embodiment*** As described above, according to the land cover classification system 1 of embodiment 1, when an actual satellite image is input into a trained model for learning a land cover classification model for optical satellite images, it is possible to apply land cover classification with relatively high accuracy. Furthermore, according to embodiment 1, when creating a trained model, it is possible to create a land cover classification AI with relatively high accuracy by using images that extract variable disturbance areas based on the results of analyzing wavelength information.

[0067] ***Other Configurations*** <Variation 1> In the first embodiment, an example in which supervised learning is adopted as the learning algorithm used by the model generation unit 202 has been described, but the learning algorithm is not limited to this example. As a specific example, reinforcement learning, unsupervised learning, or semi-supervised learning may be adopted as the learning algorithm in addition to supervised learning. Furthermore, deep learning that learns the process of extracting features themselves may be used as the learning algorithm used in the model generation unit 202. The model generation unit 202 may also perform machine learning according to other known methods, such as genetic programming, functional logic programming, or support vector machines.

[0068] <Variation 2> The model generation unit 202 may learn a reference land cover map D101 corresponding to each simulated satellite image D102 according to learning data created in multiple land cover classification systems 1, including a certain land cover classification system 1. Note that the model generation unit 202 may acquire learning data from multiple land cover classification systems 1 used in the same area, or may learn a reference land cover map D101 corresponding to each simulated satellite image D102 using learning data collected from multiple land cover classification systems 1 that operate independently in different areas.

[0069] <Variation 3> The model generation unit 202 may add the land cover classification system 1 that collects learning data to the collection targets for learning data during learning, or may remove the land cover classification system 1 from the collection targets for learning data during learning. Furthermore, the model generation unit 202 may apply a trained model that has trained a reference land cover map D101 corresponding to a simulated satellite image D102 for a certain land cover classification system 1 to a land cover classification system 1 other than the certain land cover classification system 1, and update the trained model by re-training the reference land cover map D101 corresponding to the simulated satellite image D102 for the other land cover classification system 1.

[0070] <Variation 4> When generating the reference land cover map D101, the disturbance location classification image D103 may be used to add land cover classification labels such as cloud labels or shadow labels. In other words, the reference land cover map D101 may be the result of inferring the land cover classification of the earth's surface shown by the reference image DIN1 using an image in which land cover classification labels have been added to disturbance locations in the reference image DIN1 and a pre-trained model. In this case, the simulated satellite image D102 may be an image generated by applying degradation processing to the reference image DIN1. Furthermore, by storing the land cover inference result D104 and the disturbance location classification image D103 in two layers, land cover information present under occlusion, such as "clouds" and "urban area," can also be input to the learning device 20. Using such input enables learning of more diverse information, enabling relatively detailed land cover classification. In other words, the reference land cover map D101 may be data composed of an image in which land cover classification labels have been added to disturbance locations in the reference image DIN1, and the result of inferring the land cover classification of the earth's surface shown in the reference image DIN1 using the reference image DIN1 and a pre-trained model. In this case, the simulated satellite image D102 may be an image generated by applying degradation processing to the reference image DIN1.

[0071] <Variation 5> When generating the reference land cover map D101, the disturbance location classification image D103 may be used as input to the inference unit 102 in order to remove occlusion elements. The disturbance location classification image D103 is also called a disturbance location-removed image. In other words, in this modification, the reference land cover map D101 may be the result of inferring the land cover classification of the earth's surface indicated by the reference image DIN1 using a disturbance location-removed image in which disturbance locations have been removed from the reference image DIN1 and a pre-trained model. In this case, the simulated satellite image D102 may be an image generated by applying degradation processing to the disturbance location-removed image. 10 is a block diagram showing an example of the configuration of an input image generation device 10b according to this modification. An inference unit 102 according to this modification infers a reference land cover map D101 using a disturbance location classification image D103 as input. According to this modification, during inference by the inference unit 102, the reference land cover map D101 can be generated with relatively high accuracy without erroneously detecting occlusion elements.

[0072] <Variation 6> When a target satellite image DIN3 that observes the same area as the observation range of the reference image DIN1 can be obtained, learning data consisting of the target satellite image DIN3 and the reference land cover map D101 can be input into the learning device 20 to train a learned model. 11 is a block diagram showing an example of the configuration of an input image generation device 10c according to this modification. Compared to the input image generation device 10, the input image generation device 10c further includes a position adjustment unit 104. The alignment unit 104 performs alignment processing as necessary to generate an aligned target satellite image D105. The alignment processing is processing for aligning the reference image DIN1 and the target satellite image DIN3 when generating learning data. The aligned target satellite image D105 corresponds to the simulated satellite image D102. In other words, the simulated satellite image D102 according to this modification is an image obtained by performing alignment on the observation area image based on the observation area. The observation area image is an image showing the observation area indicated by the reference image DIN1, and is an image captured by the target satellite. Note that the alignment processing corresponds to degradation processing. It is desirable that the observation period of the target satellite image DIN3 be close to that of the reference image DIN1, but this is not always the case in areas where there is little change in land cover.

[0073] <Variation 7> 12 is a block diagram showing an example of the configuration of an inference device 30 according to this modification. Compared to the inference device 30, the inference device 30b further includes a wavelength information analysis unit 303. The wavelength information analysis unit 303 generates a disturbance location classification image D301 by analyzing the wavelength information of the target satellite image DIN3. At this time, the wavelength information analysis unit 303 may exclude occlusion elements from the target satellite image DIN3, and may classify variable disturbance locations in the target satellite image DIN3 as cloud labels or shadow labels. The generated disturbance location classification image D301 is input to the inference unit 302. The estimated land cover map DOUT in this variant may be the result of inferring the land cover classification of the earth's surface shown in the input image by inputting an image in which land cover classification labels have been added to disturbance points in the input image into a trained model.

[0074] <Variation 8> FIG. 13 shows an example of the hardware configuration of a learning device 20 according to this modification. The learning device 20 includes a processing circuit 98 instead of the processor 91 , the processor 91 and memory 92 , the processor 91 and auxiliary storage device 93 , or the processor 91 , memory 92 , and auxiliary storage device 93 . The processing circuitry 98 is hardware that realizes at least a part of the components of the learning device 20 . The processing circuitry 98 may be dedicated hardware, or may be a processor that executes a program stored in the memory 92 .

[0075] When processing circuitry 98 is dedicated hardware, processing circuitry 98 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The learning device 20 may include a plurality of processing circuits that replace the processing circuit 98. The plurality of processing circuits share the role of the processing circuit 98.

[0076] In the learning device 20, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0077] Processing circuitry 98 is illustratively implemented in hardware, software, firmware, or a combination thereof. The processor 91, memory 92, auxiliary storage device 93, and processing circuit 98 are collectively referred to as "processing circuitry." In other words, the functions of each functional component of learning device 20 are realized by the processing circuitry. The learning device 20 according to other embodiments may also have the same configuration as that of this modified example. Furthermore, the other devices included in the land cover classification system 1 may also have the same configuration as that of this modified example.

[0078] Embodiment 2 The following mainly describes the differences from the above-described embodiment with reference to the drawings.

[0079] ***Configuration Description*** 14 is a block diagram schematically illustrating an example of the configuration of a land cover classification system 1b according to Embodiment 2. The configuration of the land cover classification system 1b is the same as that of Embodiment 1, except that it includes a wavelength information analysis device 40.

[0080] The wavelength information analysis device 40 generates an analyzed land cover result D401 by analyzing land cover based on the wavelength information of the target satellite image DIN3. The analyzed land cover result D401 is also called a wavelength information analyzed land cover result.

[0081] The estimated land cover map DOUT according to the second embodiment is data generated based on the results of inferring the land cover classification of the earth's surface shown in the input image by inputting an image in which land cover classification labels have been added to disturbance locations in the input image into the trained model, and the results of analyzing the land cover classification of the earth's surface shown in the input image by analyzing the wavelength information in the input image.

[0082] ***Explanation of Operation*** FIG. 15 is a flow diagram showing the process of generating an estimated land cover map DOUT by combining the land cover classification results generated by the inference device 30b and the wavelength information analysis device 40.

[0083] The wavelength information analyzer 40 generates an analyzed land cover result D401 from the target satellite image DIN3 using an index that allows land cover classification with a relatively high degree of accuracy, such as NDVI.

[0084] In the inference device 30b, a disturbance location classification image D301 is generated from the target satellite image DIN3, and the disturbance location classification image D301 is input to the learned model stored in the learned model memory unit 203, and an inferred land cover result D302 corresponding to the input target satellite image DIN3 is obtained. The estimated land cover map DOUT is a combination of the inferred land cover results D302 and the analyzed land cover results D401.

[0085] ***Explanation of the effect of the second embodiment*** As described above, in this embodiment, even when the inference device 30b uses a trained model (such as for building extraction or road extraction) that is specialized for only specific features, the inference results from the inference device 30b and the analysis results from the wavelength information analyzer 40 are used together to generate the estimated land cover map DOUT. Here, in the process of generating the estimated land cover map DOUT, each device can be said to share the responsibility of classifying features according to the characteristics of each device. Therefore, by utilizing this embodiment, it is possible to provide a land cover classification map with relatively high accuracy.

[0086] ***Other embodiments*** The above-described embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted. Furthermore, the embodiments are not limited to those shown in Embodiments 1 and 2, and various modifications are possible as necessary. The procedures explained using flowcharts and the like may be modified as appropriate.

[0087] Various aspects of the present disclosure are summarized below as appendices.

[0088] (Appendix 1) an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; a model generation unit that generates a trained model for inferring land cover classification of the earth's surface shown in a target satellite image by inputting a target satellite image captured by the target satellite using training data configured from the A learning device comprising:

[0089] (Appendix 2) the reference land cover map is a result of inferring a land cover classification of the earth's surface shown in the reference image using a disturbance-area-removed image obtained by excluding the disturbance area from the reference image and the pre-trained model; and 2. The learning device according to claim 1, wherein the simulated satellite image is an image generated by applying the degradation processing to the image from which the disturbance portion has been removed.

[0090] (Appendix 3) the reference land cover map is a result of inferring a land cover classification of the earth's surface shown in the reference image using an image in which a land cover classification label is added to the disturbance location in the reference image and the pre-trained model; and 2. The learning device according to claim 1, wherein the simulated satellite image is an image generated by applying the degradation processing to the reference image.

[0091] (Appendix 4) the reference land cover map is data composed of an image in which a land cover classification label is added to the disturbance location in the reference image, and a result of inferring a land cover classification of the earth's surface shown in the reference image using the reference image and the pre-trained model; 2. The learning device according to claim 1, wherein the simulated satellite image is an image generated by applying the degradation processing to the reference image.

[0092] (Appendix 5) an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; and an inference unit that obtains an estimated land cover map indicating the land cover classification of the earth's surface shown in the input image based on the trained model, which is a model for inferring the land cover classification of the earth's surface shown in the target satellite image, using a target satellite image taken by the target satellite as an input, and the input image taken by the target satellite. A utilization device equipped with the above.

[0093] (Appendix 6) The utilization device described in Appendix 5, wherein the estimated land cover map is the result of inferring the land cover classification of the earth's surface shown in the input image by inputting an image in which land cover classification labels have been added to the disturbance locations in the input image into the trained model.

[0094] (Appendix 7) The estimated land cover map includes: a result of inferring a land cover classification of the earth's surface shown in the input image by inputting an image in which a land cover classification label has been added to the disturbance location in the input image into the trained model; and an analysis result obtained by analyzing wavelength information in the input image to analyze the land cover classification of the ground surface shown in the input image; 6. The utilization device according to claim 5, wherein the data is generated based on the data.

[0095] (Appendix 8) A land cover classification system comprising the learning device according to any one of Supplementary Notes 1 to 4 and the utilization device according to any one of Supplementary Notes 5 to 7, A land cover classification system in which the reference image is either an image taken by an aerial photography means or an image taken by a satellite that takes images having a higher resolution than the resolution of the images taken by the target satellite.

[0096] (Appendix 9) 9. The land cover classification system according to claim 8, wherein the degradation process is based on a combination of one or more of the sensor arrangement of the target satellite, the point spread function (PSF) characteristics of the target satellite, the ground sampling distance (GSD) of the target satellite, the sensitivity performance of the target satellite, the noise characteristics of the target satellite, data compression, and quantization.

[0097] (Appendix 10) 10. The land cover classification system of claim 9, wherein the degradation process involves downsampling the reference image with a different offset for each band.

[0098] (Appendix 11) 11. The land cover classification system according to any one of appendices 8 to 10, wherein the simulated satellite image is an image showing the observation area indicated by the reference image, and is an image that has been aligned with an observation area image, which is an image captured by the target satellite, using the observation area as a reference.

[0099] (Appendix 12) 12. The land cover classification system according to any one of claims 8 to 11, wherein the target satellite image is an image captured by exposing to light having a wavelength in the visible light band.

[0100] (Appendix 13) 13. The land cover classification system according to any one of appendices 8 to 12, wherein the target satellite image is an image captured by exposing it to light having a wavelength in the infrared light band. [Explanation of symbols]

[0101] 1,1b Land cover classification system, 10,10b,10c Input image generation device, 101 Wavelength information analysis unit, 102 inference unit, 103 degradation processing unit, 104 alignment unit, 20 Learning device, 201 data acquisition unit, 202 model generation unit, 203 learned model memory unit, 30, 30b inference device, 301 data acquisition unit, 302 inference unit, 303 wavelength information analysis unit, 40 wavelength information analysis device, 91 processor, 92 memory, 93 auxiliary storage device, 94 input / output IF, 95 communication device, 98 processing circuit, 99 signal line, D101 reference land cover map, D102 simulated satellite image, D103, D301 disturbance location classification image, D104 land cover inference result, D105 aligned target satellite image, D302 inferred land cover result, D401 analyzed land cover result, DIN1 reference image, DIN2 target satellite information, DIN3 target satellite image, DOUT estimated land cover map.

Claims

1. an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; a model generation unit that generates a trained model for inferring land cover classification of the earth's surface shown in a target satellite image by inputting a target satellite image captured by the target satellite using training data configured from the A learning device comprising:

2. the reference land cover map is a result of inferring a land cover classification of the earth's surface shown in the reference image using a disturbance-area-removed image obtained by excluding the disturbance area from the reference image and the pre-trained model; and The learning device according to claim 1 , wherein the simulated satellite image is an image generated by applying the degradation processing to the disturbance portion removed image.

3. the reference land cover map is a result of inferring a land cover classification of the earth's surface shown in the reference image using an image in which a land cover classification label is added to the disturbance location in the reference image and the pre-trained model; and The learning device according to claim 1 , wherein the simulated satellite image is an image generated by applying the degradation processing to the reference image.

4. the reference land cover map is data composed of an image in which a land cover classification label is added to the disturbance location in the reference image, and a result of inferring a land cover classification of the earth's surface shown in the reference image using the reference image and the pre-trained model; The learning device according to claim 1 , wherein the simulated satellite image is an image generated by applying the degradation processing to the reference image.

5. an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; and an inference unit that obtains an estimated land cover map indicating the land cover classification of the earth's surface shown in the input image based on the trained model, which is a model for inferring the land cover classification of the earth's surface shown in the target satellite image, using a target satellite image taken by the target satellite as an input, and the input image taken by the target satellite. A utilization device equipped with the above.

6. The utilization device described in claim 5, wherein the estimated land cover map is the result of inferring the land cover classification of the earth's surface shown in the input image by inputting an image in which land cover classification labels have been added to the disturbance locations in the input image into the trained model.

7. The estimated land cover map includes: a result of inferring a land cover classification of the earth's surface shown in the input image by inputting an image in which a land cover classification label has been added to the disturbance location in the input image into the trained model; and an analysis result obtained by analyzing wavelength information in the input image to analyze the land cover classification of the ground surface shown in the input image; The utilization device according to claim 5, wherein the data is generated based on the above.

8. A land cover classification system comprising the learning device according to any one of claims 1 to 4 and the utilization device according to any one of claims 5 to 7, A land cover classification system in which the reference image is either an image taken by an aerial photography means or an image taken by a satellite that takes images having a higher resolution than the resolution of the images taken by the target satellite.

9. 9. The land cover classification system according to claim 8, wherein the degradation process is based on a combination of one or more of the following: a sensor arrangement of the target satellite; a point spread function (PSF) characteristic of the target satellite; a ground sampling distance (GSD) of the target satellite; a sensitivity performance of the target satellite; a noise characteristic of the target satellite; data compression; and quantization.

10. The land cover classification system according to claim 9 , wherein the degradation process involves downsampling the reference image with a different offset for each band.

11. 9. The land cover classification system according to claim 8, wherein the simulated satellite image is an image showing the observation area indicated by the reference image, and is an image that has been aligned with the observation area image, which is an image captured by the target satellite, using the observation area as a reference.

12. The land cover classification system according to claim 8 , wherein the target satellite image is an image captured by exposing it to light having a wavelength in the visible light band.

13. The land cover classification system according to claim 8 , wherein the target satellite image is an image captured by exposing it to light having a wavelength in the infrared light band.

14. The computer an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; A learning method for generating a trained model for inferring land cover classification of the earth's surface shown in a target satellite image by inputting a target satellite image taken by the target satellite, using learning data consisting of the above.

15. The computer an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; and a utilization method for obtaining an estimated land cover map showing the land cover classification of the earth's surface shown by the input image based on the trained model, which is a model for inferring the land cover classification of the earth's surface shown by the target satellite image, using a target satellite image taken by the target satellite as input, and the input image taken by the target satellite.

16. an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; A model generation process for generating a trained model for inferring land cover classification on the earth's surface shown in a target satellite image by inputting a target satellite image captured by the target satellite using learning data composed of the above. A learning program that executes the above on a learning device, which is a computer.

17. an image obtained by extracting a disturbance location that varies with time from a reference image captured by a remote sensing technique by analyzing wavelength information of the reference image; and a result of inferring a land cover classification of the earth's surface shown in the reference image based on the reference image and a pre-trained model for inferring land cover classification; a reference land cover map generated based on a simulated satellite image, which is an image generated based on the reference image and is generated based on a degradation process that generates an image corresponding to an image captured by the target satellite based on information about the target satellite; and an inference process for obtaining an estimated land cover map showing the land cover classification of the earth's surface shown in the input image based on the trained model, which is a model for inferring the land cover classification of the earth's surface shown in the target satellite image, using a target satellite image taken by the target satellite as input, and the input image taken by the target satellite. A utilization program that causes a utilization device, which is a computer, to execute the above.

Images