Missing satellite data compensation method and system

Abstract

The present invention provides a missing data compensation method comprising: sequentially receiving time-series satellite images; extracting a region of interest on the basis of predetermined coordinates so as to generate a plurality of local images; detecting a missing region in the plurality of local images on the basis of a pre-trained missing data detection model; and performing, on the basis of another plurality of second local images, spatio-temporal interpolation on the missing region detected from one or more first local images from among the plurality of local images, so as to compensate for the missing region.

Description

Satellite data missing supplementation method and system

[0001] The present invention relates to a method and system for supplementing missing satellite data.

[0002] Satellite imagery is utilized in a variety of fields, including the observation of vegetation data and the creation of digital maps and satellite image maps. Depending on the intended use, satellite imagery is utilized in a variety of ways, ranging from low- to mid-resolution satellite images to high-resolution images. For example, satellite imagery is captured by various satellites, including Kompsat-3, Kompsat-3A, Landsat, Sentinel-1, and Sentinel-2.

[0003] Meanwhile, these satellite images may have missing data (or data loss) due to weather conditions or environment in the shooting area, and in order to utilize the satellite images, these missing data need to be supplemented (or corrected).

[0004] Accordingly, a method to compensate for missing data has been proposed by aligning satellite images taken from different satellites or at different times.

[0005] The present invention relates to a method and system for compensating for missing satellite data.

[0006] In addition, the present invention relates to a satellite data missingness supplementation method and system that supplements missingness by considering temporal changes appearing in satellite images.

[0007] In addition, the present invention relates to a satellite data missing supplementation method and system that can accurately supplement missing data that is difficult to supplement depending on circumstances such as season and environment.

[0008] In order to solve the problem discussed above, a missing complementation method according to the present invention may include the steps of sequentially receiving time-series satellite images; extracting a region of interest from each of the satellite images based on predetermined coordinates to generate a plurality of local images; detecting a missing region in the plurality of local images based on a pre-learned missing detection model; and performing spatiotemporal interpolation on a first missing region detected from at least one first local image among the plurality of local images based on a plurality of second local images captured at different times from the at least one first local image among the plurality of local images to complement the first missing region.

[0009] In addition, a missing complement system according to the present invention includes an input unit that sequentially receives time-series satellite images; and a control unit that specifies at least one of the plurality of satellite images and complements a missing area for the specified at least one satellite image based on a plurality of satellite images captured at different times from the specified at least one satellite image among the plurality of satellite images, wherein the control unit extracts a region of interest from each of the satellite images based on a predetermined coordinate to generate a plurality of local images, detects a missing area in the plurality of local images based on a pre-learned missing detection model, and performs spatiotemporal interpolation on a first missing area detected from at least one first local image among the plurality of local images based on a plurality of second local images captured at different times from the at least one first local image among the plurality of local images to complement the first missing area.

[0010] In addition, a program stored in a computer-readable recording medium according to the present invention is a program stored in a computer-readable recording medium, which is executed by one or more processes in an electronic device, and which comprises commands to perform the steps of: sequentially receiving time-series satellite images; extracting a region of interest from each of the satellite images based on a predetermined coordinate to generate a plurality of local images; detecting a missing region in the plurality of local images based on a pre-learned missing detection model; and performing spatiotemporal interpolation on a first missing region detected from at least one first local image among the plurality of local images, based on a plurality of second local images captured at different times from the at least one first local image among the plurality of local images, to supplement the first missing region.

[0011] According to various embodiments of the present invention, a method and system for supplementing missing satellite data detects missing from satellite images and supplements previously detected missing based on time-series satellite data, thereby taking into account temporal changes appearing in the satellite images and supplementing missing more accurately.

[0012] In addition, according to various embodiments of the present invention, the satellite data missing supplementation method and system can accurately supplement missing data that is difficult to supplement depending on the season, environment, etc., by supplementing previously detected missing data based on SAR images when the period of missing data detected from satellite images for a specific region is long.

[0013] Figure 1 illustrates a missing complement system according to the present invention.

[0014] Figure 2 illustrates a missing detection model learning system.

[0015] Figures 3 and 4 illustrate an embodiment of modifying the scale parameter of a missing detection model.

[0016] Figure 5 illustrates an image generation model learning system.

[0017] Figure 6 is a flowchart showing a missing complementation method according to the present invention.

[0018] Figure 7 illustrates an embodiment of generating a local image.

[0019] Figure 8 illustrates an embodiment of specifying a first local image and a second local image.

[0020] Figures 9 and 10 illustrate an embodiment of compensating for missingness according to linear regression.

[0021] Figure 11 is a flowchart showing a method for supplementing missing areas using radar images.

[0022] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0023] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0024] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0025] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0026] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0027] Figure 1 illustrates a missingness compensation system according to the present invention. Figure 2 illustrates a missingness detection model learning system. Figures 3 and 4 illustrate an embodiment of modifying the scale parameters of a missingness detection model. Figure 5 illustrates an image generation model learning system.

[0028] Referring to FIG. 1, a missing area supplementation system (100) according to the present invention detects a missing area from a time series satellite image (10) using a pre-learned missing area detection model (121), and supplements the missing area using a plurality of satellite images (10) taken at different times from the satellite image (10) in which the missing area was detected, thereby generating a satellite image (11) in which the missing area is supplemented.

[0029] Here, the satellite image (10) may be an image captured by an artificial satellite, and may include information in a wavelength band corresponding to visible light and near infrared (NIR) channels. For example, the satellite image (10) may be captured to include R, G, B, and NIR wavelength bands, and may include satellite images (10) such as Sentinel-2 and Landsat, depending on the embodiment.

[0030] In this regard, the missing complement system (100) receives a plurality of satellite images (10) taken over a predetermined period of time for a specific region, specifies one or more of the plurality of satellite images (10) as a satellite image (10) to be used to complement a missing area, and specifies a plurality of satellite images (10) taken at a different time from one or more of the previously specified satellite images (10) among the plurality of satellite images (10) as satellite images (10) to be used to complement a missing area.

[0031] That is, the missing image supplementation system (100) can use multiple satellite images (10) taken in the past (or future) for the same area as the satellite image (10) in order to supplement the missing image of a specific satellite image (10).

[0032] In addition, the missing area is an area where the photographing of the ground surface failed due to environmental factors in the satellite image (10), for example, it may be an area where the ground surface is obscured by clouds and cloud shadows.

[0033] Meanwhile, these missing areas can be detected from satellite images (10) using a missing detection model (121), and thus, based on learning data used in the process of training the missing detection model (121), it can be implemented to detect missingness due to different factors.

[0034] In this regard, referring to FIG. 2, the missing detection model (121) may be learned according to a missing detection model learning method, and the missing detection model learning method may be performed by a missing detection model learning system (200) to be described later.

[0035] The missing detection model learning system (200) according to the present invention receives a learning satellite image (12) and a correct missing area (13), which is labeling data for the learning satellite image (12), and can train a missing detection model (121) using the learning satellite image (12) and the correct missing area (13) to detect a missing area (15) from a predetermined satellite image (14).

[0036] At this time, the learning satellite image (12) may include information of a wavelength band corresponding to visible light and near-infrared light, and may include satellite images such as Kompsat-3 and Kompsat-3A, depending on the embodiment.

[0037] Here, the missing detection model learning system (200) can divide the learning satellite image (12) into a predetermined size (e.g., 1024X1024) to generate a segmented image, and divide the correct missing area (13) to correspond to the segmented image. In this case, the missing detection model (121) can be trained using the segmented image and the correct missing area (13) divided to correspond to the segmented image.

[0038] In addition, the missing detection model learning system (200) can perform data augmentation techniques such as rotation, crop, flip, scale adjustment, and color (or brightness) adjustment on the learning satellite image (12) (or segmented image), and train the missing detection model (121) using the augmented data.

[0039] In addition, the correct answer missing area (13) may represent an area corresponding to a missing area in the learning satellite image (12). For example, the correct answer missing area (13) may include an area corresponding to a cloud and an area corresponding to a cloud shadow in the learning satellite image (12).

[0040] Accordingly, the missing detection model (121) can be trained using the training satellite image (12) and the correct missing area (13) so that when a satellite image (14) is input, the missing area (15) corresponding to the missing area in the input satellite image (14) is output.

[0041] Meanwhile, in one embodiment, the missing detection model (121) may be a Swin (Shifted Windows) Transformer model, and this Swin Transformer model can effectively learn missing areas (15) (e.g., clouds and cloud shadows) that appear in various sizes and thicknesses from a satellite image (14) by dividing the training satellite image (12) into patch units and processing them hierarchically.

[0042] Furthermore, the missing detection model learning system (200) according to the present invention can correct the scale parameter of the missing detection model (121) based on the resolution of a predetermined satellite image (14), which is input data for the missing detection model (121).

[0043] For example, the missing detection model learning system (200) can modify the value of the scale parameter (e.g., the img_scale parameter of the MultiScaleFlipAug component) of the missing detection model (121) based on the difference in size (or resolution) between the learning satellite image (12) used for learning the missing detection model (121) and the satellite image (14) for detecting the missing area (15) using the missing detection model (121).

[0044] Referring to FIG. 3 together, in one embodiment, the missing detection model learning system (200) can confirm that the scale parameter value of the missing detection model (121) learned using Kompsat-3 and Kompsat-3A is (1024, 1024). In this case, the missing detection model learning system (200) can modify the value of the scale parameter to (512, 512) when the satellite image (14) to detect the missing area (15) using the missing detection model (121) is Landsat.

[0045] Referring to FIG. 4, in another embodiment, the missing detection model learning system (200) may use the missing detection model (121) to detect the missing area (15) when the satellite image (14) is Sentinel-2, and may modify the value of the scale parameter to (3072, 3072).

[0046] At this time, the missing detection model learning system (200) may correct the value of the scale parameter to a predetermined value, or may prepare a test data set for a satellite image (14) for detecting a missing area (15) using the missing detection model (121), modify the value of the scale parameter within a predetermined numerical range, calculate accuracy based on the test data set, and determine the value of the scale parameter based on the calculated accuracy.

[0047] Meanwhile, although the missing detection model learning system (200) is described as being implemented as a separate system (or device) from the missing complementation system (100) according to the present invention, depending on the embodiment, the missing detection model learning system (200) and the missing complementation system (100) may be implemented as integrated with each other.

[0048] Furthermore, if it is determined that it is impossible to supplement the missing area using a plurality of other satellite images that are adjacent in time series for a satellite image in which a missing area is detected, the missing area supplementation system (100) receives a radar image corresponding in time series to the satellite image in which a missing area is detected, obtains a simulated image corresponding to the previously received radar image using a pre-learned image generation model, and supplements the missing area of ​​the satellite image using the obtained simulated image.

[0049] Here, the radar image may be a Synthetic Aperture Radar (SAR) image, for example, generated based on the time difference measured by projecting radar toward the Earth's surface (and ocean) from an artificial satellite and receiving the reflected radar. In one embodiment, the training radar image may be a SAR image from Sentinel-1.

[0050] A simulated image is a simulated satellite image. It can be an image created by inputting a radar image into a pre-trained image generation model and estimating a satellite image of the same area from the input radar image.

[0051] In this regard, referring to FIG. 5, the image generation model (123) may be learned according to an image generation model learning method, and the image generation model learning method may be performed by an image generation model learning system (300) to be described later.

[0052] The image generation model learning system (300) according to the present invention receives a learning radar image (16) and a correct satellite image (17), which is labeling data for the learning radar image (16), and can train the image generation model (123) using the learning radar image (16) and the correct satellite image (17) to generate a satellite image corresponding to a predetermined radar image (18) as a simulated image (19).

[0053] At this time, the training radar image (16) may be a SAR image, and may be generated based on the time difference measured by irradiating radar toward the ground surface (and ocean) from an artificial satellite and receiving the reflected radar. In one embodiment, the training radar image (16) may be a SAR image of Sentinel-1.

[0054] In addition, the correct satellite image (17) is a satellite image that is provided to correspond to the learning radar image (16), and may be a satellite image taken within a predetermined time range (e.g., one week) for the same area as the learning radar image (16).

[0055] In one embodiment, the correct satellite image (17) may be a Sentinel-2 image, and may be a satellite image having a 10 m spatial resolution.

[0056] At this time, the correct satellite image (17) may have some differences from the learning radar image (16) with respect to the photographed coordinate range, and in this case, preprocessing such as rotation, cropping, and resizing may be performed on at least one of the learning radar image (16) and the correct satellite image (17) to align the coordinate ranges between the learning radar image (16) and the correct satellite image (17).

[0057] Accordingly, the image generation model (123) can be trained using the training radar image (16) and the correct satellite image (17) to estimate and output a satellite image from the input radar image (18) when the radar image (18) is input.

[0058] In one embodiment, the image generation model learning system (300) is described as being implemented as a separate system (or device) from the missing complement system (100) according to the present invention, but depending on the embodiment, the image generation model learning system (300) and the missing complement system (100) may be implemented as an integrated system.

[0059] Meanwhile, referring again to FIG. 1, the missing complement system (100) according to the present invention may include an input unit (110), a storage unit (120), a control unit (130), and an output unit (140).

[0060] The input unit (110) can receive satellite images (10) and radar images. To this end, the input unit (110) can be connected to a separate server (or device) in which at least one of the satellite images (10) and radar images is stored, via a wireless or wired network.

[0061] At this time, the input unit (110) may be connected to an artificial satellite equipped to capture at least one of a satellite image (10) and a radar image via a wireless network. In this case, the input unit (110) may receive at least one of a satellite image (10) and a radar image from the artificial satellite.

[0062] The storage unit (120) can store data and commands necessary for the operation of the missing complement system (100) according to the present invention.

[0063] For example, the storage unit (120) can store a missing detection model (121), satellite image (10), and information related to missing areas.

[0064] In addition, the storage unit (120) may store information generated in the process of detecting a missing area from a satellite image (10) using a missing detection model (121) and supplementing the detected missing area, as well as a satellite image (11) in which the missing area has been supplemented.

[0065] Additionally, the storage unit (120) can store an image generation model (123) and information related to radar images and simulated images.

[0066] In addition, the storage unit (120) can store information generated in the process of generating a simulated image from a radar image using an image generation model (123) and supplementing a missing area of ​​a satellite image (10) using the simulated image.

[0067] The control unit (130) can control the overall operation of the missing complement system (100) according to the present invention.

[0068] For example, the control unit (130) can detect a missing area from a satellite image (10) using a missing detection model (121), and supplement the previously detected missing area using the satellite image (10) and a plurality of other satellite images (10).

[0069] In addition, if the control unit (130) fails to supplement the previously detected missing area using the other plurality of satellite images (10) described above, it can generate a simulated image from the radar image using the image generation model (123) and supplement the previously detected missing area using the generated simulated image.

[0070] The output unit (140) can output a satellite image (11) with missing information supplemented. To this end, the output unit (140) can be connected to a separate display device wirelessly or via a wired network.

[0071] In addition, the output unit (140) can output various information generated by the control unit (130), and can be connected to a separate server (or device) via a wireless or wired network to transmit a satellite image (11) with missing information supplemented and various information generated by the control unit (130).

[0072] Based on the configuration of the missing complement system (100) discussed above, the missing complement method will be described in more detail below.

[0073] Figure 6 is a flowchart illustrating a missing area supplementation method according to the present invention. Figure 7 illustrates an embodiment of generating a local image. Figure 8 illustrates an embodiment of specifying a first local image and a second local image. Figures 9 and 10 illustrate an embodiment of supplementing missing areas using linear regression. Figure 11 is a flowchart illustrating a method of supplementing missing areas using radar images.

[0074] Referring to FIG. 6, the missing complement system (100) according to the present invention sequentially receives time-series satellite images (S100), and extracts an area of ​​interest from each of a plurality of satellite images based on predetermined coordinates to generate a plurality of local images (S200).

[0075] Specifically, the missing complement system (100) can receive multiple satellite images taken in a time series in a specific area.

[0076] For example, the missing complement system (100) can receive a plurality of satellite images captured over a predetermined period of time and covering a predetermined coordinate range.

[0077] At this time, the predetermined period is a period set to receive multiple satellite images, and may be set to 1 month, 2 months, or 6 months, depending on the embodiment, but the present invention is not limited to these embodiments.

[0078] As another example, the missing complement system (100) may receive a specific satellite image and further receive a plurality of satellite images of the same area taken during a predetermined period of time in the past from the time the received satellite image was taken.

[0079] Furthermore, the missing complement system (100) can create multiple local images from which the same region is extracted by cutting out (or cropping) a region of interest of a predetermined size from each of multiple previously received satellite images based on predetermined coordinates.

[0080] Referring to FIG. 7, for example, the missing complement system (100) can specify coordinates corresponding to a region of interest (22) based on user input, and crop a region of interest (22) of a predetermined size having the specified coordinates as center coordinates (21) from each of a plurality of satellite images (20) to generate a plurality of local images.

[0081] Through this, each of the multiple local images can be extracted to have the same coordinate range.

[0082] Referring again to FIG. 6, the missing complement system (100) according to the present invention detects a missing area in a plurality of local images based on a pre-learned missing detection model (S300), and, for a first missing area detected from at least one first local image among the plurality of local images, performs spatiotemporal interpolation based on a plurality of second local images captured at different times from at least one first local image among the plurality of local images to complement the previously detected first missing area (S400).

[0083] Specifically, the missing complement system (100) can input a plurality of local images into a pre-learned missing detection model, detect missing areas corresponding to the plurality of local images, and mask previously detected missing areas in the plurality of local images.

[0084] For example, the missing complement system (100) can input each of a plurality of local images into a pre-learned missing detection model, and detect a cloud region and a cloud shadow region as missing regions corresponding to each of the plurality of local images.

[0085] Accordingly, the missing complement system (100) can mask the cloud area and cloud shadow area detected in advance of each local image, and at this time, masking can be performed so that the cloud area and the cloud shadow area have different values.

[0086] Furthermore, the missing complement system (100) can specify one or more first local images among a plurality of local images in which the missing area is masked.

[0087] For example, the missing area supplementation system (100) can specify a local image extracted from one or more satellite images that meet preset conditions among a plurality of previously received satellite images as a first local image to supplement the missing area.

[0088] At this time, the preset conditions may be set so that any satellite image is specified, or one or more satellite images captured most recently in a time series are specified, or one or more satellite images may be specified based on the location or size of the missing area detected in each satellite image.

[0089] Referring to FIG. 8, as another example, the missing complement system (100) can specify one of the previously generated multiple local images (23), corresponding to the satellite image most recently captured according to the time series (t), as the first local image (24) to complement the missing area.

[0090] As another example, the missing complement system (100) can sequentially specify each of the plurality of previously generated local images as a first local image.

[0091] At this time, sequentially specifying as the first local image may be to specify one of the plurality of local images as the first local image to supplement the missing area, and then re-specify the next local image of the local image whose missing area has been supplemented as the first local image to supplement the missing area.

[0092] Here, the order of multiple local images can also be specified according to time series.

[0093] As another example, the missing complement system (100) may separate one or more local images from among a plurality of local images, each of which has a masked missing area wider than a predetermined ratio (e.g., 50%), and may specify one of the remaining multiple local images as a first local image.

[0094] In this case, the missing complement system (100) can perform missing complement based on the radar image described below for one or more local images separated according to the ratio of the missing area.

[0095] Furthermore, the missing complement system (100) can specify a plurality of second local images to be used for missing complementation among a plurality of local images captured at different times from one or more first local images specified above.

[0096] Referring again to FIG. 8, for example, the missing complement system (100) can specify, as second local images, a plurality of local images (25) that are captured at a different time point from one or more of the previously specified first local images (24) and in which the missing area masked in each local image is lower than a predetermined ratio (e.g., 50%).

[0097] As another example, the missing complement system (100) may specify, as second local images, a plurality of local images, excluding one or more first local images, in which the missing area masked in the same area as the previously specified first local image is lower than a predetermined ratio (e.g., 50%).

[0098] Furthermore, the missing complement system (100) performs linear regression according to the shooting time and pixel color of each of the plurality of second local images for each pixel (or coordinate) of one or more previously specified first local images, and can complement the first missing region detected in one or more first local images based on the linear regression result.

[0099] Referring to FIG. 9, for example, the missing complement system (100) extracts color values ​​(33) of a plurality of pixels corresponding to the same coordinates (30) among a plurality of coordinates (or pixels) belonging to one or more first local images (24), among a plurality of coordinates (or pixels) belonging to each of a plurality of second local images (26), for one coordinate (30), and performs linear regression by listing the color values ​​(33) of the plurality of pixels previously extracted according to the shooting time point (t) (e.g., time series value) of each of the plurality of second local images (26).

[0100] Referring to FIG. 10, through this, the missing complement system (100) can obtain a linear regression function (34) for the color value (32) of a pixel that changes according to the shooting time (t) at a specific coordinate (30), and based on the obtained linear regression function (34), calculate the color value (42) of the pixel for each coordinate corresponding to the shooting time (41) of one or more first local images (24) specified in advance, thereby compensating for the first missing area.

[0101] At this time, the missing complement system (100) can generate a first local image in which the missing area is complemented by performing spatial interpolation (e.g., Mean Filter) based on surrounding pixels on the first local image in which the missing area is complemented according to the time series, as described above.

[0102] Furthermore, referring to FIG. 11, if the missing complement system (100) fails to specify a plurality of second local images to be used for missing complement of one or more first local images, it receives radar images corresponding to one or more first local images (S410), and can generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model (S420).

[0103] For example, the missing complement system (100) may determine that the number of local images, excluding one or more first local images previously specified, in which the number of missing areas masked in each local image is narrower than a predetermined ratio (e.g., 50%) is less than a predetermined threshold or does not exist, has failed to specify the plurality of second local images.

[0104] Accordingly, the missing complement system (100) can receive radar images captured within a predetermined time range for a time series of one or more first local images, and generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model.

[0105] That is, if the missing complement system (100) fails to specify other multiple local images to complement the missing area of ​​a specific local image among multiple local images generated based on multiple satellite images having a predetermined time range, the system can receive a radar image captured within a predetermined time range from the specific local image and generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model.

[0106] As another example, if the number of local images in which the masked missing area is narrower than a predetermined ratio (e.g., 50%) in the same area as the masked missing area in one or more of the first local images among the plurality of local images excluding the first local images specified above is less than a predetermined threshold or does not exist, it can be determined that the plurality of second local images have failed to be specified.

[0107] Accordingly, the missing complement system (100) can receive radar images captured within a predetermined time range for a time series of one or more first local images, and generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model.

[0108] Furthermore, the missing complement system (100) may receive radar images corresponding to one or more first local images and generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model when the missing area masked in one or more first local images is higher than a predetermined ratio.

[0109] For example, the missing complement system (100) may specify, among a plurality of previously generated local images, one or more local images in which a masked missing area in each local image is wider than a predetermined ratio (e.g., 50%) as a first local image, receive radar images captured within a predetermined time range for a time series of the specified one or more first local images, and generate a simulated image corresponding to the previously received radar image using a pre-learned image generation model.

[0110] Furthermore, the missing complement system (100) can complement missing areas of one or more first local images using previously generated simulated images (S430).

[0111] For example, the missing region supplementation system (100) can supplement the missing region of one or more first local images by extracting pixel values ​​of an area corresponding to a missing region masked in one or more first local images from a previously generated simulated image and inserting the extracted pixel values ​​into one or more first local images.

[0112] At this time, the missing complement system (100) may remove noise from the first local image in which the missing area is complemented by performing post-processing (e.g., Mean Filter) on the first local image in which the missing area is complemented.

[0113] Through the above configurations, the missingness supplementation system (100) according to the present invention detects missingness from satellite images and supplements previously detected missingness based on time series satellite data, thereby taking into account temporal changes appearing in the satellite images and supplementing missingness more accurately.

[0114] In addition, the missingness supplementation system (100) according to the present invention can accurately supplement missingness that is difficult to supplement depending on the season, environment, etc. by supplementing previously detected missingness based on SAR image when the period of missingness detected from satellite image for a specific region is long.

[0115] Furthermore, the present invention discussed above can be implemented as a program executed by one or more processes in an electronic device and stored in a computer-readable recording medium.

[0116] Accordingly, the present invention can be implemented as computer-readable code or instructions on a program-recorded medium. That is, the various control methods according to the present invention can be provided in the form of integrated or individual programs.

[0117] Meanwhile, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0118] Furthermore, the computer-readable medium may include a storage device and may be a server or cloud storage device accessible via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage device via wired or wireless communication.

[0119] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and there is no particular limitation on its type.

[0120] Meanwhile, the detailed description above should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A step of sequentially receiving time series satellite images; A step of generating multiple local images by extracting an area of ​​interest from each of the satellite images based on predetermined coordinates; A step of detecting a missing area in the plurality of local images based on a pre-learned missing detection model; and A missing area supplementation method, comprising a step of supplementing the first missing area by performing spatiotemporal interpolation based on a plurality of second local images captured at different times from the one or more first local images among the plurality of local images, for a first missing area detected from at least one first local image among the plurality of local images.

2. In the first paragraph, the step of supplementing the first missing area is: A step of performing linear regression according to the shooting time and pixel color of each of the plurality of second local images for each pixel of the one or more first local images; and A missing region supplementation method, comprising a step of supplementing the first missing region detected in the one or more first local images based on the linear regression results.

3. In the first paragraph, the step of supplementing the first missing area is: A missingness supplementation method, comprising a step of specifying, as the plurality of second local images, a plurality of local images that are captured at a different time point from the at least one first local image among the plurality of local images and in which the missing area masked in each local image is lower than a predetermined ratio.

4. In the first paragraph, the step of generating the plurality of local images is: A missing part supplementation method for generating multiple local images in which the same region is extracted by cropping a region of interest of a predetermined size from each of the plurality of received satellite images based on the predetermined coordinates.

5. In paragraph 1, The above missing detection model is learned according to the missing detection model learning method, The above missing detection model learning method is, A step of receiving a training satellite image and a missing correct answer area, which is labeling data for the training satellite image; A step of training the missing detection model using the training satellite image and the correct missing area to detect the missing area from a given satellite image; and A missingness supplementation method, comprising a step of correcting a scale parameter of the missingness detection model based on the resolution of the predetermined satellite image, which is input data for the missingness detection model.

6. In paragraph 5, the missing correct answer area is A missing complement method for the above learning satellite image, including an area corresponding to a cloud and an area corresponding to a cloud shadow.

7. In paragraph 1, If it fails to specify the plurality of second local images to be used for missing supplementation of the one or more first local images, the step of generating a simulated image that simulates a satellite image using a radar image and supplementing the first missing area using the simulated image is further included. The step of supplementing the first missing area using the above-mentioned simulated image is as follows: A step of receiving the radar image corresponding to the shooting time of the one or more first local images; A step of generating a simulated image corresponding to the radar image using a pre-learned image generation model; and A missing region supplementation method, comprising a step of supplementing a first missing region of the at least one first local image using the generated simulated image.

8. In paragraph 7, The above image generation model is learned according to the image generation model learning method, The above image generation model learning method is, A step of receiving a training radar image and a correct satellite image, which is labeling data for the training radar image; and A missingness supplementation method, comprising a step of training the image generation model using the training radar image and the correct satellite image to generate a satellite image corresponding to the radar image as the simulated image.

9. An input unit that sequentially receives time-series satellite images; and A control unit is included that specifies at least one of the plurality of satellite images, and complements a missing area for the specified at least one satellite image based on a plurality of satellite images taken at different times from the specified at least one satellite image among the plurality of satellite images. The above control unit, A missing area supplementation system, which extracts a region of interest from each of the satellite images based on predetermined coordinates to generate a plurality of local images, detects a missing area in the plurality of local images based on a pre-learned missing area detection model, and supplements the first missing area by performing spatiotemporal interpolation on a first missing area detected from at least one first local image among the plurality of local images based on a plurality of second local images captured at different times from the at least one first local image among the plurality of local images.

10. A program that is executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of sequentially receiving satellite images of a time series; A step of generating multiple local images by extracting an area of ​​interest from each of the satellite images based on predetermined coordinates; A step of detecting a missing area in the plurality of local images based on a pre-learned missing detection model; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing a step of supplementing the first missing area by performing spatiotemporal interpolation based on a plurality of second local images captured at different times from the one or more first local images among the plurality of local images, for a first missing area detected from at least one first local image among the plurality of local images.

Images