Method and system for improving resolution on basis of composite of satellite images

Abstract

The present invention relates to a method and a system for combining images captured by different types of satellites. The present invention can provide a resolution improvement method for combining a first satellite image captured at a first spatial resolution and a second satellite image captured at a second spatial resolution, comprising a step of generating a composite image by inputting the first and second satellite images into a deep learning model, wherein the first spatial resolution is higher than the second spatial resolution, a first radiation resolution of the first satellite image is lower than a second radiation resolution of the second satellite image, and the composite image is formed at the first spatial resolution and the second radiation resolution.

Description

Resolution improvement method and system based on the synthesis of satellite images

[0001] The present invention relates to a method and system for improving resolution based on the synthesis of satellite images, and more particularly, to a method and system for providing satellite images with improved resolution by synthesizing satellite images taken from different satellites.

[0002] Satellites are being developed in various sizes, depending on their intended use and intended use. More specifically, satellites can be categorized by size and weight into large satellites, medium satellites, mini satellites, micro satellites, nano satellites, pico satellites, and femto satellites. In addition to the above terms, the terms CANSAT and CUBESAT are also often used.

[0003] Recently, the development of small satellites or micro-satellites with short development periods and low manufacturing costs has been attracting attention.

[0004] These small satellites (or microsatellites) can be primarily used to monitor wide areas through constellation operations. Small satellites offer high spatial resolution and photograph the same area more frequently than medium- and large-sized satellites, but their ability to distinguish color differences is inferior to that of medium- and large-sized satellites.

[0005] In contrast, medium and large satellites have a longer period of time to image the same area than small satellites, but their ability to distinguish color differences is better than that of small satellites.

[0006] As illustrated in Figure 1, when different types of satellites (11 and 12) capture images of a specific area, the quality of the images may vary depending on the performance of the satellites. The present invention proposes a method and a system for synthesizing satellite images to provide satellite images with improved resolution by incorporating the strengths of images captured from different types of satellites.

[0007] The present invention relates to a method and system for improving resolution based on the synthesis of satellite images, which synthesize satellite images taken from different satellites to provide satellite images with improved resolution.

[0008] In addition, the present invention relates to a method and system for improving resolution based on the synthesis of satellite images, which synthesizes satellite images having different spatial and radial resolutions to provide satellite images having higher spatial and radial resolutions.

[0009] In order to solve the problem discussed above, a resolution improvement method according to the present invention is a resolution improvement method for synthesizing a first satellite image captured with a first spatial resolution and a second satellite image captured with a second spatial resolution, the method including a step of inputting the first and second satellite images into a deep learning model to generate a composite image, wherein the first spatial resolution is higher than the second spatial resolution, the first radial resolution of the first satellite image is lower than the second radial resolution of the second satellite image, and the composite image is formed of the first spatial resolution and the second radial resolution.

[0010] In addition, a resolution improvement system according to the present invention may include a communication unit configured to receive a first satellite image captured with a first spatial resolution and a second satellite image captured with a second spatial resolution; and a control unit configured to input the first and second satellite images into a deep learning model to generate a synthetic image, wherein the first spatial resolution is higher than the second spatial resolution, the first radial resolution of the first satellite image is lower than the second radial resolution of the second satellite image, and the synthetic image is formed of the first spatial resolution and the second radial resolution.

[0011] According to various embodiments of the present invention, a method and system for improving resolution based on the synthesis of satellite images can synthesize satellite images having different resolutions to obtain a composite image having a higher resolution among the satellite images.

[0012] That is, the resolution improvement method and system based on the synthesis of satellite images according to the present invention can provide a composite image with improved resolution so as to have the highest spatial resolution and the highest radial resolution simultaneously by synthesizing a plurality of satellite images having different spatial resolutions and radial resolutions.

[0013] In addition, according to various embodiments of the present invention, a method and system for improving resolution based on the synthesis of satellite images can, in the process of improving the resolution of a satellite image, provide a synthetic image with improved resolution based on a satellite image having a relatively high temporal resolution, such that the synthetic image has the same cycle as the shooting cycle of the corresponding satellite image.

[0014] In addition, according to various embodiments of the present invention, a method and system for improving resolution based on the synthesis of satellite images can provide information according to various wavelengths at high resolution by considering different wavelengths in the process of improving the resolution of satellite images.

[0015] Figure 1 is a conceptual diagram showing different satellites photographing the same area.

[0016] Figure 2 is a conceptual diagram showing a resolution improvement system according to the present invention.

[0017] Figure 3 is a flowchart showing a resolution improvement method according to the present invention.

[0018] Figure 4 is a conceptual diagram showing the result of improving the resolution of satellite images according to the present invention.

[0019] Figure 5 is a conceptual diagram showing the difference between images used in deep learning model training and images used in image synthesis.

[0020] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0021] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0022] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0023] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0024] In this specification, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0025] In this specification, “spatial resolution” refers to the minimum unit pixel size of a target that can be observed by a sensor in satellite remote sensing. The smaller the spatial resolution value, the higher the spatial resolution. For example, if the spatial resolution of a specific satellite is 3.3 x 3.3 m, it can be said that the specific satellite has a higher spatial resolution than a satellite with a spatial resolution of 10 x 10 m.

[0026] In this specification, "radiometric resolution" refers to how small differences in signal size can be distinguished in satellite remote sensing. Specifically, the higher the radiometric resolution, the more detailed the color range of data can be recorded when radiant energy is recorded by the sensor.

[0027] In this specification, "temporal resolution" refers to how frequently image data can be acquired for a specific area using satellite remote sensing. The more frequently a region is visited, the higher the temporal resolution.

[0028] Below, a satellite resolution improvement system according to the present invention is described in detail.

[0029] Figure 2 is a conceptual diagram showing a resolution improvement system according to the present invention.

[0030] The satellite resolution improvement system (100) according to the present invention may include a communication unit (110), a storage unit (120), and a control unit (130). However, the location provision device (100) according to the present invention is not limited to the above components, and any one of the above components may be omitted, or other components may be further included.

[0031] The communication unit (110) can receive data from any one of the plurality of satellites (11 and 12). Here, the data can include image information captured from any one of the plurality of satellites (11 and 12).

[0032] Additionally, the communication unit (110) can receive images captured from each of the plurality of satellites (11 and 12).

[0033] Meanwhile, the storage unit (120) may also be called a database (DB) and may store image information received from a satellite, location information of the satellite, synthesized image information, and learning data for deep learning.

[0034] Meanwhile, the control unit (130) can control the overall operation of the satellite resolution improvement system (100) according to the present invention.

[0035] The control unit (130) can control the communication unit to receive images of a specific region from each of a plurality of satellites.

[0036] Meanwhile, the control unit (130) may include a deep learning model for image synthesis, and may train the deep learning model using training data stored in the storage unit (120). In this specification, image synthesis is described as being performed by the control unit (130) or the deep learning model, and both expressions described above may be interpreted as image synthesis being performed by the control unit (130).

[0037] Above, the roles of the components constituting the resolution improvement system (100) according to the present invention have been described.

[0038] Hereinafter, the resolution improvement method (hereinafter referred to as “resolution improvement method”) of the resolution improvement system (100) according to the present invention will be specifically examined with reference to the drawings.

[0039] FIG. 3 is a flowchart showing a resolution improvement method according to the present invention, FIG. 4 is a conceptual diagram showing a result of resolution improvement of satellite images according to the present invention, and FIG. 5 is a conceptual diagram showing the difference between an image used in deep learning model training and an image used in image synthesis.

[0040] Referring to FIG. 3, a step (S110) of receiving a first satellite image captured with a first spatial resolution and a step (S120) of receiving a second satellite image captured with a second spatial resolution are performed.

[0041] S110 and S120 may be performed at different times, and the chronological order of each step is not specifically limited. Furthermore, since the first and second satellite images may utilize previously stored images, S110 and S120 are not essential components of the present invention.

[0042] The above first satellite image is an image captured from the first satellite, and the second satellite image is an image captured from the second satellite.

[0043] The first and second satellites are different types of satellites. The first satellite has a higher spatial resolution than the second satellite, and the first spatial resolution is higher than the second spatial resolution. For example, the first spatial resolution may be 3.3 x 3.3 m, and the second spatial resolution may be 13.2 x 13.2 m.

[0044] Meanwhile, the first satellite has a lower radial resolution than the second satellite, and the first radial resolution of the first satellite image is lower than the second radial resolution of the second satellite image.

[0045] That is, the first satellite can distinguish objects more precisely than the second satellite, but its ability to distinguish colors in detail is relatively low. In one embodiment, the first satellite may be a Dove satellite, and the second satellite may be a Sentinel 2 satellite, but this is not limited thereto.

[0046] Meanwhile, the photographing cycle of the first satellite for a specific point is shorter than the photographing cycle of the second satellite for the specific point. In other words, the temporal resolution of the first satellite is higher than that of the second satellite.

[0047] Next, a step (S130) is performed to input the first and second satellite images into a deep learning model to generate a synthetic image.

[0048] The above composite image may be composed of a first spatial resolution and a second radial resolution. That is, the composite image is an image that includes both the high spatial resolution of the image captured from the first satellite and the high radial resolution of the image captured from the second satellite.

[0049] For example, referring to FIG. 4, the deep learning model receives a first satellite image (410) and a second satellite image (420) as input, and then outputs a composite image (430). The composite image (430) has the spatial resolution of the first satellite image (410) and the radial resolution of the second satellite image (420).

[0050] Meanwhile, a second satellite image of a specific point captured by a second satellite at a specific point in time can be synthesized with each of a plurality of satellite images of the specific point captured by the first satellite during the second satellite's imaging cycle. That is, an image captured by a second satellite with a relatively low temporal resolution can be synthesized with each of a plurality of images captured by a first satellite with a relatively high temporal resolution.

[0051] As described above, according to the present invention, by synthesizing different types of satellite images, an image with improved resolution can be obtained that incorporates the strengths of each satellite. Specifically, the resolution improvement method according to the present invention provides a composite image with high spatial resolution and high radial resolution by synthesizing an image with high spatial resolution and low radial resolution and an image with low spatial resolution and high radial resolution.

[0052] In addition, according to the present invention, by utilizing an image captured from a satellite having a low temporal resolution for synthesis with a plurality of images captured from a satellite having a high temporal resolution, an image having a high spatial resolution and a high radial resolution can be obtained at the same cycle as the image capturing cycle of the satellite having a high temporal resolution.

[0053] Meanwhile, the resolution improvement method according to the present invention further includes a step of training the deep learning model. Below, the training method for the aforementioned deep learning model is described.

[0054] The step of training the deep learning model includes the step of downsampling a first sample image having a first spatial resolution to generate a first learning image having a spatial resolution lower than the first spatial resolution, the step of downsampling a second sample image having a second spatial resolution to generate a second learning image having a spatial resolution lower than the second spatial resolution, and the step of training a deep learning model that synthesizes two images into one image using at least one of the first and second learning images and the first and second sample images.

[0055] Here, the first sample image is an image captured from the first satellite, and the second sample image is an image captured from the second satellite. However, in the present invention, the spatial resolution of the training image used when training the deep learning model and the spatial resolution of the image used during image synthesis are different.

[0056] Specifically, a first sample image having a first spatial resolution is converted into a first training image having a spatial resolution lower than the first spatial resolution through downsampling.

[0057] For example, referring to FIG. 5, if the spatial resolution of the first sample image (Dove) is 3.3X3.3m (or expressed as 3.3m), the spatial resolution of the first learning image may be 13.2X13.2m (or expressed as 13.2m).

[0058] Here, the spatial resolution of the first learning image may be the same as the spatial resolution of the second sample image.

[0059] The second sample image having the second spatial resolution is converted into a second learning image having a spatial resolution lower than the second spatial resolution through downsampling.

[0060] For example, referring to FIG. 5, if the second spatial resolution of the second sample image (Sentinel 2) is 13.2X13.2m (or expressed as 13.2m), the spatial resolution of the second learning image may be 52.8X52.8m (or expressed as 52.8m).

[0061] When training a deep learning model, the first and second training images are input into the deep learning model to generate a synthetic image. The generated synthetic image is then compared to the correct image to determine the error. The deep learning model can be run until the calculated error falls below a threshold value.

[0062] Here, the correct image may be a second sample image. The spatial resolution of the correct image is the second spatial resolution, which may be identical to the spatial resolution of the first training image, which is a downsampled version of the first sample image.

[0063] A first satellite image having a first spatial resolution and a second satellite image having a second spatial resolution are input to the trained deep learning model, and a synthetic image having the first spatial resolution is generated.

[0064] Here, the first satellite image used in the step of generating the synthetic image and the first learning image used in the step of training the deep learning model correspond to each other, and the second satellite image used in the step of generating the synthetic image and the second learning image used in the step of training the deep learning model correspond to each other.

[0065] The spatial resolutions of the corresponding first satellite image and the corresponding first learning image are different from each other, and the spatial resolutions of the corresponding second satellite image and the corresponding second learning image are different from each other. In other words, in the present invention, the spatial resolution of the image used for training the deep learning model is different from the spatial resolution of the image used for image synthesis.

[0066] For example, referring to FIG. 5, the spatial resolutions of the first training image (Dove, spatial resolution: 13.2 m) used in the deep learning model training phase and the first satellite image (Dove, spatial resolution: 3.3 m) corresponding to the first training image used in the image synthesis (Testing Phase) are different from each other, and the spatial resolutions of the second training image (Sentinel 2, spatial resolution: 52.8 m) used in the deep learning model training phase and the second satellite image (Sentinel 2, spatial resolution: 13.2 m) corresponding to the second training image used in the image synthesis (Testing Phase) are different from each other.

[0067] As described above, the present invention provides a deep learning model that increases resolution (e.g., quadruples spatial resolution) through the synthesis of satellite images. In order to train such a deep learning model, an image with a higher resolution than the image actually used for synthesis is required. However, the present invention downsamples the image actually used for synthesis and uses it for training the deep learning model, thereby enabling training of a deep learning model that increases resolution even without an image with a higher resolution than the image actually used for synthesis. Meanwhile, the first and second satellite images used for synthesis may include pixel values ​​of different wavelengths.

[0068] Specifically, each of the plurality of pixels constituting the first satellite image may include a pixel value corresponding to a first wavelength band, each of the plurality of pixels constituting the second satellite image may include a pixel value corresponding to the first wavelength band and a second wavelength band different from the first wavelength band, and each of the pixels constituting the composite image may include a pixel value corresponding to the first and second wavelength bands.

[0069] In one embodiment, the first wavelength band may include at least one of a visible light region and a near-infrared region, and the second wavelength band may include at least one of a red edge region and a shortwave infrared (SWIR) region.

[0070] The composite image may include pixel values ​​corresponding to a wider range of wavelengths than the pixel values ​​of the pixels included in the first satellite image, and may include pixel values ​​in a wavelength range corresponding to the second satellite image. In other words, the composite image may be richer than the first satellite image and may include the same color information as the second satellite image.

[0071] In this regard, referring to (a) of FIG. 6, it is possible to confirm the result of improving (or enhancing) the spatial resolution corresponding to the red edge region and the shortwave infrared (SWIR) region of the second wavelength band from 20 m to 10 m, and referring to (b), it is possible to confirm the result of improving (or enhancing) the spatial resolution corresponding to the NDWI (Normalized Difference Water Index) according to the near-infrared and shortwave infrared spectral bands of the first wavelength band from 20 m to 10 m.

[0072] These improved results can be understood as a result of the resolution improvement system according to the present invention improving the resolution of satellite images through a resolution improvement method.

[0073] As described above, the present invention can diversify the types of colors that can be expressed in a synthesized image by synthesizing images containing pixel values ​​of different wavelengths.

[0074] While the embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A resolution improvement method for synthesizing a first satellite image captured with a first spatial resolution and a second satellite image captured with a second spatial resolution, A step of inputting the first and second satellite images into a deep learning model to generate a synthetic image is included. The first spatial resolution is higher than the second spatial resolution, The first radial resolution of the first satellite image is lower than the second radial resolution of the second satellite image, A resolution improvement method, characterized in that the synthetic image is composed of the first spatial resolution and the second radial resolution.

2. In paragraph 1, The above first satellite image is an image taken from the first satellite, The above second satellite image is an image taken from the second satellite, A resolution improvement method, characterized in that the photographing cycle for a specific point of the first satellite is shorter than the photographing cycle for the specific point of the second satellite.

3. In paragraph 2, Further comprising a step of training the above deep learning model, The steps for training the above deep learning model are: A step of downsampling a first sample image having a first spatial resolution to generate a first learning image having a spatial resolution lower than the first spatial resolution; A step of downsampling a second sample image having a second spatial resolution to generate a second learning image having a spatial resolution lower than the second spatial resolution; and A resolution improvement method, characterized by comprising a step of training a deep learning model that synthesizes two images into one image using at least one of the first and second learning images and the first and second sample images.

4. In paragraph 3, A resolution improvement method, characterized in that the spatial resolution of the first sample image is higher than the spatial resolution of the second sample image.

5. In paragraph 4, The first satellite image used in the step of generating the above synthetic image and the first learning image used in the step of training the deep learning model correspond to each other, The second satellite image used in the step of generating the above synthetic image and the second learning image used in the step of training the deep learning model correspond to each other, The spatial resolutions of the first satellite image and the first learning image corresponding to each other are different, A resolution improvement method, characterized in that the spatial resolutions of the second satellite image and the second learning image, which correspond to each other, are different from each other.

6. In paragraph 5, In the step of generating the first learning image, A resolution improvement method characterized in that the first learning image is downsampled to a second spatial resolution.

7. In paragraph 6, Each of the plurality of pixels constituting the first satellite image includes a pixel value corresponding to the first wavelength band, Each of the plurality of pixels constituting the second satellite image includes a pixel value corresponding to a first wavelength band and a second wavelength band different from the first wavelength band, Each pixel that constitutes the above composite image is A resolution improvement method characterized by including pixel values ​​corresponding to the first and second wavelength bands.

8. A communication unit configured to receive a first satellite image captured with a first spatial resolution and a second satellite image captured with a second spatial resolution; and A control unit for inputting the first and second satellite images into a deep learning model to generate a synthetic image, The first spatial resolution is higher than the second spatial resolution, The first radial resolution of the first satellite image is lower than the second radial resolution of the second satellite image, A resolution improvement system, characterized in that the synthetic image is composed of the first spatial resolution and the second radial resolution.

9. In paragraph 8, The above first satellite image is an image taken from the first satellite, The above second satellite image is an image taken from the second satellite, A resolution improvement system, characterized in that the photographing cycle for a specific point of the first satellite is shorter than the photographing cycle for the specific point of the second satellite.

10. In paragraph 9, The above control unit, Train the above deep learning model, Training of the above deep learning model is A step of downsampling a first sample image having a first spatial resolution to generate a first learning image having a spatial resolution lower than the first spatial resolution; A step of downsampling a second sample image having a second spatial resolution to generate a second learning image having a spatial resolution lower than the second spatial resolution; and A resolution improvement system characterized by comprising a step of training a deep learning model that synthesizes two images into one image using at least one of the first and second learning images and the first and second sample images.

Images