Image restoration method and imaging system for executing same

Abstract

An image restoration device according to one disclosed embodiment comprises: a memory for storing an artificial intelligence model and an original image captured by a camera; and a processor for training the artificial intelligence model, wherein the artificial intelligence model includes: an image restoration model for cropping the original image into a preset patch, and generating a restoration image on the basis of input data in which coordinate information of the patch is embedded in the cropped patch image; and a discrimination model for Fourier-transforming the restoration image generated by the image restoration model and discriminating the Fourier-transformed restoration image, and the processor performs adversarial learning on the image restoration model and the discrimination model, and generates, from the trained image restoration model, a restoration image for a new original image captured by the camera.

Description

Image restoration method and imaging system for executing the same

[0001] The present invention relates to an image restoration method using a chi information embedding technique and an adversarial learning method performed in Fourier space for improving the performance of an ultra-small imaging system having optical aberrations, and to an imaging system executing the same.

[0002] Metalens, ultra-thin lenses with ultra-short wavelength structures, are attracting attention as a technology that can overcome the limitations of conventional lenses. However, recent research has shown that large-area, broadband metalens suffer from a fundamental trade-off between broadband focusing efficiency and diameter. Consequently, currently available broadband metalens suffer from chromatic aberration or low focusing efficiency across a wide bandwidth, hindering the commercialization of compact metalens-based imaging systems.

[0003] Meanwhile, previously published image restoration technologies are as follows.

[0004] Prior paper 1 discusses an image restoration method using deconvolution. It assumes that performance degradation does not vary with location, and deconvolves the damaged image using a pre-measured point spread function (PSF). In other words, the technique in Prior paper 1 assumes that performance degradation does not vary with location, which leads to the problem of not being able to account for location-dependent performance degradation.

[0005] Prior Paper 2 describes an image restoration technique using deep learning. Due to the physical limitations of GPUs (Graphics Processing Units) and to improve learning efficiency, this technique randomly crops patches of a given image rather than using the full-resolution image. This approach, however, has limitations: when training a model on the given data, the image's positional information is completely lost, making it impossible to learn about performance degradation caused by lens aberrations.

[0006] Prior paper 3 discloses a patch-wise deconvolution method using patch-based PSFs. Prior paper 3 requires training by directly measuring PSFs for each patch unit region on the image to restore image damage caused by common optical aberrations, which have PSFs that vary with viewing angle. This approach poses challenges in terms of cost and time.

[0007] [Previous Paper 1]

[0008] Krishnan, Dilip, and Rob Fergus. “Fast image deconvolution using hyper-Laplacian priors.” Advances in neural information processing systems 22 (2009).

[0009] [Previous Paper 2]

[0010] Zamir, Syed Waqas, et al. “Multi-stage progressive image restoration.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021.

[0011] [Previous Paper 3]

[0012] Li, Xiu, et al. “Universal and flexible optical aberration correction using deep-prior based deconvolution.” Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021.

[0013] The disclosed embodiment relates to an image restoration method and an imaging system that executes the same, which can solve image quality degradation in an ultra-small imaging system by using an artificial intelligence model trained on image data directly captured by a lens with strong aberration and directly learning clear correct answer data in the frequency domain, and can restore image damage in a high spatial frequency domain.

[0014] An imaging restoration device according to one embodiment of the present disclosure includes: a memory for storing an original image captured by a camera and an artificial intelligence model; and a processor for training the artificial intelligence model; wherein the artificial intelligence model includes: an image restoration model for cropping the original image into a preset patch and generating a restored image based on input data in which coordinate information of the patch is embedded in a cropped patch image; and a discriminant model for Fourier transforming the restored image generated by the image restoration model and distinguishing the Fourier-transformed restored image; wherein the processor performs adversarial learning of the image restoration model and the discriminant model, and generates a restored image for a new original image captured by the camera from the image restoration model for which the learning is completed.

[0015] The above processor can generate the coordinate information based on the middle pixel of the patch image, convert the coordinate information into 2D coordinate data through a meshgrid method, and concatenate the coordinate information converted into 2D data to the patch image through a 1x1 convolution layer to generate the input data.

[0016] The above processor can input the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and output the restored image from the first neural network.

[0017] The first neural network may include a CNN (Convolution Neural Network) model including at least one of MIRNet, MPRNet, or NAFNet, or a Transformer model including at least one of Restormer or Uformer.

[0018] The above-mentioned discrimination model includes a CNN model including at least one of GoogleNet, AlexNet, and VGG Network for determining whether the Fourier-transformed restored image is genuine, and the processor can compare the Fourier-transformed restored image and the correct image through the discrimination model, and based on the comparison result, adversarially train the image restoration model so that the image restoration model generates a low-frequency original image into a high-frequency restored image.

[0019] The processor can train the discrimination model to output a value higher than a preset first reference value for the correct image, and to output a value lower than a preset second reference value for the restored image.

[0020] An imaging system according to another disclosed embodiment includes a camera including a metalens or a diffractive optical lens; a memory for receiving an original image captured by the camera and storing an artificial intelligence model; a processor for training the artificial intelligence model; and a display for outputting a restored image obtained by restoring the original image through the artificial intelligence model for which the processor has completed training; wherein the artificial intelligence model includes an image restoration model for cropping the original image into a patch of a preset size and generating a restored image based on input data in which coordinate information of the patch is embedded in a cropped patch image; and a discrimination model for Fourier transforming the restored image generated by the image restoration model and distinguishing the Fourier-transformed restored image.

[0021] The processor can generate the coordinate information based on the middle pixel of the patch image, convert the coordinate information into 2D coordinate data through a mesh grid method, and combine the coordinate information converted into 2D data with the patch image through a 1x1 convolution layer to generate the input data.

[0022] The above processor can input the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and output the restored image from the first neural network.

[0023] The processor can compare the restored image and the correct image that have been Fourier transformed through the discrimination model, and can adversarially train the image restoration model to generate a low-frequency original image into a high-frequency restored image based on the comparison result.

[0024] The processor can train the discrimination model to output a value higher than a preset first reference value for the correct image, and to output a value lower than a preset second reference value for the restored image.

[0025] Another embodiment of the present invention provides an image restoration method, comprising: training an artificial intelligence model; obtaining an original image by a camera including a metalens; and generating a restored image of the original image through the trained artificial intelligence model; wherein the artificial intelligence model comprises: an image restoration model that crops an image of training data into a patch of a preset size and generates a restored image based on input data in which coordinate information of the patch is embedded in a cropped patch image; and a discriminant model that Fourier transforms the restored image generated by the image restoration model and distinguishes the Fourier-transformed restored image; wherein training the artificial intelligence model comprises: comparing the Fourier-transformed restored image with a correct image through the discriminant model, and adversarially training the image restoration model so that the image restoration model generates a low-frequency original image as a high-frequency restored image based on a result of the comparison.

[0026] The image restoration model may generate the restored image by generating the coordinate information based on the middle pixel of the patch image, converting the coordinate information into 2D coordinate data through a mesh grid method, and embedding the input data by combining the coordinate information converted into 2D data with the patch image through a 1x1 convolution layer.

[0027] The image restoration model may generate the restored image by inputting the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and outputting the restored image from the first neural network.

[0028] Training the artificial intelligence model may include training the discrimination model to output a value higher than a preset first reference value for the correct image, and training the discrimination model to output a value lower than a preset second reference value for the restored image.

[0029] An image restoration method according to one embodiment of the invention and an imaging system executing the same can solve image quality degradation in an ultra-small imaging system by using an artificial intelligence model trained on image data directly captured by a lens with strong aberration.

[0030] In addition, the disclosed image restoration method and the imaging system executing the same can restore image damage in a high spatial frequency domain by directly learning clear correct answer data in the frequency domain. Therefore, the disclosed image restoration method and the imaging system executing the same, when installed in a wearable device such as a smartphone, can solve the camera protrusion phenomenon of the smartphone and enable capturing high-quality images / videos.

[0031] In addition, the disclosed image restoration method and the imaging system executing the same can also exhibit excellent performance for downstream applications that can be performed from images captured by cameras installed in drones and AR (Augmented Reality) / VR (Virtual Reality) devices based on improved image / video quality.

[0032] The disclosed image restoration method and the imaging system executing the same can realize a lightweight lens, thereby reducing the weight of the imaging system to be mounted, and thus, a weight reduction of unmanned aerial vehicles, drones, etc. can be expected, and thus, an increase in power efficiency can also be expected. In addition, the disclosed image restoration method and the imaging system executing the same can reduce the weight of the device itself by mounting an ultra-small imaging system on the imaging system that acts as the eye of an AR / VR device, and can significantly reduce user fatigue, thereby improving the user experience.

[0033] FIG. 1 is a schematic drawing for explaining each component of an imaging system according to one disclosed embodiment.

[0034] FIG. 2 is a control block diagram of an imaging restoration device according to one embodiment of the disclosure.

[0035] FIG. 3 is a general flowchart of an image restoration method according to one disclosed embodiment.

[0036] FIG. 4 is a drawing specifically explaining an artificial intelligence model according to an embodiment of the disclosed method.

[0037] Figures 5 and 6 are drawings for explaining a method of generating and combining coordinate information.

[0038] FIG. 7 is a flowchart specifically explaining a method of performing learning in an image restoration method according to an embodiment disclosed.

[0039] Figure 8 is a drawing for explaining a determination process according to one embodiment.

[0040] Figure 9 is an example of the results of the disclosed imaging system.

[0041] Figure 10 is a table showing the performance comparison results of the disclosed image restoration method.

[0042] Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and any general information within the technical field to which the present invention pertains or any information that overlaps between the embodiments is omitted.

[0043] Throughout the specification, when we say that a part is "connected" to another part, this includes both direct connections as well as indirect connections.

[0044] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0045] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0046] Additionally, terms such as "~part", "~device", "~block", "~absence", and "~module" may refer to a unit that processes at least one function or operation. For example, the terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) / ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0047] The symbols attached to each step are used to identify each step and do not indicate the order of the steps, and the steps may be performed in a different order than stated unless the context clearly indicates a specific order.

[0048] Hereinafter, an embodiment of a speaker verification device according to one aspect will be described in detail with reference to the attached drawings.

[0049] FIG. 1 is a schematic drawing for explaining each component of an imaging system according to one disclosed embodiment.

[0050] An imaging system (1) according to the disclosed embodiment includes a camera (5) for photographing an object (Object, Ob), an imaging restoration device (10-1, 10-2, 10-3, 10) for receiving an original image photographed (or acquired) by the camera (5) and generating a restored image through the disclosed image restoration method.

[0051] A camera (5) according to a specifically disclosed embodiment may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor including an ultra-small lens such as a metalens and a diffractive optics lens. Typical commercial lenses are formed by overlapping multiple lenses due to the performance limitations of a single lens. As a result, the weight and size of various electronic devices using commercial lenses, such as smartphones (10-2) and drones (10-3), have increased, and there has been a problem of reduced power efficiency. To solve these problems, a metalens (or singlet lens) has recently been developed, and a camera (5) including such an ultra-small lens is expected to reduce the weight of the user terminal (10) and increase power efficiency.

[0052] However, the camera (5) equipped with such an ultra-small lens had a problem in that the performance of the restored image was deteriorated due to the limitations of the process and theoretical limitations. In particular, the camera (5) including such an ultra-small lens had strong aberrations due to the limitations of the small and thin lens. In addition, the camera (5) including the ultra-small lens had a significant difference in the performance of the restored image depending on the shooting space, and it was difficult to use a conventional general restoration model (including an artificial intelligence model).

[0053] The user terminal (10) that receives the original image captured by the camera (5), i.e., the original image with strong aberration, includes an artificial intelligence model (20 in FIG. 2) trained through an adversarial learning method, and can generate an image (hereinafter, a restored image) that restores the received original image through the trained artificial intelligence model (20).

[0054] The user terminal (10) may be implemented as a server (10-1), a smartphone (10-2), or a drone (10-3), as shown in FIG. 1, or may be implemented as a computer or portable terminal capable of receiving an original image through a network. Here, the computer includes, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a WEB Browser, and the portable terminal includes, for example, a wireless communication device that ensures portability and mobility, such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone, etc., and all kinds of handheld-based wireless communication devices, and wearable devices such as a watch, glasses, contact lenses, or head-mounted-device (HMD), etc. That is, the user terminal (10) may be a device that uses a learned artificial intelligence model (20). After saving, it can include various configurations that can create a restored image from the original image captured through the camera (5). Hereinafter, a user terminal (10) including a camera (5) will be described as an embodiment (image restoration device).

[0055] FIG. 2 is a control block diagram of an imaging restoration device according to one embodiment of the disclosure.

[0056] Referring to FIG. 2, the disclosed imaging restoration device (10) may include a camera (5), a communication unit (11) capable of communicating with the outside, a processor (12) that generates a restored image by inputting an original image transmitted through the camera (5) or the communication unit (11) into an artificial intelligence model (20) that has completed learning, a memory (13) that stores the received original image, an image required for learning (correct image), or the generated restored image, as well as the artificial intelligence model (20), and an output unit (14) that displays the original image or the generated restored image.

[0057] A camera (5) according to a specifically disclosed embodiment can obtain an original image with strong aberration by operating under the control of a processor (12) with an image sensor including an ultra-small lens, for example, a metalens or a diffractive optical lens. The camera (5) converts the original image into an electrical signal and transmits it to the processor (12) or memory (13).

[0058] The communication unit (11) is configured to receive an original image captured externally through an ultra-small lens via a communication network. The communication unit (11) may include one or more components that enable communication with the outside world, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0059] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at a short distance, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0060] The wired communication module may include various wired communication modules such as a Controller Area Network (CAN) communication module, a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0061] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution), in addition to a WiFi module and a WiBro (Wireless broadband) module.

[0062] Memory (13) may be implemented as at least one of non-volatile memory elements such as cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, or volatile memory elements such as RAM (Random Access Memory), or storage media such as a hard disk drive (HDD) or CD-ROM, but is not limited thereto.

[0063] The output unit (14) is a display that displays an original image or a restored image, and may be provided as a digital light processing (DLP) panel, a plasma display panel, a liquid crystal display (LCD) panel, an electroluminescence (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0064] The processor (12) is a component that controls the overall imaging restoration device (10), and in particular, can train an artificial intelligence model (20), generate a restoration image through the trained artificial intelligence model (20), and display the image through the output unit (14). In addition, the processor (12) can also provide the generated restoration image to a wearable device carried by the user through the communication unit (11).

[0065] Specifically, the artificial intelligence model (20) can be divided into an image restoration model (21) that crops an original image into a preset patch and generates a restored image based on input data that embeds coordinate information of the patch into the cropped patch image, and a discriminant model (22) that Fourier transforms the restored image generated by the image restoration model (21) and distinguishes the Fourier-transformed restored image. The processor (12) performs adversarial learning of the image restoration model (21) and the discriminant model (22), thereby improving the performance of the image restoration model (21) and restoring the original image received from the camera (5) or the communication unit (11) later. A specific description of the image restoration method in which the processor (12) trains the artificial intelligence model (20) and performs the image restoration using the artificial intelligence model (20) for which training is completed will be described later with reference to other drawings.

[0066] A processor (12) may refer to a data processing device built into hardware that has a physically structured circuit to perform a function expressed by a code or command included in a program. Examples of such a data processing device built into hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and a graphics processing unit (GPU). The processor (12) may be provided in a plurality of configurations including one or more processors.

[0067] Meanwhile, the imaging restoration device (10) may include various configurations in addition to the configuration described above in FIG. 2, and some specific configurations may be omitted as needed.

[0068] FIG. 3 is a general flowchart of an image restoration method according to one disclosed embodiment.

[0069] Referring to Figure 3, the image restoration method first trains an artificial intelligence model (20) (100).

[0070] The artificial intelligence model (20) is trained through adversarial learning. Specifically, among the artificial intelligence models, the image restoration model (21) acts as a generator in adversarial learning, which generates a restored image based on original data (training data) input by the user. The discriminant model (22) acts as a discriminator in adversarial learning, which performs a Fourier transform on the restored image generated by the image restoration model (21) and then compares the Fourier-transformed restored image with the correct data. Here, the correct data is an image with a clear image quality that the user wishes to restore.

[0071] The disclosed image restoration method trains the entire artificial intelligence model (20) by continuously updating (training) the image restoration model (21) and the discrimination model (22) so that the image restoration model (21) can generate a high-frequency restored image from a low-frequency original image.

[0072] When the learning of the artificial intelligence model (20) is completed, the camera (5) takes a new picture of the object (200).

[0073] Here, the image captured and transmitted by the camera (5) is a new original image captured of an object (Ob), not the original image used for learning the artificial intelligence model (20).

[0074] The image restoration method inputs the original image captured by the camera (5) into the artificial intelligence model (20) that has completed learning (300).

[0075] Specifically, the original image captured by the camera (5) is input only to the image restoration model (21) for which learning has been completed, and the image restoration model (21) for which learning has been completed generates a high-frequency restored image.

[0076] The image restoration method outputs (400) a restored image generated (output) from an artificial intelligence model (20).

[0077] There may be various ways to output the restored image generated by the image restoration method. If the image restoration method is implemented in an imaging restoration device (10) including a display, the restored image may be displayed to the user via an output unit (14). If the imaging restoration device (10) is provided in a configuration that does not include an output unit (14) (e.g., a drone without a display), the imaging restoration device (10) may transmit the generated restored image to a wearable device such as a smartphone via a communication unit (11).

[0078] FIG. 4 is a drawing specifically explaining an artificial intelligence model according to an embodiment of the disclosed method.

[0079] Referring to FIG. 4, the artificial intelligence model (20) performs an adversarial learning process, i.e., an iterative process in which an image restoration model (21) generates a restored image, a discrimination model (22) distinguishes between authenticity and falsehood by comparing the restored image with the correct image, and an image restoration model (21) with an adjusted loss function based on the discrimination result of the discrimination model (22) generates a restored image again.

[0080] The artificial intelligence model (20) is characterized by operating the image restoration model (21) in the spatial domain and the discrimination model (22) in the Fourier domain.

[0081] Specifically, the image restoration model (21) learns the original image as a restored image based on the patch image (211). The image restoration model (21) according to the disclosed embodiment uses a patch image (211) randomly cropped from the original image (training data) in order to overcome the physical capacity limitations of the GPU, etc. provided in the imaging restoration device (10). That is, the patch image is a plurality of images in which the original image is cropped into patch units having a size set by the user.

[0082] Meanwhile, a randomly cropped patch image was also used in the conventional image restoration method using deconvolution. However, the disclosed image restoration model (21) generates coordinate information (212) of a patch image (211) to improve the restoration performance of an original image with optical aberrations, and embeds the coordinate information (212) thus generated into the patch image (211).

[0083] Specifically, the image restoration model (21) generates coordinate information (212) based on the patch size of the patch image (211). When the coordinate information is generated for each patch, the image restoration model (21) converts each coordinate information in the patch image (211) into two-dimensional coordinate data (2D coordinate data) through the mesh grid method.

[0084] In order to convert into two-dimensional coordinate data, the image restoration model (21) includes a 1x1 convolution layer (213). The image restoration model (21) generates a patch image (211) in which the coordinate information (212) is concatenated, i.e., input data (214), by passing the generated coordinate information (212) and the patch image (211) through the 1x1 convolution layer (213). The positional embedding method performed by the image restoration model (21) is described in detail through other drawings below.

[0085] The image restoration model (21) inputs input data (214) into the first neural network (215). The first neural network (215) is a neural network in which the size of the input data (214) and the size of the output data, i.e., the restored image, are the same. The first neural network (215) of Fig. 4 is illustrated to represent a skip connection, which is a basic concept of a Nonlinear Activation Free Network (NAFNet). However, the first neural network (215) is not necessarily limited to NAFNet, and may include a CNN (Convolution Neural Network) model including at least one of MIRNet or MPRNet, or a Transformer model including at least one of Restormer or Uformer.

[0086] When the image restoration model (21) generates a restoration image, the discrimination model (22) performs a Fourier transform (221) of the restoration model.

[0087] Original images damaged by optical aberration often have significant loss of high-frequency components. Therefore, the disclosed discriminant model (22) performs adversarial learning using domain-transformed restored images, thereby enabling the trained image restoration model (21) to effectively generate high-frequency information based on low-frequency information.

[0088] The Fourier-transformed restored image is input to the second neural network (222). Here, the discrimination model (22) compares the restored image with the original image and outputs whether the restored image is correct (Real) or restored (Fake). The second neural network (222) may include a CNN model including at least one of GoogleNet, AlexNet, and VGG Network.

[0089] Figures 5 and 6 are diagrams illustrating a method for generating and combining coordinate information. To avoid redundant explanation, they are described together below.

[0090] Referring first to FIG. 5, the disclosed image restoration method crops the original image in patch units (111).

[0091] The original images included in the training data are cropped into prepared patch units of a preset size (same size) within the same batch, and the locations are cropped randomly.

[0092] The disclosed image restoration method generates coordinate information (112) based on the middle pixel of the patch image (211).

[0093] In the embodiment of Fig. 6, a patch image (211) can be cropped at a specific location of the original image (201). The image restoration method can generate coordinate information (212) based on the value of the midpoint (x, y,) based on the pixel (0, 0) at the upper left of the patch image (211).

[0094] The image restoration method converts coordinate information (212) into 2D coordinate data (113) using the Meshgrid method.

[0095] The mesh grid method relates to a method for returning two-dimensional (or three-dimensional) grid coordinates based on coordinates included in vector x and vector y, and the disclosed image restoration method converts coordinate information (212) of a patch image (211) into two-dimensional coordinate data, thereby allowing the patch image (211) and coordinate information (212) to be combined later through a 1x1 convolution layer (213). The mesh grid method illustrated in FIG. 6 relates to an example of converting specific coordinates (x, y) into grid coordinate data.

[0096] Referring again to Figure 5, coordinate information (212) converted into 2D data is combined with a patch image (211) through a 1x1 convolution layer (213) (114).

[0097] Here, the combination of the patch image (211) and the coordinate information (212) can be generated by inputting the patch image (211) and the coordinate information (212) into a 1x1 convolution layer (213) in a manner of combining a specific sequence.

[0098] The image restoration method generates input data (214) in which a patch image (211) and coordinate information (212) are combined, and generates a restoration image based on the input data through a first neural network (215). Here, the sizes of the input data and the restoration image output through the first neural network (215) are identical.

[0099] Figure 7 is a flowchart specifically illustrating a learning process in an image restoration method according to one disclosed embodiment. Figure 8 is a diagram illustrating a determination process according to one embodiment. To avoid redundant explanation, these two processes are described together below.

[0100] Referring first to FIG. 7, the image restoration method embeds coordinate information (212) into a patch image (211) (110), and the image restoration model (21) generates a restored image (120).

[0101] The method of generating coordinate information (212) and combining it with a patch image (211) to generate input data (or learning data, 214) is omitted as described above in FIGS. 5 and 6.

[0102] When the image restoration model (21) generates a restoration image, the discrimination model (22) performs a Fourier transform of the restoration image (130) and determines the Fourier transformed restoration image (140).

[0103] Referring to FIG. 8, the discrimination model (22) according to one embodiment can control the output from the second neural network (222) to be output as a preset reference value.

[0104] Specifically, the second neural network (222) can output a value greater than or equal to 1 if the Fourier-transformed restored image is well restored to the extent that it can be judged as the correct image. If the Fourier-transformed restored image is judged as an image that is not restored to the extent of the correct image, it can output a value less than or equal to -1. That is, the reference value (1: first reference value, 2: second reference value) set as the value output by the second neural network (222) can be changed by the user. In addition, the range of the Decision Boundary and Margin illustrated in FIG. 8 can be set in various ways.

[0105] Referring back to Figure 7, the image restoration method determines whether learning is complete using the learning data. If learning is not complete (No in 150), the image restoration method adjusts the loss function of the artificial intelligence model (20) and then performs the image restoration step (120) again, thereby proceeding with learning.

[0106] The disclosed artificial intelligence model (20) is updated in the direction of reducing the loss function, and the overall loss function (L) of the disclosed artificial intelligence model (20) Total ) is as shown in mathematical formula 1 below.

[0107] [Mathematical Formula 1]

[0108]

[0109] Here, λ is a hyperparameter set by the user in advance and can be changed in various ways.

[0110] Specifically L PSNR is the loss function between the Fourier transformed restored image (restored result) and the correct image. L PSNR is defined by the mathematical formula 2 below.

[0111] [Equation 2]

[0112]

[0113] Here, X^ is the result restored from the image restoration model (21), and X represents the correct image. R is the maximum pixel value of the correct image (X^), and MSE is an indicator representing the distance between X^ and X, and can be defined by the following mathematical expression 3.

[0114] [Equation 3]

[0115]

[0116] L in Equation 1 a is the loss function (L) of the image restoration model (21). G ) and the loss function (L) of the discriminant model (22) D ) is defined by the mathematical formula 4 below.

[0117] [Equation 4]

[0118]

[0119] The image restoration model is the loss function (L) of the image restoration model (21). G ) and the loss function (L) of the discriminant model (22) D ) is learned in the direction that minimizes the sum.

[0120] Loss function (L) of the discriminant model (22) D ) is the loss function (L) of the image restoration model (21) as shown in the mathematical expression 5 below. G ) is defined by the mathematical formula 6 below.

[0121] [Equation 5]

[0122]

[0123] [Equation 6]

[0124]

[0125] Here, F() represents the Fourier transform and D() represents the output of the discriminant model (22).

[0126] Meanwhile, when learning is completed (Yes of 150), the image restoration method generates a new restored image from the image restoration model (21) for which learning has been completed (401).

[0127] Here, the new restored image is a restored image output by an image restoration model (21) that has completed learning from an original image newly captured by a camera (5).

[0128] Figure 9 is an example of the results of the disclosed imaging system.

[0129] The imaging system (1) can obtain an image with strong aberration, such as the original image (302) of Fig. 9. The stronger the aberration, the more high-frequency components disappear from the original image (302).

[0130] The disclosed imaging system (1) trains an image restoration model (21) through adversarial learning while training a discrimination model (22) based on the correct image (301) of FIG. 9. The imaging system (1) that has completed training can output a restored image (304) for the original image (302) through the image restoration model (21) that has completed training.

[0131] Comparing the image (303) restored using conventional techniques with the restored image (304), the image (303) restored using conventional deep learning models fails to restore edge damage. However, it can be confirmed that the image (304) restored using the disclosed imaging system (1) restores the high-frequency portion of the image more distinctly than the image (303) restored using conventional techniques.

[0132] Figure 10 is a table showing the performance comparison results of the disclosed image restoration method.

[0133] Table 10 compares the disclosed image restoration method with currently disclosed image restoration models such as MIRNet V2, SFNet, HINet, and NAFNet. The performance of each artificial intelligence model was compared using PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index Map), and LPIPS (Learned Perceptual Image Patch Similarity).

[0134] Specifically, the disclosed image restoration method shows a PSNR of 22.095 / 2.423, which is higher than that of the conventional model, and SSIM and LPIPS are also 0.692 / 0.103 and 0.432 / 0.096, which are higher than that of the conventional model.

[0135] The image restoration method disclosed herein and the imaging system executing it can resolve image quality degradation in ultra-small imaging systems by using an artificial intelligence model trained on image data directly captured with a lens with strong aberrations.

[0136] In addition, the disclosed image restoration method and the imaging system executing the same can restore image damage in a high spatial frequency domain by directly learning clear correct answer data in the frequency domain. Therefore, the disclosed image restoration method and the imaging system executing the same, when installed in a wearable device such as a smartphone, can solve the camera protrusion phenomenon of the smartphone and enable capturing high-quality images / videos.

[0137] In addition, the disclosed image restoration method and the imaging system executing the same can also exhibit excellent performance for downstream applications that can be performed from images captured by cameras installed in drones and AR (Augmented Reality) / VR (Virtual Reality) devices based on improved image / video quality.

[0138] The disclosed image restoration method and the imaging system executing the same can realize a lightweight lens, thereby reducing the weight of the imaging system to be mounted, and thus, a weight reduction of unmanned aerial vehicles, drones, etc. can be expected, and thus, an increase in power efficiency can also be expected. In addition, the disclosed image restoration method and the imaging system executing the same can reduce the weight of the device itself by mounting an ultra-small imaging system on the imaging system that acts as the eye of an AR / VR device, and can significantly reduce user fatigue, thereby improving the user experience.

[0139] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0140] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0141] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. Memory for storing original images and artificial intelligence models captured from the camera; and A processor for training the artificial intelligence model is included; The above artificial intelligence model is, An image restoration model that crops the original image into a preset patch and generates a restored image based on input data that embeds coordinate information of the patch into the cropped patch image; and A discriminant model that Fourier transforms the restored image generated by the image restoration model and distinguishes the restored image obtained by Fourier transformation; The above processor, An imaging restoration device that performs adversarial learning of the image restoration model and the discrimination model, and generates a restoration image for a new original image captured from the camera from the image restoration model for which the learning has been completed.

2. In paragraph 1, The above processor, Generate the coordinate information based on the middle pixel of the above patch image, The above coordinate information is converted into 2D coordinate data using the Meshgrid method, An imaging restoration device that generates the input data by concatenating coordinate information converted into 2D data into the patch image through a 1x1 convolution layer.

3. In paragraph 2, The above processor, An imaging restoration device that inputs the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and outputs the restored image from the first neural network.

4. In paragraph 3, The above first neural network, An image restoration device comprising a CNN (Convolution Neural Network) model including at least one of MIRNet, MPRNet, or NAFNet, or a Transformer model including at least one of Restormer or Uformer.

5. In paragraph 3, The above discrimination model is, A CNN model including at least one of GoogleNet, AlexNet, and VGG Network for determining whether the restored image is authentic or not, The above processor, An imaging restoration device that compares the restored image and the correct image obtained through the Fourier transformation using the above discrimination model, and adversarially trains the image restoration model to generate a low-frequency original image into a high-frequency restored image based on the comparison result.

6. In paragraph 5, The above processor, An imaging restoration device that trains the above-mentioned discrimination model to output a value higher than a preset first reference value for the correct image and to output a value lower than a preset second reference value for the restored image.

7. Camera including a metalens; A memory that receives the original image captured by the above camera and stores the artificial intelligence model; a processor that trains the artificial intelligence model; and The above processor includes a display that outputs a restored image obtained by restoring the original image through an artificial intelligence model for which learning has been completed; The above artificial intelligence model is, An image restoration model that crops the original image into a patch of a preset size and generates a restored image based on input data that embeds coordinate information of the patch into the cropped patch image; and An imaging system comprising a Fourier transform model for transforming a restored image generated by the image restoration model and a discrimination model for distinguishing the Fourier transformed restored image.

8. In paragraph 7, The above processor, Generate the coordinate information based on the middle pixel of the above patch image, Convert the above coordinate information into 2D coordinate data using the mesh grid method, An imaging system that generates the input data by combining coordinate information converted into the 2D data into the patch image through a 1x1 convolution layer.

9. In paragraph 8, The above processor, An imaging system that inputs the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and outputs the restored image from the first neural network.

10. In paragraph 9, The above processor, An imaging system that compares the restored image and the correct image obtained through the Fourier transformation using the above discrimination model, and adversarially trains the image restoration model to generate a low-frequency original image into a high-frequency restored image based on the comparison result.

11. In paragraph 10, The above processor. An imaging system that trains the above-mentioned discrimination model to output a value higher than a preset first reference value for the correct image and to output a value lower than a preset second reference value for the restored image.

12. Train the artificial intelligence model; A camera including a metalens or a rotating optical lens acquires the original image; generating a restored image of the original image using the artificial intelligence model for which the above learning has been completed; The above artificial intelligence model is, An image restoration model that crops an image of learning data into a patch of a preset size and generates a restored image based on input data in which coordinate information of the patch is embedded in the cropped patch image; and A discriminant model that Fourier transforms the restored image generated by the image restoration model and distinguishes the restored image obtained by Fourier transformation; Learning the above artificial intelligence model is An image restoration method comprising: comparing the Fourier-transformed restored image and the correct image through the discriminant model, and adversarially training the image restoration model to generate a low-frequency original image into a high-frequency restored image based on the comparison result; 13. In paragraph 12, The above image restoration model generates the above restoration image, Generate the coordinate information based on the middle pixel of the above patch image, Convert the above coordinate information into 2D coordinate data using the mesh grid method, An image restoration method comprising: embedding the input data by combining coordinate information converted into 2D data into the patch image through a 1x1 convolution layer.

14. In paragraph 13, The above image restoration model generates the above restoration image, An image restoration method comprising: inputting the input data output from the 1x1 convolution layer into a first neural network having the same size as the output data, and outputting the restored image from the first neural network.

15. In paragraph 12, Training the above artificial intelligence model is as follows: An image restoration method comprising: training the above-mentioned discrimination model to output a value higher than a first preset reference value for the correct image, and training the model to output a value lower than a second preset reference value for the restored image.

Images