Image restoration method and imaging system for performing the same

Abstract

An imaging repair apparatus according to one embodiment of the present invention includes a memory for storing original images captured by a camera and an artificial intelligence model, and a processor for training the artificial intelligence model, wherein the artificial intelligence model includes an image repair model that crops the original image into a patch of a predetermined size and generates a repaired image based on input data in which the coordinate information of the patch is embedded in the cropped patch image, and a discrimination model that performs a Fourier transform on the repaired image generated by the image repair model and distinguishes the Fourier transformed repaired image, wherein the processor performs adversarial training on the image repair model and the discrimination model and generates a repaired image for a new original image captured by the camera from the trained image repair model.

Description

[Technical Field]

[0001] This invention relates to an image restoration method using a dimensional information embedding technique and an adversarial learning method performed in Fourier space, and an imaging system that implements this method, for improving the performance of a microimaging system having optical aberrations.

[0002] This research was supported by a grant from the Korea Research Foundation (NRF), which is supported by the Korean government (MSIT: Ministry of Science and ICT) (RS-2024-00338048), and the Global Research Support Program in the Digital Field, administered by the Institute for Information and Communications Planning and Evaluation (IITP) with the support of the Ministry of Science and ICT (MSIT) of Korea (RS-2024-00412644). It also received support from the Culture, Sports and Tourism R&D Program (RS-2024-00332210), which is supported by the Ministry of Culture, Sports and Tourism through the Korea Creative Content Agency in 2024. [Background technology]

[0003] Metalenses, which are ultrathin-film lenses composed of subwavelength structures, are attracting attention as a technology that can overcome the limitations of existing lenses. However, recent research has shown that large-area, broadband metalenses have a fundamental trade-off between broadband focal efficiency and diameter. Therefore, currently known broadband metalenses have chromatic aberration or low focal efficiency over a wide bandwidth, which is a bottleneck for the commercialization of compact imaging systems based on metalenses.

[0004] On the other hand, known conventional image restoration techniques are as follows:

[0005] Non-Patent Document 1 concerns an image restoration method using deconvolution. Non-Patent Document 1 performs deconvolution on a damaged image using a pre-measured point spread function (PSF), under the assumption that the performance degradation with respect to location does not change. In other words, the technique in Non-Patent Document 1 has the problem that it cannot take into account the performance degradation with respect to location, because it assumes that the performance degradation with respect to location does not change.

[0006] Non-patent document 2 describes an image restoration technique using deep learning. Due to the physical limitations of GPUs (Graphics Processing Units) and the efficiency of training, the technique in Non-patent document 2 does not use the full-resolution image as is, but instead randomly crops patch images from a given image and uses them. In this case, when the model is trained on the given data, the positional information of the image is completely lost, and there is a limitation that it cannot learn about performance degradation due to lens aberrations.

[0007] Non-patent document 3 discloses a patch-based patch-wise deconvolution method using PSF. Non-patent document 3 addresses the issue of cost and time in repairing image damage caused by common optical aberrations, which have PSFs that vary depending on the field of view. This method involves directly measuring the PSF for each patch-unit area on the image and using that for training. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Krishnan, Dilip, and Rob Fergus. “Fast image deconvolution using hyper-Laplacian priors.” Advances inneural information processing systems 22 (2009). [Non-Patent Document 2] Zamir, Syed Waqas, et al. “Multi-stage progressive image restoration.” Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2021. [Non-Patent Document 3] Li, Xiu, et al. “Universal and flexible optical aberration correction using deep-prior based deconvolution.” Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021. [Overview of the project] [Problems that the invention aims to solve]

[0009] One embodiment of the present invention relates to an image restoration method and an imaging system that implements this method, which can solve the problem of image quality degradation in a microimaging system and repair image damage in the high spatial frequency domain by using an artificial intelligence model trained on image data directly captured with a lens with strong aberrations to directly learn vivid ground truth data in the frequency domain. [Means for solving the problem]

[0010] An imaging repair apparatus according to one embodiment of the present invention includes a memory for storing original images captured by a camera and an artificial intelligence model, and a processor for training the artificial intelligence model, wherein the artificial intelligence model includes an image repair model that crops the original image into a preset patch and generates a repaired image based on input data in which the coordinate information of the patch is embedded in the cropped patch image, and a discrimination model that performs a Fourier transform on the repaired image generated by the image repair model and distinguishes the Fourier transformed repaired image, wherein the processor performs adversarial learning on the image repair model and the discrimination model and generates a repaired image for a new original image captured by the camera from the trained image repair model.

[0011] The processor can generate the coordinate information based on the intermediate pixels of the patch image, convert the coordinate information into 2D coordinate data using a mesh grid method, and concatenate the converted 2D coordinate information to the patch image via a 1x1 convolutional layer to generate the input data.

[0012] The processor can input the input data output from the 1x1 convolutional layer into a first neural network whose output data is of the same size, and output the restored image from the first neural network.

[0013] The first neural network may include a CNN (Convolutional Neural Network) model that includes at least one of MIRNet, MPRNet, or NAFNet, or a transformer model that includes at least one of Restormer or Uformer.

[0014] The discriminant model includes a CNN model that includes at least one of GoogleNet, AlexNet, and VGGNet for determining the truth or falsity of the Fourier-transformed restored image, and the processor compares the Fourier-transformed restored image with the ground truth image and, based on the results of the comparison, can train the image restoration model adversarially to generate a high-frequency restored image from a low-frequency original image.

[0015] The processor can be trained to output a value equal to or greater than a pre-set first reference value for the correct image, and to output a value equal to or less than a pre-set second reference value for the repaired image.

[0016] An imaging system according to another embodiment of the present invention includes a camera including a metalens or diffractive optical lens, a memory that receives a raw image captured by the camera and stores an artificial intelligence model, a processor that trains the artificial intelligence model, and a display that outputs a restored image obtained by the processor from the trained artificial intelligence model, wherein the artificial intelligence model includes an image restoration model that crops the raw image into a patch of a preset size and generates a restored image based on input data in which the coordinate information of the patch is embedded in the cropped patch image, and a discrimination model that performs a Fourier transform on the restored image generated by the image restoration model and distinguishes the Fourier transformed restored image.

[0017] The processor can generate the coordinate information based on the intermediate pixels of the patch image, convert the coordinate information into 2D coordinate data using a mesh grid method, and combine the coordinate information converted into 2D data with the patch image via a 1x1 convolutional layer to generate the input data.

[0018] The processor can input the input data output from the 1×1 convolutional layer, which has the same size as the output data, into a first neural network, and output the restored image from the first neural network.

[0019] The processor can compare the Fourier-transformed restored image and the correct image by the discrimination model, and adversarially train the image restoration model based on the result of the comparison so that the image restoration model generates a low-frequency original image into a high-frequency restored image.

[0020] The processor can train the discrimination model to output a value equal to or greater than a first reference value set in advance for the correct image, and train the discrimination model to output a value equal to or less than a second reference value set in advance for the restored image.

[0021] An image restoration method according to another embodiment of the present invention includes training an artificial intelligence model, acquiring an original image by a camera including a metalens, and generating a restored image of the original image by the trained artificial intelligence model. The artificial intelligence model includes an image restoration model that crops an image of training data into patches of a preset size and generates a restored image based on input data in which the coordinate information of the patches is embedded in the cropped patch images, and a discrimination model that Fourier-transforms the restored image generated by the image restoration model and distinguishes the Fourier-transformed restored image. Training the artificial intelligence model includes adversarially training the image restoration model based on the result of comparing the Fourier-transformed restored image and the correct image by the discrimination model so that the image restoration model generates a low-frequency original image into a high-frequency restored image.

[0022] The generation of the restored image by the image restoration model may include generating the coordinate information based on the intermediate pixels of the patch image, converting the coordinate information into 2D coordinate data by a mesh grid method, and embedding the input data by combining the coordinate information converted into the 2D data with the patch image through a 1×1 convolutional layer.

[0023] The generation of the restored image by the image restoration model may include inputting the input data output from the 1×1 convolutional layer and the output data having the same size into a first neural network, and outputting the restored image from the first neural network.

[0024] Training the artificial intelligence model may include training the discrimination model to output a value equal to or greater than a preset first reference value for the correct image, and training the discrimination model to output a value equal to or less than a preset second reference value for the restored image.

Advantages of the Invention

[0025] The image restoration method and the imaging system for executing the same according to an embodiment of the present invention can solve the degradation of image quality in a micro-imaging system by using an artificial intelligence model trained with image data directly captured by a lens having strong aberration.

[0026] In addition, the image restoration method and the imaging system for executing the same according to the present invention can repair image damage in a high spatial frequency region by directly learning vivid correct data in the frequency domain. Therefore, when the image restoration method and the imaging system for executing the same according to the present invention are mounted on a wearable device such as a smartphone, it can solve the protrusion of the smartphone camera and enable high-quality images and videos to be captured.

[0027] Furthermore, the image restoration method and imaging system of the present invention can demonstrate excellent performance in downstream applications that can be performed using images captured by cameras installed in unmanned aerial vehicles and AR (Augmented Reality) / VR (Virtual Reality) devices, based on improved image and video quality.

[0028] The image restoration method and imaging system of the present invention can achieve lighter lenses, thereby reducing the weight of the imaging system on which it is mounted, and thus reducing the weight of unmanned aerial vehicles, drones, etc., and thereby increasing power efficiency. Furthermore, by mounting a microimaging system on the imaging system that acts as the eyes in AR / VR devices, the image restoration method and imaging system of the present invention can reduce the weight of the device itself, increasing power efficiency, significantly reducing user fatigue, and improving the user experience. [Brief explanation of the drawing]

[0029] [Figure 1] This is a schematic diagram illustrating the various components of an imaging system according to one embodiment of the present invention. [Figure 2] This is a control block diagram of an imaging repair device according to one embodiment of the present invention. [Figure 3] This is a flowchart of the image restoration method according to one embodiment of the present invention. [Figure 4] This is a diagram illustrating an artificial intelligence model according to one embodiment of the present invention. [Figure 5] This is a diagram illustrating a method for generating and combining coordinate information. [Figure 6] This is a diagram illustrating a method for generating and combining coordinate information. [Figure 7] This flowchart provides a detailed explanation of how to proceed with learning using the image restoration method according to one embodiment of the present invention. [Figure 8] This is a diagram illustrating the discrimination process according to one embodiment of the present invention. [Figure 9] This is an illustrative diagram showing the results of the imaging system of the present invention. [Figure 10] This table shows the results of a performance comparison of the image restoration methods of the present invention. [Modes for carrying out the invention]

[0030] Throughout this specification, identical components are denoted by the same reference numerals. This specification does not describe all elements of these embodiments, and general content in the art to which the present invention pertains, or content that overlaps with each other in these embodiments, is omitted.

[0031] When a part of a specification is considered to be "connected" to another part, this includes not only direct connections but also indirect connections.

[0032] Furthermore, when a part is said to "include" a certain component, unless otherwise specified, this does not exclude other components, but rather means that it may include other components.

[0033] Unless otherwise clearly stated in the context, singular expressions include plural forms.

[0034] Furthermore, terms such as "~part," "~machine," "~block," "~component," and "~module" can refer to a unit that processes at least one function or operation. For example, these terms can refer to at least one piece of hardware such as an FPGA (field-programmable gate array) / ASIC (application-specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0035] The symbols assigned to each stage are used to identify each stage, and do not indicate the procedures between the stages. Each stage may be performed in a manner different from the specified procedure unless the context clearly indicates otherwise.

[0036] Hereinafter, an embodiment of an imaging system according to one aspect will be described in detail with reference to the attached drawings.

[0037] Figure 1 is a schematic diagram illustrating the various components of an imaging system according to one embodiment of the present invention.

[0038] An imaging system 1 according to one embodiment of the present invention includes a camera 5 that images an object (Ob), and an imaging restoration device 10 that receives the original image captured (or acquired) by the camera 5 and generates a restored image by the image restoration method of the present invention, the imaging device 10 can be embodied by 10-1, 10-2, and 10-3.

[0039] Specifically, camera 5 according to one embodiment of the present invention may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor that includes microlenses such as a metalens and a diffractive optics lens. Conventional commercial lenses are made by stacking several lenses due to the performance limitations of a single lens. As a result, many electronic devices using commercial lenses, such as smartphones 10-2 and drones 10-3, have been increasing in weight and size and have problems with poor power efficiency. To solve these problems, metalens (or singlet lenses) have recently been developed, and camera 5 including such microlenses is expected to reduce the weight of the user terminal 10 and increase power efficiency.

[0040] However, camera 5 equipped with such microlenses had the problem of reduced performance of restored images due to process limitations and theoretical limitations. In particular, camera 5 with such microlenses suffers from strong aberration due to the limitations of small and thin lenses. Furthermore, camera 5 with microlenses exhibits significant differences in restored image performance depending on the imaging space, making it difficult to use conventional general restoration models (including artificial intelligence models).

[0041] The user terminal 10, upon receiving the original image captured by camera 5, i.e., the original image with strong aberrations, includes an artificial intelligence model (reference numeral 20 in Figure 2) trained by an adversarial learning method, and can generate a restored image (hereinafter referred to as the restored image) by repairing the received original image via the trained artificial intelligence model 20.

[0042] As shown in Figure 1, the user terminal 10 can be implemented not only by a server 10-1, a smartphone 10-2, or a drone 10-3, but also by a computer or portable terminal that can receive the original image via a network. Here, "computer" includes, for example, notebook computers, desktop computers, laptop computers, tablet computers, and slate computers equipped with a web browser, and "portable terminal" includes, for example, wireless communication devices that ensure portability and mobility, and can include all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, and smartphones, as well as wearable devices such as watches, glasses, contact lenses, or head-mounted devices (HMDs). In other words, the user terminal 10 can include any of the various configurations that allow it to store the learned artificial intelligence model 20 and then generate a restored image from the original image captured by the camera 5. Next, the user terminal 10 including the camera 5 will be described with reference to an embodiment (imaging restoration device).

[0043] Figure 2 is a control block diagram of an imaging repair device according to one embodiment of the present invention.

[0044] Referring to Figure 2, the imaging repair apparatus 10 of the present invention may include a camera 5, a communication unit 11 capable of communicating with the outside, a processor 12 that generates a repaired image by inputting the original image transmitted via the camera 5 or the communication unit 11 into a trained artificial intelligence model 20, a memory 13 that stores not only the artificial intelligence model 20 but also the received original image, images necessary for training (ground truth images), or the generated repaired image, and an output unit 14 that displays the original image or the generated repaired image.

[0045] Specifically, camera 5 according to one embodiment of the present invention is an image sensor including a microlens, such as a metalensor or diffractive optical lens, which operates under the control of processor 12 to acquire a raw image with strong aberrations. Camera 5 converts the raw image into an electrical signal and transmits it to processor 12 or memory 13.

[0046] The communication unit 11 is configured to receive the original image captured externally via a microlens through a communication network. The communication unit 11 may include one or more components that enable communication with the outside world, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0047] Short-range communication modules can include a variety of modules that transmit and receive signals over short distances using wireless communication networks, such as Bluetooth® modules, infrared communication modules, RFID (Radio Frequency Identification) communication modules, WLAN (Wireless Local Access Network) communication modules, NFC communication modules, and Zigbee communication modules.

[0048] Wired communication modules can include a variety of wired communication modules such as Controller Area Network (CAN) modules, Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as a variety of cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0049] In addition to WiFi modules and WiBro (Wireless broadband) modules, wireless communication modules can also include wireless communication modules that support a variety of wireless communication methods, such as GSM (Global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

[0050] Memory 13 can be embodied in at least one of the following: a non-volatile memory element such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory; a volatile memory element such as RAM (Random Access Memory); or a storage medium such as a hard disk drive (HDD) or CD-ROM.

[0051] The output unit 14 is a display that shows the original image or the restored image, This can be provided in, but is not limited to, digital light processing (DLP) panels, plasma display panels, liquid crystal display (LCD) panels, electroluminescence (EL) panels, electrophoretic display (EPD) panels, electrochromic display (ECD) panels, light-emitting diode (LED) panels, or organic light-emitting diode (OLED) panels.

[0052] The processor 12 controls the entire imaging repair device 10, and in particular, it can train an artificial intelligence model 20, generate a repaired image using the trained artificial intelligence model 20, and display it through the output unit 14. The processor 12 may also provide the generated repaired image to a wearable device held by the user via the communication unit 11.

[0053] Specifically, the artificial intelligence model 20 can be divided into an image repair model 21 that crops the original image into pre-set size patches and generates a repaired image based on input data in which the coordinate information of the patches is embedded in the cropped patch image, and a discrimination model 22 that performs a Fourier transform on the repaired image generated by the image repair model 21 and distinguishes between the Fourier-transformed repaired images. The processor 12 improves the performance of the image repair model 21 by adversarial learning of the image repair model 21 and the discrimination model 22, and subsequently repairs the original image received from the camera 5 or the communication unit 11. A detailed explanation of the image repair method performed by the trained artificial intelligence model 20, after the processor 12 has trained it, will be described later with reference to the following diagrams.

[0054] The processor 12 can mean a data processing device embedded in hardware that has a physically structured circuit for performing functions expressed by code or instructions contained within a program. Examples of such data processing devices embedded in hardware include, but are not limited to, microprocessors, central processing units (CPUs), processor cores, multiprocessors, ASICs (application-specific integrated circuits), FPGAs (field programmable gate arrays), and GPUs (Graphics Processing Units). The processor 12 may be provided in a configuration of multiple units, each containing one or more processors.

[0055] On the other hand, the imaging repair device 10 can include a variety of other configurations in addition to the configuration described above in Figure 2, and certain configurations may be omitted as needed.

[0056] Figure 3 is an overall flowchart of the image restoration method according to one embodiment of the present invention.

[0057] Referring to Figure 3, the image restoration method first involves training the artificial intelligence model 20 (100).

[0058] The artificial intelligence model 20 is trained through adversarial learning. Specifically, within the artificial intelligence model, the image repair model 21 acts as an adversarial learning generator, generating repaired images based on raw data (training data) input from the user. The discrimination model 22 acts as an adversarial learning discriminator, performing a Fourier transform on the repaired images generated by the image repair model 21 and then comparing the Fourier-transformed repaired images with the ground truth data. Here, the ground truth data refers to the clear image quality that the user is trying to repair.

[0059] The image restoration method of the present invention trains all artificial intelligence models 20 by continuously updating (training) the image restoration model 21 and the discrimination model 22 so that the image restoration model 21 can generate a high-frequency restored image from a low-frequency original image.

[0060] Once the training of the artificial intelligence model 20 is complete, camera 5 will capture images of a new object (200).

[0061] Here, the image captured and transmitted by camera 5 is a new original image of object Ob, which is not the original image used to train the artificial intelligence model 20.

[0062] The image restoration method involves inputting the original image captured by camera 5 into a trained artificial intelligence model 20 (300).

[0063] Specifically, the original image captured by camera 5 is input only to the trained image repair model 21, which then generates a high-frequency repaired image.

[0064] The image restoration method outputs a restored image generated (output) from the artificial intelligence model 20 (400).

[0065] The method for outputting the generated repaired image can vary. If the image repair method is implemented in an imaging repair device 10 that includes a display, the repaired image can be displayed to the user through an output unit 14. If the imaging repair device 10 is provided in a configuration that does not include an output unit 14 (for example, a drone without a display), the imaging repair device 10 may transmit the generated repaired image to a wearable device such as a smartphone via a communication unit 11.

[0066] Figure 4 is a diagram illustrating an artificial intelligence model according to one embodiment of the present invention.

[0067] Referring to Figure 4, the artificial intelligence model 20 proceeds with an adversarial learning process in which the image repair model 21 generates a repaired image, the discrimination model 22 distinguishes between true and false by comparing the repaired image with the ground truth image, and the image repair model 21, with its loss function adjusted based on the discrimination result of the discrimination model 22, generates a repaired image again.

[0068] The artificial intelligence model 20 is characterized by operating the image restoration model 21 in the spatial domain and the discrimination model 22 in the Fourier domain.

[0069] Specifically, the image restoration model 21 learns the original image to be a restored image based on the patch image 211. The image restoration model 21 according to one embodiment of the present invention uses patch images 211 that are randomly cropped from the original image training data in order to overcome the limitations of the physical capacity of the GPU and other components provided in the imaging restoration device 10. That is, a patch image is a series of images in which the original image has been cropped into patch units of a size set by the user.

[0070] Conventional image restoration methods using convolution also utilize randomly cropped patch images. However, the image restoration model 21 of the present invention generates coordinate information 212 of the patch image 211 and embeds the thus generated coordinate information 212 into the patch image 211 in order to improve the restoration performance of the original image having optical aberrations.

[0071] Specifically, the image restoration model 21 generates coordinate information 212 based on the size of the patches in the patch image 211. Once coordinate information is generated for each patch, the image restoration model 21 converts the coordinate information into two-dimensional coordinate data (2D coordinate data) for the patch image 211 using a mesh grid method.

[0072] To convert the data into two-dimensional coordinate data, the image restoration model 21 includes a 1x1 convolutional layer 213. The image restoration model 21 generates a patch image 211 in which the coordinate information 212 is concatenated, i.e., input data 214, by passing the generated coordinate information 212 and patch image 211 through the 1x1 convolutional layer 213. The positional embedding method performed by the image restoration model 21 will be specifically explained by the following other diagrams.

[0073] The image restoration model 21 inputs the input data 214 into the first neural network 215. The first neural network 215 is a neural network in which the size of the input data 214 is the same as the size of the output data, i.e., the restored image. The first neural network 215 in Figure 4 is illustrated to show the skip connection, which is a basic concept of NAFNet (Nonlinear Activation Free Network). However, the first neural network 215 is not necessarily limited to NAFNet, and may include a CNN (Convolutional Neural Network) model that includes at least one MIRNet or MPRNet, or a transformer model that includes at least one Restormer or Uformer.

[0074] When the image restoration model 21 generates a restored image, the discrimination model 22 performs a Fourier transform 221 on the restored image.

[0075] Original images damaged by optical aberrations often lose many high-frequency components. Therefore, the discrimination model 22 of the present invention performs adversarial learning using the restored image with transformed domains, enabling the trained image restoration model 21 to successfully generate high-frequency information based on low-frequency information.

[0076] The Fourier-transformed restored image is input to a second neural network 222. Here, the discrimination model 22 compares the restored image to the original image and outputs whether the restored image is real or fake. The second neural network 222 can include a CNN model that contains at least one of GoogleNet, AlexNet, or VGGNet.

[0077] Figures 5 and 6 illustrate the method for generating and combining coordinate information. To avoid repetition, they are described together below.

[0078] First, referring to Figure 5, the image restoration method of the present invention crops the original image in patch units (111).

[0079] The original images included in the training data are cropped into pre-defined patch units of the same size within the same batch, and the positions are cropped randomly.

[0080] The image restoration method of the present invention generates coordinate information based on the intermediate pixels of the patch image 211 (112).

[0081] In the embodiment shown in Figure 6, the patch image 211 can be cropped at a specific location in the original image 201. The image restoration method can generate coordinate information 212 by using the pixel (0,0) at the top left corner of the patch image 211 as the reference point and the value of the midpoint (x,y).

[0082] The image restoration method converts the coordinate information 212 into 2D coordinate data using the Meshgrid method (113).

[0083] The mesh grid method is a method for transforming two-dimensional (or three-dimensional) grid coordinates based on the coordinates contained in vectors x and y. The image restoration method of the present invention converts the coordinate information 212 of the patch image 211 into two-dimensional coordinate data, thereby enabling the patch image 211 and the coordinate information 212 to be combined via a 1x1 convolutional layer 213. The mesh grid method shown in Figure 6 is an example of converting specific coordinates (x, y) into grid coordinate data.

[0084] Referring again to Figure 5, the coordinate information 212, which has been converted to 2D data, is combined with the patch image 211 via a 1x1 convolutional layer 213 (114).

[0085] Here, the combination of the patch image 211 and the coordinate information 212 is a method of combining specific sequences, which can be generated by inputting the patch image 211 and the coordinate information 212 into a 1x1 convolutional layer 213.

[0086] The image restoration method generates input data 214 by combining the patch image 211 and coordinate information 212, and generates a restored image based on this data via a first neural network 215. Here, the size of the input data and the restored image output via the first neural network 215 are the same as each other.

[0087] Figure 7 is a flowchart illustrating the method for learning using an image restoration method according to one embodiment of the present invention. Figure 8 is a diagram illustrating the discrimination process according to one embodiment. To avoid repetition in the explanation, they will be described together below.

[0088] First, referring to Figure 7, the image restoration method embeds the coordinate information 212 into the patch image 211 (110), and the image restoration model 21 generates a restored image (120).

[0089] The method for generating coordinate information 212 and combining it with the patch image 211 to generate input data (or training data) 214 has been described above in Figures 5 and 6, so it will be omitted here.

[0090] When the image restoration model 21 generates a restored image, the discrimination model 22 performs a Fourier transform on the restored image (130) and discriminates against the Fourier-transformed restored image (140).

[0091] Referring to Figure 8, the discrimination model 22 according to one embodiment can be controlled so that the result output from the second neural network 222 is output as a pre-set reference value.

[0092] Specifically, the second neural network 222 can output a value of 1 or greater if the Fourier-transformed restored image is deemed to have been restored well enough to be considered the ground truth image. If the Fourier-transformed restored image is deemed not to have been restored as well as the ground truth image, it can output a value of -1 or less. In other words, the reference values ​​set for the values ​​output by the second neural network 222 (1: first reference value, 2: second reference value) can be changed by the user. Furthermore, as shown in Figure 8, the ranges of the decision boundary and margin can be set in various ways.

[0093] Referring again to Figure 7, the image restoration method determines whether or not it has been trained through the training data. If it has not been trained (150 No), the image restoration method adjusts the loss function of the artificial intelligence model 20, and then repeats the image restoration step 120 while continuing to train.

[0094] The artificial intelligence model 20 of the present invention is updated in a direction that reduces the loss function, and the overall loss function L of the artificial intelligence model 20 of the present invention Total This is as shown in equation 1 below.

[0095]

number

[0096] Here, λ can be varied in many ways using hyperparameters pre-configured by the user.

[0097] Specifically, L PSNR This is the loss function between the Fourier-transformed restored image (the restored result) and the ground truth image. LPSNR is defined by equation 2 below.

[0098]

number

[0099] Here, \(\hat{x}\) is the result restored from the image restoration model 21, and \(x\) represents the correct image. \(R\) is the maximum pixel value of the correct image \(x\), and MSE is an index indicating the distance between \(\hat{x}\) and \(x\), which can be defined by the following formula 3.

[0100]

Number

[0101] \(L\) in Equation 1 a is the loss function \(L\) of the image restoration model 21 G and the loss function \(L\) of the discriminant model 22 D and is defined by the following formula 4.

[0102]

Number

[0103] The image restoration model is learned in the direction where the sum of the loss function \(L\) of the image restoration model 21 G and the loss function \(L\) of the discriminant model 22 D becomes the minimum.

[0104] The loss function \(L\) of the discriminant model 22 D is defined by the following mathematical formula 5, and the loss function \(L\) of the image restoration model 21 G is defined by the following formula 6.

[0105]

Number

[0106]

Number

[0107] Here, \(F\) represents the Fourier transform, and \(D\) represents the output of the discriminant model 22.

[0108] On the other hand, once training is complete (branch 150 "yes"), the image restoration method generates a new restored image from the trained image restoration model 21 (401).

[0109] Here, the new restored image refers to the restored image output by the trained image restoration model 21 from the original image newly captured by camera 5.

[0110] Figure 9 shows an example of the results obtained from the imaging system of the present invention.

[0111] The imaging system 1 can acquire images with strong aberrations, as shown in the original image 302 in Figure 9. The stronger the aberration, the more high-frequency components are lost in the original image 302.

[0112] The imaging system 1 of the present invention trains a discrimination model 22 based on the ground truth image 301 in Figure 9, and also trains an image restoration model 21 through adversarial learning. The trained imaging system 1 can output a restored image 304 for the original image 302 through the trained image restoration model 21.

[0113] Comparing image 303, restored using conventional technology, with the restored image 304, it can be seen that image 303, restored using a conventional deep learning model, failed to repair damage to the edges. However, it can be confirmed that image 304 restored using the imaging system 1 of the present invention restores the high-frequency region with greater clarity than image 303, which was restored using conventional technology.

[0114] Figure 10 is a table showing the results of a performance comparison of the image restoration method of the present invention.

[0115] The table in Figure 10 compares the image restoration method of the present invention with currently known image restoration models such as MIRNetV2, SFNet, HINet, and NAFNet. The performance of each artificial intelligence model was compared using PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index Map), and LPIPS (Learned Perceptual Image Patch Similarity).

[0116] Specifically, the image restoration method of the present invention showed a PSNR of 22.095 / 2.423, which was found to be higher than that of conventional models. SSIM and LPIPS were also found to be higher than those of conventional models, at 0.692 / 0.103 and 0.432 / 0.096, respectively.

[0117] As a result, the image restoration method and imaging system of the present invention can solve the problem of image quality degradation in microimaging systems by using an artificial intelligence model trained on image data directly captured with a lens having strong aberrations.

[0118] Furthermore, the image repair method and imaging system of the present invention can repair image damage in the high spatial frequency domain by directly learning vivid ground truth data in the frequency domain. Therefore, when the image repair method and imaging system of the present invention are installed in wearable devices such as smartphones, they solve the problem of the smartphone's camera protrusion and enable the capture of high-quality images / videos.

[0119] Furthermore, the image restoration method and imaging system of the present invention can demonstrate excellent performance in downstream applications that can be performed on images captured by cameras installed on unmanned aerial vehicles and AR (Augmented Reality) / VR (Virtual Reality) devices, based on improved image and video quality.

[0120] The image restoration method and imaging system of the present invention can achieve lighter lenses, reducing the weight of the imaging system on which it is mounted, which can lead to a reduction in the weight of unmanned aerial vehicles, drones, etc., and thereby an increase in power efficiency can also be expected. Furthermore, by mounting a microimaging system on the imaging system that acts as the eyes in AR / VR devices, the image restoration method and imaging system of the present invention can reduce the weight of the device itself, which can lead to an increase in power efficiency, significantly reducing user fatigue and improving the user experience.

[0121] On the other hand, embodiments of the present invention can be realized in the form of a recording medium that stores computer-executable instruction words. The instruction words can be stored in the form of program code, and when executed by a processor, they can generate a program module to perform the operations of the embodiments of the present invention. The recording medium can be a computer-readable recording medium.

[0122] Computer-readable recording media include all types of recording media that store computer-decipherable instruction words. Examples include ROM (read-only memory), RAM (random access memory), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0123] Embodiments of the present invention have been described above with reference to the attached drawings. A person with ordinary skill in the art to which the present invention pertains will understand that the present invention can be carried out in forms different from those of the embodiments without changing the technical idea or essence of the present invention. The embodiments of the present invention are illustrative and should not be construed as limiting. [Explanation of Symbols]

[0124] 1: Imaging System 5: Camera 10: Imaging repair device, user terminal 10-1: Server 10-2: Smartphone 10-3: Drone 11: Communications Department 12: Processor 13: Memory 14: Output section

Claims

1. Memory for storing the original image captured by the camera and the artificial intelligence model, A processor for training the aforementioned artificial intelligence model, The artificial intelligence model, An image restoration model that crops the original image into a patch of a predetermined size and generates a restored image based on input data in which the coordinate information of the patch is embedded in the cropped patch image, The image restoration model performs a Fourier transform on the restored images generated by the aforementioned image restoration model, and includes a discrimination model that distinguishes the Fourier-transformed restored images. The aforementioned processor, An imaging restoration apparatus that performs adversarial learning on the image restoration model and the discrimination model, and generates a restored image for a new original image captured by the camera from the trained image restoration model.

2. The aforementioned processor, The coordinate information is generated based on the intermediate pixels of the patch image. The coordinate information is converted into 2D coordinate data using the MeshGrid method. The imaging repair apparatus according to claim 1, which generates the input data by concatenating the coordinate information converted to 2D coordinate data with the patch image via a 1x1 convolutional layer.

3. The aforementioned processor, The imaging restoration apparatus according to claim 2, wherein the input data output from the 1x1 convolutional layer is input to a first neural network having the same size as the input data, and the restored image is output from the first neural network.

4. The first neural network described above is The imaging repair apparatus according to claim 3, comprising a CNN (Convolution Neural Network) model including at least one of MIRNet, MPRNet, or NAFNet, or a transformer model including at least one of Restorer or Uformer.

5. The aforementioned discrimination model is, The system includes a CNN model that determines the authenticity of the Fourier-transformed restored image, which includes at least one of GoogleNet, AlexNet, and VGGNet. The aforementioned processor, The imaging restoration apparatus according to claim 3, which compares the Fourier-transformed restored image with the ground truth image using the discrimination model, and causes the image restoration model to undergo adversarial training to generate a high-frequency restored image from a low-frequency original image based on the results of the comparison.

6. The aforementioned processor, The imaging repair apparatus according to claim 5, wherein the discrimination model is trained to output a value equal to or greater than a first predetermined reference value for the correct image, and to output a value equal to or less than a second predetermined reference value for the repaired image.

7. A camera including a metal lens, The camera receives the original image captured and stores the artificial intelligence model in a memory, A processor for training the aforementioned artificial intelligence model, The processor includes a display that outputs a restored image obtained by restoring the original image using a trained artificial intelligence model, The artificial intelligence model, An image restoration model that crops the original image into a patch of a predetermined size and generates a restored image based on input data in which the coordinate information of the patch is embedded in the cropped patch image, An imaging system comprising: a Fourier transform of the restored images generated by the image restoration model; and a discrimination model that distinguishes the Fourier transformed restored images.

8. The aforementioned processor, The coordinate information is generated based on the intermediate pixels of the patch image. The coordinate information is converted into 2D coordinate data using a mesh grid method. The imaging system according to claim 7, wherein the input data is generated by combining the coordinate information converted to 2D coordinate data with the patch image via a 1x1 convolutional layer.

9. The aforementioned processor, The imaging system according to claim 8, wherein the input data output from the 1x1 convolutional layer is input to a first neural network having the same size for both the input and output data, and the first neural network outputs the restored image.

10. The aforementioned processor, The imaging system according to claim 9, which compares the Fourier-transformed restored image with the ground truth image using the discriminant model, and causes the image restoration model to undergo adversarial training to generate a high-frequency restored image from a low-frequency original image based on the results of the comparison.

11. The aforementioned processor, The imaging system according to claim 10, wherein the discrimination model is trained to output a value equal to or greater than a first predetermined reference value for the correct image, and to output a value equal to or less than a second predetermined reference value for the repaired image.

12. Training an artificial intelligence model, A camera including a metalensor or diffractive optical lens acquires the original image, This includes generating a restored image of the original image using the trained artificial intelligence model, The artificial intelligence model, An image restoration model that crops images of training data into pre-set size patches and generates a restored image based on input data in which the coordinate information of the patches is embedded in the cropped patch image, The image restoration model performs a Fourier transform on the restored images generated by the aforementioned image restoration model, and includes a discrimination model that distinguishes the Fourier-transformed restored images. Training the aforementioned artificial intelligence model means An image restoration method comprising comparing the Fourier-transformed restored image with the ground truth image using the discriminant model, and adversarial training the image restoration model so that it generates a high-frequency restored image from a low-frequency original image based on the results of the comparison.

13. The image repair model generates the repaired image, The coordinate information is generated based on the intermediate pixels of the patch image, Converting the aforementioned coordinate information into 2D coordinate data using a mesh grid method, The image restoration method according to claim 12, comprising embedding the input data by combining the coordinate information converted to 2D coordinate data with the patch image via a 1x1 convolutional layer.

14. The image repair model generates the repaired image, The image restoration method according to claim 13, further comprising inputting the input data output from the 1x1 convolutional layer into a first neural network having the same size for both input and output data, and outputting the restored image from the first neural network.

15. Training the aforementioned artificial intelligence model is The image restoration method according to claim 12, which includes training the discrimination model to output a value equal to or greater than a first predetermined reference value for the correct image, and training the model to output a value equal to or less than a second predetermined reference value for the restored image.

Images