Image Restoration Apparatus And Face Image Restoration Method OF Thereof
The image restoration method using a deep learning model with face parsing and text-image alignment enhances face recognition in surveillance cameras by improving image quality in adverse weather and low-light conditions.
Patent Information
- Application Number
- KR1020250059334
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-16
AI Technical Summary
Intelligent surveillance cameras face significant challenges in maintaining accurate face recognition due to adverse weather and lighting conditions, which degrade image quality and impair performance in outdoor environments.
An image restoration method using a deep learning model that includes a face parsing module, FC-SFT module, TIA-SDE module, and fusion module to estimate statistical distributions from both image and text information, enhancing face image restoration capabilities.
The method effectively restores image quality in poor shooting environments, providing stable image analysis services by accurately enhancing face recognition in low-light and adverse weather conditions.
Smart Images

Figure PAT00032_ABST
Abstract
Description
Technology Field
[0001] The embodiments relate to an image restoration device and a method for restoring a face image of the image restoration device. Background Technology
[0003] Along with artificial intelligence technology, intelligent surveillance cameras, or intelligent CCTVs, are developing rapidly. While conventional surveillance cameras aimed solely at improving video resolution, intelligent surveillance cameras are focusing on developing "intelligent video analysis software" capable of analyzing video information to automatically detect abnormal behavior and send alerts to administrators. In other words, emphasis is placed on providing video analysis service functions that can mimic high-level human thinking abilities such as cognition, reasoning, and creativity. Currently, intelligent video analysis software technology is replacing high-level human information processing functions, including abnormal behavior detection, intrusion detection, early detection of arson, object counting, and suspect face detection.
[0004] In a hyper-connected society linked by the Internet and CCTV, facial recognition technology is necessary for authentication, security, disaster response, and the detection of abnormal behavior and objects in various devices and systems, such as traffic control, mobile devices, autonomous vehicles, and surveillance cameras.
[0005] However, since most intelligent surveillance cameras are installed outdoors, the accuracy and performance of video analysis are significantly affected by shooting conditions such as weather and lighting.
[0006] Unlike indoor shooting environments, outdoor environments involve a variety of shooting conditions. For instance, one may encounter adverse weather conditions (fog, heavy rain, heavy snow, etc.) or diverse lighting environments (low light and color temperature). These various shooting conditions in outdoor environments degrade the image quality of facial footage, thereby impairing face recognition capabilities. For instance, fog reduces the contrast of facial images, making them appear hazy, while heavy rain and snow obscure key parts of the face due to raindrops and snowflakes. Furthermore, in low-light environments, the overall low brightness causes the captured face area to appear dark. Additionally, faces captured from a distance, such as by CCTV, become blurry due to reduced resolution. The problem to be solved
[0008] The embodiment is intended to provide an image restoration method capable of accurately restoring the image quality of degraded images in various shooting environments, for the purpose of providing a stable image analysis service in poor shooting environments such as bad weather and low light conditions. means of solving the problem
[0010] A method for restoring a face image using an image restoration device may be provided, comprising the steps of: acquiring a low-quality image including a face region and text information related to the face region; acquiring a parsing map for the face region; estimating a first statistical distribution for the face region based on the parsing map through a first neural network; estimating a second statistical distribution from the text information through a second neural network; and generating a restored image with improved resolution of the low-quality image using the first statistical distribution and the second statistical distribution.
[0011] The step of estimating a second statistical distribution from the text information may include the step of estimating the mean and standard deviation of the reconstructed image from the text information through the second neural network including an image encoder and a text encoder.
[0012] The second neural network above may be a neural network trained such that the output of the image encoder for an input image and the output of the text encoder for an input text are the same.
[0013] The step of generating a restored image with improved resolution of the above low-quality image may include the step of summing the first statistical distribution and the second statistical distribution to calculate a feature map corresponding to the restored image from an arbitrary random number vector.
[0014] The step of estimating a first statistical distribution for the face region may include: a step of extracting face elements for the face region using the parsing map; and a step of estimating the mean and standard deviation for each of the face elements.
[0015] The step of estimating a first statistical distribution for the face region may include: a step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region; a step of determining a weight for each of the face elements by operating an attention function on the concatenation layer of the feature maps; and a step of obtaining the first statistical distribution by multiplying the corresponding feature maps according to the weights.
[0016] The step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region may include the step of obtaining upsampled feature maps for the current layer by statistically transforming the upsampled feature maps from the previous layer.
[0017] The image restoration device can be trained with a loss function to reduce the error between the original high-resolution image and the restored image of the image restoration device while keeping the weights of the second neural network fixed.
[0018] An image restoration device may be provided, comprising one or more processors and a memory for storing instructions, wherein when the instructions are executed by the one or more processors, the device performs the steps of: acquiring a low-quality image including a face region and text information related to the face region; acquiring a parsing map for the face region; estimating a first statistical distribution for the face region based on the parsing map through a first neural network; estimating a second statistical distribution from the text information through a second neural network; and generating a restored image with improved resolution of the low-quality image using the first statistical distribution and the second statistical distribution. Effects of the invention
[0020] The embodiment can provide an image restoration method capable of accurately restoring the image quality of degraded images in various shooting environments to provide a stable image analysis service in poor shooting environments such as bad weather and low light conditions. Brief explanation of the drawing
[0022] FIG. 1 is a diagram illustrating the input and output of an image restoration device in an embodiment. FIG. 2 is a structural diagram for explaining the structure and operation of an image restoration device in an embodiment. Figure 3 is a diagram illustrating a method for estimating a parsing map in an embodiment. Figure 4 is a drawing illustrating the detailed structure of the FC-SFT module in an embodiment. FIG. 5 is a diagram illustrating the structure and operation of the TIA-SDE module in an embodiment. FIG. 6 is a flowchart illustrating a face image restoration method of an image restoration device in an embodiment. FIG. 7 is a diagram illustrating an example of image restoration of an image restoration device in an embodiment. FIG. 8 is a diagram illustrating the configuration of an image restoration device in an embodiment. Specific details for implementing the invention
[0023] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, various modifications may be made to the embodiments, and thus the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, and substitutions to the embodiments are included within the scope of the rights.
[0024] The terms used in the embodiments are for illustrative purposes only and should not be interpreted as intended to be limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0025] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the embodiments pertain. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0027] In addition, when describing with reference to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the embodiments, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the embodiments, such detailed description is omitted.
[0028] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments. These terms are intended merely to distinguish the components from other components, and the nature, order, or sequence of the components is not limited by these terms. Where it is stated that a component is "connected," "combined," or "connected" to another component, it should be understood that while the component may be directly connected or connected to the other component, another component may also be "connected," "combined," or "connected" between each component.
[0029] Components included in any one embodiment and components having common functions shall be described using the same names in other embodiments. Unless otherwise stated, the description in any one embodiment may also apply to other embodiments, and specific descriptions shall be omitted to the extent of overlap.
[0031] FIG. 1 is a diagram illustrating the input and output of an image restoration device in an embodiment.
[0032] The image restoration device (100) is intended to restore a degraded image into a high-quality image by overcoming various weather and lighting environments, such as bad weather and low light environments.
[0033] The image restoration device (100) is an input random number vector is the final restored video It operates to convert to. At this time, the original high-resolution image To convert the low-resolution image into the mean and standard deviation, which are the statistical distributions of And original high-resolution video or text information depending on the training and testing phases It requires additional information. Here, the input low-resolution video may include video provided through a camera or various devices, such as a vehicle dashcam or a drone. Additionally, the original high-resolution video or text may be set as an option depending on the situation. Therefore, the original high-resolution video may be used during the training phase, and a text sentence may be received as input during the test phase.
[0034] The image restoration device (100) is implemented as a deep learning model, and the parameters of the image restoration device (100) can be learned using a random number vector, a low-quality image, and an original high-quality image. At this time, the learning can be performed using a loss function to reduce the error between the original high-quality image and the restored image of the image restoration device (100). For example, mean squared error, adversarial loss, perception loss, etc., can be used, and optimization techniques such as Adam (Adaptive Moment Estimation) and SGD (Stochastic Gradient Descent) can be applied to update the model parameters.
[0035] Based on learned parameters, the image restoration device (100) can restore a final restored image from inputs of a random number vector, a low-quality image, and text information. The input random number vector can be considered as a function that maps to the output image of the image restoration device (100).
[0037] FIG. 2 is a structural diagram for explaining the structure and operation of an image restoration device in an embodiment.
[0038] The image restoration device (100) may have a structure including a face parsing module (110), an FC-SFT module (120), a TIA-SDE module (130), and a fusion module (140) as core modules to enhance face image restoration capabilities.
[0039] The face parsing module (110) receives a low-quality image (201) as input and can estimate a parsing map (202) for the image.
[0040] The FC-SFT (Fully Connected-Statistical Feature Transformation) module (120) can calculate statistical distributions, such as the mean and standard deviation, from a low-quality image (201) and a parsing map (202), and perform statistical feature transformation. Images downsampled at different rates for the input low-quality image (201) can be processed in multiple layers of the FC-SFT module (120). The operation of the FC-SFT module (120) will be explained in detail later.
[0041] The TIA-SDE (Text-Image Alignment Statistical Distribution Extraction) module (130) can estimate the statistical distribution of a target from input text by utilizing a text-image alignment model. To this end, when training the TIA-SDE module (130), it can be trained so that the mean and standard deviation estimated from the original high-resolution image match the mean and standard deviation estimated from the input text information.
[0042] Through the TIA-SDE module (130), the standard deviation and mean, which are statistical distributions for the target image, are extracted from the text information to supplement the statistical distribution estimated from the FC-SFT module (120). The text information corresponds to text information related to the overall face image, such as skin color, hairstyle, gender, and whether glasses are worn, and can help restore lost image information.
[0043] As illustrated, the FC-SFT module (120) and the TIA-SDE module (130) can be implemented in multiple layers, and as the number of layers increases, the accuracy of the statistical distribution can be increased through stepwise upsampling.
[0044] The fusion module (140) can perform the role of performing a final statistical feature transformation by fusing the mean and standard deviation, which are the outputs of the TIA-SDE module (130) and the FC-SFT module (120). The TIA-SDE module (130) can output the mean and standard deviation for face components, and this mean and standard deviation can be converted into a single mean and standard deviation through the fusion module (140) and applied to the input feature map. Therefore, the fusion module (140) can estimate a feature map corresponding to the final reconstructed image (205) from the random number vector (204) by summing the mean and standard deviation output from the two modules.
[0046] Figure 3 is a diagram illustrating a method for estimating a parsing map in an embodiment.
[0047] The face parsing module (300) is a parsing map that divides face components such as eyes, nose, and mouth from an input low-quality image. It can generate.
[0048] FIG. 3 shows an example of the operation result of the face parsing module (300). It can be seen that face components such as eyes, nose, and mouth are separated by region. The face parsing module (300) can be implemented as an encoder-decoder type deep learning model. It can be implemented as a representative UNet or FCN model.
[0049] In the embodiment, the face parsing module (300) is not implemented within the model but is implemented as a separate module and can be implemented in a form that obtains a parsing map from the image restoration device (100).
[0051] Figure 4 is a drawing illustrating the detailed structure of the FC-SFT module in an embodiment.
[0052] Figure 4(a) is the structure of an FC-SFT module, and Figure 4(b) is intended to explain the structure of an SFT fusion block among FC-SFT modules.
[0053] The FC-SFT module plays the role of estimating the statistical distribution of facial elements by separately extracting regions for facial features such as eyes, nose, and mouth based on the parsing map.
[0054] According to FIG. 4(a), the FC-SFT module may be configured to include a face element extraction block (410), an SDE block (420), and an SFT fusion block (430).
[0055] The face element extraction block (410) can crop the eye, nose, and mouth regions in the estimated parsing map. The cropping method includes the process of extracting regions for rectangular eyes, nose, and mouth using the maximum and minimum values of the left, right, top, and bottom coordinates of the eye, nose, and mouth regions of the parsing map.
[0056] Examples of regions for each of the face elements, such as eyes, nose, and mouth, are shown as outputs of the face element extraction block (410) of FIG. 4. For example, the eye region can be assumed to be a region including eyebrows. Regions for many face components, such as hair, ears, and cheekbones, including eyes, nose, and mouth, can be extracted.
[0057] The SDE block (420) can estimate the mean and standard deviation required for statistical feature transformation for the parsing map of the entire face, the input low-resolution image, and each cropped parsing map and face element.
[0058] The SDE block (420) can estimate a statistical distribution for each index of the face element. By estimating the statistical distribution for the downsampled low-quality input image and the parsing map or the cropped parsing map and the face element, the mean and standard deviation for each face element can be derived.
[0059] The actual SDE block (420) can be implemented with a convolution layer and a ReLU layer.
[0060] The SFT fusion block (430) can calculate a single average and standard deviation value by summing the average and standard deviation extracted for the entire face and for each face element. The SFT fusion block (430) can be configured as shown in FIG. 4(b).
[0061] The operation of the SFT fusion block (430) can be expressed by the following mathematical formula.
[0062] [Mathematical Formula 1]
[0063]
[0064] represents the standard deviation and mean calculated in the i-th FC-SFT module for index j representing the face element. represents the operation function of the SFT fusion block, and is generated from the random number vector presented in Fig. 2 This is the nth feature map. is an upsampling layer and serves to increase the resolution of the feature map of the previous layer.
[0065] Upsampled input features and estimated statistics Using to get the final standard deviation and average It can generate. In this block, the SFT layer uses the input features of the previous layer. Converts to the standard deviation and mean of the current hierarchy.
[0066] The SFT layer of Fig. 4(b) can be computed as follows.
[0067] [Mathematical Formula 2]
[0068]
[0069] μ and σ are upsampled These are the mean and standard deviation per channel. Through the statistical feature transformation of Equation 2, the input feature map Statistical distribution of target videos feature map that becomes It can be converted into.
[0070] Statistical feature transformed feature maps for the entire face and by facial elements After stacking along the channel axis through a concatenation layer, the weights can be calculated by passing through an attention function. Here, the attention function can be implemented through convolution, ReLU, and sigmoid layers.
[0071] Weights predicted through the attention function is an input feature map through skip connection Create the final output feature map through element-wise multiplication and element-wise sum, and pass it through the SDE block to fuse the statistical distributions of the entire face and face elements into a single standard deviation. and average It can be converted into.
[0073] FIG. 5 is a diagram illustrating the structure and operation of the TIA-SDE module in an embodiment.
[0074] In the embodiments, the effectiveness of image restoration can be increased by adding a module for statistical feature transformation based on text-image alignment. This module is intended to extract the statistical distribution of the restored image from input text information.
[0075] The mean and standard deviation, which are the outputs of the TIA-SDE module (500), can be combined with the mean and standard deviation of the FC-SFT module to determine the statistical distribution of the final target reconstructed image.
[0076] As described, the TIA-SDE module (500) operates differently in the learning phase and the testing phase (inference phase).
[0077] The path passing through the image encoder (510) operates during the learning phase, and the path passing through the text encoder (520) can operate during the testing phase. During the learning phase, the TIA-SFT module (500) inputs the original high-resolution image to the pre-trained image encoder (510) to map it into an embedding space, and then passes it through the fully connected layer so that the weights of the fully connected layer can be learned to output the standard deviation and mean of the original high-resolution image. The image encoder and the text encoder are aligned. That is, they can be pre-trained so that the output of the image encoder (510) and the output of the text encoder (520) match.
[0078] Since there is no high-resolution image in the test phase, text information is used as input. The standard deviation and average of the target high-resolution image can be output by passing through a pre-trained text encoder (520) and a fully connected layer.
[0079] The image encoder (510) and text encoder (520) of the TIA-SDE module (500) are pre-trained, so the weights of the model are not changed when the image restoration device is trained.
[0080] The TIA-SDE module (500) can enhance the ability to transform statistical features because it extracts statistics from the original high-resolution image during the training phase and extracts them from the input text during the test phase.
[0082] FIG. 6 is a flowchart illustrating a face image restoration method of an image restoration device in an embodiment.
[0083] In step (610), the image restoration device obtains a low-quality image below the standard including a face region and text information related to the face region.
[0084] Low-resolution video may include footage taken in adverse weather conditions such as heavy rain or snow, or footage whose quality has been degraded for various other reasons. Low-resolution video may include facial areas. Text information may include phrases describing the overall style, such as skin color, hairstyle, gender, and whether glasses are worn.
[0085] In step (620), the image restoration device obtains a parsing map for the face region.
[0086] The face restoration device can acquire a parsing map of the face region within a low-resolution image using a face parsing module. The face parsing module may be included in the image restoration device or included as a separate device, allowing the face restoration device to operate by receiving the parsing map.
[0087] In step (630), the image restoration device estimates a first statistical distribution for a face region based on a parsing map through a first neural network.
[0088] The first neural network may include the FC-SFT module described above.
[0089] The image restoration device can calculate a statistical distribution for restoring a low-quality image by using a connection layer of the low-quality image and the parsing map. As described through FIG. 4, the mean and standard deviation for the target high-quality image can be estimated using a feature map that has been progressively upsampled by a predetermined number of steps.
[0090] In step (640), the image restoration device estimates a second statistical distribution from text information through a second neural network.
[0091] The image restoration device can estimate the mean and standard deviation targeting the restored image from the input text information.
[0092] The second neural network may include a pre-trained TIA-SDE module. When training the second neural network, it may be trained so that the mean and standard deviation estimated from the original high-resolution image match the mean and standard deviation estimated from the text information.
[0093] In step (650), the image restoration device generates a restored image with improved resolution of the low-quality image using the first statistical distribution and the second statistical distribution.
[0094] In the embodiment, the first statistical distribution and the second statistical distribution can be summed to progressively derive the final statistical distribution. The first statistical distribution estimated from the low-resolution input image can be supplemented using the second statistical distribution estimated from the text information.
[0095] The final statistical distribution can be summed as shown in mathematical formula 3 below.
[0096] [Mathematical Formula 3]
[0097]
[0098] Here, is the standard deviation and mean by the second neural network. According to Equation 3, the standard deviation and mean extracted from low-quality images can be added to the standard deviation and mean extracted from text without degradation.
[0099] In other words, it can be interpreted as multimodal fusion estimated from images and text. Although the fusion process in the formula is expressed as element-wise addition, it can be implemented through various fusion techniques, such as stacking layers like concatenation and convolution.
[0101] FIG. 7 is a diagram illustrating an example of image restoration of an image restoration device in an embodiment.
[0102] The first and second rows of Figure 7 show the input text and the corresponding original face images used during the test phase. It can be seen that the text accurately describes the style of the input images, such as hair color and components (eyes, nose, mouth). It is predicted that using this text information will allow for the effective extraction of statistical distributions from text embedding vectors aligned with the original images. For reference, the text is provided along with the images in the MM-CelebA-HQ dataset.
[0103] The input face rain image placed in the third row of Figure 7 is the result of applying a blind image degradation model. As shown, differences in resolution, rain streaks, and the intensity of the rain can be observed. It can be seen that the existing rain removal in the resulting image, the HiFaceGAN in the fourth row, and the FSRNet model in the fifth row failed to restore the high-resolution image. In other words, it can be seen that the texture is distorted or the color representation ability is degraded in the resulting image. Additionally, it can be seen that rain streaks or blurring were not completely removed.
[0104] On the other hand, as shown in the sixth row, it can be seen that the PSFRGAN model and the proposed model show relatively improved sharpness or effective removal of rain streak defects. However, when comparing the PSFRGAN model to the proposed model, as seen in the red box area, the detail of the hair is reduced or the sharpness of the facial components is relatively lower. Additionally, texture distortion can be observed in the face area. The colors also appear to have a reddish tint compared to the original.
[0105] In the example, since the restored image by the image restoration device of the seventh row extracts a statistical distribution from texture information, it can be observed that the facial elements, such as the eyes, nose, and mouth, are expressed more clearly and accurately.
[0107] FIG. 8 is a diagram illustrating the configuration of an image restoration device in an embodiment.
[0108] The image restoration device (600) includes a memory (850), one or more processors (830) and a communication interface (810), and may include one or more programs stored in the memory (850) and configured to be executed by the one or more processors (830).
[0109] The communication interface (810) receives low-quality video and text information. Alternatively, it may receive a parsing map for the low-quality video.
[0110] The communication interface (810) is for transmitting and receiving data wirelessly or via wired means and can be implemented in the form of a wireless interface such as Wi-Fi, Bluetooth, ZigBee, or LoRa, a wired interface such as Ethernet, USB (Universal Serial Bus), NFC (Near Field Communication), etc. The communication interface (810) may include a user interface for receiving input from a user.
[0111] The processor (830) estimates a first statistical distribution and a second statistical distribution from low-quality video and text information received through the communication interface (810) using a learned neural network. The processor (830) can derive a statistical distribution corresponding to the restored image by summing the first statistical distribution and the second statistical distribution, and generate the restored image.
[0112] The memory (850) can store various information generated during the processing of the processor (830) described above. In addition, the memory (850) can store various data and programs. The memory (850) may include volatile memory or non-volatile memory. The memory (850) may store various data by being equipped with a large-capacity storage medium such as a hard disk.
[0113] Additionally, the processor (830) may perform at least one method or an algorithm corresponding to at least one method described above through FIGS. 1 to 7. The processor (830) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program. The processor (630) may be composed of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a NPU (Neural Network Processing Unit). For example, the image restoration device (800) implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or a FPGA (Field Programmable Gate Array).
[0114] The processor (830) can execute a program and control the image restoration device (800). The program code executed by the processor (830) can be stored in memory (850).
[0116] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may contain program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.
[0118] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.
[0120] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.
[0121] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.
Claims
Claim 1 A method for restoring a face image using an image restoration device, comprising: a step of obtaining a low-quality image including a face region and text information related to the face region; a step of obtaining a parsing map for the face region; a step of estimating a first statistical distribution for the face region based on the parsing map through a first neural network; a step of estimating a second statistical distribution from the text information through a second neural network; and a step of generating a restored image with improved resolution of the low-quality image using the first statistical distribution and the second statistical distribution. Claim 2 A face image restoration method according to claim 1, wherein the step of estimating a second statistical distribution from the text information includes the step of estimating the mean and standard deviation of the restored image from the text information through the second neural network including an image encoder and a text encoder. Claim 3 A face image restoration method according to claim 1, wherein the second neural network is a neural network trained such that the output of the image encoder for an input image and the output of the text encoder for an input text are identical. Claim 4 A face image restoration method according to claim 1, wherein the step of generating a restored image with improved resolution of the low-quality image comprises the step of calculating a feature map corresponding to the restored image from an arbitrary random number vector by summing the first statistical distribution and the second statistical distribution. Claim 5 A method for restoring a face image according to claim 1, wherein the step of estimating a first statistical distribution for the face region comprises: a step of extracting face elements for the face region using the parsing map; and a step of estimating the mean and standard deviation for each of the face elements. Claim 6 A method for restoring a face image according to claim 1, wherein the step of estimating a first statistical distribution for the face region comprises: a step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region; a step of determining a weight for each of the face elements by operating an attention function on the concatenation layer of the feature maps; and a step of obtaining the first statistical distribution by multiplying the corresponding feature maps according to the weights. Claim 7 A face image restoration method according to claim 6, wherein the step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region comprises the step of obtaining upsampled feature maps for the current layer by statistically transforming the upsampled feature maps from the previous layer. Claim 8 A face image restoration method according to claim 1, wherein the image restoration device is trained with a loss function to reduce the error between the original high-resolution image and the restored image of the image restoration device while keeping the weights of the second neural network fixed. Claim 9 A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 through 8. Claim 10 An image restoration device comprising one or more processors and a memory for storing instructions, wherein when the instructions are executed by the one or more processors, the device performs the steps of: acquiring a low-quality image including a face region and text information related to the face region; acquiring a parsing map for the face region; estimating a first statistical distribution for the face region based on the parsing map through a first neural network; estimating a second statistical distribution from the text information through a second neural network; and generating a restored image with improved resolution of the low-quality image using the first statistical distribution and the second statistical distribution. Claim 11 An image restoration device according to claim 10, wherein the step of estimating a second statistical distribution from the text information includes the step of estimating the mean and standard deviation of the restored image from the text information through the second neural network comprising an image encoder and a text encoder. Claim 12 An image restoration device according to claim 10, wherein the second neural network is a neural network trained such that the output of the image encoder for an input image and the output of the text encoder for an input text are identical. Claim 13 An image restoration device according to claim 10, wherein the step of generating a restored image with improved resolution of the low-quality image comprises the step of summing the first statistical distribution and the second statistical distribution to calculate a feature map corresponding to the restored image from an arbitrary random number vector. Claim 14 An image restoration device according to claim 10, wherein the step of estimating a first statistical distribution for the face region comprises: a step of extracting face elements for the face region using the parsing map; and a step of estimating the mean and standard deviation for each of the face elements. Claim 15 In claim 10, the step of estimating a first statistical distribution for the face region comprises: a step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region; a step of determining a weight for each of the face elements by operating an attention function on the concatenation of the feature maps; and a step of obtaining the first statistical distribution by multiplying the corresponding feature maps according to the weights. Claim 16 An image restoration device according to claim 15, wherein the step of obtaining upsampled feature maps using a statistical distribution for each of the face elements of the face region comprises the step of obtaining upsampled feature maps for the current layer by statistically transforming the upsampled feature maps from the previous layer. Claim 17 In claim 10, the image restoration device is an image restoration device that is trained with a loss function to reduce the error between the original high-resolution image and the restored image of the image restoration device while keeping the weights of the second neural network fixed.
