Video restoration method and apparatus

Abstract

To provide a video restoration method and apparatus.SOLUTION: A video restoration method includes the steps of: receiving a deteriorated video from a camera; determining deterioration information indicating deterioration elements of the deteriorated video; tuning the deterioration information in accordance with a tuning condition; and executing a video restoration network with the deteriorated video and the deterioration information to generate a restored video corresponding to the deteriorated video. A video restoration apparatus includes a processor and a memory including an instruction word to be executed on the processor, and executes the video restoration method when the instruction word is executed on the processor.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The following embodiments relate to a video restoration method and apparatus.

Background Art

[0002] A camera, which is a device for shooting videos, is widely installed and used in various electronic devices. In a mobile device such as a smartphone, a camera is an essential component, becoming more and more high-performance over time, and its size has been miniaturized. Generally, a smartphone includes a front camera and a rear camera. The front camera is disposed in the upper region of the smartphone and is often used for taking selfies. A UDC (under display camera) system provides a camera hidden behind a display panel.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The purpose of the following embodiments is to provide a video restoration method and apparatus.

Means for Solving the Problems

[0004] According to one embodiment, a video restoration method includes receiving a degraded video from a camera, determining degradation information indicating degradation elements of the degraded video, tuning the degradation information according to tuning conditions, and running a video restoration network with the degraded video and the degradation information to generate a restored video corresponding to the degraded video.

[0005] The degradation information may include at least one of a noise parameter and a blur parameter. The tuning conditions may include a user preference regarding a removal level of the degradation elements.

[0006] The step of generating the restored video may include inputting input data corresponding to the degraded video into the video restoration network, and adjusting the output data of at least one layer of the video restoration network to map data corresponding to the degradation information.

[0007] The step of determining the degradation information may include a step of analyzing the degraded video and determining a noise parameter that indicates the level of noise contained in the degraded video. The step of tuning the degradation information may further include a step of tuning the noise parameter based on environmental information of the degraded video. The step of tuning the noise parameter may include a step of tuning the noise parameter to indicate a high noise level when the environmental information corresponds to a low-light environment, and a step of tuning the noise parameter to indicate a low noise level when the environmental information corresponds to a high-light environment.

[0008] The camera is an under-display camera (UDC), and the step of determining the degradation information may include obtaining blur parameters corresponding to the hardware characteristics of the UDC. The UDC receives light through holes between display pixels of a display panel, and the hardware characteristics may include at least one of the size, shape, depth, and arrangement pattern of the holes. The blur parameters may include at least one of a first parameter indicating the intensity of the blur, a second parameter indicating the spacing between the double images, and a third parameter indicating the intensity of the double images.

[0009] According to one embodiment, the video restoration device includes a processor and a memory containing instructions that can be executed by the processor. When the instructions are executed by the processor, the processor receives degraded video from a camera, determines degradation information indicating the degradation elements of the degraded video, tunes the degradation information according to tuning conditions, and executes a video restoration network with the degraded video and the degradation information to generate a restored video corresponding to the degraded video.

[0010] According to one embodiment, the electronic device includes a camera and a processor that receives degraded video from the camera, determines degradation information indicating the degradation elements of the degraded video, tunes the degradation information according to tuning conditions, and executes a video restoration network with the degraded video and the degradation information to generate a restored video corresponding to the degraded video.

[0011] According to one embodiment, the electronic device includes a camera and a processor that receives degraded video captured by the camera, estimates the amount of noise contained in the degraded video via noise parameters indicating noise information for each pixel in the degraded video, and generates a restored video by executing a deep neural network based on the degraded video and the noise parameters.

[0012] The processor can generate the restored image by removing noise elements corresponding to the noise parameters from the degraded image. The processor can generate a noise map corresponding to the noise parameters and input the noise map to the deep neural network to adjust the output of the layers of the deep neural network. The processor can tune the noise parameters based on tuning conditions and apply the tuned noise parameters to the deep neural network. [Effects of the Invention]

[0013] According to the present invention, a method and apparatus for restoring video can be provided. [Brief explanation of the drawing]

[0014] [Figure 1] A schematic diagram of the restoration process for degraded video according to one embodiment is shown. [Figure 2] This shows the process of restoring degraded video using noise parameters according to one embodiment. [Figure 3] The tuning process for noise parameters according to one embodiment is shown. [Figure 4] This shows the process of restoring degraded video using blur parameters according to one embodiment. [Figure 5] This document shows the process of simulating the point diffusion function for the hole pattern of a UDC according to one embodiment. [Figure 6] This shows the process of restoring degraded video using all noise and blur parameters according to one embodiment. [Figure 7] This shows the training process of a video restoration network according to one embodiment. [Figure 8] A video restoration method according to one embodiment is shown. [Figure 9] The configuration of a video restoration device according to one embodiment is shown. [Figure 10] An electronic device providing a UDC according to one embodiment is shown. [Figure 11] The arrangement of the display panel and UDC according to one embodiment is shown. [Figure 12] The configuration of an electronic device according to one embodiment is shown. [Modes for carrying out the invention]

[0015] The specific structural or functional descriptions disclosed herein are illustrative for the purpose of illustrating embodiments, and embodiments can be carried out in various different forms. The present invention is not limited to the embodiments described herein, and the scope of the present invention includes modifications, equivalents, or substitutions that are included in the technical ideas described in the embodiments.

[0016] Terms such as "first" or "second" may be used to describe multiple components, but such terms should only be construed for the purpose of distinguishing one component from another. For example, the first component can be named the second component, and similarly, the second component can also be named the first component.

[0017] When it is mentioned that any component is "connected" or "attached" to another component, it should be understood that it is directly connected or attached to the other component, but there may be other components in between.

[0018] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprising" or "having" indicate the presence of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should not be construed as precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0019] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this embodiment belongs. Commonly used pre-defined terms should be construed to have a meaning consistent with the meaning in the context of the relevant art and should not be construed in an idealized or overly formal sense unless clearly defined herein.

[0020] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be assigned to the same components regardless of the reference signs, and redundant descriptions thereof will be omitted.

[0021] Figure 1 schematically shows the process of restoring a degraded image according to one embodiment. Referring to Figure 1, the image restoration device 100 generates a restored image 120 by removing image quality degradation elements from the degraded image 110 through image restoration. Here, the image restoration device 100 can remove degradation elements by considering the degradation causes and degradation patterns specific to the degraded image 110. For example, the degraded image 110 may be UDC (under display camera) image, and degradation elements specific to UDC may be considered in the image restoration.

[0022] The video restoration device 100 may include a video restoration network 101. The video restoration network 101 is a deep neural network (DNN) with multiple layers. The multiple layers include an input layer, a hidden layer, and an output layer. The neural network may include a fully connected network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. For example, some of the multiple layers in the neural network may be a CNN, and other parts may be an FCN. In this case, the CNN may be called a convolutional layer, and the FCN may be called a fully connected layer. The neural network may also include residual connections.

[0023] In the case of a CNN, the data input to each layer may be called an input feature map, and the data output from each layer may be called an output feature map. The input feature map and output feature map may also be called activation data. If the convolutional layer is the input layer, the input feature map of the input layer may be the input image.

[0024] Neural networks, after being trained based on deep learning, can perform inference suitable for the training purpose by mapping non-linearly related input and output data to each other. Deep learning is a machine learning method for solving problems such as image or speech recognition from big datasets. Deep learning is understood as the process of solving an optimization problem by training a neural network using prepared training data to find the point where energy is minimized.

[0025] Through supervised or unsupervised deep learning, the structure of a neural network, or the weights corresponding to the model, are determined, and input and output data are mapped to each other through these weights. If the width and depth of the neural network are sufficiently large, it can have enough capacity to realize any function. If the neural network learns a sufficiently large amount of training data through an appropriate training process, it can achieve optimal performance.

[0026] A neural network is described as being "pre-trained," where "pre-trained" refers to the state before the neural network is "started." The "starting" of a neural network means that it is ready for inference. For example, the "starting" of a neural network may include the neural network being loaded into memory, or the neural network being loaded into memory and then receiving input data for inference.

[0027] The video restoration device 100 determines degradation information 111 indicating the degradation elements of the degraded video 110, and generates a restored video 120 by executing the video restoration network 101 with the degraded video 110 and the degradation information 111. For example, the degradation information 111 may include noise parameters indicating the noise elements of the degraded video 110 and blur parameters indicating the blur elements of the degraded video 110. The video restoration network 101 may be inferred by executing the video restoration network 101.

[0028] The video restoration device 100 inputs the degraded video 110 and degradation information 111 as input data to the video restoration network 101 to generate the restored video 120. Alternatively, the video restoration device 100 may input the degraded video 110 as input data to the video restoration network 101 and generate the restored video 120 by adjusting the output of the layers of the video restoration network 101 based on the degradation information 111. For example, the video restoration device 100 can adjust the output of any layer of the video restoration network 101 as map data corresponding to the degradation information 111. Here, the video restoration network 101 may use the information of the restored video 120 via an attention mechanism.

[0029] The video restoration device 100 can tune and use the degradation information 111 according to tuning conditions. Tuning may include scaling the parameter values ​​of the degradation information 111 by a certain ratio, adding a certain value to the parameter values, clipping the parameter values ​​to a certain value, or assigning a specific value to the parameter values. For example, when noise reduction is performed, environmental information of the environment in which the degraded video 110 was filmed may be considered as a tuning condition. Degraded video 110 was filmed in various environments, but when noise is removed from the degraded video 110 using a single neural network, the noise may not be removed appropriately depending on the noise intensity. For example, when a neural network trained to remove noise from low-light video removes noise from high-light video, over-smoothing may occur during over-smoothing restoration. Conversely, when a neural network trained to remove noise from high-light video removes noise from low-light video, noise may remain in the video.

[0030] Therefore, instead of using a single neural network for the image restoration network 101, the image restoration device 100 may tune and use the degradation information 111 as needed. For example, if the degraded image 110 was shot in a low-light environment, the image restoration network 101 can tune the degradation information 111 to perform more noise reduction so that no noise remains in the degraded image 110. If the noise parameter is tuned to show a high noise level through scaling or addition, the image restoration network 101 will consider the degraded image 110 to have more noise than the actual noise and can perform noise reduction work to address more noise. Conversely, if the degraded image 110 was shot in a high-light environment, the image restoration device 100 can tune the noise parameter to show a low noise level to prevent over-smoothing.

[0031] As a different example, user preference regarding the level of degradation removal may be considered as a tuning condition. For example, some users may prefer older-looking images with visible noise, while others may prefer clean images with little to no noise. Similarly, some users may prefer a softer look with some blur, while others may prefer a sharper look with little to no blur. Therefore, user preference may include preferred noise level and preferred blur level. The image restoration device 100 can adjust the noise and / or blur parameters according to the user's preference, thereby adjusting the level of noise and / or blur removal by the image restoration network 101.

[0032] Figure 2 shows the process of restoring a degraded image using noise parameters according to one embodiment. Referring to Figure 2, the image restoration device analyzes the degraded image 210 and estimates the level of noise contained in the degraded image 210. The image restoration device estimates the noise level via the noise parameter 211. For example, the image restoration device shows the noise level as a normal distribution N(0, σ), where σ is the noise parameter 211. The noise parameter 211 indicates noise information for each pixel of the degraded image 210 via parameter values ​​corresponding to each pixel. For example, the first parameter value σ1, which indicates the noise level of the first pixel of the degraded image 210, and the second parameter value σ2, which indicates the noise level of the second pixel, may be different.

[0033] The video restoration device generates restored video 220 by executing a video restoration network 201 based on the degraded video 210 and noise parameters 211. For example, the video restoration network 201 may generate restored video 220 by removing noise components corresponding to noise parameters 211 from the degraded video 210. To this end, the noise parameters 211 are converted into a map form. More specifically, the video restoration device generates a noise map corresponding to the noise parameters 211, and the video restoration network 201 can adjust the output of any layer of the video restoration network 201 (e.g., input layer, hidden layer, or output layer) to the noise map. For example, the video restoration network may remove noise components shown via the noise map from the output via an operation between the output and the noise map. The noise parameters 211 may be converted into a noise map showing spatial information to indicate the noise level per pixel of the degraded video 210.

[0034] The video restoration device tunes the noise parameter 211 according to the tuning conditions and applies the tuned noise parameter 211 to the video restoration network 201. If a noise map is used, tuning the noise parameter 211 includes tuning the noise map. More specifically, the video restoration device may, depending on the tuning conditions, tune the noise parameter 211, convert the tuned noise parameter 211 into a noise map, and apply it to the video restoration network 201, or it may first convert the noise parameter 211 into a noise map and then tune the noise map according to the tuning conditions.

[0035] Tuning conditions may include environmental information of the degraded video 210 and / or user preferences. The video restoration device may determine environmental information via metadata of the degraded video 210. For example, the environmental information may include illuminance information indicating the illuminance at which the degraded video 210 was filmed. The video restoration device classifies illuminances above a threshold as high illuminance and illuminances below the threshold as low illuminance. Low illuminance means an illuminance level where there is so little light that it is difficult for a person to identify objects. High illuminance is a term used to distinguish it from low illuminance and is understood to mean general illuminance at which a person can easily identify objects, rather than extremely high illuminance at which saturation occurs. High illuminance may also be called general illuminance. User preferences may be stored in the video restoration device as set values. For example, user preferences may include preferred noise level and preferred blur level.

[0036] Figure 3 shows the noise parameter tuning process according to one embodiment. Referring to Figure 3, the video restoration device determines the noise parameter 311 in step S310 and tunes the noise parameter 311 according to the tuning conditions in step S320. The degradation information may also include blur parameters in addition to the noise parameter 311, and the following description of the noise parameter 311 may also apply to the blur parameter.

[0037] The video restoration device can determine a first tuned noise parameter 321 by tuning the noise parameter 311 according to a first tuning condition, and determine a second tuned noise parameter 322 by tuning the noise parameter 311 according to a second tuning condition. The video restoration device executes the video restoration network with the tuned noise parameter other than noise parameter 311 and / or the tuned noise parameter. Therefore, depending on the tuning condition, the effect of applying other data to the video restoration network can be obtained.

[0038] According to one embodiment, the tuning conditions may include environmental information. If the environmental information corresponds to a low-light environment, the video restoration device may tune the noise parameter 311 to show a high noise level, thereby determining the first tuned noise parameter 321. If the environmental information corresponds to a high-light environment, the video restoration device may tune the noise parameter 311 to show a low noise level, thereby determining the second tuned noise parameter 322. For example, if the noise parameter 311 shows 10% noise, the first tuned noise parameter 321 shows 15% noise, and the second tuned noise parameter 322 shows 5% noise. Since the noise parameter 311 may include other parameter values ​​for each pixel of the degraded video, the 10%, 15%, and 5% noise represent the average noise of the entire video, respectively. The video restoration device may adjust the parameter values ​​of each pixel to a certain ratio, adjusting the 10% average noise to 15% or 5%. Therefore, the video restoration network can remove more noise from the degraded video in accordance with the first tuned noise parameter 321.

[0039] In another embodiment, the tuning conditions may include user preferences. For example, user preferences may include preferred noise level and preferred blur level, and the video restoration device may tune the noise parameter 311 according to the preferred noise level. More specifically, if the user prefers noisy, aged-looking video, the video restoration device can tune the noise parameter 311 to show a low noise level, and if the user prefers clean video with little noise, the video restoration device can tune the noise parameter 311 to show a high noise level. In another embodiment, the tuning conditions may include environmental information and user preferences, and the video restoration device can tune the noise parameter 311 considering both conditions.

[0040] When a noise map is used, the video restoration device can generate a noise map corresponding to the noise parameter 311, and then tune the noise map to determine the tuned noise map. Alternatively, the video restoration device may tune the noise parameter 311 and then determine the tuned noise map corresponding to the tuned noise parameter. The video restoration network uses the tuned noise map to remove noise from the degraded video. Blur parameters may also be converted into a map format such as a blur map, and the description of blur parameters and / or noise parameters 311 on the blur map may apply.

[0041] Figure 4 shows the process of restoring a degraded image using blur parameters according to one embodiment. Referring to Figure 4, the image restoration device obtains blur parameters 421 from the database 420. For example, if the degraded image 410 is a UDC image, the database 420 can database and store sample blur parameters for various hardware characteristics of the UDC. The blur of the image generated by the UDC is dependent on the hardware characteristics of the UDC. For example, the hardware characteristics of the UDC may include at least one of the hole size, shape, depth, and arrangement pattern. The hole arrangement pattern may include the spacing between adjacent holes. As will be explained later with reference to Figures 10 and 11, etc., the UDC receives light through holes between display pixels and generates an image, and in this process the holes act as multi-slits, causing blur in the image.

[0042] Each hardware characteristic of a UDC may be determined through design data or measured data, and the point spread function (PSF) for each pixel of a sample UDC image can be determined through simulation based on the hardware characteristics. The sample blur parameters for each sample in database 420 are determined based on the characteristics of the PSF. For example, if a sample image without blur is the ground truth (GT), then a blurred sample UDC image is the result of a convolution operation between each pixel of the GT and the corresponding PSF. Using such a correspondence and deconvolution based on the GT, sample UDC image, and the PSF of each pixel, the sample blur parameters for each UDC can be determined.

[0043] Unlike noise parameters, which can change depending on the variable shooting environment, blur parameters 421 have a constant value because they depend on fixed hardware characteristics. Therefore, the video restoration device can acquire and use blur parameters 421 from the database 420 that are suitable for the hardware characteristics of the UDC that generated the degraded video 410. There is no need to repeatedly acquire blur parameters 421 each time the degraded video 410 is restored, and after acquiring blur parameters 421 from the database 420, the video restoration device can continue to use the previously acquired blur parameters 421 without any further acquisition operation. If there are UDCs with different hardware characteristics, instead of using the video restoration network 401 as a single neural network, video restoration may be performed by applying different blur parameters to each UDC. Therefore, video restoration for UDCs with various specifications can be performed without training a separate neural network for each UDC.

[0044] Similar to the noise parameter, the blur parameter 421 may have other values ​​for each pixel of the degraded image 410. For example, a first blur parameter may be determined for the first pixel of the degraded image 410, and graph 422 may show the PSF corresponding to the first blur parameter. The blur parameter 421 may include at least one of a first parameter value indicating the intensity of the blur, a second parameter value indicating the interval between double images, and a third parameter value indicating the intensity of the double images.

[0045] In Graph 422, k1 is the width of the main lobe, and may also be called the blur bandwidth. k2 is the distance between the main lobe and the first side lobe, and may also be called the peak-to-peak distance. k3 is the size of the side lobe. The peak-to-peak ratio is derived from the size of the main lobe and the size of the main lobe. k1, k2, and k3 correspond to the first, second, and third parameters of the blur parameter 421. The larger the value of k1, the more blurred the GT is in the degraded image 410, so k1 is considered to indicate the intensity of the blur. Also, because the GT is shown as a double image in the degraded image 410 due to the distance and ratio between peaks, k2 and k3 are considered to indicate the spacing and intensity of the double image.

[0046] The video restoration device tunes the blur parameter 421 according to the tuning conditions and applies the tuned blur parameter 421 to the video restoration network 401. If a blur map is used, tuning the blur parameter 421 includes tuning the blur map. More specifically, the video restoration device can tune the blur parameter 421 and convert the tuned blur parameter 421 into a blur map and apply it to the video restoration network 401 according to the tuning conditions, or it can first convert the blur parameter 421 into a blur map and then tune the blur map according to the tuning conditions.

[0047] The tuning conditions may include user preferences. For example, user preferences may include preferred noise level and preferred blur level, and the image restoration device tunes the blur parameter 421 according to the preferred blur level. More specifically, if the user prefers a soft look with some blur, the image restoration device may tune the blur parameter 421 to show a low blur level. For example, the image restoration device may set the first parameter value, which indicates the intensity of the blur, low. As a result, the image restoration network 401 can assume that the degraded image 410 has less blur than the actual blur and perform blur removal work corresponding to the less blur. Conversely, if the user prefers a sharp look with almost no blur, the image restoration device tunes the blur parameter 421 to show a high noise level. As a result, the image restoration network 401 can assume that the degraded image 410 has more blur than the actual blur and perform blur removal work corresponding to the more blur.

[0048] Figure 5 shows the process of simulating a point diffusion function with respect to a hole pattern of a UDC according to one embodiment. Referring to Figure 5, one region 505 of the display has a diameter 510, and holes 520, 525 are arranged between pixels 515 within the region 505 according to a certain hole pattern. The hole pattern of holes 520, 525 is determined based on at least one of the size, shape, depth, and arrangement pattern (e.g., spacing between holes) of the holes 520, 525, and the PSF 540 can be simulated according to such a hole pattern.

[0049] PSF540 is a mathematical or numerical representation of how light spreads in relation to each pixel of a degraded image, and the degraded image is the result of the convolution between the PSF and GT of each pixel. Therefore, the blur information shown in the degraded image via the hole patterns of holes 520, 525 and / or PSF540 is estimated. For example, the size of holes 520, 525 determines the distance 550 between the x-intercept and the main lobe of the PSF540 envelope, and the shape of the envelope, while the spacing 530 between adjacent holes 520, 525 determines the position and size of the first side lobe. Also, the ratio of the spacing between holes 520, 525 to the size of each hole 520, 525 determines the size of the first side lobe.

[0050] The larger the size of holes 520 and 525, the greater the distance 550 between the envelope's x-intercept and the main lobe, and the smaller the size of the first side lobe. Conversely, the narrower the spacing 530 between holes 520 and 525, the greater the distance 545 between the main lobe and the first side lobe, and the smaller the size of the first side lobe. For example, if the spacing 530 between holes 520 and 525 is large, a strong double image may be shown in the degraded image, and if the spacing 530 between holes 520 and 525 is small, a strong blur may occur in the degraded image. Depending on these characteristics of the PSF540, blur parameters for each UDC module are determined, and the image restoration device can remove double images and blur in the degraded image using the corresponding blur parameters.

[0051] Figure 6 shows the process of restoring a degraded image using all the noise and blur parameters for one example. Referring to Figure 6, the image restoration device executes the image restoration network 601 with the degraded image 610 and the degradation information 630 to generate the restored image 640. The degradation information 630 includes noise parameters 631 and blur parameters 632. The image restoration device can analyze the degraded image 610 to estimate the noise parameters 631 and obtain the blur parameters 632 from the database 620. The image restoration device can tune the degradation information 630 and execute the image restoration network 601 with the tuned degradation information 630. For example, the image restoration device may tune the noise parameters 631 and blur parameters 632 according to tuning conditions. The tuning conditions may include environmental information and / or user preferences.

[0052] The video restoration device uses the degradation information 630 as input data for the video restoration network 601, or to adjust the output of any layer of the video restoration network 601. The video restoration device generates map data corresponding to the noise parameter 631 and the blur parameter 632, and can adjust the output data of any layer of the video restoration network 601 using the map data. For example, the video restoration device can generate a noise map corresponding to the noise parameter 631 and a blur map corresponding to the blur parameter 632, and can use the noise map and blur map as map data to adjust the output data of any layer of the video restoration network 601. As a different example, the noise map and the blur map can be integrated to generate integrated map data, and the output data of any layer of the video restoration network 601 can be adjusted using the integrated map data.

[0053] Figure 7 shows the training process of a video restoration network according to one embodiment. Referring to Figure 7, the training device can run the video restoration network 701 with training video 710 and degradation information 711, and train the video restoration network 701 while updating it based on the loss 740 between the output video 720 and GT730. The degradation information 711 indicates the degradation elements present in the training video 710. In such a training process, the parameters (e.g., weights) of the video restoration network 701 can be continuously adjusted so that the loss 740 decreases, and the video restoration network 701 has the ability to generate an output video 720 that is close to GT730 based on the training video 710 and degradation information 711.

[0054] According to one embodiment, the training device generates training video 710 by applying degradation information 711 to GT730. For example, the training device performs a convolution operation between each pixel of GT730 and the corresponding value of the degradation information 711, and determines the result of the operation as training video 710. In order to train the video restoration network 701 that restores degraded video (e.g., UDC video), it is necessary to first construct training data, but it is not easy to construct training data through actual shooting. To construct training data for a UDC video infrastructure, it is necessary to alternately photograph UDC equipment with UDC cameras and general equipment with general cameras, or to alternately photograph UDC equipment with and without display panels attached, but this process may cause motion, minute vibrations, changes in focus, etc. By applying degradation information 711 to GT730 to generate training video 710, training data can be constructed relatively easily and efficiently.

[0055] Figure 8 shows a video restoration method according to one embodiment. Referring to Figure 8, the video restoration device receives degraded video from the camera in step S810, determines degradation information indicating the degradation elements of the degraded video in step S820, tunes the degradation information according to tuning conditions in step S830, and executes a video restoration network with the UDC video and degradation information in step S840 to generate a restored video corresponding to the degraded video. In addition, the descriptions in Figures 1 to 7 and Figures 9 to 12 apply to the video restoration device.

[0056] Figure 9 shows the configuration of a video restoration device according to one embodiment. Referring to Figure 9, the video restoration device 900 includes a processor 910 and a memory 920. The memory 920 is connected to the processor 910 and stores instructions that can be executed by the processor 910, data that the processor 910 calculates, or data that has been processed by the processor 910. The memory 920 may also include a non-temporary computer-readable recording medium, such as a high-speed random-access memory and / or a non-volatile computer-readable storage medium (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0057] The processor 910 executes instructions stored in the memory 920 and performs the operations described with reference to Figures 1 to 8 and 10 to 12. For example, the processor 910 can receive degraded video from the camera, determine degradation information indicating the degradation elements of the degraded video, tune the degradation information according to tuning conditions, and run a video restoration network with the degraded video and degradation information to generate restored video corresponding to the degraded video. In addition, the descriptions in Figures 1 to 8 and 10 to 12 may apply to the video restoration device 900.

[0058] Figure 10 shows an electronic device providing a UDC according to one embodiment. Referring to Figure 10, the electronic device 1010 may include a UDC (under display camera) positioned below a region 1030 of the display 1020. By positioning the camera inside the electronic device 1010, the camera area previously allocated for camera exposure can also be realized as part of the display area. Therefore, it is not necessary to implement the display in a notch form or to allocate a camera area within the display area to achieve the maximum display area, and the display 1020 can be realized in a perfectly rectangular form. In Figure 10, a smartphone is shown as the electronic device 1010, but the electronic device 1010 may be a different device other than a smartphone that includes a display 1020.

[0059] The display area 1040 is an enlarged representation of a display panel of area 1030 and includes display pixels 1050 and holes 1060. The holes 1060 are not limited to a circular shape and can be implemented in various forms such as elliptical or square shapes. The holes 1060 may also be referred to as micro-holes. The display pixels 1050 and holes 1060 may be arranged in area 1030 in a certain pattern. Such an arrangement pattern may be called a hole pattern. For example, the holes 1060 may be placed between display pixels, as close to the display pixels as possible. The UDC generates an image (in other words, a degraded image or UDC image) based on light provided from outside the electronic device 1010 through the holes 1060. The display pixels 1050 can output a panel image together with other display pixels other than those in the display area 1040.

[0060] Figure 11 shows the arrangement of a display panel and UDC according to one embodiment. Figure 11 is a cross-sectional view of a region 1030 shown in Figure 10. The display panel 1110 includes display pixels 1130 that can represent color and holes 1140 that transmit external light 1150. The display pixels 1130 and holes 1140 may be arranged alternately with respect to each other. Each display pixel 1130 may include subpixels that detect a specific color.

[0061] A protective layer 1160 made of a transparent material is placed on the display panel 1110 to protect it. The protective layer 1160 may be made of, for example, tempered glass or reinforced plastic. In addition to the display pixels 1130, the display panel 1110 may also include several other components for realizing the display panel 1110, and through such components, a display method such as LCD or OLED can be realized.

[0062] The image sensor 1120 is positioned below the display panel 1110 and detects external light 1150 that has passed through the hole 1140 to generate an image (in other words, a degraded image or UDC image). The image sensor 1120 may be designed to be ultra-compact and there may be multiple images. Since the light 1150 that reaches the image sensor 1120 is a portion of the light incident on the display panel 1110 that passes through the hole 1140, the UDC image generated by the image sensor 1120 may have low brightness and contain a lot of noise. In addition, multiple holes may each act like slits, causing blur due to light diffraction in the UDC image. Such degradation factors of UDC image quality can be removed through an image restoration process specifically for UDC images.

[0063] Figure 12 shows an electronic device according to one embodiment. Referring to Figure 12, the electronic device 1200 may include a processor 1210, a memory 1220, a camera 1230, a storage device 1240, an input device 1250, an output device 1260, and a network interface 1270, which can communicate via a communication bus 1280. For example, the electronic device 1200 may be implemented as at least part of a mobile device such as a mobile phone, smartphone, PDA, netbook, tablet computer, or laptop computer; a wearable device such as a smartwatch, smart band, or smart glasses; a computing device such as a desktop or server; a home appliance such as a television, smart TV, or refrigerator; a security device such as a door rack; or a vehicle such as a smart car.

[0064] The electronic device 1200 generates video (e.g., degraded video and / or UDC video), restores the video, and generates restored video. The electronic device 1200 may also perform subsequent operations related to restored imaging, such as user authentication. The electronic device 1200 corresponds to the electronic device shown in Figure 10 and may structurally and / or functionally include the video restoration device 100 shown in Figure 1 and / or the video restoration device 900 shown in Figure 9.

[0065] The processor 1210 executes functions and instructions for execution within the electronic device 1200. The processor 1210 processes instructions stored in the memory 1220 or the storage device 1240. The processor 1210 can perform one or more operations as described with reference to Figures 1 to 11.

[0066] Memory 1220 stores data for face detection. Memory 1220 may include a computer-readable storage medium or a computer-readable storage device. Memory 1220 can store instruction words to be executed by the processor 1210 and can store related information while the software and / or application is executed by the electronic device 1200.

[0067] Camera 1230 takes photographs and / or videos. For example, camera 1230 may be a UDC camera. A UDC camera is located below the display panel and generates UDC images based on light received through holes between display pixels. For example, the UDC images may include the user's face, and user authentication based on the user's face is performed via the reconstructed image of the UDC images. Camera 1230 may be a 3D camera that provides 3D images including depth information about objects.

[0068] The storage device 1240 includes a storage medium or storage device that is readable by a computer. The storage device 1240 can store a larger amount of information than the memory 1220 and can store information for a longer period of time. For example, the storage device 1240 may include a magnetic hard disk, an optical disk, flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0069] The input device 1250 can receive input from the user via traditional input methods such as keyboards and mice, as well as newer input methods such as touch input, voice input, and image input. For example, the input device 1250 may include a keyboard, mouse, touchscreen, microphone, or any other device that can detect input from the user and transmit the detected input to the electronic device 1200.

[0070] The output device 1260 provides the user with the output of the electronic device 1200 via a visual, auditory, or tactile channel. The output device 1260 may include, for example, a display, a touchscreen, a speaker, a vibration generator, or any other device capable of providing output to the user. The network interface 1270 can communicate with external devices via a wired or wireless network.

[0071] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device can run an operating system (OS) and one or more software applications run on the OS. The processing device can also access, store, manipulate, process, and generate data in response to software execution. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will see that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0072] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by a processing unit or for providing instructions or data to a processing unit. Software can be distributed across a network of computer systems and stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0073] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0074] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0075] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0076] Therefore, the scope of the present invention is not limited to the disclosed embodiments, but is defined by the claims and equivalents thereof. [Explanation of symbols]

[0077] 100, 900: Video restoration device 910, 1210: Processors 920, 1220: Memory 1010, 1200: Electronic equipment 1230: Camera

Claims

1. The steps include receiving degraded video from the camera, The steps include determining degradation information that indicates the degradation elements of the aforementioned degraded video, A step of tuning the degradation information according to the tuning conditions, The steps include: running a video restoration network with the degraded video and the degradation information to generate a restored video corresponding to the degraded video; Includes, The aforementioned camera is an under-display camera (UDC), The step of determining the degradation information includes the step of obtaining blur parameters corresponding to the hardware characteristics of the UDC, The UDC receives light through holes between the display pixels of the display panel, The hardware characteristics include at least one of the size, shape, depth, and arrangement pattern of the holes, in a method for restoring video.

2. The video restoration method according to claim 1, wherein the degradation information includes at least one of noise parameters and blur parameters.

3. The video restoration method according to claim 1 or 2, wherein the tuning conditions include the user's preference regarding the level of removal of the degradation elements.

4. The step of generating the restored video is: The steps include inputting input data corresponding to the degraded video into the video restoration network, The steps include adjusting the output data of at least one layer of the video restoration network to map data corresponding to the degradation information, A method for restoring video according to any one of claims 1 to 3, including the method described in any one of claims 1 to 3.

5. The video restoration method according to any one of claims 1 to 4, wherein the step of determining the degradation information includes a step of analyzing the degraded video and determining a noise parameter indicating the level of noise contained in the degraded video.

6. The video restoration method according to claim 5, wherein the step of tuning the degradation information further includes the step of tuning the noise parameters based on environmental information of the degraded video.

7. The step of tuning the noise parameters is: When the environmental information corresponds to a low-light environment, the step of tuning the noise parameter to show a high noise level, When the aforementioned environmental information corresponds to a high-illumination environment, the step of tuning the noise parameter to show a low noise level, The video restoration method according to claim 6, including the method described in claim 6.

8. The image restoration method according to any one of claims 1 to 7, wherein the blur parameter includes at least one of a first parameter indicating the intensity of blur, a second parameter indicating the interval between double images, and a third parameter indicating the intensity of the double images.

9. A computer program stored on a computer-readable recording medium for use in conjunction with hardware to perform the method according to any one of claims 1 to 8.

10. Processor and A memory containing an instruction word that can be executed by the aforementioned processor, Includes, If the instruction is executed by the processor, the processor performs the video restoration method according to any one of claims 1 to 8. Video restoration device.

11. Camera and, A processor that receives degraded video from the camera, determines degradation information indicating the degradation elements of the degraded video, tunes the degradation information according to tuning conditions, and executes a video restoration network with the degraded video and the degradation information to generate a restored video corresponding to the degraded video. Includes, The camera is an under-display camera (UDC) located below the display panel, which generates an image based on light received through holes between the display pixels of the display panel. The processor determines the degradation information using blur parameters corresponding to the hardware characteristics of the UDC. The hardware characteristics include at least one of the size, shape, depth, and arrangement pattern of the holes in the electronic device.

12. The electronic device according to claim 11, wherein the processor analyzes the degraded video, determines noise parameters indicating the level of noise contained in the degraded video, and tunes the noise parameters based on environmental information of the degraded video.

Images