Image restoration method and apparatus, and electronic device

Abstract

An image restoration method and apparatus and an electronic device are disclosed. The image restoration method includes determining degradation information indicating a degradation factor of a degraded image, adjusting the degradation information based on an adjustment condition, and generating a restored image corresponding to the degraded image by performing an image restoration network with the degraded image and the degradation information.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0143565, filed October 30, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to an image restoration method and apparatus. BACKGROUND

[0003] Cameras (devices configured to photograph images) are provided in various electronic devices. As time passes, cameras have become an essential part of mobile devices such as smartphones, and have become higher in performance and smaller in size. Typically, a smartphone includes a front camera and a rear camera. The front camera is arranged in an upper region of the smartphone, and is often used to photograph a self-portrait. An under-display camera (UDC) system provides a camera hidden behind a display panel. SUMMARY

[0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented image restoration method includes receiving a degraded image, determining degradation information indicating a degradation factor of the degraded image, adjusting the degradation information based on an adjustment condition, and generating a restored image corresponding to the degraded image by performing an image restoration network with the degraded image and the degradation information.

[0006] The degradation information can include at least one or both of a noise parameter and a blur parameter. The adjustment condition can include a user preference for a level of removal of the degradation factor. The generating of the restored image can include inputting input data corresponding to the degraded image to the image restoration network, and adjusting output data of at least one layer of the image restoration network using graph data corresponding to the degradation information.

[0007] The determining of the degradation information can include determining a noise parameter indicating a level of noise included in the degraded image by analyzing the degraded image. The adjusting of the degradation information can further include adjusting the noise parameter based on environmental information of the degraded image. The adjusting of the noise parameter can include adjusting the noise parameter such that the noise parameter indicates a high noise level in response to the environmental information corresponding to a low-illumination environment, and adjusting the noise parameter such that the noise parameter indicates a low noise level in response to the environmental information corresponding to a high-illumination environment.

[0008] The degraded image can be photographed by a camera, which can be an under-display camera (UDC), and the determining of the degradation information can include obtaining a blur parameter corresponding to a hardware characteristic of the UDC. The UDC can receive light through an aperture disposed between display pixels of a display panel. The hardware characteristic can include one or more of a size, a shape, a depth, and a disposition pattern of the aperture. The blur parameter can include one or more of a first parameter indicating a blur strength, a second parameter indicating a spacing between artifacts, and a third parameter indicating a strength of the artifacts.

[0009] In another general aspect, an image restoration device includes a processor; and a memory including instructions executable in the processor. When the instructions are executed in the processor, the processor can receive a degraded image, determine degradation information indicating a degradation factor of the degraded image, adjust the degradation information based on an adjustment condition, and generate a restored image corresponding to the degraded image by executing an image restoration network with the degraded image and the degradation information.

[0010] In another general aspect, an electronic device includes a camera; and a processor configured to: receive a degraded image from the camera, determine degradation information indicating a degradation factor of the degraded image, adjust the degradation information based on an adjustment condition, and generate a restored image corresponding to the degraded image by executing an image restoration network with the degraded image and the degradation information.

[0011] In another general aspect, an electronic device includes a camera; and a processor configured to: receive a degraded image photographed by the camera, estimate an amount of noise included in the degraded image by a noise parameter representing noise information of each pixel in the degraded image, and generate a restored image by executing a deep neural network based on the degraded image and the noise parameter.

[0012] The processor can be configured to generate the restored image by removing a noise component corresponding to the noise parameter from the degraded image.

[0013] The processor can be configured to generate a noise map corresponding to the noise parameter, and input the noise map to the deep neural network to adjust an output of a layer of the deep neural network.

[0014] The processor can be configured to adjust the noise parameter based on an adjustment condition, and apply the adjusted noise parameter to the deep neural network.

[0015] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 An example of image restoration is illustrated.

[0017] Figure 2 An example of restoring a degraded image using a noise parameter is shown.

[0018] Figure 3 An example of adjusting a noise parameter is shown.

[0019] Figure 4 An example of restoring a degraded image using a blur parameter is shown.

[0020] Figure 5 An example of simulating a point spread function (PSF) of an aperture pattern of an under-display camera (UDC) is shown.

[0021] Figure 6 An example of restoring a degraded image using both a noise parameter and a blur parameter is shown.

[0022] Figure 7 An example of training an image restoration network is shown.

[0023] Figure 8 An example of an image restoration method is shown.

[0024] Figure 9 An example of an image restoration device is shown.

[0025] Figure 10 An example of an electronic device providing a UDC is shown.

[0026] Figure 11 An example of an arrangement of a display panel and a UDC is shown.

[0027] Figure 12 An example of an electronic device is shown.

[0028] Throughout the drawings and the detailed description, the same reference labels are used to refer to the same elements throughout the detailed description and the drawings. The drawings can not be to scale, and the relative dimensions, proportions, and depiction of elements in the drawings can be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0029] The following detailed description is provided to aid the reader in understanding the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the present disclosure. For example, the order of the operations described herein is merely an example and is not limited to those set forth herein, but can be changed as will be clear to one of ordinary skill in the art after understanding the present disclosure, except for operations that must occur in a particular order. Also, descriptions of features known after understanding the present disclosure can be omitted for the sake of clarity and conciseness.

[0030] The features described herein can be implemented in different ways depending upon the particular application. Rather than be bound by the examples described herein, the examples have been provided to illustrate some of the many possible ways in which the methods, devices, and / or systems described herein can be implemented.

[0031] Throughout the specification, where assemblies are described as "connected to" or "coupled to" other assemblies, it will be understood that the assembly can be directly connected or coupled to the other assembly, or intervening assemblies can be present. Conversely, where an element is described as being "directly connected to" or "directly coupled to" another element, it will be understood that no intervening assembly is present. Similarly, similar expressions such as "between," "adjacent to," and "adjacent" can be interpreted in the same way. As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0032] Although terms such as "first," "second," and "third" can be used herein to describe various elements, components, regions, layers, or sections, these elements, components, regions, layers, or sections are not limited by these terms. Rather, these terms are used only to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section referred to in the examples described herein can also be referred to as a second element, component, region, layer, or section without departing from the teachings of the examples.

[0033] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the articles "a," "an," and "the" are intended to include one or more items, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "including," and "having" specify the presence of stated features, numbers, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or combinations thereof.

[0034] The use herein of the term "may" (e.g., with respect to what an example or embodiment can include or implement) means that at least one example or embodiment includes or implements that feature, and no example is limited to such.

[0035] Furthermore, in the description of the various examples, it will be understood that no detailed description of structure or function is necessary to understand the disclosure, and that such description is omitted to avoid obscuring the disclosure.

[0036] Hereinafter, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements.

[0037] Figure 1 An example of image restoration is illustrated. Referring to Figure 1 , the image restoration device 100 can remove a degradation factor degrading image quality from the degraded image 110 through image restoration to generate a restored image 120. The image restoration device 100 can remove the degradation factor based on a cause and a pattern specific to degradation of the degraded image 110. For example, the degraded image 110 can be an image from an under-display camera (UDC) (hereinafter, simply referred to as a UDC image). In this example, a UDC-specific degradation factor can be considered for image restoration.

[0038] The image restoration device 100 includes an image restoration network 101. The image restoration network 101 can be a deep neural network (DNN) including a plurality of layers. The layers can include an input layer, a hidden layer, and an output layer. The neural network described herein can include, for example, a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, a portion of the layers included in the neural network can correspond to the CNN, and another portion of the layers can correspond to the FCN. In this example, the CNN can be referred to as a convolutional layer, and the FCN can be referred to as a fully connected layer. The neural network can include a residual connection.

[0039] In the case of the CNN, data input to each layer can be referred to as an input feature map, and data output from each layer can be referred to as an output feature map. The input feature map and the output feature map can also be referred to as activation data. For example, when the convolutional layer corresponds to the input layer, the input feature map of the input layer can be an input image.

[0040] After training based on deep learning, the neural network can perform inference suitable for a purpose of training by mapping input data and output data in a non-linear relationship with each other. Here, deep learning denotes a machine learning method for solving a problem such as image recognition or voice recognition from a large data set. Deep learning can also be interpreted as an optimization problem solving process that finds a point of energy minimization when a neural network is trained using prepared training data.

[0041] Through supervised learning or unsupervised learning of deep learning, weights corresponding to the architecture of the neural network or model can be obtained. Through such weights, input data and output data can be mapped. When the neural network has a width and a depth large enough, the neural network can have a capacity sufficient to implement a function. When the neural network learns a sufficient amount of training data through an appropriate training process, the neural network can achieve optimal performance.

[0042] The neural network can be represented as being pre-trained, where "pre-trained" means "before" the neural network is launched. Here, the neural network is launched means that the neural network is ready for inference. For example, the neural network is launched can mean that the neural network is loaded into a memory, or after the neural network is loaded into the memory, input data for inference is input to the neural network.

[0043] As illustrated, the image restoration device 100 can determine the degradation information 111 indicating a degradation factor of the degraded image 110, and generate the restored image 120 by executing the image restoration network 101 with the degraded image 110 and the degradation information 111. For example, the degradation information 111 can include a noise parameter indicating a noise factor in the degraded image 110 and a blur parameter indicating a blur factor in the degraded image 110. When the image restoration network 101 is executed, inference of the image restoration network 101 can be performed.

[0044] The image restoration device 100 can generate the restored image 120 by inputting the degraded image 110 and the degradation information 111 as input data to the image restoration network 101. Alternatively, the image restoration device 100 can generate the restored image 120 by inputting the degraded image 110 as input data to the image restoration network 101 and adjusting the output of the layer of the image restoration network 101 based on the degradation information 111. For example, the image restoration device 100 can adjust the output of the layer of the image restoration network 101 using graph data corresponding to the degradation information 111. In this example, the image restoration network 101 can use information of the restored image 120 through an attention mechanism.

[0045] The image restoration device 100 can adjust the degradation information 111 based on an adjustment condition and the use information obtained by the adjustment. Here, the adjustment can include scaling, adding, clipping, and assigning, the scaling scaling a parameter value of the degradation information 111 by a ratio, the adding adding a value to the parameter value, the clipping limiting the parameter value to a value, and the assigning assigning a specific value as the parameter value. For example, for noise removal, environment information associated with an environment in which the degraded image 110 is photographed can be used as an adjustment condition. For example, in a case where noise of the degraded image 110 is removed with a single neural network when the degraded image 110 is photographed in various environments, it can not be possible to appropriately remove the noise according to the intensity of the noise. For example, in a case where restoration is over-smoothed (for example, when noise is removed from a high-illumination image with a neural network trained to remove noise from a low-illumination image), over-smoothing can occur. Conversely, in a case where noise is removed from a low-illumination image with a neural network trained to remove noise from a high-illumination image, the noise can remain in the image.

[0046] Accordingly, instead of using the image restoration network 101 as a single neural network, the image restoration device 100 can adjust the degradation information 111 according to a situation, and then use information obtained by the adjustment. For example, in a case where the degraded image 110 is captured in a low-illumination environment, the image restoration device 100 can adjust the degradation information 111 so that the image restoration network 101 removes a larger amount of noise to prevent noise from remaining in the degraded image 110. In a case where the adjustment is performed by scaling or addition so that the noise parameter indicates a high noise level, the image restoration network 101 can consider that a larger amount of noise is present in the degraded image 110 than an actual amount, and can perform a noise removal operation corresponding to the larger amount of noise. In contrast, in a case where the degraded image 110 is captured in a high-illumination environment, the image restoration device 100 can adjust the noise parameter so that the noise parameter indicates a low noise level to prevent over-smoothing.

[0047] Alternatively, a user's preference for a level of removal of a degradation factor can be used as an adjustment condition. Hereinafter, the user's preference will be referred to as a user preference for a simpler expression. For example, a user can prefer a retro-style image with noise, and another user can prefer a clean image with little noise. For another example, a user can prefer a soft image with some blur, and another user can prefer a clear image with little blur. Accordingly, the user preference can include a preferred noise level and a preferred blur level. The image restoration device 100 can adjust the noise parameter and / or the blur parameter based on the user preference, and a level of removal of noise and / or blur by the image restoration network 101 can be adjusted accordingly.

[0048] Figure 2 An example of restoring a degraded image using a noise parameter is illustrated. Referring to Figure 2 , the image restoration device can analyze the degraded image 210 to estimate a noise level as a level of noise included in the degraded image 210. The image restoration device can estimate the noise level through the noise parameter 211. For example, the image restoration device can express the noise level through a normal distribution N(0, σ), where σ denotes the noise parameter 211. The noise parameter 211 can express noise information of each pixel through a parameter value corresponding to each pixel in the degraded image 210. For example, a first parameter value σ1 indicating a noise level of a first pixel in the degraded image 210 can be different from a second parameter value σ2 indicating a noise level of a second pixel in the degraded image 210.

[0049] The image restoration device can generate the restored image 220 by executing the image restoration network 201 based on the degraded image 210 and the noise parameter 211. For example, the image restoration network 201 can generate the restored image 220 by removing a noise component corresponding to the noise parameter 211 from the degraded image 210. To this end, the noise parameter 211 can be transformed into a graph form. That is, the image restoration device can generate a noise graph corresponding to the noise parameter 211, and the image restoration network 201 can use the noise graph to adjust an output of a layer (e.g., an input layer, a hidden layer, or an output layer) of the image restoration network 201. For example, the image restoration network 201 can use an operation between the output and the noise graph to remove a noise component indicated by the noise graph in the output. The noise parameter 211 can indicate a noise level of each pixel in the degraded image 210, and thus be transformed into a noise graph representing spatial information.

[0050] The image restoration device can adjust the noise parameter 211 based on an adjustment condition, and apply the adjusted noise parameter 211 to the image restoration network 201. In the case of using the noise graph, the adjustment of the noise parameter 211 can include adjusting the noise graph. That is, the image restoration device can adjust the noise parameter 211 based on the adjustment condition, transform the adjusted noise parameter 211 into a noise graph, and then apply the noise graph to the image restoration network 201. Alternatively, the image restoration device can transform the noise parameter 211 into a noise graph, and then adjust the noise graph based on the adjustment condition.

[0051] The adjustment condition can include environmental information of the degraded image 210 and / or a user preference. The image restoration device can determine the environmental information through metadata of the degraded image 210. For example, the environmental information can include illumination information indicating an illumination at which the degraded image 210 is photographed. The image restoration device can classify an illumination higher than a threshold value as high illumination, and classify an illumination lower than the threshold value as low illumination. The low illumination can be interpreted as a level of illumination at which light is too little for a person to recognize an object. The high illumination is a term provided to distinguish from the low illumination condition, and can not necessarily mean an extremely high illumination at which saturation can be generated, but mean a general level of illumination at which a person can easily recognize an object. Accordingly, the high illumination can be referred to as general illumination. The user preference can be stored in the image restoration device as a setting value. For example, the user preference can include a preferred noise level and a preferred blur level.

[0052] Figure 3 An example of adjusting the noise parameter is illustrated. Referring to FIG. 2, the image restoration device can generate a noise graph 230 corresponding to the noise parameter 211. The image restoration device can adjust the noise graph 230 based on an adjustment condition, and apply the adjusted noise graph 230 to the image restoration network 201. The image restoration device can generate the restored image 220 by executing the image restoration network 201 based on the degraded image 210 and the adjusted noise graph 230. Figure 3, the image restoration device determines the noise parameter 311 in operation 310, and adjusts the noise parameter 311 based on the adjustment condition in operation 320. The degradation information can include a blur parameter in addition to the noise parameter 311, and thus the following description of the noise parameter 311 can also apply to the blur parameter.

[0053] The image restoration device can determine a first adjusted noise parameter 321 by adjusting the noise parameter 311 based on a first adjustment condition, and determine a second adjusted noise parameter 322 by adjusting the noise parameter 311 based on a second adjustment condition. The image restoration device can perform the image restoration network with the first adjusted noise parameter 321 and / or the second adjusted noise parameter 322 instead of using the noise parameter 311. Thus, there can be an effect that different data is applied to the image restoration network according to the adjustment condition.

[0054] In one example, the adjustment condition can include environment information. For example, in a case where the environment information corresponds to a low-illumination environment, the image restoration device can adjust the noise parameter 311 such that the noise parameter 311 indicates a high noise level. In this example, the first adjusted noise parameter 321 can be determined. For example, in a case where the environment information corresponds to a high-illumination environment, the image restoration device can adjust the noise parameter 311 such that the noise parameter 311 indicates a low noise level. In this example, the second adjusted noise parameter 322 can be determined. For example, when the noise parameter 311 indicates 10% noise, the first adjusted noise parameter 321 can indicate 15% noise, and the second adjusted noise parameter 322 can indicate 5% noise. The noise parameter 311 can include different parameter values for each pixel in the degraded image, and thus each of 10% noise, 15% noise, and 5% noise can indicate an average noise of the entire image. The image restoration device can adjust the parameter value of each pixel by a ratio of adjusting the average 10% noise to 15% noise or 5% noise. Thus, in response to the first adjusted noise parameter 321, the image restoration network can remove a larger amount of noise from the degraded image.

[0055] In another example, the adjustment condition can include a user preference. For example, the user preference can include a preferred noise level and a preferred blur level. The image restoration device can adjust the noise parameter 311 based on the preferred noise level. For example, in a case where the user preference has an old-looking image with noise, the image restoration device can adjust the noise parameter 311 such that the noise parameter 311 indicates a low noise level. In a case where the user preference has a clean image with little noise, the image restoration device can adjust the noise parameter 311 such that the noise parameter 311 indicates a high noise level. In yet another example, the adjustment condition can include both the environment information and the user preference, and the image restoration device can adjust the noise parameter 311 based on both conditions.

[0056] In a case where a noise map is used, the image restoration device can generate a noise map corresponding to the noise parameter 311, and then adjust the noise map to determine an adjusted noise map. Alternatively, the image restoration device can adjust the noise parameter 311, and then determine an adjusted noise map corresponding to the adjusted noise parameter 311. The image restoration network can use the adjusted noise map to remove noise from the degraded image. In addition, the blur parameter can be transformed into the form of a map such as a blur map. The foregoing description of the noise parameter 311 can also apply to the blur parameter and / or the blur map.

[0057] Figure 4 An example of restoring a degraded image using a blur parameter is illustrated. Referring to Figure 4 , the image restoration device can obtain a blur parameter 421 from a database (DB) 420. For example, in a case where the degraded image 410 is a UDC image, the DB 420 can store a database of sample blur parameters according to hardware characteristics of each of various UDCs. The blur of an image generated by a UDC can depend on the hardware characteristics of the UDC. For example, the hardware characteristics of the UDC can include at least one of the size, shape, depth, and arrangement pattern of the holes. The arrangement pattern of the holes can include the interval between adjacent holes. Hereinafter, as described with reference to Figure 10 and Figure 11 , a UDC can generate an image by receiving light through holes arranged between display pixels, and the holes can function as multi-slit that generates blur in the image.

[0058] The hardware characteristics of each UDC can be determined through design data or ground truth (GT) data. Through simulation based on such hardware characteristics, a point spread function (PSF) of each pixel in a sample UDC image can be determined. Each sample blur parameter in the DB 420 can be determined based on the characteristics of the PSF. For example, when a sample image without blur corresponds to a GT, a sample UDC image with blur can correspond to the result of a convolution operation between each pixel of the GT and the corresponding PSF. The sample blur parameter of each UDC can be determined using the GT, the sample UDC image, and the correspondence based on the PSF of each pixel and deconvolution.

[0059] Unlike the noise parameter, which varies according to a variable shooting environment, the blur parameter 421 can depend on an unchanging hardware characteristic, and thus can have a consistent value. Accordingly, the image restoration device can obtain and use the blur parameter 421 that is suitable for the hardware characteristic of the UDC that generated the degraded image 410. However, the blur parameter 421 can not need to be repeatedly obtained each time the degraded image 410 is restored. Accordingly, the image restoration device can continuously use the previously obtained blur parameter 421, without additionally obtaining the blur parameter 421 after obtaining the blur parameter 421 from the DB 420. For example, in the case where there are UDCs having different hardware characteristics, the image restoration device can perform image restoration by applying different blur parameters, rather than using the image restoration network 401 as a single neural network. Accordingly, image restoration can be performed for various kinds of UDCs without training a separate neural network for each UDC.

[0060] Like the noise parameter, the blur parameter 421 can have different values for each pixel in the degraded image 410. For example, a first blur parameter can be determined for a first pixel in the degraded image 410. The graph 422 illustrates a PSF corresponding to the first blur parameter. The blur parameter 421 can include at least one of a first parameter value indicating a blur strength, a second parameter value indicating a distance between artifacts (e.g., ghosting), and a third parameter value indicating a strength of the artifacts.

[0061] In the graph 422, k1 denotes a width of a main lobe, which is also referred to as a blur bandwidth. In addition, k2 denotes a distance between the main lobe and a first side lobe, which is also referred to as a peak-to-peak distance. In addition, k3 denotes a size of the side lobe. Based on the size of the main lobe and the size of the side lobe, a peak-to-peak ratio between peaks can be derived. k1, k2, and k3 can correspond to the first parameter, the second parameter, and the third parameter of the blur parameter 421, respectively. When the value of k1 increases, the degraded image 410 can correspond to a more blurred image than the GT, and thus k1 can indicate a blur strength. In addition, based on the distance and the ratio between the peaks, the GT can be shown as artifacts in the degraded image 410, and thus k2 and k3 can indicate a distance between artifacts and a strength of the artifacts, respectively.

[0062] The image restoration device can adjust the blur parameter 421 based on the adjustment condition, and apply the adjusted blur parameter 421 obtained by the adjustment to the image restoration network 401. In the case of using a blur map, the adjustment of the blur parameter 421 can include adjusting the blur map. For example, the image restoration device can adjust the blur parameter 421 based on the adjustment condition, and transform the adjusted blur parameter 421 into a blur map and apply the blur map to the image restoration network 401. Alternatively, the image restoration device can transform the blur parameter 421 into a blur map, and then adjust the blur map based on the adjustment condition.

[0063] The adjustment condition can include a user preference. For example, the user preference can include a preferred noise level and a preferred blur level. In this example, the image restoration device can adjust the blur parameter 421 based on the preferred blur level. For example, in the case of a user preference for soft images with some blur, the image restoration device can adjust the blur parameter 421 such that the blur parameter 421 indicates a low blur level. For example, the image restoration device can decrease the first parameter value indicating the blur strength. Accordingly, the image restoration network 401 can consider that there is less blur in the degraded image 410 than the actual blur, and perform a removal operation corresponding to less blur. However, in the case of a user preference for clear images with little blur, the image restoration device can adjust the blur parameter 421 such that the blur parameter 421 indicates a high blur level. Accordingly, the image restoration network 401 can consider that there is more blur in the degraded image 410 than the actual blur, and perform a removal operation corresponding to more blur.

[0064] Figure 5 An example of a PSF showing an analog UDC hole pattern is shown. Referring to Figure 5 The region 505 of the display has a diameter 510, and the holes 520 and 525 are arranged between the pixels 515 in the region 505 according to a hole pattern. The hole pattern of the holes 520 and 525 can be determined based on at least one of the size, shape, depth, and arrangement pattern (e.g., spacing between the holes) of the holes 520 and 525. The PSF 540 can be simulated based on such a hole pattern.

[0065] The PSF 540 is a mathematical or numerical representation of how light corresponding to each pixel of the degraded image is diffused. Here, the degraded image can correspond to a result of a convolution operation between each pixel's PSF and the GT. Thus, blur information associated with blur that would appear in the degraded image can be estimated by the aperture pattern of the aperture 520 and the aperture 525 and / or the PSF 540. For example, the size of the aperture 520 and the aperture 525 can determine the distance 550 between the x-axis intercept of the envelope of the PSF 540 and the main lobe of the PSF 540 and the shape of the envelope. The interval 530 between the adjacent apertures 520 and 525 can determine the position and size of the first side lobe. In addition, the ratio of the interval between the adjacent apertures (e.g., the aperture 520 and the aperture 525) to the size of each of the apertures (e.g., the aperture 520 and 525) can determine the size of the first side lobe.

[0066] When the size of the aperture 520 and the aperture 525 increases, the distance 550 between the x-axis intercept of the envelope and the main lobe can increase, and the size of the first side lobe can decrease. When the interval 530 between the aperture 520 and the aperture 525 decreases, the distance 545 between the main lobe and the first side lobe can increase, and the size of the first side lobe can decrease. For example, when the interval 530 between the aperture 520 and the aperture 525 is large, there can be strong artifacts or ghosting in the degraded image. In contrast, when the interval 530 between the aperture 520 and the aperture 525 is small, there can be strong blur in the degraded image. Based on such characteristics of the PSF 540, the blur parameters of each UDC module can be determined. The image restoration device can remove such artifacts and blur from the degraded image using the corresponding blur parameters.

[0067] Figure 6 An example of restoring a degraded image using both a noise parameter and a blur parameter is illustrated. Referring to Figure 6 , the image restoration device can generate a restored image 640 by executing the image restoration network 601 with the degraded image 610 and the degradation information 630. The degradation information 630 can include a noise parameter 631 and a blur parameter 632. The image restoration device can estimate the noise parameter 631 by analyzing the degraded image 610, and obtain the blur parameter 632 from the DB 620. The image restoration device can adjust the degradation information 630, and execute the image restoration network 601 with the adjusted degradation information 630. For example, the image restoration device can adjust the noise parameter 631 and / or the blur parameter 632 based on an adjustment condition. The adjustment condition can include environmental information and / or user preference.

[0068] The image restoration device can use the degradation information 630 as input data of the image restoration network 601, or adjust the output of a layer of the image restoration network 601 using the degradation information 630. The image restoration device can generate map data corresponding to the noise parameter 631 and the blur parameter 632, and adjust the output data of a layer of the image restoration network 601 using the map data. For example, the image restoration device can generate a noise map corresponding to the noise parameter 631 and a blur map corresponding to the blur parameter 632, and adjust the output data of a layer of the image restoration network 601 using the noise map and the blur map as the map data. For another example, the image restoration device can generate combined map data by combining the noise map and the blur map, and adjust the output data of a layer of the image restoration network 601 using the combined map data.

[0069] Figure 7 An example of training an image restoration network is illustrated. Referring to Figure 7 , the training device can execute the image restoration network 701 with the training image 710 and the degradation information 711, and train the image restoration network 701 while updating the image restoration network 701 based on the loss 740 between the output image 720 and the GT 730. The degradation information 711 can represent a degradation factor present in the training image 710. In such a training process, the parameters (e.g., weights) of the image restoration network 701 can be continuously adjusted so that the loss 740 decreases, and the image restoration network 701 can have the ability to generate an output image 720 similar to the GT 730 based on the training image 710 and the degradation information 711.

[0070] In one example, the training device can generate the training image 710 by applying the degradation information 711 to the GT 730. For example, the training device can perform a convolution operation between each pixel of the GT 730 and a corresponding value of the degradation information 711, and determine the result of the convolution operation as the training image 710. To train the image restoration network 701 that restores a degraded image (e.g., a UDC image), it can be necessary to first perform configuration of training data. However, it can not be easy to configure training data by actually taking a photograph. To configure training data based on a UDC image, it can be necessary to perform a photograph by alternately using a UDC device having a UDC and a normal device having a normal camera, or by alternately displaying a state in which a display panel is combined with a UDC device and a state in which the display panel is removed from the UDC device. However, in such a process described in the foregoing, movement, fine vibration, focus change, etc. can occur. Therefore, by generating the training image 710 by applying the degradation information 711 to the GT 730, it can be relatively simple and effective to configure training data.

[0071] Figure 8 An example of an image restoration method is illustrated. Referring to Figure 8, the image restoration device receives a degraded image from the camera in operation 810, determines degradation information indicating a degradation factor of the degraded image in operation 820, adjusts the degradation information based on an adjustment condition in operation 830, and generates a restored image corresponding to the degraded image by performing an image restoration network with the degraded image (e.g., a UDC image) and the degradation information in operation 840. For a more detailed description of the operations performed by the image restoration device, reference can be made to the above description with reference to Figures 1 to 7 what has been described above.

[0072] Figure 9 An example of an image restoration device is illustrated. Referring to Figure 9 , the image restoration device 900 includes a processor 910 and a memory 920. The memory 920 can be connected to the processor 910 and store instructions executable by the processor 910, data to be processed by the processor 910, or data processed by the processor 910. The memory 920 can be a non-transitory computer readable medium (e.g., a high-speed random access memory (RAM)) and / or a non-volatile computer readable storage medium (e.g., at least one disk storage device, a flash device, or other non-volatile solid state memory device).

[0073] The processor 910 can execute instructions stored in the memory 920 to perform the operations described above with reference to Figures 1 to 8 In one example, the processor 910 can receive a degraded image from a camera, determine degradation information indicating a degradation factor of the degraded image, adjust the degradation information based on an adjustment condition, and generate a restored image corresponding to the degraded image by performing an image restoration network with the degraded image and the degradation information. For a more detailed description of the image restoration device 900, reference can be made to the above description with reference to Figures 1 to 8 what has been described above.

[0074] Figure 10 An example of an electronic device providing UDC is illustrated. Referring to Figure 10 , the electronic device 1010 includes UDC disposed below a region 1030 of a display 1020. When a camera is disposed inside the electronic device 1010, it is feasible to include a camera region for exposure allocated for the camera as a display region. Accordingly, it is possible to dispose the display 1020 in a complete rectangular shape without needing to dispose the display shape as a notch or allocate a camera region in the display region in order to obtain a maximum size of the display region. Although a smartphone is illustrated as an example of the electronic device 1010 in Figure 10 , the electronic device 1010 can be a device including the display 1020 other than a smartphone.

[0075] The display area 1040 shows an enlarged display panel of the area 1030, and includes display pixels 1050 and holes 1060. The shape of the holes 1060 can not be limited to a circular shape, but can be provided in various shapes such as an elliptical shape and a rectangular shape. The holes 1060 can also be referred to as micro-holes. The display pixels 1050 and the holes 1060 can be arranged in a certain pattern in the area 1030. Such an arrangement pattern can be referred to as a hole pattern. For example, the holes 1060 can be arranged between the display pixels 1050 as close as possible to the display pixels 1050. The UDC can generate an image (e.g., a degraded image or a UDC image) based on light provided after passing through the holes 1060 from the outside of the electronic device 1010. The display pixels 1050 can output a panel image together with other display pixels outside the display area 1040.

[0076] Figure 11 An example of an arrangement of a display panel and a UDC is shown. Figure 11 Figure 10 A cross-sectional view of the area 1030. Referring to Figure 11 The display panel 1110 can include display pixels 1130 representing colors and holes 1140 allowing external light 1150 to pass therethrough. The display pixels 1130 and the holes 1140 can be arranged alternately. Each display pixel 1130 can include sub-pixels, each of which senses a certain color.

[0077] Above the display panel 1110, a protective layer 1160 of a transparent material can be arranged to protect the display panel 1110. For example, the protective layer 1160 can be tempered glass or reinforced plastic. In addition to the display pixels 1130, the display panel 1110 can include other various components for implementing the display panel 1110. Through these components, display types such as, for example, liquid crystal displays (LCDs) and organic light emitting diodes (OLEDs) can be implemented.

[0078] The image sensor 1120 can be arranged below the display panel 1110 and generate an image (e.g., a degraded image or a UDC image) by sensing external light 1150 transmitted via the holes 1140. The image sensor 1120 can be designed to be ultra-small and provided as a plurality of image sensors. The light 1150 reaching the image sensor 1120 can be a portion of incident light on the display panel 1110, which is transmitted through the holes 1140. Thus, the UDC image generated by the image sensor 1120 can be low in brightness and include a relatively large amount of noise. In addition, each hole 1140 can act as a slit, so the UDC image can have blurring due to diffraction of light. Such degrading factors degrading the image quality of the UDC image can be removed through image restoration dedicated to the USD image.

[0079] Figure 12 ​An example of an electronic device is illustrated. Referring to Figure 12 The electronic device 1200 includes a processor 1210, a memory 1220, a camera 1230, a storage 1240, an input device 1250, an output device 1260, and a network interface 1270, which can communicate with each other via a communication bus 1280. For example, the electronic device 1200 can be implemented as at least a portion of a mobile device (e.g., a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (e.g., a smart watch, a smart band, smart glasses, etc.), a computing device (e.g., a desktop computer, a server, etc.), a home appliance (e.g., a television (TV), a smart TV, a refrigerator, etc.), a security device (e.g., a door lock, etc.), or a vehicle (e.g., a smart vehicle, etc.).

[0080] The electronic device 1200 can generate an image (e.g., a degraded image and / or a UDC image), and generate a restored image by restoring the generated image. In addition, the electronic device 1200 can perform a subsequent operation (e.g., user authentication) associated with image restoration. The electronic device 1200 can correspond to the electronic device 1010 of Figure 10 , or structurally and / or functionally include the image restoration apparatus 100 of Figure 1 , and / or the image restoration apparatus 900 of Figure 9 .

[0081] The processor 1210 can execute functions and instructions to be performed in the electronic device 1200. The processor 1210 can process instructions stored in the memory 1220 or the storage 1240. The processor 1210 can perform one or more or all of the operations described above with reference to Figures 1 to 11 .

[0082] The memory 1220 can store data for face detection. The memory 1220 can include a computer readable storage medium or a computer readable storage device. The memory 1220 can store instructions to be executed by the processor 1210, and store related information during execution of software and / or applications by the electronic device 1200.

[0083] The camera 1230 can photograph an image and / or a video. For example, the camera 1230 can be a UDC. The UDC can be disposed under a display panel and generate a UDC image based on light received through a hole disposed between display pixels. For example, the UDC image can include a user's face. In this example, through image restoration of the UDC image, user authentication can be performed based on the user's face. The camera 1230 can be a three-dimensional (3D) camera that provides a 3D image including depth information of an object.

[0084] The storage 1240 can include a computer readable storage medium or a computer readable storage device. The storage 1240 can store a larger amount of information for a longer period of time than the memory 1220. The storage 1240 can include, for example, a magnetic hard disk, a compact disk, a flash memory, a floppy disk, or another form of non-volatile storage known in the related art.

[0085] The input device 1250 can receive input from a user by a conventional input method through a keyboard and a mouse, and by a new input method such as, for example, touch input, voice input, and image input. The input device 1250 can include, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from a user and transmit the detected input to the electronic device 1200.

[0086] The output device 1260 can provide output of the electronic device 1200 to a user through a visual channel, an auditory channel, or a tactile channel. The output device 1260 can include, for example, a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to a user. The network interface 1270 can communicate with an external device through a wired network or a wireless network.

[0087] Herein Figures 1 to 12The described image restoration device, training device, electronic apparatus, and other devices, apparatuses, units, modules, and components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application, where appropriate, include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic gates, controllers, and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and implement instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories that store instructions or software for execution by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or additional processors and additional controllers. The one or more processors, or a processor and a controller, can implement a single hardware component or two or more hardware components. The hardware components can have any one or more of various processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessor, single-instruction multiple-data (SIMD) multiprocessor, multiple-instruction single-data (MISD) multiprocessor, and multiple-instruction multiple-data (MIMD) multiprocessor.

[0088] Figures 1 to 12The methods illustrated in the figures, which perform the operations described in this application, are performed by computing hardware (e.g., by one or more processors or computers) implemented as instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. The one or more processors, or a processor and a controller, can perform a single operation or two or more operations.

[0089] The instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above can be written as a computer program, a code segment, instructions, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as produced by a compiler) directly executable by a processor or computer. In another example, the instructions or software include high-level code executable by a processor or computer using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flow charts illustrated in the figures and corresponding descriptions used herein, which disclose algorithms for performing the operations performed by the hardware components and methods as described above. Programmers of ordinary skill in the art can readily write the instructions or software based on the block diagrams and flow charts illustrated in the figures and corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0090] Instructions or software for controlling a computing hardware processor or computer to implement the hardware components and perform the methods as described above, as well as any associated data, data files, and data structures, can be recorded, stored, or fixed in one or more non-transitory computer-readable storage media or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, hard disk drive (HDD), solid-state drive (SSD), card-type memory such as a multimedia card or a micro card (e.g., secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor and computer can execute the instructions.

[0091] Although the present disclosure includes specific examples, it will be clear to those skilled in the art following the disclosure herein that various changes can be made to form and details of these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered as merely illustrative, and not restrictive. The description of features or aspects in each example should be considered to apply to similar features or aspects in other examples. Suitable results can be achieved if the described techniques are performed in a different order, and / or if the components of the described systems, architectures, devices, or circuits are combined in different ways, and / or are replaced or supplemented by other components or their equivalents.

[0092] Therefore, the scope of the disclosure is not limited by the specific embodiments described herein, but only by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

Claims

1. An image restoration method, comprising: Receive degraded images; Degradation information is determined to indicate degradation factors in a degraded image. The degradation information includes one or both of a noise parameter and a blur parameter, wherein the noise parameter represents the noise information of each pixel in the degraded image, and the blur parameter represents the blur information of each pixel in the degraded image. Based on adjustment conditions, one or both of the noise parameters and fuzzy parameters of the deterioration information are adjusted, and the adjustment conditions include one or both of environmental information and user preferences. Generate and adjust graph data corresponding to one or both of the noise parameters and fuzziness parameters; and A restored image corresponding to the degraded image is generated by performing an image restoration network with the degraded image and one or both of the adjusted noise and blur parameters. The step of generating the restored image includes: inputting input data corresponding to the degraded image into an image restoration network; and using operations between output data from at least one layer of the image restoration network and graph data to adjust the components in the output data indicated by the graph data, in order to generate the restored image corresponding to the degraded image. The adjustment to one or both of the noise parameters and fuzzy parameters of the degradation information based on the adjustment conditions includes at least one of scaling, adding, cropping, and assigning. Scaling scales one or both of the noise parameters and fuzzy parameters of the degradation information by a ratio; adding adds a value to one or both of the noise parameters and fuzzy parameters of the degradation information; cropping restricts one or both of the noise parameters and fuzzy parameters of the degradation information to a value; and assigning assigns a specific value to one or both of the noise parameters and fuzzy parameters of the degradation information. The step of adjusting the component indicated by the graph data in the output data using an operation between the output data and graph data of at least one layer of the image restoration network includes: using the operation between the output data and graph data to remove the component indicated by the graph data in the output data.

2. The image restoration method according to claim 1, wherein, The steps to determine degradation information include: Noise parameters that indicate the level of noise included in the degraded image are determined by analyzing the degraded image.

3. The image restoration method according to claim 2, wherein, The steps of adjusting one or both of the noise parameter and the fuzziness parameter include: Noise parameters are adjusted based on environmental information from the degraded image.

4. The image restoration method according to claim 3, wherein, The steps for adjusting noise parameters include: In response to environmental information corresponding to the first illuminance environment, the noise parameters are adjusted so that the noise parameters indicate the first noise level; and In response to environmental information corresponding to the second illuminance environment, the noise parameters are adjusted so that the noise parameters indicate the second noise level. The illuminance of the second illuminance environment is higher than that of the first illuminance environment, and the second noise level is lower than the first noise level.

5. The image restoration method according to any one of claims 1 to 4, wherein, The degraded image was captured by a camera, and the camera is an under-display camera. The steps for determining the degradation information include: Obtain the blur parameters corresponding to the hardware characteristics of the under-display camera.

6. The image restoration method according to claim 5, wherein, The under-display camera is configured to receive light through holes arranged between the display pixels of the display panel. The hardware characteristics include one or more of the size, shape, depth, and arrangement pattern of the holes.

7. The image restoration method according to claim 6, wherein, The blur parameters include one or more of the following: a first parameter indicating the blur intensity, a second parameter indicating the spacing between artifacts, and a third parameter indicating the intensity of the artifacts.

8. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the image restoration method according to any one of claims 1 to 7.

9. An image restoration device, comprising: processor; and Memory includes instructions that can be executed in the processor. When the instruction is executed in the processor, the processor is configured to: Receive degraded images; Degradation information is determined to indicate degradation factors in a degraded image. The degradation information includes one or both of a noise parameter and a blur parameter, wherein the noise parameter represents the noise information of each pixel in the degraded image, and the blur parameter represents the blur information of each pixel in the degraded image. Based on adjustment conditions, one or both of the noise parameters and fuzzy parameters of the deterioration information are adjusted, and the adjustment conditions include one or both of environmental information and user preferences. Generate and adjust graph data corresponding to one or both of the noise parameters and fuzziness parameters; and A restored image corresponding to the degraded image is generated by performing an image restoration network with the degraded image and one or both of the adjusted noise and blur parameters. The processor is configured to: input input data corresponding to the degraded image into an image restoration network; and adjust the components in the output data indicated by the graph data in the output data using operations between the output data of at least one layer of the image restoration network and the graph data, to generate a restored image corresponding to the degraded image. The adjustment to one or both of the noise parameters and fuzzy parameters of the degradation information based on the adjustment conditions includes at least one of scaling, adding, cropping, and assigning. Scaling scales one or both of the noise parameters and fuzzy parameters of the degradation information by a ratio; adding adds a value to one or both of the noise parameters and fuzzy parameters of the degradation information; cropping restricts one or both of the noise parameters and fuzzy parameters of the degradation information to a value; and assigning assigns a specific value to one or both of the noise parameters and fuzzy parameters of the degradation information. The step of adjusting the component indicated by the graph data in the output data using an operation between the output data and graph data of at least one layer of the image restoration network includes: using the operation between the output data and graph data to remove the component indicated by the graph data in the output data.

10. The image restoration device according to claim 9, wherein, The processor is configured as follows: Noise parameters that indicate the level of noise included in the degraded image are determined by analyzing the degraded image.

11. The image restoration device according to claim 10, wherein, The processor is configured as follows: Noise parameters are adjusted based on environmental information from the degraded image.

12. The image restoration apparatus according to any one of claims 9 to 11, wherein, The degraded image was taken by a camera, and the camera is an under-display camera. The processor is configured as follows: Degradation information is determined using fuzzy parameters that correspond to the hardware characteristics of the under-display camera.

13. The image restoration device according to claim 12, wherein, The under-display camera is configured to receive light through holes arranged between the display pixels of the display panel, and Hardware characteristics include one or more of the size, shape, depth, and arrangement pattern of the holes.

14. An electronic device comprising: camera; and The processor is configured to receive degraded images from the camera; The process involves: determining degradation information indicating degradation factors in a degraded image, the degradation information including one or both of noise parameters and blur parameters, wherein the noise parameter represents the noise information of each pixel in the degraded image, and the blur parameter represents the blur information of each pixel in the degraded image; adjusting one or both of the noise parameter and blur parameter of the degradation information based on adjustment conditions, including one or both of environmental information and user preferences; generating image data corresponding to the adjusted noise parameter and blur parameter; and generating a restored image corresponding to the degraded image by performing an image restoration network using the degraded image and the adjusted noise parameter and blur parameter. The processor is configured to: input input data corresponding to the degraded image into an image restoration network; and adjust the components in the output data indicated by the graph data in the output data using operations between the output data of at least one layer of the image restoration network and the graph data, to generate a restored image corresponding to the degraded image. The adjustment to one or both of the noise parameters and fuzzy parameters of the degradation information based on the adjustment conditions includes at least one of scaling, adding, cropping, and assigning. Scaling scales one or both of the noise parameters and fuzzy parameters of the degradation information by a ratio; adding adds a value to one or both of the noise parameters and fuzzy parameters of the degradation information; cropping restricts one or both of the noise parameters and fuzzy parameters of the degradation information to a value; and assigning assigns a specific value to one or both of the noise parameters and fuzzy parameters of the degradation information. The step of adjusting the component indicated by the graph data in the output data using an operation between the output data and graph data of at least one layer of the image restoration network includes: using the operation between the output data and graph data to remove the component indicated by the graph data in the output data.

15. The electronic device according to claim 14, wherein, The processor is configured as follows: Noise parameters indicating the level of noise included in the degraded image are determined by analyzing the degraded image; and Noise parameters are adjusted based on environmental information from the degraded image.

16. The electronic device according to claim 14 or 15, wherein, The camera is an under-display camera, which is positioned below the display panel and configured to generate images based on light received through holes arranged between the display pixels of the display panel. The processor is configured to use blur parameters corresponding to the hardware characteristics of the under-display camera to determine degradation information, and The hardware characteristics include one or more of the size, shape, depth, and arrangement pattern of the holes.

Images