Image enhancement method and electronic device

Abstract

The invention provides an image enhancement method based on fuzzy segmentation and an electronic device. The method of image enhancement includes: generating a blur segmentation map including indications of blur characteristics of respective pixels of a blurred image, where the blur characteristics are in predetermined blur characteristic categories, where the generating step is performed by classifying the blur characteristics of each pixel of the blurred image as one of the predetermined blur characteristic categories; converting the blurred segmented image into an image residual corresponding to a blurred component of the blurred image; and generating a deblurred image based on the blurred image and the image residual.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0145879 filed on October 27, 2023, and Korean Patent Application No. 10-2024-0014083 filed on January 30, 2024, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a method and an electronic device having image enhancement based on fuzzy segmentation. Background Art

[0003] Neural networks based on deep learning are often used for different kinds of image processing. Neural networks can be trained based on deep learning and then perform inference for the desired purpose by mapping input data and output data that are in a nonlinear relationship to each other. The training ability to generate such a mapping can be referred to as the learning ability of the neural network. A neural network trained for a specific purpose (such as image enhancement) can have a generalization ability to generate relatively accurate outputs in response to input patterns that have not been specifically trained. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a method for enhancing an image comprises: generating a blur segmentation map comprising an indication of blur characteristics of individual pixels of a blurred image, wherein the blur characteristics are in predetermined blur characteristic categories, wherein the generating step is performed by classifying the blur characteristics of each pixel of the blurred image into one of the predetermined blur characteristic categories; converting the blur segmentation map into an image residual corresponding to a blur component of the blurred image; and generating a deblurred image based on the blurred image and the image residual.

[0006] The indication may be a feature representation, and each feature representation may represent a blur characteristic of a corresponding pixel of the blurred image.

[0007] The feature representations may be paired with positionally corresponding pixels of the blurred image, and a first pair of the pairs may include a first feature representation of the feature representations and a first pixel of the pixels, and the first feature representation may represent a blur characteristic of the first pixel.

[0008] An image residual may be generated by performing discrete-to-continuous conversion on the blur segmentation map based on the blur characteristic class, and the image residual may include pixel values ​​in a continuous format.

[0009] A blur segmentation map may be generated using a first neural model pre-trained to segment an input blur image based on clustering blur characteristics of the input blur image, and the blur segmentation map may be converted into an image residual using a second neural network pre-trained to convert the segmentation map into an image with continuous pixel values.

[0010] The step of converting the fuzzy segmentation map into an image residual may include: inputting the fuzzy image and the fuzzy segmentation map into a second neural network.

[0011] The first neural network model can be trained based on the following steps: using a neural kernel estimation model to generate a base kernel of a first training blurred image; generating a first intermediate deconvolution result by performing deconvolution of the first training blurred image using the base kernel; generating a first training blurred segmentation map of the first sample blurred image using the first neural network model; sampling pixels of a first final deconvolution result from the first intermediate deconvolution result by using feature representation of the first sample blurred segmentation map to generate a first final deconvolution result; and training the first neural network model and the second neural network model so that the difference between the first final deconvolution result and the first training clear image is reduced.

[0012] The number of base kernels is set to be the same as the number of predetermined fuzzy characteristic categories.

[0013] The step of generating a first final deconvolution result may include: based on the feature representation at the first position among the feature representations of the first training fuzzy segmentation map, by selecting the pixel value at the position corresponding to the first position from among the pixel values ​​of the first intermediate deconvolution result, determining the pixel value at the position corresponding to the first position of the first final deconvolution result.

[0014] The second neural network model can be trained based on the following steps: after the training of the first neural network model is completed, using the first neural network model to generate a second training fuzzy segmentation map of the second training blurred image; using the second neural network model to convert the second training fuzzy segmentation map into a training image residual, the training image residual corresponding to the difference between the second training blurred image and the second training deblurred image; generating a second training deblurred image based on the second training blurred image and the training image residual; and training the second neural network model so that the difference between the second training deblurred image and the second training clear image is reduced.

[0015] The step of training the second neural network model may include repeatedly updating weights of the second neural network model when no weights of the first neural network model are updated.

[0016] In another general aspect, an electronic device includes: one or more processors; and a memory storing instructions configured to cause the one or more processors to: generate a blur segmentation map from a blurry image by classifying blur characteristics of pixels of the blurry image into predetermined blur characteristic categories and storing indications of the determined blur characteristic categories of individual pixels of the blurry image in the blur segmentation map; generate an image residual based on the blur segmentation map, the image residual corresponding to a blur component of the blurry image; and generate a deblurred image by applying the image residual to the blurry image to remove the blur component of the blurry image.

[0017] The pixels of the blurred image may each have a frequency component and a motion component, and the predetermined blur characteristic classes may correspond to respective clusters of pixels in the frequency-motion domain.

[0018] In which predetermined blur characteristic class a pixel of a blurred image is classified depends on the motion component of the pixel and the blur component of the pixel.

[0019] The blur segmentation map may be a discretization of the frequency-motion values ​​of the pixel values ​​in the blurry image.

[0020] The instructions may also be configured to cause the one or more processors to: generate a blurred segmentation map by inputting the blurred image into a first neural network model, the first neural network model having been pre-trained to cluster pixels of the blurred image according to frequency components and motion components of the pixels of the blurred image, the first neural network model generating the blurred segmentation map; and generate an image residual by inputting the blurred segmentation map into a second neural network model, the second neural network model having been pre-trained to convert the segmentation map of the blurred image into a non-segmented image residual.

[0021] The first neural network model can be trained based on the following steps: using a third neural model trained to estimate the blur kernel of a blurred image to generate a base kernel of a first training blurred image; generating a first intermediate deconvolution result by performing deconvolution of the first training blurred image using the base kernel; generating a first training blurred segmentation map of the first training blurred image using the first neural network model; sampling pixels of a first final deconvolution result from the first intermediate deconvolution result by using frequency-motion feature indications of the first training blurred segmentation map to generate a first final deconvolution result; and training the first neural network model and the third neural network model so that the difference between the first final deconvolution result and the first training clear image is reduced.

[0022] The base kernels may correspond to predetermined fuzzy characteristic categories, respectively.

[0023] In another general aspect, a method for generating a deblurred image from a blurred image is performed by one or more processors, and the method includes: determining a frequency-motion blur category of pixels of the blurred image based on a frequency-motion blur component of the pixels of the blurred image, and storing indications of the determined frequency-motion blur categories in a blur segmentation map, wherein each indication in the blur segmentation map represents the determined frequency-motion blur category of a pixel in the blurred image to which it corresponds in position; and generating a deblurred image based on the blur segmentation map.

[0024] The frequency-motion blur categories may be in a frequency-motion domain of the blurred image, wherein the frequency-motion blur categories correspond to clusters of pixels of the blurred image in the frequency-motion domain, and the method may further include: generating an image residual corresponding to a blur component of the blurred image, and applying the image residual to the blurred image to generate a deblurred image.

[0025] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example of an image enhancement model is shown in accordance with one or more embodiments.

[0027] Figure 2 Example blur characteristics of blur scenarios are shown in accordance with one or more embodiments.

[0028] Figure 3 An example of clustering discretized blur characteristics of pixels of a blurry image to obtain a blur segmentation map according to one or more embodiments is shown.

[0029] Figure 4 An example of using a hybrid method to obtain a fuzzy segmentation map and an image residual according to one or more embodiments is shown.

[0030] Figure 5 Example blurred segmentation maps and image residuals are shown in accordance with one or more embodiments.

[0031] Figure 6 An example training process of a neural clustering model and a neural kernel estimation model according to one or more embodiments is shown.

[0032] Figure 7 An example neural conversion model is shown in accordance with one or more embodiments.

[0033] Figure 8 Example image enhancement methods according to one or more embodiments are shown.

[0034] Fig. 9 An example electronic device is shown in accordance with one or more embodiments.

[0035] Throughout the drawings and detailed description, unless otherwise described or provided, the same or similar reference numerals will be understood to refer to the same or similar elements, features and structures. The drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0036] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is merely an example and is not limited to the order of operations set forth herein, but the order of operations can be changed as clearly after understanding the disclosure of the application, except for the operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the application can be omitted.

[0037] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0038] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form of the article is also intended to include the plural form. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As non-limiting examples, the terms "include", "comprise" and "have" illustrate the existence of the features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0039] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, the component or element may be directly "connected to," "coupled to," or "engaged to" another component or element, or one or more other components or elements may reasonably exist between them. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to," another component or element, there may not be other elements between them. Similarly, expressions such as "between" and "immediately between," as well as "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0040] Although terms such as "first," "second," and "third," or A, B, (a), (b), etc., may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Each of these terms is not used to define, for example, the nature, order, or sequence of the corresponding member, component, region, layer, or portion, but is only used to distinguish the corresponding member, component, region, layer, or portion from other members, components, regions, layers, or portions. Therefore, without departing from the teachings of the examples, the first member, first component, first region, first layer, or first portion mentioned in the examples described herein may also be referred to as the second member, second component, second region, second layer, or second portion.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs based on an understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the disclosure of the present application, and should not be interpreted in an idealized or overly formal sense. The use of the term "may" with respect to an example or embodiment herein (e.g., with respect to what an example or embodiment may include or may achieve) indicates that there is at least one example or embodiment that includes or achieves such a feature, and all examples are not limited thereto.

[0042] Figure 1 An example image enhancement model is shown in accordance with one or more embodiments. Figure 1 , the image enhancement model 100 may include a neural clustering model 110 and a neural conversion model 120. The image enhancement performed by the image enhancement model 100 may include deblurring the input image. For such a task, the input image is often referred to as a blurred image 101. When the image enhancement model 100 is applied to the blurred image 101, the result may be a deblurred image 102.

[0043] The image enhancement task or deblurring may be performed by the image enhancement model 100. The image enhancement model 100 may also be referred to as a deblurring model. The image enhancement model 100 may be performed by one or more processors of an electronic device. However, the image enhancement task performed by the image enhancement model 100 is merely an example, and the example is not limited thereto. For example, in the absence of the image enhancement model 100, the image enhancement task may be performed by one or more processors of an electronic device. The image enhancement model 100 may generate a blur segmentation map 111 representing blur characteristics of each pixel of the blur image 101. The blur segmentation map 111 may be generated by classifying each of the blur characteristics of the pixels of the blur image 101 into one of predetermined blur characteristic categories. There may be a predetermined number of blur characteristic categories, and the blur characteristics of each pixel of the blur image 101 may be classified into one of the blur characteristic categories. For example, there may be a first blur characteristic category, a second blur characteristic category, and a third blur characteristic category, and the blur characteristics of each pixel of the blur image 101 may be determined as one of the first blur characteristic category, the second blur characteristic category, and the third blur characteristic category; the number of blur characteristic categories is not limited.

[0044] The fuzzy segmentation map 111 may include information about the fuzzy category of each pixel of the fuzzy image 101. The fuzzy category of each pixel may correspond to one of the fuzzy characteristic categories determined by the image enhancement model 100. For the purpose of understanding the terminology, the fuzzy category is equivalent to the fuzzy characteristic category, but is a specific form in the fuzzy segmentation map 111. The fuzzy segmentation map 111 may include feature representations for representing the fuzzy characteristics of the pixels of the fuzzy image 101. The pixels of the fuzzy image 101 and the feature representations of the fuzzy segmentation map 111 may form respective corresponding pairs (e.g., may correspond to each other in position on a one-to-one basis). That is, each pair may have a position-based correspondence (the pixels and feature representations in each pair may have the same position in their respective images / maps). For example, the first pixel at the first position of the fuzzy image 101 and the first feature representation at the same position in the fuzzy segmentation map 111 may form a first pair. The first feature representation of the first pair may show / may represent the fuzzy characteristics of the first pixel of the first pair.

[0045] For example, each feature representation may be a one-hot vector. Among the vector elements of the one-hot vector, one element having a value of 1 corresponds to (e.g., identifies) the blur characteristic of the corresponding blurred image pixel, and the other elements of the vector having a value of 0 indicate the blur characteristic that the first pixel lacks. Referring to the previous example of a first pair of a first pixel and a first feature representation, when the first pixel of the first pair has a blur characteristic that is a first blur characteristic category, the first feature representation may represent the vector value corresponding to the first blur characteristic category by 1, and represent the vector values ​​corresponding to the remaining blur characteristic categories (e.g., the second blur characteristic category and the third blur characteristic category) by 0. The one-hot vector is merely an example, and the feature representation may have other formats that may selectively represent one of the blur characteristic categories based on the blur characteristic of the pixel of the blurred image 101.

[0046] The image enhancement model 100 may use the neural clustering model 110 to generate a fuzzy segmentation map 111 corresponding to the blurred image 101. The neural clustering model 110 may be pre-trained to generate a fuzzy segmentation map representing the fuzzy characteristics of the pixels of the blurred image input to the neural clustering model 110 by classifying the fuzzy characteristics of each pixel into one of the predetermined fuzzy characteristic categories for a given input fuzzy image. The training of the image enhancement model 100 is discussed below.

[0047] Based on the generated blurred segmentation map 111, the image enhancement model 100 may convert / transform the blurred segmentation map 111 into an image residual 121, which corresponds to the difference between the blurred image 101 and the deblurred image 102. The image residual 121 is a type of information / image generally obtained for deblurring (although by the techniques described herein). The blurred segmentation map 111 represents the information loss of the blurred image 101 (the information lost due to blurring when the blurred image was taken). The image enhancement model 100 may fuse the blurred image 101 and the blurred segmentation map 111, and apply the neural conversion model 120 to their fusion to generate the image residual 121.

[0048] The difference between the blurred image 101 and the deblurred image 102 may correspond to a blur component of the blurred image 101. The image residual 121 may represent such a blur component. The blur segmentation map 111 may also reflect the blur component, but may reflect other blur-related information of the blurred image (i.e., by representing the blur characteristics of the pixels of the blurred image 101 via the feature representation (discussed above). In other words, the feature representation may represent the type of blur information related to the blur component of the blurred image 101. Since the blurred image 101 and the deblurred image 102 each have a continuous format (i.e., not segmented), the image residual 121 may have a continuous format. Since the number of blur characteristic categories may be predetermined (e.g., three), the blur segmentation map 111 has a discrete format because it selectively / classifiably represents the blur characteristic categories. The blur segmentation map 111 may represent the pixel values ​​of the image residual 121 (which is a continuous format / non-segmented format) in a discrete format based on the blur characteristic categories.

[0049] The image enhancement model 100 may generate an image residual 121 using the neural conversion model 120. As described above, the fuzzy segmentation map 111 may have a discrete format (discretized fuzzy information), and the image residual 121 may have a continuous format. The fuzzy segmentation map 111 may be considered as a discrete version of the image residual 121, and conversely, the image residual 121 may be considered as a continuous version of the fuzzy segmentation map 111. The image residual 121 may represent a feature representation of the fuzzy segmentation map 111 (which is based on the fuzzy characteristic category as pixel values) in a discrete format in a continuous format.

[0050] Since the information loss of the blurred image 101 can be represented in the form of a blurred segmentation map 111 (e.g., inferred based on the blurred image 101), the blurred segmentation map 111 can be converted into an image residual 121 by executing the neural conversion model 120 using input data including the blurred image 101 and the blurred segmentation map 111. For example, the input data can be a concatenation of the blurred segmentation map 111 and the blurred image 101.

[0051] The image residual 121 may represent the blur component of the blurred image 101 in a continuous format. As described above, the blur segmentation map 111 may represent the blur component of the blurred image 101 in a discrete format using blur characteristic categories. The conversion of the blur segmentation map 111 to the image residual 121 may correspond to the conversion of the blur segmentation map 111 in a discrete format to the image residual 121 in a continuous format. Therefore, the conversion may be referred to as a discrete to continuous (D2C) conversion. In other words, the image residual may be generated by performing a D2C conversion on the blur segmentation map based on the blur characteristic category. The neural conversion model 120 may perform the D2C conversion. The neural conversion model 120 may be pre-trained to convert the blur segmentation map into an image residual corresponding to the difference between the blurred image and the deblurred image.

[0052] The image enhancement model 100 may generate a deblurred image 102 based on the blurred image 101 and the image residual 121. For example, the blurred image 101 and the image residual 121 may be fused by an element-by-element (e.g., pixel-by-pixel) addition operation to generate the deblurred image 102. The deblurred image 102 may be a result of an image enhancement task (e.g., deblurring) on ​​the blurred image 101. The deblurred image 102 may be a sharpened version of the blurred image 101 excluding the blur component of the blurred image 101.

[0053] The neural clustering model 110 and / or the neural conversion model 120 may each include / may be a neural network. Such a neural network may include / may be a deep neural network (DNN) including multiple layers. The layers may include an input layer, at least one hidden layer, and an output layer. Each layer may include a node having a connection to another (e.g., adjacent) layer, and the connection may have a trained / learned weight.

[0054] Just to name some non-limiting examples, the DNN may include at least one of a fully connected network (FCN), a convolutional neural network (CNN), and / or a recurrent neural network (RNN). For example, at least some of the layers included in the neural network may correspond to a CNN, and other layers may correspond to an FCN. A CNN may be referred to as a convolutional layer, and an FCN may be referred to as a fully connected layer.

[0055] In the case of CNN, data input to each layer may be referred to as an input feature map, and data output from each layer may be referred to as an output feature map. The input feature map and the output feature map may also be referred to as activation data. When a convolutional layer corresponds to an input layer, its input feature map may be an image (e.g., blurred image 101, blurred segmentation map 111, etc.).

[0056] The neural network of the image enhancement model 100 can be trained based on deep learning, and performs inference suitable for the training purpose by mapping input data and output data in a nonlinear relationship to each other. Deep learning is a machine learning technique for solving problems such as image recognition or speech recognition based on large data sets. Deep learning can be interpreted as an optimization problem solving process of finding a point that minimizes energy while training a neural network using prepared training data.

[0057] Through supervised learning or unsupervised learning of deep learning, the structure of the neural network or the weights corresponding to the model can be obtained, and the input data and the output data can be mapped to each other through the weights. When the width and depth of the neural network are sufficient, the neural network can have sufficient capabilities to achieve the predetermined function. The neural network can achieve optimized performance by learning a sufficiently large amount of training data through an appropriate training process.

[0058] In discussion Figure 2 Previously, it was considered that an image may have a frequency domain / dimension and a motion domain / dimension. That is, the pixels of the image may have varying values ​​in both domains / dimensions. With respect to the frequency domain, generally, the higher frequency portion of the image corresponds to a higher level of information (e.g., a higher level of texture, a higher incidence of features (such as edges), etc.). The lower frequency portion may be relatively more uniform, smoother and / or lacking in features. An image may have blur-related features as well as frequency-related features. Each pixel of an image (e.g., a blurred image) may be quantified or characterized according to a motion component and a frequency component. Some pixels may have a higher frequency and a lower motion, other pixels may have a higher motion and a lower frequency, and so on. The blur characteristic categories discussed above may include, for example, categories associated with different combinations of motion and frequency. For example, a first blur characteristic category may correspond to high frequency and low motion, a second blur characteristic category may correspond to high motion and high frequency, and a third blur characteristic category may correspond to low motion and low frequency. The pixels of the blurred image may be classified into these categories according to the extent of their motion components and frequency components. Figure 2 An example is shown to help understand the motion domain and the frequency domain.

[0059] Figure 2 1 shows blur characteristics of example blur scenarios according to one or more embodiments. Figure 2, the image residual 213 of the first blur situation 210 may include (i) a non-motion region 213a having pixel values ​​reflecting relatively small blur / motion of a region corresponding in position to the first blur situation 210 (e.g., low amplitude (dark)), and (ii) a motion region 213b having pixel values ​​reflecting relatively high blur / motion of a region corresponding in position to the first blur situation 210 (e.g., high amplitude (bright)). The image residual 223 of the second blur situation 220 may include (i) a low-frequency region 223a having pixel values ​​reflecting relatively low frequencies of a region corresponding in position to the second blur situation 220 (e.g., low amplitude), and (ii) a high-frequency region 223b having pixel values ​​reflecting high frequencies of a region corresponding in position to the second blur situation 220 (e.g., high amplitude). The image residual 213 may correspond to a difference between a clear image 211 of the first blurry situation 210 and a blurred image 212 of the first blurry situation 210. The image residual 223 may correspond to a difference between a clear image 221 of the second blurry situation 220 and a blurred image 222 of the second blurry situation 220.

[0060] The blurred image 212 of the first blur situation 210 may include local motion (motion component in a local area / limited area (e.g., a moving leg)). The motion area 213b may be an area affected by the local motion, and the non-motion area 213a may be an area not affected by the local motion. The difference between the motion area 213b and the corresponding area of ​​the clear image 211 may be greater than the difference between the non-motion area 213a and the corresponding area of ​​the clear image 211. The image residual value of the motion area 213b may be greater than the image residual value of the non-motion area 213a.

[0061] The blurred image 222 of the second blurred situation 220 may include a global motion (e.g., corresponding to the camera motion). Both the low-frequency region 223a and the high-frequency region 223b may be affected by the global motion. Compared with the distribution of the pixel values ​​in the low-frequency region 223a, the distribution of the pixel values ​​in the high-frequency region 223b may be relatively uneven, and the difference between the high-frequency region 223b and the corresponding region of the clear image 221 may be relatively greater than the difference between the low-frequency region 223a and the corresponding region of the clear image 211. The image residual value of the high-frequency region 223b may be greater than the image residual value of the low-frequency region 223a.

[0062] The blur characteristics of the blurred image may be determined based on factors that affect blur. For example, factors that affect blur may include the type of motion (e.g., motion and non-motion) and the distribution of pixel values ​​of neighboring pixels (e.g., high frequency or low frequency). For example, pixels in each of the non-motion region 213a, the motion region 213b, the low-frequency region 223a, and the high-frequency region 223b may correspond to different blur characteristics. The blur characteristic category may correspond to the result of clustering different blur characteristics in both the motion and frequency dimensions / domains.

[0063] Figure 3 An example of clustering the discretized blur characteristics of pixels of a blur image to obtain a blur segmentation map according to one or more embodiments is shown. Figure 3 , graphs 310 and 320 may each illustrate blur characteristics of a pixel of blurred image 311 as a function of motion of blurred image 311 and frequency of a local neighborhood of the pixel of blurred image 311. In each of graphs 310 and 320, the horizontal axis indicates a motion dimension (e.g., motion type (motion or non-motion type) according to motion intensity), and the vertical axis may indicate a frequency dimension (e.g., high frequency and low frequency).

[0064] Figure 310 shows the initial blur characteristics of example pixels at positions ①, ②, and ③ of the blurred image 311; although Figure 3 Only three examples are shown, but each pixel of the blurred image 311 may have motion and frequency quantization (e.g., as potential features within a neural network). FIG. 320 shows pixels at positions ①, ②, and ③ of the blurred image 311 in FIG. 310 and Figure 3 320, the initial blur characteristics may be clustered into clusters 321 to 323. As a result of, for example, applying a clustering algorithm (or network / layer) to the blur characteristics of the pixels of the blurred image 311, clusters may be formed, including: pixels having a motion frequency value in the first cluster 321, pixels having a motion frequency value in the second cluster 322, and pixels having a motion frequency value in the third cluster 323. Clusters 321 to 323 are merely examples, and examples are not limited thereto.

[0065] As shown in Figure 320, in this example, all pixels in Figure 310 can be classified as belonging to one of clusters 321 to 323 (or, can be formed into such clusters by clustering on their frequency-motion values / components). Clusters 321 to 323 can define blur characteristic categories. For example, a first cluster 321 can correspond to a first blur characteristic category, a second cluster 322 can correspond to a second blur characteristic category, and a third cluster 323 can correspond to a third blur characteristic category. Through clustering, each pixel of the blurred image 311 is placed into one of the determined clusters, and the cluster to which the pixel is clustered is the blur characteristic category set for the pixel.

[0066] A blur segmentation map 324 representing blur characteristics (e.g., blur characteristic categories (clusters)) of each pixel of the blurred image 311 may be generated by classifying the blur characteristics of each pixel of the blurred image 311 into one of predetermined blur characteristic categories. For example, blur characteristics of pixels belonging to the first cluster 321 may be classified into a first blur characteristic category, blur characteristics of pixels belonging to the second cluster 322 may be classified into a second blur characteristic category, and blur characteristics of pixels belonging to the third cluster 323 may be classified into a third blur characteristic category.

[0067] A fuzzy segmentation map 324 may be generated based on the classification result. The feature representation of the fuzzy segmentation map 324 may selectively represent clusters 321 to 323 and fuzzy characteristic categories. For example, each feature representation in the fuzzy segmentation map 324 may correspond to one of the first feature representation of the first cluster 321 and the first fuzzy characteristic category, the second feature representation of the second cluster 322 and the second fuzzy characteristic category, or the third feature representation of the third cluster 323 and the third fuzzy characteristic category.

[0068] Figure 4 An example of using a hybrid method to obtain a fuzzy segmentation map and an image residual according to one or more embodiments is shown. Figure 4 , the hybrid method 450 may include blur pixel classification 410 and format conversion 420. Pixels of the blur image 401 may each be classified into one of blur characteristic categories (discretized into motion frequency categories) according to the blur pixel classification 410, and a blur segmentation map 411 may be generated according to the classification result. The blur segmentation map 411, which may be considered as a discretized image residual (a set of discrete (classified) values), may be converted into a continuous format image residual 421 in which the discrete values ​​of the blur segmentation map 411 have been dispersed into a continuous format (non-discrete) according to the format conversion 420.

[0069] A regression method 460 may be used that includes pixel regression 430. An image residual 431 of the blurred image 402 may be generated from the pixel regression 430. The regression method 460 is described to illustrate how the general nature of the image residual 421 (blur error) is similar to a conventional image residual 431 from conventional techniques (regression method 460).

[0070] According to an example, a hybrid method 450 may be used instead of a regression method 460 to generate an image residual 421. Blur pixel classification 410 may correspond to a classification problem. Compared to the regression method 460 that directly solves the regression problem of pixel regression 430 to generate an image residual 431, the hybrid method 450 classifies blur characteristics of the blur image 401 into blur pixel classification 410, which is then used to generate a blur segmentation map 411. The hybrid method 450 then performs a format conversion 420 on the blur segmentation map 411 to generate an image residual 421, which may generally be a more efficient problem solving method and may be more suitable for repairing large motion blur.

[0071] Figure 5 An example blurred segmentation map and image residual are shown according to one or more embodiments. Figure 5 , the blurred segmentation map 510 and the image residual 520 may have similar shapes. The blurred segmentation map 510 may represent pixels in a blurred image in a discrete format, and the image residual 520 may represent the blurred component of the blurred image in a continuous format. The image residual 520 may be obtained from the blurred segmentation map 510 by performing a format conversion (e.g., a D2C conversion). The format conversion may disperse or scatter the values ​​of the blurred segmentation map 510. In some examples, this may be accomplished using additional information (i.e., the blurred image from which the blurred segmentation map 510 is obtained). That is, a discrete to continuous format conversion may be achieved by blurring the image (or information obtained from the blurred image). For example, in Figure 1 , see the solid and dashed lines from the blurred image 101 to the neural conversion model 120.

[0072] Figure 6 An example of training a neural clustering model and a neural kernel estimation model according to one or more embodiments is shown. Figure 6 For the training phase, a neural kernel estimation model 620 may be used to generate a base kernel 621 of the sample blurred image 601. An intermediate deconvolution result 631 (intermediate deblurred image) may be generated by performing deconvolution on the sample blurred image 601 using the base kernel 621.

[0073] about Figure 6In the following description, in order to distinguish the description of the training phase from the description of the inference phase (the inference phase is based on the training phase), the term "sample" is added to the data of the training phase; such data may be referred to as sample blurred images 601, sample blurred segmentation maps 611, and sample clear images 602, in contrast to non-training terms of the inference phase (such as blurred images, blurred segmentation maps, and clear images). Training may be performed on a training set with many sample blurred images (possibly with multiple rounds), and the sample blurred image 601 may be just one of the multiple sample blurred images. The term "training" may be used instead of "sample" with the same meaning.

[0074] A neural clustering model 610 (which after training can be used as Figure 1 The neural clustering model 110 in the sample blur segmentation map 601 generates a sample blur segmentation map 611 of the sample blur image 601. The sample blur segmentation map 611 may be composed of feature representations (e.g., motion frequency blur classification) corresponding to the pixels of the sample blur image 601 in position. The final deconvolution result 641 may be generated by sampling pixels from the intermediate deconvolution result 631 (included in the final deconvolution result 641) using the feature representation of the sample blur segmentation map 611. For example, the pixel value of the intermediate deconvolution result 631 (at the position corresponding to the first position in the intermediate deconvolution result 631) may be selected based on the feature representation (e.g., motion frequency blur classification) at the first position in the feature representation of the sample blur segmentation map 611 to determine the pixel value of the final deconvolution result 641 (at the position corresponding to the first position in the deconvolution result 641).

[0075] The neural clustering model 610 may also generate a latent sharp image 612 based on the sample blurred image 601. The latent sharp image 612 may correspond to a temporarily generated sharp image and will not be confused with the sample sharp image 602 (true value (GT)). The neural kernel estimation model 620 may generate a base kernel 621 based on the sample blurred image 601 and the latent sharp image 612. The performance of the base kernel 621 may be improved by using the latent sharp image 612.

[0076] The number of base kernels 621 and the number of intermediate deconvolution results 631 may be set equal to the number of predetermined blur characteristic categories. Figure 6Three base kernels 621 and three intermediate deconvolution results 631 for three blur characteristic categories are shown; three are non-limiting examples. The base kernels 621 and the intermediate deconvolution results 631 may correspond to the blur characteristic categories, respectively. For example, the base kernel 621 may include a first sub-base kernel, a second sub-base kernel, and a third sub-base kernel corresponding to the first blur characteristic category, the second blur characteristic category, and the third blur characteristic category, respectively, and the intermediate deconvolution result 631 may include a first sub-intermediate deconvolution result, a second sub-intermediate deconvolution result, and a third sub-intermediate deconvolution result corresponding to the first blur characteristic category, the second blur characteristic category, and the third blur characteristic category, respectively. The first sub-intermediate deconvolution result, the second sub-intermediate deconvolution result, and the third sub-intermediate deconvolution result 631 may be generated by performing a deconvolution operation 630 on the first base kernel, the second base kernel, and the third base kernel 621.

[0077] The sampling operation 640 may be performed for each blur characteristic category represented by each feature representation of the sample blur segmentation map 611. For example, when the feature representation at the aforementioned example first position (among the feature representations of the sample blur segmentation map 611) represents / is the first blur characteristic category, the sampling operation 640 may be performed based on the first sub-intermediate deconvolution result generated using the first sub-base kernel corresponding to the first blur characteristic category. The pixel value of the first sub-intermediate deconvolution result at the position corresponding to the first position may be sampled as the pixel value (at the position corresponding to the first position) of the final deconvolution result 641.

[0078] The deconvolution operation 630 may be performed in the frequency domain. For example, a sample blurred image in the logarithmic frequency domain and a potential clear image in the logarithmic frequency domain may be generated by performing a logarithmic frequency transform (e.g., logarithmic Fourier transform, logarithmic operation + fast Fourier transform (FFT)) on each of the sample blurred image 601 and the potential clear image 612. The neural kernel estimation model 620 may generate a base kernel in the logarithmic frequency domain based on the sample blurred image in the logarithmic frequency domain and the potential clear image in the frequency domain. The deconvolution operation 630 may be a subtraction operation in the logarithmic frequency domain, and when the deconvolution operation 630 is replaced by a subtraction operation, the computational load may be reduced. An intermediate deconvolution result 631 may be generated by performing an inverse frequency transform (e.g., inverse logarithmic Fourier transform, exponential operation + inverse FFT (IFFT)) on the operation result of the subtraction operation. The spatial domain data may be transformed into the frequency domain data according to the frequency transform, and the frequency domain data may be transformed into the spatial domain data according to the inverse frequency transform.

[0079] The neural clustering model 610 and the neural kernel estimation model 620 can be trained so that the difference between the final deconvolution result 641 and the sample clear image 602 is reduced. For example, the weights of the layers of the relevant model can be adjusted to reduce the back-propagation gradient loss (the loss between the final deconvolution result 641 and the sample clear image 602). The sample clear image 602 can be a GT image for an image enhancement task (e.g., deblurring). When the potential clear image 612 is used, the neural clustering model 610 and the neural kernel estimation model 620 can be trained so that the difference between the final deconvolution result 641 and the sample clear image 602 and the difference between the potential clear image 612 and the sample clear image 602 are reduced. As described above, the neural clustering model 610 and the neural kernel estimation model 620 can be repeatedly trained using other sample blurred images and corresponding sample clear images.

[0080] Figure 7 An example of the training process of a neural conversion model is shown. Figure 7 , the image enhancement model 700 may include a neural clustering model 710 and a neural conversion model 720. Figure 7 The training process can be Figure 6 After the training process is completed. Figure 6 When the training process of the neural clustering model 710 is completed, the training process of Figure 7 The training process is used to train the neural conversion model 720.

[0081] More specifically, when the training of the neural clustering model 710 is completed, the trained neural clustering model 710 can be used to generate a sample blur segmentation map 711 of the sample blur image 701. The sample blur image 701, the sample blur segmentation map 711, and the sample clear image 702 can be used together with Figure 6 The sample blurred image 601, the sample blurred segmentation map 611, and the sample clear image 602 are the same or different. In order to distinguish the description of the inference stage and the training stage, the data of the training stage are represented by the beginning of "sample", that is, the sample blurred image 701, the sample blurred segmentation map 711, the sample image residual 721, the sample deblurred image 731, and the sample clear image 702 can be used in the training stage to contrast with the terminology of the inference stage (that is, blurred image, blurred segmentation map, image residual, deblurred image, and clear image) omitting "sample".

[0082] The sample blur segmentation map 711 may be converted into a sample image residual 721 (the sample image residual 721 corresponds to the difference between the sample blur image 701 and the sample deblurred image 731) using the neural conversion model 720. Since information loss of the sample blur image 701 may occur when the sample blur segmentation map 711 is generated based on the sample blur image 701, the sample blur segmentation map 711 may be converted into the sample image residual 721 by executing / applying the neural conversion model 720 using input data including the sample blur image 701 and the sample blur segmentation map 711. For example, the sample blur segmentation map 711 and the sample blur image 701 may be concatenated to form an input of the neural conversion model 720.

[0083] A sample deblurred image 731 may be generated based on the sample blurred image 701 and the sample image residual 721. For example, the sample blurred image 701 and the sample image residual 721 may be fused through an element-by-element (eg, pixel-by-pixel) addition operation 730 to generate the sample deblurred image 731.

[0084] The neural conversion model 720 may be trained while the neural clustering model 710 is frozen (not updated) so that the difference between the sample deblurred image 731 and the sample sharp image 702 decreases. The neural conversion model 720 may be repeatedly trained using other sample blurry images and sample sharp images paired therewith.

[0085] Figure 8 An example of an image enhancement method is shown. Figure 8 In operation 810, a blur segmentation map representing blur characteristics of each of the pixels of the blurred image may be generated by classifying blur characteristics of the pixels of the blurred image into one of predetermined blur characteristic categories. In operation 820, the blur segmentation map may be converted into an image residual that may be a difference between the blurred image and the deblurred image. In operation 830, a deblurred image may be generated based on the blurred image and the image residual.

[0086] The blurred segmentation map may include a feature representation, and the feature representation may be used to represent blur characteristics of each pixel of the blurred image.

[0087] The feature representation may be paired with a corresponding pixel of the blurred image. A first pair may include a first feature representation in the feature representation and a first pixel in the pixels, and the first feature representation may represent a blur characteristic of the first pixel.

[0088] The image residual may represent the feature representation of the fuzzy segmentation map in a discrete format into pixel values ​​in a continuous format based on the fuzzy characteristic category.

[0089] Operation 810 may include generating a blurred segmentation map corresponding to the blurred image using a pre-trained neural clustering model, and operation 820 may include converting the blurred segmentation map into an image residual using the pre-trained neural conversion model.

[0090] Operation 820 may include converting the blurred segmentation map into an image residual by executing a neural conversion model using input data including the blurred image and the blurred segmentation map.

[0091] The neural clustering model can be trained based on the following steps: using a neural kernel estimation model to generate a base kernel of a first sample blurred image, generating a first intermediate deconvolution result by performing deconvolution of the first sample blurred image using the base kernel, generating a first sample blurred segmentation map of the first sample blurred image using the neural clustering model, sampling pixels of the first intermediate deconvolution result by using feature representation of the first sample blurred segmentation map to generate a first final deconvolution result, and training the neural clustering model and the neural kernel estimation model so that the difference between the first final deconvolution result and the first sample clear image is reduced.

[0092] The number of base kernels may be set to be the same as the number of predetermined fuzzy characteristic categories.

[0093] The step of generating a first final deconvolution result may include: based on the feature representation at the first position among the feature representations of the first sample fuzzy segmentation map, by selecting the pixel value at the position corresponding to the first position from among the pixel values ​​of the first intermediate deconvolution result, determining the pixel value at the position corresponding to the first position of the first final deconvolution result.

[0094] The neural conversion model can be trained based on the following steps: in response to the completion of the training of the neural clustering model, a second sample blur segmentation map of the second sample blurry image is generated using the neural clustering model, the second sample blurry segmentation map is converted into a sample image residual using the neural conversion model, the sample image residual corresponds to the difference between the second sample blurry image and the second sample deblurred image, a second sample deblurred image is generated based on the second sample blurry image and the sample image residual, and the neural conversion model is trained when the neural clustering model is frozen so that the difference between the second sample deblurred image and the second sample clear image is reduced.

[0095] Fig. 9 An example configuration of an electronic device according to one or more embodiments is shown. Fig. 9, the electronic device 900 may include a processor 910, a memory 920, a camera 930, a storage device 940, an input device 950, an output device 960, and a network interface 970. The processor 910, the memory 920, the camera 930, the storage device 940, the input device 950, the output device 960, and the network interface 970 may communicate with each other via a communication bus 980. For example, the electronic device 900 may be implemented as a mobile device (e.g., a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (e.g., a smart watch, a smart bracelet, smart glasses, etc.), a computing device (e.g., a desktop computer, a server, etc.), a home appliance (e.g., a television (TV), a smart TV, a refrigerator, etc.), a security device (e.g., a door lock, etc.), a vehicle (e.g., an automatic vehicle, a smart vehicle, etc.), or at least a portion of an aircraft (drone). The electronic device 900 may execute the above-referenced Figures 1 to 8 Describe the image enhancement task (e.g., deblurring) and / or training task.

[0096] The one or more processors 910 may execute instructions or functions to be executed in the electronic device 900. For example, the one or more processors 910 may process instructions stored in the memory 920 or the storage device 940. The one or more processors 910 may execute the instructions or functions to be executed in the electronic device 900. Figures 1 to 8 The memory 920 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory 920 may store instructions to be executed by one or more processors 910, and may store relevant information when the electronic device 900 is executing software and / or applications. For example, the memory 920 may store one or more of a neural kernel estimation model, a neural clustering model, and a neural conversion model.

[0097] The camera 930 may take photos and / or videos. The photos and / or videos may be used as input blurred images, and a deblurring task may be performed on the input blurred images. The storage device 940 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The storage device 940 may store a larger amount of information than the memory 920, and store information for a longer period of time. For example, the storage device 940 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other forms of non-volatile memory known in the art (excluding the signal itself).

[0098] The input device 950 may receive input from a user through a traditional input scheme using a keyboard and a mouse and through new input schemes such as touch input, voice input, and image input. For example, the input device 950 may detect input from a keyboard, a mouse, a touch screen, a microphone, or a user, and may include any other device configured to transmit the detected input to the electronic device 900. The output device 960 may provide the output of the electronic device 900 to the user through a visual channel, an auditory channel, or a tactile channel. The output device 960 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device configured to provide output to the user. The network interface 970 may communicate with an external device via a wired network or a wireless network.

[0099] The examples described herein can be implemented using hardware components, software components, and / or combinations thereof. The processing device can be implemented using one or more general-purpose computers or special-purpose computers (such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of responding to and executing instructions in a defined manner). The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and generate data in response to the execution of the software. For the sake of brevity, the description of the processing device is used as a singular form; however, those skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors, or a single processor and a single controller. In addition, different processing configurations are possible (such as parallel processors).

[0100] Software may include a computer program, a piece of code, instructions, or a specific combination thereof to independently or collectively instruct or configure a processing device to operate as desired. Software and data may be stored in any type of machine, component, physical or virtual device, or computer storage medium or computer storage device that can provide instructions or data to a processing device or parse instructions or data by a processing device. Software may also be distributed on networked computer systems so that the software is stored and executed in a distributed manner. Software and data may be stored by one or more non-transitory computer-readable recording media.

[0101] The method according to the above example may be recorded in a non-transitory computer-readable medium including program instructions for implementing various operations of the above example. The medium may also include data files, data structures, etc., alone or in combination with program instructions. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purpose of the example, or they may be types known and available to technicians in the field of computer software. Examples of non-transitory computer-readable media include: magnetic media (such as hard disks, floppy disks, and tapes; optical media (such as CD-ROM disks, DVDs, and / or Blu-ray discs); magneto-optical media, (such as optical discs); and hardware devices specifically configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drive, memory card, memory stick, etc.), etc.). Examples of program instructions include both machine code (such as generated by a compiler) and files including higher-level code that can be executed by a computer using a parser.

[0102] Here about Figures 1 to 9The computing devices, electronic devices, processors, memories, image sensors / cameras, displays, information output systems and hardware, storage devices, and other devices, devices, units, modules, and components described are implemented by hardware components or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components for performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. A hardware component implemented by a processor or a computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware component may also access, manipulate, process, create and store data in response to the execution of instructions or software. For the sake of brevity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0103] Figures 1 to 9The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as instructions or software described above for performing the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0104] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include higher-level code that is executed by one or more processors or computers using a parser. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0105] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory, card memory (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0106] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be descriptive only and not for limiting purposes. The description of features or aspects in each example is considered to be applicable to similar features or similar aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents.

[0107] Therefore, in addition to the above disclosure, the scope of the disclosure may be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. A method for image enhancement, the method comprising: generating a blur segmentation map comprising an indication of blur characteristics of respective pixels of the blurred image, wherein the blur characteristics are in a predetermined blur characteristic class, wherein the step of generating is performed by classifying the blur characteristics of each pixel of the blurred image into one of the predetermined blur characteristic classes; converting the blurred segmentation map into an image residual corresponding to a blurred component of the blurred image; and A deblurred image is generated based on the blurred image and the image residual.

2. The method according to claim 1, wherein: The indications are feature representations, and each feature representation represents a blur characteristic of a corresponding pixel of the blurred image.

3. The method according to claim 2, wherein: The feature representation is paired with the corresponding pixels in the blurred image, and A first one of the pairs includes a first one of the feature representations and a first one of the pixels, and the first feature representation represents a blur characteristic of the first pixel.

4. The method according to any one of claims 1 to 3, wherein: An image residual is generated by performing a discrete-to-continuous conversion on the fuzzy segmentation map based on the fuzzy characteristic category, and the image residual includes continuous pixel values.

5. The method according to any one of claims 1 to 3, wherein: generating a fuzzy segmentation map using a first neural network model, the first neural network model being pre-trained to segment an input fuzzy image based on clustering fuzzy features of the input fuzzy image, and The blurred segmentation map is converted into an image residual using a second neural network model, which is pre-trained to convert the segmentation map into an image with continuous pixel values.

6. The method according to claim 5, wherein: The step of converting the fuzzy segmentation map into an image residual comprises: inputting the fuzzy image and the fuzzy segmentation map into a second neural network model.

7. The method according to claim 5, wherein: The first neural network model is trained based on the following steps: Generate a basis kernel of a first training blurred image using a neural kernel estimation model; generating a first intermediate deconvolution result by performing deconvolution of the first training blurred image using the base kernel; Generate a first training fuzzy segmentation map of a first training fuzzy image using a first neural network model; By using the feature representation of the first training fuzzy segmentation map, sampling the pixels of the first intermediate deconvolution result to generate a first final deconvolution result; as well as The first neural network model and the neural kernel estimation model are trained so that the difference between the first final deconvolution result and the first training clear image is reduced.

8. The method according to claim 7, wherein: The number of base kernels is set to be the same as the number of predetermined fuzzy characteristic categories.

9. The method according to claim 7, wherein: The steps of generating the first final deconvolution result include: Based on the feature representation at the first position among the feature representations of the first training fuzzy segmentation map, the pixel value at the position corresponding to the first position of the first final deconvolution result is determined by selecting the pixel value at the position corresponding to the first position from the pixel values ​​of the first intermediate deconvolution result.

10. The method according to claim 5, wherein: The second neural network model is trained based on the following steps: After the training of the first neural network model is completed, using the first neural network model to generate a second training fuzzy segmentation map of the second training fuzzy image; converting the second training blurred segmentation map into a training image residual using a second neural network model, the training image residual corresponding to a difference between the second training blurred image and the second training deblurred image; generating a second training deblurred image based on the second training blurred image and the training image residual; as well as The second neural network model is trained so that the difference between the second training deblurred image and the second training clear image is reduced.

11. The method according to claim 10, wherein: The step of training the second neural network model includes repeatedly updating weights of the second neural network model when no weights of the first neural network model are updated.

12. An electronic device comprising: one or more processors; as well as A memory storing instructions configured to cause the one or more processors to: generating a blur segmentation map from the blurred image by classifying blur characteristics of pixels of the blurred image into predetermined blur characteristic classes and storing in the blur segmentation map an indication of the determined blur characteristic class for respective pixels of the blurred image; Generate an image residual based on the fuzzy segmentation map, where the image residual corresponds to a fuzzy component of the fuzzy image; as well as A deblurred image is generated by applying the image residual to the blurred image to remove the blur component of the blurred image.

13. The electronic device according to claim 12, wherein: The pixels of the blurred image each have a frequency component and a motion component, wherein predetermined blur characteristic classes correspond to respective clusters of pixels in the frequency-motion domain.

14. The electronic device according to claim 13, wherein: In which predetermined blur characteristic class a pixel of a blurred image is classified depends on the motion component of the pixel and the blur component of the pixel.

15. The electronic device according to any one of claims 12 to 14, wherein: The fuzzy segmentation map is a discretization of the frequency-motion values ​​of the pixel values ​​in the blurred image.

16. The electronic device according to any one of claims 12 to 14, wherein: The instructions are further configured to cause the one or more processors to: Generating a blurred segmentation map by inputting the blurred image into a first neural network model, the first neural network model having been pre-trained to cluster pixels of the blurred image according to frequency components and motion components of the pixels of the blurred image to generate a blurred segmentation map; as well as An image residual is generated by inputting the blurred segmentation map into a second neural network model, which has been pre-trained to convert the segmentation map of the blurred image into a non-segmented image residual.

17. The electronic device according to claim 16, wherein: The first neural network model is trained based on the following steps: generating a basis kernel of the first training blurred image using a third neural network model trained to estimate a basis kernel of the blurred image; generating a first intermediate deconvolution result by performing deconvolution of the first training blurred image using the base kernel; Generate a first training fuzzy segmentation map of a first training fuzzy image using a first neural network model; By sampling pixels of the first intermediate deconvolution result using the frequency-motion feature indication of the first training fuzzy segmentation image, a first final deconvolution result is generated; as well as The first neural network model and the third neural network model are trained so that the difference between the first final deconvolution result and the first training clear image is reduced.

18. The electronic device according to claim 17, wherein: The base kernels correspond to predetermined fuzzy characteristic categories respectively.

19. A method for generating a deblurred image from a blurred image, the method being performed by one or more processors, the method comprising: determining a frequency-motion blur class of pixels of the blurred image based on a frequency-motion blur component of the pixels of the blurred image, and storing indications of the determined frequency-motion blur classes in a blur segmentation map, wherein each indication in the blur segmentation map represents the determined frequency-motion blur class of a positionally corresponding pixel in the blurred image; and Generate a deblurred image based on the blurred segmentation map.

20. The method according to claim 19, wherein: The frequency-motion blur categories are in a frequency-motion domain of the blurred image, wherein the frequency-motion blur categories correspond to clusters of pixels of the blurred image in the frequency-motion domain, wherein the method further comprises: generating an image residual corresponding to a blur component of the blurred image, and applying the image residual to the blurred image to generate a deblurred image.

Images