Image enhancement method and apparatus

Abstract

Image enhancement methods and apparatuses are disclosed. The image enhancement method includes obtaining an input image, estimating a noise distribution of the input image by performing a noise model based on the input image, and generating an enhanced image by performing an image enhancement model based on the input image and the estimated noise distribution.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2021-0076599, filed June 14, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. TECHNICAL FIELD

[0002] The following description relates to an image enhancement method and apparatus. BACKGROUND

[0003] As a non-limiting example, a deep learning-based neural network can be used for image enhancement, for example, after the deep learning-based neural network has been trained based on deep learning. For example, a neural network can perform inference by mapping input data and output data in a non-linear relationship to each other according to a purpose of training, for example. The ability to be trained to generate such a mapping can be referred to as a learning ability. A neural network can have a generalization ability to generate a relatively accurate output for an input pattern for which the neural network has not been trained. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in determining the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method having image enhancement includes obtaining an input image, estimating a noise distribution of the input image by executing a noise model based on the input image, and generating an enhanced image by executing an image enhancement model based on the input image and the estimated noise distribution.

[0006] The step of obtaining the image can include capturing the input image using a camera.

[0007] The step of estimating the distribution of the noise can include executing the noise model based on pixel data of the input image, pixel position information of the input image, and a shooting parameter of the input image.

[0008] The shooting parameter can include at least one of an International Organization for Standardization (ISO) value, an exposure time, and Bayer pattern information related to the input image.

[0009] The noise model can be a regression model that has been trained before the input image is obtained, and the noise model can provide a mapping relationship between input image data, pixel position information of the input image data, and a corresponding shooting parameter of the input image data, and output noise distribution data.

[0010] The method can further include generating the noise model by training the noise model in training to have a mapping relationship between pixel data of an input image, pixel position information of the input image data, and a corresponding shooting parameter of the input image data and output noise distribution data.

[0011] The method can further include generating the image enhancement model by training the image enhancement model in training to have a mapping relationship between output enhanced image data and input image data, pixel position information of the input image data, and a corresponding shooting parameter of the input image data.

[0012] The noise model can be a noise model in training, and / or the image enhancement model can be an image enhancement model in training.

[0013] The noise model can be configured to estimate a noise distribution of an input image based on respective distances from a center of the input image to each of a plurality of pixels of the input image.

[0014] The noise distribution can include a variance or a standard deviation of noise values of the plurality of pixels of the input image.

[0015] The step of generating an enhanced image can include generating input data by concatenating the input image and the estimated noise distribution, and executing the image enhancement model using the generated input data.

[0016] The image enhancement model can be a machine learning model that has been pre-trained before obtaining the input image, and the trained enhancement model can provide a mapping relationship between input image data and a corresponding estimated noise distribution and output enhanced image data.

[0017] The method can further include generating the image enhancement model by training the image enhancement model based on training images generated via use of the noise model.

[0018] In one general aspect, there is provided a non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform any one, any combination, or all of the operations or methods described herein.

[0019] In one general aspect, a processor-implemented method includes training a noise model based on training images and corresponding pixel position information, executing the trained noise model to generate a degraded training image of a clean training image provided to the trained noise model, determining a training data set based on the clean training image and the degraded training image, and training an image enhancement model based on the training data set.

[0020] The method can further include capturing an input image, estimating a noise distribution of the input image by executing the trained noise model based on the input image, generating a degraded image of the input image based on the input image and the noise distribution of the input image, and generating an enhanced image by executing the trained image enhancement model based on the input image and the degraded image of the input image.

[0021] The training of the noise model can include generating a first clean training image from the first noise training image, determining a noise component of the first noise training image based on a difference between the first noise training image and the first clean training image, and training the noise model to have a mapping relationship of a distribution based on the noise component, pixel data, and pixel position information of the first clean training image.

[0022] The determining of the noise component can include determining Gaussian noise pixel data and Gaussian clean pixel data by applying a Gaussian data transform to the first noise pixel data of the first noise training image and the first clean pixel data of the first clean training image, and determining the noise component based on a difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

[0023] The training of the noise model can include training the noise model based on a shooting parameter of the first noise training image, wherein the shooting parameter can include at least one of an International Organization for Standardization (ISO) value, an exposure time, or Bayer pattern information.

[0024] The training of the noise model can include removing a clipping effect of a pixel value of the training image by de-clipping.

[0025] The generating of the degraded training image can include determining a noise distribution corresponding to a degradation condition by providing the clean training image and the degradation condition to the noise model, and generating the degraded training image by applying the noise distribution to the clean training image.

[0026] The determining of the training data set can include generating a training input by concatenating the noise distribution and the degraded training image.

[0027] The training of the degraded training image can include applying a blur effect to the clean training image using a blur model.

[0028] In one general aspect, an apparatus having image enhancement includes a processor configured to estimate a noise distribution of an input image by executing a noise model based on the obtained input image, and generate an enhanced image of the input image by executing an image enhancement model based on the input image and the estimated noise distribution.

[0029] The processor can be further configured to execute the noise model based on pixel data, pixel position information, and a shooting parameter of the input image.

[0030] The noise model can be a trained or training-in-progress noise model, and / or the image enhancement model can be a trained or training-in-progress image enhancement model.

[0031] The device can further include a camera to obtain the input image and a memory to store instructions that, when executed by the processor, configure the processor to perform the steps of estimating the noise distribution and generating the enhanced image, wherein the device can be a mobile phone electronic device.

[0032] In one general aspect, an electronic device includes a processor configured to generate a plurality of noise distributions of an image based on a plurality of shooting parameters of the image and pixel information of the image, respectively, generate a plurality of degraded images of the image using the generated plurality of noise distributions, and train an image enhancement model based on the plurality of degraded images.

[0033] The noise model can be a trained noise model, and the electronic device can be a mobile phone.

[0034] In one general aspect, an electronic device includes a camera to shoot an input image and a processor configured to estimate a noise distribution of the input image by executing a noise model that considers pixel information of the input image and considers shooting parameters and / or a plurality of corresponding pixel position information of the input image when the input image is shot, and generate an enhanced image of the input image by executing an image enhancement model, wherein the image enhancement model is executed based on the input image and the estimated noise distribution.

[0035] The electronic device can be a mobile phone.

[0036] Other features and aspects will become apparent from the following specific description, drawings and claims. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 An example of image enhancement according to one or more embodiments is illustrated.

[0038] Figure 2 An example of estimating noise by a noise model according to one or more embodiments is illustrated.

[0039] Figure 3 An example of training a noise model according to one or more embodiments is illustrated.

[0040] Figure 4 An example of a mapping model based on pixel data and noise distribution according to one or more embodiments is illustrated.

[0041] Figure 5An example of a mapping model based on pixel data, pixel location information, and a noise distribution is shown in accordance with one or more embodiments.

[0042] Figure 6 An example of a real noise image and an example of a noise distribution mapping is shown in accordance with one or more embodiments.

[0043] Figure 7 An example of applying a Gaussian distribution through data transformation is shown in accordance with one or more embodiments.

[0044] Figure 8 An example of suppressing clipping effects through de-clipping is shown in accordance with one or more embodiments

[0045] Figure 9 An example of generating a training image is shown in accordance with one or more embodiments.

[0046] Figure 10 An example of generating a degraded training image by estimating a noise distribution from a noise model is shown in accordance with one or more embodiments.

[0047] Figure 11 An example of data transformation in the process of generating a degraded training image is shown in accordance with one or more embodiments.

[0048] Figure 12 and Figure 13 An example of generating a blur is shown in accordance with two or more embodiments.

[0049] Figure 14 An example of training an image enhancement model is shown in accordance with one or more embodiments.

[0050] Figure 15 An example of enhancing an image using a noise model and a graph enhancement model is shown in accordance with one or more embodiments.

[0051] Figure 16 An example of a training device is shown in accordance with one or more embodiments.

[0052] Figure 17 An example of an image enhancement device is shown in accordance with one or more embodiments.

[0053] Figure 18 An example of a method with image enhancement is shown in accordance with one or more embodiments.

[0054] Figure 19 An example of a method with training is shown in accordance with one or more embodiments.

[0055] Figure 20 An example of an electronic device is shown in accordance with one or more embodiments.

[0056] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals are to be understood to represent same or similar elements, features, and structures. The drawings can not be to scale and the dimensions, proportions, and shapes of the components in the drawings can be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0057] The following detailed description is provided to help the reader obtain a thorough understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the present disclosure. For example, the order of the operations described herein is merely an example and is not limited to the order described herein, but can be changed as will be clear to one of ordinary skill in the art after understanding the present disclosure, except for operations that must occur in a specific order. Also, descriptions of features known after understanding the present disclosure can be omitted for clarity and conciseness.

[0058] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, these examples have been provided so that this disclosure will be thorough and complete, and will fully convey the scope of the methods, devices, and / or systems described herein to one of ordinary skill in the art after understanding the present disclosure.

[0059] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including," and "has," "having," and the like are meant to be open-ended and non-limiting.

[0060] Throughout the specification, when an element is described as "connected to" or "coupled to" another element, it can be "connected to" or "coupled to" the other element directly or via one or more other elements. In contrast, when an element is described as being "directly connected to" or "directly coupled to" another element, there are no other elements interposed between them. Also, expressions such as "between," "among," and "adjacent to," for example, are to be interpreted in the same fashion as "directly connected to" and "directly coupled to," respectively. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0061] Although terms such as "first", "second", and "third" can be used herein to describe various components, assemblies, regions, layers or portions, these components, assemblies, regions, layers or portions should not be limited by these terms. Rather, these terms are only used to distinguish one component, assembly, region, layer or portion from another component, assembly, region, layer or portion. Thus, the "first", "second", "third", etc. components, assemblies, regions, layers or portions described in the examples herein can also be referred to as "second", "third", "fourth" etc. components, assemblies, regions, layers or portions without departing from the teachings of the examples.

[0062] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs and based on the understanding of the present disclosure. Unless specifically defined otherwise herein, terms such as those defined in a generally used dictionary are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense. The use of the term "may" with respect to examples or embodiments (e.g., as to what an example or embodiment can include or implement) indicates that there are at least one example or embodiment for which the feature is included or implemented, and that not all examples are limited to this.

[0063] Figure 1 Examples of image enhancement according to one or more embodiments are illustrated. Referring to Figure 1 , an example electronic device 1 can include an image enhancement device 100, which can receive an input image 101 and generate an enhanced image 103 by enhancing the image quality of the input image 101 using a noise model 110 and an image enhancement model 120. For example, the electronic device 1 can be or can be implemented as at least a portion of, for example, a mobile device (such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (such as a smart watch, a smart band, smart glasses, etc.), a computing device (such as a desktop, a server, etc.), a home appliance (such as a television (TV), a smart TV, a refrigerator, etc.), a security device (such as a door lock, etc.), and a vehicle (such as an autonomous vehicle, a smart vehicle, etc.), as non-limiting examples.

[0064] The input image 101 can be generated by a camera, received through a network interface, and / or pre-stored in a storage device of the example electronic device 1, for example, as non-limiting examples. The electronic device 1 can be or correspond to, for example, a mobile device (such as a mobile phone, a smart phone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (such as a smart watch, a smart band, smart glasses, etc.), a computing device (such as a desktop, a server, etc.), a home appliance (such as a television (TV), a smart TV, a refrigerator, etc.), a security device (such as a door lock, etc.), and a vehicle (such as an autonomous vehicle, a smart vehicle, etc.), as non-limiting examples. Figure 20electronic device 2000. The input image 101 can include a degraded aspect (e.g., noise, blur, etc.), and the image enhancement device 100 can perform removal of the degraded aspect using the noise model 110 and the image enhancement model 120. For example, the degraded aspect can be removed or substantially removed.

[0065] The image enhancement device 100 can estimate a noise distribution 102 of the input image 101 by performing the noise model 110 based on the input image 101. The noise model 110 can be, for example, a regression model that is pre-learned and / or trained using a mapping relationship between input image data and output data corresponding to a noise distribution prior to an inference operation with respect to the input image 101. The output noise distribution 102 can represent a distribution of noise values of pixels of the input image 101. The noise distribution 102 can include, for example, a variance and / or a standard deviation of the noise values.

[0066] The image enhancement device 100 can generate an enhanced image 103 by performing the image enhancement model 120 based on the input image 101 and the noise distribution 102 generated by the noise model. The image enhancement model 120 can be, for example, a machine learning model that is pre-learned and / or trained to have a mapping relationship between input data corresponding to the input image and the noise distribution and output data corresponding to the enhanced image. As a non-limiting embodiment, the image enhancement model 120 can include, for example, a deep learning network. In such an example, for example, the image enhancement model 120 and the noise model 110 can respectively perform inference operations according to one or more respectively trained, e.g., purposes, by mapping input data and output data in a non-linear relationship to each other. The training or learning of the noise model 110 and / or the image enhancement model 120 can be based on supervised learning and / or unsupervised learning.

[0067] Figure 2 An example of estimating noise by a trained / learned noise model according to one or more embodiments is illustrated. Referring to FIG. 1, the image enhancement device 100 can include a noise model 110 and an image enhancement model 120. The image enhancement device 100 can receive an input image 101 from an electronic device 2000. The input image 101 can include a degraded aspect (e.g., noise, blur, etc.), and the image enhancement device 100 can perform removal of the degraded aspect using the noise model 110 and the image enhancement model 120. For example, the degraded aspect can be removed or substantially removed. Figure 2The noise model 200 may be pre-trained / learned, for example, based on various training images, to have a mapping relationship between input data and output data. Given input data 210 of an input image, the noise model 200 may estimate output data 220 corresponding to the input data 210. As a non-limiting example, the input data 210 may include at least one of pixel data 211, pixel location information 212, and shooting parameters 213 (e.g., shooting parameters 213 of the captured input image or various modeling training shooting parameters 213). For example, the output data 220 may include a noise distribution 221. For example, given pixel data 211, pixel location information 212, and shooting parameters 213, the noise model 200 may estimate a noise distribution 221 corresponding to the pixel data 211, pixel location information 212, and shooting parameters 213. Similarly, the noise model 200 may be trained based on inputting various types of input data (i.e., not just pixel data) into the noise model during training.

[0068] For example, given an input image, noise model 200 can estimate the noise distribution 221 of the input image based on each of pixel data 211, pixel location information 212, and shooting parameters 213. For example, pixel data 211 can include pixel values ​​of the input image. As a non-limiting example, pixel location information 212 can include pixel coordinate values ​​and / or distance from the center. As pixel data 211 of the input image, clean pixel data from a clean version of the input image can be used. As a non-limiting example, shooting parameters 213 can include image shooting information of the input image (e.g., at least one of International Organization for Standardization (ISO) values, exposure time, and Bayer pattern information). For example, exposure time can correspond to shutter speed. For example, Bayer pattern information can include information associated with color filters (e.g., red, green, blue (RGB)) used in pixel data 211. For example, noise model 200 can use various information sets (including, for example, pixel data 211, pixel location information 212, and shooting parameters 213) to estimate noise distribution 221, which is an estimate that approximates the real or natural noise distribution.

[0069] Figure 3 Examples of training noise models according to one or more embodiments are shown. (Refer to...) Figure 3A clean training image 320 (i.e., a clean training modeling image 320) can be generated based on the noisy training image 310 (i.e., the noisy training modeling image 310). For example, the noisy training image 310 can be obtained by photographing a flat monochrome (e.g., gray) plate (or, in this case, by modeling such a photographed image, for example, based on a pre-stored image considered as a photographed image), and the clean training image 320 can be obtained by performing a blur filter on the noisy training image 310. In this example, the clean training image 320 may correspond to a pseudo-clean image; for example, the noisy training image 310 may correspond to the original version of the photographed image or the original version of the modeled photographed image (e.g., based on a pre-stored image considered as a photographed image), or a version obtained by applying some post-processing to the example photographed image, or the original version of the modeled or pre-stored photographed image. The noise component 330 and noise distribution 340 of the noisy training image 310 can be determined based on the difference between the noisy training image 310 and the clean training image 320. The noise distribution 340 may include the distribution (e.g., variance and / or standard deviation) of the noise values ​​of the pixels in the noisy training image 310. For example, the noise distribution 340 may include the squared value of each difference.

[0070] Input data 301 can be constructed based on training images 310 and 320. Output data 302 can be constructed based on noise data (including, for example, noise component 330 and noise distribution 340). For example, input data 301 can be determined by clean pixel data of clean training image 320, pixel position information of training images 310 and 320, and shooting parameters of noisy training image 310, and output data 302 can be determined by noise distribution 340. Clean pixel data may include pixel values ​​of clean training image 320. As an example, training images 310 and 320 may each be the same size, and pixel position information may include pixel coordinate values ​​and / or distances from the center of training images 310 and 320 as non-limiting examples. As a non-limiting example, shooting parameters may include ISO values, exposure time, and Bayer pattern information used when noisy training image 310 is captured or modeled as a captured image (e.g., based on a pre-stored image considered as a captured image).

[0071] Noise model 300 can be learned or trained using the mapping relationship between input data 301 and output data 302, or has been learned or trained using the mapping relationship between input data 301 and output data 302. For example, regression analysis can be used to train noise model 300, and noise model 300 can correspond to a regression model. As a non-limiting example, regression analysis can correspond to ridge regression. Therefore, noise model 300 can estimate output data 302 corresponding to given input data 301. For example, when given pixel data (e.g., clean pixel data), pixel location information, and shooting parameters of an input image as input data 301, noise model 300 can estimate noise distribution 340 corresponding to input data 301.

[0072] In one example, a two-dimensional (2D) map can be determined based on pixel data, pixel location information, and shooting parameters, respectively. The map can constitute input data. Each component of the 2D map based on pixel data can have a pixel value based on the pixel location of the pixel data. As a non-limiting example, each component of the 2D map based on pixel location information can have a distance value from the center. All components of the 2D map based on shooting parameters can have the same parameter values. For example, all components of a 2D map based on ISO parameters can have the same values ​​according to the ISO setting. Noise distribution 340 can represent the distribution (e.g., variance and / or standard deviation) of the noise values ​​of the pixels in the pixel data as a 2D map based on the pixel location of the pixel data.

[0073] Training of the noise model 300 can be repeatedly performed using various training images (i.e., various training modeling images). For example, under various image capturing environments (e.g., illumination, etc.) and under various image capturing conditions (e.g., ISO value, exposure time, etc.), various noise training images can be generated from images of the captured object (e.g., a flat monochrome panel) or from modeling images based on the object (e.g., based on pre-stored images considered as captured images), and the corresponding clean training images, noise components, and noise distribution can be derived from this. When a "modeling pair or generation pair of input data 301 and output data 302" corresponding to the noise training images is constructed, the noise model 300 can be repeatedly learned or trained to have a mapping relationship with respect to the modeling / generation pairs.

[0074] For example, noise model 300 can be learned or trained on the mapping between "2D images corresponding to pixel data, pixel location information, and shooting parameters, respectively" and "2D images corresponding to noise distribution 340". In this example, the squared values ​​of the graph components of each 2D image can be mapped instead of directly mapping the values ​​of the graph components. Therefore, noise model 300 can output noise distribution 340 of the input image in response to a 2D image input corresponding to pixel data, pixel location information, and shooting parameters of the input image. Noise distribution 340 may have heteroscedastic characteristics, in which the standard deviation is concentrated in a narrow range in the portion with low pixel values ​​and widely distributed in the portion with high pixel values. Therefore, weighted regression (e.g., weighted linear regression) that assigns high weights to low pixels can be used. Furthermore, for example, to apply the actual noise of bias frames obtained in environments with little or no light, modified weighted regression (e.g., weighted linear regression) can be applied to the portion with extremely low pixel values.

[0075] Figure 4 Examples of mapping models based on pixel data and noise distribution according to one or more embodiments are shown. (Refer to...) Figure 4 The points in Figure 400, as shown, indicate distribution values ​​based on pixel values. For example, the pixel values ​​used here may indicate pixel brightness, and the distribution values ​​used here may indicate noise distribution (e.g., variance and / or standard deviation). The points in Figure 400 are derived based on pixel values ​​and distribution values ​​from various images. For example, these points in a corresponding graph similar to Figure 400 may be based on the above regarding… Figure 3 The pixel values ​​and distribution values ​​of the various training images discussed are derived, or derived using a trained noise model based on the pixel values ​​and distribution values ​​of various inference images suitable for inference operations. For example, the clean pixel values ​​of a portion of an image can be derived based on the average pixel values ​​of that portion of the image, and the distribution values ​​of the noise components can be derived based on the difference between the pixel values ​​of that portion and the clean pixel values. The pair of clean pixel values ​​(or average pixel values) and distribution values ​​derived as described above can be represented as points corresponding to the portions in Figure 400.

[0076] Curve 401, as shown, was derived through regression analysis based on the points in Figure 400. Curve 401 represents the mapping between pixel values ​​and distribution values. Based on curve 401, the noise distribution corresponding to the pixel values ​​of the input image can be determined. Curve 401 can correspond to a heteroscedasticity model that assumes the distribution values ​​vary with pixel values. The noise model can more realistically model the noise based on information other than pixel data (e.g., pixel location information and shooting parameters).

[0077] Figure 5Examples of mapping models based on pixel data, pixel location information, and noise distribution according to one or more embodiments are shown. (Refer to...) Figure 5 The points in Figure 500 shown represent an example of mapping pixel data and pixel location information to noise distribution. Pixel data, pixel location information, and noise distribution can correspond to the pixel values, distances, and distribution values ​​in Figure 500. The mapping model of Figure 500 has been derived based on the pixel values, distances, and distribution values ​​of various images, and can be derived based on the pixel values, distances, and distribution values ​​of various images in various examples. For example, a corresponding diagram similar to Figure 500 can be based on the above regarding... Figure 3 The pixel values, distance values, and distribution values ​​of the various training images discussed are derived using a trained noise model, or based on the pixel values, distance values, and distribution values ​​of various inference images suitable for inference operations, using a trained noise model. For example, multiple 2D maps can be derived based on pixel data, pixel location information, and the noise distribution of the image, and a mapping model can be derived based on the mapping relationships between these multiple 2D maps. Figure 4 Compared to the model shown in the figure, the model in Figure 500 also estimates the noise distribution based on, for example, the distance from the image center to each pixel. Furthermore, the model in Figure 500 can be learned or trained using more detailed or complex mapping relationships based on the shooting parameters.

[0078] Figure 6 Examples of actual noisy images and noise distribution maps according to one or more embodiments are shown. The noisy image 610 can be obtained by capturing an image of a monochrome (e.g., gray) panel of the example flat surface, or by capturing an image based on a model of the monochrome panel of the example flat surface (e.g., based on a pre-stored image considered as a captured image), and the noise distribution map 620 can be derived from the noise image 610 through training. The noise distribution map 620 can represent a mapping based on pixel data, pixel location information, and noise distribution. For example, the noise distribution map 620 can correspond to... Figure 5 The model shown in Figure 620 can be applied to the noise distribution map 620 by changing the distribution values ​​based on the distance from the image center to, for example, each pixel. When the shooting parameters are further applied, a mapping relationship can be derived between the input data corresponding to pixel data, pixel location information, and shooting parameters, and the output data corresponding to the noise distribution. Compared to a model used to estimate noise that is learned or trained using only pixel data, a noise model (or trained noise model) that has been learned or trained (or trained) to have such a mapping relationship can take into account additional information (such as pixel location information and shooting parameters), thus estimating the actual noise more realistically.

[0079] Figure 7Examples of applying a Gaussian distribution through data transformation are shown according to one or more embodiments. It can be assumed that the data being modeled follows a Gaussian distribution when modeling noise. However, the actual data used to model the noise (e.g., captured data) may not follow a Gaussian distribution (e.g., during training or inference operations). For example, in this case, the actual data can be transformed to follow a Gaussian distribution and then used for training. (Refer to...) Figure 7 New pixel data conforming to a Gaussian distribution can be determined by performing data transformations 771 and 721 on the pixel data of images 710 and 720, for example, during training or inference operations. Data transformations 711 and 721 can also be referred to as Gaussian data transformations. The new pixel data of noisy image 710 can also be referred to as Gaussian noisy pixel data, and the new pixel data of clean image 720 can also be referred to as Gaussian clean pixel data.

[0080] The difference between images 710 and 720 can be calculated based on the new pixel data, and a noise component 730 can be derived based on said difference. As a result, the noise component 730 may follow a Gaussian distribution. For example, data transformation 711 may include, for instance,... Figure 7 The T(Y) = Y shown in the figure λ The power transform of λ. In the equation shown, T represents the power transform function, Y represents the actual data, and λ represents the transform value. Therefore, Figure 701 shown can represent the difference between the distribution of the actual data based on the value of λ and a Gaussian distribution (or a normal distribution). When the value of λ that minimizes the difference (e.g., 0.94) is determined, the power transform based on the value of λ can be applied to images 710 and 720.

[0081] Figure 8 Examples of suppressing the clipping effect by declipping, according to one or more embodiments, are shown. Figure 8 The points in Figure 800 represent example relationships between pixel values ​​and distribution values. Pixels of an image sensor and / or images may not fully reflect the brightness of real light as presented in nature. For example, a first range 801 may represent brightness that can be represented by pixels, and a second range 802 may represent real light brightness that cannot be represented by pixels. Based solely on pixel values ​​in the first range 801, a clipping effect, as shown in the first region 810, occurs due to the limitations of pixel value representation. That is, clipping can occur as one gets closer to the limit line of representation because no actual data is applied.

[0082] The clipping effect can be removed by performing de-clipping, allowing data from the second range 802 to be applied. For example, de-clipping based on the data distribution of the image (e.g., during training or inference operations) can remove the clipping effect caused by limitations in the representation of the image's pixel values. In another example, data from the third range 803 can be extracted and used for noise modeling. In this example, the remaining data in the first range 801, excluding the third range 803, can be estimated (e.g., interpolated or extrapolated) based on the data from the third range 803. Therefore, in addition to the third range 803 being used for noise modeling, the combination of the third range 803 and the estimated remaining data from the first range 801 can be used for noise modeling to obtain... Figure 8 The data in the second region 820 shown in the figure.

[0083] Figure 9 Examples of generating training images according to one or more embodiments are shown. (Refer to...) Figure 9 For example, degradation model 920 may include blur model 921 and noise model 922. Blur model 921 may add a blur effect to clean training image 910, and noise model 922 may add a noise effect to clean training image 910. When the blur effect and noise effect are applied to clean training image 910, a degraded training image 930 may be generated. For example, when blur model 921 generates a blurred image by adding a blur effect to clean training image 910, noise model 922 can generate a degraded training image 930 by adding a noise effect to the blurred image. However, the order in which the blur effect and noise effect are added is not limited to the foregoing example.

[0084] Degradation model 920 can generate a degraded training image 930 based on degradation conditions 923. Each degradation aspect can be set based on degradation conditions 923. For example, degradation conditions 923 may include blur conditions associated with blurring effects and noise conditions associated with noise effects. As an example, blur conditions may include motion information, and blur model 921 can generate a blurred image based on a clean training image 910 and motion information. Noise conditions may include pixel position information and shooting parameters, and noise model 922 can generate a degraded training image 930 based on pixel data, pixel position information, and shooting parameters of the blurred image.

[0085] As described in further detail below, a training dataset can be determined based on a clean training image 910 and a degraded training image 930, and an image enhancement model can be trained based on this training dataset. Various training datasets may be required to train the image enhancement model. By changing the degradation condition 923, various versions of the degraded training image 930 can be generated from the same clean training image 910. Furthermore, by changing the degradation condition 923 and the clean training image 910, even more different versions of the degraded training image 930 can be generated. For example, by adjusting the exposure time, long-exposure degraded training images and short-exposure degraded training images can be generated as the degraded training image 930. In this example, the realism of the degraded training image 930 can be enhanced by adding the blurring characteristics of long and short exposures to it.

[0086] Figure 10 Examples are shown of generating degraded training images by estimating the noise distribution from a noise model, according to one or more embodiments. (Refer to...) Figure 10 A noise model 1020 (e.g., a trained / learned noise model 1020) can estimate a noise distribution 1030 based on a clean training image 1010 and a degradation condition 1021. For example, the noise model 1020 can estimate the noise distribution 1030 based on pixel data, pixel location information, and shooting parameters of the clean training image 1010. When applying a degradation effect to the clean training image 1010, pixel data from the degraded image can be used instead of the pixel data from the clean training image 1010. A degraded training image 1040 can be derived based on the clean training image 1010 and the noise distribution 1030. For example, a noise map can be generated by concatenating (e.g., multiplying) a noise distribution map representing the noise distribution 1030 with a Gaussian random graph, and a degraded training image 1040 can be generated by applying (e.g., adding) the noise map to the clean training image 1010. The Gaussian random graph can be extracted from a Gaussian distribution with a mean of 0 and a standard deviation of 1. Clean training image 1010, noisy distribution 1030, and degraded training image 1040 can be included in the training dataset.

[0087] Figure 11 Examples of data transformations during the generation of degraded training images are shown according to one or more embodiments. As described above, it can be assumed that during noise modeling using the noise model 1140 (e.g., a trained / learned noise model 1140), the modeling data follows a Gaussian distribution. However, the actual data used for modeling may not follow a Gaussian distribution. In such cases, as a non-limiting example, the actual data can be transformed to follow a Gaussian distribution and then used to train the image enhancement model. The process of generating training images can also be applied to inference operations to generate degraded images. See also... Figure 11Blur model 1120 can generate blurred image 1130 based on clean training image 1110. By performing a data transformation on the pixel data of blurred image 1130, new pixel data following a Gaussian distribution can be determined. Then, noise model 1140 can estimate noise distribution 1150 based on the new pixel data, pixel location information, and shooting parameters. Noise modeling can be performed in Gaussian region 1101 through the data transformation. Noise distribution 1150 can be applied to Gaussian region 1101, thereby determining intermediate image 1160. Degraded training image 1170 can then be determined by performing an inverse transformation on intermediate image 1160. For example, a power transform function can be used for data transformation, and an inverse power transform function can be used for inverse data transformation.

[0088] Figure 12 and Figure 13 Examples of generating blurs according to two or more embodiments are shown. The blur generation steps can be applied to training or inference operations. In one example, the blur model can use... Figure 12 The Gaussian distribution shown in the image generates a blurred image. (Refer to...) Figure 12 The blurring model can shift the input image based on shift points, including shift point 1202. As a non-limiting example, the input image can be a clean training image. Shift points can be extracted from the distribution region 1200 based on a Gaussian distribution with a standard deviation σ (e.g., a bivariate Gaussian distribution). The blurring model can generate a blurred image corresponding to the target point 1201 by averaging the shifted image. The target point 1201 can correspond to the average position among the shift points.

[0089] In another example, the fuzzy model can be used Figure 13 The motion blur kernel shown generates a blurred image. (Refer to...) Figure 13 Pixel 1301 can represent a shift point in the input image. Pixel 1301 can be obtained through a motion blur kernel and / or a sensor (e.g., an accelerometer, gyroscope, etc.). The blur model can generate a shifted image corresponding to the shift point. The pixel value of each pixel 1301 can represent the weight of the corresponding shift point, and the blur model can generate a blurred image by calculating a weighted average of the shifted images based on the weights. Figure 12 Examples can represent discontinuous motions that can have high stiffness. Figure 13 The example represents continuous motion.

[0090] Figure 14 Examples of training image augmentation models according to one or more embodiments are shown. (Refer to...) Figure 14The image enhancement model 1410 can be trained based on a degraded training image 1401, a noise distribution 1402, and a clean training image 1403. The degraded training image 1401, noise distribution 1402, and clean training image 1403 can correspond to a training dataset. As described above, the degradation model can generate the degraded training image 1401 and noise distribution 1402 based on the clean training image 1403, and the degraded training image 1401, noise distribution 1402, and clean training image 1403 can constitute a training dataset. The image enhancement model 1410 can be trained to have a mapping relationship between "input data corresponding to the degraded training image 1401 and noise distribution 1402" and "output data corresponding to the clean training image 1403". Therefore, given an input image and its noise distribution, using the trained image enhancement model, the image enhancement model 1410 can have the ability to infer an enhanced image of the input image.

[0091] For example, image enhancement model 1410 can be a machine learning model. As a non-limiting example, image enhancement model 1410 may include, for example, a deep learning network. Image enhancement model 1410 can perform image enhancement; for example, image enhancement model 1410 can be trained based on supervised and / or unsupervised learning to have a mapping between input and output data with a non-linear relationship. Training input data for image enhancement model 1410 can be determined based on degraded training image 1401 and noise distribution 1402. For example, training input data can be determined by concatenating degraded training image 1401 and noise distribution 1402. Clean training image 1403 can be determined as training output data. Image enhancement model 1410 can be repeatedly trained to generate training output data based on training input data.

[0092] Figure 15 Examples are shown of image enhancement using a trained noise model and a trained image enhancement model, according to one or more embodiments. (Refer to...) Figure 15 The noise model 1510 can estimate the noise distribution 1505 of the input image 1501 based on the input image 1501. For example, the noise model 1510 can estimate the noise distribution 1505 based on, for example, pixel data 1502, pixel location information 1503, and shooting parameters 1504 of the input image 1501. The noise distribution 1505 may include the variance or standard deviation of the noise values ​​of the pixels of the input image 1501.

[0093] As a non-limiting example, pixel data 1502 may include pixel values ​​of the input image 1501, and pixel location information 1503 may include pixel coordinate values ​​and / or distances from the center. As pixel data 1502, clean pixel data of a clean version of the input image 1501 may be used. For example, a pseudo-clean image can be obtained by applying a blur filter to the input image 1501, and clean pixel data can be determined from the pixel values ​​of the pseudo-clean image. Capture parameters 1504 may include capture parameters of the input image 1501. For example, a noise model 1510 may estimate a noise distribution 1505 based on, for example, the distance from the center of the input image 1501 to each pixel.

[0094] Image enhancement model 1520 can generate enhanced image 1506 based on input image 1501 and noise distribution 1505. For example, input data can be generated by concatenating input image 1501 and noise distribution 1505, and then image enhancement model 1520 can generate enhanced image 1506 based on the input data. Image enhancement model 1520 can be a machine learning model pre-learned or trained to have a mapping relationship between "input data corresponding to input image 1501 and noise distribution 1505" and "output data corresponding to enhanced image 1506". Furthermore, image enhancement model 1520 can be pre-trained based on training images generated using noise model 1510.

[0095] Figure 16 Examples of devices with training according to one or more embodiments are shown. As a non-limiting example, see [reference needed]. Figure 16 The device 1600 may include a processor 1610 and a memory 1620. As a non-limiting example, the memory 1620 may be connected to the processor 1610 and may store instructions executable by the processor 1610, data to be computed by the processor 1610, and / or data processed by the processor 1610. For example, the memory 1620 may store a noise model 1621, a training dataset 1622, and an image enhancement model 1623. For example, the noise model 1621 and the image enhancement model 1623 may be machine learning models in training or already trained machine learning models. As a non-limiting example, the memory 1620 may include non-transitory computer-readable media (such as high-speed random access memory (RAM)) and / or non-volatile computer-readable media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0096] Processor 1610 executable configuration processor 1610 execution reference here Figures 1 to 20Any one, any combination, or all of the instructions described in the operation. For example, processor 1610 may be configured to train a noise model 1621 based on training modeling images and corresponding pixel location information (e.g., during the training of noise model 1621 or by using a trained noise model 1621), generate degraded training images of clean training images using noise model 1621, determine a training dataset 1622 based on the clean training images and the degraded training images, and train an image enhancement model 1623 based on the training dataset 1622.

[0097] Figure 17 Examples of devices with image enhancement according to one or more embodiments are shown. As a non-limiting example, reference is made to... Figure 17 The device 1700 may include a processor 1710 and a memory 1720. The memory 1720 may be connected to the processor 1710 and may store instructions executable by the processor 1710, data to be computed by the processor 1710, and / or data processed by the processor 1710. For example, the memory 1720 may store a noise model 1721 and an image enhancement model 1723, for example, as corresponding models under training or already trained models. The processor 1710 is configurable and can execute actions described herein. Figures 1 to 20 Any one, any combination, or all of the instructions described in the operation. For example, processor 1710 may receive an input image, estimate the noise distribution of the input image by executing a trained noise model 1721 based on the input image, and generate an enhanced image by executing a trained or training image enhancement model 1723 based on the input image and the estimated noise distribution.

[0098] Figure 18 Examples of image enhancement methods according to one or more embodiments are shown. (Refer to...) Figure 18 In operation 1810, the device may receive an input image. In operation 1820, the device may estimate the noise distribution of the input image by performing a noise model based on the input image. For example, the device may perform the noise model (e.g., a trained noise model) based on pixel data, pixel location information, and shooting parameters of the input image. The noise model may be pre-learned or trained to have a mapping between input data corresponding to pixel data, pixel location, and shooting parameters and output data corresponding to the noise distribution. As a non-limiting example, the shooting parameters may include at least one of ISO value, exposure time, and Bayer pattern information. As a non-limiting example, the noise model may estimate the noise distribution of the input image based on distance from the center. For example, the noise distribution may include the variance or standard deviation of the noise values ​​of the pixels of the input image.

[0099] In operation 1830, the device can generate an enhanced image by executing an image enhancement model (e.g., a trained image enhancement model) based on the input image and an estimated noise distribution. For example, the device can generate input data by concatenating the input image and the estimated noise distribution, and then execute the image enhancement model using the generated input data. The image enhancement model can be a machine learning model that has been pre-learned or trained to have a mapping between the input data corresponding to the input image and the estimated noise distribution, and the output data corresponding to the enhanced image. For example, the image enhancement model can be pre-trained based on training images generated using a noise model.

[0100] Figure 19 Examples of training methods according to one or more embodiments are shown. (Refer to...) Figure 19 In operation 1910, the device can train a noise model based on training images and corresponding pixel location information. For example, the device can generate a first clean training image from a first noisy training image, determine the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image, and train the noise model to have a mapping relationship based on the distribution of noise components, pixel data of the first clean training image, and pixel location information.

[0101] The device can determine Gaussian noise pixel data and Gaussian clean pixel data by applying a Gaussian data transform to first noisy pixel data of a first noisy training image and first clean pixel data of a first clean training image, and determine noise components based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data. As a non-limiting example, the device can also derive a mapping relationship based on the first noisy training image / the shooting parameters used for the first noisy training image (including at least one of ISO value, exposure time, and Bayer pattern information). The device can remove the clipping effect caused by limitations in the representation of pixel values ​​in the training image by de-clipping.

[0102] In operation 1920, the device can generate a degraded training image of a clean training image using a noise model (e.g., a trained or training noise model). For example, the device can determine a noise distribution corresponding to the degraded conditions by providing a clean training image and degradation conditions to the noise model, and generate a degraded training image by applying the noise distribution to the clean training image. The device can use a blurring model to apply a blurring effect to the clean training image.

[0103] In operation 1930, the device can determine the training dataset based on clean and degraded training images. The device can generate training input by concatenating a noise distribution with the degraded training images. In operation 1940, the device can train an image augmentation model based on the training dataset.

[0104] Figure 20 Examples of electronic devices according to one or more embodiments are shown. (Refer to...) Figure 20 Electronic device 2000 may include processor 2010, memory 2020, camera 2030, storage device 2040, input device 2050, output device 2060, and network interface 2070, and these components may communicate with each other via communication bus 2080. As a non-limiting example, electronic device 2000 may be, or can be implemented as, at least a portion of, mobile devices (such as mobile phones, smartphones, personal digital assistants (PDAs), netbooks, tablet computers, laptop computers, etc.), wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), computing devices (such as desktop computers, servers, etc.), home appliances (such as televisions (TVs), smart TVs, refrigerators, etc.), security devices (such as door locks, etc.), and vehicles (such as autonomous vehicles, intelligent vehicles, etc.). Electronic device 2000 may be... Figure 1 Electronic device 1, or which includes in its structure and / or operation Figure 1 Image enhancement equipment 100 Figure 16 Equipment 1600 Figure 17 Equipment 1700, and / or this information regarding Figures 1 to 20 Any or all of the components, devices, and hardware discussed.

[0105] Processor 2010 (e.g., one or more processors) can perform various operations in electronic device 2000. For example, processor 2010 can process or execute instructions stored in memory 2020 or storage device 2040, which, when executed by processor 2010, can configure or cause processor 2010 to perform any, any combination, or all of the operations or methods described herein. Memory 2020 may include computer-readable storage media or devices. Memory 2020 may, for example, store instructions executable by processor 2010, related information (such as any or all training and / or trained models described herein), and instructions executable by processor 2010 based on such related information, such as to control any or all remaining components of electronic device 2000, while other software and / or applications of electronic device 2000 are being executed by electronic device 2000.

[0106] Camera 2030 (e.g., one or more cameras) can capture images and / or videos. As a non-limiting example, such images and / or videos may correspond to captured images, training images, and / or input images. Images and / or videos may be temporarily or non-temporarily stored in memory 2020 and / or storage device 2040.

[0107] Storage device 2040 may include a computer-readable storage medium or device. Storage device 2040 may store more or more information than memory 2020, and may store the information for a longer or longer period of time. Storage device 2040 may include, for example, a magnetic hard disk, optical disk, flash memory, floppy disk, or other types of non-volatile memory.

[0108] As a non-limiting example, input device 2050 can receive input from a user using input methods such as keyboard, mouse, touch input, voice input, and / or image input. As a non-limiting example, input device 2050 may include a keyboard, mouse, touchscreen, microphone, and / or any other device that detects input from the user. Input device 2050 may be a component of electronic device 2000 and / or a component connected to or used by electronic device 2000 to transmit or otherwise provide detected input to electronic device 2000. As a non-limiting example, output device 2060 can provide output from electronic device 2000 to a user through visual, auditory, and / or tactile channels. Output device 2060 may include, for example, a display, touchscreen, speaker, vibration generating device, and / or any other device that provides output to the user. Output device 2060 may be a component of electronic device 2000 and / or a component connected to or used by electronic device 2000 to transmit or otherwise provide detected input to electronic device 2000. For example, network interface 2070 can communicate with external devices via wired and wireless networks. As an example of a non-limiting, more detailed description of electronic device 1 and / or electronic device 2000, the above refers to Figures 1 to 19 The content described can also be applied to Figure 20 For example, an electronic device may include any one, any combination, or all of the devices described above, and may perform any one, any combination, or all of the operations or methods described above.

[0109] Regarding Figures 1 to 20The described image enhancement device, device with image enhancement, device with training, electronic device, memory or memory device, storage device, processor, network interface, camera, input device, output device, communication bus, and other devices, apparatus, units, modules, and components are implemented or represent hardware components by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components of the hardware components describing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For brevity, the singular terms "processor" or "computer" are used in the description of the examples described herein; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0110] Figures 1 to 20The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0111] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0112] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card-type storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other means configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0113] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0114] Therefore, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be understood to be included in the disclosure.

Claims

1. A processor-implemented image enhancement method, comprising: Obtain the input image; The noise distribution of the input image is estimated by performing a noise model based on the input image. as well as Enhanced images are generated by performing an image enhancement model based on the input image and the estimated noise distribution. The noise model is trained through the following steps: generating a first clean training image from a first noisy training image; determining the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image; and training the noise model to establish a mapping relationship between the distribution of the noise component and the pixel data and pixel location information of the first clean training image. The step of determining the noise component includes: determining Gaussian noise pixel data and Gaussian clean pixel data by applying Gaussian data transformation to the first noise pixel data of the first noisy training image and the first clean pixel data of the first clean training image; and determining the noise component based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

2. The image enhancement method according to claim 1, wherein, The steps to obtain the input image include: Use a camera to capture the input image.

3. The image enhancement method according to claim 1, wherein, The steps for estimating the noise distribution include: The noise model is executed based on at least one of the pixel data of the input image, the pixel position information of the input image, and the shooting parameters of the input image.

4. The image enhancement method according to claim 3, wherein, The shooting parameters include at least one of the following: the International Organization for Standardization (ISO) value associated with the input image, the exposure time, and the Bayer pattern information.

5. The image enhancement method according to claim 3, in, A noise model is a regression model that has been pre-trained before the input image is obtained. The noise model provides a mapping relationship between input data corresponding to at least one of pixel data, pixel location information, and shooting parameters, and output data corresponding to the noise distribution.

6. The image enhancement method according to claim 3 further includes: A noise model is generated by training a noise model in training to have a mapping relationship between input data corresponding to at least one of pixel data, pixel position information, and shooting parameters and output data corresponding to the noise distribution.

7. The image enhancement method according to claim 6 further includes: Image enhancement models are generated by training an image enhancement model in training to have a mapping relationship between the output enhanced image and the input image and noise distribution.

8. The image enhancement method according to claim 1, wherein, The noise model is the noise model in training, and / or the image augmentation model is the image augmentation model in training.

9. The image enhancement method according to claim 1, wherein, The noise model is configured to estimate the noise distribution of the input image based on the respective distance from the center of the input image to each of the multiple pixels in the input image.

10. The image enhancement method according to claim 1, wherein, Noise distribution includes the variance or standard deviation of noise values ​​of multiple pixels in the input image.

11. The image enhancement method according to claim 1, wherein, The steps to generate an enhanced image include: Input data is generated by concatenating the input image and the estimated noise distribution; and The generated input data is used to execute an image enhancement model.

12. The image enhancement method according to claim 1, in, Image augmentation models are machine learning models that have been pre-trained before the input image is obtained, and... The trained image enhancement model provides a mapping between the input data corresponding to the input image and the corresponding estimated noise distribution, and the output data corresponding to the enhanced image.

13. The image enhancement method according to claim 1, further comprising: The image enhancement model is trained based on training images generated using a noise model.

14. A processor-implemented method, comprising: A noise model is trained based on the training images and corresponding pixel location information; Execute the trained noisy model to generate degraded training images that are provided to the trained noisy model for clean training images. The training dataset is determined based on clean and degraded training images. as well as The image enhancement model is trained based on the training dataset. The steps of training the noise model include: generating a first clean training image from a first noisy training image; determining the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image; and training the noise model to establish a mapping relationship between the distribution of the noise component and the pixel data and pixel position information of the first clean training image. The step of determining the noise component includes: determining Gaussian noise pixel data and Gaussian clean pixel data by applying Gaussian data transformation to the first noise pixel data of the first noisy training image and the first clean pixel data of the first clean training image; and determining the noise component based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

15. The method according to claim 14, wherein, The method further includes: Capture the input image; The noise distribution of the input image is estimated by executing a trained noise model based on the input image. A degraded image of the input image is generated based on the input image and its noise distribution; and Enhanced images are generated by executing a trained image enhancement model based on the input image and a degraded image of the input image.

16. The method of claim 14, wherein, The steps for training a noise model include: The noise model is trained based on the shooting parameters of the first noise training image, including at least one of the International Organization for Standardization ISO value, exposure time, and Bayer pattern information.

17. The method according to claim 14, wherein, The steps for training a noisy model include removing the clipping effect of pixel values ​​in the training images by de-clipping.

18. The method according to claim 14, wherein, The steps to generate degraded training images include: The noise distribution corresponding to the degradation condition is determined by providing the noise model with clean training images and degradation conditions; and Degraded training images are generated by applying noise distribution to clean training images.

19. The method according to claim 18, wherein, The steps to determine the training dataset include: Training input is generated by connecting a noise distribution and a degraded training image.

20. The method of claim 14, wherein, The steps to generate degraded training images include: Use a blur model to apply a blur effect to clean training images.

21. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 20.

22. An image enhancement device, comprising: The processor is configured as follows: The noise distribution of the input image is estimated by performing a noise model based on the obtained input image; as well as, An enhanced image of the input image is generated by performing an image enhancement model based on the input image and the estimated noise distribution. The processor is further configured to: generate a first clean training image from a first noisy training image; determine the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image; and train the noise model to have a mapping relationship between the distribution of the noise component and the pixel data and pixel position information of the first clean training image, thereby training the noise model. The process of determining the noise component includes: applying Gaussian data transformation to the first noisy pixel data of the first noisy training image and the first clean pixel data of the first clean training image to determine Gaussian noise pixel data and Gaussian clean pixel data, and determining the noise component based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

23. The image enhancement apparatus according to claim 22, wherein, The processor is configured as follows: A noise model is executed based on at least one of the pixel data, pixel location information, and shooting parameters of the input image.

24. The image enhancement apparatus according to claim 22, wherein, The noise model is a trained or in-training noise model, and / or the image augmentation model is a trained or in-training image augmentation model.

25. The image enhancement apparatus according to claim 22, further comprising: Camera, to acquire input image; as well as The memory stores instructions that, when executed by the processor, configure the processor to perform the steps of estimating noise distribution and generating an enhanced image. The device in question is a mobile phone electronic device.

26. An electronic device comprising: The processor is configured as follows: Multiple noise distributions of the image are generated by performing noise models based on multiple shooting parameters and pixel information of the image, respectively. Multiple degraded images of the image are generated using the generated multiple noise distributions; as well as An image enhancement model is trained based on the aforementioned degraded images. The processor is further configured to: generate a first clean training image from a first noisy training image; determine the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image; and train the noise model to have a mapping relationship between the distribution of the noise component and the pixel data and pixel position information of the first clean training image, thereby training the noise model. The process of determining the noise component includes: applying Gaussian data transformation to the first noisy pixel data of the first noisy training image and the first clean pixel data of the first clean training image to determine Gaussian noise pixel data and Gaussian clean pixel data, and determining the noise component based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

27. The electronic device according to claim 26, wherein, The noise model is a trained noise model, and the electronic device is a mobile phone.

28. An electronic device comprising: Camera, captures the input image; as well as The processor is configured as follows: The noise distribution of the input image is estimated by executing a noise model that takes into account the pixel information of the input image and the shooting parameters when the input image was captured and / or multiple corresponding pixel position information of the input image. as well as An enhanced image of the input image is generated by executing an image enhancement model, wherein the image enhancement model is executed based on the input image and the estimated noise distribution. The processor is further configured to: generate a first clean training image from a first noisy training image; determine the noise component of the first noisy training image based on the difference between the first noisy training image and the first clean training image; and train the noise model to have a mapping relationship between the distribution of the noise component and the pixel data and pixel position information of the first clean training image, thereby training the noise model. The process of determining the noise component includes: applying Gaussian data transformation to the first noisy pixel data of the first noisy training image and the first clean pixel data of the first clean training image to determine Gaussian noise pixel data and Gaussian clean pixel data, and determining the noise component based on the difference between the Gaussian noise pixel data and the Gaussian clean pixel data.

29. The electronic device according to claim 28, wherein, The electronic device is a mobile phone.

Images