Method of correcting image using neural network model and computing device executing neural network model for image correction

Abstract

A method for correcting an image by using a neural network model according to an embodiment of the present disclosure comprises the steps of: inputting an input image and a correction parameter to a first neural network model; and obtaining an inferred image corresponding to the correction parameter from the first neural network model. The first neural network model is a model trained to minimize a difference between an inferred image output when the input image and the correction parameter are input and a tag image corresponding to the correction parameter. The label image is an image obtained by correcting the input image by the at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm.

Description

Technical Field

[0001] The present disclosure relates to a method of correcting an image by using a neural network model and a computing device for executing the neural network model for image correction. Background Art

[0002] With the recent development of neural network technology, neural networks have been used in various fields. Specifically, by using neural networks in the field of image signal processing to perform image correction or reconstruction, processing speed and image characteristics are improved. Summary of the invention

[0003] Solution to the problem

[0004] According to an embodiment of the present disclosure, a method for correcting an image by using a neural network model includes: inputting an input image and correction parameters into a first neural network model by a processor of a computing device, and obtaining an inference image corresponding to the correction parameters from the first neural network model by the processor, wherein the first neural network model is a model trained to minimize the difference between an inference image output when the input image and the correction parameters are input and a label image corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is an image obtained in the following manner: when the correction parameters are applied to at least one image correction algorithm, the input image is corrected by using at least one image correction algorithm, wherein the first neural network model is trained on the relationship between the correction parameters and the label image, so that the inference image output from the first neural network model can be controlled by changing the correction parameters.

[0005] According to an embodiment of the present disclosure, a computing device includes: an input / output interface, through which input is received from a user; a memory, in which a program for correcting an input image is stored; and a processor, the processor being configured to execute the program to input an input image and correction parameters into a first neural network model, and to obtain an inferred image corresponding to the correction parameters from the first neural network model, wherein the first neural network model is a model trained to minimize the difference between an inferred image output when the input image and the correction parameters are input and a label image corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is an image obtained by correcting the input image by using at least one image correction algorithm when the correction parameters are applied to at least one image correction algorithm, wherein the first neural network model is trained on the relationship between the correction parameters and the label image so that the inferred image output from the first neural network model can be controlled by changing the correction parameters.

[0006] According to an embodiment of the present disclosure, a computer-readable recording medium stores therein a program for executing at least one of the embodiments of the method on a computer.

[0007] According to an embodiment of the present disclosure, a computer program is stored in a medium so as to execute at least one of the embodiments of the method on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram for describing the operation of a first neural network model according to an embodiment of the present disclosure.

[0009] Figure 2 is a diagram for describing the operation of the second neural network model according to an embodiment of the present disclosure.

[0010] Figure 3 is a diagram for describing a process of training a first neural network model according to an embodiment of the present disclosure.

[0011] Figure 4 is a diagram for describing a process of training a second neural network model according to an embodiment of the present disclosure.

[0012] Figure 5 is a diagram for describing a process in which a label image generation module generates a label image from an input image according to an embodiment of the present disclosure.

[0013] Figure 6 is a diagram for describing intensity values ​​applied to a contrast enhancement algorithm according to an embodiment of the present disclosure.

[0014] Figure 7 is a diagram for describing a gamma value applied to a brightness correction algorithm according to an embodiment of the present disclosure.

[0015] Figure 8 is a diagram for describing an alpha value applied to an exposure fusion algorithm according to an embodiment of the present disclosure.

[0016] Fig. 9 is a diagram for describing a method of applying a correction parameter as input to a first neural network model according to an embodiment of the present disclosure.

[0017] Fig.10 is a diagram schematically illustrating elements included in a mobile terminal according to an embodiment of the present disclosure.

[0018] Fig.11 is a diagram schematically illustrating a detailed configuration of an image signal processor (ISP) according to an embodiment of the present disclosure.

[0019] Fig.12is a diagram showing an embodiment of the present disclosure wherein the first neural network model and the second neural network model are included in Fig.10 FIG. 4 is a diagram of a specific embodiment of an image correction module.

[0020] Figures 13 to 15 is a flowchart for describing a method for correcting an image by using a neural network model according to an embodiment of the present disclosure.

[0021] Fig.16 is a diagram for describing changes in an inferred image when a correction parameter input to a first neural network model changes according to an embodiment of the present disclosure.

[0022] Fig.17 is a diagram for describing an embodiment in which a user adjusts a correction parameter for brightness adjustment through a user interface (UI) displayed on a screen of a mobile terminal according to an embodiment of the present disclosure.

[0023] Fig.18 is a block diagram showing a configuration of a computing device for performing training on a first neural network model and a second neural network model according to an embodiment of the present disclosure.

[0024] Fig.19 is a diagram for describing an embodiment in which a cloud server corrects an image received from a mobile terminal by using a first neural network model and a second neural network model according to an embodiment of the present disclosure.

[0025] Fig. 20 and Fig.21 is a flowchart for describing a method for training a first neural network model and a second neural network model according to an embodiment of the present disclosure.

[0026] Fig. 22 and Fig.23 is a diagram for describing a method for updating a neural network model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0028] In this detailed description, descriptions of technologies that are well known in the art and not directly related to the disclosed embodiments are omitted. This is to clearly convey the main points of the disclosed embodiments by omitting unnecessary descriptions. The terms used herein are defined in consideration of the functions in the disclosed embodiments and may vary according to the intention of the user or operator, precedents, etc. Therefore, the terms used herein should be defined based on the meaning of the terms and the description of the entire specification.

[0029] For the same reason, some elements in the accompanying drawings are enlarged, omitted or schematically shown. In addition, the size of each element may not substantially reflect its actual size. In each of the accompanying drawings, the same or corresponding elements are represented by the same reference numerals.

[0030] The advantages and features of the disclosed embodiments and their implementation methods will become apparent with reference to the embodiments described in detail below in conjunction with the accompanying drawings. However, the disclosed embodiments may be embodied in many different forms and should not be construed as being limited to any particular disclosed embodiment set forth herein. The embodiments are provided so that the detailed description will be thorough and complete, and will fully convey the scope of the disclosed embodiments to those of ordinary skill in the art. The embodiments may be defined according to the claims. In the specification, the same reference numerals represent the same elements. Detailed descriptions of related well-known functions or configurations that may obscure the main points of the disclosed embodiments are omitted. The terms used herein are defined in view of the functions and may vary according to the intentions of the user or operator, precedents, etc. Therefore, the terms used herein should be defined based on the meaning of the terms and the description of the entire specification.

[0031] It should be understood that each box of the flowchart diagram and the combination of boxes in the flowchart diagram can be implemented by computer program instructions. These computer program instructions can be loaded into a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and the instructions executed by the processor of the computer or other programmable data processing device can generate a device for implementing the function specified in the flowchart box. These computer program instructions can also be stored in a computer-available or computer-readable memory, which can guide the computer or other programmable data processing device to work in a specific manner, and the instructions stored in the computer-available or computer-readable memory can produce a product including an instruction device that implements the function specified in the flowchart box. The computer program instructions can also be loaded onto a computer or other programmable data processing device.

[0032] In addition, each frame of the flowchart diagram can represent a module, a fragment or a portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that the functions mentioned in the frame may not occur in the order shown. For example, two frames shown in succession can actually be executed substantially simultaneously, or these frames can sometimes be executed in reverse order, depending on the functions involved.

[0033] The term "... unit" used in the present embodiment refers to a software or hardware component that performs certain tasks, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, "... unit" is not meant to be limited to software or hardware. "... unit" can be configured in an addressable storage medium, or can be configured to operate one or more processors. In the disclosed embodiments, "... unit" may include components (such as software components, object-oriented software components, class components, and task components), processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcodes, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided in components and "... units" can be combined into fewer components and "... units", or further divided into additional components and "... units". In addition, "... units" may include one or more processors.

[0034] The embodiment relates to a method for correcting an image by using a neural network model. Before describing the specific embodiment, the meanings of terms frequently used in this specification are defined.

[0035] An "input image" refers to an image input to a neural network model, and an "inferred image" refers to an image output after the neural network model corrects the input image. The input image may be a linear RGB image generated from a plurality of original images. An 'original image' may be an image output from an image sensor (e.g., a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor), and may refer to a Bayer format image having only one color channel per pixel. In addition, for a plurality of original images taken before and after a specific point in time, a linear RGB image may be obtained by using the temporal information of the original images and performing demosaicing and denoising.

[0036] “Image characteristics” may refer to the brightness, contrast, color temperature, etc. of an image.

[0037] “Image correction” may refer to the operation of adjusting the characteristics of an image, and an “image correction algorithm” may refer to an algorithm used to adjust the characteristics of an image.

[0038] "Correction parameters" may refer to parameters applied to an image correction algorithm when correcting an image by using the image correction algorithm. That is, the degree to which the characteristics of the image are adjusted may be determined according to the value of the correction parameters. Specific examples of the correction parameters will be described below.

[0039] The "label image" may be an image used as training data for performing supervised learning on a neural network model, and specifically refers to an image used as ground truth data. The "label image generation module" may be an element that generates a label image by correcting an input image using at least one image correction algorithm.

[0040] The neural network according to an embodiment of the present disclosure may include a "first neural network model" that corrects an input image and outputs an inferred image and a "second neural network model" that infers the optimal correction parameters for a given input image. Fig.10 and Fig.11 A detailed description is given of how the first neural network and the second neural network can be used in a practical device.

[0041] 1. Description of the basic operations of the first neural network model and the second neural network model

[0042] First, refer to Figure 1 and Figure 2 The functions of the first neural network model 100 and the second neural network model 200 will be briefly described, and then reference will be made to Figure 3 and Figure 4 Describe how to train the first neural network model 100 and the second neural network model 200 to perform functions.

[0043] Figure 1 is a diagram for describing the operation of the first neural network model 100 according to an embodiment of the present disclosure. Figure 1 , the first neural network model 100 may receive an input image 1 and a correction parameter 10, and may output an inferred image 2 corresponding to the correction parameter 10. In this case, the inferred image 2 corresponds to an image obtained by correcting the input image 1 based on the correction parameter 10. Figure 3 How to train the first neural network model 100 to output the inferred image 2 is described in detail.

[0044] Generally, the input image 1 may have a problem that all or part of the image is dark or has low contrast, so the object in the image cannot be well recognized. The first neural network model 100 can increase the brightness of the input image 1 or enhance the contrast of the input image 1 to solve the problem.

[0045] The first neural network model 100 can output Figure 1The inferred image 2 shown in the figure, but the first neural network model 100 can output a filter or mapping information for converting the input image 1 into the inferred image 2, instead of outputting the image itself. That is, the input image 1 can be converted into the inferred image 2 by using the filter or mapping information output by the first neural network model 100.

[0046] The first neural network model 100 may include any of various types of deep learning networks, and may be implemented as, for example but not limited to, a residual network (ResNet), which is a type of convolutional neural network (CNN).

[0047] Figure 2 is a diagram for describing the operation of the second neural network model 200 according to an embodiment of the present disclosure. Figure 2 , the second neural network model 200 can receive the input image 1 and can infer and output the correction parameter 10. The correction parameter 10 output from the second neural network model 200 can be input to the first neural network model 100, and the first neural network model 100 can output the inferred image 2 corresponding to the correction parameter 10, as described above with reference to Figure 1 described.

[0048] The second neural network model 200 may infer correction parameters 10 for correcting the input image 1 to have image characteristics that the user may like. In this case, how to correct the input image 1 by the correction parameters 10 inferred by the second neural network model 200 may be determined in the process of training the second neural network model 200, which will be referred to below. Figure 4 Detailed description.

[0049] The second neural network model 200 may include any of various types of deep learning networks, and may be implemented as, for example but not limited to, ResNet as a type of CNN.

[0050] 2. (Training process) Method for training the first neural network model and the second neural network model

[0051] Reference Figure 3 and Figure 4 A method of training the first neural network model 100 and the second neural network model 200 is described.

[0052] According to an embodiment of the present disclosure, the first neural network model 100 and the second neural network model 200 may be generated by an external device (e.g., Fig.18The training is performed by a computing device 1800) rather than by a device on which the first neural network model 100 and the second neural network model 200 are installed (e.g., Fig.10 However, a device on which the first neural network model 100 and the second neural network model 200 are installed can train the first neural network model 100 and the second neural network model 200. For ease of explanation, it is assumed that Fig.18 The processor 1830 of the computing device 1800 executes the program stored in the memory 1820 to refer to Figure 3 and Figure 4 The first neural network model 100 and the second neural network model 200 are trained as described. That is, Figure 3 and Figure 4 In the example, the operations performed by the label image generation module 300 or the optimizer 400 and the calculation of the loss function can actually be represented by Fig.18 Executed by processor 1830 of computing device 1800.

[0053] Fig.18 The processor 1830 of the computing device 1800 may include a central processing unit (CPU) 1831, a graphics processing unit (GPU) 1832, and a neural processing unit (NPU) 1833, and may execute a program stored in the memory 1820 to train the first neural network model 100 and the second neural network model 200 by using at least one of the CPU 1831, the GPU 1832, or the NPU 1833. The input / output interface 1810 of the computing device 1800 may receive a command related to the training of the first neural network model 100 and the second neural network model 200, or may display information.

[0054] Figure 3 is a diagram for describing a process of training the first neural network model 100 according to an embodiment of the present disclosure. Figure 3 , the label image generation module 300 can output a label image 3 obtained by correcting the input image 1 using at least one image correction algorithm. When the label image generation module 300 corrects the input image 1, the correction parameter 10 is applied to the image correction algorithm. Figures 5 to 8 The image correction algorithm and correction parameters 10 used by the label image generation module 300 are described in detail.

[0055] Figure 5 3 is a diagram for describing a process in which the label image generation module 300 generates the label image 3 from the input image 1 according to an embodiment of the present disclosure. Figure 5 , the label image generation module 300 can use the contrast enhancement algorithm 310, the brightness correction algorithm 320 and the exposure fusion algorithm 330. However, Figure 5 The image correction algorithm used by the label image generation module 300 is merely an example, and the label image generation module 300 may use various other types of image correction algorithms.

[0056] Figure 5 It is shown which parameters are specifically included in the correction parameter 10 and which image correction algorithm corresponds to each parameter. Figure 5 , the intensity value 11 can be applied to the contrast enhancement algorithm 310, the gamma value 12 can be applied to the brightness correction algorithm 320, and the alpha value 13 can be applied to the exposure fusion algorithm 330. Figures 6 to 8 Describe the parameters (eg, 11, 12, and 13) that are applied to the image correction algorithm, respectively.

[0057] Figure 6 is a diagram for describing an intensity value 11 applied to a contrast enhancement algorithm 310 according to an embodiment of the present disclosure. The contrast enhancement algorithm 310 may be a method of adjusting contrast by using a curve.

[0058] refer to Figure 6 From the graph 600 of FIG. 600 , it is found that the curvature of the S-curve varies according to the intensity value I1, and specifically, the curvature of the S-curve increases as the intensity value I1 increases. When the slope of the S-curve increases, the tone is stretched, and thus the contrast increases. Therefore, as the intensity value I1 increases, the contrast enhancement algorithm 310 can increase the contrast of the input image 1.

[0059] Figure 7 3 is a diagram for describing a gamma value 12 applied to the brightness correction algorithm 320 according to an embodiment of the present disclosure. Figure 7 In the graph 700 of FIG. 700 , a gamma curve according to a gamma value of 12 is shown. The x-axis of the graph 700 represents the contrast value, and the y-axis represents the brightness. Figure 7 From the graph 700, it is found that as the gamma value 12 decreases, the brightness increases. Therefore, as the gamma value 12 decreases, the brightness correction algorithm 320 can increase the brightness of the input image 1.

[0060] Figure 8 is a diagram for describing an alpha value 13 applied to the exposure fusion algorithm 330 according to an embodiment of the present disclosure. The exposure fusion algorithm 330 may generate a processed image X' having an exposure different from that of the base image X, and may output a fused image F by multiplying the base image X and the processed image X' by a weight W and a weight (1-W), respectively, and adding the multiplication results. The base image X may correspond to Figure 5The input image 1, and the output fused image F can correspond to Figure 5 Label image 3.

[0061] The exposure fusion algorithm 330 may generate a processed image X′ from a base image X according to Equation 1.

[0062] [Equation 1]

[0063] X'=αX β

[0064] When the exposure fusion algorithm 330 is executed by using the above method, as the α value 13 increases, the fused image F may become brighter than the basic image X, and as the β value decreases, the fused image F may become brighter than the basic image X. Figure 5 The α value of 13 is used as the correction parameter of the exposure fusion algorithm, but the β value may be used instead of the α value as the correction parameter, or both the α value and the β value may be used as the correction parameter.

[0065] The label image generation module 300 may output a label image 3 obtained by applying correction parameters (e.g., 11, 12, and 13) to an image correction algorithm (e.g., 310, 320, and 330) to adjust at least one characteristic of the input image 1. The label image generation module 300 may use Figure 5 A combination of at least one of the image correction algorithms shown in (e.g., 310, 320, and 330), or a combination of at least one of the image correction algorithms shown in Figure 5 Another type of image correction algorithm not shown in FIG.

[0066] The order of parameter values ​​(e.g., 11, 12, and 13) included in the correction parameters 10 can remain the same during training and inference. In other words, the order of parameter values ​​(e.g., 11, 12, and 13) in the correction parameters 10 used during training of the neural network models 100 and 200 can remain the same during use of the neural network models 100 and 200.

[0067] For example, when the correction parameters 10 is a column vector, the elements of each row may correspond to the same type of parameters (parameters applied to the same image correction algorithm) during training and inference.

[0068] A specific example will be described as follows.

[0069] As in Figure 5In an embodiment, when the elements of the first row of the correction parameters (column vector) 10 used during training are parameters applied to the contrast enhancement algorithm 310 (e.g., 11), the elements of the second row are parameters applied to the brightness correction algorithm 320 (e.g., 12), and the elements of the third row are parameters applied to the exposure fusion algorithm 330 (e.g., 13), the elements of the first row of the correction parameters used during inference may be parameters applied to the contrast enhancement algorithm 310 (e.g., 11), the elements of the second row may be parameters applied to the brightness correction algorithm 320 (e.g., 12), and the elements of the third row may be parameters applied to the exposure fusion algorithm 330 (e.g., 13).

[0070] This will refer to Fig. 9 Give a description.

[0071] refer to Fig. 9 , two parameter values ​​(e.g., 11 and 12) are concatenated with the input image 1 and the grayscale image converted from the input image 1 (see Fig.12 1201), and then input to the first neural network model 100. That is, the grayscale image multiplied (or divided, or added) by the parameter values ​​(e.g., 11 and 12) is attached to the input image 1 in the channel direction and is input to the first neural network model 100. In this case, because the channel order should be the same as during training and inference, the order of the parameter values ​​(e.g., 11, 12, and 13) included in the correction parameter 10 should remain the same during training and inference as described above.

[0072] In addition, the number of bits of each parameter value (e.g., 11, 12, and 13) included in the correction parameter 10 can remain the same during training and inference. An embodiment in which the order of parameter values ​​(e.g., 11, 12, and 13) included in the correction parameter 10 and the number of bits of each parameter value (e.g., 11, 12, and 13) remain the same during training and inference will be described in detail below.

[0073] Assume that the correction parameter 10 used during the training of the first neural network model 100 and the second neural network model 200 is binary data "01010110". Assume that the first 3 bits of data "010" are parameters to be applied to the contrast enhancement algorithm 310 (e.g., 11), the next 2 bits of data "10" are parameters to be applied to the brightness correction algorithm 320 (e.g., 12), and the last 3 bits of data "110" are parameters to be applied to the exposure fusion algorithm 330 (e.g., 13).

[0074] The number of bits of each parameter (e.g., 11, 12, and 13) and the order of the parameters (e.g., 11, 12, and 13) should remain the same even during the use (inference) of the first neural network model 100 and the second neural network model 200. For example, when binary data "10111100" is input as correction parameter 10 to the first neural network model 100 during inference, the first 3 bits of data "101" are parameters (e.g., 11) to be applied to the contrast enhancement algorithm 310, the next 2 bits of data "11" are parameters (e.g., 12) to be applied to the brightness correction algorithm 320, and the last 3 bits of data "100" are parameters (e.g., 13) to be applied to the exposure fusion algorithm 330.

[0075] Return to reference Figure 3 , the label image 3 output from the label image generation module 300 can be used as training data (specifically, benchmark truth data) for training the first neural network model 100. The first neural network model 100 can be trained so that the inferred image 2 output from the first neural network model 100 when the input image 1 and the correction parameter 10 are input to the first neural network model 100 is as similar to the label image 3 as possible. That is, the first neural network model 100 can be a model trained to minimize the difference between the inferred image 2 output when the input image 1 and the correction parameter 10 are input and the label image 3 corresponding to the correction parameter 10. In this case, the label image 3 corresponding to the correction parameter 10 can be an image obtained by correcting the input image 1 by using at least one image correction algorithm when the correction parameter 10 is applied to the at least one image correction algorithm.

[0076] In other words, the optimizer 400 may update the first neural network model 100 to minimize a loss value of the loss function 20, the loss value of the loss function 20 indicating the difference between the inferred image 2 and the label image 3. In this case, the loss function 20 may include a combination of mean absolute error (MAE), mean square error (MSE), and structural similarity index measure (SSIM).

[0077] As described above, during the training of the first neural network model 100, the input image 1, the label image 3, and the correction parameter 10 can be used as training data. A plurality of label images 3 can be generated by changing the correction parameter 10 of several input images 1, and the first neural network model 100 can be trained by using the combination of the input image 1, the label image 3, and the correction parameter 10 collected in this manner as training data.

[0078] In this way, by using the correction parameter 10 as the input of the first neural network model 100 during the training of the first neural network model 100, the first neural network model 100 can be trained on the relationship between the correction parameter 10 and the label image 3. That is, the first neural network model 100 can be trained as to which label image 3 is generated when which correction parameter 10 is applied to the input image 1. When only the correction parameter 10 is changed while the input image 1 input to the trained first neural network model 100 remains the same, the inferred image 2 output from the first neural network model 100 also changes. Therefore, the user can control the inferred image 2 output from the first neural network model 100 to have desired image characteristics by adjusting the correction parameter 10 input to the first neural network model 100. Referring to Fig. 9 A specific embodiment in which the correction parameter 10 is applied as input to the first neural network model 100 is described.

[0079] Fig. 9 is a diagram for describing a method of applying a correction parameter as an input to a first neural network model according to an embodiment of the present disclosure. Fig. 9 In the embodiment of FIG. 1 , it is assumed that the correction parameter 10 includes a first parameter 11 and a second parameter 12 .

[0080] refer to Fig. 9 , the input image 1 is converted into a grayscale image by the decolorization module 500. The first operation unit 911 performs multiplication on the grayscale image and the first parameter 11, and the second operation unit 912 performs multiplication on the grayscale image and the second parameter 12. The third operation unit 920 may perform concatenation on the output of the first operation unit 911 and the output of the second operation unit 912, and may output the result to the first neural network model 100. That is, each of the first parameter 11 and the second parameter 12 included in the correction parameter 10 may be applied to the grayscale image converted from the input image 1, and may then be input to the first neural network model 100 together with the input image 1.

[0081] Although in Fig. 9 The grayscale image is multiplied by the first parameter 11 and the second parameter 12 and then concatenated with the input image 1, but the grayscale image and the first parameter 11 and the second parameter 12 may be processed by using an operation other than multiplication. For example, the grayscale image may be divided by each of the first parameter 11 and the second parameter 12 and then concatenated with the input image, or each of the first parameter 11 and the second parameter 12 may be added to the grayscale image and then concatenated with the input image 1.

[0082] Figure 4 is a diagram for describing a process of training the second neural network model 200 according to an embodiment of the present disclosure.

[0083] First, the reason for using the second neural network model 200 will be described.

[0084] The second neural network model 200 can infer correction parameters for correcting the input image 1 to have image characteristics that may be generally liked by many users (for example, image characteristics determined to be optimal by the designer of the neural network model).

[0085] Therefore, using the second neural network model 200, even if the user does not set or adjust the correction parameters each time, correction parameters for correcting the input image 1 to have optimal image characteristics can be automatically generated (inferred), so that the input image 1 can be corrected accordingly and provided to the user.

[0086] For example, when a user captures an image through a terminal on which the first neural network model 100 and the second neural network model 200 are installed, the captured image can be corrected according to the correction parameters inferred by the second neural network model 200 and can be displayed as a preview on the screen of the terminal.

[0087] Therefore, the user of the terminal can first check the image with good image characteristics (brightness, contrast, etc.) through preview, and can adjust the correction parameters to change the image characteristics only when the preview is not satisfactory.

[0088] refer to Figure 4 , add a second neural network model 200 that receives the input image 1 and infers and outputs the correction parameter 10. Figure 4 As shown, two neural network models, namely, the first neural network model 100 and the second neural network model 200, may be trained simultaneously in a state where the first neural network model 100 and the second neural network model 200 are connected, or the first neural network model 100 may be trained first, and then the second neural network model 200 may be trained in a state where the trained first neural network model 100 is fixed (the parameters included in the first neural network model 100 are fixed). In addition, each of the first neural network model 100 and the second neural network model 200 may be trained separately.

[0089] During the training of the second neural network model 200, the measured characteristic value 40 that quantifies the characteristic of the label image 3 (e.g., brightness, contrast, or color temperature) may be compared with the preset target characteristic value 50, and the second neural network model 200 may be updated to minimize the difference between the measured characteristic value 40 and the preset target characteristic value 50. The target characteristic value 50 may be preset to a value desired by the user (manager). That is, the second neural network model 200 may be trained to minimize the difference between the correction parameter 10 inferred by the second neural network model 200 when the input image 1 is input and the correction parameter that allows the label image 3 to have the preset image characteristics (allowing the label image generation module 300 to output the same as the label image 3). Figure 4 The model is based on the difference between the target feature value 50 in the image corresponding to the correction parameters).

[0090] In other words, the optimizer 400 may update the second neural network model 200 to minimize the loss value of the second loss function 30 indicating the difference between the measured characteristic value 40 and the target characteristic value 50. In this case, the second loss function 30 may include a combination of MAE, MSE, and SSIM.

[0091] The measured characteristic values ​​40 and the target characteristic values ​​50 used during the training of the second neural network model 200 may include a plurality of values ​​corresponding to a plurality of image characteristics. For example, the measured characteristic values ​​40 and the target characteristic values ​​50 may include a first characteristic value quantifying the brightness of the image and a second characteristic value quantifying the color temperature of the image.

[0092] exist Figure 4 In the embodiment, the measured characteristic value 40 of the label image 3 is obtained to train the second neural network model 200, and the obtained measured characteristic value 40 is compared with the preset target characteristic value 50. However, the measured characteristic value 40 of an image generated in the middle of the process of converting the input image 1 into the label image 3 (hereinafter, referred to as an “intermediate label image”) may be obtained, and the second neural network model 200 may be trained by comparing the obtained measured characteristic value 40 with the preset target characteristic value 50.

[0093] The following will further refer to Figure 5 A specific example is described.

[0094] When measuring specific values ​​to train the second neural network model 200, characteristic values ​​of an image (intermediate label image) to which only some of the multiple image correction algorithms (e.g., 310, 320, and 330) included in the label image generation module 300 are applied may be measured, and the second neural network model 200 may be trained by comparing the measured characteristic values ​​with preset target characteristic values.

[0095] exist Figure 5 , it is assumed that when an input image 1 is input to the label image generation module 300 , the input image 1 is sequentially converted into a label image 3 through a contrast enhancement algorithm 310 , a brightness correction algorithm 320 , and an exposure fusion algorithm 330 .

[0096] For example, a measured characteristic value quantifying the "brightness" of the intermediate label image (input image of the exposure fusion algorithm 330) obtained by applying only the contrast enhancement algorithm 310 and the brightness correction algorithm 320 to the input image 1 can be obtained, and the second neural network model 200 can be trained to minimize the difference (loss) between the obtained measured characteristic value and the preset target characteristic value of "brightness". Thus, when the second neural network model 200 is trained in this manner, the parameter (13) applied to the exposure fusion algorithm 330 can be trained to minimize the individual loss regardless of the target brightness (brightness corresponding to the target characteristic value).

[0097] The second neural network model 200 may include a plurality of neural network models. For example, when the second neural network model 200 infers the first correction parameter and the second correction parameter, the neural network model for inferring each correction parameter may exist separately. If necessary, a neural network model for inferring the third correction parameter may be added to the second neural network model 200, and the second neural network model 200 may be changed to infer the first correction parameter to the third correction parameter.

[0098] 3. (Process used) Method for correcting an image by using a first neural network and a second neural network

[0099] A method of correcting an image by using the first neural network model 100 and the second neural network model 200 trained according to the above method will be described. First, how the first neural network model 100 and the second neural network model 200 according to an embodiment of the present disclosure can be used in an actual device will be described. Fig.10 and Fig.11 First, a method of correcting an image by using the first neural network model 100 and the second neural network model 200 will be described, and then a method of correcting an image by using the first neural network model 100 and the second neural network model 200 will be described.

[0100] Fig.10 1 is a diagram schematically showing elements included in the mobile terminal 1000 according to an embodiment of the present disclosure. Although the device on which the first neural network model 100 and the second neural network model 200 are installed is Fig.10 The mobile terminal 1000 in the embodiment is not limited thereto, and the first neural network model 100 and the second neural network model 200 may be installed on any type of device (such as a camera) having an image capturing function. In addition, it is obvious that the first neural network model 100 and the second neural network model 200 may also be installed on any device among various types of devices that do not have an image capturing function but can perform image signal processing.

[0101] refer to Fig.10 , the mobile terminal 1000 may include a camera module 1010 and a main processor 1020 .

[0102] The camera module 1010 may include a lens module 1011 , an image sensor 1012 , and an image signal processor (ISP) 1013 , and the lens module 1011 , the image sensor 1012 , and the ISP 1013 may be mounted on one chip as a system-on-chip (SoC).

[0103] The image sensor 1012 may receive light transmitted through the lens module 1011 and may output an image, and the ISP 1013 may reconstruct and correct the image output from the image sensor 1012 and may output a final image. That is, when a user captures an image through the mobile terminal 1000, the final image reconstructed and corrected by the ISP 1013 is displayed to the user.

[0104] The ISP 1013 may include an image reconstruction module 1013a and an image correction module 1013b, and the first neural network model 100 and the second neural network model 200 may be included in the image correction module 1013b. In this case, the image reconstruction module 1013a and the image correction module 1013b may be software units that perform specific functions in a program for performing image processing.

[0105] The image reconstruction module 1013a may perform processing such as demosaicing and denoising on the original image output from the image sensor 1012 to output a linear RGB image, and the image correction module 1013b may correct the linear RGB image to output a final image. Fig.11 Specific embodiments of the image reconstruction module 1013a and the image correction module 1013b are described.

[0106] Fig.11 is a diagram schematically illustrating a detailed configuration of an ISP according to an embodiment of the present disclosure. Fig.11 The sensor 1110 may correspond to Fig.10 The image sensor 1012, and Fig.11 The NeuroISP 1120 corresponds to Fig.10 A specific embodiment of the ISP 1013. The neural ISP 1120 is an element that processes an image signal by using a neural network. The neural ISP 1120 may be an ISP based on a single neural network end-to-end, rather than a method of applying AI to a part of the ISP in units of modules.

[0107] refer to Fig.11 , the sensor 1110 may output a plurality of raw images. In this case, a "raw image" is an image output from the image sensor of the camera, and may refer to a Bayer format image having only one color channel per pixel.

[0108] Fig.11 The burst network (BurstNet) 1121 may correspond to Fig.10 The image reconstruction module 1013a, and Fig.11 The mastering network (MasteringNet) 1122 can correspond to Fig.10 The image correction module 1013b.

[0109] The burst network 1121 may receive a plurality of original images and may output a linear RGB image. The plurality of original images input to the burst network 1121 are images taken before and after a specific time point, and the burst network 1121 may output a linear RGB image by using the time information of the original images and performing demosaicing and denoising. In an embodiment of the present disclosure, the output linear RGB image may correspond to the input image 1 input to the neural network model.

[0110] The master network 1122 may perform correction on the linear RGB image. For example, the master network 1122 may adjust image characteristics by adjusting white balance (WB) on the linear RGB image, adjusting gamma values, or performing processes such as global tone mapping and local tone mapping, and may output an sRGB image as a final image.

[0111] The neural network model according to an embodiment of the present disclosure may be included in the master network 1122 and may correct a linear RGB image (input image).

[0112] Although it is assumed above that the input image 1 is a linear RGB image and the inferred image 2 is an sRGB image, the input image 1 and the inferred image 2 may be images of various formats. For example, each of the input image 1 and the inferred image 2 may be any one of a nonlinear RGB image, an sRGB image, an AdobeRGB image, a YCbCr image, and a Bayer image.

[0113] The format of each of the input image 1 and the inference image 2 may be determined according to the purpose or environment in which the neural network model is used. However, the format of the input image 1 used during training may remain the same even during inference, and by a similar principle, the format of the inference image 2 used during training may remain the same even during inference.

[0114] For example, when an input image 1 in a specific format is used during training of the first neural network model 100, the first neural network model 100 can receive an input image 1 in the same format (as the input image 1 used during training) and can process the input image to output an inferred image 2.

[0115] exist Fig.10In the embodiment of the present invention, the first neural network model 100 and the second neural network model 200 are installed on the camera module 1010. Different from this, the first neural network model 100 and the second neural network model 200 can be installed on the main processor 1020 of the mobile terminal 1000. The main processor 1020 may include a CPU 1011, a GPU 1012, an NPU 1013 and a memory 1014, and at least one of the CPU 1011, the GPU 1012 or the NPU 1013 may execute a program stored in the memory 1014 to implement the first neural network model 100 and the second neural network model 200.

[0116] The first neural network model 100 and the second neural network model 200 may not be installed on the device that captures the image, but may be implemented by an external device such as a cloud server. Fig.19 As shown, when the mobile terminal 1000 sends the original image obtained by the image sensor 1012 to the cloud server 1900, the processor 1910 of the cloud server 1900 can generate a linear RGB image from the original image by using the image reconstruction module 1911, and can correct the linear RGB image by using the first neural network model 100 and the second neural network model 200 included in the image correction module 1912, and then the corrected image can be stored in the cloud server 1900, or the corrected image can be sent to the mobile terminal 1000.

[0117] Mobile terminal 1000 can reconstruct a linear RGB image from the original image, and then can send the reconstructed linear RGB image to cloud server 1900, and processor 1910 of cloud server 1900 can correct the image by using first neural network model 100 and second neural network model 200, and then can store the corrected image in cloud server 1900, or can send the corrected image to mobile terminal 1000.

[0118] Fig.12 is a diagram showing a specific embodiment in which the first neural network model 100 and the second neural network model 200 are included in the image correction module 1013b.

[0119] refer to Fig.12 , the image correction module 1200 may include a first preprocessing unit 1210 , a second preprocessing unit 1220 , a first neural network model 100 , and a second neural network model 200 .

[0120] The first preprocessing unit 1210 for converting the input image 1 into a grayscale image can apply the auto white balance (AWB) gain, color correction matrix (CCM), gamma, etc. included in the metadata to the input image 1, and then perform channel wise MAX to extract the maximum value of RGB for each pixel to convert the input image 1 into a grayscale image and output the grayscale image.

[0121] The second preprocessing unit 1220 extracts the mean and variance of the pixel values ​​from the grayscale image output by the first preprocessing unit 1210, and scales and concatenates and outputs the extracted mean and variance as well as the input image 1 and the grayscale image. The specific operation structure for performing the scaled concatenation is shown in part 1201.

[0122] The second neural network model 200 may receive the output of the second preprocessing unit 1220 and may infer correction parameters (α, β). The correction parameters (α, β) inferred by the second neural network model 200 may be scaled and concatenated with the input image 1 and the grayscale image and may be input to the first neural network model 100.

[0123] The first neural network model 100 may output a filter 1230 corresponding to the correction parameters (α, β) and an input image, and may convert the input image 1 into an inferred image 2 by using the output filter 1230 .

[0124] The correction parameters (α, β) inferred by the second neural network model 200 may be adjusted by the user. When the user adjusts the correction parameters (α, β), the first neural network model 100 may output a filter 1230 corresponding to the adjusted correction parameters (α, β), and the input image 1 may be converted into the inferred image 2 by using the output filter 1230. As described above, the user may check the image corrected according to the correction parameters (α, β) inferred by the second neural network model 200 through preview, and may adjust the correction parameters (α, β) to have desired image characteristics.

[0125] Reference Figures 13 to 15 Describes a method for correcting an image by using a neural network model. For ease of explanation, assume that Fig.10 The ISP 1013 of the mobile terminal 1000 executes Figures 13 to 15 However, the embodiment is not limited thereto, and Figures 13 to 15 All or some of the operations of the mobile terminal 1000 may be performed by the main processor 1020 or Fig.19 Executed by processor 1910 of cloud server 1900.

[0126] refer to Fig.13 In operation 1301, the ISP 1013 may input an input image and correction parameters to the first neural network model 100. The first neural network model 100 may be a model trained to minimize the difference between an inferred image output when the input image and correction parameters are input and a label image corresponding to the correction parameters. In this case, the label image corresponding to the correction parameters may be an image obtained by correcting the input image by using at least one image correction algorithm when the correction parameters are applied to the at least one image correction algorithm.

[0127] The input image may be an image obtained when the image reconstruction module 1013a reconstructs and outputs the original image output from the image sensor 1012. The correction parameter may be a preset value (a value used during the training of the first neural network model 100), or may be a value adjusted according to the user's input. Figures 15 to 17 An embodiment of adjusting the correction parameters according to the user's input is described in detail.

[0128] In operation 1302, the ISP 1013 may obtain an inferred image corresponding to the correction parameter from the first neural network model 100. The ISP 1013 may display the obtained inferred image on the screen of the mobile terminal 1000, or may store the obtained inferred image in the memory 1014.

[0129] Fig.14 is a diagram for describing an embodiment in which a second neural network model infers correction parameters. Fig.14 The flowchart includes Fig.13 The detailed operations included in operation 1301.

[0130] refer to Fig.14 In operation 1401, the ISP 1013 may input an input image to the second neural network model 200. The second neural network model 200 may be a model trained to minimize the difference between a correction parameter inferred by the second neural network model 200 when the input image is input and a correction parameter that allows the label image to have preset image characteristics.

[0131] In operation 1402, the ISP 1013 may obtain correction parameters inferred by the second neural network model 200, and in operation 1403, the ISP 1013 may input the correction parameters to the first neural network model 100 together with the input image.

[0132] Fig.15 is a diagram for describing an embodiment in which a user adjusts correction parameters. Fig.15 The flowchart includes Fig.13The detailed operations included in operation 1301.

[0133] refer to Fig.15 In operation 1501, the ISP 1013 may receive an input for adjusting a correction parameter through an input / output interface (e.g., a touch screen or a microphone) of the mobile terminal 1000. In operation 1502, the ISP 1013 may change a preset first correction parameter to a second correction parameter based on the input received in operation 1501. In operation 1503, the ISP 1013 may input the second correction parameter to the first neural network model 100 together with the input image.

[0134] exist Fig.17 2 shows an embodiment in which a user adjusts correction parameters for brightness adjustment through a UI displayed on the screen of the mobile terminal 1000.

[0135] refer to Fig.17 , an image to be corrected (an inferred image corresponding to a preset first correction parameter) may be displayed in a first area 1710 of a UI screen 1700 for adjusting image characteristics, and a tool for adjusting image characteristics may be displayed at a lower portion of the UI screen 1700.

[0136] Assume the user selects Fig.17 The UI screen 1700 is shown as a tool for adjusting the brightness of an image. Information indicating that the currently adjusted image characteristic is "brightness" and a numerical value of the current brightness may be displayed in the second area 1720 of the UI screen 1700. A slider for adjusting the brightness may be displayed in the third area 1730 of the UI screen 1700, and the user may adjust the brightness of the input image by moving the slider through a touch input.

[0137] When the user moves Fig.17 When the slider in the third area 1730 of the UI screen 1700 is used to increase the brightness of the input image, the numerical value displayed in the second area 1720 may be increased, and the ISP 1013 may adjust the first correction parameter to the second correction parameter to increase the brightness of the image.

[0138] When a user captures an image by using mobile terminal 1000 and selects a tool for adjusting the brightness of the image, an image obtained by correcting an input image (e.g., a linear RGB image generated from an original image as the captured image) according to correction parameters inferred by second neural network model 200 may be first displayed in first area 1710 of UI screen 1700. When the user adjusts the brightness through tools displayed in second area 1720 and third area 1730 of UI screen 1700, the correction parameters (e.g., as shown in FIG. 1 ) may be adjusted accordingly. Fig.16(doubled as shown), and an image obtained by correcting the input image according to the adjusted correction parameters can be displayed in the first area 1710.

[0139] As described above, since the first neural network model 100 is trained based on the relationship between the correction parameter and the label image, when the correction parameter is changed, the label image corresponding to the changed correction parameter can be inferred. Fig.16 is a diagram for describing changes in the inferred image when the correction parameters input to the first neural network model are changed.

[0140] For ease of explanation, Fig.16 It is assumed that the correction parameters 10 only include parameters related to brightness adjustment. In addition, it is assumed that the first neural network model 100 is trained by using a label image generated by an image correction algorithm for adjusting brightness, and the image correction algorithm used during training corrects the image so that the brightness increases as the value of the correction parameter increases.

[0141] exist Fig.16 In the embodiment of the invention, the correction parameter 10 is doubled. The brightness of the label image corresponding to the doubled correction parameter 10 is higher than the brightness of the label image corresponding to the existing correction parameter 10. Therefore, the brightness of the inferred image 2 output from the first neural network model 100 also increases as the correction parameter 10 increases.

[0142] Although the user touches Fig.17 The user may adjust the correction parameters by voice input, although the user may not apply the input for adjusting the correction parameters through the UI screen 1700 in the mobile terminal 1000. For example, the user may input a voice (e.g., "Hey Bixby, make the picture brighter") instructing to adjust the image characteristics through a speaker provided in the mobile terminal 1000.

[0143] Reference Fig. 20 and 21 Describe the method of training the first neural network model 100 and the second neural network model 200. For ease of explanation, assume that Fig.18 The processor 1830 of the computing device 1800 executes Fig. 20 and 21 However, the embodiment is not limited thereto, and Fig. 20 and 21 All or some operations can be performed by Fig.10 The main processor 1020 of the mobile terminal 1000 executes.

[0144] refer to Fig. 20 In operation 2001, the processor 1830 may correct an input image by applying correction parameters to at least one image correction algorithm, thereby generating a label image.

[0145] In operation 2002, the processor 1830 may input a correction parameter and an input image to the first neural network model, and in operation 2003, the processor 1830 may obtain an inferred image output from the first neural network model.

[0146] In operation 2004, the processor 1830 may update the first neural network model to minimize the difference between the label image and the inferred image.

[0147] refer to Fig.21 , in operation 2101, the processor 1830 may input an input image to the second neural network model, and in operation 2102, the processor 1830 may obtain a correction parameter output from the second neural network model.

[0148] In operation 2103 , the processor 1830 may correct the input image by applying the correction parameters to at least one image correction algorithm, thereby generating a label image.

[0149] In operation 2104, the processor 1830 may obtain a measured characteristic value that quantifies an image characteristic of the label image, and in operation 2105, the processor 1830 may update the second neural network model to minimize a difference between the measured characteristic value and a preset target characteristic value.

[0150] When the first neural network model 100 and the second neural network model 200 according to the above-described embodiments of the present disclosure are installed on the mobile terminal 1000, the image correction module 1013b can be easily updated. Fig. 22 and 23 Describes methods for updating neural network models.

[0151] Fig. 22 Shown included in Fig.10 The image correction module 1013b in the ISP 1013 of the mobile terminal 1000 includes an example of an image correction algorithm instead of a neural network model. Fig. 22 , the image correction module 1013b may include a contrast enhancement algorithm A 2211 and a brightness correction algorithm 2220.

[0152] exist Fig. 22 In FIG. 1 , it is assumed that the image correction algorithm included in the image correction module 1013 b is updated. The updating is performed as follows.

[0153] 1) Replace contrast enhancement algorithm A with contrast enhancement algorithm B

[0154] 2) Remove the brightness correction algorithm

[0155] 3) Add exposure fusion algorithm

[0156] Updated results in Fig. 22 Reference Fig. 22 , in the image correction module 1013b, the contrast enhancement algorithm A 2211 is replaced with the contrast enhancement algorithm B 2212, the brightness correction algorithm 2220 is removed, and the exposure fusion algorithm 2230 is added.

[0157] In this way, when the image correction module 1013b is updated in a state where the image correction module 1013b includes the image correction algorithm, the following problem arises.

[0158] First, since an optimization process for improving processing speed should be performed whenever an image correction algorithm is changed (replaced, removed, or added), a lot of time and resources are required.

[0159] Secondly, because a certain image correction algorithm is implemented as separate hardware, in order to add the image correction algorithm, the hardware should be replaced or added. Specifically, when the image correction module 1013b is implemented on the mobile terminal 1000, the replacement or addition of hardware is subject to many restrictions.

[0160] Fig.23 The embodiment according to the present disclosure is shown Fig.10 An example of updating the image correction module 1013 b when the image correction module 1013 b included in the ISP 1013 of the mobile terminal 1000 includes the first neural network model 100 and the second neural network model 200 .

[0161] refer to Fig.23 , in order to update the first neural network model 100 included in the image correction module 1013b, the first neural network model 100 should be trained again. As described above, since the first neural network model 100 can be Fig.18 The separate computing device 1800 shown in the figure is trained, so in the following, it is assumed that the computing device 1800 trains the first neural network model 100.

[0162] As described above, the label image generation module 300 may include an image correction algorithm corresponding to the image correction characteristics on which the first neural network model 100 is to be trained. Fig.23 , the processor 1830 initially configures the label image generation module 300 to include a contrast enhancement algorithm A 3211 and a brightness correction algorithm 3220, trains the first neural network 100, and then installs the first neural network model 100 on the image correction module 1013b of the mobile terminal 1000.

[0163] Next, when it is necessary to adjust the image correction characteristics of the image correction module 1013b of the mobile terminal 1000, the processor 1830 may newly configure the label image generation module 300 to include the contrast enhancement algorithm B 3212 and the exposure fusion algorithm 3230, and may train the first neural network model 100 again. When the training of the first neural network model 100 is completed, the processor 1830 may send the newly trained first neural network model 100 to the mobile terminal 1000, and the mobile terminal 1000 may update the image correction module 1013b by installing the received first neural network model 100.

[0164] In this way, when the image correction module 1013b includes the first neural network model 100 and the second neural network model 200 according to an embodiment of the present disclosure, there is no need to perform an optimization process even when the image correction module 1013b is updated. In addition, generally, because the computing device 1800 that trains the first neural network model 100 and the second neural network model 200 includes sufficient hardware resources, the image correction algorithm included in the label image generation module 300 can be freely changed (replaced, removed, and added) to perform new training.

[0165] According to an embodiment of the present disclosure, a method for correcting an image by using a neural network model includes: inputting an input image and correction parameters into a first neural network model by a processor of a computing device, and obtaining, by the processor, an inferred image corresponding to the correction parameters from the first neural network model, wherein the first neural network model is a model trained to minimize the difference between an inferred image output when the input image and the correction parameters are input and a label image corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is an image obtained in the following manner: when the correction parameters are applied to at least one image correction algorithm, the input image is corrected by using at least one image correction algorithm, wherein the first neural network model is trained on the relationship between the correction parameters and the label image, so that the inferred image output from the first neural network model can be controlled by changing the correction parameters.

[0166] A first inferred image output from the first neural network model when an input image and a first correction parameter are input to the first neural network model may be different from a second inferred image output from the first neural network model when the input image and a second correction parameter are input to the first neural network model.

[0167] Inputting the correction parameters may include inputting the input image into the second neural network model by the processor, obtaining the correction parameters inferred by the second neural network model by the processor, and inputting the correction parameters together with the input image into the first neural network model by the processor, wherein the second neural network model is a model trained to minimize the difference between the correction parameters inferred by the second neural network model when the input image is input into the second neural network model and the correction parameters that allow the label image to have preset image characteristics.

[0168] The input of the correction parameters may include receiving an input for adjusting the correction parameters by the processor through an input / output interface of the computing device, changing a preset first correction parameter to a second correction parameter based on the input by the processor, and inputting the second correction parameter together with the input image into the first neural network model by the processor.

[0169] Obtaining an inferred image may include: obtaining, by a processor, an inferred image corresponding to a second correction parameter from a first neural network model, wherein the first neural network model is a model trained to minimize the difference between a first inferred image output when an input image and a first correction parameter are input and a first label image corresponding to the first correction parameter, and to minimize the difference between a second inferred image output when the input image and the second correction parameter are input and a second label image corresponding to the second correction parameter, wherein the first label image is an image obtained by correcting the input image by using at least one image correction algorithm when the first correction parameter is applied to at least one image correction algorithm, and the second label image is an image obtained by correcting the input image by using at least one image correction algorithm when the second correction parameter is applied to at least one image correction algorithm.

[0170] The first neural network model can be trained in the following manner: correcting an input image by applying correction parameters to at least one image correction algorithm to generate a label image, inputting the correction parameters and the input image to the first neural network model, obtaining an inferred image output from the first neural network model, and updating the first neural network model to minimize the difference between the label image and the inferred image.

[0171] The second neural network model can be trained in the following manner: inputting an input image to the second neural network model, obtaining correction parameters output from the second neural network model, generating a label image by correcting the input image by applying the correction parameters to at least one image correction algorithm, obtaining measured characteristic values ​​that quantify image characteristics of the label image, and updating the second neural network model to minimize the difference between the measured characteristic values ​​and a preset target characteristic value.

[0172] Inputting the input image and the correction parameters into the first neural network model may include: converting the input image into a grayscale image by the processor, generating a first grayscale image by applying a first parameter included in the correction parameters to the grayscale image by the processor, generating a second grayscale image by applying a second parameter included in the correction parameters to the grayscale image by the processor, and inputting the first grayscale image and the second grayscale image into the first neural network model by the processor.

[0173] The obtaining of the inferred image may include: obtaining, by a processor, at least one of a filter or mapping information from a first neural network model, and converting, by the processor, an input image into the inferred image by using at least one of the filter or mapping information.

[0174] Inputting the input image and the correction parameters into the first neural network model may include: receiving, by a processor, an original image obtained by an image sensor of the camera from a camera included in the computing device, generating, by the processor, an input image by using the original image, and inputting, by the processor, the input image and the correction parameters into the first neural network model.

[0175] Inputting the input image and the correction parameters into the first neural network model may include: receiving, by a computing device, an original image from an external device, generating, by a processor, an input image by using the original image, and inputting, by the processor, the input image and the correction parameters into the first neural network model.

[0176] A computing device for executing a neural network model for image correction may include: an input / output interface, wherein input is received from a user through the input / output interface; a memory, wherein a program for correcting an input image is stored; and a processor, the processor being configured to execute the program to input an input image and correction parameters into a first neural network model, and to obtain an inferred image corresponding to the correction parameters from the first neural network model, wherein the first neural network model is a model trained to minimize the difference between an inferred image output when the input image and the correction parameters are input and a label image corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is an image obtained by correcting the input image by using at least one image correction algorithm when the correction parameters are applied to at least one image correction algorithm, wherein the first neural network model is trained on the relationship between the correction parameters and the label image so that the inferred image output from the first neural network model can be controlled by changing the correction parameters.

[0177] When the processor inputs the correction parameters, the processor can input the input image into the second neural network model, obtain the correction parameters inferred by the second neural network model, and then input the correction parameters together with the input image into the first neural network model, wherein the second neural network model is a model trained to minimize the difference between the correction parameters inferred by the second neural network model when the input image is input and the correction parameters that allow the label image to have preset image characteristics.

[0178] When the processor inputs the correction parameter, the processor can receive the input for adjusting the correction parameter through the input / output interface, change the preset first correction parameter to the second correction parameter based on the input, and then input the second correction parameter into the first neural network model together with the input image.

[0179] When the processor obtains an inferred image, the processor can obtain an inferred image corresponding to a second correction parameter from a first neural network model, wherein the first neural network model is a model trained to minimize the difference between a first inferred image output when an input image and a first correction parameter are input and a first label image corresponding to the first correction parameter, and minimize the difference between a second inferred image output when an input image and a second correction parameter are input and a second label image corresponding to the second correction parameter, wherein the first label image is an image obtained by correcting the input image by using at least one image correction algorithm when the first correction parameter is applied to at least one image correction algorithm, and the second label image is an image obtained by correcting the input image by using at least one image correction algorithm when the second correction parameter is applied to at least one image correction algorithm.

[0180] The first neural network model can be trained in the following manner: correcting the input image by applying correction parameters to at least one image correction algorithm to generate a label image, inputting the correction parameters and the input image to the first neural network model, obtaining an inferred image output from the first neural network model, and then updating the first neural network model to minimize the difference between the label image and the inferred image.

[0181] The second neural network model can be trained in the following manner: inputting an input image to the second neural network model, obtaining correction parameters output from the second neural network model, correcting the input image by applying the correction parameters to at least one image correction algorithm to generate a label image, obtaining measured characteristic values ​​that quantify image characteristics of the label image, and then updating the second neural network model to minimize the difference between the measured characteristic values ​​and a preset target characteristic value.

[0182] When the processor inputs the input image and the correction parameters into the first neural network model, the processor may convert the input image into a grayscale image, generate a first grayscale image by applying a first parameter included in the correction parameters to the grayscale image, generate a second grayscale image by applying a second parameter included in the correction parameters to the grayscale image, and then input the input image, the first grayscale image, and the second grayscale image into the first neural network model.

[0183] When the processor obtains the inferred image, the processor may obtain at least one of the filter or the mapping information from the first neural network model, and then may convert the input image into the inferred image by using at least one of the filter or the mapping information.

[0184] When the processor inputs the input image and the correction parameters into the first neural network model, the processor may receive an original image obtained by an image sensor of the camera from a camera included in the computing device, generate the input image by using the original image, and then input the input image and the correction parameters into the first neural network model.

[0185] When the processor inputs the input image and the correction parameters into the first neural network model, the processor may receive an original image from an external device, generate the input image by using the original image, and then input the input image and the correction parameters into the first neural network model.

[0186] Various embodiments may be implemented or supported by one or more computer programs, each of which is formed of a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" as used herein refer to one or more computer programs, software components, instruction sets, programs, functions, objects, classes, instances, related data, or a portion thereof, suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" may include any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random-access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.

[0187] In addition, machine-readable storage media may be provided as non-transitory storage media. "Non-transitory storage media" are tangible devices and may exclude wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. In this case, "non-transitory" does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily. For example, "non-transitory storage media" may include a buffer that temporarily stores data. Computer-readable media can be any available media accessible to a computer, and may include all volatile and non-volatile media and detachable and non-detachable media. Computer-readable media include media that can permanently store data and media that can store data and rewrite the data later, such as rewritable optical disks or erasable memory devices.

[0188] The methods according to various embodiments may be provided in a computer program product. A computer program product is a product that can be purchased between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed (e.g., downloaded or uploaded) between two user devices (e.g., smart phones) through an application store or directly or online. When distributed online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0189] The above description is provided for illustration, and those of ordinary skill in the art will understand that various changes in form and detail can be easily made therein without departing from the basic characteristics and scope of the disclosed embodiments as defined by the appended claims. For example, the described techniques can be performed in a different order than the described methods, and / or the described elements (such as systems, structures, devices, or circuits) can be combined or integrated in a different form than the described methods, or can be replaced or substituted by other elements or equivalents to achieve appropriate results. Therefore, the embodiments should be considered only in a descriptive sense, and not for the purpose of limitation. For example, each component described as a single type can be implemented in a distributed manner, and similarly, the components described as distributed can be implemented in a combined form.

[0190] The scope of the disclosed embodiments is defined by the appended claims rather than the detailed description, and all changes or modifications within the scope of the appended claims and their equivalents are to be construed as being included in the scope of the disclosed embodiments.

Claims

1. A method for correcting an image, the method comprising: Inputting the input image and the correction parameters to the first neural network model (100) by a processor of the computing device; as well as obtaining, by the processor, an inferred image from the first neural network model (100), the inferred image corresponding to the correction parameter, wherein the first neural network model (100) is a model trained to minimize the difference between the inferred image output after the input image and the correction parameter are input and the label image corresponding to the correction parameter, wherein the label image corresponding to the correction parameter is an image obtained by correcting the input image by using at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm, Wherein, the first neural network model (100) is trained on the relationship between the correction parameters and the label image, so that the inferred image output from the first neural network model (100) can be controlled by changing the correction parameters.

2. The method according to claim 1, in, A first inferred image output from the first neural network model (100) when the input image and the first correction parameter are input to the first neural network model (100) is different from a second inferred image output from the first neural network model (100) when the input image and the second correction parameter are input to the first neural network model.

3. The method according to any one of claims 1 and 2, in, The correction parameter is a first correction parameter, Wherein, inputting the calibration parameters comprises: The processor inputs the input image into a second neural network model (200); obtaining, by the processor, the first correction parameter inferred by the second neural network model (200); and The processor inputs the first correction parameter together with the input image into the first neural network model (100), Wherein, the second neural network model (200) is a model trained to minimize the difference between the first correction parameter inferred by the second neural network model (200) after the input image is input into the second neural network model (200) and the second correction parameter that allows the label image to have preset image characteristics.

4. The method according to any one of claims 1 to 3, wherein: Inputting the calibration parameters includes: receiving, by the processor through an input / output interface of the computing device, an input for adjusting the correction parameter; The processor changes the preset first correction parameter to a second correction parameter based on the input; and The processor inputs the second correction parameter together with the input image into the first neural network model (100).

5. The method according to any one of claims 1 to 4, in, Obtaining the inferred image includes obtaining, by the processor, an inferred image corresponding to the second correction parameter from the first neural network model (100), wherein the first neural network model (100) is a model trained to minimize the difference between the first inferred image output after the input image and the first correction parameter are input and the first label image corresponding to the first correction parameter, and to minimize the difference between the second inferred image output after the input image and the second correction parameter are input and the second label image corresponding to the second correction parameter, wherein the first label image is an image obtained by correcting the input image by using the at least one image correction algorithm when the first correction parameter is applied to the at least one image correction algorithm, and The second label image is an image obtained by correcting the input image by using the at least one image correction algorithm when the second correction parameter is applied to the at least one image correction algorithm.

6. The method according to any one of claims 1 to 5, wherein: The first neural network model (100) is trained in the following manner: Correcting the input image by applying the correction parameters to the at least one image correction algorithm, thereby generating the label image; Inputting the correction parameters and the input image into the first neural network model (100); Obtaining the inferred image output from the first neural network model (100); as well as The first neural network model (100) is updated to minimize the difference between the label image and the inferred image.

7. The method according to any one of claims 1 to 6, wherein: The second neural network model (200) is trained in the following manner: Inputting the input image into the second neural network model (200); Obtaining the correction parameter output from the second neural network model (200); Correcting the input image by applying the correction parameters to the at least one image correction algorithm, thereby generating the label image; obtaining a measured characteristic value quantifying an image characteristic of the label image; as well as The second neural network model 200 is updated to minimize the difference between the measured characteristic value and the preset target characteristic value.

8. The method according to any one of claims 1 to 7, wherein: Inputting the input image and the correction parameters into the first neural network model (100) comprises: converting, by the processor, the input image into a first grayscale image; generating, by the processor, a second grayscale image by applying a first parameter included in the correction parameter to the first grayscale image; generating, by the processor, a third grayscale image by applying a second parameter included in the correction parameter to the first grayscale image; and The processor inputs the second grayscale image and the third grayscale image into the first neural network model (100).

9. The method according to any one of claims 1 to 8, wherein: Obtaining the inferred image includes: obtaining, by the processor, at least one of filter or mapping information from the first neural network model; and The input image is converted, by the processor, into the inferred image by using the at least one of the filter or the mapping information.

10. The method according to any one of claims 1 to 9, wherein: Inputting the input image and the correction parameters into the first neural network model comprises: receiving, by the processor, a raw image obtained by an image sensor of a camera included in the computing device; generating, by the processor, the input image by using the original image; and The processor inputs the input image and the correction parameters into the first neural network model (100).

11. The method according to any one of claims 1 to 10, wherein: Inputting the input image and the correction parameters into the first neural network model (100) comprises: Receiving, by the computing device, an original image from an external device; generating, by the processor, the input image by using the original image; and The processor inputs the input image and the correction parameters into the first neural network model (100). 12 . A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

13. A computing device comprising: an input / output interface, wherein input is received from a user through the input / output interface; a memory storing a program for executing the method for correcting an input image; as well as A processor configured to execute the method. Wherein, the method comprises: Inputting the input image and the correction parameters to the first neural network model (100), and obtaining an inferred image from the first neural network model (100), the inferred image corresponding to the correction parameter, wherein the first neural network model (100) is a model trained to minimize the difference between the inferred image output after the input image and the correction parameter are input and the label image corresponding to the correction parameter, wherein the label image corresponding to the correction parameter is an image obtained by correcting the input image by using at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm, Wherein, the first neural network model is trained on the relationship between the correction parameters and the label image, so that the inferred image output from the first neural network model (100) can be controlled by changing the correction parameters.

14. The computing device according to claim 13, in, A first inferred image output from the first neural network model (100) when the input image and the first correction parameter are input to the first neural network model (100) is different from a second inferred image output from the first neural network model (100) when the input image and the second correction parameter are input to the first neural network model (100).

15. A computing device according to any one of claims 13 and 14, in, The correction parameter is a first correction parameter, Wherein, inputting the calibration parameters comprises: The processor inputs the input image into a second neural network model (200), obtaining, by the processor, the first correction parameter inferred by the second neural network model (200), and inputting the first correction parameter together with the input image into the first neural network model (100), Wherein, the second neural network model (200) is a model trained to minimize the difference between the first correction parameter inferred by the second neural network model (200) after the input image is input and the second correction parameter allowing the label image to have preset image characteristics.

Images