Image editing method using neural network model, and electronic device for performing same

Abstract

This image editing method using a neural network model may comprise the steps of: acquiring an editing request for an input image from a user; adjusting, in response to the editing request, editing parameters generated by a neural network model with respect to the input image; editing the input image on the basis of the adjusted editing parameters; and outputting the edited image.

Description

Image editing method using a neural network model and electronic device for performing the same

[0001] The present disclosure relates to a method for editing an image using a neural network model and an electronic device for performing the same. Specifically, the present disclosure relates to a method for editing an image by adjusting editing parameters generated by a neural network model according to a user's editing request.

[0002] Recent advancements in neural network technology have led to the application of neural networks in a variety of fields. In particular, neural networks can be utilized in image signal processing, such as for video editing and restoration, to improve processing speed and image quality. For example, mobile devices like smartphones often have programs installed to edit videos captured by their cameras. Utilizing neural network models in this process can lead to significant improvements in speed and efficiency.

[0003] A method for editing an image using a neural network model may include a step of obtaining an editing request for an input image from a user, a step of adjusting an editing parameter generated by a neural network model for the input image according to the editing request, a step of editing the input image based on the adjusted editing parameter, and a step of outputting the edited image.

[0004] An electronic device for editing an image using a neural network model includes an input / output interface for receiving an input from a user and outputting an image, a memory storing a program or instructions for editing an image, and at least one processor, wherein the at least one processor executes the program or instructions stored in the memory, thereby obtaining an editing request for an input image from the user, adjusting an editing parameter generated by a neural network model for the input image according to the editing request, editing the input image based on the adjusted editing parameter, and then outputting the edited image.

[0005] A non-transitory computer-readable recording medium may have stored thereon a program for executing at least one of the embodiments of the disclosed method on a computer.

[0006] A computer program may be stored on a non-transitory storage medium for performing at least one of the embodiments of the disclosed method on a computer.

[0007] FIG. 1 is a diagram for explaining configurations for processing a video signal according to one embodiment of the present disclosure.

[0008] FIG. 2 is a diagram illustrating components included in an electronic device for editing an image using a neural network model according to one embodiment of the present disclosure.

[0009] FIG. 3 is a drawing for explaining detailed configurations and operations of an image signal processing unit (ISP) according to one embodiment of the present disclosure.

[0010] FIG. 4 is a drawing for explaining detailed configurations and operations of an image editing module according to one embodiment of the present disclosure.

[0011] FIG. 5 is a diagram illustrating UI screens displayed on an electronic device according to one embodiment of the present disclosure, which receive an editing request for an image from a user and output an edited image according to the editing request.

[0012] FIG. 6 is a diagram for explaining a process of performing exposure fusion based on editing parameters generated by a neural network model (editing parameter generation model) according to one embodiment of the present disclosure.

[0013] FIG. 7 is a diagram illustrating a method for training a neural network model (edit parameter generation model) according to one embodiment of the present disclosure.

[0014] FIG. 8 is a diagram illustrating a system that performs an image editing method using a neural network model according to one embodiment of the present disclosure through a cloud server.

[0015] FIGS. 9 to 14 are flowcharts for explaining an image editing method using a neural network model according to embodiments of the present disclosure.

[0016] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0017] Unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" are to be understood to include plural referents. Thus, for example, reference to "a surface of a composition" may also include reference to one or more of such surfaces.

[0018] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.

[0019] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0020] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the specification. In addition, when describing an embodiment of the present disclosure, if a detailed description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

[0021] It should be understood that the blocks and combinations of flowcharts in each of the flowcharts in this disclosure can be implemented by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be divided and stored across multiple different memories.

[0022] In one embodiment, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and the instructions, when executed by the processor of the computer or other programmable data processing apparatus, can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0023] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0024] All functions or operations described in the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors may be a circuitry that performs processing, and may include circuitry such as an Application Processor (AP), a Communication Processor (CP), a Graphical Processing Unit (GPU), a Neural Processing Unit (NPU), a Microprocessor Unit (MPU), a System on Chip (SoC), an Integrated Chip (IC), etc.

[0025] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided through a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In addition, in one embodiment, the '~ unit' may include one or more processors.

[0026] Below, the meanings of terms used in this disclosure are explained.

[0027] The term "development" refers to the process of processing image files on a computer. Specifically, it can refer to the process of converting raw image files obtained during video shooting into playable file formats (e.g., JPEG, TIFF, etc.). During the video development process, "restoration" of the image may be performed, and in this case, "restoration" of the image can refer to the process of improving the quality of the image by interpolating some pixel values ​​included in the image or removing noise.

[0028] "Editing" a video can refer to the act of altering its characteristics, either in part or in its entirety. "Characteristics" of the video can refer to factors such as brightness, contrast, tone, and color temperature. Terms such as "correction" may also be used in place of "editing."

[0029] The term "editing algorithm" may refer to an algorithm for performing image editing. An electronic device according to an embodiment of the present disclosure may support various editing algorithms, and as described below, editing parameters used may vary depending on the editing algorithm. Instead of the term "editing algorithm," terms such as "editing function," "correction algorithm," or "correction function" may also be used.

[0030] The term "editing parameter" may refer to a parameter used when editing an image. For example, when editing an image using an image editing algorithm, the parameter applied to the image editing algorithm may be referred to as an editing parameter. The degree to which the characteristics of the image are changed may be determined depending on the value of the editing parameter. The editing parameter may include various types of parameters (e.g., parameters affecting brightness changes, parameters affecting exposure changes, parameters affecting contrast ratio changes, etc.). According to one embodiment of the present disclosure, a neural network model may generate and output editing parameters corresponding to an image. For example, when an arbitrary image is input to the neural network model, the neural network model may output editing parameters corresponding to the arbitrary image. To this end, the neural network model according to one embodiment of the present disclosure may be a model trained to generate editing parameters for editing an arbitrary image according to preset criteria when an arbitrary image is input. Instead of 'edit parameter', terms such as 'correction parameter', 'network output parameter', or 'auto-generated parameter' may also be used.

[0031] An 'adjustment parameter' may refer to a parameter for adjusting an editing parameter, and the adjustment parameter may be determined according to a user's editing request. In other words, the adjustment parameter may be said to be a parameter used to edit an image to reflect the user's intention. According to one embodiment of the present disclosure, when a user requests editing of an image through a UI of an electronic device, the electronic device may adjust the editing parameter based on an adjustment parameter corresponding to the user's editing request, and edit the image based on the adjusted editing parameter. Instead of the 'adjustment parameter', terms such as 'control parameter', 'remapping parameter', 'editing adjustment parameter', or 'user parameter' may also be used.

[0032] An 'editing parameter generation model' may refer to a neural network model that generates editing parameters. In other words, the editing parameter generation model may refer to a neural network model that infers optimal editing parameters for a given input image. According to one embodiment of the present disclosure, when an image is input to the editing parameter generation model, the editing parameter generation model may generate editing parameters for editing the input image into an image that is pleasing to the eye (e.g., an image with characteristics preset by an experiment or a manager's decision). How to train the editing parameter generation model for this purpose will be described in detail below with reference to the drawings. Terms such as 'neural network model' may also be used instead of the 'editing parameter generation model.'

[0033] A 'transform function' may refer to a function that transforms the value of an adjustment parameter in order to adjust the editing parameter based on the adjustment parameter. In other words, the transform function may refer to a function that transforms the value of the adjustment parameter into a scale for use in adjusting the editing parameter. Or, in other words, the transform function may refer to a function that transforms the value of the adjustment parameter into a scaling factor for adjusting the editing parameter. In this case, the 'scaling factor' may refer to a value for scaling the editing parameter according to the adjustment parameter. For example, the adjustment parameter may be transformed into a scaling factor having a value within a certain range (e.g., a value from 0 to 2) by the transform function. Instead of 'transform function', terms such as 'value transform' or 'scale transform function' may also be used.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0035] The present disclosure relates to a method for editing an image using a neural network model. An electronic device according to one embodiment of the present disclosure may acquire parameters (adjustment parameters) corresponding to a user's editing request to edit an image in accordance with a user's request, and adjust editing parameters generated by a neural network model based on the acquired parameters. Below, the overall process of processing an image signal (developing and editing an image) will first be described, followed by a detailed description of a process for editing an image by adjusting editing parameters generated by a neural network model according to a user's editing request.

[0036] FIG. 1 is a diagram illustrating components for processing an image signal according to an embodiment of the present disclosure. Referring to FIG. 1, an electronic device (100) according to an embodiment of the present disclosure may include a photographing module (110) and an image signal processing unit (ISP unit) (200). The image signal processing unit (200) may include an image development module (210) and an image editing module (220).

[0037] The image development module (210) and the image editing module (220) may be software configurations implemented by executing a program or instruction stored in a memory by a processor of an electronic device (100) that processes image signals. In addition, the image development module (210) and the image editing module (220) may be virtual configurations in which no matching hardware device actually exists. Therefore, the operations described as being performed by the image development module (210) or the image editing module (220) may be viewed as actually being performed by the processor of the electronic device (100) by executing a program stored in a memory. The processor and memory of the electronic device (100) will be described below with reference to FIG. 2.

[0038] Referring to FIG. 1, the overall process of processing an image signal in an electronic device (100) is schematically described.

[0039] When the shooting module (110) shoots an image and transmits it to the image signal processing unit (200), the image signal processing unit (200) can develop the image and then edit it to generate an output image (20).

[0040] The image development module (210) is a module for performing an operation of developing an image received from the photographing module (110). For example, the image development module (210) can convert an image in the form of a raw file received from the photographing module (110) into the form of a playable file. According to one embodiment, the image development module (210) can perform at least one restoration operation among lens shading correction (LSC), bad pixel correction (BPC), demosaicing, and denoising while developing an input image received from the photographing module (110). The image development module (210) may also be implemented to include at least one neural network.

[0041] The image editing module (220) is a module for editing an image. The image editing module (220) can make the image look better by increasing the brightness or improving the contrast ratio of the image received from the image development module (210). According to one embodiment, the image editing module (220) can change the image characteristics by adjusting the white balance (WB), performing color correction (CC), adjusting the gamma value, or performing processing such as global tone mapping, local tone mapping, or HDR effect on the received image. The image editing module (220) may be implemented to include at least one neural network. According to one embodiment of the present disclosure, the image editing module (220) may include a neural network model (editing parameter generation model) for generating editing parameters, and can edit the image by adjusting the editing parameters according to a user's editing request. The detailed configurations included in the video editing module (220) are described in detail below with reference to FIG. 4.

[0042] The image signal processing unit (200) of the electronic device (100) illustrated in FIG. 1 includes an image development module (210) and an image editing module (220), but the electronic device (100) according to one embodiment of the present disclosure may include only the image editing module (220).

[0043] Referring to FIG. 3, a specific example of an image signal processing unit (200) will be described. The image signal processing unit (200) may be implemented using a neural network model. According to one embodiment, the image signal processing unit (200) may be a method of applying a neural network to a part of the ISP in units of modules. Alternatively, according to one embodiment, the image signal processing unit (200) may be a configuration based on a single neural network end-to-end. FIG. 3 illustrates a specific example of a neural network-based image signal processing unit (Neuro-ISP) (200). In the image signal processing unit (200) illustrated in FIG. 3, a neural network is applied in units of modules, and therefore, the image development module (210) and the image editing module (220) may each include a neural network.

[0044] Referring to FIG. 3, the image sensor (111) can output a plurality of raw images. The image sensor (111) is a component included in the photographing module (110) of the electronic device (100), which will be described below with reference to FIG. 2. According to one embodiment, the plurality of raw images may be images captured continuously at a certain time interval. In addition, according to one embodiment, the raw image may be an image in Bayer format having only one color channel per pixel.

[0045] The image development module (210) can receive multiple raw images and output a single linear RGB image. The multiple raw images input to the image development module (210) are multiple images taken before and after a specific point in time, and the image development module (210) can output a single linear RGB image by using temporal information of the raw images and performing LSC, BPC, align & fusion, demosaicing, and denoising.

[0046] The image editing module (220) can perform editing on a linear RGB image. According to one embodiment of the present disclosure, the image editing module (220) can change the image characteristics by adjusting the white balance, performing color correction, adjusting the gamma value, or performing processing such as global tone mapping, local tone mapping, or HDR effect on the linear RGB image, and output the sRGB image as the output image (20). To this end, the image editing module (220) can receive a white balance gain (WBG) and a color correction matrix (CCM) from the image sensor (111).

[0047] In the embodiment illustrated in FIG. 3, it is assumed and explained that the image output from the image sensor (111) is a Bayer image, the image output from the image development module (210) is a linear RGB image, and the image output from the image editing module (220) is an sRGB image. However, this is not limited to the above, and each image may be an image in various formats. For example, the above images may be any one of a non-linear RGB image, an sRGB image, an AdobeRGB image, a YCbCr image, and a Bayer image.

[0048] An electronic device (100) according to an embodiment of the present disclosure may be a device having a photographing function and an operation processing function, such as a smartphone, tablet, or digital camera, and may also be various types of devices (e.g., a laptop, a cloud server, etc.) that can receive image files or video files and perform an image signal processing process even without a photographing function. The hardware configuration of an electronic device (100) that performs image signal processing according to embodiments of the present disclosure will be described in detail below with reference to FIG. 2.

[0049] FIG. 2 is a diagram illustrating components included in an electronic device (100) according to one embodiment of the present disclosure. Referring to FIG. 2, the electronic device (100) may include a photographing module (110), a communication interface (120), an input / output interface (130), a processor (140), and a memory (150).

[0050] The photographing module (110) may include a lens module (not shown) and an image sensor (111 of FIG. 3). The image sensor may output an image by receiving light transmitted through the lens module. According to one embodiment of the present disclosure, the image sensor may output a raw image in Bayer format having only one color channel per pixel, but is not limited thereto and the image sensor may output images in various formats. The electronic device (100) according to one embodiment of the present disclosure may not include the photographing module (110) and may perform development and editing operations on images received from an external source.

[0051] The communication interface (120) is a configuration for transmitting and receiving signals (such as control commands and data) with an external device via wire or wirelessly, and may be implemented to include a communication chipset that supports various communication protocols. The communication interface (120) may receive a signal from the outside and output it to the processor (140), or transmit a signal output from the processor (140) to the outside. The electronic device (100) may also receive an image to be edited from the outside via the communication interface (120).

[0052] The input / output interface (130) may include an input interface (e.g., a touch screen, a hard button, a microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., a display panel, a speaker, etc.) for displaying the results of execution of an operation according to the user's control or the status of the electronic device (100). According to one embodiment of the present disclosure, the electronic device (100) may display an image to be edited through the input / output interface (130), and the user may input an editing request for the image through the input / output interface (130). A specific embodiment in which the user checks the image and inputs an editing request through a UI screen displayed on the input / output interface (130) will be described below with reference to FIG. 5.

[0053] The processor (140) controls a series of processes to operate the electronic device (100) according to the embodiments described below, and may be composed of one or more processors. The one or more processors included in the processor (140) may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. The one or more processors included in the processor (140) may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). When the one or more processors included in the processor (140) are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0054] The processor (140) can write data to the memory (150) or read data stored in the memory (150), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program or at least one instruction stored in the memory (150). Accordingly, the processor (140) can perform operations described in the following embodiments, and operations described as being performed by the electronic device (100) or detailed components (210 and 220 of FIG. 1, 410 to 480 of FIG. 4) included in the electronic device (100) in the following embodiments can be regarded as being performed by the processor (140) unless otherwise described.

[0055] The memory (150) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (150) may not exist separately and may be configured to be included in the processor (140). The memory (150) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program or at least one instruction for performing operations according to embodiments described below may be stored in the memory (150). The memory (150) may also provide stored data to the processor (140) upon request of the processor (140).

[0056] Hereinafter, with reference to FIGS. 4 to 6, a process of editing an image by an image editing module (220) according to an embodiment of the present disclosure will be described in detail. The image editing module (220) according to an embodiment of the present disclosure can adjust editing parameters inferred by a neural network model for an input image (10) according to a user's editing request, and edit the input image (10) using the adjusted parameters.

[0057] 1. Generation of editing parameters and image editing using a neural network model - 1st editing (automatic editing)

[0058] First, referring to FIG. 4, detailed components included in the image editing module (220) according to one embodiment of the present disclosure will be described. The detailed components (410 to 480) included in the image editing module (220) illustrated in FIG. 4 may be components classified based on function or role. The detailed components (410 to 480) illustrated in FIG. 4 may be software components implemented by the processor (140) of the electronic device (100) executing a program stored in the memory (150), or may be virtual components for which no actual matching hardware device exists. In other words, the operations performed by the processor (140) of the electronic device (100) executing the program stored in the memory (150) may be classified into a plurality of groups based on function or purpose, and the entities performing the operations included in each classified group may be expressed as the detailed components (410 to 480) of FIG. 4. Accordingly, the operations described as being performed by the detailed components (410 to 480) of the image editing module (220) of FIG. 4 can be seen as actually being performed by the processor (140) of the electronic device (100) executing a program stored in the memory (150).

[0059] The preprocessing unit (410) can perform preprocessing on the input image (10). As described above, the input image (10) may be an image developed by the image developing module (210), or may be an image directly received from the photographing module (110) or an external device. According to one embodiment of the present disclosure, the preprocessing unit (410) can perform preprocessing, such as adjusting white balance or performing color correction, on the input image (10). In addition, the preprocessing unit (410) can generate an illumination map (40) from the input image (10). The illumination map (40) is a map that expresses the degree of brightness and darkness of the image, and can be used in an editing algorithm, such as exposure fusion, which will be described later. In the embodiment illustrated in FIG. 4, the preprocessing unit (410) generates an illumination map (40) from an input image (10) and transmits it to the editing unit (440). However, the preprocessing unit (410) may also generate different types of data and transmit them to the editing unit (440) depending on the editing algorithm (editing function) performed by the editing unit (440).

[0060] The editing parameter generation model (420) is a neural network model for generating editing parameters. According to one embodiment of the present disclosure, the editing parameter generation model (420) may be an image-to-value network that outputs a value (editing parameter) corresponding to an input image when an image is input. The image-to-value network has the advantages of being smaller in size and higher controllability compared to an image-to-image network. Therefore, the editing parameter generation model (420) according to one embodiment of the present disclosure may have advantageous features for use in a mobile terminal by being implemented as an image-to-value network.

[0061] There may be various types of editing parameters, and the types of editing parameters required may be determined depending on the type of editing algorithm used by the editing unit (440). In one embodiment of the present disclosure, it is assumed that the editing unit (440) performs exposure fusion when a user requests brightness adjustment for a shadow area or brightness adjustment for a highlight area. Accordingly, the editing parameter generation model (420) can generate μ and k, which are editing parameters required to perform exposure fusion. A method for the editing unit (440) to perform exposure fusion using the editing parameters μ and k will be described in detail below with reference to FIG. 6. According to one embodiment, the editing parameter generation model (420) may further generate editing parameters for changing a contrast ratio, editing parameters for changing brightness, or editing parameters for performing image stretching.

[0062] The editing parameter generation model (420) may be a neural network model trained to generate editing parameters for editing an input image according to preset criteria when an image is input. For example, the editing parameter generation model (420) may infer and output editing parameters for editing an input image into an image that is pleasing to the eye (e.g., an image with preset characteristics determined by an experiment or an administrator). In other words, the editing parameter generation model (420) may infer editing parameters for editing an image so that it has image characteristics that a user would like. In general, an input image (10) may have a problem in that the entire or a portion of the image is dark or has a low contrast ratio, making it difficult for a user to recognize objects within the image. To solve this problem, the editing parameter generation model (420) may be trained in advance to infer editing parameters for increasing the brightness of the input image (10) or improving the contrast ratio. The direction in which the image is edited due to the editing parameters inferred by the editing parameter generation model (420) may be determined during the training process of the editing parameter generation model (420). A specific method for training the editing parameter generation model (420) is described in detail below with reference to FIG. 7.

[0063] Even if there is no editing request from the user, the editing parameter generation model (420) generates editing parameters corresponding to the input image (10), and the image editing module (220) can edit the image based on the generated editing parameters.

[0064] The editing parameter generation model (420) may be implemented to include various types of deep learning networks. For example, the editing parameter generation model (420) may be implemented using a ResNet (Residual Network), a type of convolutional neural network (CNN), but is not limited thereto.

[0065] The editing parameters generated by the editing parameter generation model (420) can be applied to the editing algorithm performed in the editing execution unit (440) or the post-processing unit (450).

[0066] The editing parameter adjustment unit (430) is configured to adjust editing parameters according to a user's editing request. The specific method by which the editing parameter adjustment unit (430) adjusts editing parameters is described in detail in the section '3. When adjusting editing parameters' below. If there is no editing request from the user, the editing parameter adjustment unit (430) can output the input editing parameters as is to the editing execution unit (440).

[0067] The editing unit (440) is configured to edit an image based on editing parameters. When the editing unit (440) receives a preprocessed input image (10) and editing parameters, it can edit the preprocessed input image (10) by applying the received editing parameters to an editing algorithm. Hereinafter, with reference to FIG. 6, an embodiment in which the editing unit (440) edits an image by performing an exposure fusion algorithm will be described in detail.

[0068] According to one embodiment of the present disclosure, the editing parameter generation model (420) infers editing parameters μ and k, and the editing execution unit (440) can perform an exposure fusion algorithm on an image using μ and k. The exposure fusion algorithm is an editing algorithm that generates an image with changed exposure (exposure image) from a given image and generates a new image by performing a weighted sum on the two images.

[0069] Referring to FIG. 6, the editing unit (440) can obtain an exposure image (61) with changed exposure from the input image (10) using the editing parameter k. For the convenience of explanation, it is said that the editing unit (440) generates the exposure image (61) from the input image (10), but in reality, the image processed by the editing unit (440) may be an input image (10) on which preprocessing has been performed after development. If a matrix including pixel values ​​of the input image (10) as elements is X, and a matrix including pixel values ​​of the exposure image (61) as elements is X', then according to one embodiment, the editing unit (440) can generate the exposure image (61) from the input image (10) by applying the editing parameter k to the following mathematical expression 1.

[0070]

[0071] At this time, a and b can be constants whose values ​​are determined according to the situation or need.

[0072] As previously described, the preprocessing unit (410) can obtain an illumination map (40) from the input image (10). The method of obtaining the illumination map (40) from the input image (10) can be implemented in various ways, and a detailed description thereof is omitted.

[0073] The editing unit (440) can obtain a first weight matrix (62) from the illumination map (40) using the editing parameter μ. If a matrix including pixel values ​​of the illumination map (40) as elements is T and the first weight matrix (62) is W, then according to one embodiment, the editing unit (440) can generate the first weight matrix (62) from the illumination map (40) by applying the editing parameter μ to the following mathematical expression 2.

[0074]

[0075] The second weight matrix (63) can be determined based on the first weight matrix (62). For example, the editing unit (440) can determine the second weight matrix such that the sum of the first weight matrix (62) and the second weight matrix (63) becomes 1. In other words, the editing unit (440) can obtain the second weight matrix (63) (1-W) by subtracting the first weight matrix (62) W from a matrix in which all elements are 1. The reason for making the sum of the weight matrices (62, 63) become 1 in this way is to adjust the brightness of only a specific area (shadow area or highlight area) while maintaining the overall brightness of the image.

[0076] The editing unit (440) can generate an output image (20) by performing a weighted sum by applying the first weight matrix (62) and the second weight matrix (63) to the input image (10) and the exposure image (61), respectively. In other words, the editing unit (440) can generate an output image (20) by adding the result of multiplying the input image (10) by the first weight matrix (62) and the result of multiplying the exposure image (61) by the second weight matrix (63).

[0077] The influence of the values ​​of the editing parameters μ and k in the exposure fusion algorithm described above is as follows. As the value of the editing parameter k increases, the brightness of the generated exposure image (61) can increase. Depending on the value of the editing parameter μ, the degree to which the bright area (highlight area) and dark area (shadow area) of the output image (20) are brightened compared to the input image (10) can be determined.

[0078] In addition to exposure fusion, the editing unit (440) may also perform editing algorithms such as a contrast enhancement algorithm or a brightness correction algorithm. To this end, the editing parameter generation model (420) may generate editing parameters used in various editing algorithms.

[0079] The post-processing unit (450) can perform various types of post-processing on the image output from the editing unit (440). For example, the post-processing unit (450) can perform post-processing such as contrast boost or chroma flattening. In addition, the post-processing unit (450) can also perform at least one editing algorithm using the editing parameters generated by the editing parameter generation model (420).

[0080] As described above, the editing parameter generation model (420) included in the image editing module (220) can infer editing parameters for appropriately changing the characteristics of the image according to the shooting environment, etc. Therefore, even if there is no editing request from the user, the image editing module (220) can generate an output image (20) by editing the input image (10) based on the editing parameters inferred by the editing parameter generation model (420). In this way, the image output by the image editing module (220) in a situation where there is no editing request from the user can be referred to as a primary 'edited image' or 'automatically edited image'.

[0081] The user adjustment module (460), the conversion unit (470), and the additional editing unit (480) are described below along with an embodiment of additionally editing an image according to a user's editing request.

[0082] 2. Additional video editing according to the user's editing request

[0083] The user can check the output image (20) (primarily edited image or automatically edited image) that has been primarily edited according to the process described above through the screen of the electronic device (100) and request additional editing (editing adjustment) of the output image (20). The processes performed by the electronic device (100) when the user requests additional editing of the primary edited image will be described below.

[0084] According to one embodiment of the present disclosure, a user may request to edit an image using one or more editing functions (editing algorithms) among a plurality of editing functions supported by the electronic device (100). According to one embodiment of the present disclosure, among the plurality of editing functions supported by the electronic device (100), there may be an editing function that uses editing parameters inferred by the editing parameter generation model (420), and there may also be an editing function that does not use editing parameters inferred by the editing parameter generation model (420). In other words, among the plurality of editing functions supported by the electronic device (100), there may be an editing function that requires a neural network model (editing parameter generation model (420)) and an editing function that does not require a neural network model.

[0085] Whether an editing function requires a neural network model can be determined by various criteria. According to one embodiment, an editing function that requires image analysis to perform editing can be classified as an editing function that requires a neural network model. For example, in order to perform an editing function such as adjusting the brightness of a shadow area or adjusting the brightness of a highlight area, the electronic device (100) must determine which part of the image is a shadow area or a highlight area. Therefore, adjusting the brightness of a shadow area or adjusting the brightness of a highlight area can be classified as an editing function that requires a neural network model. According to one embodiment, other editing functions such as contrast ratio adjustment, filtering, denoising, vertical / horizontal alignment, shadow erasing, and light erasing can also be classified as editing functions that require a neural network model. On the other hand, editing functions such as color temperature adjustment, brightness adjustment, tone adjustment, and exposure adjustment do not require image analysis and can therefore be classified as editing functions that do not require a neural network model.

[0086] According to one embodiment of the present disclosure, the electronic device (100) can perform editing in different ways depending on whether the editing function requested by the user requires a neural network model. In other words, the process by which the electronic device (100) performs editing may vary depending on the editing function requested by the user. According to one embodiment of the present disclosure, the electronic device (100) can improve processing speed and use resources efficiently by not using editing parameters inferred by a neural network model when performing an editing function that does not require analysis of an image. Hereinafter, a process by which the electronic device (100) performs additional editing of an image will be described by dividing the case in which the user's editing request requires a neural network model into the case in which the user's editing request does not require a neural network model and the case in which the user's editing request does not require a neural network model.

[0087] 3. If the user-requested editing function requires a neural network model - adjust the editing parameters.

[0088] If the editing function requested by the user is an editing function that utilizes a neural network model, the electronic device (100) can adjust editing parameters according to the user's editing request and edit the image based on the adjusted parameters.

[0089] A method for a user to check an automatically edited output image (20) through the screen of an electronic device (100) and request additional editing of the output image (20) is described with reference to FIG. 5.

[0090] FIG. 5 is a diagram illustrating UI screens displayed on an electronic device (100) according to one embodiment of the present disclosure, which receive an editing request for an image from a user and output an edited image according to the editing request.

[0091] In the first screen (510) of FIG. 5, an automatically edited first edited image (511) is displayed, and below that, UI configurations (sliders) for a user to input an editing request are displayed. For example, when an image is captured through the shooting module (110) of the electronic device (100), an editing parameter generation model (420) infers editing parameters corresponding to the captured image (input image), and a first edited image (511) edited based on the inferred editing parameters can be displayed on the first screen (510).

[0092] The user can check the first edited image (511) displayed on the first screen (510) and request editing of the first edited image (511) through the sliders displayed below. For example, if the user requests to increase the brightness of the shadow area by adjusting the slider (522) as displayed on the second screen (520), the electronic device (100) can increase the brightness of the shadow area of ​​the first edited image (511) to generate a second edited image (521) and display the generated second edited image (521) on the second screen (520). To this end, the electronic device (100) can adjust editing parameters according to the user's adjustment of the slider (522) and edit the input image based on the adjusted editing parameters.

[0093] The process of adjusting editing parameters according to a user's editing request and editing a video based on the adjusted editing parameters is described in detail below.

[0094] (1) Acquisition of adjustment parameters corresponding to the user's editing request

[0095] Referring to FIG. 4, the user adjustment module (460) included in the image editing module (220) can receive an editing request from a user and start a process corresponding to the editing function requested by the user.

[0096] As previously described, among the editing functions supported by the electronic device (100), there may be editing functions that require a neural network model and editing functions that do not require a neural network model. In the embodiment illustrated in FIG. 4, the editing function that requires a neural network model is referred to as a first editing function (461), and the editing function that does not require a neural network model is referred to as a second editing function (462).

[0097] When a user requests editing using the first editing function (461), the user adjustment module (460) can output an adjustment parameter corresponding to the user's editing request. For example, as in the second screen (520) of FIG. 5, when the user requests to increase the brightness of the shadow area by 40, the user adjustment module (460) can output an adjustment parameter with a value corresponding to 'increase the brightness of the shadow area by 40'.

[0098] According to one embodiment of the present disclosure, there may be a type of adjustment parameter corresponding to each of a plurality of editing functions supported by the electronic device (100).

[0099] (2) Adjustment of editing parameters based on adjustment parameters

[0100] Referring to FIG. 4, the conversion unit (470) can convert the adjustment parameters received from the user adjustment module (460). In order to adjust the editing parameters based on the adjustment parameters, a scale conversion of the adjustment parameters is required. Therefore, according to one embodiment, the conversion unit (470) can obtain a scaling factor by converting the values ​​of the adjustment parameters using a transform function.

[0101] When the conversion unit (470) transmits a scaling factor to the editing parameter adjustment unit (430), the editing parameter adjustment unit (430) can adjust the editing parameter using the scaling factor. For example, the editing parameter adjustment unit (430) can apply the scaling factor to the editing parameter by multiplying or adding the scaling factor to the editing parameter.

[0102] A specific example will be given of a method in which a conversion unit (470) converts an adjustment parameter into a scaling factor, and an editing parameter adjustment unit (430) adjusts an editing parameter using the scaling factor.

[0103] When a user requests brightness adjustment of a shadow area as in the embodiment of FIG. 5, the user adjustment module (460) adjusts the shadow strength S, which is an adjustment parameter related to the brightness adjustment function of the shadow area. sd can be output to the conversion unit (470). At this time, the adjustment parameter S sd The value can be determined based on the user's editing request (e.g. increasing the brightness of the shadow area by 40).

[0104] According to one embodiment of the present disclosure, the transformation function used by the transformation unit (470) may be prepared for each editing parameter. For example, the transformation function M may be prepared for each of the editing parameters μ and k used in the exposure fusion algorithm. μ (·) and M k (·) can be prepared. The conversion unit (470) is a conversion function M μ (·) and M k Using (·), the adjustment parameter S sd can be converted into a scaling factor. Since a conversion function is prepared for each editing parameter, the conversion unit (470) can obtain a scaling factor for each editing parameter. For example, the conversion unit (470) can obtain an adjustment parameter S sd By converting , the scaling factor M corresponding to the editing parameters μ and k respectively μ (S sd ) and M k (S sd ) can be obtained.

[0105] The transformation function can map the adjustment parameter to a scaling factor having a value within a certain range. In other words, the scaling factor obtained by the transformation function can have a value within a certain range. According to one embodiment of the present disclosure, the transformation function M μ (·) and M k (·) are the adjustment parameters S respectively sdcan be converted to a scaling factor with values ​​between 0 and 2.

[0106] The editing parameter adjustment unit (430) adjusts the scaling factor M μ (S sd ) and M k (S sd ) can be used to adjust the values ​​of the editing parameters μ and k. The action of 'adjusting' the editing parameters can also be expressed as 'remapping' the editing parameters. Therefore, the adjusted editing parameters from the editing parameters μ and k are called μ remap and k remap In this case, the editing parameter adjustment unit (430) adjusts the editing parameter μ using the following mathematical expressions 3 and 4. remap and k remap can be obtained.

[0107]

[0108]

[0109] According to one embodiment of the present disclosure, if there is no user edit request, the scaling factor M μ (S sd ) and M k (S sd ) can be set so that all values ​​are 1. Therefore, according to mathematical expressions 3 and 4, if there is no user edit request, the edit parameters μ and k before adjustment and the edit parameter μ after adjustment remap and k remap becomes the same.

[0110] According to mathematical expressions 3 and 4, the editing parameter adjustment unit (430) adjusts the editing parameter by multiplying the editing parameter by a scaling factor. According to one embodiment, the editing parameter adjustment unit (430) may use a method of adding the scaling factor to the editing parameter, or may adjust the editing parameter by performing a linear combination or expanding it into a polynomial.

[0111] (3) Editing of images based on adjusted editing parameters

[0112] When the editing parameter adjustment unit (430) outputs the adjusted editing parameters, the editing execution unit (440) can edit the input image (10) based on the adjusted editing parameters. A specific example is as follows.

[0113] As in the embodiment described above, if the editing parameter generation model (420) infers the editing parameters μ and k, and the user requests brightness adjustment for the shadow area, the editing parameter adjustment unit (430) adjusts the editing parameters μ according to the above mathematical expressions 3 and 4. remap and k remap , and the editing execution unit (440) outputs μ remap and k remap The exposure fusion algorithm can be performed using this. That is, the editing execution unit (440) can reflect the user's intention by editing the image using editing parameters adjusted by the user's editing request instead of the initially inferred editing parameters.

[0114] Referring to Fig. 4, the process of adjusting editing parameters according to a user's editing request and editing a video according to the adjusted editing parameters is summarized as follows.

[0115] - When preprocessing of the input image (10) is completed, the editing parameter generation model (420) can infer the editing parameter ① corresponding to the preprocessed input image (10).

[0116] - When a user checks the output image (20) edited by the editing parameter ① and requests additional editing, the user adjustment module (460) can output the adjustment parameter ② corresponding to the user's editing request.

[0117] - The conversion unit (470) can convert the adjustment parameter ② into a scaling factor ③ using a pre-prepared conversion function and output it.

[0118] - The editing parameter adjustment unit (430) can adjust the editing parameter ① using the scaling factor ③ and then output the adjusted editing parameter ④.

[0119] - The editing unit (440) can edit the input image (10) using the adjusted editing parameter ④.

[0120] 4. If the user-requested editing function does not require a neural network model - if editing parameters are not adjusted.

[0121] Referring to FIG. 4, if the editing function requested by the user corresponds to a second editing function (462) that does not require a neural network model, such as color temperature adjustment or brightness adjustment, the image editing module (220) can edit the image without adjusting the editing parameters.

[0122] When the user adjustment module (460) transmits information about the user's editing request to the additional editing unit (480), the additional editing unit (480) can additionally edit the output image (20) according to the received editing request.

[0123] In this way, the electronic device (100) according to one embodiment of the present disclosure can improve processing speed and use resources efficiently by not using the editing parameters inferred by the editing parameter generation model (420) when a user requests a second editing function (462).

[0124] 5. Learning the editing parameter generation model

[0125] A method for training the editing parameter generation model (420) is described in detail with reference to FIG. 7.

[0126] The training of the editing parameter generation model (420) may be performed by an external device (e.g., a server used for production or management of the electronic device (100)) other than the electronic device (100) on which the editing parameter generation model (420) is mounted. Of course, the electronic device (100) may also train the editing parameter generation model (420). In addition, the training of the editing parameter generation model (420) may already be completed when the electronic device (100) is shipped, or may be additionally trained during the use of the electronic device (100). For the convenience of explanation, it is assumed below that an external server (hereinafter referred to as a “training server”) trains the editing parameter generation model (420). That is, the operations performed by the optimizer (75) in FIG. 7 or the operation of calculating the loss function may be viewed as actually being performed by the training server.

[0127] Before explaining the specific learning method, to help understand the purpose of learning, the reason for using the editing parameter generation model (420) is explained as follows. The electronic device (100) can use the editing parameter generation model (420) to automatically edit the input image (10) into an image that is easy to view without user intervention. To this end, the editing parameter generation model (420) must be trained to infer editing parameters for editing the input image into an image that is easy to view. At this time, the 'image that is easy to view' may generally mean an image that many users find good (e.g., image characteristics determined to be optimal by the designer of the neural network model or image characteristics determined to be optimal through experiments, etc.), i.e., an image that has optimal image characteristics. Even if the user does not set or adjust the editing parameters each time, the editing parameter generation model (420) can automatically infer editing parameters for editing the input image (10) into having optimal image characteristics, and edit and present the input image (10) to the user accordingly. For example, when a user takes a picture using an electronic device (100), the electronic device (100) can automatically edit the pictured picture according to editing parameters inferred by the editing parameter generation model (420), create a preview, and display the preview on the screen of the electronic device (100).

[0128] Referring to Fig. 7, when an input image (10) is input to an editing parameter generation model (420), the editing parameter generation model (420) can infer an editing parameter (71) corresponding to the input image (10). In addition, the editing execution unit (440) can edit the input image (10) according to the editing parameter (71) to generate an output image (20).

[0129] The learning server can acquire measurement characteristic values ​​(73) by measuring one or more types of image characteristics (e.g., brightness, contrast ratio, color temperature, etc.) for the output image (20), and train the editing parameter generation model (420) by comparing the acquired measurement characteristic values ​​(72) with target characteristic values ​​(72). Specifically, the optimizer (75) can update the editing parameter generation model (420) so that the loss value of the loss function (74) representing the difference between the target characteristic values ​​(72) and the measured characteristic values ​​(73) is minimized. At this time, the loss function (74) can be composed of a combination of a mean absolute error (MAE), a mean square error (MSE), and a structural similarity index measure (SSIM).

[0130] The target characteristic value (72) may be a preset value for one or more types of image characteristics. The target characteristic value (72) may correspond to the optimal image characteristic of the image that is good to look at as described above. According to one embodiment, the target characteristic value (72) may be preset to a value desired by a user (administrator) or may be determined through experiments, etc. According to one embodiment, the target characteristic value (72) may include a plurality of values ​​corresponding to each of a plurality of image characteristics. For example, the target characteristic value (72) may include a first characteristic value that quantitatively quantifies the brightness of the image and a second characteristic value that quantitatively quantifies the color temperature of the image.

[0131] So far, embodiments in which the electronic device (100) processes an image signal have been described. However, the processing of the image signal (image development and editing) may also be performed by another device (e.g., a cloud server) external to the electronic device (100). For example, some of the components included in the image signal processing unit (200) of FIG. 1 or the image editing module (220) of FIG. 4 may not be installed in the electronic device (100), but may be implemented by an external device such as a cloud server.

[0132] An embodiment in which a cloud server processes an image signal will be described with reference to FIG. 8. Referring to FIG. 8, a cloud server (800) may be implemented with an image signal processing unit (200) including an image development module (210) and an image editing module (220). According to one embodiment of the present disclosure, when an electronic device (100) transmits raw images acquired through a photographing module (110) to a cloud server (800), the image development module (210) of the cloud server (800) generates a linear RGB image from the raw images, and the image editing module (220) edits the linear RGB image and then stores the edited image in the cloud server (800) or transmits it to the electronic device (100). In addition, according to one embodiment of the present disclosure, the electronic device (100) develops a linear RGB image from raw images, and then transmits the developed linear RGB image to a cloud server (800), and the cloud server (800) edits the image using an image editing module (220), and then stores the edited image in the cloud server (800) or transmits it to the electronic device (100).

[0133] The video editing module (220) can perform editing on a video according to the process described above, and when a user requests editing through an electronic device (100), the video can be edited by adjusting editing parameters according to the user's request.

[0134] In the embodiments described above, the operations performed by the image signal processing unit (200) of FIG. 1 or the image editing module (220) of FIG. 4 may actually be performed by each of the electronic device (100), the cloud server (800), or various other computing devices, or by a combination of two or more computing devices.

[0135] Hereinafter, with reference to FIGS. 9 to 14, a method for editing an image using a neural network model according to embodiments of the present disclosure will be described. The steps included in the flowcharts of FIGS. 9 to 14 are performed by the electronic device (100) or the image editing module (220) of FIGS. 1 to 4, and therefore, even if the contents described above with reference to FIGS. 1 to 8 are omitted below, they can be equally applied to FIGS. 9 to 14.

[0136] Referring to FIG. 9, in step 901, the electronic device (100) may obtain an editing request for an input image from a user. According to one embodiment of the present disclosure, the screen of the electronic device (100) displays an image that has been primarily edited according to editing parameters inferred by a neural network model, and the user may review the displayed image and request additional editing. The user may input the editing request through various input / output interfaces of the electronic device (100).

[0137] The detailed steps included in step 901 are illustrated in FIG. 10. Referring to FIG. 10, in step 1001, the electronic device (100) inputs an input image to a neural network model and obtains editing parameters output by the neural network model. The neural network model may be a model trained to infer editing parameters for editing the input image according to predetermined criteria.

[0138] In step 1002, the electronic device (100) can edit the input image based on editing parameters and output a first edited image. The user can check the first edited image through the screen of the electronic device (100) and decide whether to perform additional editing.

[0139] In step 1003, the electronic device (100) may obtain an editing request from the user requesting additional editing of the primary edited video. For example, the user may request editing of the primary edited video using at least one editing function among multiple editing functions supported by the electronic device (100) through a UI configuration (e.g., a slider) displayed on the screen of the electronic device (100).

[0140] According to one embodiment of the present disclosure, when a user's editing request is input, the electronic device (100) may determine whether to use a neural network model based on the user's editing request. For example, if analysis of the image is required to perform the editing requested by the user, the electronic device (100) may decide to use the neural network model and proceed to step 902. Alternatively, for example, if editing parameters inferred by the neural network model are required to perform the editing function requested by the user, the electronic device (100) may proceed to step 902 to adjust the editing parameters according to the user's editing request. Conversely, if analysis of the image is not required to perform the editing function requested by the user, the electronic device (100) may decide not to use the neural network model and may edit the image without adjusting the editing parameters.

[0141] Returning to FIG. 9, in step 902, the electronic device (100) can adjust the editing parameters generated by the neural network model for the input image according to the user's editing request. Detailed steps included in step 902 are illustrated in FIG. 11.

[0142] Referring to FIG. 11, in step 1101, the electronic device (100) may acquire an adjustment parameter corresponding to the user's editing request. There may be different types of adjustment parameters corresponding to multiple editing functions supported by the electronic device (100). The value of the adjustment parameter may be determined based on the content of the user's editing request (e.g., increasing the brightness of a shadow area by 40).

[0143] In step 1102, the electronic device (100) can adjust the editing parameters based on the adjustment parameters. Detailed steps included in step 1102 are illustrated in FIG. 12.

[0144] Referring to FIG. 12, in step 1201, the electronic device (100) can obtain a scaling factor by converting an adjustment parameter using a conversion function. The conversion function can map the adjustment parameter to a scaling factor having a value within a certain range. In other words, the scaling factor obtained by the conversion function can have a value within a certain range. The conversion function can be prepared for each editing parameter, and thus the electronic device (100) can obtain a scaling factor for each editing parameter.

[0145] At step 1202, the electronic device (100) can adjust the value of the editing parameter using a scaling factor. According to one embodiment of the present disclosure, the electronic device (100) can adjust the editing parameter by multiplying the editing parameter by the scaling factor. Alternatively, according to one embodiment, the electronic device (100) can adjust the editing parameter by adding the scaling factor to the editing parameter, performing linear combination, or polynomial expansion.

[0146] Returning to FIG. 9 , at step 903, the electronic device (100) can edit the input image based on the adjusted editing parameters. The process by which the electronic device (100) performs editing on the image may vary depending on the editing function requested by the user. Below, an embodiment in which the user requests editing, such as a function for adjusting the brightness of a shadow area or a function for adjusting the brightness of a highlight area, and the electronic device (100) performs an exposure fusion algorithm accordingly will be described with reference to FIGS. 13 and 14 .

[0147] The detailed steps included in step 903 are illustrated in FIG. 13. Referring to FIG. 13, in step 1301, the electronic device (100) can obtain an illumination map from an input image. The illumination map is a map expressing the degree of brightness and darkness of the image, and the electronic device (100) can obtain the illumination map using various methods, and a detailed description thereof will be omitted.

[0148] In step 1302, the electronic device (100) can perform exposure fusion on a portion or the entire area of ​​the input image using the adjusted editing parameters and illumination map. Detailed steps included in step 1302 are illustrated in FIG. 14.

[0149] Referring to FIG. 14, in step 1401, the electronic device (100) can acquire a first image with the exposure of the input image changed based on the adjusted editing parameters. For example, the electronic device (100) can acquire a first image brighter than the input image by increasing the exposure of the input image.

[0150] In step 1402, the electronic device (100) can obtain a first weight matrix and a second weight matrix from the illumination map based on the adjusted editing parameters. According to one embodiment of the present disclosure, the electronic device (100) can obtain the first weight matrix according to the mathematical expression 2 described above, and determine the second weight matrix such that the sum of the first weight matrix and the second weight matrix becomes 1.

[0151] In step 1403, the electronic device (100) can perform a weighted sum by applying the first weight matrix and the second weight matrix to the input image and the first image, respectively. In other words, the electronic device (100) can generate an output image by adding the result of multiplying the input image by the first weight matrix and the result of multiplying the first image by the second weight matrix.

[0152] Returning to FIG. 9 again, at step 904, the electronic device (100) can output the edited image through the screen.

[0153] According to the embodiments described above, an electronic device can output an image that reflects the user's intent by further editing an image initially edited using editing parameters inferred by a neural network model according to the user's editing request. Furthermore, an electronic device according to one embodiment of the present disclosure can be expected to improve processing speed and utilize resources efficiently by determining whether to use a neural network model based on the type of editing function requested by the user.

[0154] An image processing method using a neural network model according to one embodiment of the present disclosure may include a step of obtaining an editing request for an input image from a user, a step of adjusting an editing parameter generated by a neural network model for the input image according to the editing request, a step of editing the input image based on the adjusted editing parameter, and a step of outputting the edited image.

[0155] According to one embodiment, the neural network model may be a model trained to generate editing parameters for editing an arbitrary image according to preset criteria when an arbitrary image is input.

[0156] According to one embodiment, the step of obtaining the editing request may include the step of inputting the input image to the neural network model to obtain the editing parameters output by the neural network model, the step of editing the input image based on the editing parameters to output a first edited image, and the step of obtaining the editing request requesting additional editing of the first edited image from the user.

[0157] According to one embodiment, the step of adjusting the editing parameter according to the editing request may include the step of obtaining an adjustment parameter corresponding to the editing request and the step of adjusting the editing parameter based on the adjustment parameter.

[0158] According to one embodiment, the step of adjusting the editing parameter based on the adjustment parameter may adjust the editing parameter using a value obtained by converting the scale of the adjustment parameter.

[0159] According to one embodiment, the step of adjusting the editing parameter based on the adjustment parameter may include the step of obtaining a scaling factor by transforming the adjustment parameter using a pre-prepared transform function, and the step of adjusting the value of the editing parameter using the scaling factor.

[0160] According to one embodiment, the step of editing the input image based on the adjusted editing parameter may include adjusting at least one of brightness, tone, or contrast for a portion or the entire area of ​​the input image according to the adjusted editing parameter.

[0161] According to one embodiment, the step of editing the input image based on the adjusted editing parameters may include the step of obtaining an illumination map from the input image and the step of performing exposure fusion on a portion or the entire area of ​​the input image using the adjusted editing parameters and the illumination map.

[0162] According to one embodiment, the step of performing the exposure fusion may include the step of obtaining a first image in which the exposure of the input image is changed based on the adjusted editing parameter, the step of obtaining a first weight matrix and a second weight matrix from the illumination map based on the adjusted editing parameter, and the step of performing a weighted sum by applying the first weight matrix and the second weight matrix to the input image and the first image, respectively.

[0163] According to one embodiment, the step of adjusting the editing parameter according to the editing request may include the step of determining whether to use the neural network model based on the editing request, and the step of adjusting the editing parameter using an adjustment parameter corresponding to the editing request if it is determined to use the neural network model.

[0164] An electronic device for editing an image using a neural network model according to one embodiment of the present disclosure includes an input / output interface for receiving an input from a user and outputting an image, a memory storing a program or instruction for editing an image, and at least one processor, wherein the at least one processor executes the program or instruction stored in the memory, thereby obtaining an editing request for an input image from the user, adjusting an editing parameter generated by a neural network model for the input image according to the editing request, editing the input image based on the adjusted editing parameter, and then outputting the edited image.

[0165] According to one embodiment, the neural network model may be a model trained to generate editing parameters for editing an arbitrary image according to preset criteria when an arbitrary image is input.

[0166] According to one embodiment, when obtaining the editing request, the electronic device may input the input image into the neural network model to obtain the editing parameters output by the neural network model, edit the input image based on the editing parameters to output a first edited image, and then obtain the editing request requesting additional editing of the first edited image from the user.

[0167] According to one embodiment, when adjusting the editing parameter according to the editing request, the electronic device may obtain an adjustment parameter corresponding to the editing request and then adjust the editing parameter based on the adjustment parameter.

[0168] According to one embodiment, the electronic device can adjust the editing parameter based on the adjustment parameter by obtaining a scaling factor by transforming the adjustment parameter using a pre-prepared transform function, and then adjust the value of the editing parameter using the scaling factor.

[0169] According to one embodiment, the electronic device may adjust at least one of brightness, tone, or contrast for a portion or the entire region of the input image based on the adjusted editing parameter when editing the input image based on the adjusted editing parameter.

[0170] According to one embodiment, the electronic device may, when editing the input image based on the adjusted editing parameters, obtain an illumination map from the input image, and then perform exposure fusion on a portion or the entire area of ​​the input image using the adjusted editing parameters and the illumination map.

[0171] According to one embodiment, when performing the exposure fusion, the electronic device may obtain a first image in which the exposure of the input image is changed based on the adjusted editing parameter, obtain a first weight matrix and a second weight matrix from the illumination map based on the adjusted editing parameter, and then apply the first weight matrix and the second weight matrix to the input image and the first image, respectively, to perform a weighted sum.

[0172] According to one embodiment, the electronic device may determine whether to use the neural network model based on the edit request when adjusting the edit parameter according to the edit request, and if it is determined to use the neural network model, the electronic device may adjust the edit parameter using an adjustment parameter corresponding to the edit request.

[0173] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0174] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0175] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0176] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0177] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In a video editing method using a neural network model, A step of obtaining an editing request for an input image from a user; A step of adjusting editing parameters generated by a neural network model for the input image according to the editing request; a step of editing the input image based on the adjusted editing parameters; and A method comprising the step of outputting the above-mentioned edited image.

2. In paragraph 1, A method characterized in that the neural network model is a model learned to generate editing parameters for editing an arbitrary image according to preset criteria when an arbitrary image is input.

3. In either of paragraphs 1 and 2, The steps of obtaining the above editing request are: A step of inputting the input image into the neural network model and obtaining the editing parameters output by the neural network model; A step of editing the input image based on the above editing parameters and outputting a first edited image; and A method characterized by comprising the step of obtaining an editing request requesting additional editing of the first edited video from the user.

4. In any one of paragraphs 1 to 3, The step of adjusting the above editing parameters according to the above editing request is: A step of obtaining an adjustment parameter corresponding to the above editing request; and A method characterized by comprising a step of adjusting the editing parameter based on the adjustment parameter.

5. In any one of paragraphs 1 to 4, The step of adjusting the editing parameters based on the above adjustment parameters comprises: A step of obtaining a scaling factor by transforming the above adjustment parameter using a pre-prepared transform function; and A method characterized by comprising a step of adjusting the value of the editing parameter using the scaling factor.

6. In any one of paragraphs 1 to 5, The step of editing the input image based on the adjusted editing parameters is: A method characterized in that at least one of brightness, tone, or contrast is adjusted for a part or the entire area of ​​the input image according to the adjusted editing parameters.

7. In any one of paragraphs 1 to 6, The step of editing the input image based on the adjusted editing parameters is: A step of obtaining an illumination map from the above input image; and A method characterized by comprising a step of performing exposure fusion on a part or the entire area of ​​the input image using the adjusted editing parameters and the illumination map.

8. In any one of paragraphs 1 to 7, The step of performing the above exposure fusion is: A step of obtaining a first image in which the exposure of the input image is changed based on the adjusted editing parameters; A step of obtaining a first weight matrix and a second weight matrix from the illumination map based on the adjusted editing parameters; and A method characterized by comprising a step of performing a weighted sum by applying the first weight matrix and the second weight matrix to the input image and the first image, respectively.

9. In any one of paragraphs 1 to 8, The step of adjusting the above editing parameters according to the above editing request is: a step of determining whether to use the neural network model based on the above editing request; and A method characterized in that, when it is decided to use the neural network model, it comprises a step of adjusting the editing parameter using an adjustment parameter corresponding to the editing request.

10. A computer-readable, non-transitory recording medium having recorded thereon a program for performing the method of any one of clauses 1 to 9 on a computer.

11. In an electronic device (100) for editing an image using a neural network model, An input / output interface (130) for receiving input from a user and outputting an image; Memory (150) in which a program or instruction for editing an image is stored; and comprising at least one processor (140), The electronic device (100) executes a program or instruction stored in the memory (150) by the at least one processor (140). Obtain an editing request for an input image from the above user, For the above input image, the editing parameter generated by the neural network model (420) is adjusted according to the editing request, After editing the input image based on the above adjusted editing parameters, An electronic device that outputs the above-mentioned edited video.

12. In paragraph 11, An electronic device characterized in that the neural network model (420) is a model learned to generate editing parameters for editing an arbitrary image according to preset criteria when an arbitrary image is input.

13. In any one of paragraphs 11 and 12, The electronic device (100) obtains the editing request, By inputting the input image into the neural network model (420), the editing parameters output by the neural network model (420) are obtained, After editing the input image based on the above editing parameters and outputting the first edited image, An electronic device characterized in that it obtains an editing request requesting additional editing of the first edited video from the user.

14. In any one of paragraphs 11 to 13, The electronic device (100) adjusts the editing parameters according to the editing request. After obtaining the adjustment parameter corresponding to the above editing request, An electronic device characterized in that the editing parameter is adjusted based on the adjustment parameter.

15. In any one of paragraphs 11 to 14, The electronic device (100) adjusts the editing parameters based on the adjustment parameters, After obtaining the scaling factor by transforming the above adjustment parameter using a pre-prepared transform function, An electronic device characterized in that the value of the editing parameter is adjusted using the scaling factor.

Images