Method and apparatus for training a neural network model for enhancing image details
By training a neural network model for enhancing image details, using the differences between low-quality and high-quality image chunking, combined with Siamese convolutional neural subnetwork and nested convolution kernel, the problem of inconsistent image detail recovery in the existing technology is solved. , realizes details repainting and natural details recovery of blurred or overbright images, and reduces the storage requirements of the model.
Patent Information
- Application Number
- CN202080038929.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-05-27
- Filing Date
- 2020-06-08
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-06-08
AI Technical Summary
Existing image enhancement technologies are difficult to effectively restore fine details in low-quality images, especially when the image is blurred, too bright or otherwise rendered, and often lead to unrelated noise-like details, and within large areas. Lack of consistency.
By training a neural network model for enhancing image details, the model uses the difference between low-quality and high-quality image chunking as a loss function to reduce low-quality output images through Siamese convolutional neural subnetwork and nested convolution kernels. The difference with high-quality input images and the calculation of error backpropagation gradients ensures natural recovery and consistency of image details.
Re-drawing of details of blurred or over-bright images is achieved, restoring natural details, including remote parts of self-similar nature, such as human hair, and reducing the storage capacity requirements of the model.
Smart Images

Figure CN114041161B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to methods and apparatus for training a neural network model for enhancing image details. Background Art
[0002] Currently, most electronic mobile devices (such as smartphones, mobile phones, laptops, tablet computers, small cameras, etc.) include cameras for recording images, and are often used to obtain various photos and videos due to their small size. However, the quality of images obtained using such mobile devices depends on multiple factors, specifically on: the quality of the lens system in the mobile device camera (due to the use of small diameter lenses and low-quality glass, such mobile devices generally have low resolution lenses, which can significantly impair the quality of photos with intricately detailed objects or surfaces), light levels (the lower the light level, the more noise there is in the image), etc.
[0003] There are a variety of methods for enhancing the quality of images captured by a camera, which are based on denoising algorithms, generalized image enhancement algorithms, and sharpening algorithms. When using a denoising algorithm or a denoiser integrated in a camera, the fine details in the image inevitably deteriorate with denoising. That is, the image is smoothed. Generalized image enhancement algorithms aim to process the entire image by improving some or several of its main characteristics (for example, at least one of brightness or contrast). However, when processing an image, the algorithm does not take into account and therefore does not restore the fine details in the image when necessary. Sharpening algorithms also aim to process the entire image, and therefore sharpen the blurred edges in the image, and the algorithm also adds undesirable artifacts to those parts of the image where the edges originally have sufficient sharpness. In addition, the algorithm also cannot enhance the homogeneous parts of the image, in particular, it cannot restore the structural details of uniform surfaces, objects, etc. Therefore, such algorithms are not suitable for image content, but rely on general initial conditions for image enhancement.
[0004] Convolutional neural network models trained on image patch pairs and their downsampled versions are disclosed in the papers: "Fast and efficient image quality enhancement via desubpixel CNN", T. Van Vu et al., KAIST, 2018 (referred to herein as FEQE); "Enhanced deep residual networks for single image super-resolution", B. Lim et al., National University of Seoul, 2017 (referred to herein as EDSR), and "Wide Activation for Efficient and Accurate Image Super-Resolution", J. Yu et al., Adobe Research (referred to herein as WDSR), each of which is incorporated herein by reference in its entirety. FEQE teaches a method of training on a 4x downsampled version of the patch and using residual and inverse subpixel (converting spatial data blocks to depth) blocks to upsample the input image by a factor of 4. The methods disclosed in EDSR and WDSR involve training on 2x downsampled versions of the tiles and using enhanced residual blocks to upsample the input image by a factor of 2. However, this convolutional neural network training method and training samples limit the applicability to image restoration based on the downsampling algorithm used in training. That is, the input image may not be enhanced in areas that are not recognized by the network model as resulting from the specific downsampling algorithm used in training.
[0005] The methods disclosed in the papers “Deep Photo Enhancer: Unpaired Learning for Image Enhancement From Photos With GANs”, Y, S. Chen et al., National Taiwan University, 2018 (referred to herein as DPE) and “DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks”, A. Ignatov et al., ETH Zurich, 2017 (referred to herein as DPED) use a generative adversarial network (GAN) to enhance images, the entire contents of each of which are incorporated herein by reference. DPE discloses a recurrent GAN model trained on unpaired patches of an image. One of the images in the pair must be a high-quality image processed by a skilled photographer. However, this method can only improve global image characteristics, such as color rendering, brightness, display of shadows, etc., but cannot restore fine-grained image details. DPED discloses a GAN model trained on roughly aligned patch pairs of digital single-lens reflex (DSLR)-quality (acquired using a DSLR camera) images and mobile phone quality images of the same scene. However, image enhancement using this method results in an overly bright image. In addition, irrelevant noise-like details appear across the entire image. That is, the method cannot correctly restore or redraw fine details in the image. In addition, all of the above methods lack consistency in large areas. That is, even when one or more parts of a remote object in an image are enhanced, these methods do not correlate these parts to determine whether they belong to the same remote object, thereby uniformly enhancing these parts in a natural manner.
[0006] Therefore, there is a need for a method of converting an image into an image with re-rendered details that have been blurred, over-brightened, or not otherwise rendered in the image, which method will naturally display the re-rendered details in the image, including remote parts with self-similar properties, such as a person's hair.
[0007] The above information is presented as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion has been made, as to whether any of the above may be applicable as prior art with respect to the present disclosure. Summary of the invention
[0008] Technical Solutions
[0009] A neural network model training device for enhancing image details is provided. The device includes a memory and at least one processor, the at least one processor being configured to: obtain low-quality input image blocks and high-quality input image blocks; obtain low-quality output image blocks by inputting the low-quality input image blocks into a first neural network model; obtain high-quality output image blocks by inputting the high-quality input image blocks into a second neural network model; and train the first neural network model based on a loss function, the loss function being set to reduce the difference between the low-quality output image blocks and the high-quality input image blocks and the difference between the high-quality output image blocks and the high-quality input image blocks. The second neural network model is the same as the first neural network model. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:
[0011] Figure 1 shows examples of image pairs with different quality settings captured for training a convolutional neural network according to an embodiment of the present disclosure;
[0012] Figure 2 is a flowchart of a method of training a convolutional neural network to convert an image into an image with redrawn details according to an embodiment of the present disclosure;
[0013] Figure 3 The structure of a nested convolution kernel for training a convolutional neural network according to an embodiment of the present disclosure is shown;
[0014] Figure 4 The structure of a convolution kernel for training a convolutional neural network according to an embodiment of the present disclosure is shown;
[0015] Figure 5A is a detailed schematic diagram of input image processing using a Siamese convolutional neural sub-network according to an embodiment of the present disclosure;
[0016] Figure 5B is a detailed schematic diagram of input image processing using a Siamese convolutional neural sub-network according to an embodiment of the present disclosure;
[0017] Fig. 6A An input image and an image pair obtained by processing the input image according to an embodiment of the present disclosure are shown;
[0018] Figure 6B An input image and an image pair obtained by processing the input image according to an embodiment of the present disclosure are shown;
[0019] Figure 6C An input image and an image pair obtained by processing the input image according to an embodiment of the present disclosure are shown;
[0020] Fig.6D An input image and an image pair obtained by processing the input image according to an embodiment of the present disclosure are shown;
[0021] Figure 7 is a table comparing the results of quality evaluation of images processed by a method of the related art according to an embodiment of the present disclosure;
[0022] Figure 8 is a block diagram schematically showing the configuration of a neural network model training device for enhancing image details according to an embodiment of the present disclosure;
[0023] Fig. 9 is a block diagram schematically showing a configuration of a data set collection device for enhancing image details according to an embodiment of the present disclosure;
[0024] Fig.10 is a flowchart schematically illustrating the process of a method for operating a neural network model training device for enhancing image details according to an embodiment of the present disclosure; and
[0025] Fig.11 is a flow chart schematically illustrating the process of a method for operating a data set collection apparatus for enhancing image details according to an embodiment of the present disclosure.
[0026] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION
[0027] Aspects of the present disclosure will address at least the above problems and / or disadvantages, and will provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a method and apparatus for training a neural network model for enhancing image details.
[0028] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.
[0029] According to one aspect of the present disclosure, a neural network model training device for enhancing image details is provided. The neural network model training device includes: a memory configured to store one or more instructions; and at least one processor, the at least one processor configured to execute one or more instructions stored in the memory to obtain low-quality input image blocks and high-quality input image blocks, obtain low-quality output image blocks by inputting the low-quality input image blocks into a first neural network model, obtain high-quality output image blocks by inputting the high-quality input image blocks into a second neural network model, and train the first neural network model based on a loss function, the loss function is set to reduce the difference between the low-quality output image blocks and the high-quality input image blocks and the difference between the high-quality output image blocks and the high-quality input image blocks, wherein the second neural network model is the same as the first neural network model.
[0030] In an embodiment of the present disclosure, the loss function may include as terms the difference between a low-quality output image block and a high-quality input image block, and the product of the difference between the low-quality output image block and the low-quality input image block and the difference between the high-quality output image block and the high-quality input image block.
[0031] In an embodiment of the present disclosure, when at least one processor trains the first neural network model, the gradient of the loss function of error back propagation may not include the partial differential of the difference between the low-quality output image block and the low-quality input image block.
[0032] In an embodiment of the present disclosure, the loss function may include the L1 difference between the low-quality output image block and the high-quality input image block, the product of the L1 difference between the low-quality output image block and the low-quality input image block and the L1 difference between the high-quality output image block and the high-quality input image block, the visual geometry group (VGG) difference between the low-quality output image block and the high-quality input image block, and the product of the VGG difference between the low-quality output image block and the low-quality input image block and the VGG difference between the high-quality output image block and the high-quality input image block.
[0033] In an embodiment of the present disclosure, the first neural network model may use a nested convolution kernel.
[0034] In an embodiment of the present disclosure, in the encoding part of the first neural network model, for convolution kernels other than the first convolution kernel, each convolution kernel may include each convolution kernel smaller than itself and be included in each convolution kernel larger than itself.
[0035] In an embodiment of the present disclosure, the low-quality input image blocks and the high-quality input image blocks may be captured by the same image capturing device based on different capturing parameters.
[0036] In an embodiment of the present disclosure, low-quality input image blocks can be captured with higher sensitivity and shorter exposure time than high-quality input image blocks, and high-quality input image blocks can be captured with lower sensitivity and longer exposure time than low-quality input image blocks.
[0037] According to another aspect of the present disclosure, a data set collection device for enhancing image details is provided. The data set collection device includes: an image capture module; a memory configured to store one or more instructions; and at least one processor, the at least one processor being configured to: execute one or more instructions stored in the memory to capture a first quality image through the image capture module based on a first capture parameter; when the first quality image is captured, automatically capture a second quality image through the image capture module based on a second capture parameter different from the first capture parameter; and send the first quality image and the second quality image to a neural network model training device for enhancing image details.
[0038] In an embodiment of the present disclosure, the second capture parameters may include a lower sensitivity than that of the first capture parameters and a longer exposure time than that of the first capture parameters.
[0039] In an embodiment of the present disclosure, the second capture parameter may be set such that the brightness of the second quality image is equal to the brightness of the first quality image.
[0040] In an embodiment of the present disclosure, at least one processor may also be configured to: execute one or more instructions to receive a neural network model from a neural network model training device for enhancing image details; capture an image for detail enhancement; and capture an image with redrawn details by inputting the image for detail enhancement into the neural network model.
[0041] In an embodiment of the present disclosure, the neural network model may use a nested convolution kernel.
[0042] According to another aspect of the present disclosure, a method for training a neural network model for enhancing image details is provided. The method includes: obtaining low-quality input image blocks and high-quality input image blocks; obtaining low-quality output image blocks by inputting the low-quality input image blocks into a first neural network model; obtaining high-quality output image blocks by inputting the high-quality input image blocks into a second neural network model; and training the first neural network model based on a loss function, the loss function being set to reduce the difference between the low-quality output image blocks and the high-quality input image blocks and the difference between the high-quality output image blocks and the high-quality input image blocks, wherein the second neural network model is the same as the first neural network model.
[0043] According to another aspect of the present disclosure, a method for operating a data set collection device for enhancing image details is provided. The method includes: capturing a first quality image based on a first capture parameter; when the first quality image is captured, automatically capturing a second quality image based on a second capture parameter different from the first capture parameter; and sending the first quality image and the second quality image to a neural network model training device for enhancing image details.
[0044] According to another aspect of the present disclosure, there is provided a method for generating a total loss function for training a convolutional neural network to convert an image into an image with redrawn details. The method comprises:
[0045] - acquiring an image pair, including a low quality input patch (PL) and a high quality input patch (PH), the low quality input patch (PL) being an image captured at a high International Standards Organization (ISO) value and a short exposure time, and the high quality input patch (PH) being an image captured at a lower ISO value and a longer exposure time,
[0046] - Divide the input into chunks (P L ,P H ) is fed into the input of each of the Siamese convolutional neural sub-networks and processes the input block (P L ,P H ) to generate output image blocks (P' L ,P' H ), where the encoding part of each Siamese convolutional neural subnetwork uses nested convolution kernels,
[0047] - Calculate the regression difference D(P' L ,P H ),
[0048] - Calculate the modulated retention difference D(P' L ,P L)*D(P' H ,P H ),
[0049] The total loss function for training the convolutional neural network is generated by summing the regression difference and the modulated retention difference as D(P' L ,P H )+D(P' L ,P L )*D(P' H ,P H ),as well as
[0050] -Train a convolutional neural network based on the generated total loss function.
[0051] According to an embodiment of the present disclosure, the loss function gradient for the weight θ of the Siamese convolutional neural subnetwork can be calculated as
[0052]
[0053] According to an embodiment of the present disclosure, the ISO value and exposure time of the input blocks may be set so that the input blocks have the same total brightness.
[0054] According to another aspect of the present disclosure, a system for converting an image into an image with redrawn details is provided. The system comprises:
[0055] - an image capture device configured to capture an image,
[0056] - a memory configured to store a captured image and a convolutional neural network model for converting the image into an image with redrawn details, the convolutional neural network model being trained based on a total loss function generated according to a method for generating a total loss function for training a convolutional neural network to convert a defined image into an image with redrawn details, and
[0057] - A processor configured to process the captured image using the stored convolutional neural network model to convert the image into an image with redrawn details.
[0058] According to another aspect of the present disclosure, there is provided a program stored in a non-transitory computer-readable recording medium, which, when executed by a computer, performs a method.
[0059] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable recording medium having recorded thereon a program for executing a method by a computer.
[0060] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure.
[0061] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. The following description includes various specific details to assist in this understanding, but these details are to be regarded as merely exemplary. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.
[0062] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it is clear to those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only, and not for the purpose of limiting the present disclosure as defined by the attached claims and their equivalents.
[0063] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.
[0064] The term "coupling" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not those elements are in physical contact with each other. The terms "send", "receive" and "communication" and their derivatives include both direct and indirect communication. The terms "include" and "comprises" and their derivatives refer to inclusion and not limitation. The term "or" is inclusive, referring to and / or. The phrase "associated with..." and its derivatives refer to including, being included, interconnected with..., including, being included, being connected to or with..., being connected to or with..., being coupled to or with..., being coupled with..., communicating with..., collaborating with..., interweaving, juxtaposing, being close to, being bound to or with..., having, having... properties, having a relationship to... or having a relationship with..., etc. The term "controller" refers to any device, system or part thereof that controls at least one operation. Such a controller can be implemented in hardware or a combination of hardware and software and / or firmware. The functions associated with any particular controller can be centralized or distributed, whether local or remote. When used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items can be used, and only one of the items in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C.
[0065] In the accompanying drawings, elements having substantially the same functional configuration are assigned the same reference numerals and symbols as much as possible, even when these elements are shown in different figures. For ease of description, when necessary, the apparatus and method will be described together. The corresponding operations of the present disclosure do not necessarily need to be performed in the order described, and can be performed in parallel, optionally or individually.
[0066] The present disclosure provides the use of a convolutional neural network to convert an input image into an image with redrawn details, that is, to construct an image with a higher level of detail than an input image in which the details are blurred or smoothed. In other words, the convolutional neural network can be trained to convert low-quality image blocks into high-quality image blocks without losing the information content of the blocks. In order to train the convolutional neural network, an image pair, each including a low-quality block and a high-quality block, can be fed to the input of the convolutional neural network. Each of the blocks can be processed by a corresponding Siamese neural convolutional subnetwork, and the total loss function can include the difference between the input block and the output block. In addition, the Siamese neural convolutional subnetwork can use nested convolution kernels to take into account the self-similarity properties in the image and reduce overfitting, which ensures the use of a small number of weights for the convolutional neural network, so that the storage capacity required to store the model of the present disclosure is reduced.
[0067] The image pairs used to train the neural network can be captured using any suitable device including a camera and configured to capture images with different quality settings (e.g., a mobile device having an application installed thereon for capturing such images). Each input tile pair can be matched. That is, the two tiles of the same pair show the same scene at the same time or at nearly the same time. However, the tiles may be captured with different settings (e.g., exposure, ISO). The image tile pair P L and P H (Low quality block P L and high-quality blocks P H ) can be processed separately by the corresponding Siamese convolutional neural sub-network to produce the output block P' L and P' H . Output block P' L and input block P H Can be used to form the regression difference D(P' L ,P H ), which is the component responsible for the enhancement of low-quality blocks. Output block P' L and P' H And the input block P L and P H Can be used to form the modulated retention difference D(P' L ,PL)*D(P' H ,PH ), which is the component responsible for maintaining the quality of the high-quality patches. These two differences can be added together to form the total loss function for training the entire convolutional neural network. Because the error backpropagation method applied does not take into account D(P' L ,P L )(this difference is not minimized), so the loss function gradient for the weight θ of the Siamese neural convolution subnetwork can be calculated as:
[0068]
[0069] The above description will be described in more detail below.
[0070] Figure 1 An example of an image pair captured using a mobile device on which an application for capturing images with different quality settings is installed according to an embodiment of the present disclosure is shown.
[0071] refer to Figure 1 , the image shows the top of a person's head, including the person's hair. The application can set the International Organization for Standardization (ISO) parameters required when capturing each image, and the exposure and aperture parameters can be automatically adjusted by the application so that the overall brightness of the captured image pairs is the same. Optionally, these parameters can be manually set at any suitable device configured to capture images. According to an embodiment of the present disclosure, a low-quality image refers to an image captured at a high ISO and a short exposure time, and a high-quality image refers to an image captured at a low ISO and a long exposure time, compared to the same parameters applied to capture a low-quality image.
[0072] Specifically, Figure 1 An image pair is shown: a low quality image with an ISO of 1250 and an exposure time of 1 / 1563, and a high quality image with an ISO of 40 and an exposure time of 1 / 50. Figure 1 It is clearly shown that the human hair structure in the second image is rendered much better than in the first image.
[0073] According to an embodiment of the present disclosure, an end user may receive a pre-trained convolutional neural network model to convert an image into an image with redrawn details (e.g., forming a high-quality image based on a low-quality image). According to an alternative embodiment of the present disclosure, a method is provided, in which an end user independently collects a database for training a convolutional neural network (i.e., a series of training image pairs are obtained according to user interests), the database collected by the end user is loaded into a cloud storage, a convolutional neural network is trained on the database collected by the end user, and the convolutional neural network model trained on the database is provided to the end user. Therefore, according to an alternative embodiment of the present disclosure, the convolutional neural network model can be applied to user data.
[0074] Reference Figure 2 Describe in more detail the method of training a convolutional neural network to convert an image into an image with repainted details.
[0075] Figure 2 is a flowchart of a method of training a convolutional neural network to convert an image into an image with redrawn details according to an embodiment of the present disclosure.
[0076] refer to Figure 2 , in order to train the convolutional neural network, the image pairs can be obtained as described above, i.e., the low-quality blocks P L and high-quality blocks P H The image data may be processed separately by Siamese convolutional neural subnetworks 210 and 220 to form output blocks P' L and P' H , and the Siamese convolutional neural subnetwork has the same weights for uniformly processing the two images of the pair. As a result of the processing by the Siamese convolutional neural subnetworks 210 and 220, the details of the input patch are redrawn. During the processing, the Siamese convolutional neural subnetworks 210 and 220 increase the input patch P L level of detail to form the output block P' L , and in generating the output block P' H The input block P is basically unchanged H .
[0077] Next, the corresponding difference is calculated, which is used to calculate the total loss function for training the convolutional neural network to convert the image into an image with redrawn details. The regression difference can be based on the output block P' by any method of the related art. L and input block P H is calculated as D(P' L ,P H ), for example, is calculated as the absolute loss value (also known as the L1 norm ) or quadratic loss function (also known as the L2 norm ). Respectively, p i Represents block P H elements, and p' i Represents block P' L In addition, the difference D can be calculated as the perceptual difference (i.e., the function obtained by training the model to recognize image categories constructed by the Visual Geometry Group (VGG). In addition to the regression difference, the modulated retention difference is based on the output block P' L and P' H And the input block PL and P H Both are generated as D(P' L ,P L )*D(P' H ,P H ). The difference D can be similarly calculated by any of the methods of the above-mentioned related art.
[0078] The differences (i.e., regression difference and modulated retention difference) are added together to obtain the total loss function D(P') L ,P H )+D(P' L ,P L )*D(P' H ,P H ), which is used to train the entire convolutional neural network. Specifically, the total loss function can be calculated as:
[0079]
[0080] in, is the layer 4 transform of the VGG-19 neural network (see Simonyan K., Zisserman A., “Verydeep convolutional networks for large-scale image recognition”, 2014, which is incorporated herein by reference in its entirety), l is the image pyramid level index, N l is the number of pixels on level l of the image pyramid, N f is the number of VGG layer 4 features, and w is an empirically chosen positive parameter. i and j are the spatial coordinates of each tensor, and k is the color plane of the pixel in the former and the VGG image index in the latter. Specifically, according to an embodiment, based on multiple experiments, the parameter w=0.001. The transformation can be set as follows: the tiled image of dimension WxHx3 (W represents width, H represents height, and 3 represents three RGB color channels) is fed to the input of the trained VGG-19 neural network, and the transformation output is the output of the 4th layer of VGG-19 (refer to Table 1 in the above paper, the rightmost column is configuration E, and the output of the top layer is marked in bold as conv3-512), which is a tensor of dimension W / 8xH / 8x512.
[0081] Furthermore, as mentioned above, since the error back-propagation method applied does not take into account D(P' L ,P L ), so for processing block P LThe loss function gradient of the weight θ of the Siamese neural convolutional subnetwork is calculated as:
[0082]
[0083] Here, θ is the weight of the Siamese convolutional neural sub-network.
[0084] Specifically, according to the rules of differentiation, the gradient of the loss function for the weights of the sub-network should be as follows:
[0085] However, according to the present disclosure, as described above, the differential D(P') is not considered. L ,P L ). Therefore, the error propagates from the difference D(P' L ,P L ) is "blocked". This provides selective enhancement of low-quality image regions by restoring or redrawing the details of the image, even in the absence of a matching paired high-quality region. In other words, by considering the modulated preservation difference, the convolutional neural network can be trained to consider the most "valuable" regions in the high-quality image based on how much change similar regions in the low-quality image have undergone at the output processed by the Siamese convolutional neural sub-network (when the low-quality image region has not undergone a change, the similar region in the high-quality image is considered to be less valuable in training, and vice versa). Therefore, the convolutional neural network does not use the high-quality image as a reference image and learns a method to identify and enhance low-quality regions in that image.
[0086] Figure 3 The structure of a nested convolution kernel used in the encoding part of a Siamese convolutional neural subnetwork to train a convolutional neural network to convert an image into an image with redrawn details according to an embodiment of the present disclosure is shown.
[0087] In the model of the convolutional neural network of the related art, the convolution kernels used in training are updated independently of each other, which usually leads to a large model capacity (multiple network weights). This is the main reason for overfitting of the convolutional neural network, which in turn impairs the efficiency of using the convolutional neural network model when processing unfamiliar databases. The embodiments of the present disclosure assume the use of a single pool of convolution kernels, i.e., nested convolution kernels, which allows the receptive field to be increased without increasing the amount of weights used.
[0088] The single pool of convolution kernel weights has the maximum convolution dimension used in training the convolutional neural network. Each convolution (except the first convolution, which acts directly on the raw red, green, blue (RGB) data of the input patch) is assigned a corresponding subset of kernel weights. Figure 3 , shows an example in which three consecutive KxK convolutions successively transform a feature map with dimensions HxWxF into feature maps with dimensions HxWx2F, HxWx4F, and HxWx8F. The largest convolution in this example uses a kernel with dimensions KxKx8Fx4F (for clarity, KxK is merged into K in the figure). 2 ). According to an embodiment of the present disclosure, in this example, the largest convolution is distributed to three convolutions: the first convolution receives a KxKx2FxF sub-tensor from a common single pool, the second convolution receives a KxKx4Fx2F sub-tensor, and the third convolution receives a full KxKx8Fx4F tensor. During training, the convolution kernel weights are updated according to such a distribution. That is, updates to the weight pool always take into account that a portion of the larger convolution weights is always consistent with the smaller convolution weights.
[0089] This application of nested convolutions can substantially expand the receptive field of the convolutional neural network without increasing the amount of convolutional neural network weights, providing a greater visual context for image processing without the risk of overfitting. In addition, this approach provides consistency of fine details in the image, including long-range details (e.g., human hair) that possess self-similar properties, because all levels of convolution have the same small convolution weights, which seek and exploit fractal image self-similarity. Therefore, the convolutional neural network model of the present disclosure makes it possible to apply the same weights to feature maps with different dimensions, thereby identifying self-similar features of objects present in the processed image.
[0090] Figure 4 The structure of a convolution kernel for training a convolutional neural network according to an embodiment of the present disclosure is shown.
[0091] refer to Figure 4 In an alternative embodiment of the present disclosure, a single pool of weights may be used, where the convolution kernels are not nested but distributed over the largest convolution. Even in an alternative embodiment of the present disclosure, the amount of convolutional neural network weights is reduced. However, this approach cannot ensure the consistency of fine details in the image, i.e., the integrity of details in the resulting high-quality image is deteriorated, compared to the image produced according to the previous embodiment of the present disclosure.
[0092] Figure 5A is a detailed diagram of processing using a Siamese convolutional neural sub-network according to an embodiment of the present disclosure.
[0093] The subnetwork can have a U-Net architecture (a compression encoder followed by a decoder). Figure 5A, each of the blocks (low quality blocks and high quality blocks) independently reaches the input of its own Siamese convolutional neural sub-network. The output of the Siamese convolutional neural sub-network is a 2-level pyramid of output blocks. Optionally, a pyramid of images with a large number of levels can be generated, but it has been found experimentally that larger image pyramids may not significantly enhance quality. The compression encoder of the sub-network uses convolutions with nested kernels as described above (except for the first convolution applied). As shown in Figure 5A As clearly shown in the figure, the input block (P L or P H ) has dimensions WxHx3, and this block is processed within the subnetwork (3x3 convolution and exponential linear unit (ELU) activation function) to produce an array with depth 32 (dimensions WxHx32). All convolutions processed later within the subnetwork can be nested convolutions as described above. Specifically, further processing in the subnetwork produces an array with similar dimensions (applying two 3x3 convolutions and ELU activation functions).
[0094] Then, 2x2 downsampling (pooling), 3x3 convolutions, and ELU activation functions are applied to generate an array in which the width and height are halved and the depth (number of feature maps) becomes 64 (W / 2xH / 2x64). Then, three 3x3 convolutions and ELU activation functions are applied to generate an array in which the width, height, and depth remain unchanged. Then, 2x2 downsampling, 3x3 convolutions, and ELU activation functions are applied to generate an array in which the width and height are halved again and the depth becomes 128 (W / 4xH / 4x128). Then, three 3x3 convolutions and ELU activation functions are applied again to generate an array in which the width, height, and depth remain unchanged. Then, 2x2 downsampling, 3x3 convolutions, and ELU activation functions are applied again to generate an array in which the width and height are halved again and the depth becomes 256 (W / 8xH / 8x256). Then, four 3x3 convolutions and ELU activation functions are applied to generate an array in which the width, height, and depth remain unchanged. Then, 2x2 down sampling, 3x3 convolution and ELU activation function are applied again, to produce an array in which width and height are halved again and depth becomes 512 (W / 16xH / 16x512). Then, 3x3 convolution and ELU activation function are applied again four times to produce an array in which width, height and depth remain unchanged. Then, 2x2 down sampling, 3x3 convolution and ELU activation function are applied again, to produce an array in which width and height are halved again and depth becomes 768 (W / 32xH / 32x768). Then, 3x3 convolution and ELU activation function are applied again four times to produce an array in which width, height and depth remain unchanged. After 2x2 down sampling, 3x3 convolution and ELU activation function are applied, an array in which width and height are halved again and depth remains unchanged (W / 64xH / 64x768) is obtained. Then, 3x3 convolution and ELU activation function are applied again four times to produce an array in which width, height and depth remain unchanged, and the processing in the coding part of the subnetwork terminates. Therefore, when processed in the compression encoding part of the subnetwork, the image height and width are reduced (to a total of 1 / 64 of the original), and the number of feature maps is increased to 768.
[0095] Then, the image size is increased again in the decoder. Figure 5A As can be clearly seen in FIG, the Siamese convolutional neural subnetwork of the present disclosure uses a skip connection (transfer link) that connects subnetwork layers that are not adjacent to each other to preserve the details of the original input block by sending the corresponding array from the compression encoding part to the decoding part and concatenating with the array with matching height and width. For example, the first residual connection (in Figure 5A The output of the second layer (i.e., an array with dimensions WxHx32) is sent to the decoding part and concatenated with the output of the next layer with dimensions WxHx16. Figure 5A In , the upsampling operation used in the decoding part is labeled as "Upsampling". Specifically, when the processing in the decoding part of the subnetwork applies 2x2 upsampling, 3x3 convolution, and ELU activation function, the result is an array with dimensions W / 32xH / 32x512. In addition, as Figure 5A As shown, an array with matching dimensions is sent to the part by a transmission operation, and the array is concatenated with the part to obtain an array with dimensions W / 32xH / 32x1024. Then, 3x3 convolution and ELU activation functions are applied to produce a W / 32xH / 32x512 array. Then, similar to the encoding part, 2x2 upsampling, 3x3 convolution and ELU activation functions are applied alternately, and 3x3 convolution and ELU activation functions are applied to the transmission of the array with matching dimensions. After obtaining a W / 2xH / 2x32 array as a result of the 3x3 convolution and ELU activation function, instead of applying 2x2 upsampling, 3x3 convolution and ELU activation functions, 3x3 convolution and tanh activation functions are applied to produce a W / 2xH / 2x3 output block (the second level of the output block pyramid). 2x2 upsampling is applied to this block, producing a WxHx3 array, which is cascaded with the WxHx32 array sent from the encoding part and the WxHx16 output of the next convolution of the decoding part. Then, a 3x3 convolution and ELU activation function are applied to the cascaded result to produce a WxHx16 array, to which a 3x3 convolution and tanh activation function are applied to produce a WxHx3 output block (the first level of the pyramid of output blocks). Due to the use of nested convolution kernels, the Siamese convolutional neural subnetwork includes a relatively small number of trainable weights (17.5M). The same architecture without nested convolution kernels includes 77.4M trainable weights (4.4 times more), but has the same receptive field.
[0096] Figure 5B is a detailed diagram of processing using a Siamese convolutional neural sub-network according to another embodiment of the present disclosure.
[0097] refer to Figure 5B , ⊕ represents element-by-element summation, and σ represents the sigmoid activation function. Figure 5A Those parts are similar, so the detailed description thereof will be omitted.
[0098] Fig. 6A , Figure 6B , Figure 6C and Fig.6D An input image and an image pair generated by processing the input image according to various embodiments of the present disclosure are shown.
[0099] refer to Fig. 6A , Figure 6B , Figure 6C and Fig.6D , shows the result of converting an image into an image with redrawn details by using a convolutional neural network trained according to one or more methods of the present disclosure. Fig. 6A , Figure 6B , Figure 6C and Fig.6D In each of the examples, an input image (e.g., a low-quality image) is shown on the left, and a result based on a convolutional neural network trained using a method according to the present disclosure (specifically, a high-quality image) is shown on the right. Fig. 6A , Figure 6B , Figure 6C and Fig.6D , the rectangular areas highlight areas that clearly show the enhancements resulting from using the convolutional neural network of the present disclosure, specifically, more clearly drawn hair structure, more clearly drawn clothing / fabric structure, clearer skin structure in darker images, etc., and all of these clearer details are drawn in a natural-looking manner.
[0100] A system for converting an image into an image with redrawn details according to an embodiment of the present disclosure may include an image capture device configured to capture an image to be processed. The image capture device may also be configured to capture images with different quality settings to collect a personal training set, as described above. The system may also include a memory configured to store a convolutional neural network model trained according to the present disclosure for converting an image into an image with redrawn details, as well as captured images and personal training sets collected using the image capture device. In addition, the system may include a processing device configured to process the captured image using the stored convolutional neural network model for converting an image into an image with redrawn details. In addition, the memory may be configured to store the result of processing the captured image.
[0101] The quality of images generated according to the present disclosure may be evaluated using the Cumulative Probability of Blur Detection (CPBD) and the Weighted Average Deep Image Quality Measure for No-Reference (WaDIQaM-NR).
[0102] Figure 7is a table comparing the results of the method trained according to the embodiments of the present disclosure and the method of the related art described in the background technology section, as well as the widely used "Let's Enhance" (i.e., an algorithm for adding texture and an algorithm for enhancing resolution) and "Photoshop" (i.e., an unsharp mask filter).
[0103] refer to Figure 7 , because the CPBD image quality assessment metric only evaluates image sharpness / blurriness, the higher the score, the better the image quality. WaDIQaM-NR is a trained non-reference image quality metric that evaluates the entire image. As in Figure 7 As can be clearly seen in the table, the average quality score of the images produced according to the present disclosure obtained by the CPBD metric is 0.65, which is comparable to similar scores obtained by the methods disclosed in EDSR and WDSR. The average score obtained using the WaDIQaM-NR metric is 32.1, which is the best result among all the methods analyzed.
[0104] Figure 8 It is a block diagram schematically showing the configuration of a neural network model training device for enhancing image details according to an embodiment of the present disclosure.
[0105] refer to Figure 8 According to an embodiment of the present disclosure, the neural network model training device 800 may include a memory 810 configured to store one or more instructions, and a processor 820 configured to execute one or more instructions stored in the memory 810. The memory 810 may be a single memory or multiple memories. The processor 820 may be a single processor or multiple processors. The instructions executed by the processor 820 and referred to Fig.10 The operation of the neural network model training device 800 described again is the same as the method for training a neural network model described above. The neural network model training device 800 may be a mobile device.
[0106] Fig. 9 is a block diagram schematically showing the configuration of a data set collection device for enhancing image details according to an embodiment of the present disclosure.
[0107] refer to Fig. 9 According to an embodiment of the present disclosure, the data set collection device 900 may include a memory 910 configured to store one or more instructions, a processor 920 configured to execute one or more instructions stored in the memory 910, and an image capture module 930. The memory 910 may be a single memory or multiple memories. The processor 920 may be a single processor or multiple processors. The processing executed by the processor 920 and referred to Fig.11The operation of the data set collection device 900 described again is the same as the data set collection method described above. The data set collection device 900 can be a mobile device. The data set collection device 900 and the neural network model training device 800 can be integrated into one device.
[0108] Fig.10 It is a flowchart schematically illustrating the process of a method for operating a neural network model training device for enhancing image details according to an embodiment of the present disclosure.
[0109] refer to Fig.10 In operation S1010, the processor 820 of the neural network model training apparatus 800 may obtain a low-quality input image block P L and high-quality input image blocks P H The low-quality input image block P can be obtained from the data set collection device 900 L and high-quality input image blocks P H In operation S1020, the processor 820 may divide the low-quality input image into blocks P L Input to the first neural network model to obtain low-quality output image blocks P' L In operation S1030, the processor 820 may divide the high-quality input image into blocks P H Input to the second neural network model to obtain high-quality output image blocks P' H As described above, operations S1020 and S1030 need not be performed in sequence. In operation S1040, the processor 820 may be configured to reduce the low-quality output image block P'. L and high-quality input image blocks P H The difference between D(P' L ,P H ) and high-quality output image blocks P' H and high-quality input image blocks P H The difference between D(P' H ,P H ) loss function to train the first neural network model.
[0110] The first neural network model and the second neural network model are the same neural network model. The first neural network model and the second neural network model being the same may mean that the structures and weights of the two network models are the same. The first neural network model and the second neural network model being the same may mean that the two neural network models are a single neural network model. The first neural network model and the second neural network model may each be a Siamese convolutional neural sub-network. The first neural network model and the second neural network model may be stored in the memory 810. Because the two models are the same, the two models may not be stored redundantly and may therefore be stored once.
[0111] The loss function may have as a term the difference D(P') between a low quality output image patch and a high quality input image patch. L ,P H ) and the product D(P') of the difference between the low quality output image block and the low quality input image block and the difference between the high quality output image block and the high quality input image block L ,P L )*D(P' H ,P H ). In this case, each term can be multiplied by a constant of proportionality.
[0112] The gradient of the loss function for error back propagation may not include the difference D(P') between the low-quality output image patch and the low-quality input image patch. L ,P L The gradient of the loss function refers to the gradient of each weight, while the partial derivative refers to the partial derivative of each weight. The gradient of the loss function can include the difference D(P') between the low-quality output image block and the high-quality input image block. L ,P H ) and the difference D(P') between the high-quality output image block and the high-quality input image block H ,P H ) is the partial derivative of .
[0113] The loss function may include the L1 difference between the low-quality output image block and the high-quality input image block, the product of the L1 difference between the low-quality output image block and the low-quality input image block and the L1 difference between the high-quality output image block and the high-quality input image block, the VGG difference between the low-quality output image block and the high-quality input image block, and the product of the VGG difference between the low-quality output image block and the low-quality input image block and the VGG difference between the high-quality output image block and the high-quality input image block. Each of these can be each term of the loss function, and each term can be multiplied by a proportional constant. For example, the product of the VGG difference between the low-quality output image block and the low-quality input image block and the VGG difference between the high-quality output image block and the high-quality input image block can be multiplied by a constant w=0.001.
[0114] The first neural network model may use nested convolution kernels. The first neural network model may have a U-Net structure. In the encoding portion of the first neural network model, for convolution kernels other than the first convolution kernel, each convolution kernel may include each convolution kernel smaller than itself and be included in each convolution kernel larger than itself.
[0115] The low-quality input image blocks and the high-quality input image blocks can be captured by the same image capture device based on different quality settings (i.e., different capture parameters). The low-quality input image blocks can be captured with high sensitivity and short exposure time, and the high-quality input image blocks can be captured with low sensitivity and long exposure time. In this case, the capture parameters can be set so that the brightness of the low-quality image is equal to the brightness of the high-quality image.
[0116] Fig.11 is a flow chart schematically illustrating a method of operating a data set collection apparatus for enhancing image details according to an embodiment of the present disclosure.
[0117] refer to Fig.11 In operation S1110, the processor 920 of the data set collection device 900 may capture a first quality image through the image capture module 930 based on the first capture parameter. In operation S1120, when the first quality image is captured, the processor 920 of the data set collection device 900 may automatically capture a second quality image through the image capture module 930 based on the second capture parameter. In this case, the first capture parameter and the second capture parameter are different parameters. For example, the second capture parameter may include a lower sensitivity than the first capture parameter sensitivity and an exposure time longer than the first capture parameter exposure time. The first quality image and the second quality image may be captured sequentially at short time intervals. The first quality image and the second quality image may be captured simultaneously through adjacent lenses.
[0118] In operation S1130, the processor 920 may transmit the first quality image and the second quality image to the neural network model training device 800 for enhancing image details. When the data set collection device 900 and the neural network model training device 800 are a single device, this may be data transmission within the device.
[0119] The data set collection device 900 may convert the captured image into an image with redrawn details. That is, the processor 920 of the data set collection device 900 may receive the neural network model from the neural network model training device 800, capture an image for enhancing details, and obtain the image with redrawn details by inputting the captured image into the neural network model. The received neural network model may be the weight of the received neural network model. The received neural network model may be the structure and weight of the received neural network model. The received neural network model may be stored in the memory 910.
[0120] The present disclosure can be applied to multiple fields requiring image processing, as follows:
[0121] - Use a special application to process the photo. The end user can independently collect a personal training database for appropriate image processing. (Thus, when the enhancement of a portrait photo is desired, the user can collect an array of training data including multiple portraits, and when the enhancement of an animal or plant photo is desired, the user can independently collect an array of training data including multiple animals or landscapes, respectively. In addition, any known array of collected training data can also be used);
[0122] - Photo beautification. For example, by collecting an array of training data that includes beautiful faces (e.g., with makeup, without "blemishes", etc.), a convolutional neural network can be trained to apply "beauty" effects to facial images;
[0123] - Security camera image enhancement. For example, enhancement of images of car numbers or criminals’ faces, etc.;
[0124] -Virtual reality image enhancement;
[0125] -Text image enhancement for OCR; and
[0126] - Similar object detection in images.
[0127] Embodiments of the present disclosure may be implemented as computer executable code on a computer-readable recording medium. Computer-readable recording media include any recording media, such as magnetic media, optical media, read-only memory (ROM), random access memory (RAM), etc. The computer-readable recording medium may be provided in the form of a non-transitory storage medium. "Non-transitory storage medium" refers to a tangible device and refers to that it does not include a signal (e.g., an electromagnetic wave). The term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.
[0128] According to an embodiment of the present disclosure, the method according to various embodiments of the present disclosure may be provided in a state included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in a state stored in a computer-readable recording medium, or may be available through an application store (e.g., Play Store TM ) or two user devices (e.g., smart phones) directly or online (e.g., downloading or uploading). In the case of online distribution, at least a part of the computer program product (e.g., downloadable application) may be at least temporarily stored in a computer-readable recording medium (such as a memory of a manufacturer's server, an application store's server, or a relay server), or may be temporarily generated.
[0129] According to an embodiment of the present disclosure, a method and apparatus for training a neural network model for enhancing image details can be provided. While retaining the integrity of remote objects with self-similarity properties, an image with fine details restored and redrawn in a natural manner can be obtained. In addition, due to a very small number of weights, the convolutional neural network model according to the present disclosure requires a small storage capacity.
[0130] The present disclosure has been described in detail with reference to the embodiments of the present disclosure shown in the accompanying drawings. These embodiments of the present disclosure are not intended to limit the present disclosure, but are only illustrative, and should be considered from an explanatory perspective rather than a restrictive perspective. Those of ordinary skill in the art will understand that these embodiments of the present disclosure can be easily modified to other specific forms without changing the technical spirit or main features of the present disclosure. For example, an element described as a single type can be implemented in a distributed manner. Similarly, an element described as distributed can be implemented in a combined form. Although specific terms are used in the specification, they are only used for the purpose of describing the concept of the present disclosure, and are not used to limit the scope of the present disclosure as defined in the claims.
[0131] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.
Claims
1. A neural network model training device for enhancing image details, the neural network model training device include: a memory configured to store one or more instructions; as well as At least one processor configured to execute one or more instructions stored in the memory to: Obtain low-quality input image blocks and high-quality input image blocks; Obtaining low-quality output image patches by inputting the low-quality input image patches into a first neural network model; Obtaining high-quality output image blocks by inputting the high-quality input image blocks into a second neural network model; as well as training a first neural network model based on a loss function, wherein the loss function is set to reduce the difference between low quality output image patches and high quality input image patches, and the difference between high quality output image patches and high quality input image patches, Wherein, the second neural network model is the same as the first neural network model; Wherein, the loss function includes as a term: the difference between the low quality output image patch and the high quality input image patch, and The product of the difference between the low quality output image patch and the low quality input image patch and the difference between the high quality output image patch and the high quality input image patch.
2. The neural network model training device according to claim 1, in, When the at least one processor trains the first neural network model, the gradient of the loss function of error back propagation does not include a partial differential of a difference between a low quality output image patch and a low quality input image patch.
3. The neural network model training device according to claim 1, in, The loss function includes: The L1 difference between the low-quality output image patch and the high-quality input image patch, The product of the L1 difference between the low quality output image patch and the low quality input image patch and the L1 difference between the high quality output image patch and the high quality input image patch, The visual geometry group (VGG) difference between the low-quality output image patch and the high-quality input image patch, and The product of the VGG difference between the low quality output image patch and the low quality input image patch and the VGG difference between the high quality output image patch and the high quality input image patch.
4. The neural network model training device according to claim 1, in, The first neural network model uses nested convolution kernels.
5. The neural network model training device according to claim 1, in, In the encoding part of the first neural network model, for convolution kernels other than the first convolution kernel, each convolution kernel includes each convolution kernel smaller than itself and is included in each convolution kernel larger than itself.
6. The neural network model training device according to claim 1, in, The low-quality input image blocks and the high-quality input image blocks are captured by the same image capture device based on different capture parameters.
7. The neural network model training device according to claim 1, in, The low quality input image patches are captured at a higher sensitivity and a shorter exposure time than the high quality input image patches, and Among them, high-quality input image patches are captured at lower sensitivity and longer exposure time than low-quality input image patches.
8. A method for training a neural network model for enhancing image details, the method include: Obtain low-quality input image blocks and high-quality input image blocks; Obtaining low-quality output image patches by inputting the low-quality input image patches into a first neural network model; Obtaining high-quality output image blocks by inputting the high-quality input image blocks into a second neural network model; as well as training a first neural network model based on a loss function, wherein the loss function is set to reduce the difference between low quality output image patches and high quality input image patches, and the difference between high quality output image patches and high quality input image patches, Wherein, the second neural network model is the same as the first neural network model; Wherein, the loss function includes as a term: the difference between the low quality output image patch and the high quality input image patch, and The product of the difference between the low quality output image patch and the low quality input image patch and the difference between the high quality output image patch and the high quality input image patch. 9 . A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to claim 8 .
Citation Information
Cited By
Method and apparatus for super resolution
US12670548B2
