Computer-based image processing methods, image-enhancing convolutional neural networks, and computer products

Abstract

A computer-implemented image processing method is provided. The computer-implemented image processing method includes: obtaining a pair of training samples, the pair including a training image with a first sharpness and a reference image with a second sharpness greater than the first sharpness, wherein at least a portion of the training image and at least a portion of the reference image in the same pair have the same content; inputting the training image into an image enhancement convolutional neural network to generate a training enhancement image; inputting the training enhancement image into an edge detector; generating a plurality of first edge maps by the edge detector; inputting the reference image into the edge detector; generating a plurality of second edge maps by the edge detector; and adjusting parameters in the image enhancement convolutional neural network to minimize at least one or more of the first and second losses.

Description

Technical Field

[0001] This invention relates to artificial intelligence, and more specifically, to a computer-implemented image processing method, an image enhancement convolutional neural network, and a computer product. Background Technology

[0002] Image enhancement is a commonly used image processing technique to improve image quality and the user's viewing experience. For example, when an image is overexposed, it appears brighter overall, and the low contrast makes it difficult to distinguish scenes within the overexposed image. Image enhancement can be used to improve the contrast of overexposed images. Summary of the Invention

[0003] In one aspect, this disclosure provides a computer-implemented image processing method, comprising: inputting a first image into a trained image augmentation convolutional neural network; processing the first image through the trained image augmentation convolutional neural network to obtain a second image, the second image having a higher resolution than the first image; wherein the trained image augmentation convolutional neural network is obtained by training an original image augmentation convolutional neural network; wherein training the original image augmentation convolutional neural network comprises: obtaining a pair of training samples, the pair of training samples including a training image having a first resolution and a reference image having a second resolution, the second resolution being greater than the first resolution, at least a portion of the training image and at least a portion of the reference image in the same pair having the same content; inputting the training image into the original image augmentation convolutional neural network; The process involves generating a training enhanced image; inputting the training enhanced image into an edge detector; generating a plurality of first edge maps from the edge detector; inputting the reference image into the edge detector; generating a plurality of second edge maps from the edge detector; calculating one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, and calculating a corresponding one of the one or more first losses based on a corresponding one of the plurality of first edge maps and a corresponding one of the plurality of second edge maps, wherein the corresponding one of the plurality of first edge maps and the corresponding one of the plurality of second edge maps have the same scale; comparing the training enhanced image with the reference image to calculate a second loss; and adjusting the parameters in the original image-enhanced convolutional neural network to at least minimize the one or more first losses and the second losses.

[0004] Optionally, the edge detector is an edge detection neural network, which includes N edge map generators, each configured to generate N edge maps; wherein the nth edge map generator among the N edge map generators includes an nth set of convolutional layers and an nth side layer configured to output the nth edge map of the same scale, 1≤n≤N; and when 1≤n<N, the nth edge map generator further includes an nth downsampling layer.

[0005] Optionally, generating the plurality of first edge maps by the edge detector includes: generating an nth first edge map of the same scale by the nth edge map generator; wherein generating the nth first edge map includes: inputting the (n-1)th first image into the nth group of convolutional layers; processing the (n-1)th first image by the nth group of convolutional layers to generate the nth processed first image; downsampling the nth processed first image by the nth downsampling layer to generate the nth downsampled first image; processing the nth processed first image by the nth side layer to generate the nth first edge map of the same scale; and when n < N, inputting the nth downsampled first image into the (n+1)th group of convolutional layers; wherein when n = 1, the (n-1)th first image is the training enhancement image; and when n > 1, the (n-1)th first image is the (n-1)th downsampled first image.

[0006] Optionally, generating the plurality of second edge maps by the edge detector includes: generating an nth second edge map of the same scale by the nth edge map generator; wherein generating the nth second edge map includes: inputting the (n-1)th second image into the nth group of convolutional layers; processing the (n-1)th second image by the nth group of convolutional layers to generate the nth processed second image; downsampling the nth processed second image by the nth downsampling layer to generate the nth downsampled second image; processing the nth processed second image by the nth side layer to generate the nth second edge map of the same scale; and when n < N, inputting the nth downsampled second image into the (n+1)th group of convolutional layers; wherein when n = 1, the (n-1)th second image is the reference image; and when n > 1, the (n-1)th second image is the (n-1)th downsampled second image.

[0007] Optionally, the nth group of convolutional layers includes multiple convolutional layers and multiple activation layers, each of the multiple convolutional layers includes (p×64) 3×3 convolutional kernels, where p is a positive integer; and the nth side layer includes a convolutional layer and a deconvolutional layer connected in sequence.

[0008] Optionally, the edge detector is an edge detection neural network, which includes: a first group of convolutional layers, a first pooling layer, a first lateral layer, a second group of convolutional layers, a second pooling layer, a second lateral layer, a third group of convolutional layers, a third pooling layer, a third lateral layer, a fourth group of convolutional layers, a fourth pooling layer, a fourth lateral layer, a fifth group of convolutional layers, and a fifth lateral layer; wherein, the first group of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 64 3×3 convolutional kernels; the second group of convolutional layers... The first group of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 128 3×3 convolutional kernels; the second group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 256 3×3 convolutional kernels; the third group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3×3 convolutional kernels; and the fourth group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3×3 convolutional kernels.

[0009] Optionally, calculating the one or more first losses based on the plurality of first edge maps and the plurality of second edge maps includes: calculating the nth loss based on the nth first edge map and the nth second edge map of the same scale, where 1≤n≤N.

[0010] Optionally, one or more first losses include Where Lossn represents the nth loss.

[0011] Optionally, calculating the nth loss includes calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein, the expression is sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of f(x), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation.

[0012] Optionally, calculating the nth loss includes calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein the L1 loss is calculated using the L1 loss function expressed as sum(|Yf(x)|), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation.

[0013] Optionally, calculating the nth loss includes calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein, the expression is avg(|Yf(x)| 2The MSE loss is calculated using the MSE loss function of f(x), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0014] Optionally, calculating the second loss includes calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein, the expression is sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of f(x), where Y represents the pixel value in the reference image, f(x) represents the pixel value in the training enhancement image, and sum represents the summation operation.

[0015] Optionally, calculating the second loss includes: calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein the L1 loss is calculated using the L1 loss function expressed as sun(|Yf(x)|), where Y represents the pixel value in the reference image, f(x) represents the pixel value in the training enhancement image, and sum represents the summation operation.

[0016] Optionally, calculating the second loss includes: calculating one or more of the L2 loss, L1 loss, and MSE loss; wherein, the expression avg(|Yf(x)| is used. 2 The MSE loss is calculated using the MSE loss function of f(x) in the reference image, where Y represents the pixel value in the reference image, f(x) represents the pixel value in the training enhancement image, and avg represents the averaging operation.

[0017] Optionally, the computer-implemented method further includes: repeatedly training the original image-enhanced convolutional neural network using different pairs of training images and reference images.

[0018] In another aspect, this disclosure provides an image augmentation convolutional neural network trained by the following steps: obtaining a pair of training samples, the pair comprising a training image having a first sharpness and a reference image having a second sharpness greater than the first sharpness, wherein at least a portion of the training image and at least a portion of the reference image in the same pair have the same content; inputting the training image into the image augmentation convolutional neural network to generate a training augmented image; inputting the training augmented image into an edge detector; generating a plurality of first edge maps by the edge detector; inputting the reference image into the edge detector; generating a plurality of second edge maps by the edge detector; calculating one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, calculating a corresponding one of the one or more first losses based on a corresponding one of the plurality of first edge maps and a corresponding one of the plurality of second edge maps, wherein the corresponding one of the plurality of first edge maps and the corresponding one of the plurality of second edge maps have the same scale; comparing the training augmented image with the reference image to calculate a second loss; and adjusting the parameters in the image augmentation convolutional neural network to at least minimize the one or more first losses and the second losses.

[0019] Optionally, the image enhancement convolutional neural network includes: a plurality of encoding modules; a plurality of decoding modules; and at least one cascade between a corresponding one of the plurality of encoding modules and a corresponding one of the plurality of decoding modules; wherein each of the plurality of encoding modules includes a downsampling convolutional layer and a plurality of first residual blocks arranged in series; and each of the plurality of decoding modules includes a plurality of second residual blocks arranged in series and an upsampling convolutional layer.

[0020] On the other hand, this disclosure provides an apparatus for training an image augmentation convolutional neural network configured to augment an original image, the apparatus comprising: a memory; and one or more processors; wherein the memory and the one or more processors are connected to each other; and the memory stores computer-executable instructions for controlling the one or more processors to perform the following operations: obtaining a pair of training samples, the pair of training samples comprising a training image having a first sharpness and a reference image having a second sharpness, wherein at least a portion of the training image and at least a portion of the reference image in the same pair have the same content; inputting the training image into the image augmentation convolutional neural network to generate a training augmented image; and inputting the training image into the image augmentation convolutional neural network to generate a training augmented image. The training enhancement image is input into an edge detector; a plurality of first edge maps are generated by the edge detector; the reference image is input into the edge detector; a plurality of second edge maps are generated by the edge detector; one or more first losses are calculated based on the plurality of first edge maps and the plurality of second edge maps, and a corresponding one of the one or more first losses is calculated based on a corresponding one of the plurality of first edge maps and a corresponding one of the plurality of second edge maps, wherein the corresponding one of the plurality of first edge maps and the corresponding one of the plurality of second edge maps have the same scale; and the parameters in the image enhancement convolutional neural network are adjusted to at least minimize the one or more first losses.

[0021] On the other hand, this disclosure provides a computer program product for training an image augmentation convolutional neural network configured to augment an original image, the computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to: obtain a pair of training samples, the pair of training samples comprising a training image having a first sharpness and a reference image having a second sharpness, at least a portion of the training image and at least a portion of the reference image in the same pair having identical content; input the training image into the image augmentation convolutional neural network to generate a training augmented image; input the training augmented image into... The reference image is input into the edge detector; a plurality of first edge maps are generated by the edge detector; a plurality of second edge maps are generated by the edge detector; one or more first losses are calculated based on the plurality of first edge maps and the plurality of second edge maps, and a corresponding one of the one or more first losses is calculated based on a corresponding one of the plurality of first edge maps and a corresponding one of the plurality of second edge maps, wherein the corresponding one of the plurality of first edge maps and the corresponding one of the plurality of second edge maps have the same scale; and the parameters in the image enhancement convolutional neural network are adjusted to at least minimize the one or more first losses. Attached Figure Description

[0022] The following figures are merely illustrative examples based on various disclosed embodiments and are not intended to limit the scope of the invention.

[0023] Figure 1 This is a schematic diagram of a convolutional neural network according to some embodiments of the present disclosure.

[0024] Figure 2 A computer implementation of a method for training an image-enhanced convolutional neural network according to some embodiments of the present disclosure is shown.

[0025] Figure 3 The structure of an edge detection neural network according to some embodiments of the present disclosure is shown.

[0026] Figure 4 A computer-implemented method for training an image-enhancing convolutional neural network is shown according to some embodiments of the present disclosure.

[0027] Figure 5 A computer-implemented method for training an image-enhancing convolutional neural network is shown according to some embodiments of the present disclosure.

[0028] Figure 6 A computer-implemented method for training an image-enhancing convolutional neural network is shown according to some embodiments of the present disclosure.

[0029] Figure 7 This is a schematic diagram illustrating the structure of an image-enhancing convolutional neural network according to some embodiments of the present disclosure.

[0030] Figure 8 This is a schematic diagram illustrating the structure of a corresponding one of a plurality of coding modules according to some embodiments of the present disclosure.

[0031] Figure 9 This is a schematic diagram illustrating the structure of a corresponding one of a plurality of decoding modules according to some embodiments of the present disclosure.

[0032] Figure 10 The structure of a VGG16 neural network according to some embodiments of the present disclosure is shown.

[0033] Figure 11 This is a schematic diagram of the structure of a device according to some embodiments of the present disclosure. Detailed Implementation

[0034] This disclosure will now be described in more detail with reference to the following embodiments. It should be noted that the following description of some embodiments presented herein is for illustrative and descriptive purposes only. It is not exhaustive or limited to the precise forms disclosed.

[0035] This disclosure provides, in particular, a computer-implemented method for training an image augmentation convolutional neural network, an image augmentation convolutional neural network, and a computer product that substantially eliminates one or more problems caused by the limitations and disadvantages of related technologies. In one aspect, this disclosure provides a computer-implemented image processing method. In some embodiments, the computer-implemented method includes inputting a first image into a trained image augmentation convolutional neural network; processing the first image through the trained image augmentation convolutional neural network to obtain a second image, the second image having greater sharpness than the first image. Optionally, the trained image augmentation convolutional neural network is obtained by training the image augmentation convolutional neural network. Optionally, training the image augmentation convolutional neural network includes: obtaining a pair of training samples, the pair including a training image with a first sharpness and a reference image with a second sharpness greater than the first sharpness, wherein at least a portion of the training image and at least a portion of the reference image in the same pair have the same content; inputting the training image into the image augmentation convolutional neural network to generate a training augmented image; inputting the training augmented image into an edge detector; generating a plurality of first edge maps by the edge detector; inputting the reference image into the edge detector; generating a plurality of second edge maps by the edge detector; calculating one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, calculating a corresponding one of the one or more first losses based on a corresponding one of the plurality of first edge maps and a corresponding one of the plurality of second edge maps, wherein the corresponding one of the plurality of first edge maps and the corresponding one of the plurality of second edge maps have the same scale; comparing the training augmented image with the reference image to calculate a second loss; and adjusting the parameters in the image augmentation convolutional neural network to at least minimize the one or more first losses and the second loss.

[0036] As used herein, the term "scale" refers to one or any combination of three dimensions of an image, including one or any combination of the image's width, height, and depth. In one example, the scale of an image (e.g., feature map, data, signal) refers to the image's "volume," which includes the image's width, height, and depth. In another example, the spatial scale of an image (e.g., feature map, data, signal) refers to the image's width and length, for example, width × length.

[0037] As used herein, the term "sharpness" refers to the contrast between individual image objects, such as the sharpness of image outlines and associated fine shadow patterns within an image. Higher sharpness is perceived better by the human eye. For example, by processing an input image using image processing methods provided in this disclosure (e.g., denoising and / or deblurring or resolution enhancement processing), the sharpness of the output image can be increased compared to the sharpness of the input image, resulting in a sharper output image than the input image. In one example, sharpness can be measured by utilizing mathematical calculations of the modulation transfer function method.

[0038] Figure 1 This is a schematic diagram of a convolutional neural network according to some embodiments of this disclosure. Reference Figure 1 Convolutional Neural Networks (CNNs) can be used in fields including image recognition, face recognition, text recognition, animal recognition, and image processing. For example, CNNs are used to process images. The input and output of a CNN are images. The convolutional kernels of a CNN replace scalar weights. In some embodiments, Figure 1 The convolutional neural network shown has only three layers. Optionally, the convolutional neural network includes an input layer IL, a hidden layer HL, and an output layer OL. Optionally, four inputs IPT are input to the input layer IL. Three outputs OPHL are outputs from the hidden layer HL. Two outputs OPT are output from the output layer OL. Therefore, the convolutional neural network can output two output images (e.g., two outputs OPT). For example, the four inputs IPT input to the input layer IL are four images, or four features of one image. The three outputs OPHL output from the hidden layer HL can be three feature maps of four images, or three feature maps of four features of one image input to the input layer IL.

[0039] As used herein, the term "convolutional neural network" refers to a deep feedforward artificial neural network. Optionally, a convolutional neural network includes multiple convolutional layers, multiple upsampling layers, and multiple downsampling layers. For example, each of the multiple convolutional layers can process an image. Upsampling and downsampling layers can resize the input image to a size corresponding to a specific convolutional layer. The output from an upsampling or downsampling layer can then be processed by a convolutional layer of the corresponding size. This allows the convolutional layers to add or extract features with a size different from that of the input image.

[0040] As used herein, the term "convolution kernel" refers to a two-dimensional matrix used in the convolution process. Optionally, each item in a plurality of terms in the two-dimensional matrix has a specific value.

[0041] As used herein, the term "convolution" refers to the process of processing an image. A convolution kernel is used for convolution. Each pixel of the input image has a value, and the convolution kernel starts at one pixel in the input image and moves sequentially over each pixel in the input image. At each location of the convolution kernel, the kernel overlaps with several pixels in the image based on the kernel's scale. At the location of the convolution kernel, the value of one pixel among the several overlapping pixels is multiplied by the corresponding value of the convolution kernel to obtain the multiplied value of one pixel among the several overlapping pixels. Subsequently, all the multiplied values ​​of the overlapping pixels are summed to obtain the sum corresponding to the location of the convolution kernel in the input image. By moving the convolution kernel over each pixel of the input image, all sums corresponding to all locations of the convolution kernel are collected and output to form the output image. In one example, convolution can use different convolution kernels to extract different features from the input image. In another example, the convolution process can use different convolution kernels to add more features to the input image.

[0042] As used herein, the term "convolutional layer" refers to a layer in a convolutional neural network. A convolutional layer is used to perform convolution on an input image to obtain an output image. Optionally, different convolutional kernels can be used to perform different convolutions on the same input image. Optionally, different convolutional kernels can be used to perform convolutions on different parts of the same input image. Optionally, different convolutional kernels can be used to perform convolutions on different input images; for example, multiple images are input into the convolutional layer, and corresponding convolutional kernels are used to perform convolutions on images from multiple images. Optionally, different convolutional kernels can be used depending on the specific input image.

[0043] refer to Figure 1 In some embodiments, the hidden layer HL includes a first hidden convolutional layer HCL1 and a second hidden convolutional layer HCL2. Optionally, both the first hidden convolutional layer HCL1 and the second hidden convolutional layer HCL2 have weights. and deviation Weight Represents the convolution kernel. Bias. This is a scalar value added to the output of the convolutional layer. k represents the k-th input layer of the convolutional neural network. i represents the i-th input image among multiple input images input to the corresponding input layer. j represents the j-th output among multiple outputs obtained by processing the corresponding input image. k, i, and j are positive integers. For example, the first hidden convolutional layer HCL1 includes a first set of convolutional kernels (e.g., ) and the first set of weights (e.g., The second hidden convolutional layer HCL2 includes a second set of convolutional kernels (e.g., ) and the second set of weights (e.g., Typically, each convolutional layer consists of hundreds of convolutional kernels. For example, in a deep convolutional neural network, the hidden layers of a deep convolutional neural network consist of at least five convolutional layers.

[0044] refer to Figure 1 The hidden layer HL of the convolutional neural network further includes a first hidden activation layer HAL1 and a second hidden activation layer HAL2. The first hidden activation layer HAL1 is located between the first hidden convolutional layer HCL1 and the second hidden convolutional layer HCL2. The second hidden activation layer HAL2 is sequentially located after the second hidden convolutional layer HCL2. Optionally, the activation layers (e.g., the first hidden activation layer HAL1 and the second hidden activation layer HAL2) include activation functions. By using active functions to add nonlinear elements to the convolutional neural network, the convolutional neural network can effectively solve relatively complex problems.

[0045] As used herein, the term "activation layer" refers to a layer in a convolutional neural network. An activation layer performs a nonlinear mapping on the output signal from a convolutional layer. Various functions can be used in activation layers. Examples of functions suitable for use in activation layers include, but are not limited to, the rectified linear unit (ReLU) function, the sigmoid function, and the hyperbolic tangent function (e.g., the tanh function). In one example, the activation layer is not included in the convolutional layer. In another example, the convolutional layer includes the activation layer. For example, the first hidden convolutional layer HCL1 includes the first hidden activation layer HAL1. The second hidden convolutional layer HCL2 includes the second hidden activation layer HAL2.

[0046] In some embodiments, in the first hidden convolutional layer HCL1, the first set of convolutional kernels and the first set of weights The input is applied to one of the four inputs IPT to generate multiple outputs from the first hidden convolutional layer HCL1. The multiple outputs from the first hidden convolutional layer HCL1 are processed by the first hidden activation layer HAL1 to generate multiple outputs from the first hidden activation layer HAL1.

[0047] In some embodiments, in the second hidden convolutional layer HCL2, the second set of convolutional kernels Second group of weights The outputs from the first hidden activation layer HAL1 are applied to one of the corresponding outputs to generate multiple outputs from the second hidden convolutional layer HCL2. The multiple outputs from the second hidden convolutional layer HCL2 are processed by the second hidden activation layer HAL2 to generate multiple outputs from the second hidden activation layer HAL2.

[0048] In one example, generating the corresponding output from multiple outputs of the first hidden convolutional layer HCL1 includes: converting the first set of convolutional kernels... The corresponding convolutional kernels in the first set of inputs are applied to the corresponding inputs in the four input IPTs to generate the first hidden intermediate output, and the first set of weights are applied to the first set of weights. The corresponding weights in the middle are added to the intermediate outputs to generate the corresponding outputs from the multiple outputs of the first hidden layer HCL1.

[0049] In another example, generating one of the multiple outputs from the second hidden convolutional layer HCL2 includes: applying the second set of convolutional kernels... The corresponding one from the multiple outputs of the first hidden convolutional layer HCL1 is applied to generate the second hidden intermediate output, and the second set of weights is applied to the corresponding one. The corresponding one from the second hidden intermediate output is added to generate the corresponding one of the multiple outputs from the second hidden convolutional layer HCL2.

[0050] For example, the output from the first hidden excitation layer HAL1 is the three outputs OPHL of the hidden layer HL. The output from the second hidden excitation layer HAL2 is the two outputs OPT of the output layer OL.

[0051] In some embodiments, convolutional layers are the core layers of a convolutional neural network. In a convolutional layer, neurons are connected to neurons in directly adjacent convolutional layers. Optionally, a convolutional layer applies multiple convolutional kernels to the input image to extract multiple features from the input image. The convolutional layer can extract one type of feature from the input image. Optionally, the initialized convolutional kernel is a random fractional matrix. During the pre-training process of the convolutional neural network, the convolutional kernel learns to obtain reasonable values.

[0052] Optionally, the result obtained by applying convolutional kernels to the input image is called a feature map. The number of feature maps is equal to the number of convolutional kernels. Each feature map corresponds to a specific convolutional kernel.

[0053] Optionally, a corresponding neuron in one of the multiple feature maps is formed by a rectangular arrangement of neurons. The neurons of a corresponding neuron in the multiple feature maps share a corresponding one of the multiple convolutional kernels.

[0054] Optionally, the convolutional neural network has multiple convolutional layers. A feature map from a corresponding output of one of the multiple convolutional layers is input into a downstream convolutional layer. The downstream convolutional layer processes the feature map output from the corresponding convolutional layer and outputs a downstream feature map from the multiple feature maps.

[0055] In some embodiments, the downsampling layer is located between two adjacent convolutional layers. In one example, a pooling layer is used to reduce the size of the input image to simplify computational complexity and reduce overfitting. In another example, a pooling layer can compress the features of the input image and extract the main features. Optionally, the pooling layer reduces the size of a single feature map within the feature maps, but does not change the number of feature maps. For example, an input image of size 12×12 is sampled using a 6×6 filter, and subsequently, the 6×6 filter outputs a 2×2 output image. This means that the 144 pixels of the 12×12 input image are divided into four parts, each of which has 36 pixels. After pooling using the 6×6 filter, the 36 pixels in each part are combined into 1 pixel, and the resulting output image has a size of 2×2.

[0056] As used herein, the term "pooling" refers to a type of downsampling. Various methods can be used for pooling. Examples of suitable pooling methods include, but are not limited to, max-pooling, average-polling, decimation, and DEMUXOUT. As used herein, the term "downsampling" refers to the process of extracting features from an input image and outputting an image with a smaller size. As used herein, the term "upsampling" refers to the process of adding more information to an input image and outputting an image with a larger scale.

[0057] Figure 2 A computer-implemented method for training an image-enhancing convolutional neural network according to some embodiments of the present disclosure is shown. References Figure 2In some embodiments, the method includes obtaining a pair of training samples, a training image TI with a first sharpness and a reference image RI with a second sharpness, wherein at least a portion of the training image TI and the reference image RI in the same pair has the same content. Optionally, the second sharpness is greater than the first sharpness. In one example, the training image TI and the reference image RI have the same content, but the image quality of the training image is lower than that of the reference image. In another example, the training image TI and the reference image RI have the same content, but the resolution of the training image TI is lower than that of the reference image RI. In yet another example, the training image TI and the reference image RI have the same content, but the contrast of the training image TI is lower than that of the reference image RI. As used herein, the term "image quality" broadly encompasses a variety of characteristics that affect the visual experience of an image, which can be affected by a variety of factors, including but not limited to, the resolution of the image data, the color depth of the image data, the contrast of the image data, and whether the image is compressed with high or low compression quality. The image enhancement convolutional neural network (IEN) is configured to enhance the original image.

[0058] In some embodiments, the pair of training images TI and reference images RI can be selected from a sample database prepared for training an image-enhancing convolutional neural network configured to enhance the original image. Multiple pairs of training and reference images exist, with each pair comprising a training image TI and a reference image RI. In one example, the training image TI and reference image RI are obtained by photographing the same scene under different exposure conditions. One image with relatively high contrast is the reference image RI, while the other image with relatively low contrast is the training image TI.

[0059] refer to Figure 2 In some embodiments, the method further includes: inputting the training image TI into the original image enhancement convolutional neural network IEN to generate a training enhanced image TEI; inputting the training enhanced image TEI into an edge detector (e.g., Figure 2 In the edge detection neural network (EDN) shown, a reference image RI is input into the edge detection neural network EDN. The edge detection neural network EDN is configured to: upon receiving a training augmented image TEI, generate multiple first edge maps based on the training augmented image TEI (e.g., such as...). Figure 2 The 1EM1, ..., 1EM shown n ... 1EM N The edge detection neural network (EDN) is configured to generate multiple second edge maps based on the reference image RI upon receiving the reference image RI (e.g., such as...). Figure 2 The 2EM1, ..., 2EM shown n... 2EM N Optionally, multiple second edge maps (e.g., such as...) Figure 2 The 2EM1, ..., 2EM shown n ... 2EM N They have the same scale. In some embodiments, the method further includes: calculating one or more losses (e.g., based on a plurality of first edge maps and a plurality of second edge maps) based on a plurality of first edge maps and a plurality of second edge maps. Figure 2 The L1, ..., L shown n ... L N One of one or more losses is calculated based on a corresponding one in a plurality of first edge maps and a corresponding one in a plurality of second edge maps, wherein the corresponding one in the plurality of first edge maps and the corresponding one in the plurality of second edge maps have the same scale. In this disclosure, various suitable edge detectors can be used to detect edges.

[0060] In some embodiments, the method further includes: adjusting the parameters in the original image augmentation convolutional neural network (IEN) at least in part based on one or more losses. Various adjustment methods can be used to adjust the parameters in the original image augmentation convolutional neural network (IEN). In one example, the parameters of the original image augmentation convolutional neural network (IEN) can be adjusted to at least minimize one or more losses.

[0061] In some embodiments, the method further includes: repeatedly training the original image-enhancing convolutional neural network IEN using the pair of training images TI and reference images RI, or different pairs of training images and reference images, to obtain a trained image-enhancing convolutional neural network. Optionally, repeatedly training the original image-enhancing convolutional neural network IEN includes: repeatedly training the original image-enhancing convolutional neural network IEN using the pair of training images TI and reference images RI. Optionally, repeatedly training the original image-enhancing convolutional neural network IEN includes: repeatedly training the original image-enhancing convolutional neural network IEN using different pairs of training images and reference images. Optionally, during the repeated training of the original image-enhancing convolutional neural network IEN, the training image TI in the pair of training images TI and reference images RI is retained, but the reference image RI in the pair of training images TI and reference images RI is replaced with a differential reference image that also has the same content as the training image. Optionally, during the repeated training of the original image-enhancing convolutional neural network IEN, the reference image RI in the pair of training images TI and reference images RI is retained, but the training image TI in the pair of training images TI and reference images RI is replaced with a differential training image that also has the same content as the reference image RI.

[0062] Alternatively, to avoid selecting the same pair of training and reference images during repetition, the pair of training and reference images used during training can be removed from the sample database.

[0063] Figure 3 The structure of an edge detection neural network according to some embodiments of this disclosure is shown. Reference Figure 3 In some embodiments, the edge detection neural network (EDN) includes components configured to generate N edge maps (e.g., ...). Figure 3 The EM1, ..., EM shown are shown. n ... EM N N edge graph generators (e.g., such as) Figure 3 The EMG1, ..., EMG shown n ... EMG N Depending on the input, the N edge maps can be edge maps corresponding to either the training augmented image or the reference image. In one example, the input to the edge detection neural network is the training augmented image, and the N edge maps are multiple first edge maps (e.g., such as...). Figure 2 The 1EM1, ..., 1EM shown n ... 1EM N In another example, the input to the edge detection neural network is a reference image, and the N edge maps are multiple second edge maps (e.g., such as...). Figure 2 The 2EM1, ..., 2EM shown n ... 2EM N ).

[0064] In some embodiments, the nth edge graph generator (EMG) among the N edge graph generators n Including the nth convolutional layer CLG n And connected to the last convolutional layer and configured to output the nth edge map EM of the same scale. n The nth side layer SL n Where 1≤n≤N, optionally, when 1≤n<N, the nth edge graph generator EMG in the N edge graph generators. n It also includes the nth downsampling layer PL connected to the last convolutional layer in the nth group of convolutional layers CLG. n (For example, the nth pooling layer).

[0065] Figure 4 A computer-implemented method for training an image-enhancing convolutional neural network according to some embodiments of the present disclosure is shown. References Figure 4 In some embodiments, the method includes: inputting the (n-1)th image into the nth convolutional layer CLG. n In the middle; by the nth convolutional layer CLGn Process the (n-1)th image to generate the nth processed image; pass through the nth downsampling layer PL n Downsampling (e.g., pooling) is performed on the nth processed image to generate the nth downsampled image; this is achieved through the nth side layer SL. n The nth processed image is processed to generate the nth edge map of the same scale; when n < N, the nth downsampled image is input into the (n+1)th convolutional layer CLG. (n+1) middle.

[0066] In some embodiments, the (n-1)th image is the (n-1)th first image; the method includes the generation of the nth edge map by the edge map generator EMG. n Generate the nth first edge map of the same scale. Figure 5 A computer-implemented method for training an image-enhancing convolutional neural network according to some embodiments of the present disclosure is shown. References Figure 5 In some embodiments, generating the nth first edge map includes: inputting the (n-1)th first image into the nth convolutional layer CLG. n In the middle; by the nth group of convolutional layers CLG n Process the (n-1)th first image to generate the nth processed first image; pass through the nth downsampling layer PL n The nth processed first image is downsampled (e.g., pooled) to generate the nth downsampled first image; by the nth side layer SL n Process the nth processed first image to generate the nth first edge at the same scale. Figure 1 EM n When n < N, the first image after the nth downsampling is input into the (n+1)th convolutional layer CLG. (n+1) In one example, when n = 1, the (n-1)th first image is the training augmentation image. In another example, when n > 1, the (n-1)th first image is the (n-1)th downsampled first image.

[0067] In some embodiments, the (n-1)th image is the (n-1)th second image; the method includes the generation of the nth edge map by the edge map generator EMG. n Generate the nth second edge map of the same scale. Figure 6 A computer-implemented method for training an image-enhancing convolutional neural network according to some embodiments of the present disclosure is shown. References Figure 6 In some embodiments, generating the nth second edge map includes: inputting the (n-1)th second image into the nth convolutional layer CLG. n In the middle; by the nth group of convolutional layers CLG nProcess the (n-1)th second image to generate the nth processed second image; pass through the nth downsampling layer PL n The nth processed second image is downsampled (e.g., pooled) to generate the nth downsampled second image; this is achieved by the nth side layer SL. n Process the nth processed second image to generate the nth second edge at the same scale. Figure 2 EM n When n < N, the second image after the nth downsampling is input into the (n+1)th convolutional layer CLG. (n+1) In one example, when n = 1, the (n-1)th second image is the reference image. In another example, when n > 1, the (n-1)th second image is the (n-1)th downsampled second image.

[0068] In some embodiments, the nth convolutional layer CLG n It includes multiple convolutional layers and multiple activation layers. Optionally, each of the multiple convolutional layers includes (p×64) 3×3 convolutional kernels, where p is a positive integer. In some embodiments, the nth side layer SL n It includes sequentially connected convolutional and deconvolutional layers, with the convolutional layers connected to the last convolutional layer in the nth group. The nth lateral layer SL... n The convolutional layer converts the multi-channel feature image into a single-channel feature image. The nth lateral layer SL... n The deconvolutional layers are upsampling layers. The input images fed into the N side layers each have different scales. The deconvolutional layers in the N side layers upsample the input images by different factors (e.g., 1, 2, 4, 8, 16, etc.). In one example, the N edge maps from the N side layers each have the same scale.

[0069] In one example, the edge detection neural network includes a first set of convolutional layers, a first pooling layer, a first lateralization layer, a second set of convolutional layers, a second pooling layer, a second lateralization layer, a third set of convolutional layers, a third pooling layer, a third lateralization layer, a fourth set of convolutional layers, a fourth pooling layer, a fourth lateralization layer, a fifth set of convolutional layers, and a fifth lateralization layer. Optionally, the first set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 64 3×3 convolutional kernels. Optionally, the second set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 128 3×3 convolutional kernels. Optionally, the third set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 256 3×3 convolutional kernels. Optionally, the fourth set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels. Optionally, the fifth group of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels.

[0070] In some embodiments, calculating one or more losses based on a plurality of first edge maps and a plurality of second edge maps includes: based on the nth first edge at the same scale. Figure 1 EM n and the nth second edge of the same scale Figure 2 EM n To calculate the nth loss L n , 1 ≤ n ≤ N. Optionally, one or more first losses include Among them, Loss n Let L represent the nth loss. Various appropriate losses can be used to train image augmentation convolutional neural networks. In some embodiments, the nth loss L is calculated. n This includes calculating one or more of the following: L2 loss, L1 loss, and MSE loss. In some embodiments, the nth loss L is calculated. n This includes using the expression sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. In some embodiments, the nth loss L is calculated... n This includes calculating the L1 loss using the L1 loss function expressed as sum(|Yf(x)|). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. In some embodiments, calculating the nth loss Ln includes using the expression avg(|Yf(x)|). 2The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0071] In this method, one or more losses are computed based on multiple first edge maps and multiple second edge maps. By minimizing one or more losses, the edges of the trained augmented image TEI can be generated to be more consistent with the edges of the reference image RI. Using this method, more edge details can be preserved in the augmented image generated by the image augmentation convolutional neural network IEN.

[0072] refer to Figure 2 In some embodiments, the method further includes comparing the training augmented image TEI with a reference image RI to calculate a total loss LT (second loss). Adjusting the parameters in the image augmentation convolutional neural network IEN includes adjusting the parameters in the image augmentation convolutional neural network IEN to minimize the total loss LT. Optionally, adjusting the parameters in the image augmentation convolutional neural network IEN includes adjusting the parameters in the image augmentation convolutional neural network IEN to at least minimize one or more first losses and the total loss LT. In some embodiments, calculating the total loss LT includes calculating one or more of the following: L2 loss, L1 loss, and MSE loss. In some embodiments, calculating the total loss LT includes using the expression sum(|Y f(x)| 2 The L2 loss is calculated using the L2 loss function of f(x) = ... 2 The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0073] In this method, the total loss is calculated based on a comparison between the trained augmented image TEI and the reference image RI. By minimizing the total loss, it is ensured that the content of the original image can be preserved in the augmented image generated by the image augmentation convolutional neural network IEN.

[0074] In some embodiments, the method further includes comparing the training augmented image TEI with a reference image RI to calculate a third loss. Adjusting the parameters in the image augmentation convolutional neural network IEN includes adjusting the parameters in the image augmentation convolutional neural network IEN to minimize the third loss. Optionally, adjusting the parameters in the image augmentation convolutional neural network IEN includes adjusting the parameters in the image augmentation convolutional neural network IEN to at least minimize one or more first losses and the third loss. Optionally, adjusting the parameters in the image augmentation convolutional neural network IEN includes adjusting the parameters in the image augmentation convolutional neural network IEN to at least minimize one or more first losses, the total loss LT, and the third loss.

[0075] In some embodiments, the method includes: inputting a training augmented image TEI and a reference image RI into a VGG neural network, respectively, to generate outputs from the VGG neural network; and calculating a third loss based on the outputs from the VGG neural network. Optionally, the VGG neural network is a VGG16 neural network. Figure 10 The structure of a VGG16 neural network according to some embodiments of the present disclosure is shown. Reference Figure 10 In some embodiments, the VGG16 neural network includes a first set of convolutional layers CLG1, a first pooling layer PL1, a second set of convolutional layers CLG2, a second pooling layer PL2, a third set of convolutional layers CLG3, a third pooling layer PL3, a fourth set of convolutional layers CLG4, a fourth pooling layer PL4, a fifth set of convolutional layers CLG5, a fifth pooling layer PL5, a first fully connected layer FC1, a second fully connected layer FC2, and a third fully connected layer FC3. Optionally, the first set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 64 3×3 convolutional kernels. Optionally, the second set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 128 3×3 convolutional kernels. Optionally, the third set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 256 3×3 convolutional kernels. Optionally, the fourth group of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels. Optionally, the fifth group of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels.

[0076] In some embodiments, the method includes: inputting a training enhancement image TEI into a VGG16 neural network; generating a first output from the m-th convolutional layer in the n-th group of convolutional layers in the VGG16 neural network; inputting a reference image RI into the VGG16 neural network; generating a second output from the m-th convolutional layer in the n-th group of convolutional layers in the VGG16 neural network; and calculating a third loss based on the first output and the second output; wherein m is a positive integer, n is a positive integer, 1≤n≤N, 1≤m≤M, and M is the total number of convolutional layers in the n-th group of convolutional layers.

[0077] In some embodiments, m = 1 and n = 3. The method includes: inputting a training enhancement image TEI into a VGG16 neural network; generating a first output from a first convolutional layer in a third set of convolutional layers (CLG3) of the VGG16 neural network; inputting a reference image RI into the VGG16 neural network; generating a second output from the first convolutional layer in the third set of convolutional layers (CLG3) of the VGG16 neural network; and calculating a third loss based on the first output and the second output. Optionally, the third set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 256 3×3 convolutional kernels.

[0078] In some embodiments, calculating the third loss includes calculating one or more of the following: L2 loss, L1 loss, and MSE loss. In some embodiments, calculating the third loss includes using the expression sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of the first output. Here, Y represents the pixel value in the second output, f(x) represents the pixel value in the first output, and sum represents the summation operation. In some embodiments, calculating the third loss includes calculating the L1 loss using the L1 loss function of the expression sum(|Yf(x)|). Here, Y represents the pixel value in the second output, f(x) represents the pixel value in the first output, and sum represents the summation operation. In some embodiments, calculating the third loss includes calculating the L1 loss using the expression avg(|Yf(x)|). 2 The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the second output, f(x) represents the pixel value in the first output, and avg represents the averaging operation.

[0079] On the other hand, this disclosure provides an apparatus for training an image enhancement convolutional neural network configured to enhance the original image. Figure 11 This is a schematic diagram of the structure of a device according to some embodiments of this disclosure. Reference Figure 11In some embodiments, the device includes a central processing unit (CPU) configured to perform actions according to computer-executable instructions stored in ROM or RAM. Optionally, data and programs required by the computer system are stored in RAM. Optionally, the CPU, ROM, and RAM are electrically connected to each other via a bus. Optionally, an input / output interface is electrically connected to the bus.

[0080] In some embodiments, the device includes a memory and one or more processors. The memory and the one or more processors are connected to each other. The memory stores computer-executable instructions for controlling the one or more processors to obtain a pair of training samples, the pair of training samples having a training image of first sharpness and a reference image of second sharpness, at least a portion of the training image and at least a portion of the reference image in the same pair having the same content; inputting the training image into the image augmentation convolutional neural network to generate a training augmented image; inputting the training augmented image into an edge detection neural network; generating a plurality of first edge maps by the edge detection neural network based on the training augmented image; inputting the reference image into the edge detection neural network; generating a plurality of second edge maps by the edge detection neural network based on the reference image; calculating one or more losses based on the plurality of first edge maps and the plurality of second edge maps, each of the one or more losses being calculated based on a corresponding first edge map and a corresponding second edge map of the plurality of second edge maps, the corresponding first edge map and the corresponding second edge map of the plurality of second edge maps having the same scale; and adjusting parameters in the image augmentation convolutional neural network to at least minimize the one or more losses.

[0081] In some embodiments, the edge detection neural network includes N edge map generators, each configured to generate N edge maps. Optionally, the nth edge map generator among the N edge map generators includes an nth set of convolutional layers and an nth side layer, the nth side layer being connected to the last convolutional layer and configured to output an nth edge map of the same scale, 1 ≤ n ≤ N. Optionally, when 1 ≤ n < N, the nth edge map generator among the N edge map generators further includes an nth downsampling layer (e.g., an nth pooling layer) connected to the last convolutional layer in the nth set of convolutional layers. Optionally, the memory stores computer-executable instructions for controlling the one or more processors to input the (n-1)th image into the nth group of convolutional layers; to process the (n-1)th image by the nth group of convolutional layers to generate the nth processed image; to pool the nth processed image by the nth downsampling layer to generate the nth downsampled image; to process the nth processed image by the nth side layer to generate the nth edge map of the same scale; and when n < N, to input the nth downsampled image into the (n+1)th group of convolutional layers.

[0082] In some embodiments, the memory further stores computer-executable instructions for controlling one or more processors to generate an nth first edge map of the same scale by an nth edge map generator. Optionally, the memory further stores computer-executable instructions for controlling the one or more processors to input the (n-1)th first image into the nth group of convolutional layers; to process the (n-1)th first image by the nth group of convolutional layers to generate an nth processed first image; to pool the nth processed first image by the nth downsampling layer to generate an nth downsampled first image; to process the nth processed first image by the nth side layer to generate an nth first edge map of the same scale; and when n < N, to input the nth downsampled first image into the (n+1)th group of convolutional layers. Optionally, when n = 1, the (n-1)th first image is a training enhancement image. Optionally, when n>1, the (n-1)th first image is the (n-1)th downsampled first image.

[0083] In some embodiments, the memory further stores computer-executable instructions for controlling one or more processors to generate an nth second edge map of the same scale by an nth edge map generator. Optionally, the memory further stores computer-executable instructions for controlling the one or more processors to input the (n-1)th second image into the nth set of convolutional layers; to process the (n-1)th second image by the nth set of convolutional layers to generate an nth processed second image; to pool the nth processed second image by the nth downsampling layer to generate an nth downsampled second image; to process the nth processed second image by the nth side layer to generate an nth second edge map of the same scale; and when n < N, to input the nth downsampled second image into the (n+1)th set of convolutional layers. Optionally, when n = 1, the (n-1)th second image is a training enhancement image. Optionally, when n > 1, the (n-1)th second image is the (n-1)th downsampled second image.

[0084] In some embodiments, the nth group of convolutional layers includes multiple convolutional layers and multiple activation layers, each of the multiple convolutional layers including (p×64) 3×3 convolutional kernels. In one example, the multiple convolutional layers in the nth group of convolutional layers include 64, 128, 256, 512, and 512 3×3 convolutional kernels, respectively. Optionally, the nth side layer includes sequentially connected convolutional layers and deconvolutional layers, the convolutional layers being connected to the last convolutional layer in the nth group of convolutional layers.

[0085] In some embodiments, the edge detection neural network includes a first set of convolutional layers, a first pooling layer, a first lateralization layer, a second set of convolutional layers, a second pooling layer, a second lateralization layer, a third set of convolutional layers, a third pooling layer, a third lateralization layer, a fourth set of convolutional layers, a fourth pooling layer, a fourth lateralization layer, a fifth set of convolutional layers, and a fifth lateralization layer. Optionally, the first set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 64 3×3 convolutional kernels. Optionally, the second set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 128 3×3 convolutional kernels. Optionally, the third set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 256 3×3 convolutional kernels. Optionally, the fourth set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels. Optionally, the fifth group of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels.

[0086] In some embodiments, the memory further stores computer-executable instructions for controlling one or more processors to compute an nth loss based on an nth first edge map and an nth second edge map of the same scale. Optionally, the memory further stores computer-executable instructions for controlling one or more processors to compute one or more of the L2 loss, L1 loss, and MSE loss. Optionally, the memory further stores computer-executable instructions for controlling one or more processors to use the expression sum(|Yf(x)| 2 The memory stores an L2 loss function of expression sum(|Yf(x)|) to calculate the L2 loss, where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. Optionally, the memory also stores computer-executable instructions for controlling one or more processors to use an L1 loss function of expression sum(|Yf(x)|) to calculate the L1 loss, where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. Optionally, the memory also stores computer-executable instructions for controlling one or more processors to use an L1 loss function of expression sum(|Yf(x)|) to calculate the L1 loss. 2 The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0087] In some embodiments, the memory further stores computer-executable instructions for controlling one or more processors to compare training augmented images with reference images to calculate a total loss (second loss). Optionally, the memory further stores computer-executable instructions for controlling one or more processors to adjust parameters in the image augmentation convolutional neural network to at least minimize one or more losses and the total loss. Optionally, the memory further stores computer-executable instructions for controlling one or more processors to calculate one or more of the following: L2 loss, L1 loss, and MSE loss. Optionally, the memory further stores computer-executable instructions for controlling one or more processors to use the expression sum(|Yf(x)| 2The memory also stores computer-executable instructions to control one or more processors to calculate the L2 loss using the L1 loss function expressed as sum(|Yf(x)|), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. Optionally, the memory also stores computer-executable instructions to control one or more processors to calculate the L1 loss using the L1 loss function expressed as sum(|Yf(x)|). 2 The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0088] In some embodiments, the memory also stores computer-executable instructions for controlling one or more processors to repeatedly train images to enhance a convolutional neural network using the pair of training images and reference images or different pairs of training images and reference images.

[0089] In another aspect, this disclosure provides a computer program product for training an image enhancement convolutional neural network configured to enhance an original image. The computer program product includes a non-transitory tangible computer-readable medium having computer-readable instructions thereon. The computer-readable instructions are executable by a processor to cause the processor to perform: obtaining a pair of training samples, the pair of training samples having a training image of first sharpness and a reference image of second sharpness, at least a portion of the training image and at least a portion of the reference image in the same pair having the same content; inputting the training image into the image augmentation convolutional neural network to generate a training augmented image; inputting the training augmented image into an edge detection neural network; generating a plurality of first edge maps by the edge detection neural network based on the training augmented image; inputting the reference image into the edge detection neural network; generating a plurality of second edge maps by the edge detection neural network based on the reference image; calculating one or more losses based on the plurality of first edge maps and the plurality of second edge maps, each of the one or more losses being calculated based on a corresponding first edge map and a corresponding second edge map of the plurality of second edge maps, the corresponding first edge map and the corresponding second edge map of the plurality of second edge maps having the same scale; and adjusting the parameters in the image augmentation convolutional neural network to at least minimize one or more losses.

[0090] In some embodiments, the edge detection neural network includes N edge map generators, each configured to generate N edge maps. Optionally, the nth edge map generator among the N edge map generators includes an nth set of convolutional layers and an nth side layer, the nth side layer being connected to the last convolutional layer and configured to output an nth edge map of the same scale, 1 ≤ n ≤ N. Optionally, when 1 ≤ n < N, the nth edge map generator among the N edge map generators further includes an nth downsampling layer (e.g., an nth pooling layer) connected to the last convolutional layer in the nth set of convolutional layers. Optionally, the memory stores computer-executable instructions for controlling the one or more processors to input the (n-1)th image into the nth group of convolutional layers; to process the (n-1)th image by the nth group of convolutional layers to generate the nth processed image; to pool the nth processed image by the nth downsampling layer to generate the nth downsampled image; to process the nth processed image by the nth side layer to generate the nth edge map of the same scale; and when n < N, to input the nth downsampled image into the (n+1)th group of convolutional layers.

[0091] In some embodiments, the computer-readable instructions may be executed by a processor to further cause the processor to generate an nth first edge map of the same scale by an nth edge map generator. Optionally, the memory also stores computer-executable instructions for controlling the one or more processors to input the (n-1)th first image into the nth group of convolutional layers; to process the (n-1)th first image by the nth group of convolutional layers to generate an nth processed first image; to pool the nth processed first image by the nth downsampling layer to generate an nth downsampled first image; to process the nth processed first image by the nth side layer to generate an nth first edge map of the same scale; and when n < N, to input the nth downsampled first image into the (n+1)th group of convolutional layers. Optionally, when n = 1, the (n-1)th first image is a training enhancement image. Optionally, when n>1, the (n-1)th first image is the (n-1)th downsampled first image.

[0092] In some embodiments, the computer-readable instructions may be executed by a processor to further cause the processor to generate an nth second edge map of the same scale by an nth edge map generator. Optionally, the memory also stores computer-executable instructions for controlling the one or more processors to input the (n-1)th second image into the nth group of convolutional layers; to process the (n-1)th second image by the nth group of convolutional layers to generate an nth processed second image; to pool the nth processed second image by the nth downsampling layer to generate an nth downsampled second image; to process the nth processed second image by the nth side layer to generate an nth second edge map of the same scale; and when n < N, to input the nth downsampled second image into the (n+1)th group of convolutional layers. Optionally, when n = 1, the (n-1)th second image is a training enhancement image. Optionally, when n>1, the (n-1)th second image is the (n-1)th downsampled second image.

[0093] In some embodiments, the nth group of convolutional layers includes a plurality of convolutional layers and a plurality of activation layers, each of the plurality of convolutional layers including (p×64) 3×3 convolutional kernels. Optionally, the nth side layer includes a sequentially connected convolutional layer and a deconvolutional layer, the convolutional layers being connected to the last convolutional layer in the nth group of convolutional layers.

[0094] In some embodiments, the edge detection neural network includes a first set of convolutional layers, a first pooling layer, a first lateralization layer, a second set of convolutional layers, a second pooling layer, a second lateralization layer, a third set of convolutional layers, a third pooling layer, a third lateralization layer, a fourth set of convolutional layers, a fourth pooling layer, a fourth lateralization layer, a fifth set of convolutional layers, and a fifth lateralization layer. Optionally, the first set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 64 3×3 convolutional kernels. Optionally, the second set of convolutional layers includes two convolutional layers and two activation layers, each of the two convolutional layers including 128 3×3 convolutional kernels. Optionally, the third set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 256 3×3 convolutional kernels. Optionally, the fourth set of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels. Optionally, the fifth group of convolutional layers includes three convolutional layers and three activation layers, each of the three convolutional layers including 512 3×3 convolutional kernels.

[0095] In some embodiments, the computer-readable instructions may be executed by a processor to further cause the processor to compute an nth loss based on an nth first edge map and an nth second edge map of the same scale, 1 ≤ n ≤ N. Optionally, one or more first losses include Among them, Loss n This represents the nth loss. Optionally, the computer-readable instructions may be executed by a processor to further cause the processor to perform calculations of one or more of the following: L2 loss, L1 loss, and MSE loss. Optionally, the computer-readable instructions may be executed by a processor to further cause the processor to perform calculations using the expression sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of f(x) = f(x), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. Optionally, the computer-readable instruction can be executed by a processor to further cause the processor to perform the calculation of the L1 loss using the L1 loss function of the expression sum(|Yf(x)|), where Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and sum represents the summation operation. Optionally, the computer-readable instruction can be executed by a processor to further cause the processor to perform the calculation of the L1 loss using the expression avg(|Yf(x)|). 2 The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0096] In some embodiments, the computer-readable instructions may be executed by a processor to further cause the processor to perform a comparison of the training augmented image with a reference image to calculate a total loss (second loss). Optionally, the computer-readable instructions may be executed by a processor to further cause the processor to perform adjustments to the parameters in the image augmentation convolutional neural network to at least minimize one or more losses and the total loss. Optionally, the computer-readable instructions may be executed by a processor to further cause the processor to perform calculations of one or more of the following: L2 loss, L1 loss, and MSE loss. Optionally, the computer-readable instructions may be executed by a processor to further cause the processor to perform calculations using the expression sum(|Yf(x)| 2 The L2 loss is calculated using the L2 loss function of f(x) = ... 2The MSE loss is calculated using the MSE loss function of f(x). Here, Y represents the pixel value in the nth second edge map, f(x) represents the pixel value in the nth first edge map, and avg represents the averaging operation.

[0097] In some embodiments, the computer-readable instructions may be executed by a processor to further cause the processor to perform repeated training of the convolutional neural network using the pair of training and reference images or different pairs of training and reference images.

[0098] On the other hand, this disclosure provides an image augmentation convolutional neural network trained by the method described herein. This method can train various suitable image augmentation convolutional neural networks. Figure 7 This is a schematic diagram illustrating the structure of an image enhancement convolutional neural network according to some embodiments of the present disclosure. Reference Figure 7 In some embodiments, the image-enhancing convolutional neural network 110 includes a plurality of encoding modules 1110, a plurality of decoding modules 1120, and at least one concatenation between a corresponding one of the plurality of encoding modules 1110 and a corresponding one of the plurality of decoding modules 1120. Optionally, through at least one concatenation between a corresponding one of the plurality of encoding modules 1110 and a corresponding one of the plurality of decoding modules 1120, the output from a corresponding one of the plurality of encoding modules 1110 and the output from a decoding module immediately preceding the corresponding one of the plurality of decoding modules 1120 are concatenated as the input to the corresponding one of the plurality of decoding modules 1120.

[0099] Figure 8 This is a schematic diagram illustrating the structure of a corresponding one of a plurality of coding modules according to some embodiments of the present disclosure. Reference Figure 8 In some embodiments, a corresponding one of the plurality of encoding modules 1110 includes a cascaded downsampled convolutional layer 1111 and a plurality of first residual blocks 1112. In one example, the downsampled convolutional layer 1111 is a 2x downsampled convolutional layer. Optionally, the downsampled convolutional layer 1111 includes a convolutional layer (e.g., a 5x5 convolutional layer with a stride of 2, Conv 5x5) followed by a leaky rectified linear unit (LeakyReLU). Optionally, each of the plurality of first residual blocks 1112 includes a convolutional layer (e.g., a 5x5 convolutional layer with a stride of 2), followed by a leaky rectified linear unit, then another convolutional layer (e.g., a 5x5 convolutional layer with a stride of 2), and again followed by another leaky rectified linear unit. In some embodiments, each of the plurality of first residual blocks 1112 further includes one or more skip connections for performing the same mapping.

[0100] Figure 9 This is a schematic diagram illustrating the structure of a corresponding one of a plurality of decoding modules according to some embodiments of the present disclosure. Reference Figure 9 In some embodiments, a corresponding decoding module in the plurality of decoding modules 1120 includes a plurality of second residual blocks 1121 arranged in series and an upsampled convolutional layer 1122. In one example, the upsampled convolutional layer 1122 is a 2x downsampled convolutional layer. Optionally, the upsampled convolutional layer 1122 includes a deconvolutional layer (e.g., a 5x5 deconvolutional layer with a stride of 2), followed by a leaky rectified linear unit. Optionally, each of the plurality of second residual blocks 1121 includes a convolutional layer (e.g., a 5x5 convolutional layer with a stride of 2), followed by a leaky rectified linear unit, then another convolutional layer (e.g., a 5x5 convolutional layer with a stride of 2), and again followed by another leaky rectified linear unit. In some embodiments, each of the plurality of second residual blocks 1121 also includes one or more skip connections for performing the same mapping.

[0101] In some embodiments, the image enhancement convolutional neural network includes 2n sequentially connected sampling units; and a plurality of processing units, wherein each processing unit is located between two adjacent sampling units in the 2n sampling units. Optionally, the first to the nth sampling units in the 2n sampling units are DeMux units, each DeMux unit being configured to rearrange pixels from each input image to each DeMux unit in the DeMux unit according to a first scrambling rule to obtain rearranged images, and being configured to segment each rearranged image into m segmented images. Optionally, the (n+1)th to the (2n)th sampling units in the 2n sampling units are Mux units, each Mux unit being configured to combine corresponding m' input images to each Mux unit in the Mux unit to obtain combined images, and being configured to rearrange the pixels of each combined image according to a second scrambling rule to obtain an output image. Optionally, each of the multiple processing units comprises a plurality of sequentially connected convolutional blocks. Optionally, m is an integer greater than or equal to 2, m' is an integer greater than or equal to 2, and n is a positive integer.

[0102] As used herein, the term "Mux unit" refers to a scale-up unit that performs pixel interleaving and rearrangement on multiple input images to obtain at least one output image, the scale of which is larger than that of one of the multiple input images. As a result, the Mux unit increases the number of pixels in each image by arranging and combining pixels from the multiple input images. Optionally, four input images INPUT 4n, INPUT 4n+1, INPUT 4n+2, and INPUT 4n+3 are input into the Mux unit, assuming the number of pixels in the input images is a*b; after the pixel rearrangement processing of the Mux unit, at least one output image OUTPUT with 2a*2b pixels is output.

[0103] In one example, four input images INPUT 4n, INPUT 4n+1, INPUT 4n+2, and INPUT 4n+3 are input into a Mux unit, assuming the number of pixels in the input images is a*b. After pixel rearrangement processing in 2*2 Mux units, four images OUTPUT 4n, OUTPUT 4n+1, OUTPUT 4n+2, and OUTPUT4n+3 with 2a*2b pixels are output.

[0104] In another example, four input images INPUT 4n, INPUT 4n+1, INPUT 4n+2, and INPUT 4n+3 are input into the Mux unit, assuming that the number of pixels in the input images is a*b; after the pixel rearrangement processing in the Mux unit, an image OUTPUT 4n with 2a*2b pixels is output.

[0105] As used herein, the term "DeMux unit" refers to a scale-down unit that performs pixel rearrangement and segmentation of an input image to obtain multiple output images at a smaller scale than the input image. As a result, by rearranging the pixels in the input image and segmenting it into multiple output images, the DeMux unit reduces the number of pixels in each image. Optionally, an input image INPUT with 2a*2b pixels is input to the DeMux unit, and four output images INPUT 4n, INPUT 4n+1, INPUT 4n+2, and INPUT 4n+3 with a*b pixels are output from the DeMux unit.

[0106] In some embodiments, the DeMux unit is configured to rearrange the pixels of each input image to obtain rearranged images, and then segment each rearranged image into m segmented images (m is an integer greater than or equal to 2). Optionally, the pixels of each input image are rearranged according to a first scrambling rule, which includes: moving pixels at position A to position B, moving pixels previously at position B to position C, and so on. After rearranging the pixels of each input image to obtain rearranged images, the rearranged images are segmented into multiple segmented images.

[0107] In some embodiments, the spatial scale (width × height) of each input image is 4*4. After inputting each input image into the DeMux unit, four segmented images with a spatial scale of 2*2 are output. Optionally, the pixels of each input image are assigned to the four segmented images.

[0108] Since the scale of the segmented image output from the DeMux unit is smaller than the scale of each input image of the DeMux unit, the processing performed by the DeMux unit is downsampling, which reduces the scale of the input images. Simultaneously, the DeMux unit only rearranges and segments the pixels of each input image to the DeMux unit without discarding any pixel values, thus preventing any information loss.

[0109] In some embodiments, the Mux unit is configured to combine m' input images into the Mux unit to obtain individual combined images, and to arrange the pixels of the individual combined images according to a second scrambling rule to obtain an output image. Optionally, the first and second scrambling rules are opposites of each other. For example, the second scrambling rule includes: moving a pixel at position C to position B, moving a pixel previously at position B to position A, and so on, so that the pixels in the output image are arranged at the same positions as the corresponding pixels in the original images input to the DeMux unit.

[0110] In some embodiments, the Mux unit combines four input images with a spatial scale of 2*2 into combined images with a spatial scale of 4*4. Optionally, pixels of a corresponding one of the four input images are distributed to regions with a spatial scale of 2*2 in the respective combined images with a spatial scale of 4*4.

[0111] Because the scale of the output image from the Mux unit is larger than the scale of the m' input images to the Mux unit, the processing performed by the Mux unit is upsampling, which increases the scale of the m' input images. At the same time, the Mux unit only combines and rearranges the pixels of the m' input images to the Mux unit without discarding any pixel values, thus preventing any information loss.

[0112] The various illustrative neural networks, layers, units, channels, blocks, and other operations described in conjunction with the configurations disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Such neural networks, layers, units, channels, blocks, and other operations can be implemented or executed using a general-purpose processor, digital signal processor (DSP), ASIC or ASSP, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to produce the configurations disclosed herein. For example, such configurations can be implemented at least partially as hardwired circuitry, circuit configurations fabricated into application-specific integrated circuits, or firmware loaded into non-volatile memory, or software programs loaded or added to data storage media as machine-readable code, such code being instructions executable by an array of logic elements such as a general-purpose processor or other digital signal processing unit. The general-purpose processor can be a microprocessor, but alternatively, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Software modules may reside in non-transitory storage media, such as RAM (Random Access Memory), ROM (Read-Only Memory), non-volatile RAM (NVRAM), such as flash RAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, or CD-ROMs; or in any other form of storage media known in the art. The illustrative storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in the user terminal. Alternatively, the processor and storage medium may reside as discrete components in the user terminal.

[0113] For illustrative and descriptive purposes, the foregoing description of embodiments of the invention has been provided. It is not exhaustive, nor is it intended to limit the invention to the precise forms or exemplary embodiments disclosed. Therefore, the foregoing description should be considered illustrative rather than restrictive. Clearly, many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to explain the principles of the invention and its best mode of practical application, thereby enabling those skilled in the art to understand the various embodiments of the invention and the various modifications suitable for the particular use or implementation contemplated. The scope of the invention is intended to be defined by the appended claims and their equivalents, wherein, unless otherwise stated, all terms are to be interpreted in their broadest reasonable sense. Therefore, the terms “the invention,” “the present invention,” etc., do not necessarily limit the scope of the claims to the specific embodiments, and references to exemplary embodiments of the invention do not imply limitation of the invention, nor should such limitation be inferred. The invention is defined only by the spirit and scope of the appended claims. Furthermore, these claims may involve the use of “first,” “second,” etc., followed by nouns or elements. These terms should be understood as nomenclature and should not be construed as limiting the number of elements modified by these nomenclatures unless a specific number has been given. Any advantages and benefits described may not apply to all embodiments of the invention. It should be understood that changes to the described embodiments can be made by those skilled in the art without departing from the scope of the invention as defined by the appended claims. Furthermore, the elements and components in this disclosure are not intended for public distribution, whether or not they are expressly recited in the appended claims.

Claims

1. A computer-implemented image processing method comprising: inputting a first image into a trained image enhancement convolutional neural network; processing the first image by the trained image enhancement convolutional neural network to obtain a second image, the second image having a higher definition than the first image; wherein the trained image enhancement convolutional neural network is obtained by training an original image enhancement convolutional neural network; wherein training the original image enhancement convolutional neural network comprises: obtaining a pair of training samples, the pair of training samples comprising a training image having a first definition and a reference image having a second definition, the second definition being higher than the first definition, at least a portion of the training image and at least a portion of the reference image in the same pair having the same content; inputting the training image into the original image enhancement convolutional neural network to generate a training enhanced image; inputting the training enhanced image into an edge detector; generating, by the edge detector, a plurality of first edge maps; inputting the reference image into the edge detector; generating, by the edge detector, a plurality of second edge maps; computing one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, a respective one of the one or more first losses being computed based on a respective one of the plurality of first edge maps and a respective one of the plurality of second edge maps, the respective one of the plurality of first edge maps and the respective one of the plurality of second edge maps having the same scale; comparing the training enhanced image with the reference image to compute a second loss; and adjusting parameters in the original image enhancement convolutional neural network to at least minimize the one or more first losses and the second loss; wherein the edge detector is an edge detection neural network, the edge detection neural network comprising: a first group of convolutional layers, a first pooling layer, a first side layer, a second group of convolutional layers, a second pooling layer, a second side layer, a third group of convolutional layers, a third pooling layer, a third side layer, a fourth group of convolutional layers, a fourth pooling layer, a fourth side layer, a fifth group of convolutional layers, and a fifth side layer; each of the side layers comprising a convolutional layer and a de-convolutional layer connected in sequence; wherein the first group of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 64 3x3 convolutional kernels; the second group of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 128 3x3 convolutional kernels; the third group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 256 3x3 convolutional kernels; the fourth group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels; and the fifth group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels.

2. The computer-implemented image processing method of claim 1, wherein, The edge detection neural network comprises N edge map generators, each of the N edge map generators is configured to generate N edge maps; wherein, an nth edge map generator of the N edge map generators comprises an nth group of convolutional layers and an nth side layer configured to output an nth edge map of the same scale, 1≤n≤N; and When 1≤n<N, the nth edge map generator further comprises an nth down-sampling layer.

3. The computer-implemented image processing method of claim 2, wherein, Generating, by the edge detector, the plurality of first edge maps comprises: generating, by the nth edge map generator, an nth first edge map of the same scale; wherein, generating the nth first edge map comprises: inputting an (n-1)th first image into the nth group of convolutional layers; processing the (n-1)th first image by the nth group of convolutional layers to generate an nth processed first image; down-sampling the nth processed first image by the nth down-sampling layer to generate an nth down-sampled first image; processing the nth processed first image by the nth side layer to generate the nth first edge map of the same scale; and When n<N, inputting the nth down-sampled first image into an (n+1)th group of convolutional layers; wherein, when n=1, the (n-1)th first image is the training enhanced image; and When n>1, the (n-1)th first image is an (n-1)th down-sampled first image.

4. The computer-implemented image processing method of claim 2 or 3, wherein, Generating, by the edge detector, the plurality of second edge maps comprises: generating, by the nth edge map generator, an nth second edge map of the same scale; wherein, generating the nth second edge map comprises: inputting an (n-1)th second image into the nth group of convolutional layers; processing the (n-1)th second image by the nth group of convolutional layers to generate an nth processed second image; down-sampling the nth processed second image by the nth down-sampling layer to generate an nth down-sampled second image; processing the nth processed second image by the nth side layer to generate the nth second edge map of the same scale; and When n<N, inputting the nth down-sampled second image into an (n+1)th group of convolutional layers; wherein, when n=1, the (n-1)th second image is the reference image; and When n>1, the (n-1)th second image is an (n-1)th down-sampled second image.

5. The computer-implemented image processing method of claim 2, wherein, The nth side layer comprises a convolutional layer and a deconvolutional layer connected in sequence.

6. The computer-implemented image processing method of claim 1, wherein, Calculating, based on the plurality of first edge maps and the plurality of second edge maps, the one or more first losses comprises: calculating an nth loss based on an nth first edge map of the same scale and an nth second edge map of the same scale, 1≤n≤N.

7. The computer-implemented image processing method of claim 6, wherein, The one or more first losses include wherein Loss n represents the nth loss.

8. The computer-implemented image processing method of claim 6, wherein, Calculating the nth loss comprises: calculating one or more of an L2 loss, an L1 loss, and an MSE loss; wherein the L2 loss is computed using an L2 loss function with an expression of sum(|Y-f(x)| 2 ) where Y represents a pixel value in the nth second edge map, f(x) represents a pixel value in the nth first edge map, and sum represents a summation operation.

9. The computer-implemented image processing method of claim 6, wherein, The computing the nth loss includes computing one or more of an L2 loss, an L1 loss, and an MSE loss; where the L1 loss is computed using an L1 loss function expressed as sum(|Y-f(x)|), where Y represents a pixel value in the nth second edge map, f(x) represents a pixel value in the nth first edge map, and sum represents a summation operation.

10. The computer-implemented image processing method of claim 6, wherein, The computing the nth loss includes computing one or more of an L2 loss, an L1 loss, and an MSE loss; wherein the MSE loss is calculated using an expression for the MSE loss function avg(|Y-f(x)| 2 ) where Y represents a pixel value in the nth second edge map, f(x) represents a pixel value in the nth first edge map, and avg represents an averaging operation.

11. The computer-implemented image processing method of claim 1, wherein, The computing the second loss includes computing one or more of an L2 loss, an L1 loss, and an MSE loss; wherein the L2 loss is computed using an L2 loss function with an expression of sum(|Y-f(x)| 2 ) where Y represents a pixel value in the reference image, f(x) represents a pixel value in the training enhanced image, and sum represents a summation operation.

12. The computer-implemented image processing method of claim 1, wherein computing the second loss comprises: The computing includes computing one or more of an L2 loss, an L1 loss, and an MSE loss; where the L1 loss is computed using an L1 loss function expressed as sum(|Y-f(x)|), where Y represents a pixel value in the reference image, f(x) represents a pixel value in the training enhanced image, and sum represents a summation operation.

13. The computer-implemented image processing method of claim 1, wherein, The computing the second loss includes computing one or more of an L2 loss, an L1 loss, and an MSE loss; wherein the MSE loss is calculated using an expression for the MSE loss function avg(|Y-f(x)| 2 ) where Y represents a pixel value in the reference image, f(x) represents a pixel value in the training enhanced image, and avg represents an averaging operation.

14. The computer-implemented image processing method of claim 1, further comprising: The training the original image enhancement convolutional neural network is repeated using different pairs of training images and reference images.

15. A method of training an image enhancement convolutional neural network, comprising the steps of: obtaining a pair of training samples including a training image having a first definition and a reference image having a second definition, the second definition being greater than the first definition, at least portions of the training image and at least portions of the reference image in the same pair having the same content; inputting the training image into the image enhancement convolutional neural network to generate a training enhanced image; inputting the training enhanced image into an edge detector; generating a plurality of first edge maps by the edge detector; inputting the reference image into the edge detector; generating a plurality of second edge maps by the edge detector; computing one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, a respective one of the one or more first losses being computed based on a respective one of the plurality of first edge maps and a respective one of the plurality of second edge maps, the respective one of the plurality of first edge maps and the respective one of the plurality of second edge maps having the same scale; comparing the training enhanced image to the reference image to compute a second loss; and and adjusting parameters in the image enhancement convolutional neural network to at least minimize the one or more first losses and the second loss; wherein the edge detector is an edge detection neural network comprising: a first set of convolutional layers, a first pooling layer, a first side layer, a second set of convolutional layers, a second pooling layer, a second side layer, a third set of convolutional layers, a third pooling layer, a third side layer, a fourth set of convolutional layers, a fourth pooling layer, a fourth side layer, a fifth set of convolutional layers, and a fifth side layer; each of the side layers comprising a convolutional layer and a deconvolutional layer connected in sequence; wherein the first set of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 64 3x3 convolutional kernels; the second set of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 128 3x3 convolutional kernels; the third set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 256 3x3 convolutional kernels; the fourth set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels; and the fifth set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels.

16. The method of training an image enhancing convolutional neural network of claim 15, wherein, the image enhancement convolutional neural network comprises: a plurality of encoding modules; a plurality of decoding modules; and at least one cascade between a respective one of the plurality of encoding modules and a respective one of the plurality of decoding modules; wherein each of the plurality of encoding modules comprises a down-sampling convolutional layer and a plurality of first residual blocks arranged in series; each of the plurality of decoding modules comprises a plurality of second residual blocks arranged in series and an up-sampling convolutional layer.

17. An apparatus for training an image enhancement convolutional neural network configured to enhance an original image, the apparatus comprising: a memory; and one or more processors; wherein the memory and the one or more processors are connected to each other; and the memory stores computer-executable instructions for controlling the one or more processors to perform the following operations: obtaining a pair of training samples, the pair of training samples comprising a training image having a first definition and a reference image having a second definition, at least a portion of the training image and at least a portion of the reference image in the same pair having the same content; inputting the training image into the image enhancement convolutional neural network to generate a training enhanced image; inputting the training enhanced image into an edge detector; generating, by the edge detector, a plurality of first edge maps; inputting the reference image into the edge detector; generating, by the edge detector, a plurality of second edge maps; computing one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, a respective one of the one or more first losses being computed based on a respective one of the plurality of first edge maps and a respective one of the plurality of second edge maps, the respective one of the plurality of first edge maps and the respective one of the plurality of second edge maps having a same scale; and adjusting parameters in the image enhancement convolutional neural network to at least minimize the one or more first losses; wherein the edge detector is an edge detection neural network, the edge detection neural network comprising: a first set of convolutional layers, a first pooling layer, a first side layer, a second set of convolutional layers, a second pooling layer, a second side layer, a third set of convolutional layers, a third pooling layer, a third side layer, a fourth set of convolutional layers, a fourth pooling layer, a fourth side layer, a fifth set of convolutional layers, and a fifth side layer; each of the side layers comprising a convolutional layer and a de-convolutional layer connected in sequence; wherein the first set of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 64 3x3 convolutional kernels; the second set of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprising 128 3x3 convolutional kernels; the third set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 256 3x3 convolutional kernels; the fourth set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels; and the fifth set of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprising 512 3x3 convolutional kernels.

18. A computer program product for training an image enhancement convolutional neural network configured to enhance an original image, the computer program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions embodied thereon, the computer-readable instructions executable by a processor to cause the processor to perform: obtaining a pair of training samples, the pair of training samples comprising a training image having a first definition and a reference image having a second definition, at least a portion of the training image and at least a portion of the reference image in the same pair having a same content; inputting the training image into the image enhancement convolutional neural network to generate a training enhanced image; inputting the training enhanced image into an edge detector; generating, by the edge detector, a plurality of first edge maps; inputting the reference image into the edge detector; generating, by the edge detector, a plurality of second edge maps; computing one or more first losses based on the plurality of first edge maps and the plurality of second edge maps, a respective one of the one or more first losses being computed based on a respective one of the plurality of first edge maps and a respective one of the plurality of second edge maps, the respective one of the plurality of first edge maps and the respective one of the plurality of second edge maps having a same scale; and adjusting parameters in the image enhancement convolutional neural network to at least minimize the one or more first losses. adjusting parameters in the image enhancement convolutional neural network to at least minimize the one or more first losses; wherein the edge detector is an edge detection neural network comprising: a first group of convolutional layers, a first pooling layer, a first side layer, a second group of convolutional layers, a second pooling layer, a second side layer, a third group of convolutional layers, a third pooling layer, a third side layer, a fourth group of convolutional layers, a fourth pooling layer, a fourth side layer, a fifth group of convolutional layers, and a fifth side layer; each of the side layers comprises a convolutional layer and a deconvolutional layer connected in sequence; wherein the first group of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprises 64 3x3 convolutional kernels; the second group of convolutional layers comprises two convolutional layers and two activation layers, each of the two convolutional layers comprises 128 3x3 convolutional kernels; the third group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprises 256 3x3 convolutional kernels; the fourth group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprises 512 3x3 convolutional kernels; and the fifth group of convolutional layers comprises three convolutional layers and three activation layers, each of the three convolutional layers comprises 512 3x3 convolutional kernels.

Images