Image processing method, device and electronic device

Abstract

The embodiment of the present invention discloses an image processing method and device, and an electronic device. The method comprises: processing a first image through a U-net network to obtain a second image, wherein the second image is a noise map of the first image, wherein the U-net network comprises an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network, wherein the bottleneck network comprises a global pooling layer, a bilinear upscaling layer, and a 1×1 convolution layer; and generating a third image according to the first image and the second image, wherein the third image is a denoised map of the first image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device, and an electronic device. Background Art

[0002] After taking an image with a short exposure in low-light conditions, the image is not visually friendly because the image is dark and the colors and details are invisible to customers. It is necessary to denoise and enhance low-light images (i.e., images taken with a short exposure time under low illumination conditions). Existing denoising and image enhancement methods have poor image processing effects and require a lot of processing resources. Summary of the invention

[0003] Embodiments of the present invention provide an image processing method and apparatus, and an electronic device.

[0004] In a first aspect, the present invention provides an image processing method, comprising the following operations. A first image is processed through a U-net network to obtain a second image. The second image is a noise map of the first image, and the U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network, and the bottleneck network includes a global pooling layer, a bilinear magnification layer, and a 1×1 convolution layer. A third image is generated based on the first image and the second image, and the third image is a denoised image of the first image.

[0005] In a second aspect, the present invention provides an image processing device, comprising a first processing unit and a second processing unit. The first processing unit is used to process a first image through a U-net network to obtain a second image. The second image is a noise map of the first image, and the U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network, and the bottleneck network includes a global pooling layer, a bilinear magnification layer, and a 1×1 convolution layer. The second processing unit is used to generate a third image based on the first image and the second image. The third image is a denoised image of the first image.

[0006] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor. The memory stores a computer program. The processor is used to call and execute the computer program in the memory to execute the image processing method in the first aspect.

[0007] In a fourth aspect, the present invention provides a chip for implementing the image processing method in the first aspect. Specifically, the chip includes a processor. The processor is used to call and execute one or more computer programs in a memory so that a device equipped with the chip executes the image processing method in the first aspect.

[0008] In a fifth aspect, the present invention provides a computer-readable storage medium storing one or more computer programs, wherein the computer programs enable a processor to execute the image processing method in the first aspect.

[0009] In a sixth aspect, the present invention provides a computer program product comprising computer program instructions, wherein the computer program instructions enable a processor to execute the image processing method in the first aspect.

[0010] In a seventh aspect, the present invention provides a computer program, which, when executed by a processor, causes the processor to execute the image processing method in the first aspect.

[0011] By applying the above technical solution of the present invention, the image can be enhanced and the noise in the image can be completely removed, so that the image can be better displayed on the mobile terminal with rich details and vivid colors. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein, which are incorporated into the present invention and constitute a part of the present invention, are for a better understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to illustrate the present invention, but should not be interpreted as undue limitations on the present invention. In the drawings:

[0013] Figure 1 It is a flowchart of an image processing method according to an embodiment of the present invention.

[0014] Figure 2 4 is a network architecture diagram of a U-net network according to an embodiment of the present invention.

[0015] Figure 3 It is a schematic diagram of a process of packaging a Bayer array into four channels according to an embodiment of the present invention.

[0016] Figure 4 Schematic diagram of test results of an embodiment of the present invention.

[0017] Figure 5 is a schematic diagram of an image processing device according to an embodiment of the present invention.

[0018] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0019] Figure 7 It is a schematic diagram of the structure of a chip according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and operations, numerical expressions and numerical values ​​described in the embodiments is not intended to limit the scope of the present invention.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the invention, its application, or uses.

[0022] The well-known techniques, methods and devices described by ordinary technicians in the field will not be described in detail here, but when appropriate, these techniques, methods and devices should be considered as part of the description.

[0023] It should be noted that like reference numerals and letters refer to like elements in the drawings, and once an element is defined in one figure, it does not require further discussion in subsequent figures.

[0024] Embodiments of the present invention may be applied to computer systems / servers that may operate with other general or special computing system environments or configurations. Computing systems, environments and / or configurations suitable for computer systems / servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing technology environments including any of the above, and the like.

[0025] Taking photos with good perceptual quality in low-light conditions is extremely challenging due to the low signal-to-noise ratio (SNR). Extending the exposure time can obtain images with good visual effects. However, it is easy to introduce motion blur and is not always applicable in real life. In order to make low-light images with short exposure times visually credible, extensive research has been conducted, including denoising techniques aimed at removing noise in images caused by low-light conditions, and enhancement techniques developed to improve the perceptual quality of digital images. However, related denoising methods are usually proposed and evaluated based on synthetic data, which do not generalize well to real images, and low-light enhancement methods do not explicitly take noise into account. In addition, existing methods do not consider global context / color information. Although convolutional neural networks (CNNs) have driven many computer vision applications, CNN networks are usually too large to be implemented on resource-limited devices, such as mobile phones. Therefore, embodiments of the present invention develop a framework for denoising and enhancing low-light images using global context / color information integrated for the original image, which can be implemented on mobile devices.

[0026] To facilitate understanding of the technical solution of the present invention, technologies related to the embodiments of the present invention will be described below.

[0027] Related terms

[0028] (1) Image denoising: removes noise caused by low-level exposure while retaining details in the image.

[0029] (2) Image enhancement: The process of improving the quality of digitally stored images by processing them using algorithms.

[0030] Related technologies

[0031] (1) Image denoising

[0032] Traditional image denoising methods are usually implemented based on specific assumptions, such as image smoothness, sparsity, low rank, or self-similarity. Recently, methods based on deep cellular neural networks have been developed for image denoising. However, these methods are usually developed and evaluated mainly on synthetic data and cannot be well applied to real images.

[0033] In addition, there is a set of methods that utilize a set of images taken simultaneously to perform denoising. Although it usually yields good performance, they are very expensive in design and computation.

[0034] (2) Low-light image enhancement:

[0035] Many techniques have been developed for image enhancement, such as histogram equalization and gamma correction. Recently, more advanced methods have been proposed to deal with low-light image enhancement. However, these models are based on a strong assumption that the input image has a clean display without any noise. Therefore, for low-light image enhancement, a separate denoising operation should be used beforehand.

[0036] One particular approach is the “see-in-the-dark” model (SID), in which an encoder-decoder CNN is used for denoising and image enhancement at the same time. However, since global information is not considered in SID, color inconsistencies are found in the output image.

[0037] Most existing methods perform image denoising and enhancement separately, which is not only time-consuming but also computationally expensive. In addition, although SID jointly performs image denoising and enhancement and achieves good results, it does not consider global information that is critical to the color consistency of the output image. Embodiments of the present invention utilize the integration of global context to perform low-light image denoising and enhancement in a single shot. This enables the network to understand the global context / color information in order to better generate the final output. In addition, embodiments of the present invention develop new network architectures to make them usable on mobile devices with limited computing resources.

[0038] Figure 1 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the convolution method includes the following operations.

[0039] In 101, a first image is processed by a U-shaped convolutional neural network (U-shaped convolutional neural network) to obtain a second image, and the second image is a noise map of the first image, wherein the U-net network includes an encoding network, a decoding network and a bottleneck network located between the encoding network and the decoding network, and the bottleneck network includes a global pooling layer, a bilinear upscaling layer and a 1×1 convolution layer.

[0040] In an embodiment of the present invention, the U-net network is referred to as an encoder-decoder network. Figure 2 As shown in the figure, the U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network. The bottleneck network includes a global pooling layer, a bilinear upscaling layer, and a 1×1 convolution layer.

[0041] In an embodiment of the present invention, the first image is an original image to be processed. The first image may be an image captured by a mobile phone, or may be an image obtained in other ways, such as an image copied from an external storage device. The present invention has no limitation on the source of the first image.

[0042] In one embodiment, the first image is a low-light image. A low-light image refers to an image taken with a short exposure under low-light conditions. The solution of the embodiment of the present invention performs image denoising and enhancement through an encoder-decoder network to improve the perceptual quality of images taken under extremely low-light conditions.

[0043] In an embodiment of the present invention, the first image is processed by a U-net network to obtain a second image based on the following formula:

[0044] ΔI=f(I:w) (1)

[0045] Among them, ΔI represents the second image, I represents the first image, w represents a set of learnable parameters of the U-net network, and f represents the mapping function.

[0046] In a specific embodiment, given an input image (ie, the first image) I, a U-net network is used to learn a mapping ΔI=f(I:w) to estimate a noise map of the input image (ie, the second image).

[0047] In 102 , a third image is generated based on the first image and the second image, and the third image is a denoised image of the first image.

[0048] In a specific embodiment, by performing element-by-element aggregation between the input image (i.e., the first image) and the predicted noise map (i.e., the second image), i.e. The final denoised output (ie, the third image) is obtained.

[0049] Figure 2 The pathway of the U-net network provided in the present invention is described. The following subsections introduce the main components and methods.

[0050] In one embodiment of the present invention, before the first image is processed by the U-net network, the first image is preprocessed in the following manner: The Bayer arrays of the first image are packaged into multiple channels to obtain a packaged first image. The black level is subtracted from the packaged first image, and the obtained first image is multiplied by the magnification factor.

[0051] The plurality of channels are, for example, four channels. In a specific embodiment, Figure 3 As shown, the input original image (i.e., the first image) is a Bayer array, which is packed into four channels corresponding to R, G1, G2, and B channels, respectively. Then, the packed image is subtracted from the black level and multiplied by the magnification factor to match the brightness of the corresponding ground-truth image (also called the ground truth image).

[0052] As mentioned above, the U-net network is trained in an end-to-end manner to generate noise maps of input images taken under low light conditions. The input images have low contrast, low dynamic range, and are extremely noisy. The architecture and training strategy of the U-net network are described below.

[0053] (1) U-net network architecture

[0054] In an embodiment of the present invention, the U-net network includes an encoding network (also referred to as an encoder), a bottleneck network and a decoding network (also referred to as a decoder). Each network structure in the U-net network is described as follows.

[0055] A) Structure of the encoding network

[0056] like Figure 2 As shown, the encoding network includes multiple downsampling stages. Any of the multiple downsampling stages includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a bilinear downsampling layer. The output of the first 3×3 convolutional layer is connected to the input of the second 3×3 convolutional layer, and the output of the second 3×3 convolutional layer is connected to the input of the bilinear downsampling layer.

[0057] In one embodiment, the downsampling factor of the first downsampling stage among the multiple downsampling stages is 2.6, and the downsampling factors of the other downsampling stages except the first downsampling stage are all 2.7.

[0058] In one embodiment, if Figure 2 As shown, the encoding network also includes a full convolution stage. The full convolution stage is located after multiple downsampling stages, including the fifth 3×3 convolution layer and the sixth 3×3 convolution layer. The output of the last downsampling stage in the multiple downsampling stages is connected to the input of the fifth 3×3 convolution layer, the output of the fifth 3×3 convolution layer is connected to the input of the sixth 3×3 convolution layer, and the output of the sixth 3×3 convolution layer is connected to the global pooling layer of the bottleneck network.

[0059] B) Structure of the bottleneck network

[0060] like Figure 2 As shown, the bottleneck network includes a global pooling layer, a bilinear upscaling layer, and a 1×1 convolutional layer.

[0061] In the bottleneck network, the output of the global pooling layer is connected to the input of the bilinear amplification layer, and the output of the bilinear amplification layer and the output of the sixth 3×3 convolutional layer in the full convolution stage in A) are connected as the input of the 1×1 convolutional layer.

[0062] C) Structure of the decoding network

[0063] like Figure 2 As shown, the decoding network includes multiple amplification stages. The multiple amplification stages in the decoding network correspond to the various downsampling stages in the encoding network according to the resolution.

[0064] Any of the multiple amplification stages includes a first 1×1 convolutional layer, a bilinear upsampling layer, a second 1×1 convolutional layer, a third 3×3 convolutional layer, and a fourth 3×3 convolutional layer. The output of the first 1×1 convolutional layer is connected to the input of the bilinear upsampling layer, the output of the downsampling stage corresponding to the upsampling stage is connected to the input of the second 1×1 convolutional layer, the output of the bilinear upsampling layer and the output of the second 1×1 convolutional layer are connected as the input of the third 3×3 convolutional layer, and the output of the third 3×3 convolutional layer is connected to the input of the fourth 3×3 convolutional layer.

[0065] like Figure 2As shown in Figure 1, the U-net network consists of a stack of processing layers. The input image first passes through an encoder that contains a set of downsampling stages to extract abstract features and reduce the spatial resolution. After the bottleneck network (referred to as the bottleneck), the feature map will pass through a decoder that includes a set of upscaling operations.

[0066] (2) Reduce computing costs

[0067] Traditional convolutional neural networks use pooling layers to downsample feature maps to reduce computational costs. Specifically, the maximum value or average value is selected from a sliding window with a stride (also called a step size) on the input feature map (H×W) to generate an output feature map with a spatial size of H / stride×W / stride. Although common choices for stride size are 2 or 3, it lacks the flexibility to downsample feature maps using arbitrary downsampling rates. For example, in the experiments of the present invention, it was noted that using a stride size of 2 can produce good performance but requires a higher computational cost, while using a stride size of 3 can significantly reduce the computational cost, but the final performance is much lower. In order to solve the above problems, an embodiment of the present invention uses bilinear interpolation to downsample in the network to further reduce the computational cost of the network while maintaining the final performance. Specifically, the embodiment of the present invention does not downsample the feature map by 2x in width and height (or 2x downsampling) each time using a pooling layer with a stride of 2 as in a conventional U-net network, but downsamples the feature map after the first convolution level by 2.6x and the feature map after other convolution levels by 2.7x in the encoder of the proposed U-net network. It should be noted that the convolution level refers to the convolution layer in the downsampling level.

[0068] like Figure 2 As shown in Figure 1, to reduce memory and time costs, in the upscaling stage, the input layer is first processed using a 1×1 convolutional layer to reduce the number of channels, and then upscaled using bilinear interpolation. For the purpose of channel reduction, the layer with the same resolution from the downsampling stage is also processed using a 1×1 convolutional layer. The two output feature maps are then concatenated as the input of the subsequent layer.

[0069] (3) Local and global context information

[0070] Since the input image can have any resolution, the size of the feature map of the bottleneck in the network may still be large, and the global context / color information cannot be observed. Therefore, the generated image may have inconsistent colors at different locations. To solve this problem, embodiments of the present invention develop strategies to introduce global context / color information into the network. Figure 2As shown, a global pooling layer represented by a red block is introduced to extract global context information. After the pooling layer, an upscaling operation is used to rescale the feature map back to the same size as the input feature map to perform the following connection process as well as the convolution operation. However, since the input of the framework has an arbitrary size, this means that the feature map in the bottleneck has an arbitrary size, so the size of the kernel in the deconvolution layer is dynamic, which is unquestionable. Therefore, instead of using a deconvolution layer, a bilinear upscaling operation (represented by a black block) is adopted. The extracted feature maps are connected in a channel-wise manner after the 1×1 convolution layer to generate a final representation that contains the global information of the input image.

[0071] (4) Cost function

[0072] During the training process, the low-light image (i.e., the first image) is fed as input to the network (i.e., the U-net network), and the loss function between the system output and the corresponding long-exposure original image is calculated. The adopted loss function is the pixel value l 2 Distance and pixel gradient l 1 The weighted joint loss of distance is defined as follows:

[0073]

[0074] Among them, λ 1 and λ 2 They are all set to 1 according to experience. is defined by the following equation 1 loss:

[0075]

[0076] in, and I are the output mosaic image (demosaicked image) and the real mosaic image, respectively. is to divide its input by [-1,1] and [-1,1] T The finite difference operator for the convolution, Γ is the sRGB transfer function:

[0077]

[0078] a=0.0055 (3.2)

[0079] l 2 The loss is defined by the following equation:

[0080]

[0081] During the inference process, given an input original image, the output of the trained network is added to the input original image to obtain the output original image. Figure 4Some qualitative results are shown. Figure 4 As shown, column (a) shows a low-light original image input in the sRGB color space, and column (b) shows the system output of the U-net network according to an embodiment of the present invention in the sRGB color space.

[0082] Table 1 shows the results of a model using a max pooling model with a stride of 2 and a model using bilinear downsampling of 2.6x, 2.7x, 2.7x, and 2.7x after four convolutional stages in the encoder. The FLOPs (floating point operations per second) of this model is about 1 / 5 of that of the model using max pooling, while the PSNR (Peak Signal-to-Noise Ratio) measured on the Sony dataset is very similar, demonstrating the effectiveness of using bilinear downsampling to reduce computational cost while maintaining network accuracy.

[0083]

[0084] Table 1 compares the model using max pooling (stride 2) and the model using bilinear downsampling (bilinear downsampling is 2.6x, 2.7x, 2.7x, 2.7x) after the four convolutional stages of the encoder.

[0085] The present invention has at least the following beneficial effects: (1) greatly reducing the cost of developing advanced hardware for capturing better images; (2) greatly saving the cost of introducing an engine, allowing the mobile phone to remain stable under low-light conditions; (3) fully exploring the resolution without using super-pixel technology; (4) being able to save the space originally required for large sensors for better engineering design.

[0086] The embodiment of the present invention further provides an image processing device 500. Figure 5 As shown, the image processing apparatus 500 includes a first processing unit 501 and a second processing unit 502 .

[0087] The first processing unit 501 is used to process the first image through a U-net network to obtain a second image. The second image is a noise map of the first image. The U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network. The bottleneck network includes a global pooling layer, a bilinear upscaling layer, and a 1×1 convolution layer.

[0088] The second processing unit 502 is used to generate a third image according to the first image and the second image. The third image is a denoised image of the first image.

[0089] In at least one embodiment of the present invention, the apparatus further includes a third processing unit 503. The third processing unit 503 is configured to perform the following preprocessing on the first image: packing the Bayer array of the first image into multiple channels to obtain a packed first image, subtracting a black level from the packed first image, and then multiplying the obtained first image by a magnification factor.

[0090] In at least one embodiment of the present invention, the encoding network may include multiple downsampling stages. At least one downsampling stage of the multiple downsampling stages includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a bilinear downsampling layer. The output of the first 3×3 convolutional layer is connected to the input of the second 3×3 convolutional layer, and the output of the second 3×3 convolutional layer is connected to the input of the bilinear downsampling layer.

[0091] In at least one embodiment of the present invention, the downsampling factor of the first downsampling stage among the multiple downsampling stages may be 2.6, and the downsampling factors of the other downsampling stages except the first downsampling stage may be 2.7.

[0092] In at least one embodiment of the present invention, the decoding network includes a plurality of amplification stages, and the plurality of amplification stages in the decoding network correspond to the respective downsampling stages in the encoding network according to the resolution.

[0093] Each of the multiple amplification stages includes a first 1×1 convolution layer, a bilinear upsampling layer, a second 1×1 convolution layer, a third 3×3 convolution layer and a fourth 3×3 convolution layer.

[0094] The output of the first 1×1 convolutional layer is connected to the input of the bilinear upsampling layer. The output of the downsampling stage corresponding to the amplification stage is connected to the input of the second 1×1 convolutional layer. The output of the bilinear upsampling layer and the output of the second 1×1 convolutional layer are connected as the input of the third 3×3 convolutional layer, and the output of the third 3×3 convolutional layer is connected to the input of the fourth 3×3 convolutional layer.

[0095] In at least one embodiment of the present invention, the encoding network further comprises a full convolutional stage. The full convolutional stage is located after the multiple downsampling stages and comprises a fifth 3×3 convolutional layer and a sixth 3×3 convolutional layer.

[0096] The output of the last downsampling stage in the multiple downsampling stages is connected to the input of the fifth 3×3 convolutional layer. The output of the fifth 3×3 convolutional layer is connected to the input of the sixth 3×3 convolutional layer, and the output of the sixth 3×3 convolutional layer is connected to the global pooling layer of the bottleneck network.

[0097] In at least one embodiment of the present invention, in the bottleneck network, the global pooling layer is connected to the input of the bilinear upscaling layer, and in the full convolution stage, the output of the bilinear upscaling layer and the output of the sixth 3×3 convolutional layer are connected as the input of the 1×1 convolutional layer.

[0098] It should be understood that, in the embodiments of the present invention, the description of the image processing apparatus may be understood by referring to the above description of the image processing method.

[0099] Figure 6 6 is a schematic diagram of the structure of an electronic device 600 according to an embodiment of the present invention. The electronic device may be a terminal, such as a mobile phone, a tablet computer, etc. Figure 6 The electronic device 600 shown includes a processor 610. The processor 610 can call and execute a computer program in a memory to implement the method in the embodiment of the present invention.

[0100] In at least one embodiment, Figure 6 As shown, the communication device 600 may further include a memory 620. The processor 610 may call and execute a computer program in the memory 620 to implement the method in the embodiment of the present invention.

[0101] The memory 620 may be a device provided separately from the processor 610 , or may be integrated into the processor 610 .

[0102] In at least one embodiment, Figure 6 As shown, the electronic device 600 further includes a transceiver 630. The processor 610 controls the transceiver 630 to communicate with another device. Specifically, the processor 610 controls the transceiver 630 to send information or data to another device, or to receive information or data from another device.

[0103] The transceiver 630 includes a transmitter and a receiver. The transceiver 630 also includes one or more antennas.

[0104] In at least one embodiment, the electronic device 600 is specifically a network device in an embodiment of the present invention. In each method embodiment of the present invention, the electronic device 600 can implement a corresponding process implemented by the network device, and for the sake of brief description, the process will not be elaborated here.

[0105] In at least one embodiment, the communication device 600 may be specifically a terminal / mobile terminal in the embodiment of the present invention. In each method embodiment of the present invention, the communication device 600 may implement the corresponding processing implemented by the terminal / mobile terminal, which will not be elaborated here for the sake of brief description.

[0106] Figure 7 is a schematic diagram of the structure of a chip according to an embodiment of the present invention. Figure 7 As shown, the chip 700 includes a processor 710. The processor 710 calls and executes a computer program in a memory to implement the method in the embodiment of the present invention.

[0107] In at least one embodiment, Figure 7 As shown, the chip 700 further includes a memory 720. The processor 710 calls and executes the computer program in the memory 720 to implement the method in the embodiment of the present invention.

[0108] The memory 720 is a device provided separately from the processor 710 , or may be integrated into the processor 710 .

[0109] In at least one embodiment, the chip 700 further includes an input interface 730. The processor 710 controls the input interface 730 to communicate with another device or chip. Specifically, the processor 710 can control the input interface 730 to obtain information or data from another device or chip.

[0110] In at least one embodiment, the chip 700 further includes an output interface 740. The processor 710 controls the output interface 740 to communicate with another device or chip. Specifically, the processor 710 controls the output interface 740 to send information or data to another device or chip.

[0111] In at least one embodiment, the chip can be applied to the network device in the embodiment of the present invention. In each method embodiment of the present invention, the chip can implement the corresponding processing implemented by the network device, which will not be elaborated here for the sake of brief description.

[0112] In at least one embodiment, the chip can be applied to the terminal / mobile terminal in the embodiment of the present invention. In each method embodiment of the present invention, the chip can implement the corresponding processing implemented by the terminal / mobile terminal, which will not be elaborated here for the sake of brief description.

[0113] It should be understood that in the embodiments of the present invention, the chip is also referred to as a system-on-chip, a system-on-chip, a chip system, or a system on a chip.

[0114] It should be understood that in an embodiment of the present invention, the processor is an integrated circuit chip with signal processing capabilities. In the implementation process, each operation of the method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), an application-specific integrated circuit (ASIC, Application Specific Integrated Circuit), a field programmable gate array (FPGA, Field Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices and discrete hardware components. Each method, operation and logic block diagram disclosed in the embodiment of the present invention is implementable or executable. The general-purpose processor can be a microprocessor, or the processor can also be any related processor, etc. The operation of the method disclosed in conjunction with the embodiment of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in a decoding processor. The software module is located in a common storage medium in the art, such as a random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), or register. The storage medium is located in the memory. The processor reads the information in the memory and completes the operation of the above method in combination with the hardware of the processor.

[0115] It is understood that the memory in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include a volatile memory and a non-volatile memory. The non-volatile memory may be a ROM, a PROM, an erasable PROM (EPROM), an EEPROM or a flash memory. The volatile memory may be a RAM and used as an external cache. RAM in various forms is described illustratively but not restrictively, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DR RAM). It should be appreciated that memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

[0116] An embodiment of the present invention also provides a computer-readable storage medium for storing one or more computer programs.

[0117] In at least one embodiment, the computer readable storage medium can be applied to the network device of the embodiment of the present invention. In each method embodiment of the present invention, the computer program can enable the processor to perform the corresponding processing implemented by the network device, which will not be elaborated here for the sake of brevity.

[0118] In at least one example, the computer-readable storage medium may be applied to the terminal / mobile terminal of the embodiment of the present invention. In each method embodiment of the present invention, the computer program may enable the processor to perform the corresponding processing implemented by the terminal / mobile terminal, which will not be described in detail here for the sake of brief description.

[0119] An embodiment of the present invention further provides a computer program product, which includes one or more computer program instructions.

[0120] In at least one embodiment, the computer program product can be applied to the network device of the embodiment of the present invention. In each method embodiment of the present invention, the computer program instructions can enable the processor to perform the corresponding processing implemented by the network device, which will not be elaborated here for the sake of brevity.

[0121] In at least one example, the computer program product may be applied to the terminal / mobile terminal of the embodiment of the present invention. In each method embodiment of the present invention, the computer program instructions may enable the processor to perform the corresponding processing implemented by the terminal / mobile terminal, which will not be described in detail here for the sake of brevity.

[0122] An embodiment of the present invention also provides a computer program.

[0123] In at least one embodiment, the computer program can be applied to the network device of the embodiment of the present invention. When executed by a processor, the computer program enables the processor to perform the corresponding processing implemented by the network device in each method embodiment of the present invention, which will not be elaborated here for the sake of brief description.

[0124] In at least one example, a computer program may be applied in a terminal / mobile terminal of an embodiment of the present invention. When executed by a processor, the computer program enables the processor to perform the corresponding processing implemented by the terminal / mobile terminal in each method embodiment of the present invention, which will not be described in detail here for the sake of brevity.

[0125] Those skilled in the art will recognize that the units and algorithmic operations of each example described in conjunction with the embodiments disclosed in the present invention can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the functions of each specific application, but such implementation should fall within the scope of the present invention.

[0126] Those skilled in the art can clearly understand that the specific working processes of the above-mentioned systems, devices and units can refer to the corresponding processes in the method embodiments, and for the sake of convenience and brief description, they will not be repeated here.

[0127] In some embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in another way. For example, the device embodiments described above are only schematic, and for example, the division of units is only a logical function division, and other division methods can be adopted during actual implementation. For example, multiple units or components can be combined or integrated into another system, or certain features can be ignored or not performed. In addition, the coupling or direct coupling or communication connection between each component shown or discussed can be an indirect coupling or communication connection, implemented through some interfaces of the device or unit, and can be electrical and mechanical connections or in other forms.

[0128] The units described as independent parts may or may not be physically separated, and the parts shown as units may or may not be physical units, i.e., they may be located at the same location, or they may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing unit, each unit may exist physically independently, and two or more units may be integrated into one unit.

[0130] When implemented in the form of a software functional unit and sold or used as an independent product, the function can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution or part of the technical solution that the present invention makes a substantial contribution or partial contribution to the known technology can be implemented in the form of a software product, and the computer software product is stored in a storage medium, which includes a computer device (which can be a personal computer, server, network device, etc.) configured to enable the execution of all or part of the operation of the method in each embodiment of the present invention. The above-mentioned storage medium includes: various media capable of storing program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk.

[0131] The above are only specific embodiments of the present invention and do not constitute a limitation on the protection scope of the present invention. Within the technical scope disclosed by the present invention, any changes or substitutions obvious to those skilled in the art should belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Processing the first image through a U-net network to obtain a second image, wherein the second image is a noise map of the first image, wherein the U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network, and the bottleneck network includes a global pooling layer, a bilinear upscaling layer, and a 1×1 convolution layer; as well as A third image is generated according to the first image and the second image, and the third image is a denoised image of the first image.

2. The method according to claim 1, characterized in that Before processing the first image through the U-net network, the method further includes: Packing the Bayer array of the first image into a plurality of channels to obtain a packed first image; and The black level is subtracted from the packaged first image, and the obtained first image is multiplied by the magnification factor.

3. The method according to claim 1 or 2, characterized in that: The encoding network comprises a plurality of downsampling stages, at least one of the plurality of downsampling stages comprising a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a bilinear downsampling layer; The output of the first 3×3 convolutional layer is connected to the input of the second 3×3 convolutional layer, and the output of the second 3×3 convolutional layer is connected to the input of the bilinear downsampling layer.

4. The method according to claim 3, characterized in that A downsampling factor of a first downsampling stage among the multiple downsampling stages is 2.6, and a downsampling factor of the other downsampling stages except the first downsampling stage is 2.

7.

5. The method according to claim 3 or 4, characterized in that: The decoding network comprises a plurality of amplification stages, and the plurality of amplification stages in the decoding network correspond to the plurality of downsampling stages in the encoding network according to resolutions; Each of the plurality of amplification stages comprises a first 1×1 convolutional layer, a bilinear upsampling layer, a second 1×1 convolutional layer, a third 3×3 convolutional layer, and a fourth 3×3 convolutional layer; Among them, the output of the first 1×1 convolutional layer is connected to the input of the bilinear upsampling layer, the output of the downsampling stage corresponding to the amplification stage is connected to the input of the second 1×1 convolutional layer, the output of the bilinear upsampling layer and the output of the second 1×1 convolutional layer are connected as the input of the third 3×3 convolutional layer, and the output of the third 3×3 convolutional layer is connected to the input of the fourth 3×3 convolutional layer.

6. The method according to claim 5, characterized in that The encoding network further includes a full convolutional stage, the full convolutional stage is located after the multiple downsampling stages, and the full convolutional stage includes a fifth 3×3 convolutional layer and a sixth 3×3 convolutional layer; Among them, the output of the last downsampling stage among the multiple downsampling stages is connected to the input of the fifth 3×3 convolutional layer, the output of the fifth 3×3 convolutional layer is connected to the input of the sixth 3×3 convolutional layer, and the output of the sixth 3×3 convolutional layer is connected to the global pooling layer of the bottleneck network.

7. The method according to claim 6, characterized in that In the bottleneck network, the output of the global pooling layer is connected to the input of the bilinear amplification layer, and the output of the bilinear amplification layer and the output of the sixth 3×3 convolutional layer are connected as the input of the 1×1 convolutional layer.

8. An image processing device, characterized in that: include: A first processing unit, configured to process the first image through a U-net network to obtain a second image, wherein the second image is a noise map of the first image, wherein the U-net network includes an encoding network, a decoding network, and a bottleneck network located between the encoding network and the decoding network, wherein the bottleneck network includes a global pooling layer, a bilinear upscaling layer, and a 1×1 convolution layer; as well as The second processing unit is configured to generate a third image according to the first image and the second image, wherein the third image is a denoised image of the first image.

9. The device according to claim 8, characterized in that The device further includes: a third processing unit, which is configured to, before the first processing unit processes the first image, Packing the Bayer array of the first image into a plurality of channels to obtain a packed first image; and The black level is subtracted from the packaged first image, and the obtained first image is multiplied by the magnification factor.

10. The device according to claim 8 or 9, characterized in that The encoding network comprises a plurality of downsampling stages, at least one of the plurality of downsampling stages comprising a first 3×3 convolutional layer, a second 3×3 convolutional layer, and a bilinear downsampling layer; The output of the first 3×3 convolutional layer is connected to the input of the second 3×3 convolutional layer, and the output of the second 3×3 convolutional layer is connected to the input of the bilinear downsampling layer.

11. The device according to claim 10, characterized in that A downsampling factor of a first downsampling stage among the multiple downsampling stages is 2.6, and a downsampling factor of the other downsampling stages except the first downsampling stage is 2.

7.

12. The device according to claim 10 or 11, characterized in that The decoding network comprises a plurality of amplification stages, and the plurality of amplification stages in the decoding network correspond to the plurality of downsampling stages in the encoding network according to resolutions; Each of the plurality of amplification stages comprises a first 1×1 convolutional layer, a bilinear upsampling layer, a second 1×1 convolutional layer, a third 3×3 convolutional layer, and a fourth 3×3 convolutional layer; Among them, the output of the first 1×1 convolutional layer is connected to the input of the bilinear upsampling layer, the output of the downsampling stage corresponding to the amplification stage is connected to the input of the second 1×1 convolutional layer, the output of the bilinear upsampling layer and the output of the second 1×1 convolutional layer are connected as the input of the third 3×3 convolutional layer, and the output of the third 3×3 convolutional layer is connected to the input of the fourth 3×3 convolutional layer.

13. The device according to claim 12, characterized in that The encoding network further includes a full convolutional stage, the full convolutional stage is located after the multiple downsampling stages, and the full convolutional stage includes a fifth 3×3 convolutional layer and a sixth 3×3 convolutional layer; Among them, the output of the last downsampling stage among the multiple downsampling stages is connected to the input of the fifth 3×3 convolutional layer, the output of the fifth 3×3 convolutional layer is connected to the input of the sixth 3×3 convolutional layer, and the output of the sixth 3×3 convolutional layer is connected to the global pooling layer of the bottleneck network.

14. The device according to claim 13, characterized in that In the bottleneck network, the global pooling layer is connected to the input of the bilinear amplification layer, and the output of the bilinear amplification layer and the output of the sixth 3×3 convolutional layer are connected as the input of the 1×1 convolutional layer.

15. An electronic device, characterized in that: include: memory for storing computer programs; as well as A processor, configured to call and execute the computer program stored in the memory to perform the method according to any one of claims 1 to 7.

16. A chip, characterized in that: The device comprises a processor for calling and executing a computer program stored in a memory so that a device equipped with the chip executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that include: The computer program instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

19. A computer program, characterized in that When the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images