Image processing methods, apparatus and electronic devices

Abstract

This application provides an image processing method, apparatus, and electronic device, relating to the field of image processing technology. These methods can more effectively suppress false color and restore more high-frequency details during image depigmentation, denoising, and super-resolution processing, thereby improving image quality. The image processing method includes: acquiring an RGBW image from an image sensor, the RGBW image including a red channel, a green channel, a blue channel, and a white channel; inputting the RGBW image into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model, the convolutional neural network model including a cascaded feature extraction module, a reconstruction module, and an upsampling module, the feature extraction module and the reconstruction module being used for the depigmentation and denoising processes, and the upsampling module being used for the super-resolution processing process. The RGB image includes a red channel, a green channel, and a blue channel, and the resolution of the RGB image is greater than that of the RGBW image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and electronic device. Background Technology

[0002] In computational photography, obtaining high-quality, high-resolution images is attracting increasing attention from industry and academia. However, for electronic devices such as mobile phones, acquiring such images is extremely difficult due to hardware limitations. First, the way images are captured by a single sensor results in incomplete color sampling rates; second, noise is inevitably introduced during image acquisition; third, the limited number of pixels in the sensors of mobile devices restricts image resolution; finally, how to efficiently complete image restoration tasks under limited computing resources is also a crucial problem that urgently needs to be solved. Therefore, the image signal processing (ISP) workflow needs to perform a hybrid task of image demosaicing, denoising, and super-resolution (SR).

[0003] However, in existing technologies, images processed through depigmentation, denoising, and super-resolution are prone to false colors and blurred details. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and electronic device that can more effectively suppress false color and restore more high-frequency details during image depigmentation, denoising, and super-resolution processing, thereby improving image quality.

[0005] In a first aspect, an image processing method is provided, comprising: acquiring an RGBW image from an image sensor, the RGBW image including a red channel, a green channel, a blue channel, and a white channel; inputting the RGBW image into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model, the convolutional neural network model including a feature extraction module, a reconstruction module, and an upsampling module cascaded in sequence, the feature extraction module and the reconstruction module being used for de-mosaicing and denoising processes, the upsampling module being used for super-resolution processing, the RGB image including a red channel, a green channel, and a blue channel, the resolution of the RGB image being greater than the resolution of the RGBW image.

[0006] In one possible implementation, the feature extraction module includes a first ordinary block and three first residual blocks cascaded together; each first residual block includes two cascaded residual processing units, each residual processing unit includes a second ordinary block and an activation function layer cascaded together, and the output of each residual processing unit is the result of adding the output of the activation function layer therein to the input of the residual processing unit therein; either the first ordinary block or the second ordinary block includes a convolutional layer, an activation function layer and a convolutional layer cascaded together.

[0007] In one possible implementation, the kernel size of any convolutional layer is 3×3, the stride is 1, and the padding is 1. The first convolutional layer in the first normal block has 4 input channels and 64 output channels. Except for the first convolutional layer in the first normal block, the input channels and output channels of any activation function layer and any convolutional layer in the feature extraction module are 64 and 64 respectively.

[0008] In one possible implementation, the reconstruction module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit; the first processing unit includes a third ordinary block and a second residual block cascaded sequentially, the third ordinary block being used to acquire the output of the feature extraction module, and the second residual block being used to output to the second processing unit and the fifth processing unit; the second processing unit includes a first convolutional layer, a fourth ordinary block, and a third residual block cascaded sequentially, the first convolutional layer being used to acquire one output of the second residual block, and the third residual block being used to output to the third processing unit and the fourth processing unit; the third processing unit includes a second convolutional layer, a fifth ordinary block, and a third residual block cascaded sequentially. The system comprises a pass block, a fourth residual block, a sixth ordinary block, and a first transposed convolutional layer. A second convolutional layer is used to obtain one output of the third residual block, and the first transposed convolutional layer is used to output to the fourth processing unit. The fourth processing unit includes a first splicing layer, a seventh ordinary block, a fifth residual block, and a second transposed convolutional layer, which are cascaded in sequence. The first splicing layer is used to obtain another output of the third residual block and the output of the first transposed convolutional layer and splice them together. The second transposed convolutional layer is used to output to the fifth processing unit. The fifth processing unit includes a second splicing layer and an eighth ordinary block, which are cascaded in sequence. The second splicing layer is used to obtain another output of the second residual block and the output of the second transposed convolutional layer and splice them together.

[0009] In one possible implementation, the kernel size of the first and second convolutional layers is 3×3, the stride is 2, and the padding is 1; the kernel size of the first and second transposed convolutional layers is 2×2, the stride is 2, and the padding is 0; the number of input channels and output channels of the third normal block and the second residual block is 64; the number of input channels of the first convolutional layer is 64 and the number of output channels is 96; the number of input channels and output channels of the fourth normal block, the third residual block, and the fifth residual block is 96; and the number of input channels of the second convolutional layer is 96 and the number of output channels is 12. 8; The fifth normal block, the fourth residual block, and the sixth normal block have 128 input channels and 96 output channels; the first transposed convolutional layer has 96 input channels and 192 output channels; the seventh normal block has 192 input channels and 96 output channels; the second transposed convolutional layer has 96 input channels and 64 output channels; the second splicing layer has 64 input channels and 128 output channels; the eighth normal block has 128 input channels and 64 output channels.

[0010] In one possible implementation, the upsampling module includes a third convolutional layer, a first activation function layer, a pixel shuffling layer, a fourth convolutional layer, a second activation function layer, and a fifth convolutional layer cascaded in sequence.

[0011] In one possible implementation, the third convolutional layer has 64 input channels and 256 output channels; the first activation function layer has 256 input channels and 256 output channels; the pixel shuffling layer has 256 input channels and 64 output channels; the fourth convolutional layer and the second activation function layer have 64 input channels and 64 output channels; and the fifth convolutional layer has 64 input channels and 3 output channels.

[0012] In one possible implementation, before inputting the RGBW image into the convolutional neural network model, the method further includes: acquiring an RGB sample image; downsampling, adding noise, and adding a white channel to the RGB sample image to obtain an RGBW sample image, wherein the resolution of the RGBW sample image is smaller than that of the RGB sample image; and using the RGB sample image and the RGBW sample image as training data to train the convolutional neural network model.

[0013] In one possible implementation, noise addition includes adding Gaussian noise, salt and pepper noise, speckle noise, Poisson noise, Gaussian blur, and light intensity variation.

[0014] Secondly, an image processing apparatus is provided, comprising: an image acquisition unit for acquiring an RGBW image from an image sensor, the RGBW image including a red channel, a green channel, a blue channel, and a white channel; and an image processing unit for inputting the RGBW image into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model, the convolutional neural network model including a feature extraction module, a reconstruction module, and an upsampling module cascaded in sequence, the feature extraction module and the reconstruction module being used for de-mosaicing and denoising processes, the upsampling module being used for super-resolution processing, the RGB image including a red channel, a green channel, and a blue channel, the resolution of the RGB image being greater than the resolution of the RGBW image.

[0015] Thirdly, an image processing apparatus is provided, comprising: a processor and a memory, wherein the memory is used to store at least one instruction, which is loaded by the processor and executed to implement the image processing method described above.

[0016] Fourthly, an electronic device is provided, including the image processing apparatus described above.

[0017] In the image processing methods, apparatuses, and electronic devices of this application, RGBW images are used as input to a convolutional neural network. Based on the process of joint demosaicing, denoising, and super-resolution processing using a convolutional neural network, thanks to the high signal-to-noise ratio white channel additional information captured by RGBWCFA, false color can be suppressed more effectively, more high-frequency details can be restored, and a higher-quality reconstructed image can be generated, thereby improving image quality. In addition, in this application embodiment, through the model structure of a cascaded feature extraction module, reconstruction module, and upsampling module, image demosaicing, denoising, and super-resolution processing can be completed more efficiently, end-to-end, and synchronously through the same neural network. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an image channel distribution in RGB CFA mode;

[0019] Figure 2 This is a schematic diagram of another image channel distribution in RGB CFA mode;

[0020] Figure 3 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a neural network architecture in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the channel distribution of an RGBW image in an embodiment of this application;

[0023] Figure 6 for Figure 4 A schematic diagram of a specific architecture of the feature extraction module in the middle;

[0024] Figure 7 for Figure 4 A schematic diagram of a specific architecture of the reconstruction module;

[0025] Figure 8 for Figure 4 A schematic diagram of a specific architecture of the upsampling module;

[0026] Figure 9 This is a flowchart illustrating another image processing method in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram illustrating the effect of image processing using a convolutional neural network model in an embodiment of this application.

[0028] Figure 11 A diagram comparing the results obtained by using a convolutional neural network model with different inputs;

[0029] Figure 12 This is a diagram comparing the output results of the model after processing with different noise addition methods. Detailed Implementation

[0030] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0031] Before describing the embodiments of this application, related technologies are first introduced. One related technology for image de-mosaicing, denoising, and super-resolution processing first uses de-mosaic technology to perform channel-by-channel interpolation on the sparse raw data of the red, green, and blue channels to recover a complete color image. Then, traditional or deep learning methods are used to perform denoising and super-resolution on the color image step by step. However, this sequential approach is prone to error accumulation and propagation, and artifacts introduced in the preceding steps can severely affect the performance of subsequent steps. For example, de-mosaic easily introduces pseudo-colors, which are further amplified by super-resolution. Furthermore, the superposition of different methods increases the complexity of the algorithm. Additionally, the image processing in related technologies only focuses on the traditional RGB Color Filter Array (CFA) mode, represented by Bayer. In the RGB CFA mode, such as... Figure 1 and Figure 2As shown, since the sampling frequency of the R and B components is only half that of the G component, they are very sparse, making the recovery of the R and B components difficult. This easily leads to false color and loss of detail, resulting in a decrease in visual quality. Therefore, the technical solution of the embodiments of this application is provided to solve the above-mentioned technical problems. The technical solution of the embodiments of this application is described below.

[0032] like Figure 3 and Figure 4 As shown, this application provides an image processing method, including:

[0033] Step 101: Acquire an RGBW image from the image sensor, such as a raw image. Figure 5 As shown, an RGBW image includes a red channel (R), a green channel (G), a blue channel (B), and a white channel (W). An RGBW image can be decomposed into a Bayer image plus a grayscale image of the same size white channel. By introducing an additional grayscale image of the same size white channel, a red / green / blue / white four-color filter array (RGBW CFA) mode is formed, which can increase rich details and edge information, alleviate the problems existing in the RGB CFA mode, and help reconstruct high-resolution and high-quality color images.

[0034] Step 102: Input the RGBW image into the convolutional neural network model to obtain the RGB image output by the convolutional neural network model. The convolutional neural network model includes a feature extraction module 100, a reconstruction module 200, and an upsampling module 300 cascaded in sequence. The feature extraction module 100 and the reconstruction module 200 are used for the de-mosaic and denoising process, and the upsampling module 300 is used for the super-resolution process. The RGB image includes a red channel R, a green channel G, and a blue channel B. The resolution of the RGB image is greater than that of the RGBW image.

[0035] Specifically, the convolutional neural network model is a pre-trained model, and the specific training process will be explained later. Because the information in the raw data of the RGB CFA mode is very sparse across all channels (especially the R and B channels), it results in significant information loss and discontinuities at edges, making it very easy for false color and blurred details to appear in the reconstructed image. This situation is even more pronounced when noise is present. Therefore, in this embodiment, an RGBW image is used as the input to the convolutional neural network. Based on the process of combining demosaicing, denoising, and super-resolution processing using the convolutional neural network, thanks to the high signal-to-noise ratio white channel information captured by the RGBW CFA, false color can be suppressed more effectively, more high-frequency details can be restored, and a higher-quality reconstructed image can be generated, thereby improving image quality. Furthermore, in this embodiment, through the cascaded feature extraction module, reconstruction module, and upsampling module model structure, image demosaicing, denoising, and super-resolution can be completed more efficiently, end-to-end, and synchronously using the same neural network.

[0036] In one possible implementation, such as Figure 6 As shown, the feature extraction module 100 includes a first simple block and three first residual blocks cascaded together. Each first residual block includes two cascaded residual processing units. Each residual processing unit includes a second simple block and an activation function layer cascaded together. The output of each residual processing unit is the result of adding the output of the activation function layer and the input of the residual processing unit. Either the first simple block or the second simple block includes a convolutional layer, an activation function layer, and a convolutional layer cascaded together.

[0037] In one possible implementation, the kernel size of any convolutional layer is 3×3, the stride is 1, and the padding is 1. The first convolutional layer in the first normal block has 4 input channels and 64 output channels. Except for the first convolutional layer in the first normal block, the activation function layer and any convolutional layer in the feature extraction module have 64 input channels and 64 output channels. The numbers within the rectangles in the diagram represent the number of channels; for example, 4 / 64 means 4 input channels and 64 output channels, and 64 / 64 means both input and output channels are 64. A 4-channel RGBW image is input to the first normal block of the feature extraction module, and after processing by the feature extraction module, 64 channels of extracted features are output.

[0038] In one possible implementation, such as Figure 7As shown, the reconstruction module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit. The first processing unit includes a third ordinary block and a second residual block cascaded together. The third ordinary block is used to obtain the output of the feature extraction module, i.e., to obtain the features extracted after processing by the feature extraction module. The second residual block is used to output to the second and fifth processing units. The second processing unit includes a first convolutional layer, a fourth ordinary block, and a third residual block cascaded together. The first convolutional layer is used to obtain one output of the second residual block, and the third residual block is used to output to the third and fourth processing units. The third processing unit includes a second convolutional layer, a fourth ordinary block, and a third residual block cascaded together. The system consists of a first concatenation layer, a fifth ordinary block, a fourth residual block, a sixth ordinary block, and a first transposed convolutional layer. A second convolutional layer is used to obtain one output of the third residual block, and the first transposed convolutional layer outputs to the fourth processing unit. The fourth processing unit comprises a first concatenation layer, a seventh ordinary block, a fifth residual block, and a second transposed convolutional layer, all cascaded together. The first concatenation layer obtains the other output of the third residual block and the output of the first transposed convolutional layer, and concatenates them. The second transposed convolutional layer outputs to the fifth processing unit. The fifth processing unit comprises a second concatenation layer and an eighth ordinary block, all cascaded together. The second concatenation layer obtains the other output of the second residual block and the output of the second transposed convolutional layer, and concatenates them. The first and second convolutional layers reduce the feature map size by half, and the first and second transposed convolutional layers double the original size. The first concatenation layer and the second impoverished layer represent the concatenation of two input features along the channel dimension. The reconstruction module has a U-shaped structure designed to further reduce the impact of noise and blurring, reconstructing high-quality features. This module can increase the receptive field of the network through downsampling, which helps to restore details and textures. At the same time, the fusion of features at different scales also helps to improve the network's performance. Figure 7 The structure of any one of the second, third, fourth, and fifth residual blocks can be the same as... Figure 6 The structure of the first residual block is the same, so it will not be described again here. Figure 7 The arrows between some structural boxes are omitted. The extracted features output by the feature extraction module are input to the third ordinary block of the reconstruction module. After processing by the reconstruction module, the reconstructed features with 64 channels are output.

[0039] In one possible implementation, the kernel size of the first and second convolutional layers is 3×3, the stride is 2, and the padding is 1; the kernel size of the first and second transposed convolutional layers is 2×2, the stride is 2, and the padding is 0; the number of input channels and output channels of the third normal block and the second residual block is 64; the number of input channels of the first convolutional layer is 64 and the number of output channels is 96; the number of input channels and output channels of the fourth normal block, the third residual block, and the fifth residual block is 96; and the number of input channels of the second convolutional layer is 96 and the number of output channels is 12. 8; The fifth normal block, the fourth residual block, and the sixth normal block have 128 input channels and 96 output channels; the first transposed convolutional layer has 96 input channels and 192 output channels; the seventh normal block has 192 input channels and 96 output channels; the second transposed convolutional layer has 96 input channels and 64 output channels; the second splicing layer has 64 input channels and 128 output channels; the eighth normal block has 128 input channels and 64 output channels.

[0040] In one possible implementation, such as Figure 8 As shown, the upsampling module comprises a third convolutional layer, a first activation function layer, a pixel shuffle layer, a fourth convolutional layer, a second activation function layer, and a fifth convolutional layer, cascaded sequentially. The upsampling module receives high-quality reconstructed features from the reconstruction module, upscales them, and aggregates their channels, ultimately outputting a complete, super-resolution RGB image. The pixel shuffle layer performs upsampling, i.e., feature map magnification.

[0041] In one possible implementation, the third convolutional layer has 64 input channels and 256 output channels; the first activation function layer has 256 input channels and 256 output channels; the pixel shuffling layer has 256 input channels and 64 output channels; the fourth convolutional layer and the second activation function layer have 64 input channels and 64 output channels; and the fifth convolutional layer has 64 input channels and 3 output channels.

[0042] In one possible implementation, such as Figure 9 As shown, before inputting the RGBW image into the convolutional neural network model, the following steps are also included:

[0043] Step 201: Obtain RGB sample images;

[0044] Step 202: Downsample the RGB sample image, add noise and add white channel to obtain the RGBW sample image. The resolution of the RGBW sample image is smaller than that of the RGB sample image.

[0045] In this process, high-quality, high-resolution RGB images are used as ground-truth (GT) images, i.e., RGB sample images. These RGB sample images are then downsampled to obtain low-resolution image data. To simulate a real-world environment, in addition to conventional random cropping and flipping, data augmentation simulating a real degradation process is applied to the low-resolution image data to add noise. Finally, the downsampled and noise-added images are subjected to RGBW sampling (adding the white channel) to obtain low-resolution RGBW images, i.e., RGBW sample images. The RGBW sample images and their corresponding RGB sample images constitute the training data pairs.

[0046] Step 203: Use RGB sample images and RGBW sample images as training data to train the convolutional neural network model to obtain a trained convolutional neural network model, so that subsequent image processing can be performed based on the model.

[0047] Specifically, the L1 loss function can be used to train a convolutional neural network model. The L1 loss function is defined as follows:

[0048]

[0049] Where L represents the loss function, F JDDS (·) represents a convolutional neural network, W represents the network parameters of the convolutional neural network, N represents the number of training samples, and x i Let y represent the i-th RGBW sample image input. i This indicates that the corresponding x i The RGB sample images are given, and ||·||1 represents the L1 norm. For example, a convolutional neural network model is implemented using the PyTorch architecture. The loss function value between the output of the convolutional neural network model and the RGB sample images is calculated, and the network parameters are updated using the Adam optimizer. The network is trained for a total of, for example, 240,000 iterations, with an initial learning rate of 1*10^24. -4 The learning rate is halved when the number of iterations reaches 60,000, 120,000, and 180,000. After the iteration stops, the trained network parameters are output and the model is saved, thus completing the training of the convolutional neural network model. The convolutional neural network model can then be used to process images, i.e., proceed to steps 101 and 102 above.

[0050] In one possible implementation, noise addition includes adding Gaussian noise, salt and pepper noise, speckle noise, Poisson noise, Gaussian blur, and light intensity variation.

[0051] The following uses actual images obtained through image processing methods to illustrate the effects of the embodiments of this application. For example... Figure 10 As shown, Figure 10The diagram illustrates the splitting of an RGBW image into RGB and W channels. It demonstrates that inputting a noisy, low-resolution image into a trained convolutional neural network can produce a clean, high-resolution RGB color image. This embodiment, by introducing additional white channel information, compensates for the loss of color and detail information due to insufficient sampling rate compared to the RGB CFA mode, thereby improving network performance and robustness. Figure 11 As shown, Figure 11 The high-resolution RGB image A is downsampled and noise is added to obtain RGB image B. RGB image B is directly input into the network model of this embodiment to obtain RGB output result C. Using the method of this embodiment, a white channel is added to RGB image B to form an RGBW image, which is then input into the network model of this embodiment to obtain RGB output result D. To ensure the comparison effect, the experimental settings are the same for obtaining results C and D, except for the difference in the input image. It can be seen that result D, obtained by using the RGBW image as input, performs better than result C, obtained by using the RGB image as input, in terms of detail preservation and false color suppression. Figure 12 As shown, result E is the output of the model trained using the first noise addition method, which only adds Gaussian noise. Result F is the output of the model trained using the second noise addition method, which adds Gaussian noise, salt-and-pepper noise, speckle noise, Poisson noise, Gaussian blur, and light intensity variations. It is evident that using various types of noise to simulate real noise yields cleaner results compared to simulating only Gaussian noise. This avoids explicitly estimating the noise distribution and improves the network's generalization performance.

[0052] It should be noted that in the neural network model described in this embodiment, some convolutional layers can be replaced with channel attention layers or spatial attention layers. This allows for more inter-channel and spatial interactions between features, but it increases the complexity of the neural network. Furthermore, in the process of acquiring RGB sample images, another network model can be used to learn the distribution of real image degradation, and then this network model can be used to generate the required RGB sample images, rather than explicitly simulating real degradation.

[0053] This application also provides an image processing apparatus, including: an image acquisition unit for acquiring an RGBW image from an image sensor, the RGBW image including a red channel, a green channel, a blue channel, and a white channel; and an image processing unit for inputting the RGBW image into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model, the convolutional neural network model including a feature extraction module, a reconstruction module, and an upsampling module cascaded in sequence, the feature extraction module and the reconstruction module being used for de-mosaicing and denoising processes, the upsampling module being used for super-resolution processing, the RGB image including a red channel, a green channel, and a blue channel, and the resolution of the RGB image being greater than the resolution of the RGBW image.

[0054] The image processing device can apply the image processing methods in any of the above embodiments. The specific process and principle are the same as those in the above embodiments, and will not be repeated here.

[0055] It should be understood that the above division of the image processing device is merely a logical functional division. In actual implementation, all or part of these modules can be integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely through software calls from processing elements; they can be implemented entirely in hardware; or some modules can be implemented through software calls from processing elements, while others are implemented in hardware. For example, either the image acquisition unit or the image processing unit can be a separate processing element, or it can be integrated into the image processing device, for example, integrated into a chip within the image processing device. Alternatively, it can be stored as a program in the memory of the image processing device, and called and executed by a processing element of the image processing device. The implementation of other modules is similar. Moreover, these modules can be integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0056] For example, the image acquisition unit and image processing unit can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Furthermore, when one of the above modules is implemented through a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Additionally, these modules can be integrated together as a system-on-a-chip (SOC).

[0057] In one possible implementation, the feature extraction module includes a first ordinary block and three first residual blocks cascaded together; each first residual block includes two cascaded residual processing units, each residual processing unit includes a second ordinary block and an activation function layer cascaded together, and the output of each residual processing unit is the result of adding the output of the activation function layer therein to the input of the residual processing unit therein; either the first ordinary block or the second ordinary block includes a convolutional layer, an activation function layer and a convolutional layer cascaded together.

[0058] In one possible implementation, the kernel size of any convolutional layer is 3×3, the stride is 1, and the padding is 1. The first convolutional layer in the first normal block has 4 input channels and 64 output channels. Except for the first convolutional layer in the first normal block, the input channels and output channels of any activation function layer and any convolutional layer in the feature extraction module are 64 and 64 respectively.

[0059] In one possible implementation, the reconstruction module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit; the first processing unit includes a third ordinary block and a second residual block cascaded sequentially, the third ordinary block being used to acquire the output of the feature extraction module, and the second residual block being used to output to the second processing unit and the fifth processing unit; the second processing unit includes a first convolutional layer, a fourth ordinary block, and a third residual block cascaded sequentially, the first convolutional layer being used to acquire one output of the second residual block, and the third residual block being used to output to the third processing unit and the fourth processing unit; the third processing unit includes a second convolutional layer, a fifth ordinary block, and a third residual block cascaded sequentially. The system comprises a pass block, a fourth residual block, a sixth ordinary block, and a first transposed convolutional layer. A second convolutional layer is used to obtain one output of the third residual block, and the first transposed convolutional layer is used to output to the fourth processing unit. The fourth processing unit includes a first splicing layer, a seventh ordinary block, a fifth residual block, and a second transposed convolutional layer, which are cascaded in sequence. The first splicing layer is used to obtain another output of the third residual block and the output of the first transposed convolutional layer and splice them together. The second transposed convolutional layer is used to output to the fifth processing unit. The fifth processing unit includes a second splicing layer and an eighth ordinary block, which are cascaded in sequence. The second splicing layer is used to obtain another output of the second residual block and the output of the second transposed convolutional layer and splice them together.

[0060] In one possible implementation, the kernel size of the first and second convolutional layers is 3×3, the stride is 2, and the padding is 1; the kernel size of the first and second transposed convolutional layers is 2×2, the stride is 2, and the padding is 0; the number of input channels and output channels of the third normal block and the second residual block is 64; the number of input channels of the first convolutional layer is 64 and the number of output channels is 96; the number of input channels and output channels of the fourth normal block, the third residual block, and the fifth residual block is 96; and the number of input channels of the second convolutional layer is 96 and the number of output channels is 12. 8; The fifth normal block, the fourth residual block, and the sixth normal block have 128 input channels and 96 output channels; the first transposed convolutional layer has 96 input channels and 192 output channels; the seventh normal block has 192 input channels and 96 output channels; the second transposed convolutional layer has 96 input channels and 64 output channels; the second splicing layer has 64 input channels and 128 output channels; the eighth normal block has 128 input channels and 64 output channels.

[0061] In one possible implementation, the upsampling module includes a third convolutional layer, a first activation function layer, a pixel shuffling layer, a fourth convolutional layer, a second activation function layer, and a fifth convolutional layer cascaded in sequence.

[0062] In one possible implementation, the third convolutional layer has 64 input channels and 256 output channels; the first activation function layer has 256 input channels and 256 output channels; the pixel shuffling layer has 256 input channels and 64 output channels; the fourth convolutional layer and the second activation function layer have 64 input channels and 64 output channels; and the fifth convolutional layer has 64 input channels and 3 output channels.

[0063] In one possible implementation, the system further includes: a training unit for acquiring RGB sample images; downsampling, adding noise, and adding a white channel to the RGB sample images to obtain RGBW sample images, wherein the resolution of the RGBW sample images is smaller than that of the RGB sample images; and using the RGB sample images and RGBW sample images as training data to train the convolutional neural network model.

[0064] In one possible implementation, noise addition includes adding Gaussian noise, salt and pepper noise, speckle noise, Poisson noise, Gaussian blur, and light intensity variation.

[0065] This application also provides an image processing apparatus, including a processor and a memory, wherein the memory is used to store at least one instruction, which is loaded and executed by the processor to implement the image processing method described above.

[0066] The number of processors can be one or more; for example, processors may include image signal processors (ISPs). Processors and memory can be connected via a bus or other means. Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the image processing apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in any of the above method embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, and necessary data, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device.

[0067] This application also provides an electronic device, including the image processing apparatus described above. The electronic device involved in this application may be any product such as a mobile phone, tablet computer, personal computer (PC), personal digital assistant (PDA), smartwatch, wearable electronic device, augmented reality (AR) device, virtual reality (VR) device, in-vehicle device, drone device, smart car, smart speaker, robot, smart glasses, etc.

[0068] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method described in any of the above embodiments.

[0069] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0070] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0071] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Acquire an RGBW image from an image sensor, the RGBW image including a red channel, a green channel, a blue channel, and a white channel; The RGBW image is input into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model. The convolutional neural network model includes a feature extraction module, a reconstruction module, and an upsampling module cascaded in sequence. The feature extraction module and the reconstruction module are used for de-mosaicing and noise reduction. The upsampling module is used for super-resolution processing. The RGB image includes a red channel, a green channel, and a blue channel. The resolution of the RGB image is greater than that of the RGBW image. The feature extraction module includes a first ordinary block and three first residual blocks cascaded in sequence; Each of the first residual blocks includes two cascaded residual processing units, each of the residual processing units includes a second ordinary block and an activation function layer cascaded in sequence, and the output of each of the residual processing units is the result of adding the output of the activation function layer to the input of the residual processing unit; Either the first ordinary block or the second ordinary block includes a convolutional layer, an activation function layer, and a convolutional layer cascaded in sequence.

2. The method according to claim 1, characterized in that, The kernel size of any convolutional layer is 3×3, the stride is 1, and the padding is 1. The first convolutional layer in the first ordinary block has 4 input channels and 64 output channels. Except for the first convolutional layer in the first ordinary block, the input channels and output channels of any activation function layer and any convolutional layer in the feature extraction module are 64 and 64 respectively.

3. The method according to claim 1, characterized in that, The reconstruction module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit; The first processing unit includes a third ordinary block and a second residual block cascaded in sequence. The third ordinary block is used to obtain the output of the feature extraction module, and the second residual block is used to output to the second processing unit and the fifth processing unit. The second processing unit includes a first convolutional layer, a fourth ordinary block, and a third residual block cascaded in sequence. The first convolutional layer is used to obtain an output of the second residual block, and the third residual block is used to output to the third processing unit and the fourth processing unit. The third processing unit includes a second convolutional layer, a fifth ordinary block, a fourth residual block, a sixth ordinary block, and a first transposed convolutional layer, which are cascaded in sequence. The second convolutional layer is used to obtain an output of the third residual block, and the first transposed convolutional layer is used to output to the fourth processing unit. The fourth processing unit includes a first splicing layer, a seventh ordinary block, a fifth residual block, and a second transposed convolutional layer cascaded in sequence. The first splicing layer is used to obtain another output of the third residual block and the output of the first transposed convolutional layer and splice them together. The second transposed convolutional layer is used to output to the fifth processing unit. The fifth processing unit includes a second splicing layer and an eighth ordinary block cascaded in sequence. The second splicing layer is used to obtain another output of the second residual block and the output of the second transposed convolutional layer and splice them together.

4. The method according to claim 3, characterized in that, The kernel size of the first convolutional layer and the second convolutional layer is 3×3, the stride is 2, and the padding is 1. The kernel size of the first transposed convolutional layer and the second transposed convolutional layer is 2×2, the stride is 2, and the padding is 0. The number of input channels and the number of output channels of the third ordinary block and the second residual block are 64; The first convolutional layer has 64 input channels and 96 output channels; The fourth ordinary block, the third residual block, and the fifth residual block have a total of 96 input channels and output channels. The second convolutional layer has 96 input channels and 128 output channels; The number of input channels and the number of output channels for the fifth normal block, the fourth residual block and the sixth normal block are 128; The first transposed convolutional layer has 128 input channels and 96 output channels; The first splicing layer has 96 input channels and 192 output channels; The seventh ordinary block has 192 input channels and 96 output channels; The second transposed convolutional layer has 96 input channels and 64 output channels; The second splicing layer has 64 input channels and 128 output channels; The eighth ordinary block has 128 input channels and 64 output channels.

5. The method according to claim 1, characterized in that, The upsampling module includes a third convolutional layer, a first activation function layer, a pixel shuffling layer, a fourth convolutional layer, a second activation function layer, and a fifth convolutional layer, which are cascaded in sequence.

6. The method according to claim 5, characterized in that, The third convolutional layer has 64 input channels and 256 output channels; The first activation function layer has 256 input channels and 256 output channels; The pixel shuffling layer has 256 input channels and 64 output channels; The fourth convolutional layer and the second activation function layer have 64 input channels and 64 output channels. The fifth convolutional layer has 64 input channels and 3 output channels.

7. The method according to any one of claims 1 to 6, characterized in that, Before inputting the RGBW image into the convolutional neural network model, the following steps are also included: Obtain RGB sample images; The RGB sample image is downsampled, noise is added, and a white channel is added to obtain an RGBW sample image, the resolution of which is smaller than that of the RGB sample image; The RGB sample images and the RGBW sample images are used as training data to train the convolutional neural network model.

8. The method according to claim 7, characterized in that, The noise addition includes: adding Gaussian noise, salt and pepper noise, speckle noise, Poisson noise, Gaussian blur, and light intensity variation.

9. An image processing apparatus, characterized in that, include: An image acquisition unit is used to acquire an RGBW image from an image sensor, wherein the RGBW image includes a red channel, a green channel, a blue channel, and a white channel; An image processing unit is used to input the RGBW image into a convolutional neural network model to obtain an RGB image output by the convolutional neural network model. The convolutional neural network model includes a feature extraction module, a reconstruction module, and an upsampling module cascaded in sequence. The feature extraction module and the reconstruction module are used for de-mosaicing and noise reduction. The upsampling module is used for super-resolution processing. The RGB image includes a red channel, a green channel, and a blue channel. The resolution of the RGB image is greater than the resolution of the RGBW image. The feature extraction module includes a first ordinary block and three first residual blocks cascaded in sequence; Each of the first residual blocks includes two cascaded residual processing units, each of the residual processing units includes a second ordinary block and an activation function layer cascaded in sequence, and the output of each of the residual processing units is the result of adding the output of the activation function layer to the input of the residual processing unit; Either the first ordinary block or the second ordinary block includes a convolutional layer, an activation function layer, and a convolutional layer cascaded in sequence.

10. An image processing apparatus, characterized in that, include: A processor and a memory, the memory being used to store at least one instruction, which, when loaded and executed by the processor, implements the image processing method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, Includes the image processing apparatus as described in claim 10.

Images