Image processing method, device and electronic equipment
The pre-trained down-resolution model converts the RAW image into RGB images with a specified resolution, which solves the image quality damage caused by reducing the resolution of RAW image in the prior art, and achieves better image processing effects.
Patent Information
- Application Number
- CN202310119858.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-01-18
AI Technical Summary
The prior art seriously damages image quality while reducing RAW image resolution, resulting in edge jagging and loss of detail.
The pre-trained down-resolution model is used to convert the target RAW image into an RGB image with a specified resolution size, and calculate the model loss using the target RGB image and the first true image. The trained model can reduce the resolution of the RAW image. At the same time, better ensure image quality.
Through this method, it is possible to reduce image quality damage while reducing RAW image resolution, maintain image details and edge clarity, and provide better image processing effects.
Smart Images

Figure CN118379197B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device and electronic equipment. Background Art
[0002] RAW (unprocessed) images refer to the original images directly output by the image sensor, which are not processed and compressed. After obtaining the RAW images, computing resources are still needed to perform subsequent image processing on the RAW images, including: converting the RAW images into RGB (Red\Green\Blue) images, and other ISP (Image Signal Processing) processing processes.
[0003] In order to save computing resources, it is sometimes necessary to reduce the resolution of the RAW image first, and then perform subsequent image processing on the RAW image with the reduced resolution. In the related art, the RAW image is downsampled to reduce the resolution of the RAW image through traditional downsampling algorithms, such as pixel merging, bilinear interpolation and other algorithms. However, the RAW image processed in this way has obvious edge jagged effects after being converted into an RGB image, and the details are seriously lost. It can be seen that the related art seriously damages the image quality while reducing the resolution of the RAW image, and cannot well guarantee the image quality.
[0004] Therefore, how to reduce the resolution of RAW images while better ensuring the image quality is an urgent problem to be solved. Summary of the invention
[0005] The purpose of the embodiment of the present invention is to provide an image processing method to achieve better image quality while reducing the resolution of the RAW image. The specific technical solution is as follows:
[0006] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising:
[0007] Acquire the target RAW image whose resolution is to be reduced;
[0008] The resolution of the target RAW image is converted into a specified resolution size by using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using a model loss calculated using a target RGB image and a first true value image; the target RGB image is an image obtained by converting an output RAW image into an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into the specified resolution size; the first true value image is an image of the specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image.
[0009] Optionally, the training method of the resolution reduction model includes:
[0010] Acquire the first sample RAW image;
[0011] Converting the first sample RAW image into an RGB image, and performing downsampling processing on the obtained RGB image to obtain the first true value image;
[0012] Inputting the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into the specified resolution size, thereby obtaining the output RAW image;
[0013] Convert the output RAW image into an RGB image to obtain a target RGB image;
[0014] Inputting the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained;
[0015] Based on the obtained model loss, the model parameters in the neural network model to be trained are adjusted.
[0016] Optionally, converting the output RAW image into an RGB image to obtain a target RGB image includes:
[0017] The output RAW image is converted into an RGB image using a pre-trained image conversion model to obtain a target RGB image; wherein the image conversion model is a neural network model trained using a second sample RAW image and a second true value image corresponding to the second sample RAW image; and the second true value image is an RGB image obtained by performing image conversion on the second sample RAW image using a demosaicing algorithm.
[0018] Optionally, converting the first sample RAW image into an RGB image includes:
[0019] The first sample RAW image is converted into an RGB image using the image conversion model.
[0020] Optionally, the target RAW image is an image containing noise;
[0021] Before inputting the first sample RAW image into the neural network model to be trained, the method further includes:
[0022] Performing noise processing on the first sample RAW image to obtain the first sample RAW image containing noise;
[0023] The method of converting the resolution of the target RAW image to a specified resolution size by using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution includes:
[0024] The target RAW image is input into a pre-trained resolution reduction model so that the resolution reduction model converts the resolution of the target RAW image into a specified resolution size, and performs noise reduction on the target RAW image to obtain a denoised RAW image of the specified resolution size.
[0025] Optionally, the first sample RAW image is: a RAW image captured by a specified image sensor under sufficient lighting conditions and with a sensitivity less than a preset threshold.
[0026] Optionally, acquiring the first sample RAW image includes:
[0027] Get the initial RAW image captured by the specified image sensor;
[0028] The pixel values of each pixel point of the initial RAW image are subjected to a plurality of preset calculation processes to obtain a RAW image after calculation corresponding to each of the preset calculation processes as a first sample RAW image.
[0029] Optionally, performing a plurality of preset calculation processes on the pixel value of each pixel point of the initial RAW image to obtain a RAW image after calculation process corresponding to each preset calculation process includes:
[0030] The pixel value of each pixel point of the initial RAW image is divided by different preset magnification values to obtain a calculated RAW image corresponding to each preset magnification value.
[0031] In a second aspect, an embodiment of the present invention provides an image processing device, the device comprising:
[0032] An acquisition module, used for acquiring a target RAW image whose resolution is to be reduced;
[0033] A resolution reduction module is used to convert the resolution of the target RAW image into a specified resolution size using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using a model loss calculated using a target RGB image and a first true value image; the target RGB image is an image obtained by converting an output RAW image into an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into the specified resolution size; the first true value image is an image of the specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image.
[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0035] Memory, used to store computer programs;
[0036] The processor is used to implement the steps of the above-mentioned image processing method when executing the program stored in the memory.
[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image processing method are implemented.
[0038] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.
[0039] Beneficial effects of the embodiments of the present invention:
[0040] In the image processing method provided by the embodiment of the present invention, the target RGB image is an image of specified resolution size obtained by converting the output image output by the resolution reduction model for the first sample RAW image into an RGB image, and the first true value image is an image of specified resolution size obtained by converting the first sample RAW image into an RGB image and then downsampling the obtained RGB image. It can be seen that the target RGB image and the first true value image are both RGB images of specified resolution size, so the resolution reduction model can be trained using the model loss calculated by the target RGB image and the first true value image, so that the resolution reduction model can learn the mapping relationship between the RAW image before and after resolution reduction, and the first true value image is obtained by converting the first sample RAW image into an RGB image and then downsampling it, and downsampling the RGB image has less damage to the image quality than directly downsampling the RAW image using a traditional algorithm, so the resolution reduction model trained using the first true value image can better ensure the image quality while reducing the resolution of the RAW image. Therefore, through this solution, the target RAW image to be downgraded is input into the trained downscaling model to obtain the RAW image with reduced resolution, which can better ensure the image quality while reducing the resolution of the RAW image.
[0041] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0043] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;
[0044] Figure 2 A schematic diagram of the structure of a resolution reduction model in the image processing method provided in an embodiment of the present invention;
[0045] Figure 3 A schematic diagram of a de-mosaicing algorithm in an image processing method provided by an embodiment of the present invention;
[0046] Figure 4 A schematic diagram of the structure of an image conversion model in the image processing method provided by an embodiment of the present invention;
[0047] Figure 5Another flow chart of the image processing method provided by the embodiment of the present invention;
[0048] Figure 6 A flowchart of model training in the image processing method provided in an embodiment of the present invention;
[0049] Figure 7 Schematic diagram showing a comparison of the effects of resolution reduction and noise reduction by different processing methods in an embodiment of the present invention;
[0050] Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;
[0051] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.
[0053] At present, the image resolution of smartphones is getting higher and higher. For example, the image sensor of the main camera of the mobile phone has an imaging resolution of 4080×3060. Considering the computing power and power consumption of the NPU (Neural network Processing Unit, embedded neural network processor), as well as actual needs, such as night scene video 1080p (resolution of 1920×1080) is sufficient, then it is necessary to reduce the resolution of the RAW image output by the image sensor, and then input the RAW image with reduced resolution into the NPU. In related technologies, reducing the resolution of the RAW image by downsampling the RAW image through traditional downsampling algorithms, such as pixel merging, bilinear interpolation, etc., seriously damages the image quality.
[0054] Therefore, in order to better ensure the image quality while reducing the resolution of the RAW image, the present invention provides an image processing method, device and electronic device.
[0055] The following first introduces the image processing method provided by the embodiment of the present invention. The method may include the following steps:
[0056] Acquire the target RAW image whose resolution is to be reduced;
[0057] The resolution of the target RAW image is converted into a specified resolution size by using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using the model loss calculated using the target RGB image and the first true value image; the target RGB image is an image obtained by converting the output RAW image into an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into a specified resolution size; the first true value image is an image of a specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image.
[0058] In this embodiment, the target RGB image is an image of specified resolution size obtained by converting the output image output by the resolution reduction model for the first sample RAW image into an RGB image, and the first true value image is an image of specified resolution size obtained by converting the first sample RAW image into an RGB image and then downsampling the obtained RGB image. It can be seen that the target RGB image and the first true value image are both RGB images of specified resolution size. Therefore, the model loss calculated by the target RGB image and the first true value image can be used to train the resolution reduction model, so that the resolution reduction model can learn the mapping relationship between the RAW image before and after resolution reduction. In addition, the first true value image is obtained by converting the first sample RAW image into an RGB image and then downsampling it. Downsampling the RGB image has less damage to the image quality than directly downsampling the RAW image using a traditional algorithm. Therefore, the resolution reduction model trained using the first true value image can better ensure the image quality while reducing the resolution of the RAW image. Therefore, through this solution, the target RAW image to be downgraded is input into the trained downscaling model to obtain the RAW image with reduced resolution, which can better ensure the image quality while reducing the resolution of the RAW image.
[0059] The image processing method provided in the embodiment of the present invention can be applied to various electronic devices, such as computers, servers, smart phones, smart cameras and other devices with data processing capabilities. In addition, the image processing method can be implemented by software, hardware or a combination of software and hardware.
[0060] The image processing method provided by the embodiment of the present invention is introduced below with reference to the accompanying drawings.
[0061] like Figure 1 As shown, the image processing method provided by the embodiment of the present invention may include the following steps:
[0062] S101, obtaining a target RAW image to be reduced in resolution;
[0063] The target RAW image is a RAW image to be down-sampled, which may be any RAW image, for example, a RAW image captured by a specified image sensor. In one implementation, the target RAW image may be acquired in real time by acquiring the RAW image captured by the specified image sensor. In one implementation, the target RAW image may be a RAW image that is pre-captured and stored in a specified storage location, and the target RAW image may be acquired from the specified storage location.
[0064] S102, using a pre-trained resolution reduction model to convert the resolution of the target RAW image into a specified resolution size, to obtain a RAW image with reduced resolution; wherein the resolution reduction model is: a model trained using a model loss calculated using a target RGB image and a first true value image; the target RGB image is: an image obtained by converting an output RAW image into an RGB image; the output RAW image is: an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into a specified resolution size; the first true value image is: an image of a specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image.
[0065] In this step, the specified resolution can be any pre-set resolution size, such as 2K resolution (2560×1440), 1080p resolution (1920×1080), 720p resolution (1280×720), etc., and can also be a proportional size, for example, 1 / 2, 1 / 3, etc.
[0066] The resolution reduction model may be a preset neural network model. For example, the structure of the resolution reduction model may be as follows: Figure 2As shown, it can include three parts: preprocessing stage (pre), encoder-decoder, and postprocessing stage (post), wherein the preprocessing stage and the postprocessing stage are both stacked by multiple convolutional layers and ReLU (Rectified Linear Unit, linear rectified activation function) layers. The encoder-decoder can first encode the feature map, that is, reduce the spatial resolution and increase the number of channels, and then decode it, that is, increase the spatial resolution and reduce the number of channels. The encoder-decoder also includes multiple convolutional layers and ReLU layers, and also includes two deconvolutional layers. The network layer in the encoder-decoder adopts a jump connection structure to achieve the fusion of feature maps at different stages. In order to make the resolution reduction model have the function of resolution reduction, it can be achieved by setting the parameters of the first convolutional layer in the preprocessing stage. For example, when the shape of the RAW image to be input is (h, w, 1) and the resolution size of the image to be output is (h / 2, w / 2, 1), the parameters of the first convolutional layer in the preprocessing stage can be set to kernel_size (convolution kernel size) = 3, stride (step length) = 2, and padding (filling) = 1. If you need to reduce the resolution to other sizes, you can also set the parameters of the convolution layer according to your needs.
[0067] The structure of the above-mentioned resolution reduction model is only an exemplary introduction, and the resolution reduction model may of course also be other structures.
[0068] When training the down-resolution model, the target RGB image and the first true value image need to be used to calculate the model loss of the down-resolution model, so as to adjust the model parameters of the down-resolution model according to the model loss. For example, the training method of the down-resolution model may include the following steps:
[0069] Step A1, obtaining a first sample RAW image;
[0070] The first sample RAW image may be a RAW image captured by any image sensor, or may be a sample RAW image obtained from a sample set of pre-created RAW images. The number of first sample RAW images obtained each time may be determined according to demand, for example, a single image or multiple images may be obtained each time.
[0071] Step A2, converting the first sample RAW image into an RGB image, and performing downsampling processing on the obtained RGB image to obtain a first true value image;
[0072] In one implementation, the first sample RAW image may be converted into an RGB image by using a demosaic algorithm, for example, a function with a demosaic function in opencv (open source computer vision library), such as cvtColor (color space conversion function), so as to convert the RAW image bayer (original image inside the camera) format data into RGB format data. Figure 3 As shown, the RAW image is a single-channel image, in which the red pixels R, green pixels G, and blue pixels B are arranged in sequence in a certain order, and after interpolation using the demosaic algorithm, a three-channel RGB image is obtained, and each channel of the RGB image contains red pixels R, green pixels G, and blue pixels B. In another implementation, a pre-trained image conversion model can also be used to convert the first sample RAW image into an RGB image.
[0073] After converting the first sample RAW image into an RGB image, it is necessary to perform downsampling processing on the obtained RGB image. The downsampling processing method may be to use a traditional downsampling algorithm, such as pixel merging, bilinear interpolation and other algorithms for downsampling processing. Since the first sample RAW image has been converted into an RGB image, the use of a traditional downsampling algorithm for downsampling processing here has less damage to the image quality and can ensure the image quality of the first true value image obtained by the downsampling processing, thereby enabling the trained resolution reduction model to better ensure image quality.
[0074] Step A3, inputting the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into a specified resolution size, thereby obtaining an output RAW image;
[0075] Step A4, converting the output RAW image into an RGB image to obtain a target RGB image;
[0076] In this step, in order to realize model training for the resolution reduction model, the pre-trained image conversion model can be used to convert the output RAW image into an RGB image to obtain the target RGB image, thereby utilizing the characteristic of the neural network that can perform gradient propagation to provide a basis for subsequent model training using the loss function.
[0077] Among them, the image conversion model can be: a neural network model trained using the second sample RAW image and the second true value image corresponding to the second sample RAW image; the second true value image is: an RGB image obtained by converting the second sample RAW image using a de-mosaicing algorithm.
[0078] The second sample RAW image can be the first sample RAW image, or other RAW images. When training the image conversion model, the RGB image obtained by converting the second sample RAW image can be used as the second true image, and the first sample RAW image can be converted into an RGB image using the image conversion model to be trained, thereby using the converted image and the second true image to calculate the model loss of the image conversion model, and then using the obtained model loss to adjust the model parameters of the image conversion model until the image conversion model converges. During the training process, the image conversion model can learn the mapping relationship between the RAW image and the RGB image, so that the trained image conversion model can realize the function of converting the RAW image into the RGB image, and the image conversion model can also perform gradient propagation when training the above-mentioned resolution reduction model, providing a basis for the training of the resolution reduction model.
[0079] For example, Figure 4 As shown in , the image conversion model can also adopt a network structure similar to the resolution reduction model, which includes three parts: preprocessing stage (pre), encoder-decoder (encoder-decoder), and post-processing stage (post). The preprocessing stage and the post-processing stage are composed of multiple convolutional layers and ReLU layers stacked together. The encoder-decoder contains multiple convolutional layers and ReLU layers, as well as deconvolution layers, and a jump connection structure is used to achieve the fusion of feature maps at different stages. The image conversion model needs to convert the input single-channel RAW image (h, w, 1) into an output three-channel RGB image (h, w, 3). In order to achieve this function, the channel of the convolutional layer of the last layer of the post-processing stage can be set to 3.
[0080] In the above step of converting the first sample RAW image into an RGB image, the first sample RAW image may also be converted into an RGB image using the image conversion model.
[0081] Step A5, inputting the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained;
[0082] Exemplarily, the above specified loss function may be: loss=y-gt';
[0083] Among them, loss is the specified loss function, y is the target RGB image, and gt' is the first true value image.
[0084] Step A6: Based on the obtained model loss, adjust the model parameters in the neural network model to be trained.
[0085] In this step, the gradient descent method can be used to adjust the model parameters, and when training the model, the step of obtaining the first sample RAW image can be performed multiple times to use the above method to adjust the model parameters in the neural network model to be trained until the neural network model converges to obtain a trained image conversion model.
[0086] When the target RAW image needs to be down-sampled, the pre-trained down-resolution model can be used to convert the resolution of the target RAW image to a specified resolution size to obtain a RAW image with reduced resolution.
[0087] In this embodiment, the target RGB image is an image of specified resolution size obtained by converting the output image output by the resolution reduction model for the first sample RAW image into an RGB image, and the first true value image is an image of specified resolution size obtained by converting the first sample RAW image into an RGB image and then downsampling the obtained RGB image. It can be seen that the target RGB image and the first true value image are both RGB images of specified resolution size. Therefore, the model loss calculated by the target RGB image and the first true value image can be used to train the resolution reduction model, so that the resolution reduction model can learn the mapping relationship between the RAW image before and after resolution reduction. In addition, the first true value image is obtained by converting the first sample RAW image into an RGB image and then downsampling it. Downsampling the RGB image has less damage to the image quality than directly downsampling the RAW image using a traditional algorithm. Therefore, the resolution reduction model trained using the first true value image can better ensure the image quality while reducing the resolution of the RAW image. Therefore, through this solution, the target RAW image to be downgraded is input into the trained downscaling model to obtain the RAW image with reduced resolution, which can better ensure the image quality while reducing the resolution of the RAW image.
[0088] Optionally, in another embodiment of the present invention, the target RAW image may be an image containing noise;
[0089] At present, imaging in a dark environment at night has always been a difficult problem. Due to the low number of photons incident on the sensor and the large impact of noise, the imaging results are difficult to be satisfactory. Video / image denoising technology is mainly used to solve the noise problem and improve the signal-to-noise ratio and perception of the image. Noise reduction technology can be divided into two categories: traditional noise reduction technology and deep learning technology. Among them, deep learning technology can achieve better results than traditional noise reduction algorithms when used for noise reduction. In deep learning technology, images are denoised by using a pre-trained noise reduction model. A typical supervised learning noise reduction model training process is: input noisy image data (denoted as input), and the output result (denoted as output) is obtained through neural network processing. Then the output and label data (also called true value data, denoted as ground-truth, abbreviated as gt) are used to calculate the model loss, that is, loss = output-input, so as to use the model loss for back propagation to adjust the model parameters, so as to obtain a more ideal noise reduction model. Due to deep learning technology, especially supervised learning, the image quality of the image used to train the model is very critical. In recent years, more and more methods have chosen to use RAW images for noise reduction. This is because: the signal value of RAW images has a good linear relationship with the incident light; RAW images can be accurately modeled based on the physical imaging process; compared with RAW images, RGB images have undergone various complex nonlinear processing by ISP, and paired noise-true value data are not easy to obtain, making model learning more difficult.
[0090] Therefore, in order to better perform noise reduction, this solution can reduce the noise of the RAW image while reducing the resolution of the RAW image. In order to achieve the noise reduction function, in this embodiment, before the first sample RAW image is input into the neural network model to be trained, the first sample RAW image can be subjected to noise processing to obtain a first sample RAW image containing noise.
[0091] like Figure 5 As shown, in this embodiment, the following steps can be adopted when training the resolution reduction model:
[0092] S501, acquiring a first sample RAW image;
[0093] S502, performing noise processing on the first sample RAW image to obtain a first sample RAW image containing noise;
[0094] In this step, Poisson noise and / or Gaussian noise may be added to the first sample RAW image to implement noise addition processing on the first sample RAW image.
[0095] S503, converting the first sample RAW image into an RGB image, and performing downsampling processing on the obtained RGB image to obtain a first true value image;
[0096] S504, inputting the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into a specified resolution size, thereby obtaining an output RAW image;
[0097] S505, converting the output RAW image into an RGB image to obtain a target RGB image;
[0098] S506, inputting the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained;
[0099] S507: Based on the obtained model loss, adjust the model parameters in the neural network model to be trained.
[0100] The above steps S501 and S503 - S507 have been introduced above and will not be repeated here.
[0101] Furthermore, in order to ensure image quality, the first sample RAW image may be a RAW image captured by a specified image sensor under sufficient lighting conditions and with a sensitivity less than a preset threshold. It is understandable that the RAW image captured by the image sensor under sufficient lighting conditions and with a low sensitivity has less noise and can be approximately regarded as a noise-free image, so that the resolution reduction model trained using the first sample RAW image can have a better noise reduction effect.
[0102] Furthermore, when performing noise reduction and resolution reduction processing on the target RAW image, the target RAW image can be input into a pre-trained resolution reduction model so that the resolution reduction model converts the resolution of the target RAW image to a specified resolution size, and performs noise reduction processing on the target RAW image to obtain a noise-reduced RAW image of the specified resolution size.
[0103] In order to increase the number of samples, thereby improving the denoising and resolution reduction effects of the trained resolution reduction model, the step of obtaining the first sample RAW image may include:
[0104] An initial RAW image acquired by a designated image sensor is obtained; a plurality of preset calculation processes are performed on the pixel values of each pixel point of the initial RAW image to obtain a RAW image after calculation processing corresponding to each preset calculation processing as a first sample RAW image. The preset calculation processing may be at least one of a flip processing, a rotation processing, a random color dithering processing, and the like.
[0105] In addition, it is understandable that images with more noise are often collected under dark light conditions. Therefore, in order to simulate dark light conditions, the pixel values of each pixel point of the initial RAW image can be divided by a preset magnification value, thereby reducing the brightness of the initial RAW image to simulate the RAW image collected under dark light conditions, thereby improving the noise reduction effect of the resolution reduction model on the RAW image collected under simulated dark light conditions. In addition, the pixel values of each pixel point of the initial RAW image can be divided by different preset magnification values to obtain the calculated and processed RAW image corresponding to each preset magnification value, so as to increase the number of samples.
[0106] In this embodiment, the image quality can be better guaranteed while reducing the resolution of the RAW image. Further, before the first sample RAW image is input into the neural network model to be trained, the first sample RAW image is subjected to noise processing to obtain a first sample RAW image containing noise, and a resolution reduction model having the functions of reducing the resolution of the RAW image and reducing noise can be trained, so that the target RAW image is input into the resolution reduction model, and the noise of the target RAW image can be reduced while reducing the resolution of the target RAW image.
[0107] For ease of understanding, the training process of the resolution reduction model of the image processing method provided by the present invention is exemplarily introduced below with reference to the accompanying drawings.
[0108] like Figure 6 As shown, when training the resolution reduction model, the resolution reduction model can be connected to the image conversion model, so that the first sample RAW image is first processed by the resolution reduction model to obtain an intermediate result after resolution reduction, and this intermediate result is then processed by the image conversion model to obtain the target RGB image, wherein the first sample RAW image can be a RAW image after noise addition processing.
[0109] Before training the resolution reduction model, the image conversion model can be trained first: first use the function (cvtColor) with demosaic function in opencv to process the second sample RAW image to obtain the true value image of the image conversion model, that is, the second true value image. The image conversion model processes the second sample RAW image and outputs an RGB image. Then, the output RGB image and the second true value image can be used to calculate the model loss of the image conversion model to adjust the model parameters of the image conversion model. The trained image conversion model can simulate the function of the real demosaicing algorithm and has the characteristic that the neural network can perform gradient propagation, providing an implementation basis for training the resolution reduction model.
[0110] When constructing the first true value image, the noise-free image corresponding to the first sample RAW image can be input into the image conversion model, and then the RGB image output by the image conversion model is downsampled, for example, by bilinear interpolation, to obtain a down-resolution RGB image as the first true value image, so that the model loss of the down-resolution model can be calculated using the first true value image and the target RGB image to adjust the model parameters of the down-resolution model. After the training is completed, the image conversion model can be removed, and in actual use, only the trained down-resolution model is used to process the RAW image. The trained down-resolution model can better ensure the image quality while reducing the resolution of the RAW image, and also has the function of noise reduction.
[0111] It should be noted that when the relevant technology uses deep learning technology for resolution reduction and noise reduction, in the process of training the model, the binning (pixel merging) algorithm is usually used to process the noise-free RAW image, that is, the values of every four adjacent (2×2) pixels in the RAW image are added or averaged, and the obtained image is used as the true value image, and then the noise-free RAW image is processed with noise as the input of the model. However, the image quality of the true value image obtained in this way is severely damaged. After ISP, the image resolution of the true value image decreases, the edge jagged effect is obvious, and the details are seriously lost. Replacing binning with other methods such as bilinear interpolation has similar effects. Since the model is trained with images with severely damaged image quality, the image quality of the image obtained by reducing the resolution of the RAW image is bound to be difficult to guarantee. In this solution, the first true value image is obtained by converting the first sample RAW image into an RGB image and then performing downsampling processing. Downsampling the RGB image is less detrimental to the image quality than directly downsampling the RAW image using a traditional algorithm. Therefore, the resolution reduction model trained using the first true value image can better ensure the image quality while reducing the resolution of the RAW image. Figure 7 As shown, Figure 7 The figure shows a comparison of the effects of resolution reduction and noise reduction by different processing methods, from top to bottom: an image containing noise; an image obtained by binning a noise-free RAW image and training a model as a true image for resolution reduction and noise reduction; the third figure is an image after resolution reduction and noise reduction obtained by the image processing method of an embodiment of the present invention. It can be seen that the image processing method of the embodiment of the present invention better maintains image details and has a lighter jagged effect on the edges while achieving an obvious noise reduction effect.
[0112] In this embodiment, the target RGB image is an image of specified resolution size obtained by converting the output image output by the resolution reduction model for the first sample RAW image into an RGB image, and the first true value image is an image of specified resolution size obtained by converting the first sample RAW image into an RGB image and then downsampling the obtained RGB image. It can be seen that the target RGB image and the first true value image are both RGB images of specified resolution size. Therefore, the model loss calculated by the target RGB image and the first true value image can be used to train the resolution reduction model, so that the resolution reduction model can learn the mapping relationship between the RAW image before and after resolution reduction. In addition, the first true value image is obtained by converting the first sample RAW image into an RGB image and then downsampling it. Downsampling the RGB image has less damage to the image quality than directly downsampling the RAW image using a traditional algorithm. Therefore, the resolution reduction model trained using the first true value image can better ensure the image quality while reducing the resolution of the RAW image. Therefore, through this solution, the target RAW image to be downgraded is input into the trained downscaling model to obtain the RAW image with reduced resolution, which can better ensure the image quality while reducing the resolution of the RAW image.
[0113] The embodiment of the present invention also provides an image processing device, such as Figure 8 As shown, the device comprises:
[0114] An acquisition module 801 is used to acquire a target RAW image whose resolution is to be reduced;
[0115] The resolution reduction module 802 is used to convert the resolution of the target RAW image into a specified resolution size using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using a model loss calculated using a target RGB image and a first true value image; the target RGB image is an image obtained by converting an output RAW image into an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into the specified resolution size; the first true value image is an image of the specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image.
[0116] Optionally, the training method of the resolution reduction model includes:
[0117] Acquire the first sample RAW image;
[0118] Converting the first sample RAW image into an RGB image, and performing downsampling processing on the obtained RGB image to obtain the first true value image;
[0119] Inputting the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into the specified resolution size, thereby obtaining the output RAW image;
[0120] Convert the output RAW image into an RGB image to obtain a target RGB image;
[0121] Inputting the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained;
[0122] Based on the obtained model loss, the model parameters in the neural network model to be trained are adjusted.
[0123] Optionally, converting the output RAW image into an RGB image to obtain a target RGB image includes:
[0124] The output RAW image is converted into an RGB image using a pre-trained image conversion model to obtain a target RGB image; wherein the image conversion model is a neural network model trained using a second sample RAW image and a second true value image corresponding to the second sample RAW image; and the second true value image is an RGB image obtained by performing image conversion on the second sample RAW image using a demosaicing algorithm.
[0125] Optionally, converting the first sample RAW image into an RGB image includes:
[0126] The first sample RAW image is converted into an RGB image using the image conversion model.
[0127] Optionally, the target RAW image is an image containing noise; the device further comprises a noise adding module, configured to perform noise adding processing on the first sample RAW image before the resolution reduction module inputs the first sample RAW image into the neural network model to be trained, so as to obtain the first sample RAW image containing noise;
[0128] The resolution reduction module is specifically used for:
[0129] The target RAW image is input into a pre-trained resolution reduction model so that the resolution reduction model converts the resolution of the target RAW image into a specified resolution size, and performs noise reduction on the target RAW image to obtain a denoised RAW image of the specified resolution size.
[0130] Optionally, the first sample RAW image is: a RAW image captured by a specified image sensor under sufficient lighting conditions and with a sensitivity less than a preset threshold.
[0131] Optionally, acquiring the first sample RAW image includes:
[0132] Get the initial RAW image captured by the specified image sensor;
[0133] The pixel values of each pixel point of the initial RAW image are subjected to a plurality of preset calculation processes to obtain a RAW image after calculation corresponding to each of the preset calculation processes as a first sample RAW image.
[0134] Optionally, performing a plurality of preset calculation processes on the pixel value of each pixel point of the initial RAW image to obtain a RAW image after calculation process corresponding to each preset calculation process includes:
[0135] The pixel value of each pixel point of the initial RAW image is divided by different preset magnification values to obtain a calculated RAW image corresponding to each preset magnification value.
[0136] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.
[0137] Memory 903, used for storing computer programs;
[0138] The processor 901 is used to implement the steps of the above-mentioned image processing method when executing the program stored in the memory 903.
[0139] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0140] The communication interface is used for communication between the above electronic device and other devices.
[0141] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0142] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0143] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image processing methods are implemented.
[0144] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any image processing method in the above embodiments.
[0145] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0146] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0147] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire the target RAW image whose resolution is to be reduced; The resolution of the target RAW image is converted to a specified resolution size by using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using a model loss calculated using a target RGB image and a first true value image; the target RGB image is an image obtained by converting an output RAW image to an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image to the specified resolution size; the first true value image is an image of the specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image to an RGB image; The training method of the resolution reduction model includes: Acquire the first sample RAW image; Converting the first sample RAW image into an RGB image, and performing downsampling processing on the obtained RGB image to obtain the first true value image; Inputting the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into the specified resolution size, thereby obtaining the output RAW image; Convert the output RAW image into an RGB image to obtain a target RGB image; Inputting the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained; Based on the obtained model loss, the model parameters in the neural network model to be trained are adjusted.
2. The method according to claim 1, characterized in that The step of converting the output RAW image into an RGB image to obtain a target RGB image comprises: The output RAW image is converted into an RGB image using a pre-trained image conversion model to obtain a target RGB image; wherein the image conversion model is a neural network model trained using a second sample RAW image and a second true value image corresponding to the second sample RAW image; and the second true value image is an RGB image obtained by performing image conversion on the second sample RAW image using a demosaicing algorithm.
3. The method according to claim 2, characterized in that The converting the first sample RAW image into an RGB image comprises: The first sample RAW image is converted into an RGB image using the image conversion model.
4. The method according to claim 1, characterized in that: The target RAW image is an image containing noise; Before inputting the first sample RAW image into the neural network model to be trained, the method further includes: Performing noise processing on the first sample RAW image to obtain the first sample RAW image containing noise; The method of converting the resolution of the target RAW image to a specified resolution size by using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution includes: The target RAW image is input into a pre-trained resolution reduction model so that the resolution reduction model converts the resolution of the target RAW image into a specified resolution size, and performs noise reduction on the target RAW image to obtain a denoised RAW image of the specified resolution size.
5. The method according to claim 4, characterized in that The first sample RAW image is: a RAW image captured by a specified image sensor under sufficient lighting conditions and when the sensitivity is less than a preset threshold.
6. The method according to claim 1, characterized in that The acquiring the first sample RAW image comprises: Get the initial RAW image captured by the specified image sensor; The pixel values of each pixel point of the initial RAW image are subjected to a plurality of preset calculation processes to obtain a RAW image after calculation corresponding to each of the preset calculation processes as a first sample RAW image.
7. The method according to claim 6, characterized in that The step of performing a plurality of preset calculation processes on the pixel values of each pixel point of the initial RAW image to obtain a RAW image after calculation processes corresponding to each of the preset calculation processes comprises: The pixel value of each pixel point of the initial RAW image is divided by different preset magnification values to obtain a calculated RAW image corresponding to each preset magnification value.
8. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a target RAW image whose resolution is to be reduced; A resolution reduction module, used to convert the resolution of the target RAW image into a specified resolution size using a pre-trained resolution reduction model to obtain a RAW image with reduced resolution; wherein the resolution reduction model is a model obtained by training using a model loss calculated using a target RGB image and a first true value image; the target RGB image is an image obtained by converting an output RAW image into an RGB image; the output RAW image is an image output after the resolution reduction model to be trained converts the resolution of a preset first sample RAW image into the specified resolution size; the first true value image is an image of the specified resolution size obtained by downsampling the RGB image obtained after converting the first sample RAW image into an RGB image; The training method of the resolution reduction model includes: Acquire the first sample RAW image; convert the first sample RAW image into an RGB image, and downsample the obtained RGB image to obtain the first true value image; input the first sample RAW image into the neural network model to be trained, so that the neural network model to be trained converts the resolution of the first sample RAW image into the specified resolution size to obtain the output RAW image; convert the output RAW image into an RGB image to obtain a target RGB image; input the target RGB image and the first true value image into a specified loss function to determine the model loss of the neural network model to be trained; and adjust the model parameters in the neural network model to be trained based on the obtained model loss.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.
