Image processing method and storage medium

Abstract

The invention relates to an image processing method and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the preprocessing of an original first image, and obtaining a second image with noise; the second image is input into an image processing model of a U-shaped network structure for convolution processing, an image set is obtained, the image set comprises a third image, and the third image and the second image are the same in size; performing denoising and demosaicing processing on the second image and the third image to obtain a target image; the problems of low image definition, noise residue, false color and false image and the like in the existing image denoising and demosaicing method are solved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and a storage medium. Background Art

[0002] In digital photography, the raw data captured by an image sensor needs to go through two key steps of demosaicing and denoising to restore a high-quality color image. Currently, deep learning, especially convolutional neural networks, can be used for denoising and demosaicing. However, due to the opaque internal working mechanism of convolutional neural networks, it is difficult to explain and understand the decision-making process of the model, making it impossible to balance performance and effect.

[0003] To solve common problems such as noise, false color, moiré, and distortion, existing methods usually rely on a large number of parameters and complex network structures, which require a large amount of computing resources and consume electrical energy, thus limiting their application on resource-constrained platforms, such as mobile phones and cameras, etc. Platforms, and also face challenges when debugging and migrating to new tasks, and problem positioning and adjustment will be difficult and complex. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image processing method and a storage medium.

[0005] According to the first aspect of the embodiments of the present disclosure, an image processing method is provided, including:

[0006] Preprocessing the original first image to obtain a second image with noise;

[0007] Inputting the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the third image has the same size as the second image;

[0008] Performing denoising and demosaicing processing on the second image and the third image to obtain a target image.

[0009] According to the second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, including:

[0010] A first acquisition module, configured to preprocess the original first image to obtain a second image with noise;

[0011] A second acquisition module, configured to input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the third image has the same size as the second image;

[0012] A processing module, configured to perform denoising and demosaicing processing on the second image and the third image to obtain a target image.

[0013] According to a third aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0014] A processor;

[0015] A memory for storing instructions executable by the processor;

[0016] Wherein, the processor is configured to:

[0017] Perform preprocessing on an original first image to obtain a second image with noise;

[0018] Input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the third image has the same size as the second image;

[0019] Perform denoising and demosaicing processing on the second image and the third image to obtain a target image.

[0020] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enabling the mobile terminal to execute an image processing method, the method including:

[0021] Perform preprocessing on an original first image to obtain a second image with noise;

[0022] Input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the third image has the same size as the second image;

[0023] Perform denoising and demosaicing processing on the second image and the third image to obtain a target image.

[0024] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the image processing method as described in the embodiments of the first aspect.

[0025] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: By analyzing the second image with noise obtained through preprocessing of the first image, the computational amount and cost are reduced. The second image is input into an image processing model with a U-shaped network structure for convolution processing, and an image set is output. The second image is denoised and demosaicked based on the multi-channel images in the image set, and a target image with better clarity and image quality is obtained, solving the problems of low image clarity, noise residue, false color artifacts, etc. in current image denoising and demosaicking methods.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0028] Figure 1 is a flowchart of an image processing method shown according to some embodiments of the present disclosure.

[0029] Figure 2 is a flowchart of an image processing method shown according to some embodiments of the present disclosure.

[0030] Figure 3 is a schematic diagram of a U-shaped network structure shown according to some embodiments of the present disclosure.

[0031] Figure 4 is a logical flowchart of an image processing method shown according to some embodiments of the present disclosure.

[0032] Figure 5 is a structural block diagram of an image processing device shown according to some embodiments of the present disclosure.

[0033] Figure 6 is a structural block diagram of another image processing device shown according to some embodiments of the present disclosure.

[0034] Figure 7 is a structural block diagram of another image processing device shown according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] Some embodiments of the present disclosure will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatuses, and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely exemplary and is not limited to those set forth herein, but may be changed as will be apparent after understanding the present disclosure, except for operations that must be performed in a specific order. Additionally, descriptions of features known in the art may be omitted for increased clarity and conciseness.

[0036] The embodiments described in some of the following embodiments of the present disclosure do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] Image denoising is to recover a clean image from a noisy image. The noise may come from various factors, including the electronic noise of the image acquisition device, lighting conditions, interference during transmission, etc. With the development of image processing technology, image denoising methods include traditional methods, image denoising methods based on mathematical models, and learning-based denoising methods.

[0038] Traditional denoising methods can be achieved through spatial domain filters, frequency domain filters, or wavelet transforms. Spatial domain filters such as mean filtering, median filtering, and Gaussian filtering are used to reduce noise by locally averaging or replacing pixel values in the spatial domain of the image; frequency domain filters such as low-pass filters are used to denoise by converting the image to the frequency domain and suppressing high-frequency noise components; wavelet transform is used to denoise by performing multi-scale decomposition on the image and then applying threshold processing to the wavelet coefficients. Image denoising methods based on mathematical models can be based on Gaussian Mixture Models (GMMs) or the sparse representation model of K-SVD to remove noise by learning the statistical characteristics of the image; non-local means can also be used to suppress noise by utilizing non-local similarity. Learning-based denoising methods can be deep convolutional neural networks that learn the mapping from noisy images to clean images through training data; autoencoders use unsupervised or self-supervised learning to learn an effective representation of the data and then use it for denoising; generative adversarial networks use adversarial training to generate high-quality denoised images.

[0039] Image demosaicing is to restore the lost pixels in order to reconstruct a full-color image from a single-channel mosaic image. Current demosaicing methods are generally based on model-based demosaicing and learning-based demosaicing. The learning-based demosaicing method restores the image by learning the mapping from a large number of real images and CFA image pairs. The model-based demosaicing method uses mathematical models and prior knowledge of images in the spatial-spectral domain to restore the image. Traditional demosaicing methods either apply a fixed demosaicing filter to each pixel without considering other parameters as features, or use the spectral and spatial features available in neighboring pixels to interpolate unknown pixels as close as possible to the original image; due to the complexity of various color filter arrays, traditional methods become cumbersome, and there are still problems such as false colors, artifacts, and low resolution in the interpolation results.

[0040] The joint denoising and demosaicing technology fuses the denoising task and the demosaicing task, takes advantage of deep learning, adaptively learns and adapts to the complex patterns in the image data, and uses a template to perform denoising and demosaicing simultaneously to generate a clearer and less noisy image. However, current denoising and demosaicing technologies usually rely on a large amount of data, require a large amount of computing resources, limit their applications on some resource-constrained platforms, and have poor effects when migrated to new tasks.

[0041] Figure 1 It is a flowchart of an image processing method shown according to some embodiments of the present disclosure. As Figure 1 shown, the method includes the following steps:

[0042] S101, preprocess the original first image to obtain a second noisy image.

[0043] In some implementations, the original first image is a real-world image captured by a mobile phone, camera, or camera, etc., and is an unprocessed original RAW image directly captured. Each pixel in the RAW image has only the color information of a certain channel in the three channels of red, green, and blue (RGB). And among every 4 pixels, 2 pixels are G-channel information, 1 pixel is R-channel information, and 1 pixel is B-channel information, that is, the GRBG format.

[0044] Optionally, the preprocessing can be scale transformation, that is, perform scale transformation on the original first image to obtain at least one transformed-scale second noisy image; for example, if the size of the original first image is H*W, after scale transformation preprocessing, 4 second noisy images with a size of H / 2*W / 2 can be obtained.

[0045] S102, input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set.

[0046] The preset network structure may include a U-shaped network structure (UNet). The U-shaped network structure (UNet) consists of a contracting path and an expanding path. The contracting path is composed of 4 blocks. Each block uses 3 valid convolutions and 1 max pooling for downsampling. After each downsampling, the number of feature maps is multiplied by 2. Therefore, after the second image is input into the image processing model with the UNet structure, through the convolutions and downsampling of the contracting path, a feature map with a reduced size is obtained. The expanding path is also composed of 4 blocks. Before each block starts, the size of the feature map is multiplied by 2 through transposed convolution, and at the same time, its number is halved (the last layer is different), and it is merged with the feature map of the symmetric contracting path on the left.

[0047] Optionally, in this embodiment, the convolutional kernel in the image processing model with the UNet structure can be 3*3. After the second image is input into the image processing model with the UNet structure, through the convolutional processing of the contracting path and the expanding path, an image set is output. The image set includes a third image. The third image can be the images of each channel segmented from the second image with noise. The third image has the same size as the second image, that is, the image set includes one or more third images with the same size as the second image. In this embodiment, the size of the second image is H / 2*W / 2, and the size of the corresponding third image is also H / 2*W / 2.

[0048] S103. Denoise and demosaic the second image and the third image to obtain a target image.

[0049] In the image set output by the image processing model with the UNet structure in this embodiment, it may include the noise images corresponding to each second image. That is, the image processing model with the UNet structure extracts the noise data in the second image with noise and outputs the noise image corresponding to the second image. Optionally, the noise in the second image can be removed based on the noise image in the third image to obtain a denoised second image.

[0050] In some implementations, the purpose of demosaicing is to reduce the mosaic effect in the image to restore the clarity and details of the original image or video. Interpolation processing can be performed on the third image based on the denoised second image, so as to restore the colors in the image to achieve the effect of demosaicing. That is, the values of the red channel, green channel, and blue channel can be obtained respectively, and the target image is fused according to the obtained values of the red channel, green channel, and blue channel.

[0051] Optionally, the size of the target image can also be adjusted to correspond to the size of the input first image. For example, the size of the finally output target image is H*W*3, and 3 indicates that the target image after demosaicing processing includes 3 channel images such as the red channel, green channel, and blue channel.

[0052] In this embodiment, a second noisy image is obtained based on size transformation to reduce the network computing power. Further, the second image is input into an image processing model with a UNet structure for convolution processing, and an image set is output. The image set includes a noisy image obtained by segmenting the second noisy image and an image used for interpolation processing. By denoising and demosaicking the second image and the third image, a target image with better clarity and image quality is obtained, solving the problems of low image clarity, noise residue, false color artifacts, etc. in current image denoising and demosaicking methods.

[0053] Figure 2 is a flowchart of an image processing method shown according to some embodiments of the present disclosure. As Figure 2 shown, the method includes the following steps:

[0054] S201, preprocess the original first image to obtain a second noisy image.

[0055] In the embodiments of the present disclosure, the implementation method of step S201 can be implemented in any one of the embodiments of the present disclosure, and no limitation is made here and will not be elaborated further.

[0056] S202, input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set.

[0057] Among them, the preset network structure includes a U-shaped network structure. The image processing model with a U-shaped network structure includes multiple convolutional modules. Each convolutional module includes an encoder and a decoder. The encoder and decoder within a layer are connected, and the encoders between adjacent layers are connected, and the decoders between adjacent layers are connected; as Figure 3 shown, it shows the basic structure of the U-shaped network structure. Each row is a convolutional module, including an encoder, a decoder, and a skip connection. The encoder on the left side and the decoder on the right side in each convolutional module are directly connected, and the encoders and decoders of adjacent two convolutional modules are respectively connected.

[0058] In some implementations, the input of the encoder in the first convolutional module is the second image; the input of the encoder in the m-th convolutional module is: the output of the encoder in the (m - 1)-th convolutional module; the output of the encoder in the m-th convolutional module is: the input of the encoder in the (m + 1)-th convolutional module; that is, the input of the encoder in the current convolutional module is the output of the encoder in the previous convolutional module, and the output of the encoder in the current convolutional module is the input of the encoder in the next convolutional module.

[0059] Accordingly, the input of the decoder in the m-th convolutional module is: the output of the encoder in the m-th convolutional module and the output of the decoder in the (m + 1)-th convolutional module; the output of the decoder in the m-th convolutional module is the input of the decoder in the (m - 1)-th convolutional module; that is, the input of the decoder in the current convolutional module is the output of the encoder in the current convolutional module and the output of the decoder in the next convolutional module; the output of the decoder in the current convolutional module is the input of the decoder in the previous convolutional module; the output of the decoder in the last convolutional module is the third image; m is a natural number greater than 1.

[0060] S203. Determine a fourth image for denoising from the third image in the image set.

[0061] In some implementations, the channel identifier corresponding to the third image can be determined, and the third image can be sorted in ascending order of the channel identifier; where the channels are different color channels, such as the red R channel, the green G channel, and the blue B channel, and each channel has a corresponding channel identifier to reflect the channel, for example, the green channel identifier is 1, the red channel identifier is 2, and the blue channel identifier is 3; in some implementations, the third image may also include a complete noise image, and the channel identifier of the noise image can be 0; then, the third image is sorted in ascending order of the channel identifier, and the images in the third image are, in sequence, the noise image, the G-channel image, the R-channel image, and the B-channel image, and the number of images in each channel is at least the same as the number of the second images.

[0062] Further, determine the number of images of the second image, and select the fourth image in sequence according to the image data; for example, if the second image is 4, then select the first 4 images in the sorted third image as the fourth image.

[0063] S204. Perform a denoising operation on the second image and the fourth image to obtain a denoised fifth image.

[0064] In some implementations, the second image and the fourth image can be added channel by channel to obtain a denoised sixth image; that is, add the second image to the corresponding fourth image, and the fourth image includes the complete noise image in the second image, and the channel addition operation removes all the noise in the second image to obtain the denoised sixth image.

[0065] Further, in order to improve the denoising effect, a denoising adjustment parameter can be obtained, and the sixth image can be corrected based on the denoising adjustment parameter to obtain the fifth image; the denoising adjustment parameter can be preset, and the denoised sixth image is corrected through the denoising adjustment parameter. For example, set the denoising adjustment parameter to perform filtering processing on each pixel in the denoised sixth image to obtain a fifth image with a better denoising effect.

[0066] S205, perform demosaicing on the fifth image and the remaining third images in the image set to obtain a target image.

[0067] In some implementations, the remaining third images can be divided by channel to obtain third image groups for each color channel; for example, if the number of third images is 16, and excluding the fourth image used for denoising, the remaining third images are divided into G-channel images, R-channel images, and B-channel images; sort all the images by channel, and the sorting order can be: the first 4 images are noise images, that is, the fourth image, the third image group corresponding to the G-channel images can be the 5th to 8th images, the third image group corresponding to the R-channel images can be the 9th to 12th images, and the third image group corresponding to the B-channel images can be the 13th to 16th images.

[0068] Further, based on the third image group of the green G-channel and the fifth image, obtain the G-channel image; based on the G-channel image and the third image group of the red R-channel, obtain the R-channel image; based on the G-channel image and the third image group of the blue B-channel, obtain the B-channel image.

[0069] Optionally, for each color channel, add the third images in the third image group and the fifth image by channel to obtain candidate images for the color channel; add the pixel values of the candidate images for the color channel that belong to the same pixel position to obtain the target pixel value at the same pixel position, so as to obtain the channel image for the color channel.

[0070] Exemplarily, for example, the third image group corresponding to the G-channel images can be the 5th to 8th images. Add the 5th to 8th images and the 1st to 4th denoised fifth images corresponding to each other to obtain 4 candidate images for the G-channel. Add the pixel values of the 4 candidate images for the G-channel that belong to the same pixel position to obtain the target pixel value at the same pixel position, that is, the G-channel value, thereby obtaining the G-channel image; based on the G-channel image and the third image group of the red channel, that is, add the G-channel image and the 9th to 12th images of the red channel corresponding to each other to obtain 4 candidate images for the R-channel. Further add the pixel values of the 4 candidate images for the R-channel that belong to the same pixel position to obtain the target pixel value at the same pixel position, that is, the R-channel value; correspondingly, based on the G-channel value and the 13th to 16th images of the B-channel, obtain 4 candidate images for the B-channel, and then obtain the B-channel image based on the 4 candidate images for the B-channel.

[0071] Further, a target image is obtained based on the R-channel image, the G-channel image, and the B-channel image. In some implementations, a candidate target image can be obtained according to the R-channel image, the G-channel image, and the B-channel image. The candidate target image is an image with a size of H / 2 * W / 2. According to the size of the first image, the size of the candidate target image is adjusted to obtain the target image, that is, the size of the candidate target image is adjusted from H / 2 * W / 2 to the size H * W of the first image, so as to obtain a target image with a size of H * W * 3 including color values of three channels: the R channel, the G channel, and the B channel.

[0072] In this embodiment, the noisy second image is convolved by a graphics processing model with a U-shaped network structure to obtain multiple corresponding third images. The second image is denoised using the noise image in the third images to obtain a denoised fifth image. Further, based on the denoised fifth image and the third image groups of each channel, addition is performed to obtain respective color channel images, that is, the R-channel image, the G-channel image, and the B-channel image. The target image is obtained through the R-channel image, the G-channel image, and the B-channel image, improving the denoising and demosaicing effects of the image. And when processing the image, the image size is reduced without losing the original information to reduce the computing power and improve the processing speed. After processing, the size of the candidate target image is adjusted according to the size of the first image to obtain a target image with the same size, preferably solving the problems of noise residue and low clarity, and the denoising and demosaicing method has strong generalization ability.

[0073] Figure 4 It is a logic flowchart of an image processing method shown according to some embodiments of the present disclosure. As Figure 4 shown, the first image with a size of H * W is preprocessed Pre-Processing to obtain a noisy second image. The noisy second image is input into a graphics processing model Model with a U-shaped network structure for convolution processing to output a third image. Denoising processing is performed by adding the noise image in the third image to the noisy second image. Further, adjustment is performed through the denoising adjustment parameters in DenoisingAdjustment to obtain a denoised fifth image. Based on RAW2RGB, corresponding addition and other processing are performed on the fifth image and the remaining image in the third image to obtain a candidate target image, and the size of the candidate target image is adjusted to output a target image with a size of H * W * 3.

[0074] Figure 5 It is a structural block diagram of an image processing apparatus shown according to some embodiments of the present disclosure. Referring to Figure 5 , the image processing apparatus 500 includes:

[0075] The first acquisition module 501 is configured to preprocess the original first image to obtain a second image with noise;

[0076] The second acquisition module 502 is configured to input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the size of the third image is the same as that of the second image;

[0077] The processing module 503 is configured to perform denoising and demosaicing processing on the second image and the third image to obtain a target image.

[0078] In some implementations, the processing module 503 includes:

[0079] Determine a fourth image for denoising from the third images in the image set;

[0080] Perform denoising operations on the second image and the fourth image to obtain a fifth denoised image;

[0081] Perform demosaicing processing on the fifth image and the remaining third images in the image set to obtain a target image.

[0082] In some implementations, the processing module 503 includes:

[0083] Determine the channel identifiers corresponding to the third images, and sort the third images in ascending order according to the channel identifiers;

[0084] Determine the number of images of the second image, and select the fourth image in sequence according to the image data.

[0085] In some implementations, the processing module 503 includes:

[0086] Add the second image and the fourth image channel by channel to obtain a sixth denoised image;

[0087] Obtain a denoising adjustment parameter, and correct the sixth image based on the denoising adjustment parameter to obtain a fifth image.

[0088] In some implementations, the processing module 503 includes:

[0089] Divide the remaining third images channel by channel to obtain third image groups for each color channel;

[0090] Obtain a G-channel image according to the third image group of the green G channel and the fifth image;

[0091] Obtain an R-channel image according to the G-channel image and the third image group of the red R channel;

[0092] Obtain a B-channel image according to the G-channel image and the third image group of the blue B channel;

[0093] Based on the R-channel image, G-channel image, and B-channel image, a target image is obtained.

[0094] In some implementations, the apparatus 500 further includes:

[0095] For each color channel, the third image and the fifth image in the third image group are added channel by channel to obtain a candidate image for the color channel;

[0096] The pixel values of the candidate images for the color channels that belong to the same pixel position are added to obtain the target pixel value at the same pixel position, so as to obtain the channel image for the color channel.

[0097] In some implementations, the processing module 503 includes:

[0098] Based on the R-channel image, G-channel image, and B-channel image, a candidate target image is obtained;

[0099] According to the size of the first image, the size of the candidate target image is adjusted to obtain the target image.

[0100] In some implementations, the preset network structure includes a U-shaped network structure. The image processing model of the U-shaped network structure includes multiple convolutional modules. Each convolutional module includes an encoder and a decoder. The encoder and decoder within a layer are connected, the encoders between adjacent layers are connected, and the decoders between adjacent layers are connected.

[0101] In some implementations, the apparatus 500 further includes:

[0102] The input of the encoder in the first-layer convolutional module is the second image;

[0103] The input of the encoder in the m-th convolutional module is: the output of the encoder in the (m - 1)-th convolutional module;

[0104] The output of the encoder in the m-th convolutional module is: the input of the encoder in the (m + 1)-th convolutional module;

[0105] The input of the decoder in the m-th convolutional module is: the output of the encoder in the m-th convolutional module and the output of the decoder in the (m + 1)-th convolutional module;

[0106] The output of the decoder in the m-th convolutional module is the input of the decoder in the (m - 1)-th convolutional module;

[0107] The output of the decoder in the last convolutional module is the third image;

[0108] m is a natural number greater than 1.

[0109] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0110] In this embodiment, a second noisy image is obtained based on size transformation to reduce network cases. Further, the second image is input into an image processing model with a UNet structure for convolution processing, and an image set is output. The image set includes a noisy image obtained by segmenting the second noisy image and an image used for interpolation processing. By denoising and demosaicking the second image and a third image, a target image with better clarity and image quality is obtained, solving the problems of low image clarity, noise residue, false color artifacts, etc. in current image denoising and demosaicking methods.

[0111] Figure 6 It is a structural block diagram of another image processing device shown according to some embodiments of the present disclosure. For example, the image processing device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0112] Referring to Figure 6 , the image processing device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 616, and a communication component 616.

[0113] The processing component 602 generally controls the overall operation of the image processing device 600, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0114] The memory 604 is configured to store various types of data to support the operation of the device 600. Examples of such data include instructions for any application or method operating on the image processing device 600, contact data, phone book data, messages, pictures, videos, and the like. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0115] The power components 606 provide power to various components of the image processing device 600. The power components 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the image processing device 600.

[0116] The multimedia component 608 includes a screen that provides an output interface between the image processing device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the image processing device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0117] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC) that is configured to receive external audio signals when the image processing device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 606 or transmitted via the communication component 616. In some embodiments, the audio component 610 further includes a speaker for outputting audio signals.

[0118] The I / O interface 612 provides an interface between the processing component 602 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0119] The sensor assembly 616 includes one or more sensors for providing a status assessment of various aspects for the image processing device 600. For example, the sensor assembly 616 can detect the on / off state of the image processing device 600, the relative positioning of components, such as the display and keypad of the image processing device 600. The sensor assembly 614 can also detect a change in the position of the image processing device 600 or a component of the image processing device 600, the presence or absence of user contact with the image processing device 600, the orientation or acceleration / deceleration of the image processing device 600, and the temperature change of the image processing device 600. The sensor assembly 614 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0120] The communication component 616 is configured to facilitate communication between the image processing device 600 and other devices in a wired or wireless manner. The image processing device 600 can access a wireless network based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or a combination thereof. In some embodiments of the present disclosure, the communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of the present disclosure, the communication component 416 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0121] In some embodiments of the present disclosure, the image processing device 600 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0122] In some embodiments of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the above instructions can be executed by a processor 620 of the image processing device 600 to complete the above methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0123] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute an image processing method, the method comprising:

[0124] Preprocess the original first image to obtain a second image with noise;

[0125] Input the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the third image has the same size as the second image;

[0126] Denoise and demosaic the second image and the third image to obtain a target image.

[0127] Figure 7 It is a structural block diagram of another image processing device shown according to some embodiments of the present disclosure. For example, the image processing device 700 may be provided as a server. Referring to Figure 7 , the image processing device 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions executable by the processing component 722, such as application programs. The application programs stored in the memory 732 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform the above image processing method.

[0128] The image processing device 700 may further include a power supply component 726 configured to perform power management of the image processing device 700, a wired or wireless network interface 750 configured to connect the image processing device 700 to a network, and an input / output (I / O) interface 758. The image processing device 700 may operate based on an operating system stored in the memory 732, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0129] Those skilled in the art can also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of both. Whether such a function is implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art can use various methods to implement the described function for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present application.

[0130] In the foregoing detailed description, reference has been made to the accompanying drawings, in which specific aspects in which the present disclosure may be practiced are shown by way of illustration. In this regard, directional or positional relationship-indicating terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. may be used with reference to the orientation of the described figures. Since the components of the described device may be positioned in a plurality of different orientations, the directional terms may be used for illustrative purposes and not in a limiting sense. It should be understood that other aspects may be utilized and structural or logical changes may be made without departing from the concepts of the present disclosure. Accordingly, the following detailed description should not be taken in a limiting sense.

[0131] It should be understood that, unless otherwise specifically stated, the features of some embodiments of the various aspects of the present disclosure described herein may be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of any two or more thereof; similarly, "at least one of..." includes any one of the related listed items and any combination of any two or more thereof.

[0132] It should be understood that, unless otherwise clearly defined and limited, the terms such as "engage", "attach", "mount", "connect", "couple", "fix", etc. used in the embodiments of the present disclosure should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral body; it may be a mechanical connection, an electrical connection, or communicable with each other; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the internal communication of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For those of ordinary skill in the art, the specific meanings of the above terms in the present disclosure may be understood according to specific circumstances.

[0133] In addition, the word "above" used with respect to a component, element, or layer of material formed "above" or located "above" a surface may be used herein to mean that the component, element, or layer of material is "indirectly" positioned (e.g., placed, formed, deposited, etc.) on the surface such that one or more additional components, elements, or layers are disposed between the surface and the component, element, or layer of material. However, the word "above" used with respect to a component, element, or layer of material formed "above" or located "above" a surface may alternatively have a specific meaning: the component, element, or layer of material is "directly" positioned (e.g., placed, formed, deposited, etc.) on the surface, e.g., in direct contact with the surface.

[0134] Although terms such as "first", "second", and "third" may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Instead, these terms are only used to distinguish one component, part, region, layer, or section from another. Thus, the first component, part, region, layer, or section mentioned in the examples described herein may also be referred to as the second component, part, region, layer, or section without departing from the teachings of the examples. Additionally, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description herein, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0135] It should be understood that spatial relative terms, such as "above", "upper", "below", and "lower", are used herein to describe the relationship of one element shown in the figures to another element. In addition to the orientation depicted in the figures, such spatial relative terms are also intended to encompass different orientations of the device during use or operation. For example, if the device in the figures is flipped, an element described as "above" or "upper" relative to another element will then be "below" or "lower" relative to that other element. Thus, the term "above" encompasses both the above and below orientations depending on the spatial orientation of the device. The device may have other orientations (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein should be interpreted accordingly.

[0136] Furthermore, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as being advantageous compared to other aspects or designs. Instead, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified, or clear from the context, "X applies A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then "X applies A or B" is satisfied in any of the foregoing instances. Additionally, unless otherwise specified or clear from the context referring to the singular form, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0137] Similarly, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. Specifically with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. Additionally, although a particular feature of the present disclosure may have been disclosed with respect to only one of several implementations, such a feature may, as may be desired and advantageous for any given or particular application, be combined with one or more other features of other implementations. Further, with respect to the use of "comprising," "having," "including," "contains," or variants thereof in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "including."

[0138] Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0139] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Preprocessing the original first image to obtain a second image with noise; Inputting the second image into an image processing model with a preset network structure for convolution processing to obtain an image set, where the image set includes a third image, and the size of the third image is the same as that of the second image; Performing denoising and demosaicing processing on the second image and the third image to obtain a target image.

2. The method according to claim 1, wherein The performing denoising and demosaicing processing on the second image and the third image to obtain a target image includes: Determining a fourth image for denoising from the third images in the image set; Performing denoising operations on the second image and the fourth image to obtain a fifth denoised image; Performing demosaicing processing on the fifth image and the remaining third images in the image set to obtain the target image.

3. The method according to claim 2, wherein The determining a fourth image for denoising from the third images in the image set includes: Determining the channel identifiers corresponding to the third images, and sorting the third images in ascending order according to the channel identifiers; Determining the number of images of the second image, and sequentially selecting the fourth image according to the image data.

4. The method according to claim 2 or 3, characterized in that, The performing denoising operations on the second image and the fourth image to obtain a fifth denoised image includes: Adding the second image and the fourth image channel by channel to obtain a sixth denoised image; Obtaining a denoising adjustment parameter, and correcting the sixth image based on the denoising adjustment parameter to obtain the fifth image.

5. The method according to claim 2 or 3, characterized in that The performing demosaicing processing on the fifth image and the remaining third images in the image set to obtain the target image includes: Dividing the remaining third images channel by channel to obtain third image groups for each color channel; Obtaining a G-channel image according to the third image group of the green G channel and the fifth image; Obtaining an R-channel image according to the G-channel image and the third image group of the red R channel; Obtaining a B-channel image according to the G-channel image and the third image group of the blue B channel; Obtaining the target image according to the R-channel image, G-channel image, and B-channel image.

6. The method according to claim 5, characterized in that The method further includes: For each color channel, adding the third images in the third image group and the fifth image channel by channel to obtain a candidate image for the color channel; Adding the pixel values of the candidate images for the color channel that belong to the same pixel position to obtain the target pixel value at the same pixel position, so as to obtain the channel image of the color channel.

7. The method according to claim 5, characterized in that The obtaining the target image according to the R-channel image, G-channel image, and B-channel image includes: Obtaining a candidate target image according to the R-channel image, G-channel image, and B-channel image; Adjusting the size of the candidate target image according to the size of the first image to obtain the target image.

8. The method according to claim 1, wherein The preset network structure includes a U-shaped network structure. The image processing model of the U-shaped network structure includes multiple convolutional modules. Each convolutional module includes an encoder and a decoder. The encoder and decoder within a layer are connected, and the encoders between adjacent layers are connected. The decoders between adjacent layers are also connected.

9. The method according to claim 8, wherein The method further includes: The input of the encoder in the first-layer convolutional module is the second image; The input of the encoder in the m-th convolutional module is: the output of the encoder in the (m - 1)-th convolutional module; The output of the encoder in the m-th convolutional module is: the input of the encoder in the (m + 1)-th convolutional module; The input of the decoder in the m-th convolutional module is: the output of the encoder in the m-th convolutional module and the output of the decoder in the (m + 1)-th convolutional module; The output of the decoder in the m-th convolutional module is the input of the decoder in the (m - 1)-th convolutional module; The output of the decoder in the last-layer convolutional module is the third image; The m is a natural number greater than 1.

10. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enabling the mobile terminal to execute an image processing method. The method includes: Preprocessing the original first image to obtain a second image with noise; Inputting the second image into an image processing model of a preset network structure for convolutional processing to obtain an image set, where the image set includes a third image, and the size of the third image is the same as that of the second image; Performing denoising and demosaicing processing on the second image and the third image to obtain a target image.

11. A computer program product, characterized in that, Including a computer program, which when executed by a processor, implements the image processing method according to any one of claims 1 to 9.

Images