Image processing method, image processing device, storage medium, and electronic device

Abstract

The present disclosure provides an image processing method, an image processing device, a storage medium and an electronic device, and relates to the technical field of image and video processing. The image processing method comprises: acquiring a to-be-processed image; performing deconvolution processing and image reconstruction on the to-be-processed image based on point spread function prior information and noise prior information to generate a target image. The present disclosure improves the optimization effect on noise and aberration in the image, and has a lower implementation cost.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing technology, and in particular to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Noise and aberrations are common defects in images. Noise can be caused by electromagnetic interference to the imaging system, mechanical vibration of internal components, or defects in related materials. Aberrations are mainly caused by the deviation between the actual imaging results of the imaging system and the theoretically ideal imaging.

[0003] In related technologies, the optimization effect on noise and aberration during image processing is not ideal, resulting in low quality of the optimized image. Summary of the Invention

[0004] This disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device to at least partially improve the optimization effect on noise and aberration.

[0005] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be processed; performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate a target image.

[0006] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: an image acquisition module configured to acquire an image to be processed; and a deconvolution and reconstruction module configured to perform deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate a target image.

[0007] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image processing method of the first aspect described above and its possible implementations.

[0008] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of the first aspect and possible implementations thereof by executing the executable instructions.

[0009] The technical solution disclosed herein has the following beneficial effects:

[0010] On the one hand, the point spread function prior information and noise prior information reflect the aberrations and noise of the real imaging pattern. Using the point spread function prior information can effectively optimize aberrations in the image, and using the noise prior information can effectively optimize noise. Furthermore, deconvolution processing and image reconstruction can achieve progressive image optimization, thereby improving the optimization effect on noise and aberrations in the image, which is conducive to obtaining high-quality target images. On the other hand, the processing of this scheme is relatively simple and requires less information, thus having a low implementation cost and being suitable for deployment in more scenarios (such as mobile scenarios). Attached Figure Description

[0011] Figure 1 This diagram illustrates the system architecture of the operating environment for this exemplary embodiment.

[0012] Figure 2 A flowchart illustrating an image processing method in this exemplary embodiment is shown;

[0013] Figure 3 A sub-flowchart of an image processing method in this exemplary embodiment is shown;

[0014] Figure 4 This exemplary embodiment illustrates a flowchart of the iterative execution of deconvolution and image reconstruction.

[0015] Figure 5 This diagram illustrates the processing flow of ResUNet in this exemplary embodiment.

[0016] Figure 6 This example illustrates a flowchart of training a neural network.

[0017] Figure 7 A schematic flowchart illustrating the acquisition of the training dataset is shown in this exemplary embodiment;

[0018] Figure 8 A schematic flowchart illustrating the training process in this exemplary embodiment is shown;

[0019] Figure 9 A schematic flowchart illustrating the image processing method implemented using a deconvolution module and a neural network in this exemplary embodiment is shown.

[0020] Figure 10 This diagram illustrates the structure of an image processing apparatus according to this exemplary embodiment.

[0021] Figure 11 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0022] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0023] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0024] The following are explanations of the terms used in the text:

[0025] RAW images: Image format based on Bayer arrays, which can be raw images output from image sensors. Each pixel has only a single channel value, typically arranged in RGGB order.

[0026] Mosaic and Demosaic: Mosaic refers to the process of converting an RGB image to a RAW image, usually achieved through methods such as downsampling; Demosaic is the inverse process of mosaic, which refers to the process of converting an RGB image to a RAW image, usually achieved through methods such as upsampling (e.g., interpolation).

[0027] An ISP (Image Signal Processor) is a microprocessor configured on a camera module or terminal to process images. After a RAW image is acquired by an image sensor, the ISP performs a series of processes to output the image that the user can see. In this article, the ISP processing flow can be a process of converting a RAW image to an RGB image, which generally includes: demosaicing, white balance and digital gain, color correction, gamma compression, and tone mapping.

[0028] Image unprocessing: The process of converting RGB images to RAW images is the reverse of the ISP process described above. It can include: invert tone mapping, gamma decompression, invert color correction, invert white balance and digital gain, and mosaic processing.

[0029] The point spread function (PSF) is the response function of an imaging system to a light signal at a point in the real world. When a light signal originates at a point in the real world and is captured by an imaging system, the image on the image plane is not a point, but typically a diffuse spot. The PSF describes the distribution information of this diffuse spot. Functionally, the PSF is the spatial domain form of the optical transfer function of the imaging system.

[0030] Deconvolution: Based on the principle of the point spread function, a pixel in an image is the superposition of light signals from multiple points in its neighborhood in the real world on the imaging plane. This superposition achieves a filtering effect, but also causes image blurring and aberrations. Deconvolution is the inverse operation of convolution. In image processing, it uses convolution kernels to separate the image signal formed by the superposition of light signals, in order to restore the real-world light signal to a certain extent, achieving an inverse filtering effect.

[0031] Noise and aberrations are very common in images. For example, mobile devices such as smartphones often have zoom cameras, such as those that can switch from macro to microscopic photography. The imaging system is already optimized for a certain focal length, with minimal noise and aberrations. Zooming disrupts this optimal state, exacerbating the noise and aberrations. The visual perception for the user is a significant deterioration of the image during zooming.

[0032] The inventors have discovered that in related technologies, when optimizing noise and aberrations, the consideration of the actual imaging mode is often neglected, and the information related to noise and aberrations used differs from the actual information, resulting in poor optimization effects.

[0033] In view of one or more of the above-mentioned problems, the exemplary embodiments of this disclosure first provide an image processing method for optimizing noise and aberrations in an image to be processed, thereby obtaining an optimized target image. The following describes the method in conjunction with... Figure 1 The system architecture of the operating environment for this exemplary embodiment will be described.

[0034] refer to Figure 1 As shown, the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be an electronic device such as a mobile phone, tablet computer, or smart wearable device. The server 120 generally refers to the backend system providing image processing services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and the server 120 can be connected via a wired or wireless communication link for data interaction.

[0035] In one embodiment, the image processing method of this exemplary embodiment can be executed by the terminal 110. For example, an image taken by the user using the terminal 110 or an image selected by the user from the photo album of the terminal 110 is used as the image to be processed, and the terminal 110 outputs an optimized target image by executing the image processing method.

[0036] In one implementation, the image processing method of this exemplary embodiment can be executed by server 120. For example, an image processing app is installed on terminal 110. After the user selects an image to be processed in the app (such as a currently captured image or an image selected from the user's album), terminal 110 uploads the image to server 120, which then executes the image processing method to obtain an optimized target image. The target image can then be returned to terminal 110.

[0037] As can be seen from the above, in this exemplary embodiment, the entity executing the image processing method can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0038] The following is combined Figure 2 The workflow of the image processing method is explained. (Reference) Figure 2 As shown, the image processing method may include the following steps S210 and S220:

[0039] Step S210: Obtain the image to be processed;

[0040] Step S220: Based on the prior information of the point spread function and the prior information of noise, perform deconvolution processing and image reconstruction on the image to be processed to generate the target image.

[0041] Based on the above method, on the one hand, the point spread function prior information and noise prior information reflect the aberrations and noise of the real imaging mode. Using the point spread function prior information can effectively optimize aberrations in the image, and using the noise prior information can effectively optimize noise in the image. Furthermore, through deconvolution processing and image reconstruction, progressive image optimization can be achieved, thereby improving the optimization effect on noise and aberrations in the image, which is conducive to obtaining high-quality target images. On the other hand, the processing of this scheme is relatively simple and requires less information, thus having a low implementation cost and being suitable for deployment in more scenarios (such as mobile scenarios).

[0042] The following is about Figure 2 Each step in the process will be explained in detail.

[0043] refer to Figure 2 In step S210, the image to be processed is acquired.

[0044] This disclosure does not limit the source of the image to be processed. The image to be processed can be a currently captured image or any image selected by the user.

[0045] In one implementation, the above-mentioned acquisition of the image to be processed may include the following steps:

[0046] The response terminal switches from macro shooting mode to microscopic shooting mode, using the currently acquired image as the image to be processed.

[0047] Macro shooting mode is generally used at object distances in the centimeter range, such as 3cm to 10cm. Microscopic shooting mode is used at even smaller object distances (such as millimeters), such as as small as 5mm.

[0048] This exemplary embodiment supports dual-mode operation—macro mode and microscopic mode—using a single camera. Generally, switching from macro mode to microscopic mode on the same camera requires significant adjustments to the focal length and field of view (FOV), causing drastic changes in image quality and leading to increased noise and aberrations. By using the currently acquired image as the image to be processed, the image processing method in this exemplary embodiment can be executed to correct aberrations and suppress noise in a computational optical manner, satisfying various focusing scenarios at a microscopic perspective and ensuring image quality.

[0049] For example, switching from macro mode to microscopy mode involves a zoom process. Each frame of the preview image captured in real-time during this zoom can be used as the image to be processed, optimized into the target image, and then displayed. This ensures that the preview image seen by the user during the zoom process maintains high quality. Alternatively, images captured in microscopy mode after switching to microscopy mode can be used as the image to be processed, optimized into the target image, and then output. The target image is a high-quality microscopic image, making this solution applicable to portable microscopic imaging scenarios such as OCR (Optical Character Recognition) and portable diagnostics.

[0050] This disclosure does not limit the type of image to be processed. For example, the image to be processed can be any type such as RAW image, RGB image, YUV image, etc.

[0051] In one implementation, the image to be processed may include an RGB image to be processed. Obtaining the image to be processed may include the following steps:

[0052] Obtain the RAW image to be processed;

[0053] The RAW image to be processed is de-mosaiced to obtain the RGB image to be processed.

[0054] The RAW image to be processed can be the original image output from the image sensor. De-mosaic processing is performed on the RAW image without other processing steps in the ISP (Image Signal Processing) workflow (such as white balance, digital gain, color correction, etc.), resulting in the RGB image to be processed. In other words, the image processing flow is simplified as much as possible during the acquisition of the RGB image to be processed, ensuring that the RGB image to be processed primarily contains the original imaging information. This reduces other information introduced by the image processing workflow or interference with the original imaging information, which is more conducive to subsequent noise and aberration optimization of the RAW image to be processed, thus improving the optimization effect.

[0055] Continue to refer to Figure 2 In step S220, the image to be processed is deconvolved and reconstructed based on the point spread function prior information and noise prior information to generate the target image.

[0056] The point spread function prior information and noise prior information are related information about the point spread function and noise obtained based on prior knowledge and pre-calibration. The point spread function prior information can be a specific point spread function or its parameters. The noise prior information can include noise type, noise parameters, and noise distribution information. In this exemplary embodiment, the point spread function prior information and noise prior information are related to the image to be processed and can reflect the aberrations and noise of the true imaging mode of the image.

[0057] In one implementation, before performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate the target image, the image processing method may further include the following steps:

[0058] Based on the information from the imaging system of the image to be processed, the prior information of the point spread function is obtained.

[0059] The information from the imaging system can include calibration information of the imaging system for the point light source, reflecting the system's response to the point light source, thereby obtaining the prior information of the point spread function. Alternatively, the information from the imaging system can include geometric parameters, such as lens size, radian, and image distance, as well as material information, such as the physical parameters of the lens material, thereby calculating the prior information of the point spread function. This method ensures that the prior information of the point spread function matches the image to be processed, thus improving the processing effect when the image is subsequently processed based on the prior information of the point spread function.

[0060] In one implementation, before performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate the target image, the image processing method may further include the following steps:

[0061] Determine the shooting scene of the image to be processed, and use the noise calibration information corresponding to the shooting scene as the noise prior information.

[0062] The shooting scene of the image to be processed can be determined based on factors such as lighting conditions, color characteristics, and the type of object being photographed. For example, the shooting scene could be a close-up photograph of text taken under natural light. The type and characteristics of image noise may differ in different shooting scenes. Noise can be pre-calibrated for different shooting scenes to obtain noise calibration information corresponding to each scene. Generally, the more finely the shooting scene is divided, the more accurate the noise calibration information. After determining the shooting scene of the image to be processed, the noise calibration information corresponding to that scene is obtained as noise prior information. This information can more realistically reflect the noise situation in the shooting scene of the image to be processed, which is beneficial for subsequent high-quality processing of the image based on the noise prior information.

[0063] In one implementation, reference Figure 3 As shown, the above-mentioned deconvolution processing and image reconstruction of the image to be processed based on the prior information of the point spread function and the prior information of noise to generate the target image may include the following steps S310 and S320:

[0064] Step S310: Deconvolve the image to be processed based on the prior information of the point spread function to obtain the deconvolution image.

[0065] Deconvolution, a process that separates image signals formed by superimposed optical signals, is based on a realistic optical imaging model and achieves the effect of inverse image filtering. Based on prior information from the point spread function, a convolution kernel can be obtained for deconvolution. For example, based on the prior point spread function, the response values ​​within the spread region (which can be square, rectangular, circular, etc.) are obtained, and these response values ​​are normalized to form a convolution kernel. This kernel is then used to deconvolve the image to be processed, resulting in a deconvolutioned image. Deconvolution can, to some extent, recover real-world information lost due to the superposition of optical signals during imaging, such as low-frequency information, optimize aberrations in the image to be processed, and improve image blur.

[0066] In one implementation, the above-mentioned deconvolution processing of the image to be processed based on the prior information of the point spread function to obtain a deconvolutioned image may include the following steps:

[0067] The image to be processed is divided into multiple image blocks, and the convolution kernel corresponding to each image block is obtained based on the prior information of the point spread function.

[0068] Each image block is deconvolved using the convolution kernel corresponding to that image block to obtain the deconvolutioned image.

[0069] Considering the large field of view of the image to be processed and the non-uniform variation of the point spread function, convolution kernels for different regions within the field of view can be determined based on prior information about the point spread function. The image to be processed is then divided into image blocks, with each image block corresponding to a specific region. Each image block is then associated with the convolution kernel of its corresponding region. Deconvolution processing is then performed on each image block using its corresponding convolution kernel, which improves the precision of the deconvolution process compared to using the same convolution kernel to deconvolve the entire image.

[0070] Step S320: Input the noise prior information and the deconvolution image into the pre-trained neural network to reconstruct the image from the deconvolution image and output the target image.

[0071] The neural network can adopt an end-to-end structure, such as an encoder-decoder structure, for image reconstruction processing. For example, the neural network mentioned above can be ResUNet (a U-shaped network with residuals).

[0072] Noise prior information and deconvolutioned images can be concatenated and then input into a neural network, or they can be input into different input channels of the neural network. Noise prior information enables the neural network to learn noise information, eliminating or reducing noise during image reconstruction to recover a noise-free or low-noise image. The neural network processes the noise prior information and deconvolutioned images to reconstruct the deconvolutioned images, suppressing noise and further optimizing aberrations to obtain the target image.

[0073] exist Figure 3 Among the methods, image reconstruction based on deep learning is beneficial to improving the quality of image reconstruction.

[0074] In one embodiment, the image processing method may further include the following steps:

[0075] When performing deconvolution processing on the image to be processed based on the prior information of the point spread function, the control parameters used in the deconvolution process are obtained.

[0076] In deconvolution, the control parameters are used to control the optimization objective. For example, in image optimization, it may be desirable to focus on optimizing the image center or the image edges. By assigning values ​​to the control parameters, the optimization objective can be controlled to determine whether to optimize the image center or the image edges.

[0077] Accordingly, the above-mentioned inputting noise prior information and deconvolutioned image into a pre-trained neural network to reconstruct the image from the deconvolutioned image and output the target image may include the following steps:

[0078] The control parameters, noise prior information, and deconvolutioned image are input into a pre-trained neural network to reconstruct the image from the deconvolutioned image and output the target image.

[0079] In other words, besides noise prior information and the deconvolutioned image, control parameters can also be input into the neural network. This provides reference information for the neural network's image reconstruction process, influencing the optimization objective and achieving the desired goal. Furthermore, the control parameters are closely related to the deconvolution process and contain, to some extent, prior information about the point spread function. Inputting these parameters into the neural network allows it to learn this prior information, leading to better aberration optimization during image reconstruction. Therefore, both deconvolution and image reconstruction utilize prior information about the point spread function, fully leveraging this information and further improving image optimization results.

[0080] In one implementation, deconvolution and image reconstruction can be carried out as an iterative process. Specifically, refer to... Figure 4 As shown, the process of deconvolution and image reconstruction of the image to be processed, based on the prior information of the point spread function and the prior information of noise, to generate the target image, may include the following steps:

[0081] Step S410: Based on the prior information of the point spread function, deconvolve the image to be processed or the intermediate image output from the previous iteration to obtain the deconvolutioned image;

[0082] Step S420: Input the control parameters, noise prior information, and deconvolution image into the pre-trained neural network to reconstruct the image from the deconvolution image and output the intermediate image.

[0083] Step S430: Determine whether the iteration termination condition has been met. If yes, proceed to step S440; otherwise, increment the iteration count by one and jump to step S410 to start the next iteration.

[0084] Step S440: Output the intermediate image as the target image.

[0085] The iteration termination conditions may include, but are not limited to: reaching a predetermined number of iterations; the intermediate images meeting predetermined quality requirements, such as the image gradient being greater than a preset gradient threshold, the image noise being less than a preset noise level, etc.

[0086] It is evident that a single deconvolution process and image reconstruction cannot guarantee a high-quality reconstructed image. Therefore, multiple deconvolution processes and image reconstruction steps can be performed iteratively to continuously improve image quality and ultimately output a high-quality target image.

[0087] For example, the deconvolution process in the iteration can be referenced by the following formula:

[0088] (1)

[0089] Where p represents an image patch; y represents the image to be processed; y p Let represent the p-th image patch of the image to be processed; t represents the iteration number; and x represents the intermediate image output by the neural network during the iteration process. This represents the p-th image patch of the intermediate image output by the neural network in the (t-1)-th iteration; This represents the Fast Fourier Transform (FFT). This indicates the inverse Fast Fourier Transform; k represents the convolution kernel, which can be different for different image patches. express The conjugate of complex numbers; This represents a control parameter used to balance the relative sizes of the numerator and denominator terms in the formula to achieve different deconvolution optimization objectives. It can vary with the image patch and the number of iterations. It is the output of the deconvolution, representing the p-th image block of the deconvolutioned image output in the t-th iteration.

[0090] The iterative image reconstruction process can be referenced by the following formula:

[0091] (2)

[0092] in, This represents the image reconstruction process of a neural network; K and σ represent noise parameters in the noise prior information. K can be a noise constant, and σ can be the noise standard deviation (or noise variance). The parameters representing the prior information of the point spread function can be... Assign to This is equivalent to inputting control parameters, noise prior information, and the deconvolutioned image into a neural network, and then outputting an intermediate image of the fall after processing. When the iteration termination condition is met, As the target image.

[0093] Figure 5 The image reconstruction process using ResUNet is illustrated. The dimension of the deconvolutioned image is H. W 3, where H and W represent the height and width of the deconvolutioned image, and 3 represents the three channels: R, G, and B. The dimension of the control parameter is H. W 3. The dimension of the noise prior information is H. W 2. Concatenate the three elements to obtain a dimension of H. W Information 8. This information is input into ResUNet, first undergoing convolution, then processed through one or more (e.g., two) Resblocks. Each Resblock contains one convolution operation, one ReLU (Rectified Linear Unit) activation, and another convolution operation, resulting in a dimension H. W A feature image of dimension H / 2 is obtained by downsampling the feature image and processing it through one or more Resblocks. W / 2 The feature image is 128; it is downsampled and processed with Resblock step by step to obtain a dimension of H / 4. W / 4 256 feature images, with a dimension of H / 8 W / 8 The 512 feature image is used to achieve step-by-step extraction of image features, completing the encoding-side processing. The H / 8... W / 8 The 512 feature image is processed through one or more Resblocks and added to the feature image of the same dimension on the encoding side; then it is upsampled, processed through one or more Resblocks, and added to the feature image of the same dimension on the encoding side, resulting in a feature image of dimension H / 4. W / 4 A feature image of dimension H / 2 is obtained by progressively upsampling, Resblocking, and adding it to the feature image of the same dimension from the encoding side. W / 2 128 feature images, with dimension H W The feature image of dimension H is used to achieve step-by-step reconstruction of image features; finally, the feature image of dimension H is reconstructed. W The 64 feature images are convolved to complete the decoding process, and the output dimension is H. W 3. Target image.

[0094] Exemplary embodiments of this disclosure also provide methods for training neural networks. (See reference...) Figure 6 As shown, before performing deconvolution processing on the image to be processed based on the prior information of the point spread function, the image processing method may further include the following steps S610 to S640:

[0095] Step S610: Obtain the training dataset, which includes high-resolution images and corresponding low-resolution noisy images.

[0096] In this paper, high-resolution image (HR) and low-resolution image (LR) are relative concepts. A high-resolution image (HR image) has greater sharpness than a low-resolution image (LR image), but no specific sharpness value is specified. In the training dataset, high-resolution images are ideal images, which can be noise-free or low-noise images and can be used as labeled data (ground truth). Compared to high-resolution images, low-resolution noisy images have lower sharpness and also introduce more noise.

[0097] Step S620: Deconvolve the low-resolution noisy image based on the point spread function sample information to obtain the deconvolution sample image.

[0098] The point spread function (PSF) sample information is simulated PSF information used during training, which can mimic the PSF in real imaging modes. This PSF sample information is correlated with low-resolution noisy images and can reflect the aberrations in low-resolution noisy images. In one embodiment, the PSF sample information can be the same as the PSF prior information used in step S220. For example, the PSF prior information can be obtained based on the information of the imaging system of the image to be processed, and simultaneously used as the PSF sample information. In this way, the neural network can learn the real aberration information of the imaging system during training, which is beneficial for subsequent high-quality optimization of the image to be processed.

[0099] The specific implementation process of step S620 can be referred to the content of step S310. The principle of deconvolution processing is the same, so it will not be repeated here. The image after deconvolution processing of the low-resolution noisy image is called the deconvolution sample image, which is different from the deconvolution image mentioned above.

[0100] Step S630: Input the noise sample information and the deconvolution sample image into the neural network to be trained, and output the predicted reconstructed image.

[0101] The noise sample information is simulated noise information used during training, which can simulate image noise in real imaging modes. This noise sample information is associated with low-resolution noisy images and can reflect the noise situation in low-resolution noisy images. In one embodiment, the noise sample information and the noise prior information used in step S220 can be the same. For example, based on the shooting scene of the image to be processed, the noise calibration information corresponding to the shooting scene can be determined as the noise prior information and used as the noise sample information. In this way, the neural network can learn the real noise information in the shooting scene during training, which is beneficial for subsequent high-quality optimization of the image to be processed.

[0102] For the specific implementation process of step S630, please refer to step S320 or Figure 5 The processing steps for some parts are the same as those for neural networks, so they will not be repeated here. Since the neural network here is in a state of being trained, it is difficult to achieve ideal image reconstruction. Therefore, the image output by the neural network is called the predicted reconstructed image, which is different from the target image mentioned above.

[0103] Step S640: Determine the loss function value based on the high-resolution image and the predicted reconstructed image, and update the parameters of the neural network using the loss function value.

[0104] As mentioned above, since the neural network is in a state of being trained, it is difficult to achieve ideal image reconstruction. Therefore, the predicted reconstructed image output by the neural network differs from the ideal high-resolution image. The loss function value can be calculated based on this difference. This disclosure does not limit the specific form of the loss function; for example, it can use the MAE (Mean Absolute Error) of the two images, the L2 norm, etc. Based on the loss function value, the gradients of each parameter of the neural network can be calculated using backpropagation, and the gradients can be used to update each parameter. Through iterative updates of the parameters, the neural network is trained.

[0105] In one implementation, when updating the parameters of the neural network using the loss function value, the deconvolution parameters can also be updated. Here, the deconvolution process is implemented as an inverse filter, and the deconvolution parameters are learnable parameters, such as the inverse filter strength and size. The deconvolution parameters can be iteratively updated using a method similar to machine learning training, continuously optimizing the deconvolution process and improving its performance. In effect, this is equivalent to training the deconvolution process and the neural network as a single machine learning model. This model can perform deconvolution processing and image reconstruction, achieving end-to-end image processing, and is easy to deploy, making it suitable for mobile devices and other scenarios.

[0106] In one embodiment, the high-resolution image may include a high-resolution RGB image; the low-resolution noisy image may include a low-resolution noisy RGB image. (See reference) Figure 7 As shown, obtaining the training dataset can include the following steps:

[0107] Acquire high-resolution RGB images (HR RGB);

[0108] High-resolution RGB images are convolved (Conv) using point spread function sample information (PSF sample) to form low-resolution RGB images (LR RGB);

[0109] Image inversion processing is performed on low-resolution RGB images to create low-resolution RAW images (LR RAW); image inversion processing is the reverse of the ISP processing flow.

[0110] Noise is added to low-resolution RAW images using noise sample information to create low-resolution noisy RAW images (LR RAW+noise).

[0111] De-mosaicing is performed on low-resolution noisy RAW images to create low-resolution noisy RGB images (LRRGB+noise).

[0112] Among them, convolution processing of high-resolution RGB images using point spread function sample information can simulate the superposition of light signals in real imaging modes, introducing real aberration information and image blur into the resulting low-resolution RGB images.

[0113] Since the acquired RGB images (whether high-resolution or low-resolution) have undergone ISP processing, in order to restore the original image information, image inversion processing is performed to restore the low-resolution RGB images to low-resolution RAW images. In one implementation, the low-resolution RGB image can be sequentially subjected to inverse tone mapping, gamma decompression, inverse color correction, inverse white balance and digital gain, and mosaic processing to achieve image inversion processing and form a low-resolution RAW image. This strictly follows the ISP processing flow and performs completely reverse processing, which can completely offset the effects of the ISP processing flow, thereby restoring a clean RAW image without post-processing intervention.

[0114] Noise is added to low-resolution RAW images using noise sample information, followed by de-mosaic processing to form low-resolution noisy RGB images. It is evident that the noise is added within the RAW domain. In real imaging modes, noise primarily arises during the acquisition of light signals and the output of RAW images by the imaging system. Therefore, the noisy images (including low-resolution noisy RAW images and low-resolution noisy RGB images) in this exemplary embodiment effectively simulate noise conditions in real imaging modes.

[0115] In related technologies, the statistical laws governing signals and noise are studied for denoising (such as anisotropic diffusion and total variation denoising). However, these methods differ somewhat from the actual physics of image formation, leading to inaccurate denoising results. This solution, based on image inversion processing, restores the RGB image to the RAW domain and adds noise within the RAW domain. This facilitates the modeling and learning of real-world imaging noise. Furthermore, RAW format images retain more detailed and realistic data structure information, making accurate denoising easier.

[0116] In one implementation, the predicted reconstructed image includes a predicted reconstructed RGB image. The determination of the loss function value based on the high-resolution image and the predicted reconstructed image may include the following steps:

[0117] High-resolution RGB images are reverse-processed to generate high-resolution RAW images;

[0118] The predicted reconstructed RGB image is pixelated to form the predicted reconstructed RAW image.

[0119] The loss function value is determined based on the high-resolution RAW image and the predicted reconstructed RAW image.

[0120] In other words, both the high-resolution RGB image and the predicted / reconstructed RGB image can be converted to the RAW domain, and the loss function values ​​for both can be calculated in the RAW domain. This allows the loss function values ​​to more accurately reflect the essential differences between the images, improving training results. Furthermore, the image data volume in the RAW domain is smaller, and the loss function calculation process is simpler.

[0121] The deconvolution part is implemented as a deconvolution module, which can be regarded as a whole with the neural network (such as ResUNet). Figure 8 A schematic flowchart illustrating the training process is shown. The process involves: acquiring a high-resolution RGB image; performing convolution processing on the high-resolution RGB image using point spread function sample information to form a low-resolution RGB image; performing image inversion processing on the low-resolution RGB image to form a low-resolution RAW image; adding noise to the low-resolution RAW image using noise sample information to form a low-resolution noisy RAW image; and performing demosaic processing on the low-resolution noisy RAW image to form a low-resolution noisy RGB image. This is the image preprocessing process. Then, the low-resolution noisy RGB image, point spread function sample information, and noise sample information are input into the deconvolution module and ResUNet to be trained, and the output is a predicted reconstructed RGB image. The predicted reconstructed RGB image is then mosaicked to form a predicted reconstructed RAW image. The high-resolution RGB image is then reverse-processed to form a high-resolution RAW image. The high-resolution RAW image and the predicted reconstructed RAW image are then substituted into a pre-constructed loss function to calculate the loss function value. The parameters of the deconvolution module and ResUNet are updated using the loss function value to train the deconvolution module and ResUNet.

[0122] In one embodiment, if the image to be processed is an RGB image obtained by de-mosaicing a RAW image, then the target RGB image can be obtained through step S220. Furthermore, the image processing method may also include the following steps:

[0123] The target RGB image is pixelated to obtain the target RAW image.

[0124] Therefore, end-to-end processing from the RAW image to be processed to the target RAW image is achieved. This end-to-end processing flow does not conflict with the ISP processing flow. Subsequently, the target RAW image can be input into the ISP, and the final image can be obtained through the ISP processing flow. In this way, the image processing method in this exemplary embodiment is compatible with any manufacturer, platform, or model of ISP configured on the terminal, thus improving the applicability of this solution.

[0125] Figure 9 A schematic flowchart illustrating an image processing method implemented using a deconvolution module and a neural network (such as ResUNet) is shown. The process involves acquiring a RAW image to be processed, performing demosaicing to obtain an RGB image; inputting the RGB image, point spread function prior information (PSF prior), and noise prior information into the deconvolution module and ResUNet, outputting the target RGB image; and then performing mosaic processing on the target RGB image to obtain the target RAW image. This achieves optimization of the RAW domain image.

[0126] Exemplary embodiments of this disclosure also provide an image processing apparatus. (See reference...) Figure 10 As shown, the image processing apparatus 1000 may include the following modules:

[0127] The data acquisition module 1010 is configured to acquire the image to be processed;

[0128] The deconvolution and reconstruction module 1020 is configured to perform deconvolution processing and image reconstruction on the image to be processed based on the point spread function prior information and noise prior information to generate the target image.

[0129] In one implementation, the above-mentioned deconvolution processing and image reconstruction of the image to be processed based on point spread function prior information and noise prior information to generate a target image includes:

[0130] Based on the prior information of the point spread function, the image to be processed is deconvolved to obtain the deconvolved image;

[0131] The noise prior information and the deconvolutioned image are input into a pre-trained neural network to reconstruct the image from the deconvolutioned image and output the target image.

[0132] In one implementation, the deconvolution and reconstruction module 1020 is configured as follows:

[0133] When deconvolving the image to be processed based on the prior information of the point spread function, the control parameters used in the deconvolution process are obtained; the control parameters, noise prior information, and deconvolution image are input into a pre-trained neural network to reconstruct the image from the deconvolution image and output the target image.

[0134] In one implementation, the deconvolution and reconstruction module 1020 is further configured to perform the following steps before performing deconvolution processing on the image to be processed based on the point spread function prior information:

[0135] Obtain the training dataset, which includes high-resolution images and corresponding low-resolution noisy images;

[0136] Deconvolution processing is performed on low-resolution noisy images based on point spread function sample information to obtain deconvolution sample images;

[0137] Input the noise sample information and the deconvolution sample image into the neural network to be trained, and output the predicted reconstructed image;

[0138] The loss function value is determined based on the high-resolution image and the predicted reconstructed image, and the parameters of the neural network are updated based on the loss function value.

[0139] In one implementation, the high-resolution image includes a high-resolution RGB image; the low-resolution noisy image includes a low-resolution noisy RGB image; and obtaining the training dataset includes:

[0140] Acquire high-resolution RGB images;

[0141] High-resolution RGB images are convolved using point spread function sample information to form low-resolution RGB images;

[0142] Image inversion processing is performed on a low-resolution RGB image to create a low-resolution RAW image; image inversion processing is the reverse of the processing flow of an image signal processor.

[0143] Noise is added to low-resolution RAW images using noise sample information to create low-resolution noisy RAW images;

[0144] Demosaicing is performed on low-resolution noisy RAW images to generate low-resolution noisy RGB images.

[0145] In one embodiment, the above-described image inversion processing of a low-resolution RGB image to form a low-resolution RAW image includes:

[0146] The low-resolution RGB image is sequentially subjected to inverse tone mapping, gamma decompression, inverse color correction, inverse white balance and digital gain, and mosaic processing to form a low-resolution RAW image.

[0147] In one implementation, the predicted reconstructed image includes a predicted reconstructed RGB image; determining the loss function value based on the high-resolution image and the predicted reconstructed image includes:

[0148] High-resolution RGB images are reverse-processed to generate high-resolution RAW images;

[0149] The predicted reconstructed RGB image is pixelated to form the predicted reconstructed RAW image.

[0150] The loss function value is determined based on the high-resolution RAW image and the predicted reconstructed RAW image.

[0151] In one implementation, the above-mentioned deconvolution processing of the image to be processed based on the prior information of the point spread function to obtain a deconvolutioned image includes:

[0152] The image to be processed is divided into multiple image blocks, and the convolution kernel corresponding to each image block is obtained based on the prior information of the point spread function.

[0153] Each image block is deconvolved using the convolution kernel corresponding to that image block to obtain the deconvolutioned image.

[0154] In one embodiment, the image to be processed includes an RGB image to be processed; the target image includes a target RGB image; and acquiring the image to be processed includes:

[0155] Obtain the RAW image to be processed;

[0156] Perform demosaic processing on the RAW image to be processed to obtain the RGB image to be processed;

[0157] The deconvolution and reconstruction module 1020 is also configured as follows:

[0158] After generating the target RGB image, a mosaic effect is applied to the target RGB image to obtain the target RAW image.

[0159] In one embodiment, the above-mentioned acquisition of the image to be processed includes:

[0160] The response terminal switches from macro shooting mode to microscopic shooting mode, using the currently acquired image as the image to be processed.

[0161] In one implementation, the data acquisition module 1010 is further configured to:

[0162] Before the deconvolution and reconstruction module 1020 performs deconvolution processing and image reconstruction on the image to be processed based on the prior information of the point spread function and the prior information of the noise, and generates the target image, the prior information of the point spread function is obtained based on the information of the imaging system of the image to be processed.

[0163] In one implementation, the data acquisition module 1010 is further configured to:

[0164] Before the deconvolution and reconstruction module 1020 performs deconvolution processing and image reconstruction on the image to be processed based on the point spread function prior information and noise prior information to generate the target image, the shooting scene of the image to be processed is determined, and the noise calibration information corresponding to the shooting scene is used as the noise prior information.

[0165] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0166] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0167] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0169] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0170] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0171] Exemplary embodiments of this disclosure also provide an electronic device, such as the terminal 110 or server 120 described above. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes the executable instructions to perform the methods of this exemplary embodiment.

[0172] The following is based on Figure 11 Taking the mobile terminal 1100 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 11 The structure can also be applied to fixed types of equipment.

[0173] like Figure 11 As shown, the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0174] Processor 1101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an information service provider (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The method in this exemplary embodiment may be executed by one or more of the AP, GPU, ISP, and DSP.

[0175] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0176] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 1102 can also store application data, such as images, videos, and other files.

[0177] The communication function of mobile terminal 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1100.

[0178] The display screen 1106 is used to implement display functions, such as displaying the user interface.

[0179] The camera module 1107 is used to implement shooting functions, such as capturing images to be processed.

[0180] The audio module 1108 is used to implement audio functions, such as playing audio and capturing voice.

[0181] The power module 1109 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status.

[0182] The sensor module 1110 may include one or more sensors for implementing corresponding sensing and detection functions.

[0183] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0184] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0185] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; Based on the point spread function prior information and noise prior information, the image to be processed is deconvolved and reconstructed to generate the target image. The step of performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate a target image includes: The image to be processed is deconvolved based on the prior information of the point spread function to obtain a deconvolved image. Obtain the control parameters used in the deconvolution process; The control parameters, the noise prior information, and the deconvolutioned image are input into a pre-trained neural network to reconstruct the image from the deconvolutioned image and output the target image.

2. The method according to claim 1, characterized in that, The deconvolution process of the image to be processed based on the prior information of the point spread function to obtain the deconvolutioned image includes: The response values ​​within the diffusion region are obtained based on the prior point spread function. The response values ​​are then normalized to form a convolution kernel. The convolution kernel is then used to deconvolve the image to be processed to obtain a deconvolutioned image.

3. The method according to claim 1, characterized in that, The point spread function prior information and the noise prior information are used to reflect the aberrations and noise of the true imaging mode of the image to be processed.

4. The method according to claim 1, characterized in that, Before performing deconvolution processing on the image to be processed based on the prior information of the point spread function, the method further includes: Obtain a training dataset, which includes high-resolution images and corresponding low-resolution noisy images of the high-resolution images; The low-resolution noisy image is deconvolved based on the point spread function sample information to obtain a deconvolved sample image. The noise sample information and the deconvolution sample image are input into the neural network to be trained, and the predicted reconstructed image is output. The loss function value is determined based on the high-resolution image and the predicted reconstructed image, and the parameters of the neural network are updated using the loss function value.

5. The method according to claim 4, characterized in that, The high-resolution image includes a high-resolution RGB image; the low-resolution noisy image includes a low-resolution noisy RGB image; obtaining the training dataset includes: Acquire the high-resolution RGB image; The high-resolution RGB image is convolved using the point spread function sample information to form a low-resolution RGB image; The low-resolution RGB image is subjected to image inversion processing to form a low-resolution RAW image; image inversion processing is the reverse of the processing flow of the image signal processor. Noise is added to the low-resolution RAW image using noise sample information to form a low-resolution noisy RAW image; The low-resolution noisy RAW image is de-mosaiced to form the low-resolution noisy RGB image.

6. The method according to claim 5, characterized in that, The step of performing image inversion processing on the low-resolution RGB image to form a low-resolution RAW image includes: The low-resolution RGB image is sequentially subjected to inverse tone mapping, gamma decompression, inverse color correction, inverse white balance and digital gain, and mosaic processing to form the low-resolution RAW image.

7. The method according to claim 5, characterized in that, The predicted reconstructed image includes a predicted reconstructed RGB image; determining the loss function value based on the high-resolution image and the predicted reconstructed image includes: The high-resolution RGB image is reversed to form a high-resolution RAW image; The predicted reconstructed RGB image is pixelated to form a predicted reconstructed RAW image; The loss function value is determined based on the high-resolution RAW image and the predicted reconstructed RAW image.

8. The method according to claim 1, characterized in that, The deconvolution process of the image to be processed based on the prior information of the point spread function to obtain the deconvolutioned image includes: The image to be processed is divided into multiple image blocks, and the convolution kernel corresponding to each image block is obtained based on the prior information of the point spread function. The image blocks are deconvolved using the convolution kernels corresponding to each image block to obtain the deconvolutioned image.

9. The method according to claim 1, characterized in that, The image to be processed includes an RGB image to be processed; the target image includes a target RGB image; the process of acquiring the image to be processed includes: Obtain the RAW image to be processed; The RAW image to be processed is subjected to demosaic processing to obtain the RGB image to be processed; After generating the target RGB image, the method further includes: The target RGB image is pixelated to obtain the target RAW image.

10. The method according to claim 1, characterized in that, The process of acquiring the image to be processed includes: The response terminal switches from macro shooting mode to microscopic shooting mode, and uses the currently acquired image as the image to be processed.

11. The method according to claim 1, characterized in that, Before performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate the target image, the method further includes: The prior information of the point spread function is obtained based on the information of the imaging system of the image to be processed.

12. The method according to claim 1, characterized in that, Before performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate the target image, the method further includes: The shooting scene of the image to be processed is determined, and the noise calibration information corresponding to the shooting scene is used as the noise prior information.

13. An image processing apparatus, characterized in that, include: The image acquisition module is configured to acquire the image to be processed. The deconvolution and reconstruction module is configured to perform deconvolution processing and image reconstruction on the image to be processed based on the point spread function prior information and noise prior information to generate the target image. The step of performing deconvolution processing and image reconstruction on the image to be processed based on point spread function prior information and noise prior information to generate a target image includes: The image to be processed is deconvolved based on the prior information of the point spread function to obtain a deconvolved image. Obtain the control parameters used in the deconvolution process; The control parameters, the noise prior information, and the deconvolutioned image are input into a pre-trained neural network to reconstruct the image from the deconvolutioned image and output the target image.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 12.

15. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 12 by executing the executable instructions.

Images