Image processing method and device, ISP, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image processing method, and the method is arranged in an ISP, and comprises the steps: obtaining a collection image, inputting the collection image and parameters, related to target flaw removal, of the collection image into a trained neural network model, obtaining an output image, removing a target flaw in the collection image, and displaying the output image, the trained neural network model is obtained by training a preset neural network model based on the training image, the parameters of the training image related to target flaw removal and the training image after target flaw removal. The embodiment of the invention further provides an image processing device, the ISP, electronic equipment and a computer storage medium.

Description

Technical Field

[0001] This application relates to image processing technology, and in particular, to an image processing method, apparatus, ISP, electronic device, and computer storage medium. Background Art

[0002] In the related art, for image processing technology, usually after obtaining the image processed by an Image Signal Processor (ISP), image processing is performed again based on information such as the statistical characteristics of the processed image to remove problems such as noise, blur, and shadow in the image.

[0003] However, the image quality of the output image obtained by the above method of performing image processing on the image processed by the ISP is poor. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, and computer storage medium, which can improve the image quality of the output image obtained by multi-frame fusion.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image processing method, which is applied to an ISP and includes:

[0007] Obtain a captured image;

[0008] Input the captured image and the parameters related to removing target defects of the captured image into a trained neural network model to obtain an output image, so as to remove the target defects in the captured image and display the output image;

[0009] Wherein, the trained neural network model is obtained by training a preset neural network model based on a training image, the parameters related to removing target defects of the training image, and the training image after removing target defects.

[0010] In a second aspect, embodiments of this application provide an image processing apparatus, which is disposed in an ISP and includes:

[0011] An obtaining module, configured to obtain a captured image;

[0012] A processing module, configured to input the captured image and the parameters related to removing target defects of the captured image into a trained neural network model to obtain an output image, so as to remove the target defects in the captured image and display the output image;

[0013] Among them, the trained neural network model is obtained by training a preset neural network model based on training images, parameters of the training images related to removing target defects, and the training images after removing the target defects.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a storage medium storing processor-executable instructions; the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, the image processing method described in one or more of the above embodiments is executed.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions, and when the executable instructions are executed by one or more processors, the processor executes the image processing method described in one or more of the above embodiments.

[0016] An embodiment of the present application provides an image processing method, device, electronic device, and computer storage medium. An acquisition image is obtained, and the acquisition image and the parameters of the acquisition image related to removing target defects are input into a trained neural network model to obtain an output image, so as to remove the target defects in the acquisition image and display the output image. The trained neural network model is obtained by training a preset neural network model based on training images, parameters of the training images related to removing target defects, and the training images after removing the target defects. That is to say, in the embodiment of the present application, after the image is acquired, the target defects are removed by inputting the acquisition image and the parameters of the acquisition image related to removing target defects into the trained neural network model to obtain an output image and display it. In this way, in the ISP, the trained neural network model obtained by training a preset neural network model based on training images, parameters of the training images related to removing target defects, and the training images after removing the target defects is used to process the acquisition image to remove the target defects, which can achieve the removal of target defects in the ISP and improve the image quality of the output image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic flowchart of an optional image processing method provided by an embodiment of the present application;

[0018] Figure 2 It is a schematic flowchart of Example 1 of an optional image processing method provided by an embodiment of the present application;

[0019] Figure 3 It is a schematic flowchart of Example 2 of an optional image processing method provided by an embodiment of the present application;

[0020] Figure 4Schematic flowchart of the third example of an optional image processing method provided by an embodiment of the present application;

[0021] Figure 5 Schematic flowchart of the fourth example of an optional image processing method provided by an embodiment of the present application;

[0022] Figure 6a Schematic diagram of the first combination method of ISP parameters and neural network provided by an embodiment of the present application;

[0023] Figure 6b Schematic diagram of the second combination method of ISP parameters and neural network provided by an embodiment of the present application;

[0024] Figure 6c Schematic diagram of the third combination method of ISP parameters and neural network provided by an embodiment of the present application;

[0025] Figure 7 Schematic structural diagram of an optional image processing device provided by an embodiment of the present application;

[0026] Figure 8 Schematic structural diagram of an optional ISP provided by an embodiment of the present application;

[0027] Figure 9 Schematic structural diagram of an optional electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0029] In the related art, although image or video noise reduction technologies consider information such as the statistical characteristics of images or videos, they all belong to post-processing methods, that is, decoding the encoded image or video into an image or video and then performing noise reduction processing. Among them, the noise introduced by the ISP of the mobile terminal is not considered.

[0030] When the ISP hardware module of the mobile terminal processes images or videos, it will synchronously generate parameters such as International Organization for Standardization (ISO) and Dynamic Range Control Gain (DRC Gain) (which can also be called DRC gain). These parameters are closely related to the noise intensity of the current video image, resulting in an impact on noise in ISP processing.

[0031] In view of the technical problem that the image quality of the output image obtained by the image processing method based on the image after ISP processing is poor, an embodiment of the present application provides an image processing method. Figure 1 It is a schematic flowchart of an optional image processing method provided by an embodiment of the present application. As Figure 1 shown, the image processing method may include:

[0032] S101: Obtain a captured image;

[0033] S102: Input the captured image and the parameters related to removing target defects of the captured image into the trained neural network model to obtain an output image, so as to remove the target defects in the captured image and display the output image;

[0034] In order to reduce the noise of the captured image in the ISP, in an embodiment of the present application, the image sensor (sensor) of the electronic device captures an image to obtain the captured image.

[0035] After obtaining the captured image, the sensor transmits the captured image to the ISP. In the ISP, a trained neural network model is set. In the ISP, before inputting to the trained neural network model, some processing can be performed on the captured image. For example, the captured image can be subjected to color space conversion to obtain the captured image, or the captured image can be subjected to autofocus processing to obtain the captured image. Here, the embodiment of the present application does not make specific limitations.

[0036] It should be noted that the above method for obtaining the captured image is the same as the method for obtaining the training image. The processing order of the trained neural network model in the ISP corresponds to the way of obtaining the captured image. For example, if the captured image is obtained by performing color space conversion on the captured image, the trained neural network model is placed after the color space conversion; if the captured image is obtained by performing autofocus processing on the captured image, the trained neural network model is placed after the autofocus processing.

[0037] After obtaining the captured image, the captured image and the parameters related to removing target defects of the captured image are input into the trained neural network model, so as to realize the processing of removing the target defects in the captured image, obtain the output image, and display it.

[0038] Among them, the trained neural network model is obtained by training a preset neural network model based on training images, parameters related to removing target defects in the training images, and the training images after removing the target defects. That is to say, in order to obtain the trained neural network model, in the training stage, first obtain the training images, obtain the parameters related to removing target defects in the training images, and obtain the training images after removing the target defects, so as to use them to train the preset neural network model to update the parameters in the preset neural network model and obtain the trained neural network model.

[0039] Among them, the above-mentioned preset neural network model can be a convolutional neural network or a Transformer structure. Here, the network structure is not limited in the embodiments of the present application. The above network structure can include common neural network modules, for example, self-attention modules, convolutional modules, and downsampling modules, etc.

[0040] In addition, the above-mentioned preset neural network model can also be a combination of the preset neural network model and the parameters related to removing target defects. Among them, there can be various combination methods. Here, the embodiments of the present application do not make specific limitations on this.

[0041] Regarding the above parameters related to removing target defects, in an optional embodiment, the parameters related to removing target defects can include one or more of the following:

[0042] Parameters related to denoising, parameters related to deblurring, parameters related to removing shadows, and parameters related to removing moiré.

[0043] It can be understood that for the images collected by the sensor, there are generally defects such as noise, blurring, shadows, and moiré. Then, the parameters related to the target defects can be one or more of the parameters related to denoising, the parameters related to deblurring, the parameters related to removing shadows, and the parameters related to removing moiré.

[0044] It should be noted that for the relevant parameters of each type of defect among the relevant parameters of different defects, the preset neural network model is trained so that the trained neural network model can remove the defects in the image. If the preset neural network model is trained for the relevant parameters of each of the above-mentioned types of defects, then the trained neural network model can remove all the defects in the image.

[0045] In this way, the target defects can be removed through the above-mentioned trained neural network model, so that the target defects in the image can be improved in the ISP in a model-based manner, and the image quality of the output image is improved.

[0046] In order to denoise the acquired image, in an optional embodiment, the parameters related to removing target defects are the parameters related to removing noise, and S102 may include:

[0047] Input the acquired image, the parameters related to denoising of the acquired image, and the mask image of the acquired image into the trained neural network model to obtain an output image, so as to remove the noise in the acquired image and display the output image.

[0048] It can be understood that for image denoising, since the noise patterns in different brightness regions of the image are different, the signal-to-noise ratio in bright regions is high and the required noise reduction intensity is very small; the signal-to-noise ratio in dark regions is low and a strong noise reduction force is required. Therefore, here, in addition to inputting the acquired image and the parameters related to denoising of the acquired image into the trained neural network model to obtain an output image to remove the noise in the acquired image, the acquired image, the parameters related to denoising of the acquired image, and the mask image of the acquired image can also be input into the trained neural network model to obtain an output image to remove the noise in the acquired image.

[0049] Correspondingly, in training the preset neural network model, the trained neural network model is obtained by training the preset neural network model based on the training image, the parameters related to denoising of the training image, the mask image of the training image, and the denoised training image.

[0050] That is to say, in training the preset neural network model, the mask image of the training image also needs to be considered. Among them, the mask image of the training image is generated by extracting the image blocks in the training image whose brightness values are greater than the preset brightness threshold. That is, the image blocks in the training image whose brightness values are greater than the preset brightness threshold are extracted, and the extracted image blocks are used as the brightness regions. The values of the brightness regions in the mask image are set to one value, and the remaining image blocks are used as the dark regions. The values of the dark regions in the mask image are set to another value, so as to obtain the mask image of the training image. Training the preset neural network model with it, the training image, the parameters related to denoising of the training image, and the denoised training image can obtain a trained neural network model, which can better denoise the acquired image.

[0051] Among them, the mask image of the acquired image is generated by extracting the image blocks in the acquired image whose brightness values are greater than the preset brightness threshold; similar to the acquisition method of the mask image of the above training image, it will not be elaborated here.

[0052] It should be noted that, in order to ensure the operation efficiency, the above method for screening the brightness region can adopt a local highlight segmentation algorithm, which is a traditional local threshold segmentation algorithm, that is, the image is divided into blocks, and threshold segmentation is performed on each image block respectively to separate the highlight region and the dark part. In addition, other segmentation methods that can separate the highlight region and the dark part can also be used here.

[0053] In this way, through the above method of adding the mask image for training, the trained neural network model can take into account the influence of the brightness distribution of the collected image on the noise, and further improve the denoising effect of the trained neural network model on the image.

[0054] For the above parameters related to denoising, in an optional embodiment, the parameters related to denoising may include one or more of the following:

[0055] Noise type parameter, ISO parameter, DRC gain parameter, and zoom (ZOOM) parameter.

[0056] It can be understood that the above parameters related to noise reduction may include one or more of the noise type parameter, ISO parameter, DRC gain parameter, and ZOOM parameter. Among them, the above locking mode, analog gain, and DRC gain of the sensor belong to the parameters related to the sensor and noise, and the analog gain, local tone mapping function, and ZOOM parameter of the image belong to the parameters related to the ISP and noise. It can be seen that the above noise type parameter, ISO parameter, DRC gain parameter, and ZOOM parameter are the parameters related to denoising when the image is input into the ISP.

[0057] Among them, the noise type parameter is mainly related to the locking mode of the sensor, the ISO parameter is mainly related to the product of the analog gain and the digital gain of the image, the DRC gain parameter is mainly related to the DRC gain of the image, and the ZOOM parameter is mainly related to the correlation between the ZOOM ratio of the image and upsampling / downsampling.

[0058] In this way, through the above parameters related to denoising for the training of the preset neural network, that is, integrating the parameters related to denoising into the noise reduction scheme, it is beneficial to determine the noise level of the image or video and implement more accurate noise reduction. While removing the noise, protect the details in the image or video to the greatest extent possible.

[0059] In denoising, in order to obtain the noise type parameter of the collected image, in an optional embodiment, the method may further include:

[0060] Obtain the locking mode of the sensor when the image is collected;

[0061] Determine the noise type parameter of the collected image according to the locking mode.

[0062] Understandably, when acquiring a captured image, the locking mode of the sensor that captures the image can be obtained simultaneously. Among them, the locking mode includes the binning mode and the remosaic mode. As is well known, the sensor can include two modes. One is the binning mode, which is an image readout mode that adds the charges sensed by adjacent pixels and reads them out in the form of one pixel. The other is the remosaic mode, which is designed based on the principle of Demosaic and is an algorithm for restoring a RAW image in a more complex color block arrangement mode into a normal RGB three-channel image.

[0063] In practical applications, when the locking mode of the sensor during image acquisition is the binning mode, the noise type parameter of the captured image is 0.5. When the locking mode of the sensor during image acquisition is the remosaic mode, the noise type parameter of the captured image is 1.

[0064] It should be noted that when training a preset neural network model, the noise type parameter of the training image can also be obtained in the above manner, which will not be elaborated here.

[0065] In this way, by determining the noise type parameter of the captured image through the locking mode of the sensor during image acquisition, the locking mode of the sensor during image acquisition can be used as an input to train the preset neural network model, so that the trained neural network model takes into account the locking mode of the sensor during denoising, which can further improve the denoising effect.

[0066] During denoising, in order to obtain the ISO parameter of the captured image, in an optional embodiment, the above method may further include:

[0067] Obtain the analog gain of the captured image and the digital gain of the captured image;

[0068] Normalize the product of the analog gain of the captured image and the digital gain of the captured image to obtain the ISO parameter of the captured image.

[0069] Understandably, when acquiring a captured image, for the sensor during image acquisition, the parameters related to noise can include the analog gain. For the ISP during ISP processing of the captured image, the parameters related to noise can include the digital gain.

[0070] Here, the product of the analog gain of the captured image and the digital gain of the captured image can be obtained. This product is the ISO parameter of the captured image. In order to make the ISO parameter more standardized, in the embodiments of the present application, after obtaining the product, the ratio of the ISO parameter to the maximum value of the ISO parameter is used as the ISO parameter of the captured image to achieve the normalization of the ISO parameter.

[0071] Among them, the maximum value of the above ISO parameter can be a set value or the maximum value in a set. For example, in training a preset neural network, the maximum value in the ISO parameter set of the training images can be used as the maximum value of the above ISO. Here, the embodiments of the present application do not make specific limitations in this regard.

[0072] It should be noted that when training a preset neural network model, the ISO parameter of the training images can also be obtained in the above manner, which will not be elaborated here.

[0073] In this way, by normalizing the product of the analog gain and digital gain of the captured image to determine the ISO parameter of the captured image, the analog gain and digital gain of the captured image can be used as inputs to train the preset neural network model, so that the trained neural network model takes into account the analog gain and digital gain of the captured image during denoising, which can further improve the denoising effect.

[0074] During denoising, in order to obtain the DRC gain parameter of the captured image, in an optional embodiment, the above method may further include:

[0075] Obtain the DRC gain of the captured image;

[0076] Normalize the DRC gain of the captured image to obtain the DRC gain parameter of the captured image.

[0077] It can be understood that when capturing the captured image, for the sensor during image capture, the parameters related to noise may include the DRC gain.

[0078] Here, the DRC gain of the captured image can be obtained. In order to make the DRC gain more standardized, in the embodiments of the present application, after obtaining the DRC gain, the ratio of the DRC gain to the maximum value of the DRC gain is used as the DRC gain parameter of the captured image to achieve the normalization of the DRC gain.

[0079] Among them, the maximum value of the above DRC gain can be a set value or the maximum value in a set. For example, in training a preset neural network, the maximum value in the DRC gain set of the training images can be used as the maximum value of the above DRC gain. Here, the embodiments of the present application do not make specific limitations in this regard.

[0080] It should be noted that when training a preset neural network model, the DRC gain parameter of the training image can also be obtained in the above manner, which will not be elaborated here.

[0081] In this way, by using the DRC gain of the sensor during image acquisition to determine the DRC gain parameter of the acquired image, the DRC gain parameter of the acquired image can be used as an input to train the preset neural network model, so that the trained neural network model takes into account the DRC gain parameter during denoising, which can further improve the denoising effect.

[0082] In denoising, in order to obtain the ZOOM parameter of the acquired image, in an optional embodiment, the above method may further include:

[0083] Obtain the ZOOM magnification of the acquired image and determine the ZOOM magnification;

[0084] Determine the ratio of the ZOOM magnification of the acquired image to the determined ZOOM magnification as the ZOOM parameter.

[0085] It can be understood that when acquiring the acquired image, for the ISP during ISP processing of the acquired image, the parameters related to noise may include the ZOOM magnification.

[0086] Here, the ZOOM magnification of the acquired image can be obtained. In order to make the ZOOM magnification more standardized, in the embodiments of the present application, in order to save power consumption, image downsampling or upsampling will be performed at a certain stage of the ISP, and both upsampling and downsampling will affect the noise pattern in the image. The magnification of upsampling and downsampling is related to the ZOOM magnification. Usually, only the upsampling and downsampling magnifications of a certain ZOOM segment will change with the change of the ZOOM magnification. Therefore, here, the ratio of the ZOOM magnification of the acquired image to the determined ZOOM magnification is determined as the ZOOM parameter.

[0087] Among them, the determined ZOOM magnification is: the value when the ZOOM magnification of the acquired image remains unchanged with the change of the sampling magnification. That is to say, when the ZOOM magnification of the acquired image that remains unchanged with the change of the sampling magnification is determined, for example, it is 1.7, and by dividing the ZOOM magnification of the acquired image by 1.7, after normalizing the ZOOM magnification, the ZOOM parameter can be obtained.

[0088] It should be noted that when training a preset neural network model, the ZOOM parameter of the training image can also be obtained in the above manner, which will not be elaborated here.

[0089] In this way, when collecting images for ISP processing, the ZOOM magnification of the ISP can be used to determine the ZOOM parameters of the collected images. The ZOOM parameters of the collected images can be used as inputs to train a preset neural network model, so that the trained neural network model takes the ZOOM parameters into account during denoising, which can further improve the denoising effect.

[0090] To obtain the above-trained neural network model, training images need to be acquired. In an alternative embodiment, the training images are obtained through ISP simulation of a first sample image set and a second noise parameter set.

[0091] It can be understood that the first sample image set can be collected first, where the first sample image set is collected under a preset first noise parameter set; that is, after the first noise parameter set is preset, images can be collected for each parameter set under the first noise parameter set to obtain the first sample image set.

[0092] In the embodiment of the present application, to obtain the second noise parameter set, the second noise parameter set is generated based on the first noise parameter set. Here, the parameter range of a certain noise in the above first noise parameter set can be used to determine the parameter range of a certain noise in the second noise parameter set, and then the second noise parameters can be randomly generated based on this parameter range.

[0093] After the first sample image set and the second noise parameter set are obtained, each image in the first sample image set can be subjected to ISP simulation with each group of noise parameter sets in the second noise parameter set to obtain training images, which are noisy images.

[0094] In this way, training images can be obtained through the above simulation method. The first sample image set is made more abundant by the preset first noise parameter set, and the obtained training images are made more abundant by the generated second noise parameter set, which helps to improve the denoising ability of the trained neural network model.

[0095] To obtain the above-trained neural network model, denoised training images need to be obtained. In an alternative embodiment, the denoised training images are obtained through ISP simulation of a second sample image set and a second noise parameter set.

[0096] It can be understood that the second sample image set can be collected first, where the second sample image set is obtained by denoising the first sample image set under the first noise parameter set. That is, after the first noise parameter set is preset, images can be collected for each parameter set under the first noise parameter set to obtain the first sample image set, and then the images in the first sample image set are denoised. Here, a denoising algorithm can be used for denoising, for example, calculating the mean value of the images in the first sample image set to achieve denoising.

[0097] After obtaining the second sample image set, perform ISP simulation on the second sample image set and the second noise parameter set, so that denoised training images can be obtained.

[0098] In this way, denoised training images can be obtained through the above simulation method. The second sample image set is made richer by the preset first noise parameter set and the first sample image set, and the obtained training images and denoised training images are made richer by the generated second noise parameter set, which helps to improve the denoising ability of the trained neural network model.

[0099] For the above first noise parameter set and second noise parameter set, in an alternative embodiment, the first noise parameter set includes one or more of the following: digital gain, DRC gain.

[0100] That is to say, the parameters in the first noise parameter set are all parameters related to denoising for the sensor. Collecting images under this parameter to obtain the first sample image set can obtain richer training images, providing favorable training data for obtaining the denoising ability of the trained neural network model.

[0101] Among them, the second noise parameter set includes one or more of the following: analog gain, local tone mapping function, ZOOM magnification. Among them, in the training stage, the value range of the digital gain can be determined according to the value range of the analog gain, and the value range of the local tone mapping function can be generated according to the range of the DRC gain. In this way, the digital gain is generated within the value range of the digital gain, the local tone mapping function is generated within the generated value range of the local tone mapping function, and the ZOOM magnification is generated, so as to obtain the second noise parameter set.

[0102] That is to say, the parameters in the second noise parameter set are all parameters related to denoising for ISP processing. Collecting images under this parameter to obtain the second sample image set and obtaining denoised training images together with the second noise parameter can obtain richer denoised training images, providing favorable training data for obtaining the denoising ability of the trained neural network model.

[0103] In order to obtain the second sample image set, in an alternative embodiment, the second sample image set is obtained by denoising the first sample image set under the first noise parameter set using the frame stacking technique.

[0104] Understandably, in order to denoise the first sample image set under the first noise parameter set to obtain the second sample image set here, a frame stacking technique can be used to achieve this. Specifically, by calculating the mean image of the first sample image set, the second sample image can be obtained. The noise in the first sample image set can be removed through the frame stacking technique to obtain the second sample image.

[0105] In this way, the denoising of the first sample image set is achieved through the above frame stacking technique, providing a data basis for obtaining training images.

[0106] In an optional embodiment, in order to obtain a more diverse preset neural network model, the first feature map in the preset neural network model is replaced with: a mathematical operation or splicing of one or more parameters in the relevant parameters for removing target defects and the first feature map.

[0107] Understandably, in addition to using the preset neural network model, the architecture of the preset neural network model can also be improved. Specifically, the determined relevant parameters for removing target defects are combined with the preset neural network model to re-obtain the preset neural network model.

[0108] Here, a mathematical operation or splicing of one or more parameters in the relevant parameters for removing target defects and the first feature map in the preset neural network model can be used to replace the first feature map in the preset neural network model. Among them, the first feature map can be any one or more feature maps in the preset neural network.

[0109] Among them, for the relevant parameters for removing target defects, when there are multiple feature maps in the first feature map, some parameters can perform a mathematical operation or splicing with the first sub-feature map in the first feature map, and some parameters can perform a mathematical operation or splicing with the second sub-feature map in the first feature map. When there is one feature map in the first feature map, it can also be that all the parameters perform a mathematical operation or splicing with the first feature map. In this way, a variety of neural network models can be derived for training.

[0110] In this way, for the architectures of different derived neural network models, in the trained neural network models obtained through training, the parameter map corresponding to the obtained relevant parameters for removing target defects can be input into the trained neural network model to further improve the denoising ability of the trained neural network model.

[0111] Furthermore, in an optional embodiment, in order to obtain a more diverse preset neural network model, the first feature map in the preset neural network model is replaced with: a mathematical operation or splicing of the parameter map corresponding to the relevant parameters for removing target defects and the first feature map.

[0112] Understandably, in addition to using a preset neural network model, the architecture of the preset neural network model can also be improved. Specifically, the parameter map corresponding to the relevant parameters for removing target defects is combined with the preset neural network model to re-obtain the preset neural network model.

[0113] Among them, the parameter map corresponding to the relevant parameters for removing target defects is: the parameter map corresponding to any one of the relevant parameters for removing target defects. That is to say, the parameter map corresponding to any one of the relevant parameters for removing target defects can be used for mathematical operations or splicing with the first feature map in the preset neural network model to replace the first feature map in the preset neural network model.

[0114] It should be noted that for the parameter maps corresponding to different parameters, when there are multiple feature maps in the first feature map, some parameter maps can be used for mathematical operations or splicing with the first sub-feature map in the first feature map, and some parameter maps can be used for mathematical operations or splicing with the second sub-feature map in the first feature map. When there is one feature map in the first feature map, it can also be that all the parameters are used for mathematical operations or splicing with the first feature map. In this way, a variety of neural network models can be derived for training.

[0115] In this way, for the architectures of different derived neural network models, in the trained neural network models obtained through training, the relevant parameters for removing target defects obtained can be input into the trained neural network model to further improve the denoising ability of the trained neural network model.

[0116] In addition, in order to obtain a more diverse preset neural network model, in an optional embodiment, the first feature map in the preset neural network model is replaced with: the mathematical operation or splicing of the parameter map after feature extraction and the first feature map.

[0117] Understandably, in the above method of deriving multiple neural network models using parameter maps, after feature extraction of the parameter map, the mathematical operation or splicing of the parameter map after feature extraction and the first feature map in the preset neural network model can be used to replace the first feature map in the preset neural network model.

[0118] It should be noted that for the above three replacement methods, at least two of the above three replacement methods can be combined for different parameters. To derive a neural network model for training, here, the embodiments of the present application do not make specific limitations in this regard.

[0119] In this way, for the architectures of different derived neural network models, in the trained neural network models obtained through training, the parameter map after feature extraction can be input into the trained neural network model to further improve the denoising ability of the trained neural network model.

[0120] The following is an example to describe the method for processing images described in one or more of the above embodiments.

[0121] This example mainly focuses on video denoising and includes two parts: the production of video denoising data and the design and training of an adaptive denoising network.

[0122] The production of video denoising data adopts a combination of RAW image stacking and ISP emulator simulation. First, multiple frames of noisy RAW images directly output by the sensor are collected on the mobile device, and clean RAW images are obtained by using a computer for stacking. Then, the noisy RAW images and parameters such as randomly generated digital gain and ZOOM ratio are input into the ISP emulator to obtain noisy YUV images as the input of the network; the clean RAW images corresponding to the noisy RAW images and the just randomly generated parameters such as digital gain and ZOOM ratio are input into the ISP emulator together to obtain clean YUV images as the ground truth label (GT) of the data for training the neural network.

[0123] The input of the adaptive denoising network is divided into three categories: known ISP parameters (equivalent to the denoising-related parameters of the above training images), noise maps (equivalent to the above training images), and highlight information extracted from the noise maps (equivalent to the mask images of the above training images). After these three categories of inputs are input into the adaptive denoising network, they are fused by means of mathematical operations or concatenation (concat) and the like, so that the adaptive denoising network learns the relationship between the ISP parameters and the image noise, and while removing the noise, protects the highlight details in the image.

[0124] Training the neural network of this example requires not only the noise map, but also information such as the noise form, DRC gain, and ISO parameters corresponding to the noise map. The training data set needs to include various combination methods of ISP parameters to ensure the effects of the finally trained neural network in various scenarios. If all real-acquired data is used to train the network, a large amount of data needs to be collected, and the time and labor costs are high. Therefore, a combination of RAW image stacking directly output by the sensor and ISP simulation is adopted to produce the training data set to reduce the workload of data collection.

[0125] Most of the existing sensors on mobile devices have two modes: binning and remosaic. Among them, the binning mode combines four pixels in the Bayer array into one pixel; remosaic converts the quand Bayer image into a normal Bayer image. The binning mode and the remosaic mode have different noise patterns, and data needs to be collected separately, that is, lock the binning mode and collect the RAW image in the binning mode; lock the remosaic mode and collect the RAW image in the remosaic mode.

[0126] The DRC gain reflects the ISO multiple relationship between long and short exposure images. In the daytime backlight scene, the ISO is very small, only about one or two hundred, but the DRC gain is very large, resulting in a large amount of noise in the dark areas of the video image. To ensure the noise reduction effect under different DRC gains, different DRC gains need to be locked and a batch of data needs to be collected.

[0127] The RAW image directly output by the snsor only contains analog gain, and the digital gain is applied by the subsequent ISP module. To collect the RAW image directly output by the snsor, only different analog gains need to be locked.

[0128] Figure 2 It is a schematic flowchart of Example 1 of an optional image processing method provided by an embodiment of the present application. As Figure 2 shown, the specific operation steps of data collection and frame stacking are as follows:

[0129] S201: List the trigger table for locking DRC gain and analog gain;

[0130] S202: Write a script on the mobile device;

[0131] S203: Execute the script to collect multiple frames of noisy RAW images;

[0132] Specifically, first, according to the maximum DRC gain range and analog gain range, list the trigger table for locking DRC gain and analog gain, and make scripts for locking DRC gain and analog gain in the binning mode and remosaic mode respectively; execute a script for locking DRC gain and analog gain; in multiple scenes with rich details, collect multiple frames (for example, 100 frames) of noisy RAW images directly output by the sensor and the corresponding meta information files. Among them, the meta information file contains parameters such as analog gain and DRC gain; the meta information is equivalent to the above-mentioned first noise parameter set;

[0133] Repeat the above steps until all the scripts for locking DRC gain and analog gain are executed once;

[0134] Repeat the above steps to collect images in the binning mode and the remosaic mode respectively.

[0135] S204: Transmit multiple frames of noisy RAW images and the corresponding meta information files collected by the mobile device to the computer.

[0136] S205: Use the program on the computer to calculate the mean image of multiple frames of noisy RAW images, and regard this mean image as a noise-free clean RAW image.

[0137] After data collection and frame stacking are completed, a pair of noisy RAW image-clean RAW image data is obtained. Then, use the ISP emulator to simulate the noisy RAW image and the clean RAW image respectively to obtain a dataset for training the neural network. Figure 3 This is a schematic flowchart of the second example of an optional image processing method provided by the embodiment of the present application. As Figure 3 shown, the specific operation steps of the simulation are as follows:

[0138] S301: Generate ISP module parameters for each pair of image pairs;

[0139] Specifically, randomly select a pair of noisy RAW image-clean RAW image; based on the DRC gain and analog gain in the meta information of the noisy RAW image, randomly generate ISP module parameters such as digital gain, random local tone mapping function (Local Tone Mapping, LTM), and random ZOOM magnification.

[0140] Among them, the digital gain, LTM, and ZOOM magnification are equivalent to the above-mentioned second noise parameter set.

[0141] S302: Simulate the noisy RAW image and the ISP module parameters, and simulate the warning RAW image and the ISP module parameters;

[0142] S303: After simulation, obtain the noisy YUV image and the clean YUV image.

[0143] Specifically, input the noisy RAW image and the ISP module parameters generated in S302 into the ISP emulator together to obtain the noisy YUV image as the input of the neural network. Input the clean RAW image and the ISP module parameters generated in S302 into the ISP emulator together to obtain the clean YUV image as the GT of the neural network. Repeat the above steps to generate a training dataset.

[0144] The goal of the adaptive noise reduction network in this example is to adapt to noise reduction in all ISO ranges for various scenarios and protect details to the greatest extent. The input of the adaptive noise reduction network is divided into three categories: noisy YUV images, high-light mask images (mask) extracted from the noisy YUV images (equivalent to the mask images of the above training images), and ISP parameter information (equivalent to the parameters related to denoising of the above training images).

[0145] Figure 4 FIG. is a schematic flowchart of the third example of an optional image processing method provided by an embodiment of the present application. As Figure 4 shown, the noisy YUV image: The noisy YUV image is made through the above Figure 3 The high-light mask extracted from the noisy YUV image: The purpose of extracting the high-light mask is to protect high-light details. The brightness distribution of some noisy YUV images with high contrast is uneven. The bright areas have a high signal-to-noise ratio and require very little noise reduction intensity; the dark areas have a low signal-to-noise ratio and require a strong noise reduction force. This phenomenon is particularly common in scenarios with low ISO and high DRC gain.

[0146] To protect the high-light details in the YUV image, a local high-light segmentation algorithm is used to extract the regions with high signal-to-noise ratio in the image, generate a high-light region mask as prior information, and input it into the adaptive noise reduction network. To ensure the operation efficiency, this local high-light segmentation algorithm is a traditional local threshold segmentation algorithm, that is, the image is divided into blocks, and threshold segmentation is performed on each image block separately to separate the high-light region and the dark part. Other segmentation methods that can separate the high-light region and the dark part can also be used here.

[0147] In Figure 4 after obtaining the noise type, ISO parameter, DRC gain, and ZOOM magnification, they are preprocessed respectively to obtain ISP parameters.

[0148] Among them, ISP parameters: The parameters input into the adaptive noise reduction network are some parameters that will affect the noise form. Some important parameters are selected here for illustration. The solutions include but are not limited to the following parameters.

[0149] (1) Noise type parameters: mainly refer to the binning mode and the remosaic mode.

[0150] In order to be able to distinguish between the two noise forms, the noise form is encoded as a floating point number between 0 and 1. For example, the noise type parameter of the binning mode is set to 0.5, and the noise type parameter of the remosaic mode is set to 1.0.

[0151] (2) ISO parameter: This ISO parameter refers to the ISO parameter after digital gain is applied. The maximum value of the ISO parameter is much greater than 1. In order to input the ISO parameter into the adaptive neural network, the ISO parameter is normalized by its maximum value, that is, the ISO parameter is divided by the maximum value of the ISO parameter to obtain the normalized ISO parameter, where the ISO maximum value is the maximum value of the ISO parameter in the training dataset:

[0152]

[0153] where ISO norm represents the normalized ISO parameter, ISO represents the product of digital gain and analog gain, and ISO max represents the maximum value of the ISO parameter in the training dataset.

[0154] (3) DRC gain parameter: This parameter indirectly reflects the magnitude of the noise difference between the highlight and dark areas in the YUV image. The maximum value of the DRC gain is also greater than 1, and it is normalized by its maximum value:

[0155]

[0156] where DRC norm represents the normalized DRC gain parameter, DRC represents the DRC gain, and DRC max represents the maximum value of the DRC gain in the training dataset.

[0157] (4) ZOOM parameter: Common video specifications on mobile devices currently include 4K, 1080P, 720P, etc. To save power, downsampling or upsampling of the YUV image is performed at a certain stage of the ISP, and both upsampling and downsampling affect the noise pattern in the YUV image. The upsampling / downsampling ratio is related to the ZOOM parameter. Usually, only the upsampling / downsampling ratio of a certain ZOOM segment changes with the ZOOM ratio. Therefore, as long as the relationship between the upsampling / downsampling ratio and the ZOOM ratio is summarized, the noise pattern can be indirectly obtained. The specific method is as Figure 5 shown.

[0158] Figure 5 is a schematic flowchart of Example 4 of an optional image processing method provided by an embodiment of the present application. As Figure 5 shown, the method may include:

[0159] S501: Determine the ZOOM ratio of each image;

[0160] S502: Obtain the relationship between the upsampling / downsampling ratio and the ZOOM ratio;

[0161] S503: Normalize the ZOOM magnification to obtain the ZOOM parameter;

[0162] Among them, determine the value of the ZOOM magnification that does not change with upsampling and downsampling, calculate the ratio of the ZOOM magnification to the determined value of the ZOOM magnification that does not change with upsampling and downsampling, and use the ratio as the ZOOM parameter.

[0163] S504: Use the ZOOM parameter to train the neural network for the training dataset.

[0164] There are multiple fusion methods between ISP parameters and the neural network, including but not limited to Figures 6a - 6c the methods shown. Figure 6a This is a schematic diagram of an optional combination method one between ISP parameters and the neural network provided by an embodiment of the present application. As Figure 6a shown, after these parameters go through the above-mentioned preprocessing operations, the parameters after these preprocessing operations can be directly operated with the data of the adaptive network or a certain layer or several layers in the middle through mathematical operations such as addition, subtraction, multiplication, division, or operations such as concat.

[0165] Figure 6b This is a schematic diagram of an optional combination method two between ISP parameters and the neural network provided by an embodiment of the present application. As Figure 6b shown, after these parameters go through the above-mentioned preprocessing operations, these parameters after preprocessing operations can also be expanded into a parameter map, and the parameter map is concat-connected or mathematically operated with the input of the adaptive neural network or a certain layer or several layers in the middle.

[0166] Figure 6c This is a schematic diagram of an optional combination method three between ISP parameters and the neural network provided by an embodiment of the present application. As Figure 6c shown, after these parameters go through the above-mentioned preprocessing operations, these parameters after preprocessing operations can be expanded into a parameter map, and feature extraction is performed on these parameter maps. For example, feature extraction is performed using a neural network block (block), that is, after these parameter maps go through several layers of neural network processing, they are concat-connected or mathematically operated with the input of the adaptive neural network or a certain layer or several layers in the middle.

[0167] Among them, the backbone structure of the adaptive neural network can be a convolutional neural network or a transformer structure, and the network structure is not limited. The adaptive neural network can include common neural network modules, such as self-attention modules, convolutional modules, downsampling modules, etc. When training the adaptive neural network, only the loss function between the output of the adaptive network and the GT needs to be calculated, the loss function is backpropagated, and the learnable parameters of the network are updated.

[0168] In the above example, the traditional method is adopted for the local highlight segmentation algorithm. The method for segmenting the highlight area can be extended to a deep learning method, using a neural network to pre-segment the high signal-to-noise ratio area and the low signal-to-noise ratio area, generating a mask, and inputting it into the noise reduction network. Taking video noise reduction in the YUV domain as an example for illustration, in essence, this scheme can be extended to noise reduction in the RAW domain and the RGB domain. This example mainly takes the noise reduction task as an example for illustration, and this example can also be extended to other low-level vision tasks, such as deblurring, de-shadowing, de-moiré and other tasks. Only the parameters input to the neural network need to be replaced with parameters related to the video blur degree, shadow depth, and moiré pattern. The parameters listed in this example are proposed based on the current status of the sensor and ISP. If there are future technological updates, new parameters affecting the noise form and intensity can also be added to this example. The preprocessing method of the parameters and the combination method of the parameters and the neural network are not limited to the examples in this example.

[0169] This example proposes a data acquisition method for locking the DRC gain and the analog gain, which can effectively control the amount of data collected in each ISO segment and DRC gain segment, and reduce the workload of data acquisition. Using the method of simulating with random ISP parameters to obtain the training data set can expand the infinite training set based on the limited actually collected data, and reduce the labor cost of obtaining training data. By integrating multiple ISP parameters that affect the noise form and intensity into the neural network, it is realized that the ISP parameters control the video noise reduction intensity of the neural network, avoiding blurring details due to too high noise reduction intensity or excessive noise in the video due to too weak noise reduction. Subsequently, the noise reduction strength can also be controlled by adjusting the ISP parameters input to the neural network, increasing the flexibility of debugging. Extracting the highlight mask from the noise map and inputting it into the neural network as a prior can protect the details of the highlight area. By changing the values, shapes, and positions of the highlight mask, the local noise reduction intensity of the network can be controlled to achieve user-defined noise reduction.

[0170] This example discloses an adaptive noise reduction scheme combined with the ISP of a mobile device, which includes a data production method and an adaptive network structure design. When making data, a combination of actually collected data and simulation is used. Only a small amount of real data needs to be collected, and a large amount of data can be obtained for training the neural network through random ISP parameters and simulation, saving labor and time costs. In the network structure design part, the ISP parameters that affect the video noise form and intensity are analyzed, and the ISP parameters related to noise are integrated into the neural network architecture, enabling the neural network to perform adaptive noise reduction in the full ISO segment, avoiding blurring details due to too strong noise reduction or excessive video noise due to too weak noise reduction. In order to protect the highlight details in the video image, a local highlight segmentation algorithm is used to obtain the mask of the highlight area, avoiding blurring the highlight details while reducing noise and improving the quality of the video image.

[0171] An embodiment of the present application provides an image processing method. This method is applied in an ISP. After an image is acquired, the acquired image and the parameters related to removing target defects of the acquired image are input into a trained neural network model to remove the target defects, and an output image is obtained and displayed. In this way, in the ISP, a trained neural network model obtained by training a preset neural network model using a training image, the parameters related to removing target defects of the training image, and the training image after removing target defects is used to process the acquired image to remove target defects, which can achieve the removal of target defects in the ISP and improve the image quality of the output image.

[0172] Based on the same inventive concept as the foregoing embodiment, an embodiment of the present application provides an image processing device, which is arranged in an ISP. Figure 7 As shown in the structure schematic diagram of an optional image processing device provided by an embodiment of the present application, Figure 7 the image processing device includes: an acquisition module 71 and a processing module 72; where,

[0173] The acquisition module 71 is used to acquire an acquired image;

[0174] The processing module 72 is used to input the acquired image and the parameters related to removing target defects of the acquired image into a trained neural network model to obtain an output image, so as to remove the target defects in the acquired image and display the output image; where, the trained neural network model is obtained by training a preset neural network model based on a training image, the parameters related to removing target defects of the training image, and the training image after removing target defects.

[0175] In an optional embodiment, the parameters related to removing target defects include one or more of the following: parameters related to denoising, parameters related to deblurring, parameters related to removing shadows, and parameters related to removing moiré patterns.

[0176] In an optional embodiment, the parameters related to removing target defects are parameters related to denoising. The processing module 72 is specifically used to: input the acquired image, the parameters related to denoising of the acquired image, and the mask image of the acquired image into a trained neural network model to obtain an output image, so as to remove the noise in the acquired image and display the output image; where, the trained neural network model is obtained by training a preset neural network model based on a training image, the parameters related to denoising of the training image, the mask image of the training image, and the training image after denoising; the mask image of the acquired image is generated by extracting image blocks with brightness values greater than a preset brightness threshold in the acquired image; the mask image of the training image is generated by extracting image blocks with brightness values greater than a preset brightness threshold in the training image.

[0177] In an alternative embodiment, the denoising-related parameters include one or more of the following: noise type parameter, ISO parameter, DRC gain parameter, and ZOOM parameter.

[0178] In an alternative embodiment, the apparatus is further configured to: obtain the locking mode of the sensor when the acquisition image is acquired; determine the noise type parameter of the acquisition image according to the locking mode; wherein the locking mode includes binning mode and remosaic mode.

[0179] In an alternative embodiment, the apparatus is further configured to: obtain the analog gain of the acquisition image and the digital gain of the acquisition image; normalize the product of the analog gain of the acquisition image and the digital gain of the acquisition image to obtain the ISO parameter of the acquisition image.

[0180] In an alternative embodiment, the apparatus is further configured to: obtain the DRC gain of the acquisition image; normalize the DRC gain of the acquisition image to obtain the DRC gain parameter of the acquisition image.

[0181] In an alternative embodiment, the apparatus is further configured to: obtain the ZOOM magnification of the acquisition image and the determined ZOOM magnification; determine the ratio of the ZOOM magnification of the acquisition image to the determined ZOOM magnification as the ZOOM parameter; wherein the determined ZOOM magnification is the value when the ZOOM magnification of the acquisition image remains unchanged with the change of the sampling magnification.

[0182] In an alternative embodiment, the training image is obtained by performing ISP simulation on the first sample image set and the second noise parameter set; wherein the first sample image set is acquired under a preset first noise parameter set; the second noise parameter set is generated according to the first noise parameter set.

[0183] In an alternative embodiment, the denoised training image is obtained by performing ISP simulation on the second sample image set and the second noise parameter set; wherein the second sample image set is obtained by denoising the first sample image set under the first noise parameter set.

[0184] In an alternative embodiment, the first noise parameter set includes one or more of the following: digital gain, DRC gain; the second noise parameter set includes one or more of the following: analog gain, local tone mapping function, ZOOM magnification.

[0185] In an alternative embodiment, the second sample image set is obtained by denoising the first sample image set under the first noise parameter set using the frame stacking technique.

[0186] In an alternative embodiment, the first feature map in the preset neural network model is replaced with: a mathematical operation or concatenation of one or more parameters in the relevant parameters for removing target defects and the first feature map.

[0187] In an alternative embodiment, the first feature map in the preset neural network model is replaced with: a mathematical operation or concatenation of the parameter map corresponding to the relevant parameters for removing target defects and the first feature map; wherein, the parameter map corresponding to the relevant parameters for removing target defects is: the parameter map corresponding to any one of the relevant parameters for removing target defects.

[0188] In an alternative embodiment, the first feature map in the preset neural network model is replaced with: a mathematical operation or concatenation of the parameter map after feature extraction and the first feature map.

[0189] In practical applications, the above-mentioned obtaining module 71 and processing module 72 can be implemented by a processor located on an image processing device, specifically implemented by a CPU, a microprocessor (Microprocessor Unit, MPU), a digital signal processor (Digital Signal Processing, DSP), or a field programmable gate array (Field Programmable Gate Array, FPGA), etc.

[0190] An embodiment of the present application provides a chip. Figure 8 It is a schematic structural diagram of an optional ISP provided by an embodiment of the present application. As Figure 8 shown, an embodiment of the present application provides an ISP800. The ISP800 includes:

[0191] A processor 81, configured to call and run a computer program from a memory, so that a device installed with the ISP800 executes the method described in the above one or more embodiments.

[0192] A transceiver 82, configured to receive and send information during the process of receiving and sending information between the device and the ISP800.

[0193] Figure 9 It is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application. As Figure 9 shown, an embodiment of the present application provides an electronic device 900, including:

[0194] An ISP91, a processor 9 / 2, and a storage medium 93 storing executable instructions of the processor; the storage medium 93 depends on the processor 92 to execute operations through a communication bus 94.

[0195] It should be noted that in practical applications, each component in the computer device is coupled together through the communication bus 94. It can be understood that the communication bus 94 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 94 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 9 all kinds of buses are labeled as the communication bus 94.

[0196] The embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the image processing method described in the above one or more embodiments.

[0197] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0198] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.

[0199] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0200] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0202] As mentioned above, it is only a preferred embodiment of the present application and is not used to limit the protection scope of the present application.

Claims

1. A method for processing an image, characterized in that: The method is applied in ISP, and includes: Acquire the collected image; Inputting the acquired image and parameters of the acquired image related to removing target defects into a trained neural network model to obtain an output image to remove target defects in the acquired image, and displaying the output image; The trained neural network model is obtained by training a preset neural network model based on a training image, parameters of the training image related to target defect removal, and the training image after target defect removal.

2. The method according to claim 1, characterized in that The parameters related to removing the target defect include one or more of the following: Parameters related to denoising, deblurring, shadow removal, and moiré removal.

3. The method according to claim 2, characterized in that The parameters related to removing target defects are the parameters related to noise removal, the acquired image and the parameters related to removing target defects of the acquired image are input into a trained neural network model to obtain an output image to remove target defects in the acquired image, and the output image is displayed, including: Inputting the collected image, the denoising-related parameters of the collected image and the mask image of the collected image into a trained neural network model to obtain an output image to remove the noise in the collected image, and displaying the output image; Among them, the trained neural network model is obtained by training a preset neural network model based on the training image, the denoising-related parameters of the training image, the mask image of the training image, and the denoised training image; the mask image of the acquired image is generated by extracting and generating image blocks whose brightness values ​​in the acquired image are greater than a preset brightness threshold; the mask image of the training image is generated by extracting and generating image blocks whose brightness values ​​in the training image are greater than a preset brightness threshold.

4. The method according to claim 3, characterized in that The denoising-related parameters include one or more of the following: Noise type parameter, ISO parameter, DRC gain parameter and ZOOM parameter.

5. The method according to claim 4, characterized in that The method further comprises: Obtain the locking mode of the sensor when acquiring the acquired image; Determining a noise type parameter of the acquired image according to the locking mode; The locking modes include binning mode and remosaic mode.

6. The method according to claim 4, characterized in that The method further comprises: Acquire an analog gain of the acquired image and a digital gain of the acquired image; The product of the analog gain of the acquired image and the digital gain of the acquired image is normalized to obtain the ISO parameter of the acquired image.

7. The method according to claim 4, characterized in that The method further comprises: Obtaining the DRC gain of the acquired image; The DRC gain of the acquired image is normalized to obtain a DRC gain parameter of the acquired image.

8. The method according to claim 4, characterized in that The method further comprises: Obtaining the ZOOM magnification of the acquired image and determining the ZOOM magnification; Determine the ratio of the ZOOM magnification of the acquired image to the determined ZOOM magnification as the ZOOM parameter; The determined ZOOM magnification is: a value when the ZOOM magnification of the acquired image remains unchanged as the sampling magnification changes.

9. The method according to any one of claims 3 to 8, characterized in that: The training images are obtained by performing ISP simulation on the first sample image set and the second noise parameter set; The first sample image set is acquired under a preset first noise parameter set; and the second noise parameter set is generated based on the first noise parameter set.

10. The method according to claim 9, characterized in that The denoised training image is obtained by performing ISP simulation on the second sample image set and the second noise parameter set; The second sample image set is obtained by denoising the first sample image set under the first noise parameter set.

11. The method according to claim 9, characterized in that The first noise parameter set includes one or more of the following: digital gain, DRC gain; The second noise parameter set includes one or more of the following: analog gain, local tone mapping function, and ZOOM ratio.

12. The method according to claim 10, characterized in that The second sample image set is obtained by denoising the first sample image set under the first noise parameter set by using the frame stacking technology.

13. The method according to claim 1, characterized in that The first feature map in the preset neural network model is replaced by: a mathematical operation or splicing of one or more parameters of the relevant parameters for removing the target defect and the first feature map.

14. The method according to claim 13, characterized in that The first feature map in the preset neural network model is replaced by: a mathematical operation or splicing of a parameter map corresponding to the relevant parameters for removing the target defect and the first feature map; The parameter graph corresponding to the relevant parameters for removing target defects is: a parameter graph corresponding to any one of the relevant parameters for removing target defects.

15. The method according to claim 14, characterized in that The first feature map in the preset neural network model is replaced by: mathematical operation or splicing of the parameter map after feature extraction and the first feature map.

16. An image processing device, characterized in that: The device is arranged in the ISP and includes: An acquisition module, used for acquiring a captured image; A processing module, used for inputting the acquired image and parameters of the acquired image related to removing target defects into a trained neural network model to obtain an output image to remove the target defects in the acquired image, and displaying the output image; The trained neural network model is obtained by training a preset neural network model based on a training image, parameters of the training image related to target defect removal, and the training image after target defect removal.

17. An ISP, characterized in that: include: A processor, configured to call and run a computer program from a memory so that a device installed with the ISP executes the method according to any one of claims 1 to 15; A transceiver is used to send and receive information between a device or an ISP.

18. An electronic device, characterized in that: include: The ISP, the processor, and the storage medium storing instructions executable by the processor as claimed in claim 17; The storage medium executes operations dependent on the processor via a communication bus.

19. A computer storage medium, characterized in that: Executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute the image processing method described in any one of claims 1 to 15.

Images