Image noise reduction method, device, camera and camera system

Abstract

Embodiments of the present application provide an image denoising method and device, a camera and a camera system. The method comprises: mapping a to-be-denoised image to a frequency domain to obtain a plurality of to-be-denoised sub-band images, wherein different to-be-denoised sub-band images are used to represent image information of different frequencies in the to-be-denoised image; merging the plurality of to-be-denoised sub-band images in a channel dimension to obtain a first multi-channel image; inputting the first multi-channel image into a target shift convolutional neural network for denoising processing to obtain a second multi-channel image output by the target shift convolutional neural network; and mapping the second multi-channel image to a spatial domain to obtain a first denoised image. Image denoising can be achieved with less consumption of computing resources.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image noise reduction method, device, camera, and camera system. Background Art

[0002] Due to various conditions, such as the shooting environment and the hardware and software conditions of the image acquisition device itself, there may be certain noise in the images captured by the image acquisition device, and the noise will lead to a decrease in image quality.

[0003] Therefore, how to reduce or even remove noise in images has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image noise reduction method, device, camera, and camera system to achieve image noise reduction while consuming less computing resources. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a method for image noise reduction is provided, the method comprising:

[0006] Mapping the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised;

[0007] Merging the multiple sub-band images to be denoised in the channel dimension to obtain a first multi-channel image;

[0008] Inputting the first multi-channel image into a target shift convolutional neural network for noise reduction processing to obtain a second multi-channel image output by the target shift convolutional neural network;

[0009] The second multi-channel image is mapped to a spatial domain to obtain a first denoised image.

[0010] In a possible embodiment, the target shift convolutional neural network is pre-trained with multiple groups of sample data, and each group of sample data includes: a true value image and a noise image obtained by superimposing noise on the true value image.

[0011] In a possible embodiment, the target shift convolutional neural network is pre-trained in the following manner:

[0012] Mapping the noise image to the frequency domain to obtain a plurality of sample sub-band images, wherein different sample sub-band images are used to represent image information of different frequencies in the noise image;

[0013] Merging the multiple sample sub-band images in a channel dimension to obtain a third multi-channel image;

[0014] Inputting the third multi-channel image into an initial shift convolutional neural network to obtain a fourth multi-channel image output by the initial shift convolutional neural network;

[0015] Mapping the fourth multi-channel image to a spatial domain to obtain a second denoised image;

[0016] According to the difference between the second denoised image and the true value image, the network parameters of the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network.

[0017] In a possible embodiment, adjusting the network parameters of the initial shifted convolutional neural network according to the difference between the second denoised image and the true image to obtain a target shifted convolutional neural network includes:

[0018] According to the difference between the second denoised image and the true value image, the network parameters other than the shift parameters in the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network, and the shift parameters are used to control the shift direction of the initial shifted convolutional neural network when performing shifted convolution processing.

[0019] In a possible embodiment, the multiple groups of sample data include simulated sample data and / or real sample data;

[0020] The true value image and the noise image in the simulation sample data are obtained in advance by the following method:

[0021] Acquire a noise-free image as a true image; generate simulated noise using a preset noise simulation model; and superimpose the simulated noise on the true image to obtain a noise image;

[0022] The true value image and the noise image in the real sample data are obtained in advance by the following method:

[0023] A sample scene is photographed in a preset first shooting environment to obtain a true image, and the sample scene is photographed in a preset second shooting environment to obtain a noise image; wherein, the intensity of the noise signal in the preset first shooting environment is lower than a preset lower limit threshold, and the intensity of the noise signal in the preset second shooting environment is higher than a preset upper limit threshold.

[0024] In a possible embodiment, mapping the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised includes:

[0025] Performing wavelet transform on the denoised image to obtain multiple denoised sub-band images;

[0026] Mapping the second multi-channel image to a spatial domain to obtain a first denoised image includes:

[0027] Perform inverse wavelet transform on the second multi-channel image to obtain a first denoised image.

[0028] In a second aspect of the embodiments of the present application, an image noise reduction device is provided, the device comprising:

[0029] A frequency domain mapping module is used to map the image to be denoised to the frequency domain to obtain multiple sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised;

[0030] a channel domain merging module, configured to merge the plurality of sub-band images to be denoised in a channel dimension to obtain a first multi-channel image;

[0031] a shift convolution module, configured to input the first multi-channel image into a target shift convolutional neural network for noise reduction processing, and obtain a second multi-channel image output by the target shift convolutional neural network;

[0032] The spatial domain mapping module is used to map the second multi-channel image to the spatial domain to obtain a first denoised image.

[0033] In a possible embodiment, the target shift convolutional neural network is pre-trained with multiple groups of sample data, and each group of sample data includes: a true value image and a noise image obtained by superimposing noise on the true value image.

[0034] In a possible embodiment, the apparatus further includes a network training module, configured to pre-train the target shift convolutional neural network in the following manner:

[0035] Mapping the noise image to the frequency domain to obtain a plurality of sample sub-band images, wherein different sample sub-band images are used to represent image information of different frequencies in the noise image;

[0036] Merging the multiple sample sub-band images in a channel dimension to obtain a third multi-channel image;

[0037] Inputting the third multi-channel image into an initial shift convolutional neural network to obtain a fourth multi-channel image output by the initial shift convolutional neural network;

[0038] Mapping the fourth multi-channel image to a spatial domain to obtain a second denoised image;

[0039] According to the difference between the second denoised image and the true value image, the network parameters of the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network.

[0040] In a possible embodiment, the network training module adjusts the network parameters of the initial shifted convolutional neural network according to the difference between the second denoised image and the true image to obtain a target shifted convolutional neural network, including:

[0041] According to the difference between the second denoised image and the true value image, the network parameters other than the shift parameters in the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network, and the shift parameters are used to control the shift direction of the initial shifted convolutional neural network when performing shifted convolution processing.

[0042] In a possible embodiment, the multiple groups of sample data include simulated sample data and / or real sample data;

[0043] The true value image and the noise image in the simulation sample data are obtained in advance by the following method:

[0044] Acquire a noise-free image as a true image; generate simulated noise using a preset noise simulation model; and superimpose the simulated noise on the true image to obtain a noise image;

[0045] The true value image and the noise image in the real sample data are obtained in advance by the following method:

[0046] A sample scene is photographed in a preset first shooting environment to obtain a true image, and the sample scene is photographed in a preset second shooting environment to obtain a noise image; wherein, the intensity of the noise signal in the preset first shooting environment is lower than a preset lower limit threshold, and the intensity of the noise signal in the preset second shooting environment is higher than a preset upper limit threshold.

[0047] In a possible embodiment, the frequency domain mapping module maps the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, including:

[0048] Performing wavelet transform on the denoised image to obtain multiple denoised sub-band images;

[0049] The spatial domain mapping module maps the second multi-channel image to the spatial domain to obtain a first denoised image, including:

[0050] Perform inverse wavelet transform on the second multi-channel image to obtain a first denoised image.

[0051] In a third aspect of the embodiments of the present application, a camera is provided, including:

[0052] An image acquisition unit, used for capturing an image as an image to be denoised;

[0053] A denoising unit is configured to perform denoising on the image to be denoised according to any of the method steps described in the first aspect.

[0054] In a fourth aspect of the embodiments of the present application, a camera system is provided, including: a server, a camera;

[0055] The server is configured to train a target shift convolutional neural network; and send the target shift convolutional neural network to the camera;

[0056] The camera is used to capture an image as the image to be denoised; and according to the target shift convolutional neural network, the image to be denoised is denoised according to any of the method steps described in the first aspect above.

[0057] Beneficial effects of the embodiments of the present application:

[0058] The image denoising method, device, camera and camera system provided in the embodiments of the present application can convert the image to be denoised into multiple sub-band images to be denoised by mapping the image to be denoised to the frequency domain, and merge the multiple sub-band images to be denoised into a first multi-channel image by merging them in the channel dimension. The first multi-channel image is denoised using a target shift convolutional neural network. Since the target shift convolutional neural network can use shifted convolution instead of traditional convolution when processing the first multi-channel image, that is, it can shift the sub-images of each channel and then perform convolution on the sub-images of each channel in the channel domain to implement convolution operations, the receptive field of convolution when performing convolution in the channel domain can be expanded by shifting, so that a small-sized convolution kernel can be used to implement convolution operations. The amount of computation required for convolution is positively correlated with the size of the convolution kernel, and the shifting process does not involve computation. Therefore, compared with traditional convolution, the amount of computation required for shifted convolution is lower, thereby effectively reducing the amount of computation required in the image denoising process, that is, image denoising can be achieved while consuming less computing resources.

[0059] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0061] Figure 1 A schematic diagram of a process flow of an image noise reduction method provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of the principle of the shifted convolution operator provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of a target shift convolutional neural network training method provided by an embodiment of the present application is shown in FIG. 2.

[0064] Figure 4 Another flowchart of an image denoising method provided by an embodiment of the present application is shown in FIG. 3.

[0065] Figure 5a A structure diagram of a target shift convolutional neural network provided by an embodiment of the present application is shown in FIG. 4.

[0066] Figure 5b A structure diagram of a shift residual network module provided by an embodiment of the present application is shown in FIG. 5.

[0067] Figure 6 A structure diagram of an image denoising device provided by an embodiment of the present application is shown in FIG. 6.

[0068] Figure 7 A structure diagram of a camera provided by an embodiment of the present application is shown in FIG. 7.

[0069] Figure 8 A structure diagram of a camera system provided by an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application are within the scope of protection of the present application.

[0071] To more clearly describe the image denoising method provided by an embodiment of the present application, a possible application scenario of the image denoising method provided by an embodiment of the present application will be described exemplarily below. It can be understood that the following example is only a possible application scenario of the image denoising method provided by an embodiment of the present application, and the image denoising method provided by an embodiment of the present application can also be applied to other possible application scenarios in other possible embodiments, and the following example does not limit this.

[0072] A user needs to capture an image of a current scene through an image capturing device, such as a mobile phone, a camera, etc., due to actual needs. However, due to the noise signals existing in the environment and / or the hardware and software of the image capturing device, there is certain noise in the captured image, which causes the captured image to fail to meet the actual needs of the user.

[0073] Because noise in an image is often high-frequency information, a wavelet transform can be performed on the captured image to obtain multiple sub-band images of the image. Different sub-band images represent image information at different frequencies. Each sub-band image is input into a pre-trained convolutional neural network designed to reduce image noise, and the convolutional neural network outputs a denoised sub-band image. All denoised sub-band images are fused to produce a fused image, which represents the denoised image of the captured image.

[0074] However, convolutional neural networks often include multiple convolutional layers. The convolutional layer will perform convolution operations on the image input to the convolutional layer according to the preset convolution kernel. Each convolution operation will traverse each pixel in the image to perform weighted addition of the neighborhood of each pixel with the convolution kernel to obtain the output of the convolutional layer. Multiple multiplication calculations are required in the weighted addition process.

[0075] As can be seen, the process of using convolutional neural networks for noise reduction requires a large number of multiplication calculations. However, due to limited computing power, the equipment used for image noise reduction takes a long time to complete the noise reduction, or even fails to complete the noise reduction, making it difficult to implement convolutional neural networks.

[0076] For example, to improve the quality of images captured by users through cameras, camera manufacturers hope to deploy a convolutional neural network in their cameras to reduce noise in the captured images. However, as previously analyzed, running this convolutional neural network requires a large amount of computing resources, and cameras often have limited computing resources. Therefore, camera manufacturers find it difficult to deploy convolutional neural networks in their cameras to reduce noise in the images.

[0077] Based on this, the present application provides an image noise reduction method, which can be applied to any electronic device with image noise reduction capability. Figure 1 Shown, including:

[0078] S101 : Mapping the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised.

[0079] S102: Merge multiple sub-band images to be denoised in a channel dimension to obtain a first multi-channel image.

[0080] S103: Input the first multi-channel image into a target shift convolutional neural network for noise reduction processing to obtain a second multi-channel image output by the target shift convolutional neural network.

[0081] S104: Map the second multi-channel image to the spatial domain to obtain a first denoised image.

[0082] By selecting this embodiment, the image to be denoised can be converted into multiple sub-band images to be denoised by mapping the image to be denoised to the frequency domain, and the multiple sub-band images to be denoised can be merged into a first multi-channel image by merging them in the channel dimension. The first multi-channel image is denoised using a target shift convolutional neural network. Therefore, the target shift convolutional neural network can use shifted convolution instead of traditional convolution in the process of processing the first multi-channel image, that is, the sub-images of each channel can be shifted, and then the sub-images of each channel can be convolved in the channel domain to implement the convolution operation. Therefore, the receptive field of the convolution when performing convolution in the channel domain can be expanded by the shifting process, so that a small-sized convolution kernel can be used to implement the convolution operation. The amount of computation required for convolution is positively correlated with the size of the convolution kernel, and the shifting process does not involve computation. Therefore, compared with traditional convolution, the amount of computation required for shifted convolution is lower, thereby effectively reducing the amount of computation required in the image denoising process, that is, image denoising can be achieved while consuming less computing resources.

[0083] On the other hand, since this embodiment is selected, image denoising can be achieved while consuming fewer computing resources. Therefore, the image denoising method provided in the embodiment of the present application can be deployed on devices with lower computing power, which is easier to implement than the solution of image denoising through traditional convolutional neural networks.

[0084] The following describes each of the steps S101-S104:

[0085] In S101, the image to be denoised can be an image captured by the subject executing the image denoising method provided herein, an image received by the subject from another electronic device, or an image read by the subject from a preset storage medium. Furthermore, the image to be denoised can be a complete image or a sub-image within a certain area of ​​the complete image.

[0086] For example, taking the execution subject as a camera, assuming that the camera captures an image with noise, the image can be used as the image to be denoised, or a sub-image within a specific area of ​​the image can be used as the image to be denoised. The specific area can be a preset area or an area determined according to the operation instructions input by the user. This application does not impose any restrictions on this.

[0087] The number of sub-images to be denoised may vary depending on the application scenario. For example, in one possible embodiment, four sub-band images to be denoised may be obtained, which, for ease of description, are respectively referred to as a first sub-band image to be denoised, a second sub-band image to be denoised, a third sub-band image to be denoised, and a fourth sub-band image to be denoised.

[0088] Among them, the first sub-band image to be denoised is used to represent low-frequency image information in the image to be denoised, the second sub-band image to be denoised is used to represent high-frequency image information in the horizontal direction of the image to be denoised, the third sub-band image to be denoised is used to represent high-frequency image information in the vertical direction of the image to be denoised, and the fourth sub-band image to be denoised is used to represent high-frequency image information in the diagonal direction of the image to be denoised.

[0089] In other possible embodiments, other numbers of sub-band images to be denoised other than four may be obtained. For the convenience of description below, only four sub-band images to be denoised are used as an example for explanation. The same logic can be applied to other numbers of sub-band images to be denoised, and no further details will be given here.

[0090] How to map the denoised image to the frequency domain will be described in detail below and will not be repeated here.

[0091] In S102, merging the multiple denoised subband images along the channel dimension means merging the multiple denoised subband images, each serving as a subimage of a different channel, into a single multi-channel image. Therefore, the number of channels in the first multi-channel image is equal to the number of the multiple subband images to be denoised. Furthermore, the subimage of each channel in the first multi-channel image is a subband image to be denoised, and subimages in different channels are different subband images to be denoised.

[0092] For example, assuming that the multiple sub-band images to be denoised are a first sub-band image to be denoised, a second sub-band image to be denoised, a third sub-band image to be denoised, and a fourth sub-band image to be denoised, the first multi-channel image has four channels, which, for ease of description, are respectively denoted as the first channel, the second channel, the third channel, and the fourth channel. In this example, the sub-image of the first channel of the first multi-channel image is the first sub-band image to be denoised, the sub-image of the second channel is the second sub-band image to be denoised, the sub-image of the third channel is the third sub-band image to be denoised, and the sub-image of the fourth channel is the fourth sub-band image to be denoised.

[0093] In S103, the target shifted convolutional neural network is a convolutional neural network including at least one shifted convolution operator. The shifted convolution operator in the target shifted convolutional neural network is used to implement a convolution operation through a shift process. The principle of the shifted convolution operator will be described below and will not be repeated here.

[0094] The target shift convolutional neural network is used to perform noise reduction processing on the input multi-channel image. The target shift convolutional neural network can be designed based on actual experience and / or needs, or it can be trained based on deep learning. Exemplarily, the target shift convolutional neural network is pre-trained with multiple sets of sample data, and each set of sample data includes a true value image and a noisy image obtained by superimposing noise on the true value image. Since the noise image is obtained by superimposing noise on the true value image, the true value image can be regarded as the image obtained after denoising the noise image. Therefore, after training with sample data, the target shift convolutional neural network can learn how to remove noise from the image, that is, the second multi-channel image output by the target shift convolutional neural network can be regarded as the multi-channel image obtained after denoising the first multi-channel image.

[0095] In S104, as analyzed above, the second multi-channel image can be regarded as a multi-channel image obtained after denoising the first multi-channel image, and the sub-image of each channel in the first multi-channel image is used to represent the image information of each frequency in the image to be denoised. Therefore, the sub-image of each channel in the second multi-channel image is used to represent the image information obtained after denoising the image information of each frequency in the image to be denoised. In other words, it can be considered that the sub-image of each channel in the second multi-channel image is used to represent the image information of each frequency in the image obtained after denoising the image to be denoised. Therefore, the first denoised image obtained by mapping the second multi-channel image from the frequency domain to the spatial domain can be regarded as the image obtained after denoising the image to be denoised.

[0096] The principle of the shift convolution operator is explained below. Figure 2 , Figure 2 The figure shows a schematic diagram of the principle of the shift convolution operator provided by this application. The processing of the input multi-channel image by the shift convolution operator can be divided into three steps: grouping (Group), shift (Shift) and convolution (Conv)

[0097] The input of the shifted convolution operator is the output result of the sub-network located upstream of the shifted convolution operator in the shifted convolutional neural network. For example, if a shifted convolution operator is located after the Relu (RectifiedLinear, linear rectifier function) layer in the shifted convolutional neural network, the input of the shifted convolution operator is the feature image output by the Relu layer. For another example, if a shifted convolution operator is located after another shifted convolution operator in the shifted convolutional neural network, the input of the shifted convolution operator is the feature image output by the other shifted convolution operator.

[0098] The grouping step involves dividing multiple channels in the input image into multiple shift groups, where the number of channels included in each shift group can be the same or different. The shifting step involves shifting the sub-images of each channel in the input image, where the sub-images of channels belonging to the same shift group are shifted in the same manner.

[0099] Shifting a sub-image means shifting all the pixels in the sub-image by the same number of pixels in the same direction. For example, all the pixels in a sub-image are shifted by one pixel in the positive direction of the pixel row. It can be understood that shifting a sub-image can be regarded as convolving the sub-image with a one-hot convolution kernel, in which only one point has a value of 1 and the values ​​of the remaining points are 0. Therefore, shifting the sub-images of each channel in the input image can be regarded as convolving the input image in the spatial domain of the feature space. However, the shifting process is only a shifting of the pixels in the sub-image, so in theory, no calculation is required to realize the shifting process, that is, the shifted convolution operator realizes the convolution of the input image in the spatial domain of the feature space through the shifting process without the need for calculation.

[0100] The convolution step involves convolving the shifted image in the channel domain using a convolution kernel of size 1x1. It is understandable that since the convolution is performed on the shifted image in the channel domain, when convolving a pixel, there is no need to use other pixels in the neighborhood of that pixel. Instead, only pixels in the same location as that pixel in each channel need to be used. Therefore, a convolution kernel of size 1x1 can be used to convolve the shifted image in the channel domain.

[0101] Since the convolution of the input image is implemented in the spatial domain of the feature space in the shift step, and the convolution of the shifted image is implemented in the channel domain in the convolution step, the image output by the convolution step can be regarded as the image obtained by convolving the input image in the spatial domain and the channel domain of the feature space respectively, that is, the image output by the convolution step can be regarded as the image obtained by convolution of the input image.

[0102] It can be seen that the shifted convolution operator can realize the convolution processing of the input image by shifting, and compared with the traditional convolution layer, the shifted convolution operator does not need to consume computational resources for convolving the input image in the spatial domain of the feature space, so the computational resources consumed are less than those of the traditional convolution layer.

[0103] The following will explain how to train the target shift convolutional neural network. Figure 3 , Figure 3The figure shows a flow chart of the target shift convolutional neural network training method provided by the present application. The target shift convolutional neural network training method provided by the present application can be applied to any electronic device with the ability to train a shift convolutional neural network, and the electronic device can be the execution subject of the image denoising method provided by the present application, or it can be other electronic devices other than the execution subject of the image denoising method provided by the present application. The target shift convolutional neural network training method provided by the present application may include:

[0104] S301 : Map the noise image to the frequency domain to obtain a plurality of sample sub-band images.

[0105] Different sample subband images are used to represent image information of different frequencies in the noisy image. The method for mapping the noisy image to the frequency domain should be the same as the method for mapping the image to be denoised to the frequency domain, and the number of sample subband images should be the same as the number of subband images to be processed.

[0106] For the sub-band image, please refer to the related description of S101 above, which will not be repeated here.

[0107] S302: Merge the multiple sample sub-band images in the channel dimension to obtain a third multi-channel image.

[0108] Regarding how to merge multiple sub-band images into a multi-channel image, please refer to the related description of S102 above, which will not be repeated here.

[0109] S303: Input the third multi-channel image into the initial shift convolutional neural network to obtain a fourth multi-channel image output by the initial shift convolutional neural network.

[0110] The initial shift convolutional neural network contains at least one shift convolution operator. For more information about the shift convolution operator, see Figure 2 And the aforementioned Figure 2 The relevant description will not be repeated here.

[0111] The network parameters of the initial shift convolutional neural network can be obtained through pre-training or set by the user according to actual needs and / or experience. This application does not impose any restrictions on this.

[0112] S304: Map the fourth multi-channel image to the spatial domain to obtain a second denoised image.

[0113] The fourth multi-channel image should be mapped to the spatial domain in the same manner as the second multi-channel image was mapped to the spatial domain.

[0114] S305: Adjust the network parameters of the initial shifted convolutional neural network according to the difference between the second denoised image and the true image to obtain a target shifted convolutional neural network.

[0115] A loss function can be constructed based on the difference between the second denoised image and the true image, and the network parameters of the initial shifted convolutional neural network are adjusted in the direction of the gradient descent of the loss function until the preset convergence condition is reached to obtain the target shifted convolutional neural network.

[0116] Among them, the way of constructing the loss function can be different depending on the application scenario. For example, the loss function can be constructed based on the difference between the second denoised image and the true value image, or based on the Euclidean distance between the second denoised image and the true value image.

[0117] The preset convergence condition may also vary depending on the application scenario. For example, the preset convergence condition may be that the convergence of the network parameters reaches a preset convergence threshold, or that the number of times the network parameters are adjusted reaches a preset number, such as 4000 times.

[0118] As previously described with respect to the shifted convolution operator, the parameters of the shifted convolution operator that need to be determined through training may include: grouping parameters, shift parameters, and convolution parameters.

[0119] Among them, the grouping parameter is used to control the way the shift convolution operator groups the channels in the grouping step, the shift parameter is used to control the shift direction of the sub-image of each channel in the shift step, and the convolution parameter is the value of each point on the 1X1 convolution kernel used in the convolution step.

[0120] Therefore, when adjusting the network parameters of the initial shifted convolutional neural network, the network parameters adjusted may include one or more of the aforementioned grouping parameters, shift parameters, and convolution parameters. In one possible embodiment, in order to make the trained target shifted convolutional neural network more accurate, the aforementioned grouping parameters, shift parameters, and convolution parameters may be adjusted when adjusting the network parameters of the initial shifted convolutional neural network. In another possible embodiment, in order to improve the training efficiency and thus accelerate the implementation of the target shifted convolutional neural network, the aforementioned shift parameters may not be adjusted when adjusting the network parameters of the initial shifted convolutional neural network, that is, in this embodiment, each shifted convolution operator in the trained target shifted convolutional neural network shifts the sub-images of each channel according to a pre-set shift direction, without learning how to shift the sub-images of each channel during the training process.

[0121] The following describes how to obtain sample data:

[0122] In one possible embodiment, the sample data may include only simulated sample data, where the true image in the simulated sample data is a noise-free image obtained, such as a noise-free image downloaded from the Internet. The noisy image in the simulated sample data is obtained by generating simulated noise using a preset noise simulation model, such as a Gaussian-Poisson mixture model, and superimposing the simulated noise on the true image.

[0123] In another possible embodiment, the sample data may include only real sample data, where the true image in the real sample data may be an image obtained by photographing the sample scene in a preset first shooting environment, and the noise image in the real sample data may be an image obtained by photographing the sample scene in a preset second shooting environment. The intensity of the noise signal in the preset first shooting environment is lower than a preset lower threshold, and the intensity of the noise signal in the preset second shooting environment is higher than a preset upper threshold, and the upper threshold is not less than the lower threshold.

[0124] Since the noise image and the true value image in the real sample data are obtained by shooting the same scene, and the noise signal of the first preset shooting environment when shooting the true value image is relatively weak, while the noise signal of the second preset shooting environment when shooting the true value image is relatively strong, it can be considered that the noise image in the real sample data is obtained by superimposing noise on the true value image.

[0125] Depending on the application scenario, the shooting environment can be modified in different ways. For example, this can be done by controlling the interference in the shooting environment, such as setting a first shooting environment without interference and a second shooting environment with interference. Alternatively, the exposure time can be controlled, such as by increasing the exposure time to increase the interference during shooting, thereby increasing the intensity of the noise signal in the shooting environment.

[0126] In another possible embodiment, the sample data may include simulated sample data and real sample data. It is understandable that, on the one hand, the real sample data needs to be obtained through photography, and due to various conditions, the real sample data that can be obtained by photography is often limited. Therefore, including both simulated sample data and real sample data in the sample data can effectively increase the amount of sample data, thereby enabling the target shift convolutional neural network to be adequately trained with the sample data.

[0127] On the other hand, due to the limitations of the performance of the noise simulation model, the simulated noise generated by the noise simulation model may differ from the noise in the actual noisy image. Therefore, based solely on the simulated sample data, the target shift convolutional neural network may find it difficult to effectively learn how to accurately denoise images with real noise, resulting in low accuracy of the trained target shift convolutional neural network. However, if the sample data includes both simulated sample data and real sample data, the target shift convolutional neural network can learn how to accurately denoise images with real noise from the real sample data, which can effectively improve the accuracy of the trained target shift convolutional neural network.

[0128] The following will describe how to map the image to be denoised to the frequency domain and how to map the second multi-channel image to the spatial domain. For example, see Figure 4 , Figure 4 Another flow chart of the image noise reduction method provided by the present application is shown, which may include:

[0129] S401 , performing wavelet transform processing on the image to be denoised to obtain a plurality of denoised sub-band images.

[0130] In other possible embodiments, the image to be denoised may be mapped to the frequency domain via Fourier transform. The wavelet transform method used in this embodiment may vary depending on the application scenario. For ease of description, the following description uses the Haar wavelet transform as an example. In other possible embodiments, the image to be denoised may be processed using other wavelet transform methods.

[0131] The pixel values ​​of each pixel in the low-frequency sub-band image and the high-frequency sub-band image can be calculated based on the pixel values ​​of each pixel in the image to be denoised and formula (1) and formula (2):

[0132]

[0133]

[0134] Among them, a x,m is the pixel value of the pixel at the xth row and mth column in the low-frequency sub-band image, d x,m is the pixel value of the pixel at the xth row and mth column in the high-frequency sub-band image, f x,2m-1 is the pixel value of the pixel at the xth row and 2m-1th column in the image to be denoised, f x,2m is the pixel value of the pixel at the xth row and 2mth column in the image to be denoised. In formulas (1) and (2), the value range of x is 1 to H, the value range of m is 1 to W / 2, H is the total number of rows in the image to be denoised, and W is the total number of columns in the image to be denoised.

[0135] And according to the pixel value of each pixel point in the low-frequency sub-band image, and formula (3) and formula (4), the pixel value of each pixel point in the low-frequency-low-frequency sub-band image and the low-frequency-high-frequency sub-band image is calculated:

[0136]

[0137]

[0138] Among them, ll x,m is the pixel value of the pixel at the xth row and mth column in the low-frequency-low-frequency subband image, lh x,m is the pixel value of the pixel at the xth row and mth column in the low-frequency-high-frequency subband image, a 2x-1,m is the pixel value of the pixel at the 2x-1th row and the mth column in the low-frequency sub-band image, a 2x,m is the pixel value of the pixel at the 2xth row and the mth column in the low-frequency subband image. In formulas (1) and (2), x ranges from 1 to H / 2, m ranges from 1 to W / 2, H is the total number of rows of the image to be denoised, and W is the total number of columns of the image to be denoised.

[0139] And according to the pixel value of each pixel point in the high-frequency sub-band image, and formula (5) and formula (6), the pixel value of each pixel point in the high-frequency-low-frequency sub-band image and the high-frequency-high-frequency sub-band image is calculated:

[0140]

[0141]

[0142] Among them, lh x,m is the pixel value of the pixel at row x and column m in the low-frequency-high-frequency subband image, hh x,m is the pixel value of the pixel at the xth row and mth column in the high-frequency-high-frequency subband image, d 2x-1,m is the pixel value of the pixel at the 2x-1th row and the mth column in the high-frequency sub-band image, d 2x,m is the pixel value of the pixel at the 2xth row and the mth column in the low-frequency subband image. In formulas (1) and (2), x ranges from 1 to H / 2, m ranges from 1 to W / 2, H is the total number of rows of the image to be denoised, and W is the total number of columns of the image to be denoised.

[0143] The low-frequency-low-frequency subband image, low-frequency-high-frequency subband image, high-frequency-low-frequency subband image and high-frequency-high-frequency subband image obtained by the above calculation are the four subband images to be denoised obtained by performing Haar wavelet transform on the image to be denoised.

[0144] S402 : Merge multiple sub-band images to be denoised in a channel dimension to obtain a first multi-channel image.

[0145] This step is the same as the aforementioned S102. Please refer to the relevant description of the aforementioned S102 and will not be repeated here.

[0146] S403: Input the first multi-channel image into a target shift convolutional neural network for noise reduction processing to obtain a second multi-channel image output by the target shift convolutional neural network.

[0147] This step is the same as the aforementioned S103. Please refer to the relevant description of the aforementioned S103 and will not be repeated here.

[0148] S404: Perform inverse wavelet transform on the second multi-channel image to obtain a first denoised image.

[0149] The inverse wavelet transform method is the inverse of the wavelet transform method, so please refer to the aforementioned description of the wavelet transform process, which will not be repeated here.

[0150] See also Figure 5a , Figure 5a The figure shows a structural diagram of the target shift convolutional neural network provided by the present application, wherein GSM represents the shift convolution operator, SRB represents the shift residual module, and the structure of the shift residual module can be as follows: Figure 5b As shown, the network consists of multiple shifted convolution operators, a HIN layer (a type of layer in a convolutional neural network), a ReLU layer, and a Shuffle layer (a type of layer in a convolutional neural network). The circled C in the figure represents a merge operation, the circled + sign represents an element-wise addition operation (elementwise add), the right-pointing arrow with a wavy line represents a wavelet transform, and the left-pointing arrow with a wavy line represents an inverse wavelet transform. Conv 3*3 indicates convolution using a 3*3 convolution kernel.

[0151] See also Figure 6 , Figure 6 The figure shows a schematic diagram of the structure of the image noise reduction device provided by the present application, which may include:

[0152] A frequency domain mapping module 601 is configured to map the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised;

[0153] A channel domain merging module 602 is configured to merge the plurality of sub-band images to be denoised in a channel dimension to obtain a first multi-channel image;

[0154] a shifted convolution module 603, configured to input the first multi-channel image into a target shifted convolutional neural network for noise reduction processing, and obtain a second multi-channel image output by the target shifted convolutional neural network;

[0155] The spatial domain mapping module 604 is configured to map the second multi-channel image to a spatial domain to obtain a first denoised image.

[0156] In a possible embodiment, the target shift convolutional neural network is pre-trained with multiple groups of sample data, and each group of sample data includes: a true value image and a noise image obtained by superimposing noise on the true value image.

[0157] In a possible embodiment, the apparatus further includes a network training module, configured to pre-train the target shift convolutional neural network in the following manner:

[0158] Mapping the noise image to the frequency domain to obtain a plurality of sample sub-band images, wherein different sample sub-band images are used to represent image information of different frequencies in the noise image;

[0159] Merging the multiple sample sub-band images in a channel dimension to obtain a third multi-channel image;

[0160] Inputting the third multi-channel image into an initial shift convolutional neural network to obtain a fourth multi-channel image output by the initial shift convolutional neural network;

[0161] Mapping the fourth multi-channel image to a spatial domain to obtain a second denoised image;

[0162] According to the difference between the second denoised image and the true value image, the network parameters of the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network.

[0163] In a possible embodiment, the network training module adjusts the network parameters of the initial shifted convolutional neural network according to the difference between the second denoised image and the true image to obtain a target shifted convolutional neural network, including:

[0164] According to the difference between the second denoised image and the true value image, the network parameters other than the shift parameters in the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network, and the shift parameters are used to control the shift direction of the initial shifted convolutional neural network when performing shifted convolution processing.

[0165] In a possible embodiment, the multiple groups of sample data include simulated sample data and / or real sample data;

[0166] The true value image and the noise image in the simulation sample data are obtained in advance by the following method:

[0167] Acquire a noise-free image as a true image; generate simulated noise using a preset noise simulation model; and superimpose the simulated noise on the true image to obtain a noise image;

[0168] The true value image and the noise image in the real sample data are obtained in advance by the following method:

[0169] A sample scene is photographed in a preset first shooting environment to obtain a true image, and the sample scene is photographed in a preset second shooting environment to obtain a noise image; wherein, the intensity of the noise signal in the preset first shooting environment is lower than a preset lower limit threshold, and the intensity of the noise signal in the preset second shooting environment is higher than a preset upper limit threshold.

[0170] In a possible embodiment, the frequency domain mapping module 601 maps the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, including:

[0171] Performing wavelet transform on the denoised image to obtain multiple denoised sub-band images;

[0172] The spatial domain mapping module 604 maps the second multi-channel image to the spatial domain to obtain a first denoised image, including:

[0173] Perform inverse wavelet transform on the second multi-channel image to obtain a first denoised image.

[0174] The embodiment of the present application also provides a camera, such as Figure 7 Shown, including:

[0175] An image acquisition unit 701 is used to capture an image as an image to be denoised;

[0176] The denoising unit 702 is configured to perform denoising on the image to be denoised according to any of the aforementioned image denoising methods.

[0177] It is understandable that due to the limitations of the shooting environment and / or the camera's own hardware and software conditions, there may be noise in the images captured by the camera, and the computing power of the camera is often low. Therefore, it is difficult or even impossible to deploy a traditional convolutional neural network on the camera to perform noise reduction on the images captured by the camera, resulting in lower quality of the images captured by the camera.

[0178] This embodiment is selected. As analyzed above, since the image denoising method provided by the present application requires a relatively low amount of computation, it can be deployed in a camera, so that the camera can perform denoising on the captured image to improve the quality of the image captured by the camera.

[0179] Furthermore, in addition to being deployed on cameras, the image noise reduction method provided in this application can also be deployed on other devices with lower computing power, including but not limited to mobile phones, tablet computers, and driving recorders, in other possible embodiments.

[0180] The aforementioned camera may also include other components such as a memory and a processor.

[0181] The memory mentioned in the above-mentioned camera may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0182] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0183] In another embodiment provided by the present application, a camera system is also provided, such as Figure 8 Shown, including:

[0184] Camera 801 and server 802.

[0185] The server 802 is configured to train a target shift convolutional neural network and send the target shift convolutional neural network to the camera;

[0186] The camera 801 is used to capture an image as the image to be denoised; and according to the target shift convolutional neural network, the image to be denoised is denoised according to any of the above-mentioned image denoising methods.

[0187] In another embodiment provided by the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of any of the above-mentioned image noise reduction methods are implemented.

[0188] In another embodiment provided by the present application, a computer program product including instructions is further provided, which, when executed on a computer, enables the computer to execute any one of the image noise reduction methods in the above embodiments.

[0189] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0190] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0191] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the embodiments of the apparatus, camera, camera system, computer-readable storage medium, and computer program product are generally similar to the method embodiments, so their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the method embodiments.

[0192] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. An image denoising method, characterized in that: The method comprises: Mapping the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised; Merging the plurality of denoised sub-band images as sub-images of different channels into a multi-channel image to obtain a first multi-channel image; Inputting the first multi-channel image into a target shift convolutional neural network for noise reduction processing to obtain a second multi-channel image output by the target shift convolutional neural network; The second multi-channel image is mapped to a spatial domain to obtain a first denoised image.

2. The method according to claim 1, characterized in that The target shift convolutional neural network is pre-trained with multiple groups of sample data, and each group of sample data includes: a true value image and a noise image obtained by superimposing noise on the true value image.

3. The method according to claim 2, characterized in that The target shift convolutional neural network is pre-trained in the following way: Mapping the noise image to the frequency domain to obtain a plurality of sample sub-band images, wherein different sample sub-band images are used to represent image information of different frequencies in the noise image; Merging the multiple sample sub-band images in a channel dimension to obtain a third multi-channel image; Inputting the third multi-channel image into an initial shift convolutional neural network to obtain a fourth multi-channel image output by the initial shift convolutional neural network; Mapping the fourth multi-channel image to a spatial domain to obtain a second denoised image; According to the difference between the second denoised image and the true value image, the network parameters of the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network.

4. The method according to claim 3, characterized in that The adjusting the network parameters of the initial shifted convolutional neural network according to the difference between the second denoised image and the true value image to obtain the target shifted convolutional neural network includes: According to the difference between the second denoised image and the true value image, the network parameters other than the shift parameters in the initial shifted convolutional neural network are adjusted to obtain a target shifted convolutional neural network, and the shift parameters are used to control the shift direction of the initial shifted convolutional neural network when performing shifted convolution processing.

5. The method according to claim 2, characterized in that The multiple groups of sample data include simulated sample data and / or real sample data; The true value image and the noise image in the simulation sample data are obtained in advance by the following method: Acquire a noise-free image as a true image; generate simulated noise using a preset noise simulation model; and superimpose the simulated noise on the true image to obtain a noise image; The true value image and the noise image in the real sample data are obtained in advance by the following method: A sample scene is photographed in a preset first shooting environment to obtain a true image, and the sample scene is photographed in a preset second shooting environment to obtain a noise image; wherein, the intensity of the noise signal in the preset first shooting environment is lower than a preset lower limit threshold, and the intensity of the noise signal in the preset second shooting environment is higher than a preset upper limit threshold.

6. The method according to claim 1, characterized in that Mapping the image to be denoised to the frequency domain to obtain a plurality of sub-band images to be denoised includes: Performing wavelet transform on the denoised image to obtain multiple denoised sub-band images; Mapping the second multi-channel image to a spatial domain to obtain a first denoised image includes: Perform inverse wavelet transform on the second multi-channel image to obtain a first denoised image.

7. An image noise reduction device, characterized in that: The device comprises: A frequency domain mapping module is used to map the image to be denoised to the frequency domain to obtain multiple sub-band images to be denoised, wherein different sub-band images to be denoised are used to represent image information of different frequencies in the image to be denoised; a channel domain merging module, configured to merge the plurality of denoised sub-band images as sub-images of different channels into a multi-channel image to obtain a first multi-channel image; a shift convolution module, configured to input the first multi-channel image into a target shift convolutional neural network for noise reduction processing, and obtain a second multi-channel image output by the target shift convolutional neural network; The spatial domain mapping module is used to map the second multi-channel image to the spatial domain to obtain a first denoised image.

8. The device according to claim 7, characterized in that The target shift convolutional neural network is pre-trained with multiple groups of sample data, and each group of sample data includes: a true value image and a noise image obtained by superimposing noise on the true value image.

9. A camera, characterized in that: include: An image acquisition unit, used for capturing an image as an image to be denoised; A denoising unit, configured to denoise the image to be denoised according to the method steps of any one of claims 1 to 6.

10. A camera system, characterized in that: include: Servers, cameras; The server is used to train a target shift convolutional neural network; Sending the target shift convolutional neural network to the camera; The camera is used to capture an image as the image to be denoised; and according to the target shift convolutional neural network, the image to be denoised is denoised according to the method steps described in any one of claims 1 to 6.

Images