Image noise reduction method and device, equipment and storage medium

Abstract

The invention discloses an image noise reduction method and device, equipment and a storage medium, and the method comprises the steps: enabling a plurality of cameras to share one image noise reduction model for image noise reduction, achieving the noise reduction of the image data of the plurality of cameras, and not needing to execute complex switching decision logic. Different feature extraction branches are trained for noise characteristics of different cameras in the image noise reduction model to adapt to noise reduction requirements of different cameras, that is, the image noise reduction model comprises at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the noise reduction requirements of different cameras are met through the different feature extraction branches in the image noise reduction model. The image noise reduction effects of different cameras can be ensured. And according to the adaptive fusion weight, fusing the feature maps obtained by the feature extraction branches, and according to the fused feature maps, carrying out image reconstruction so as to improve the final image noise reduction effect.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image noise reduction method, device, equipment and storage medium. Background Art

[0002] As users' demand for shooting with electronic devices continues to increase, most existing electronic devices are equipped with two or more cameras. Common camera types include: ultra-wide-angle lens, wide-angle lens, telephoto lens, macro lens, etc.

[0003] In actual imaging systems, due to the limitations of image sensors and imaging systems, the final generated image is often affected by noise. Generally, in an image signal processor (ISP), a noise reduction module is set, such as the common Bayer domain denoise module, spatial denoise module, and temporal denoise module. These noise reduction modules can effectively deal with the noise in the image, and cooperate with other modules in the ISP to obtain images with better signal-to-noise ratio and details. In particular, the image noise reduction scheme based on deep learning, relying on its powerful learning ability, can obtain better noise reduction effects than traditional noise reduction algorithms. However, the deep learning scheme is highly dependent on the data set. Different cameras have obvious differences in noise intensity due to differences in lenses and sensors, and noise reduction is difficult.

[0004] Most of the existing technical solutions mix the data from all cameras together and send them into a single noise reduction network for training. There is no targeted processing for different cameras, which limits the upper limit of its noise reduction effect. Another common technical solution is to train a separate model for each camera module, and select the corresponding model for processing during inference. This processing is more complex and the switching decision logic is more cumbersome. Summary of the invention

[0005] The embodiments of the present application are intended to provide an image noise reduction method, apparatus, device and storage medium.

[0006] The technical solution of this application is implemented as follows:

[0007] In a first aspect, an image denoising method is provided, wherein the image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the method includes:

[0008] Acquire a first noise image and first camera information from a first camera;

[0009] Determining a fusion weight of the image denoising model according to the first camera information;

[0010] Performing denoising on the first noisy image by using the image denoising model to obtain a first denoised image corresponding to the first noisy image;

[0011] Wherein, the image denoising model is configured as follows:

[0012] Processing the first noise image respectively by at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image;

[0013] Performing weighted fusion on the at least two groups of feature maps according to the fusion weight to obtain a fused feature map;

[0014] The first denoised image is obtained according to the fused feature map.

[0015] In a second aspect, an image denoising device is provided, wherein the image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the device includes:

[0016] An acquisition unit, used to acquire a first noise image and first camera information from a first camera;

[0017] A determination unit, configured to determine a fusion weight of the image denoising model according to the first camera information;

[0018] a processing unit, configured to perform denoising on the first noisy image by using the image denoising model to obtain a first denoised image corresponding to the first noisy image;

[0019] Wherein, the image denoising model is configured as follows:

[0020] Processing the first noise image respectively by at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image;

[0021] Performing weighted fusion on the at least two groups of feature maps according to the fusion weight to obtain a fused feature map;

[0022] The first denoised image is obtained according to the fused feature map.

[0023] According to a third aspect, an electronic device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0024] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0025] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0026] An image denoising method, device, equipment and storage medium are provided in an embodiment of the present application. The method includes: multiple cameras share an image denoising model to perform image denoising processing, thereby achieving denoising processing on image data of multiple cameras without executing complex switching decision logic. In the image denoising model, different feature extraction branches are trained according to the noise characteristics of different cameras to adapt to the denoising requirements of different cameras, that is, the image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the image denoising effect of different cameras can be guaranteed by different feature extraction branches in the image denoising model. The feature maps obtained by the feature extraction branches are then fused according to the adaptive fusion weights, and image reconstruction based on the fused feature maps can improve the final image denoising effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 1 ;

[0028] Figure 2 Schematic diagram of the process of the image noise reduction method in the embodiment of the present application;

[0029] Figure 3 Schematic diagram of the process of implementing the image denoising model in the embodiment of the present application;

[0030] Figure 4 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 2 ;

[0031] Figure 5 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 3 ;

[0032] Figure 6 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 4 ;

[0033] Figure 7 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 5 ;

[0034] Figure 8 The structure of the image denoising model in the embodiment of the present application is shown in FIG. Figure 6 ;

[0035] Fig. 9 This is a schematic diagram of the composition structure of the image noise reduction device in the embodiment of the present application;

[0036] Fig.10 It is a schematic diagram of the composition structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0038] The noise model of image sensors in most current cameras is modeled as the Gaussian-Poisson model, a general signal-dependent noise model:

[0039] z x =y x +σ(y x ) x

[0040] Among them, z x is the observed noise image, y x is a truth term, σ(y x ) x is the noise term, which consists of two independent terms: a Poisson distribution term η that depends on the signal p and a Gaussian distribution term η that is independent of the signal g ,Right now:

[0041] σ(y x ) x =η p (y x )+η g (x)

[0042] According to the definitions of Poisson distribution and Gaussian distribution, we can know that:

[0043] χ(y x +η p (y x ))~P(χy x )

[0044] η g (x)~N(0,b)

[0045] According to the properties of Poisson distribution:

[0046] exp[χ(y x +η p (y x ))]=var[χ(y x +η p (y x ))]=χy x

[0047] Therefore, it can be deduced that:

[0048] exp[η p (y x )]=0,var[η p (y x )]=y x / x

[0049] It can be seen that the Poisson term η p has a varying variance that is similar to y x related

[0050] Let a = 1 / χ, then:

[0051] var[η p (y x )]=ay x

[0052] By finding the variance of the noise model, we can get:

[0053] z x =y x +η p (y x )+η g (x)

[0054] var[z x ]=ay x +b

[0055] Therefore, the noise calibration formula can be obtained:

[0056]

[0057] and

[0058] As shown above, for the same camera, at the same sensitivity (ISO), the Gaussian term b and Poisson term a in the image sensor output RAW image noise parameters are the same, and these two terms may be different at different ISOs. However, for different cameras, even if the ISO is the same, the noise parameters of the Gaussian term and Poisson term are inconsistent. It should be noted that under other noise models, ISO is also the main factor affecting the camera noise parameters.

[0059] In view of this situation, this technical solution proposes a method as follows Figure 1 The image denoising model shown in FIG. 1 is used to cope with the situation where different cameras have different noise parameters. Figure 1 As shown, the image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and multiple cameras specifically refer to two or more cameras.

[0060] The input noise image is input into the feature extraction branch to extract the features of the noise image, and at least two sets of feature maps are output; the fusion weight is adaptively configured according to the camera information to which the input noise image belongs, and the feature maps output by all feature extraction branches are weighted fused according to the fusion weight; the image is reconstructed according to the fused feature map to obtain the denoised image.

[0061] Figure 2 FIG. 1 is a flow chart of the image noise reduction method in the embodiment of the present application. Figure 2 As shown, the method may specifically include:

[0062] Step 201: Acquire a first noise image and first camera information of a first camera;

[0063] It should be noted that the first camera can be any one of the multiple cameras possessed by the shooting device, and the first noise image can be a RAW domain image or a YUV domain image captured by the first camera, that is, the image noise reduction processing can be located in the front, middle or back section of the image signal processing.

[0064] The first camera may also be more than two cameras, and the first noise image may be a fused image of images from the two or more cameras.

[0065] Step 202: Determine the fusion weight of the image denoising model according to the first camera information;

[0066] It should be noted that the first camera information may include any one or more of the camera parameters, environmental information, identification information, etc. of the camera.

[0067] In some embodiments, according to the first camera information, determining the fusion weight of the image denoising model includes: according to the identification information of the first camera, determining the fusion weight corresponding to the first camera as the first weight, and determining the fusion weights corresponding to other cameras as the second weight; or, according to the identification information and exposure information of the first camera, determining the fusion weight corresponding to the first camera as the first weight, and determining the fusion weights corresponding to other cameras as the second weight; wherein the first weight is greater than the second weight. The second weight is used to indicate the weights corresponding to other cameras, and does not specifically refer to a fixed value. The weights corresponding to other cameras may be equal or unequal.

[0068] Exemplarily, during the training phase, the fusion weight is determined according to the identification information of the camera.

[0069] It should be noted that, during the training phase, the first weight is greater than the second weight, and each time the noise characteristics of the camera to which the input image belongs are strengthened to train the corresponding feature extraction branch, effectively extract the image features of the camera, and weaken the training of the feature extraction branches corresponding to other cameras, so that different feature extraction branches adapt to the noise reduction requirements of different cameras, and ensure the image noise reduction effect of different cameras. Exemplarily, the value of the first weight is close to 1, and the value of the second weight is close to 0. In some embodiments, the first weight is 1 and the second weight is 0.

[0070] Exemplarily, during the use phase, the fusion weight is determined based on the identification information and exposure time of the camera.

[0071] It should be noted that the exposure information may specifically be the exposure time and ISO, and the fusion weight can be flexibly adjusted according to the exposure time and ISO. For example, the brightness of the current shooting environment can be determined based on the exposure time. When the shooting environment is bright, the feature extraction branch corresponding to the camera currently in use can extract sufficient image features. A larger weight can be set for the feature extraction branch corresponding to the camera currently in use, and smaller weights can be set for other feature extraction branches. When the shooting environment is dark, it is necessary to use the image features extracted by other feature extraction branches for image enhancement. The larger weight set for the feature extraction branch corresponding to the camera currently in use can be reduced, and the smaller weights set for other feature extraction branches can be increased, thereby ensuring the noise reduction effect in all shooting environments.

[0072] It should be noted that since the feature extraction branch corresponding to each camera is trained according to the noise characteristics of different cameras and has the best feature extraction capability, the weight of the feature extraction branch corresponding to the camera currently in use should be greater than 0.5.

[0073] Exemplarily, in the use stage, the fusion weight is determined according to the identification information and exposure information of the camera. The first weight is less than 1 and greater than 0.5, and the second weight is greater than 0 and less than 0.5.

[0074] Exemplarily, when the image denoising module is in online reasoning, it specifically includes: 1) In the camera preview screen, the image denoising module can obtain the original image data and the corresponding camera identification information in real time; 2) According to the camera identification information, the fusion weight, that is, Input_1, can be configured dynamically in real time according to information such as ISO and exposure time. For example, when processing the original image data of camera 1, Input_1 can be set to [0.8, 0.1, 0.1], instead of strictly setting it to [1, 0, 0] during training. By fusing the feature maps of each branch, more image features can be obtained to improve the denoising effect. A dynamic weight adjustment strategy can also be set according to demand, but it is necessary to ensure that the weight of the feature extraction branch corresponding to the current camera is greater than 0.5.

[0075] Step 203: performing denoising processing on the first noisy image by using an image denoising model to obtain a first denoised image corresponding to the first noisy image;

[0076] in, Figure 3 FIG. 1 is a flow chart of a method for implementing an image denoising model in an embodiment of the present application. Figure 3 As shown, the image denoising model is configured as:

[0077] Step 301: Processing a first noise image through at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image;

[0078] It should be noted that the feature extraction branch is used to perform feature encoding on the noise sample image, or the feature extraction branch is used to perform feature decoding on the noise sample image, or the feature extraction branch is used to perform feature encoding and feature decoding on the input image.

[0079] Step 302: performing weighted fusion on at least two groups of feature maps according to fusion weights to obtain a fused feature map;

[0080] It should be noted that the fusion weight includes the weights of the feature maps output by at least two feature extraction branches. The weighted fusion implementation process can be expressed as:

[0081]

[0082] Among them, N is the number of feature extraction branches, F i is the feature map output by the i-th feature extraction branch, w i is the fusion weight of the feature map output by the feature extraction branch of the i-th frame, and M is the final fused feature map.

[0083] It should be noted that each set of feature maps may include multi-channel feature maps, and the channel feature maps corresponding to different feature extraction branches are weightedly fused to obtain a multi-channel fused feature map.

[0084] Step 303: Obtain a first denoised image according to the fused feature map.

[0085] Specifically, image reconstruction is performed according to the fused feature map to obtain a denoised image.

[0086] When using any of the image denoising models provided in the embodiments of the present application for image denoising, the denoising processing of multiple camera image data can be achieved by sharing one image denoising model without executing complex switching decision logic. Through different feature extraction branches in the image denoising model, the image denoising effect of different cameras can be guaranteed. Then, the feature maps obtained by the feature extraction branches are fused according to the adaptive fusion weights, which can improve the final image denoising effect.

[0087] In some embodiments, the method further includes: constructing an image denoising model. Specifically, the method may include: determining the noise parameters of the first camera and the noise parameters of the second camera; determining that the first camera and the second camera in the image denoising model are configured with a common feature extraction branch according to the noise parameters of the first camera and the noise parameters of the second camera; or determining that the first camera and the second camera in the image denoising model are configured with two independent feature extraction branches according to the noise parameters of the first camera and the noise parameters of the second camera.

[0088] That is to say, when constructing an image denoising model for multiple cameras, first establish a noise model for each camera, and when the noise models of two or more cameras are determined to be the same or similar based on the noise parameters in the noise model, configure a shared branch for them, otherwise, configure an independent feature extraction branch for them. Exemplarily, if the noise models of two of the three cameras are similar, the two cameras share the same feature extraction branch, and the third camera is set up with a separate feature extraction branch, that is, for the three cameras, only two feature extraction branches need to be configured in the image denoising model.

[0089] In some embodiments, when the image denoising model is being trained, the method further includes: obtaining a training data set; wherein the training data set includes noise sample images and clear sample images of at least two cameras; iteratively updating the parameters of the image denoising model according to the noise sample images corresponding to the noise sample images output by the image denoising model, and the clear sample images corresponding to the noise sample images, so as to obtain a trained image denoising model. That is, in the training stage, obtaining the first noise image and the first camera information of the first camera includes: obtaining the first noise sample image and the first camera information of the camera to which the first noise sample image belongs from the training data set.

[0090] It should be noted that the training data set includes a large number of noisy sample images and their corresponding clear sample images under various shooting parameters of the camera. Exemplarily, first prepare clear sample images and noisy sample images under different shooting parameters of two or more cameras, and the clear sample images and the noisy sample images constitute an image pair; the image pairs of different cameras are placed in different files, and the image pairs obtained under different shooting parameters of each camera can be directly mixed and stored. Among them, the clear sample image can be understood as the pixel true value (Ground Ture, GT) containing the noise sample image.

[0091] It should be noted that the loss function value is determined based on the denoised sample image and the clear sample image; when it is determined based on the loss function value that the training does not converge, the parameters of the image denoising model are iteratively updated; when it is determined based on the loss function value that the training converges, the training is completed.

[0092] In some embodiments, the first weight is 1 and the second weight is 0. Iteratively updating the parameters of the image denoising model includes iteratively updating the model parameters except for the feature extraction branches corresponding to other cameras in the image denoising model; wherein the other cameras are cameras other than the camera to which the input noise sample image belongs.

[0093] That is to say, each time the reverse iteration is updated, the feature extraction branch corresponding to the camera to which the input image belongs and the branch shared by all cameras will be gradually updated to perform denoising on the image of a specific camera. After the training converges, the image denoising model obtained can effectively cope with the noise of different cameras and ensure the image denoising effect of different cameras.

[0094] The structure of the image denoising model is further illustrated below with examples.

[0095] In some embodiments, Figure 4 As shown, the feature extraction branch includes a first encoding module for performing feature encoding, and the first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, including: the first noise image is feature encoded by at least two first encoding modules respectively to obtain at least two sets of feature maps of the first noise image;

[0096] The image denoising model also includes a first decoding module; obtaining a first denoised image according to the fused feature map includes: performing feature decoding on the fused feature map through the first decoding module to obtain a clean first denoised image.

[0097] In some embodiments, the image denoising model includes at least two multipliers and one adder to achieve weighted fusion of at least two sets of feature maps.

[0098] It should be noted that any encoding module provided in the embodiments of the present application implements feature encoding through at least one convolutional layer, and any decoding module implements feature decoding through at least one convolutional layer, and the convolutional layer parameters of the encoding module and the decoding module can be determined through model training.

[0099] Exemplarily, the first encoding module includes multiple convolutional layers for feature encoding. The encoding module composed of multiple convolutional layers in the embodiment of the present application can be understood as an encoding network (Encoder). The first decoding module includes multiple convolutional layers for feature decoding. The decoding module composed of multiple convolutional layers in the embodiment of the present application can be understood as a decoding network (Decoder). In addition, the encoding module and the decoding module may also include an activation layer, a normalization layer, etc. In other words, in order to ensure the noise reduction effect, a more complex encoding network and a decoding network can be used to achieve feature encoding and feature decoding.

[0100] Exemplarily, the first encoding module includes a convolutional layer for feature encoding, and the first decoding module includes a convolutional layer for feature decoding. That is to say, for example, to reduce the complexity of the model, feature encoding can be achieved through a convolutional layer, and feature decoding can also be achieved through a convolutional layer. By designing different encoding and decoding structures in the image denoising model, the denoising function of electronic devices with different computing capabilities can be achieved.

[0101] Take the configuration of three feature extraction branches in the image denoising model as an example, where one camera corresponds to one feature extraction branch, or two or more cameras can share one feature extraction branch. The single encoding modules of the feature extraction branch are defined as encoding network 1, encoding network 2, and encoding network 3 from top to bottom, respectively, which are used to extract the image features of cameras 1, 2, and 3, respectively. During training, each time a noise sample image is read from the data set as Input_0, and then input into the three encoding networks, that is, all three encoding networks will take effect and be used to extract the features of the input single image. If the input image belongs to camera 1, the corresponding Input_1 is set to [1,0,0], that is, the features extracted by encoding network 2 and encoding network 3 will be shielded through weighted fusion. This setting can be achieved. During training, each reverse iterative update will gradually update the weight of encoding network 1 to extract the image features of camera 1. Similarly, when inputting images from camera 2 or camera 3, different Input_1 inputs are used to enable encoding network 2 or encoding network 3. After the training converges, encoding network 1, encoding network 2 or encoding network 3 will be able to effectively extract the features of the object corresponding to the camera and send them to the subsequent decoding module.

[0102] By setting multiple feature extraction branches in an image denoising model to extract features, and determining the fusion weights according to the camera information for weighted fusion, different feature extraction branches can be trained for different camera characteristics. This enables the use of an image denoising model to denoise image data from cameras with different noise characteristics, flexibly and effectively improving the denoising effect of multiple cameras.

[0103] In some embodiments, Figure 5 As shown, the image denoising model also includes a first common branch, and the first common branch includes a cascaded third encoding module and a third decoding module; before the first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, the method also includes: feature encoding and feature decoding of the first noise image in sequence by the third encoding module and the third decoding module. Furthermore, feature re-encoding is performed by the first encoding module of the feature extraction branch, and the fused feature map is decoded by the first decoding module to output a denoised image.

[0104] Exemplarily, the first encoding module includes a convolutional layer, and the first decoding module includes a convolutional layer; the third encoding module includes multiple convolutional layers, and the third decoding module includes multiple convolutional layers. Take the configuration of three feature extraction branches in the image denoising model as an example. The three cameras share a common encoding network and a decoding network for feature encoding and decoding of the input. The decoded information is respectively input into the encoding module of the feature extraction branch for re-encoding. The single encoding modules of the feature extraction branch are defined as conv1, conv2, and conv3 from top to bottom. If the input image belongs to camera 1, the corresponding Input_1 is set to [1,0,0], that is, the features extracted by conv2 and conv3 will be masked through weighted fusion. The fused feature map is then reconstructed through a layer of convolutional layer conv, and the convolution output is the final denoising result. Compared with the feature extraction branch using a multi-convolutional layer encoding network for feature extraction, this method uses a single convolutional layer and is suitable for edge devices with smaller platform computing power.

[0105] Exemplarily, the first encoding module includes multiple convolutional layers, and the first decoding module includes multiple convolutional layers; the second encoding module includes one convolutional layer, and the second decoding module includes one convolutional layer.

[0106] In some embodiments, Figure 6 As shown, the image denoising model further includes a second common branch, and the second common branch includes a cascaded fourth encoding module and a fourth decoding module;

[0107] According to the fused feature map, a first denoised image is obtained, including: performing feature decoding on the fused feature map through the first decoding module to obtain the output feature map of the first decoding module, and the first decoding module inputs the output feature map to the fourth encoding module; and performing feature encoding and feature decoding on the feature map again through the fourth encoding module and the fourth decoding module to obtain the first denoised image. In other words, multiple cameras share the fourth encoding module and the fourth decoding module to perform secondary feature encoding and decoding on the features decoded by the first decoding module to obtain a denoised image.

[0108] Exemplarily, the first encoding module includes a convolution layer, and the first decoding module includes a convolution layer; the fourth encoding module includes multiple convolution layers, and the fourth decoding module includes multiple convolution layers. For example, the image denoising model is configured with three feature extraction branches, and the input image is respectively input into the encoding modules of the three feature extraction branches for encoding. The single encoding modules of the feature extraction branches are defined as conv1, conv2, and conv3 from top to bottom, respectively, which are used to extract the image features of cameras 1, 2, and 3, respectively. During training, each time a noise sample image is read from the data set as Input_0, it is then input into the three convolution layers to extract the features of the input single image. If the input image belongs to camera 2, the corresponding Input_1 is set to [0,1,0], that is, the features extracted by conv1 and conv3 will be masked through weighted fusion. This setting can be achieved, during training, each reverse iterative update will gradually update the weight of conv2 to extract the image features of camera 2. The fused features are then fully fused through a convolutional layer conv, and the fully fused features are then input into the shared encoding network and decoding network for secondary feature encoding and decoding, and the final denoising results are output. Compared to the feature extraction branch that uses a multi-convolutional layer encoding network for feature extraction, this method uses a single convolutional layer and is suitable for edge devices with smaller platform computing power.

[0109] Exemplarily, the first encoding module includes multiple convolutional layers, and the first decoding module includes multiple convolutional layers; the fourth encoding module includes multiple convolutional layers, and the fourth decoding module includes multiple convolutional layers.

[0110] In some embodiments, Figure 7 As shown, the feature extraction branch includes a second decoding module, and the image denoising model also includes a second encoding module and a first decoding module;

[0111] The first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, including: feature encoding the first noise image by a second encoding module; the feature map output by the second encoding module is sent to the second decoding modules of the at least two feature extraction branches respectively for feature decoding to obtain at least two sets of feature maps of the first noise image;

[0112] Obtaining a first denoised image according to the fused feature map includes: performing feature decoding on the fused feature map through a first decoding module to obtain a clean first denoised image.

[0113] Exemplarily, the second encoding module includes multiple convolutional layers, the second decoding module includes multiple convolutional layers; and the first decoding module includes one convolutional layer. Taking the image denoising model configured with three feature extraction branches as an example, three cameras share a coding network to extract the features of the input image, and then send them to the decoding modules of the three feature extraction branches for decoding processing. The single decoding modules of the feature extraction branches are defined as decoding network 1, decoding network 2, and decoding network 3 from top to bottom, respectively. An additional input Input_1 will be set to perform weighted fusion on the results of the decoding network. The settings of the data set, network training, and online inference are similar to those of the aforementioned model. It can be understood that each decoding network in the model corresponds to a specific camera. After the decoding network results are weighted fused, the fused information is reconstructed through a convolutional layer conv, and the convolution output is the final denoising result.

[0114] In some embodiments, Figure 8 As shown, the feature extraction branch includes a cascaded second encoding module and a second decoding module; the first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, including: feature encoding and feature decoding of the first noise image are performed by at least two cascaded second encoding modules and second decoding modules respectively to obtain at least two sets of feature maps of the first noise image;

[0115] The image denoising model also includes a first decoding module, which obtains a first denoised image according to the fused feature map, including: performing feature decoding on the fused feature map through the first decoding module to obtain a clean first denoised image.

[0116] That is to say, each branch performs feature encoding and feature decoding on the input image, performs weighted fusion on multiple groups of feature maps output by multiple branch modules, and then reconstructs the fused feature maps through a common decoding module to obtain a denoised image.

[0117] Exemplarily, the second encoding module includes a convolutional layer, the second decoding module includes a convolutional layer; and the first decoding module includes at least one convolutional layer.

[0118] In some embodiments, the feature extraction branch includes a cascaded second encoding module and a second decoding module; processing the first noise image through at least two feature extraction branches to obtain at least two sets of feature maps of the first noise image includes: performing feature encoding and feature decoding on the first noise image through at least two cascaded second encoding modules and second decoding modules to obtain at least two sets of feature maps of the first noise image;

[0119] The image denoising model also includes a first encoding module and a first decoding module. According to the fused feature map, a first denoised image is obtained, including: feature decoding of the fused feature map is performed by the first encoding module and the first decoding module to obtain a clean first denoised image.

[0120] Exemplarily, the second encoding module includes a convolution layer, and the second decoding module includes a convolution layer; the first encoding module includes multiple convolution layers, and the first decoding module includes multiple convolution layers. Take the configuration of three feature extraction branches in the image denoising model as an example. From top to bottom, the single encoding modules in the three feature extraction branches are defined as conv11, conv12 and conv13, and the single decoding modules are defined as conv21, conv22 and conv23. The feature extraction branches perform the first feature encoding and feature decoding on the input image respectively. If the input image belongs to camera 3, the corresponding Input_1 is set to [0,0,1], that is, the features extracted by the first feature extraction branch and the second feature extraction branch will be shielded through weighted fusion. The fused features are input into the common encoding network and decoding network for secondary feature encoding and decoding, and the output is the final denoising result.

[0121] In some embodiments, the present application Figure 7 and Figure 8 The provided image denoising model may further include a first common branch, the first common branch including a cascaded third encoding module and a third decoding module; the first noise image is processed respectively by at least two feature extraction branches, and before at least two sets of feature maps of the first noise image are obtained, the first noise image is feature encoded and feature decoded in turn by the third encoding module and the third decoding module.

[0122] Exemplarily, the third encoding module includes multiple convolutional layers, and the third decoding module includes multiple convolutional layers.

[0123] In some embodiments, in some embodiments, the present application Figure 7 and Figure 8 The provided image denoising model may further include a second common branch, the second common branch including a cascaded fourth encoding module and a fourth decoding module;

[0124] According to the fused feature map, a first denoised image is obtained, including: performing feature decoding on the fused feature map through the first decoding module to obtain the output feature map of the first decoding module, and the first decoding module inputs the output feature map to the fourth encoding module; and performing feature encoding and feature decoding on the feature map again through the fourth encoding module and the fourth decoding module to obtain the first denoised image. In other words, multiple cameras share the fourth encoding module and the fourth decoding module to perform secondary feature encoding and decoding on the features decoded by the first decoding module to obtain a denoised image.

[0125] In practical applications, based on the image denoising model provided in the above embodiments, the encoding module and decoding module of the image denoising model can be flexibly selected according to the computing power of the electronic device, but different models can effectively cope with and process image data from multiple cameras, and can effectively improve the operating efficiency of multi-camera electronic devices, as well as the noise reduction effect.

[0126] It should be noted that the image denoising model provided in the embodiment of the present application can cope with any number of cameras. The above embodiment takes the case of three cameras as an example and is not used to limit the number of cameras. For different numbers of cameras, it is only necessary to modify the number of branches in the model, that is, n feature extraction branches correspond to n or more cameras.

[0127] It should be noted that the image denoising model provided in the embodiment of the present application can be adapted to the denoising module of any node in the electronic device, such as RAW domain denoising / YUV domain denoising. The image data can be denoised directly in the RAW domain, or the image data in the YUV domain can be denoised at the end of the ISP image processing flow.

[0128] In the embodiments of the present application, 1. an image denoising model including common branches and feature extraction branches is designed, which can adaptively and effectively handle the denoising requirements of multiple cameras; 2. a multi-branch training data set, a network model training method and an online inference method are set to ensure the effect of model training and inference; 3. common branches and feature extraction branch structures of different magnitudes are designed, and the corresponding structure can be adaptively selected according to the computing power of the edge device.

[0129] In order to implement the image noise reduction method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image noise reduction device, such as Fig. 9 As shown, the image noise reduction device 90 includes:

[0130] An acquisition unit 901 is used to acquire a first noise image and first camera information of a first camera;

[0131] A determination unit 902, configured to determine a fusion weight of an image denoising model according to the first camera information;

[0132] The processing unit 903 is configured to perform denoising on the first noisy image by using an image denoising model to obtain a first denoised image corresponding to the first noisy image;

[0133] Among them, the image denoising model is configured as:

[0134] Processing the first noise image respectively by at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image;

[0135] Perform weighted fusion on at least two groups of feature maps according to fusion weights to obtain a fused feature map;

[0136] According to the fused feature map, a first denoised image is obtained.

[0137] In some embodiments, the feature extraction branch includes a first encoding module for feature encoding, and the image denoising model is specifically configured as follows: feature encoding is performed on the first noise image by at least two first encoding modules respectively to obtain at least two sets of feature maps of the first noise image; and feature decoding is performed on the fused feature map by a first decoding module to obtain a first denoised image.

[0138] In some embodiments, the first encoding module includes multiple convolutional layers, and the first decoding module includes multiple convolutional layers.

[0139] In some embodiments, the feature extraction branch includes a second decoding module, and the image denoising model further includes a second encoding module;

[0140] The specific configuration of the image denoising model is as follows: feature encoding is performed on the first noisy image through the second encoding module; the feature map output by the second encoding module is sent to the second decoding modules of at least two feature extraction branches for feature decoding respectively, so as to obtain at least two sets of feature maps of the first noisy image; feature decoding is performed on the fused feature map through the first decoding module, so as to obtain the first denoised image.

[0141] In some embodiments, the second encoding module includes multiple convolutional layers, the second decoding module includes multiple convolutional layers; and the first decoding module includes one convolutional layer.

[0142] In some embodiments, the feature extraction branch includes a cascaded second encoding module and a second decoding module;

[0143] The image denoising model is specifically configured as follows: feature encoding and feature decoding are performed on the first noisy image through at least two cascaded second encoding modules and second decoding modules respectively to obtain at least two sets of feature maps of the first noisy image; and feature decoding is performed on the fused feature map through the first decoding module to obtain a first denoised image.

[0144] In some embodiments, the second encoding module includes a convolutional layer, the second decoding module includes a convolutional layer; and the first decoding module includes at least one convolutional layer.

[0145] In some embodiments, the image denoising model also includes a first common branch, which includes a cascaded third encoding module and a third decoding module; the image denoising model is specifically configured as follows: feature encoding and feature decoding are performed on the first noise image in sequence through the third encoding module and the third decoding module; feature extraction is performed on the input feature map respectively through at least two feature extraction branches, before obtaining at least two groups of feature maps of the first noise image.

[0146] In some embodiments, the first encoding module includes a convolutional layer, and the first decoding module includes a convolutional layer; the second encoding module includes multiple convolutional layers, and the second decoding module includes multiple convolutional layers.

[0147] In some embodiments, the image denoising model further includes a second common branch, the second common branch including a cascaded fourth encoding module and a fourth decoding module;

[0148] The specific configuration of the image denoising model is as follows: the fused feature map is feature decoded by the first decoding module to obtain the output feature map of the first decoding module, and the first decoding module inputs the output feature map to the fourth encoding module; the feature map is feature encoded and feature decoded again by the fourth encoding module and the fourth decoding module to obtain the first denoised image.

[0149] In some embodiments, the first encoding module includes one convolutional layer, and the first decoding module includes one convolutional layer; the fourth encoding module includes multiple convolutional layers, and the fourth decoding module includes multiple convolutional layers.

[0150] In some embodiments, the determination unit 902 is used to determine, based on the identification information of the first camera, that the fusion weight corresponding to the first camera is the first weight, and determine the fusion weights corresponding to other cameras are the second weight; or, based on the identification information and exposure information of the first camera, determine that the fusion weight corresponding to the first camera is the first weight, and determine the fusion weights corresponding to other cameras are the second weight; wherein the first weight is greater than the second weight.

[0151] In some embodiments, when the image denoising model is used, the first weight is less than 1 and greater than 0.5, and the second weight is greater than 0 and less than 0.5.

[0152] In some embodiments, when the image denoising model is being trained, the first weight is 1 and the second weight is 0.

[0153] In some embodiments, when the image denoising model is being trained, the acquisition unit 901 is further used to acquire a training data set; wherein the training data set includes noisy sample images and clear sample images of at least two cameras;

[0154] The processing unit 903 is also used to iteratively update the parameters of the image denoising model according to the denoised sample image corresponding to the noisy sample image output by the image denoising model, and the clear sample image corresponding to the noisy sample image, so as to obtain a trained image denoising model.

[0155] In some embodiments, the device further includes: a model building unit, configured to determine a noise parameter of the first camera and a noise parameter of the second camera; and, according to the noise parameter of the first camera and the noise parameter of the second camera, determining that the first camera and the second camera in the image denoising model are configured with a common feature extraction branch; or,

[0156] According to the noise parameters of the first camera and the noise parameters of the second camera, it is determined that the first camera and the second camera in the image denoising model are configured with two independent feature extraction branches.

[0157] In practical applications, the above device can be an electronic device for implementing image denoising tasks, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the image denoising method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the image denoising method, and this application will not go into details one by one.

[0158] Based on the hardware implementation of each unit in the above-mentioned image noise reduction device, the embodiment of the present application also provides an electronic device, such as Fig.10 As shown, the electronic device 100 includes: a processor 1001 and a memory 1002 configured to store a computer program that can be run on the processor; wherein the processor 1001 is configured to execute the method steps in the aforementioned embodiment when running the computer program.

[0159] Of course, in practical applications, Fig.10 As shown, the various components in the electronic device 100 are coupled together through a bus system 1003. It can be understood that the bus system 1003 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1003 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 1003 in the figure.

[0160] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0161] The above-mentioned memory may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor. The memory may be a separate device independent of the processor, or may be integrated in the processor.

[0162] Optionally, the device may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may acquire a noise image sent by other devices or chips.

[0163] Optionally, the device may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output a clear image to other devices or chips.

[0164] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned method.

[0165] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0166] Optionally, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0167] The embodiment of the present application also provides a computer program.

[0168] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0169] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0170] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0171] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0174] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0175] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image denoising method, characterized in that: The image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the method includes: Acquire a first noise image and first camera information from a first camera; Determining a fusion weight of the image denoising model according to the first camera information; Performing denoising on the first noisy image by using the image denoising model to obtain a first denoised image corresponding to the first noisy image; Wherein, the image denoising model is configured as follows: Processing the first noise image respectively by at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image; Performing weighted fusion on the at least two groups of feature maps according to the fusion weight to obtain a fused feature map; The first denoised image is obtained according to the fused feature map.

2. The method according to claim 1, characterized in that The feature extraction branch includes a first encoding module for performing feature encoding, and the first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, including: Performing feature encoding on the first noise image by at least two first encoding modules respectively to obtain at least two groups of feature maps of the first noise image; The image denoising model further includes a first decoding module; and obtaining the first denoised image according to the fused feature map includes: The first decoding module performs feature decoding on the fused feature map to obtain the first denoised image.

3. The method according to claim 2, characterized in that The first encoding module includes multiple convolutional layers, and the first decoding module includes multiple convolutional layers.

4. The method according to claim 1, characterized in that The feature extraction branch includes a second decoding module, and the image denoising model also includes a second encoding module; The step of processing the first noise image by using at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image includes: Performing feature encoding on the first noise image by the second encoding module; Sending the feature maps output by the second encoding module to the second decoding modules of the at least two feature extraction branches for feature decoding, respectively, to obtain at least two groups of feature maps of the first noise image; The image denoising model further includes a first decoding module, and obtaining the first denoised image according to the fused feature map includes: The first decoding module performs feature decoding on the fused feature map to obtain the first denoised image.

5. The method according to claim 4, characterized in that The second encoding module includes a plurality of convolutional layers, and the second decoding module includes a plurality of convolutional layers; The first decoding module includes a convolutional layer.

6. The method according to claim 1, characterized in that The feature extraction branch includes a cascaded second encoding module and a second decoding module; The step of processing the first noise image by using at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image includes: Performing feature encoding and feature decoding on the first noise image respectively by at least two cascaded second encoding modules and second decoding modules, to obtain at least two groups of feature maps of the first noise image; The image denoising model further includes a first decoding module, and obtaining the first denoised image according to the fused feature map includes: The first decoding module performs feature decoding on the fused feature map to obtain the first denoised image.

7. The method according to claim 6, characterized in that The second encoding module includes a convolutional layer, and the second decoding module includes a convolutional layer; The first decoding module includes at least one convolutional layer.

8. The method according to claim 2, 4 or 6, characterized in that: The image denoising model further includes a first common branch, the first common branch including a cascaded third encoding module and a third decoding module; before the first noise image is processed by at least two feature extraction branches respectively to obtain at least two sets of feature maps of the first noise image, the method further includes: The first noise image is sequentially feature encoded and feature decoded by the third encoding module and the third decoding module, and the output feature maps are respectively input into the at least two feature extraction branches for processing.

9. The method according to claim 8, characterized in that The first encoding module includes a convolutional layer, and the first decoding module includes a convolutional layer; The second encoding module includes multiple convolutional layers, and the second decoding module includes multiple convolutional layers.

10. The method according to claim 2, 4 or 6, characterized in that: The image denoising model further includes a second common branch, wherein the second common branch includes a cascaded fourth encoding module and a fourth decoding module; The step of obtaining the first denoised image according to the fused feature map includes: Performing feature decoding on the fused feature map by the first decoding module to obtain an output feature map of the first decoding module, and the first decoding module inputs the output feature map to the fourth encoding module; The feature map is again feature encoded and feature decoded by the fourth encoding module and the fourth decoding module to obtain the first denoised image.

11. The method according to claim 10, characterized in that The first encoding module includes a convolutional layer, and the first decoding module includes a convolutional layer; The fourth encoding module includes multiple convolutional layers, and the fourth decoding module includes multiple convolutional layers.

12. The method according to claim 1, characterized in that The determining, according to the first camera information, a fusion weight of the image denoising model includes: According to the identification information of the first camera, the fusion weight corresponding to the first camera is determined to be the first weight, and the fusion weights corresponding to other cameras are determined to be the second weight Alternatively, according to the identification information and exposure information of the first camera, determining the fusion weight corresponding to the first camera as the first weight, and determining the fusion weights corresponding to other cameras as the second weight; The first weight is greater than the second weight.

13. The method according to claim 12, characterized in that When the image denoising model is in use, the first weight is less than 1 and greater than 0.5, and the second weight is greater than 0 and less than 0.

5.

14. The method according to claim 12, characterized in that When the image denoising model is being trained, the first weight is 1 and the second weight is 0.

15. The method according to claim 1, characterized in that When the image denoising model is being trained, the method further comprises: Acquire a training data set; wherein the training data set includes noisy sample images and clear sample images of at least two cameras; According to the denoised sample image corresponding to the noisy sample image output by the image denoising model, and the clear sample image corresponding to the noisy sample image, the parameters of the image denoising model are iteratively updated to obtain a trained image denoising model.

16. The method according to claim 1, characterized in that The method further comprises: Determining a noise parameter of the first camera and a noise parameter of the second camera; According to the noise parameters of the first camera and the noise parameters of the second camera, determining that the first camera and the second camera in the image denoising model are configured with a common feature extraction branch; or, According to the noise parameters of the first camera and the noise parameters of the second camera, it is determined that the first camera and the second camera in the image denoising model are configured with two independent feature extraction branches.

17. An image noise reduction device, characterized in that: The image denoising model includes at least two feature extraction branches, each feature extraction branch corresponds to one or more cameras, and the image denoising device includes: An acquisition unit, used to acquire a first noise image and first camera information from a first camera; A determination unit, configured to determine a fusion weight of the image denoising model according to the first camera information; a processing unit, configured to perform denoising on the first noisy image by using the image denoising model to obtain a first denoised image corresponding to the first noisy image; Wherein, the image denoising model is configured as follows: Processing the first noise image respectively by at least two feature extraction branches to obtain at least two groups of feature maps of the first noise image; Performing weighted fusion on the at least two groups of feature maps according to the fusion weight to obtain a fused feature map; The first denoised image is obtained according to the fused feature map.

18. An electronic device, characterized in that: The electronic device comprises: a processor and a memory configured to store a computer program capable of running on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 16 when running the computer program.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images