Image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a storage medium. According to the image processing method, multi-scale image features are extracted from to-be-processed image data to obtain a multi-scale feature map, then feature fusion is performed on the multi-scale feature map to obtain a fused feature map, and a noise reduction image and a detail image corresponding to the to-be-processed image data are respectively predicted based on the fused feature map. And obtaining two result outputs of a noise reduction image and a detail image based on the original input. In the scheme, the multi-scale features are fused, the global features and the local detail features are combined to describe the image more accurately, and finally the noise reduction image and the detail image are output respectively, so that the single-input multi-output effect is achieved, the image noise reduction function can be realized, the image detail enhancement function can also be realized, and the image processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Image noise refers to unnecessary or redundant interference information in image data. In the process of image formation and transmission, images will inevitably be interfered by noise. The noise and signal in the image are often intertwined and difficult to distinguish. Therefore, image noise reduction is an important part of image processing.

[0003] In recent years, the use of image denoising network models based on artificial intelligence (AI) to denoise images has become increasingly common in various image applications. As the computing power of various device platforms has gradually increased, image denoising network models have begun to be used in various electronic devices. However, due to the limitations of device performance, especially mobile electronic devices, if real-time performance is required, a relatively small model is required, and the output effect of a small model is significantly worse than that of a large model, which is prone to problems such as incomplete image denoising and poor image detail restoration. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve image processing effects.

[0005] In a first aspect, the present application provides an image processing method, applied to an electronic device, comprising:

[0006] Obtain image data to be processed;

[0007] Extracting multi-scale image features from the image data to be processed to obtain a multi-scale feature map;

[0008] Performing feature fusion on the multi-scale feature map to obtain a fused feature map;

[0009] Based on the fused feature map, a denoised image and a detail image corresponding to the image data to be processed are predicted respectively.

[0010] In a second aspect, the present application provides an image processing device, applied to an electronic device, comprising:

[0011] An acquisition unit, used for acquiring image data to be processed;

[0012] An extraction unit, used for extracting multi-scale image features from the image data to be processed to obtain a multi-scale feature map;

[0013] A fusion unit, used for performing feature fusion on the multi-scale feature map to obtain a fused feature map;

[0014] A processing unit is used to predict the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map.

[0015] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the image processing method provided in the present application.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor to implement the steps in the image processing method provided in the present application.

[0017] In the embodiment of the present application, multi-scale image features are extracted from the image data to be processed to obtain a multi-scale feature map, and then the multi-scale feature map is subjected to feature fusion to obtain a fused feature map, and based on the fused feature map, the denoised image and detail image corresponding to the image data to be processed are predicted respectively, and two result outputs of the denoised image and detail image are obtained based on the original input. In this solution, by fusing multi-scale features, global features and local detail features are combined to more accurately describe the image, and finally the denoised image and detail image are output respectively, so as to achieve the effect of single input and multiple outputs, and realize both the image denoising function and the image detail enhancement function, thereby improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution in this embodiment, the drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a schematic diagram of a scene of an image processing system provided in an embodiment of the present application;

[0020] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the first structure of the image denoising network model provided in the embodiment of the present application;

[0022] Figure 4 This is a second structural diagram of the image denoising network model provided in an embodiment of the present application;

[0023] Figure 5 This is a third structural diagram of the image denoising network model provided in the embodiment of the present application;

[0024] Figure 6 This is a fourth structural diagram of the image denoising network model provided in an embodiment of the present application;

[0025] Figure 7 This is a fifth structural diagram of the image denoising network model provided in the embodiment of the present application.

[0026] Figure 8 is another flowchart of the image processing method provided in the embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of the architecture of an image processing system provided in an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that the principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0031] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] The current AI image denoising network model generally has a single output function, that is, the image denoising network model only implements the denoising function, and the network model only outputs a denoised image. However, the denoised image is blurred and has lost details compared to the original input image, and when the image with lost details is transmitted to the back-end image signal processing (ISP) module, it is more difficult to restore the details.

[0035] Based on this, the embodiments of the present application provide an image processing method, an image processing device, an electronic device, and a storage medium. The image processing method can be executed by the image processing device, or by an electronic device integrated with the image processing device.

[0036] The following will be combined with the drawings in this embodiment to clearly and completely describe the technical solution in this embodiment. Obviously, the described embodiment is only a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0037] Please refer to Figure 1 , the present application also provides an image processing system, such as Figure 1 As shown, the image processing system includes an electronic device 100, and the image processing device provided by the present application is integrated in the electronic device 100. For example, the electronic device 100 can obtain image data to be processed; then, extract multi-scale image features from the image data to be processed to obtain a multi-scale feature map; then, perform feature fusion on the multi-scale feature map to obtain a fused feature map; finally, based on the fused feature map, respectively predict the denoised image and the detail image corresponding to the image data to be processed.

[0038] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as a smart phone, a tablet computer, a PDA, a laptop computer, a virtual reality device, an augmented reality device or a mixed reality device, etc., a mobile electronic device with a processor, or a desktop computer, a television, a server, an industrial equipment, etc., a fixed electronic device with a processor.

[0039] In addition, if Figure 1As shown, the image processing system may further include a data warehouse 200 for storing original data, intermediate data and result data in the image processing process. For example, the electronic device 100 acquires the image to be processed (original data), and inputs the image to be processed into the image denoising network model based on deep learning. The feature extraction network in the image denoising network model performs multi-scale feature extraction on the image to be processed to obtain a multi-scale feature map (intermediate data), and performs feature fusion on the multi-scale feature map to obtain a fused feature map (intermediate data), and finally analyzes and processes the fused feature map. Finally, the image denoising network model outputs the denoised image and detail image (result data) corresponding to the image to be processed and stores them in the data warehouse 200.

[0040] It should be noted that Figure 1 The scene diagram of the image processing system shown is merely an example. The image processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the image processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0041] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0042] Please refer to Figure 2 and Figure 3 , Figure 2 is a flowchart of an image processing method provided in an embodiment of the present application; Figure 3 Schematic diagram of the first architecture of the image processing system provided in the embodiment of the present application. Figure 2 As shown, the process of the image processing method provided by this application is as follows:

[0043] 101. Obtain image data to be processed.

[0044] In the embodiment of the present application, firstly, the image data to be processed that needs to be subjected to image noise reduction processing is obtained, and the image data to be processed contains image signals and image noise. In practical applications, image noise comes from many aspects, which can be noise caused by components, such as thermal noise caused by electronic components, shot noise and flicker noise caused by vacuum devices, particle noise generated by transistors, photon noise of photoelectric tubes, and noise formed by different transformations of these noises caused by components forming various electronic circuits and devices; it can also be image optical noise caused by optical phenomena.

[0045] In this embodiment, the image to be processed may come from image data collected by the electronic device itself, or may come from image data of a third party. When the electronic device itself is integrated with an image acquisition component, image acquisition can be performed through the built-in image acquisition component, and the image to be processed is obtained based on the acquired image. That is, in one embodiment, the electronic device of this scheme is integrated with an image sensor. When acquiring the image to be processed, the original image data of the current shooting scene can be collected by the image sensor, and then the original image data is used as the image data to be processed. It should be noted that the original image data can be in a raw image format, and the raw image format image retains the information captured during exposure, and can capture the shooting characteristics of the scene as much as possible, that is, it contains physical information such as the light intensity and color of the scene.

[0046] In addition, the electronic device can also obtain image data to be processed from other electronic devices equipped with image acquisition components, and can also obtain pre-captured images from the server, which are uploaded to the server after being captured by other electronic devices equipped with image acquisition components.

[0047] 102. Extract multi-scale image features from the image data to be processed to obtain a multi-scale feature map.

[0048] Among them, the multi-scale feature map is a feature map of multiple scales obtained by extracting features from the image data to be processed at multiple scales. Specifically, a neural network model based on deep learning can be used to extract multi-scale features from the image data to be processed, thereby obtaining a multi-scale feature map, which constitutes the image pyramid output of the feature extraction network in the neural network model.

[0049] In the embodiment of the present application, the neural network model based on deep learning (DL) is a trained artificial intelligence network model that can be used for intelligent noise reduction processing of images. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0050] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in fields such as machine vision and natural language processing.

[0051] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding training sets, network models that implement different functions can be trained. For example, a gender classification model can be trained based on one training set, and an image optimization model can be trained based on another training set.

[0052] refer to Figure 3 In some embodiments, a trained U-Net neural network model can be used to extract multi-scale image features of the image to be processed. Specifically, the structure of U-Net can be divided into two parts: an encoder and a decoder. Among them, the encoder consists of a series of convolutional layers, activation functions (such as relu, prelu or leakly relu) and downsampling layers (such as maximum pooling layers). Each convolutional layer extracts image features and reduces the spatial dimension of the image. In the U-Net neural network model, the shallow network pays more attention to local features, and the feature map scale is larger; the deep network pays more attention to global features, and the feature map scale is smaller. As the network goes deeper, the extracted features are more advanced and abstract. The decoder consists of a series of upsampling, convolutional layers and activation functions (such as relu, prelu or leakly relu). The upsampling operation increases the size of the input feature map, so that it gradually recovers to the spatial resolution of the original input image (that is, the image data to be processed).

[0053] 103. Perform feature fusion on the multi-scale feature map to obtain a fused feature map.

[0054] Specifically, image features of different scales are fused, local detail features and global features are combined, and the obtained fused feature map constitutes a more accurate feature representation of the image data to be processed.

[0055] When implementing it, continue to refer to Figure 3Taking the structure of U-Net as an example, when restoring the image size, the low-level features in the encoder (i.e., local feature information) can be combined with the high-level features in the decoder through skip connections, so that the network can capture multi-scale information.

[0056] In some embodiments, when fusing multi-scale feature maps, an attention mechanism can also be introduced to make the model pay more attention to features related to image details and image noise, suppress useless information, and thus improve processing efficiency. That is, when fusing image features of different scales, the multi-scale feature maps can be fused based on the attention mechanism to obtain a fused feature map. In specific implementation, an attention module can be introduced between the convolutional layer in the encoder and the upsampling layer in the decoder at the same level in the U-Net structure. In practical applications, due to device performance limitations, in order to reduce model complexity and reduce model calculations, attention modules can be introduced only in the last few levels of the encoder in the U-Net structure.

[0057] 104. Based on the fused feature map, a denoised image and a detail image corresponding to the image data to be processed are predicted respectively.

[0058] Specifically, the above-mentioned neural network model based on deep learning can continue to be used, and it can be used as a trained image denoising network model to process the fused feature map, and output the denoised image and detail image corresponding to the image data to be processed respectively.

[0059] In this embodiment, multiple outputs outside the model are described, that is, the final output results of the model are two, one is the denoised image and the other is the detail image, as shown in the following formula:

[0060] model out1=denoise out

[0061] model out2 = detail

[0062] denoise out=res+input

[0063] Among them, model out1 represents an output result of the model, model out2 represents another output result of the model, denoise out represents the denoised image, res represents the residual image, and input represents the input image of the model. The denoised image denoise out is obtained by adding the residual image res and the input image input.

[0064] In specific implementation, a multi-layer convolution layer can be connected after the U-Net network structure to output a denoised image and a detail image. The multi-layer convolution layer can include 1 to 5 layers of convolution, which can be determined according to actual needs and electronic device performance.

[0065] Continue to refer Figure 3 In practical applications, the residual learning strategy can be introduced and applied to image denoising. A residual architecture is added to the image denoising network model, and a residual image (i.e., image noise) is obtained by using the residual network to denoise the image data to be processed, which can improve the image denoising speed and denoising effect. That is, in some embodiments, when predicting the denoised image and detail image corresponding to the image data to be processed based on the fused feature map, the following process can be specifically included:

[0066] Based on the fused feature map, the residual image and detail image corresponding to the image data to be processed are predicted respectively;

[0067] Image denoising is performed on the image data to be processed based on the residual image to obtain a denoised image.

[0068] The residual image is the difference between the original input image (i.e., the image data to be processed) predicted by the model and the clear image (i.e., the denoised image). Specifically, the parameter values ​​(such as grayscale values ​​or color components) of the corresponding pixels in the image data to be processed can be adjusted based on the residual image to achieve image denoising and obtain a denoised image.

[0069] As can be seen from the above, the image processing method provided by the embodiment of the present application extracts multi-scale image features from the image data to be processed to obtain a multi-scale feature map, then fuses the multi-scale feature map to obtain a fused feature map, and predicts the denoised image and detail image corresponding to the image data to be processed based on the fused feature map, respectively, and obtains two result outputs of the denoised image and detail image based on the original input. In this solution, by fusing multi-scale features, global features and local detail features are combined to more accurately describe the image, and finally the denoised image and detail image are output respectively, achieving the effect of single input and multiple outputs, which can realize both the image denoising function and the image detail enhancement function, and improve the image processing effect.

[0070] In the embodiment of the present application, the image denoising network model is deployed locally on the electronic device. Before using the image denoising network model for denoising, it is necessary to use deep learning to enable the image denoising network model to learn the noise distribution and texture structure in the image, and to pre-train the image denoising network model so that it has the ability to intelligently denoise and identify image details. That is, before obtaining the image data to be processed, the following process is also included:

[0071] Construct a basic image denoising network model;

[0072] Constructing a first training set and a second training set, wherein the first training set includes: an image pair consisting of a sample original image and a corresponding sample denoised image, and the second training set includes: an image pair consisting of a sample original image and a corresponding sample detail image;

[0073] Based on the first training set and the second training set, training the basic image denoising network model;

[0074] The model parameters of the basic image denoising network model are adjusted until the basic network denoising model converges to obtain a trained image denoising network model.

[0075] Among them, the basic image denoising network model can be a residual network model based on the U-Net structure. In addition, the basic image denoising network model can also be a U-Net neural network model with a res-block residual structure as the basic structure, a residual network model with a Transformer as the basic structure, etc. The basic structure of the network model can be expanded arbitrarily, and the type of network structure to be used can ultimately be determined by the effect of model training.

[0076] In the embodiment of the present application, the image denoising network model has multiple outputs. Therefore, it is necessary to construct a data set with the real data (Ground Truth, GT) corresponding to the denoising result (i.e., denoised image) and the image details (i.e., detail image) to train the model.

[0077] In one embodiment, the data set of the noise reduction result can be obtained by using an electronic device to actually collect multiple frames of original image data, superimpose multiple frames of original image data, and perform multi-frame image fusion in a static scene (for example, calculating the mean of each pixel based on multiple frames of images) to generate an image without noise or with less noise as the noise reduction GT data of the original image data. The noise reduction GT data of the original image data and the original image data constitute an image pair and are added to the data set.

[0078] In this embodiment, there are multiple ways to construct the data set corresponding to the detail image. In one implementation, a filter operator can be directly used on the above-mentioned denoised GT data to extract image details and obtain the corresponding detail GT data. That is, it is generated according to the following formula:

[0079] GT Detail = raw GT * filter

[0080] Among them, raw GT represents denoised GT data; filter represents a high-frequency filter, such as a Laplace filter; GT Detail represents the extracted detail GT data.

[0081] In one embodiment, a large model can also be used for training on the original image data set. In this case, the large model has the ability to reduce the noise of this type of original image data. The large model is used to process the noise data, and then the denoised GT data is Gaussian blurred to smooth the image. The output result of the large model and the result of Gaussian blur are used to calculate the detailed GT data, which can be generated according to the following formula:

[0082] GT Detail=Big model(raw noise)-gauss Blur(raw GT)

[0083] Among them, Big model (raw noise) means the output result after inputting the noisy image into the big model; Gauss Blur (raw GT) means the result of Gaussian blurring the denoised GT data. The output result after the model is subtracted from the result of Gaussian blurring to finally obtain the detailed GT data.

[0084] In the embodiment of the present application, since both the denoised image and the detail image are used as the output of the image denoising network model, the model output needs to be constrained based on the loss of the denoised image and the detail image during model training. That is, in some embodiments, the model parameters of the basic image denoising network model are adjusted until the basic network denoising model converges to obtain a trained image denoising network model, including:

[0085] Obtaining a first loss between a predicted denoised image output by a basic image denoising network based on a sample original image and the corresponding sample denoised image;

[0086] Obtain a second loss between a predicted detail image output by a basic image denoising network based on a sample original image and a corresponding sample detail image;

[0087] Based on the first loss and the second loss, the model parameters of the basic image denoising network model are updated until the basic network denoising model converges, thereby obtaining a trained image denoising network model.

[0088] Specifically, the multi-output loss function needs to constrain the denoised image and the detail image at the same time. In some implementations, when the model parameters of the basic image denoising network model are updated based on the first loss and the second loss until the basic network denoising model converges and the trained image denoising network model is obtained, the following process may be specifically included:

[0089] Construct a target loss function based on the first loss and the second loss;

[0090] According to the target loss function, the output of the basic image denoising network model is constrained;

[0091] Update the model parameters of the basic image denoising network model until the target loss function converges, and determine the model parameters of the basic image denoising network model obtained by the last update as the model parameters of the trained image denoising network model. The target loss function is expressed as follows:

[0092] Loss=a1*Loss1+a2*Loss2

[0093] Among them, Loss1 represents the first loss, Loss2 represents the second loss, and a1 and a2 are constants. Loss1 constrains the denoised image, and Loss2 constrains the output detail image. Loss is the total loss of Loss1 and Loss2. Certain coefficients can be set for each of them to represent their contribution to the total loss. Among them, a1 is the coefficient of the denoised image loss, and a2 is the coefficient of the detail image loss. In specific implementation, a1=1, a2=1, or a better coefficient setting combination in the experiment can be set.

[0094] In one implementation, the absolute value loss L1 and the multi-scale similarity loss MS-SSIM may be combined to construct the first loss Loss1 and the second loss Loss2, as shown in the following formula:

[0095] Loss 1=L1(GT-model out)+MS-SSIM(GT-model out)

[0096] Loss 2=L1(GT detail-model out detail)+MS-SSIM(GT detail-model outdetail)

[0097] Among them, GT represents the real denoised image, model out represents the predicted denoised image output by the model, L1 (GT-model out) represents the absolute value of the difference between the real denoised image and the predicted denoised image, and MS-SSIM (GT-modelout) represents the calculation of multi-scale similarity between the real denoised image and the predicted denoised image; GT detail represents the real detail image, model out detail represents the predicted detail image output by the model, L1 (GT detail-model out detail) represents the absolute value of the difference between the real detail image and the predicted detail image, and MS-SSIM (GT detail-modelout detail) represents the calculation of multi-scale similarity between the real detail image and the predicted detail image.

[0098] After the loss function and training set are constructed, the model is trained using the constructed training set. During training, the model has many hyperparameters, such as the number of training epochs, learning rate, optimizer Adam, etc. Hyperparameters can be adjusted during training, and these hyperparameters can be debugged from the perspectives of model convergence speed and model convergence effect.

[0099] For example, in the early stage of model training, a larger learning rate will be used for model optimization. As the number of iterations increases, the learning rate will gradually decrease to ensure that the model will not fluctuate too much in the later stage of training, thus getting closer to the optimal solution. The learning rate will interact with many other aspects of the optimization process, and the interaction can be nonlinear. However, in general, a smaller learning rate will require more training epochs. Conversely, a larger learning rate will require less training time. In addition, considering the noisy estimate of the error gradient, a smaller batch size is more suitable for a smaller learning rate.

[0100] refer to Figure 4 In some embodiments, the trained model can be a neural network model based on U-Net, the input of the model is the image to be processed, the multi-output function of the model is implemented inside the model, and the multiple outputs of the model are directly fused internally to obtain the final output of the model. Specifically, a multi-layer convolution layer is connected after the U-Net structure, one output of the last convolution layer is a residual image (i.e., noise), and the other output is a detail image. In practical applications, the last convolution layer can be an 8-channel convolution layer, and a split layer (split layer) (not shown) can be used to split the feature map according to the dimension of its channel number, and a 4-channel residual image and a 4-channel detail image are separated from the feature map. In specific implementation, the residual image and the original image can be fused first to perform image denoising to obtain a denoised image; after the detail image is fused with the denoised image, finally, the output of the model is the result after denoising and detail enhancement. That is, after respectively predicting the denoised image and detail image corresponding to the image data to be processed based on the fused feature map, it also includes:

[0101] The denoised image is subjected to detail enhancement processing based on the detail image to obtain a detail-enhanced denoised image.

[0102] In this embodiment, multiple outputs within the model are described, that is, both the denoised image and the detail image are output within the model, and the detail enhancement processing of the image is completed within the model. The final output result of the model is only one, that is, the denoised image with enhanced details. The formula is as follows:

[0103] model out=denoise out+detail

[0104] denoise out=res+input

[0105] Wherein, model out represents the final output result of the model, denoise out represents the denoised image outputted from the model, detail represents the detail image outputted from the model, res represents the residual image outputted from the model, and input represents the input image of the model. In this implementation, the final output result of the model, model out, is obtained by adding denoise out and detail; the denoised image denoise out is obtained by adding the residual image res and the input image input.

[0106] In practical applications, the images captured in different shooting scenes will be different, and the corresponding image clarity will also be different. For example, under the same other conditions, the original image directly captured by the image sensor in a bright environment scene is clearer, and the original image directly captured by the image sensor in a dark environment scene is blurry. Therefore, different detail adjustment parameters can be set for different scenes, and detail enhancement can be adaptively performed according to actual conditions. That is, in one embodiment, detail enhancement processing of the denoised image based on the detail image can include the following process:

[0107] Obtaining shooting scene information corresponding to the image data to be processed;

[0108] Determine corresponding adjustment parameters according to shooting scene information;

[0109] According to the adjustment parameters and the detail image, the denoised image is subjected to detail enhancement processing.

[0110] Specifically, the shooting scene is analyzed based on the information obtained from the analysis, such as scene brightness, noise, etc. For example, daytime or nighttime, high noise or low noise, dark environment or bright environment, etc. The corresponding required adjustment parameters are determined based on the analysis results. In specific implementation, the noise adjustment parameter can be a noise adjustment ratio, that is, a weighting coefficient. As shown in the following formula:

[0111] out=model out1+model out2*α

[0112] Among them, α represents the adjustment parameter, model out1 represents the denoised image, model out2 represents the detail image, and out represents the denoised image with detail enhancement. Specifically, the detail enhancement strength can be adjusted in different scenarios, and adjustable functions can be added to improve the flexibility and adjustability of the algorithm.

[0113] refer to Figure 5In some embodiments, the trained model can be a neural network model based on U-Net, the input of the model is the image to be processed, the multi-output function of the model is implemented inside the model, and the multiple outputs of the model are directly subjected to image fusion processing inside to obtain the final output of the model. Specifically, a multi-layer convolution layer is connected after the U-Net structure, and two convolution layers are connected after the multi-layer convolution layer, one convolution layer outputs a residual image (i.e., noise), and the other convolution layer outputs a detail image. Then, the original image (i.e., the image to be processed), the residual image, and the detail image output by the convolution layer are image fused to obtain the final image (i.e., a denoised image with enhanced details). In specific implementation, the residual image and the original image can be fused first to perform image denoising processing to obtain a denoised image; then the detail image and the denoised image are fused, and finally, the output of the model is the result after denoising and detail enhancement (i.e., a denoised image with enhanced details).

[0114] refer to Figure 6 In some embodiments, the trained model can be a neural network model based on U-Net, the input of the model is the image to be processed, and the output of the model is a denoised image and a detail image. Specifically, a multi-layer convolution layer is connected after the U-Net structure, and two convolution layers are connected after the multi-layer convolution layer, one convolution layer outputs a residual image (i.e., noise), and the other convolution layer outputs a detail image. In specific implementation, the residual image and the original image can be fused to perform image denoising to obtain a denoised image, and the denoised image and the detail image are used as the final output of the model.

[0115] refer to Figure 7 In some embodiments, the trained image denoising network model can be a residual network model based on an encoder-decoder structure (such as a residual network model based on U-Net), including an encoder (i.e., a model front end), a first decoder branch (i.e., a model back end 1), and a second decoder branch (model back end 1). Among them, the encoder part may include multiple convolutional layers, downsampling layers, etc., for encoding the input image, obtaining multi-scale image features, and obtaining a multi-scale feature map. The first decoder branch may include an upsampling layer, a multi-convolutional layer, and a residual structure, etc., for rescaling the feature map, obtaining a fused feature map that integrates multi-scale features, and predicting a residual image based on the fused feature map, and finally using the residual structure to perform denoising on the image to be processed, and outputting a denoised image of the image to be processed (i.e., output 1). The second decoder branch may include an upsampling layer, a multi-convolutional layer, etc., for rescaling the feature map, obtaining a fused feature map that integrates multi-scale features, and predicting image details based on the fused feature map, and finally outputting a detail image of the image to be processed (i.e., output 2).

[0116] When the fused feature map is processed based on the trained image denoising network model and the denoised image and detail image corresponding to the image data to be processed are output respectively, the fused feature map can be processed by the first decoder branch to output the denoised image corresponding to the image data to be processed; and the fused feature map can be processed by the second decoder branch to output the detail image corresponding to the image data to be processed.

[0117] In the example of the present application, the use of detail images can also be applied by subsequent hardware devices, such as time domain noise reduction algorithms and detail enhancement algorithms. That is, in one embodiment, the electronic device includes an image signal processor; after respectively predicting the noise reduction image and detail image corresponding to the image data to be processed based on the fused feature map, the following process can also be included:

[0118] According to the detail image, determine the region classification corresponding to the position of each pixel point in the denoised image;

[0119] Based on the region classification, an image signal processor is called to perform image noise reduction processing and / or detail enhancement processing on the noise-reduced image.

[0120] Among them, the region classification can include smooth regions and texture regions. The image denoising network model will output a detail image, which can be used by various hardware modules in the subsequent image signal processor to guide noise reduction or detail enhancement to improve the image processing effect. For example, the image signal processor is called to perform image denoising on the flat area in the detail-enhanced denoised image and to perform detail enhancement on the texture area in the detail-enhanced denoised image to obtain the final image.

[0121] From the above, it can be seen that the embodiment of the present application provides a multi-output image denoising network model, which can realize both the image denoising function and the image detail enhancement function. After the original image data has been processed by the model for denoising, it also has the function of detail enhancement, which can ensure that the image output by the model is clear and improve its possibility of application in mobile terminals. In addition, since the image denoising network model proposed in the present technical solution is located before the image signal processor, the image denoising network model has already undertaken most of the denoising work, and the processing difficulty faced by the image signal processor will be smaller. In addition, the image denoising network model will output a detail image to the outside, and the detail image can be used by various hardware modules in the subsequent image signal processor to guide noise reduction or guide detail enhancement to improve the image processing effect.

[0122] According to the image processing method provided by the above embodiment, the image processing method provided by this embodiment is further described in detail below by taking the image to be processed as the original image, the imaging information as the sensitivity, and the image processing device integrated in the electronic device as an example.

[0123] Please refer to Figure 8 and Fig. 9 , the process of the image processing method can be specifically as follows:

[0124] In step 301, the electronic device collects raw image data of the current shooting scene through an image sensor, and obtains shooting scene information when the raw image data is shot.

[0125] In step 302, the electronic device obtains a trained image denoising network model, and inputs the original image data into the trained image denoising network model to obtain a denoised image and a detail image output by the model.

[0126] In step 303, the electronic device determines adjustment parameters according to the shooting scene information.

[0127] In step 304, the electronic device performs detail enhancement processing on the denoised image according to the adjustment parameters and the detail image to obtain a detail-enhanced denoised image.

[0128] In step 305, the electronic device determines a flat area and a texture area in the detail-enhanced denoised image according to the detail image.

[0129] In step 306, the electronic device calls the image signal processor to perform image noise reduction processing on the flat area in the detail-enhanced denoised image and to perform detail enhancement processing on the texture area in the detail-enhanced denoised image to obtain a final image.

[0130] For example, refer to Fig. 9 The electronic device collects image data through an image sensor to obtain raw image data in raw format. Subsequently, the image sensor transmits the collected raw data in raw format to the image denoising network model, and the image denoising network model performs denoising processing and detail output on the input raw image data in raw format to obtain denoised image data in raw format and detail image data in raw format. Then, the image denoising network model transmits the denoised image data in raw format and detail image data in raw format obtained after processing to the image signal processor. When the image signal processor performs data processing, it can process the denoised image data and detail image data pixel by pixel according to the received format. For example, each image pixel can have a bit depth of 8, 10, 12 or 14 bits, and the image signal processor can perform one or more image processing operations (such as image denoising, image sharpening, etc.) on the denoised image in raw format, collect statistical information about the image data, and the image processing operation can be performed at the same or different bit depth precision, and finally output the image in YUV or RGB format.

[0131] It should be noted that for the technical details of the above steps, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0132] The embodiment of the present application provides a multi-output image denoising network model, which can realize both image denoising function and image detail enhancement function. After the original image data is processed by the model for denoising, it also has the function of detail enhancement, which can ensure the clarity of the image output by the model and improve its possibility of application in mobile terminals. In addition, since the image denoising network model proposed in the present technical solution is located before the image signal processor, the image denoising network model has undertaken most of the denoising work, and the image signal processor will face less processing difficulty. The detail image output by the image denoising network model can assist in identifying and distinguishing texture areas and smooth areas, and can improve the resolution ability of the subsequent hardware device demosaicing algorithm for details and noise. Finally, the RGB image output by the demosaicing algorithm has both better signal-to-noise ratio and better details.

[0133] In addition, the image denoising network model will output a detail image, which can be used by various hardware modules in the subsequent image signal processor to guide noise reduction or detail enhancement and improve image processing effects.

[0134] In order to facilitate better implementation of the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing device based on the above image processing method. The image processing device can be integrated in an electronic device provided with an image signal processor, such as a terminal device, and the terminal device may include a tablet computer, a laptop computer and / or a personal computer. The meaning of the terms is the same as in the above image processing method. For specific implementation details, please refer to the description in the above method embodiment.

[0135] Please refer to Fig.10 , Fig.10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may include an acquisition unit 401, an extraction unit 402, a fusion unit 403, and a processing unit 404, wherein:

[0136] An acquisition unit 401 is used to acquire image data to be processed;

[0137] An extraction unit 402 is used to extract multi-scale image features from the image data to be processed to obtain a multi-scale feature map;

[0138] A fusion unit 403 is used to perform feature fusion on the multi-scale feature map to obtain a fused feature map;

[0139] The processing unit 404 is used to predict the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map.

[0140] In some implementations, the processing unit 404 is configured to:

[0141] Based on the fused feature map, respectively predicting a residual image and a detail image corresponding to the image data to be processed;

[0142] The image data to be processed is subjected to image denoising processing based on the residual image to obtain the denoised image.

[0143] In some implementations, the processing unit 404 is configured to:

[0144] Get the trained image denoising network model;

[0145] The fused feature map is processed based on the trained image denoising network model, and a denoised image and a detail image corresponding to the image data to be processed are output respectively.

[0146] In some embodiments, the image processing apparatus further comprises:

[0147] A model building unit, used to build a basic image denoising network model before acquiring the image data to be processed;

[0148] A data construction unit, used to construct a first training set and a second training set, wherein the first training set includes: an image pair consisting of a sample original image and a corresponding sample denoised image, and the second training set includes: an image pair consisting of a sample original image and a corresponding sample detail image;

[0149] A training unit, used for training the basic image denoising network model based on the first training set and the second training set;

[0150] The adjustment unit is used to adjust the model parameters of the basic image denoising network model until the basic network denoising model converges to obtain a trained image denoising network model.

[0151] In some embodiments, the adjustment unit is configured to:

[0152] Obtaining a first loss between a predicted denoised image output by the basic image denoising network based on the sample original image and the corresponding sample denoised image;

[0153] Obtaining a second loss between a predicted detail image output by the basic image denoising network based on the sample original image and the corresponding sample detail image;

[0154] Based on the first loss and the second loss, the model parameters of the basic image denoising network model are updated until the basic network denoising model converges to obtain a trained image denoising network model.

[0155] In some embodiments, when updating the model parameters of the basic image denoising network model based on the first loss and the second loss until the basic network denoising model converges to obtain a trained image denoising network model, the adjustment unit is further used to:

[0156] Constructing a target loss function based on the first loss and the second loss;

[0157] According to the target loss function, constraining the output of the model of the basic image denoising network model;

[0158] The model parameters of the basic image denoising network model are updated until the target loss function converges, and the model parameters of the basic image denoising network model obtained by the last update are determined as the model parameters of the trained image denoising network model. The target loss function is expressed as follows:

[0159] Loss=a1*Loss1+a2*Loss2

[0160] Among them, Loss1 represents the first loss, Loss2 represents the second loss, and a1 and a2 are constants.

[0161] In some embodiments, the trained image denoising network model is a residual network model based on U-Net, including an encoder, a first decoding branch, and a second decoding branch; when the fused feature map is processed based on the trained image denoising network model and the denoised image and the detail image corresponding to the image data to be processed are output respectively, the processing unit 404 is specifically used to:

[0162] The fused feature map is processed by the first decoding branch to output a denoised image corresponding to the image data to be processed;

[0163] The fused feature map is processed by the second decoding branch to output a detail image corresponding to the image data to be processed.

[0164] In some embodiments, the image processing apparatus further comprises:

[0165] The detail enhancement unit is used to respectively predict the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map, and then perform detail enhancement processing on the denoised image based on the detail image to obtain a detail-enhanced denoised image.

[0166] In some implementations, the detail enhancement unit is specifically configured to:

[0167] Obtaining shooting scene information corresponding to the image data to be processed;

[0168] Determine corresponding adjustment parameters according to the shooting scene information;

[0169] According to the adjustment parameter and the detail image, detail enhancement processing is performed on the denoised image.

[0170] In some embodiments, the electronic device includes an image signal processor; the image processing device further includes:

[0171] A classification unit, configured to respectively predict a denoised image and a detail image corresponding to the image data to be processed based on the fused feature map, and then determine a region classification corresponding to a position of each pixel point in the denoised image according to the detail image;

[0172] The calling unit is used to call the image signal processor to perform image noise reduction processing and / or detail enhancement processing on the noise reduction image based on the region classification.

[0173] In some embodiments, the fusion unit 402 is used to:

[0174] Based on the attention mechanism, the multi-scale feature map is fused to obtain the fused feature map.

[0175] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0176] The image processing device provided in the embodiment of the present application fuses multi-scale features, combines global features and local detail features to more accurately describe the image, and finally outputs a denoised image and a detail image respectively, achieving a single-input multiple-output effect, which can realize both image denoising function and image detail enhancement function, thereby improving the image processing effect.

[0177] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps of the image processing method provided in this embodiment by calling a computer program stored in the memory.

[0178] Please refer to Fig.11 , Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0179] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, an input unit 504, and an image signal processor 505. Those skilled in the art will appreciate that Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0180] The processor 501 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, the processor 501 performs various functions of the electronic device and processes data. Optionally, the processor 501 may include one or more processing cores; optionally, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0181] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0182] The electronic device also includes a power supply 503 for supplying power to each component. Optionally, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 503 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0183] The electronic device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0184] The electronic device may also include an image signal processor 505, which is logically connected to the processor 501 and can be used to match image sensors with different performances, process image signals at high speed, and realize a high-performance image processing engine for high-definition images. In the embodiment of the present application, the image signal processor 505 can be used to perform image noise reduction or image detail enhancement processing on the image.

[0185] Although not shown, the electronic device may also include a display unit, an image acquisition component (such as an image sensor), etc., which will not be described in detail here. Specifically in this embodiment, the processor 501 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 502 according to the following instructions, and the processor 501 will execute the steps in the image processing method provided in this application, such as:

[0186] Obtain image data to be processed;

[0187] Extracting multi-scale image features from the image data to be processed to obtain a multi-scale feature map;

[0188] Perform feature fusion on the multi-scale feature map to obtain a fused feature map;

[0189] Based on the fused feature map, a denoised image and a detail image corresponding to the image data to be processed are predicted respectively.

[0190] The electronic device provided in the embodiment of the present application fuses multi-scale features, combines global features and local detail features to more accurately describe the image, and finally outputs a denoised image and a detail image respectively, achieving a single-input multiple-output effect, which can realize both image denoising function and image detail enhancement function, thereby improving the image processing effect.

[0191] It should be noted that the electronic device provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0192] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the image processing method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0193] The above is a detailed introduction to an image processing method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0194] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: Obtain image data to be processed; Extracting multi-scale image features from the image data to be processed to obtain a multi-scale feature map; Performing feature fusion on the multi-scale feature map to obtain a fused feature map; Based on the fused feature map, a denoised image and a detail image corresponding to the image data to be processed are predicted respectively.

2. The image processing method according to claim 1, characterized in that: The step of respectively predicting the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map comprises: Based on the fused feature map, respectively predicting a residual image and a detail image corresponding to the image data to be processed; The image data to be processed is subjected to image denoising processing based on the residual image to obtain the denoised image.

3. The image processing method according to claim 1, characterized in that: The step of respectively predicting the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map comprises: Get the trained image denoising network model; The fused feature map is processed based on the trained image denoising network model, and a denoised image and a detail image corresponding to the image data to be processed are output respectively.

4. The image processing method according to claim 3, characterized in that: Before obtaining the image data to be processed, it also includes: Construct a basic image denoising network model; Constructing a first training set and a second training set, wherein the first training set includes: an image pair consisting of a sample original image and a corresponding sample denoised image, and the second training set includes: an image pair consisting of a sample original image and a corresponding sample detail image; Based on the first training set and the second training set, training the basic image denoising network model; The model parameters of the basic image denoising network model are adjusted until the basic image denoising network model converges to obtain the trained image denoising network model.

5. The image processing method according to claim 4, characterized in that: The step of adjusting the model parameters of the basic image denoising network model until the basic image denoising network model converges to obtain the trained image denoising network model includes: Obtaining a first loss between a predicted denoised image output by the basic image denoising network based on the sample original image and the corresponding sample denoised image; Obtaining a second loss between a predicted detail image output by the basic image denoising network based on the sample original image and the corresponding sample detail image; Based on the first loss and the second loss, the model parameters of the basic image denoising network model are updated until the basic image denoising network model converges, thereby obtaining the trained image denoising network model.

6. The image processing method according to claim 5, characterized in that: Based on the first loss and the second loss, updating the model parameters of the basic image denoising network model until the basic network denoising model converges, to obtain the trained image denoising network model, including: Constructing a target loss function based on the first loss and the second loss; According to the target loss function, constraining the model output of the basic image denoising network model; The model parameters of the basic image denoising network model are updated until the target loss function converges, and the model parameters of the basic image denoising network model obtained by the last update are determined as the model parameters of the trained image denoising network model. The target loss function is expressed as follows: Loss=a1*Loss1+a2*Loss2 Among them, Loss1 represents the first loss, Loss2 represents the second loss, and a1 and a2 are constants.

7. The image processing method according to claim 3, characterized in that: The trained image denoising network model is a residual network model based on an encoder-decoder architecture, including an encoder, a first decoding branch and a second decoding branch; the fused feature map is processed based on the trained image denoising network model, and a denoised image and a detail image corresponding to the image data to be processed are outputted respectively, including: Processing the fused feature map through the first decoding branch, and outputting a denoised image corresponding to the image data to be processed; The fused feature map is processed by the second decoding branch to output a detail image corresponding to the image data to be processed.

8. The image processing method according to claim 1, characterized in that: After respectively predicting the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map, the method further includes: The denoised image is subjected to detail enhancement processing based on the detail image to obtain a detail-enhanced denoised image.

9. The image processing method according to claim 8, characterized in that: The performing detail enhancement processing on the denoised image based on the detail image comprises: Obtaining shooting scene information corresponding to the image data to be processed; Determining corresponding adjustment parameters according to the shooting scene information; According to the adjustment parameters and the detail image, detail enhancement processing is performed on the denoised image.

10. The image processing method according to claim 1, characterized in that: The electronic device includes an image signal processor; after respectively predicting a denoised image and a detail image corresponding to the image data to be processed based on the fused feature map, the electronic device further includes: Determining, according to the detail image, a region classification corresponding to a position of each pixel point in the denoised image; Based on the region classification, the image signal processor is called to perform image noise reduction processing and / or detail enhancement processing on the denoised image.

11. The image processing method according to any one of claims 1 to 10, characterized in that: The step of fusing the multi-scale feature maps to obtain a fused feature map includes: Based on the attention mechanism, the multi-scale feature map is subjected to feature fusion to obtain the fused feature map.

12. An image processing device, applied to an electronic device, characterized in that: The device comprises: An acquisition unit, used for acquiring image data to be processed; An extraction unit, used for extracting multi-scale image features from the image data to be processed to obtain a multi-scale feature map; A fusion unit, used for performing feature fusion on the multi-scale feature map to obtain a fused feature map; A processing unit is used to predict the denoised image and the detail image corresponding to the image data to be processed based on the fused feature map.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 11.

Images