Image processing method, device, electronic device and computer readable storage medium

Abstract

The present application relates to an image processing method, device, electronic device and storage medium. The method comprises: obtaining a first image, and obtaining a corresponding first brightness channel map based on the first image; performing noise reduction processing on the first image to obtain a second image, and obtaining a corresponding second brightness channel map based on the second image; determining a brightness channel texture map based on a difference map between the first brightness channel map and the second brightness channel map; and generating a target image based on the brightness channel texture map and the second image. The method can retain more image texture information while reducing noise.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, more and more electronic devices are equipped with image sensors, which can capture various types of images, such as YUV (Y represents brightness, UV represents chrominance) images, RGB (Red, Green, Blue) images, etc. Usually, a lot of noise will remain in the image, affecting the overall perception of the image, so it is necessary to reduce the noise of the image.

[0003] Traditional methods of image denoising include spatial domain filtering (such as bilateral filtering, non-local mean algorithm, etc.), frequency domain denoising (wavelet, discrete cosine transform, etc.), etc. However, after denoising an image using traditional image processing methods, the generated image cannot retain texture information, and there are problems such as over-smearing and loss of texture information. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can retain more image texture information while reducing noise and improve the accuracy of image processing.

[0005] An image processing method, comprising:

[0006] Acquire a first image, and acquire a corresponding first brightness channel map based on the first image;

[0007] Performing noise reduction processing on the first image to obtain a second image, and acquiring a corresponding second brightness channel image based on the second image;

[0008] Determine a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map;

[0009] A target image is generated based on the brightness channel texture map and the second image.

[0010] An image processing device, comprising:

[0011] An acquisition module, used to acquire a first image, and acquire a corresponding first brightness channel map based on the first image;

[0012] a noise reduction module, configured to perform noise reduction processing on the first image to obtain a second image, and acquire a corresponding second brightness channel image based on the second image;

[0013] A determination module, configured to determine a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map;

[0014] An image generation module is used to generate a target image based on the brightness channel texture map and the second image.

[0015] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image processing method.

[0016] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0017] The above-mentioned image processing method, device, electronic device, and computer-readable storage medium obtain a first image, and obtain a corresponding first brightness channel map based on the first image; perform noise reduction processing on the first image to obtain a second image after noise reduction, and obtain a corresponding second brightness channel map after noise reduction based on the second image. It can be understood that the first brightness channel map is a channel map representing brightness that has not been subjected to noise reduction processing, and contains more high-frequency texture information. Then, based on the difference map between the first brightness channel map and the second brightness channel map, a brightness channel texture map can be determined, and the brightness channel texture map represents the high-frequency texture information erased by the noise reduction processing in the first brightness channel map. Then, based on the brightness channel texture map and the second image, a target image that has been subjected to noise reduction processing and retains more high-frequency texture information can be generated, thereby improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flowchart of an image processing method in one embodiment;

[0020] Figure 2 A schematic diagram of constructing a training pair in one embodiment;

[0021] Figure 3 is a schematic diagram of a lens shadow phenomenon in one embodiment;

[0022] Figure 4 A schematic diagram of obtaining a YUV type label image in one embodiment;

[0023] Figure 5 A flowchart of the steps of determining a target loss value for a second image relative to a label image in one embodiment;

[0024] Figure 6 is a comparison diagram before and after low-pass filtering of an image in one embodiment;

[0025] Figure 7 is a flowchart of an image processing method in another embodiment;

[0026] Figure 8 A flowchart of the steps of performing noise reduction processing on a first image to obtain a second image in one embodiment;

[0027] Fig. 9 is a flowchart of an image processing method in another embodiment;

[0028] Fig.10 is a flowchart of an image processing method in another embodiment;

[0029] Fig.11 is a flowchart of an image processing method in another embodiment;

[0030] Fig.12 is a schematic diagram of a downsampling module in one embodiment;

[0031] Fig.13 is a schematic diagram of an upsampling module in one embodiment;

[0032] Fig.14 is a schematic diagram of a global average pooling module in one embodiment;

[0033] Fig.15 is a schematic diagram of a non-local module in one embodiment;

[0034] Fig.16 is a schematic diagram of an image processing method in another embodiment;

[0035] Fig.17 A flowchart of the steps of performing noise reduction processing on a first image to obtain a second image in another embodiment;

[0036] Fig.18 is a structural block diagram of an image processing device in one embodiment;

[0037] Fig.19 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client. Both the first client and the second client are clients, but they are not the same client.

[0040] In one embodiment, Figure 1 As shown, an image processing method is provided, and this embodiment is illustrated by applying the method to an electronic device. Among them, the electronic device can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server can be implemented as an independent server or a server cluster composed of multiple servers. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0041] Step 102: Acquire a first image, and acquire a corresponding first brightness channel map based on the first image.

[0042] The first image refers to an image acquired by an electronic device for image processing. In one embodiment, the color space type of the first image is YUV, where Y represents the brightness (Luminance or Luma) channel of the first image, and UV represents the color (Chrominance or Chroma) channel of the first image. In another embodiment, the color space type of the first image may be RGB (Red, Green, Blue). In other embodiments, the color space type of the first image may also be CMY (Cyan, Magenta, Yellow) or HSV (Hue, Saturation, Value), etc.

[0043] The electronic device acquires a first image, detects an original color space type of the first image, and if the original color space type is a non-target color space type, converts the original color space type of the first image into a target color space type; if the original color space type is a target color space type, acquires a corresponding first brightness channel map based on the first image.

[0044] The color space type refers to the type of the color system of the image. The color space type may include at least YUV, RGB, CMY, and HSV. The original color space type is the original color space type of the first image. The target color space type is the color space type expected by the first image.

[0045] If the original color space type is not the target color space type, the electronic device calls the color space conversion model to convert the original color space type into the target color space type.

[0046] For example, an electronic device obtains a first image, the original color space type of the first image is RGB, and the target color space type is YUV, that is, the original color space type RGB of the first image is a non-target color space type YUV, then the color space conversion model is called to convert the original color space type RGB of the first image into YUV.

[0047] For another example, the electronic device obtains a first image, the original color space type of the first image is YUV, and the target color space type is also YUV, then performs encoding processing on the first image to obtain a first feature map.

[0048] The first brightness channel map is a channel map representing brightness in the first image. The first image also includes a first color channel map, which is a channel map representing color in the first image. If the first image is a YUV image, the brightness channel map corresponding to the first image is the Y channel map, and the color channel map corresponding to the first image is the UV channel map.

[0049] In one embodiment, the electronic device may extract a first brightness channel map representing brightness and a first color channel map representing color from the first image. In another embodiment, the electronic device may interpolate the first brightness channel map representing brightness and the first color channel map representing color based on each pixel in the first image. In other embodiments, the electronic device may also generate the first brightness channel map and the first color channel map in other ways, which are not limited here.

[0050] Step 104: perform noise reduction processing on the first image to obtain a second image, and obtain a corresponding second brightness channel image based on the second image.

[0051] The second image refers to an image obtained by denoising the first image. The second brightness channel map is a channel map representing brightness in the second image. The second image also includes a second color channel map, which is a channel map representing color in the second image. If the second image is a YUV image, the brightness channel map corresponding to the second image is the Y channel map, and the color channel map corresponding to the second image is the UV channel map.

[0052] The electronic device performs noise reduction processing on the first image, which may specifically be filtering processing, upsampling / downsampling processing or cropping processing, etc., which is not limited here.

[0053] In one embodiment, the electronic device may extract a second brightness channel map representing brightness and a second color channel map representing color from the second image. In another embodiment, the electronic device may interpolate a second brightness channel map representing brightness and a second color channel map representing color based on each pixel in the second image. In other embodiments, the electronic device may also generate the second brightness channel map and the second color channel map in other ways, which are not limited here.

[0054] Step 106: Determine a brightness channel texture map according to the difference map between the first brightness channel map and the second brightness channel map.

[0055] The difference map is an image obtained by performing difference processing on the first brightness channel map and the second brightness channel map. The brightness channel texture map is an image in which the first brightness channel map corresponding to the first image represents high-frequency texture information.

[0056] It is understandable that the brightness channel map includes high-frequency texture information of the image, which can determine the details of the image, while the color channel map mainly includes color information. When the electronic device performs noise reduction processing on the first image, the high-frequency information in the first brightness channel is erased. Therefore, the high-frequency texture information in the erased image, i.e., the brightness channel texture map, is determined based on the difference map between the first brightness channel map and the second brightness channel map.

[0057] Specifically, the electronic device removes brightness noise in the difference map according to the difference map between the first brightness channel map and the second brightness channel map, extracts high-frequency texture information in the difference map, and thus generates a brightness channel texture map.

[0058] Step 108: Generate a target image based on the brightness channel texture map and the second image.

[0059] The target image is the final generated image.

[0060] Specifically, the electronic device fuses the brightness channel texture map and the second brightness channel map corresponding to the second image to obtain a target brightness channel map, and then generates a target image based on the target brightness channel map and the second color channel map corresponding to the second image.

[0061] The above-mentioned image processing method obtains a first image, and obtains a corresponding first brightness channel map based on the first image; performs noise reduction processing on the first image to obtain a second image after noise reduction, and obtains a corresponding second brightness channel map after noise reduction based on the second image. It can be understood that the first brightness channel map is a channel map representing brightness that has not been subjected to noise reduction processing, and contains more high-frequency texture information. Then, based on the difference map between the first brightness channel map and the second brightness channel map, a brightness channel texture map can be determined, and the brightness channel texture map represents the high-frequency texture information that is erased by the noise reduction processing in the first brightness channel map. Then, based on the brightness channel texture map and the second image, a target image that has been subjected to noise reduction processing and retains more high-frequency texture information can be generated, thereby improving the accuracy of image processing.

[0062] In one embodiment, when an electronic device captures motion images, the above-mentioned image processing method is used to process the motion captured images to generate a target image, which can effectively remove noise in the short-exposure image while retaining more image texture information and edge information.

[0063] In one embodiment, obtaining a first image includes: obtaining a first number of single-frame images and a second number of fused images; wherein the fused image is an image obtained by fusing multiple single-frame images; constructing an input image set based on the first number of single-frame images and the second number of fused images; and obtaining the first image from each image included in the input image set.

[0064] A single-frame image refers to an initial frame image. A fused image is an image obtained by fusing multiple single-frame images. The first number is the number of single-frame images. The second number is the number of fused images. The input image set is a set that stores single-frame images and fused images, and is also a set used to obtain the first image input to the denoising network.

[0065] The image color types of the above-mentioned single-frame images, fused images and first images may all be YUV. The electronic device directly fuses multiple single-frame images of the YUV type to obtain a fused image, or first converts multiple single-frame images of the YUV type into single-frame images of the RGB type, averages multiple single-frame images of the RGB type to obtain a fused image of the RGB type, and then converts the fused image of the RGB type into a fused image of the YUV type.

[0066] The electronic device obtains an initial image set, selects a first number of initial images from the initial image set, and regards each initial image as a single-frame image; obtains multiple initial images (single-frame images) from the initial image set multiple times for averaging, and regards the image obtained by each averaging process as a fused image to obtain a second number of fused images.

[0067] It should be noted that the number of single-frame images fused into the fused image is not limited herein. For example, the fused image may be obtained by fusion of 4 single-frame images, 8 single-frame images, or 10 single-frame images.

[0068] For example, the electronic device obtains an initial image set, which includes 120 initial images, selects a first number of initial images from the initial images, and uses each initial image as a single-frame image single; obtains 4 initial images from the initial image set multiple times for fusion to obtain multiple fused images avg4; obtains 8 initial images from the initial image set multiple times for fusion to obtain multiple fused images avg8. Then, an input image set {single, avg4, avg8} is constructed, including three images, namely, a single-frame image single, a fused image avg4 obtained by fusion of 4 initial images, and a fused image avg8 obtained by fusion of 8 initial images.

[0069] In one embodiment, the electronic device randomly obtains an image from each image included in the input image set as the first image. In another embodiment, the electronic device selects a type of image with a higher proportion from each image included in the input image set, and selects an image from the type of image with a higher proportion as the first image. For example, the first number of single-frame images is 20, and the second number of fused images is 10, then the first number of single-frame images has a higher proportion, and one of the single-frame images is selected as the first image. It should be noted that the manner in which the electronic device obtains the first image from each image included in the input image set can be set as needed and is not limited here.

[0070] It is understandable that the noise level of a single-frame image is high, while the noise level of a fused image is low. In this embodiment, by averaging different numbers of single-frame images to simulate first images with different noise levels in actual applications, an input image set containing images with different noise levels is constructed, and the denoising network can be trained more accurately and more specifically in the future.

[0071] In one embodiment, the above method also includes: obtaining a label image corresponding to the first image; constructing a training pair based on the label image and the first image; the training pair is used to train the first denoising network to obtain a trained second denoising network; wherein the first denoising network is used to perform denoising processing on the first image.

[0072] The label image is a predefined image used for reference with the target image. The training pair refers to a pair of images consisting of the label image and the first image used to train the first denoising network. The first denoising network is a network that performs denoising on the first image. The second denoising network is a network obtained after the first denoising network is trained.

[0073] In this embodiment, the noise level of a single-frame image is relatively high, while the noise level of a fused image is relatively low. If the input image set contains images of various noise levels, then first images of different noise levels are obtained from the input image set, and second denoising networks with different noise reduction capabilities can be trained. For example, if an electronic device obtains a single-frame image with a relatively high noise level from an input image set, the noise reduction capability of the trained second denoising network is relatively weak, and the second denoising network also erases more high-frequency texture information in the image. For another example, if an electronic device obtains a fused image with a relatively low noise level from an input image set, the noise reduction capability of the trained second denoising network is relatively weak, and the second denoising network also retains more high-frequency texture information in the image.

[0074] In one embodiment, Figure 2 As shown, the electronic device fuses 4 single-frame images to obtain a fused image avg4 obtained by fusing 4 single-frame images, and fuses 8 single-frame images to obtain a fused image avg8 obtained by fusing 8 single-frame images, thereby obtaining a total of the second number of fused images avg4 and avg8, and obtaining a first number of single-frame images. The electronic device constructs an input image set with the first number of single-frame images and the second number of fused images avg4 and avg8, and can obtain a first image from each image included in the input image set. The electronic device shoots 120 scene images of the same shooting scene, each scene image is a single-frame image, and averages the 120 single-frame images to obtain a label image corresponding to the first image, and a training pair can be constructed based on the first image and the label image.

[0075] In one embodiment, the noise level of a single frame image is higher than the noise level of a fused image; the method further includes: adjusting the proportional relationship between the first number and the second number in the input image set; and obtaining a trained second denoising network based on the input image set after the proportional relationship is adjusted.

[0076] In one embodiment, the electronic device may adjust the ratio between the first number and the second number by adjusting the first number of single-frame images obtained from the input image set and the second number of fused images. In another embodiment, the electronic device may also adjust the ratio between the first number and the second number by copying a single-frame image or copying a fused image.

[0077] The electronic device performs multiple iterative training on the first denoising network. In each iterative training, the first image is randomly obtained from the input image set, and the label image obtained from the same shooting scene is trained. Each iterative training is a forward propagation and a back propagation of the first denoising network.

[0078] It is understandable that if the noise level of a single-frame image is high and the noise level of a fused image is low, the ratio between the first number of single-frame images and the second number of fused images can determine the probability of the noise level of the acquired first image, thereby determining the denoising ability of the first denoising network. By adjusting the ratio between the first number of single-frame images and the second number of fused images, a more targeted second denoising network that better meets user needs can be trained.

[0079] For example, if the first number of single-frame images accounts for a high proportion, the probability that the first image obtained from the input image set is a single-frame image is higher, and the trained second denoising network has a stronger denoising ability, and the denoised image will lose more high-frequency texture information and image details. For another example, if the second number of fused images accounts for a high proportion, the probability that the second image obtained from the input image set is a fused image is higher, and the trained second denoising network has a weaker denoising ability, but can retain more high-frequency texture information of the image.

[0080] Furthermore, the fused image may include an image obtained by fusing different numbers of single-frame images. For example, the fused image may include an image obtained by fusing 4 single-frame images, or may include an image obtained by fusing 8 single-frame images. The more single-frame images the fused image corresponds to, the lower the noise level of the fused image. Therefore, the proportional relationship between the first number and the number of various fused images may be adjusted, and the second denoising network that has been trained is obtained based on the input image set after the proportional relationship is adjusted.

[0081] For example, the noise levels of a single-frame image single, a fused image avg4 obtained by fusing four single-frame images, and a fused image avg8 obtained by fusing eight single-frame images decrease successively. The higher the proportion of single in the input image set, the stronger the denoising ability of the trained second denoising network. The higher the proportion of avg4, the medium denoising ability of the trained second denoising network, and more high-frequency texture information can be retained. The higher the proportion of avg8, the weaker the denoising ability of the trained second denoising network, and more high-frequency texture information can be retained.

[0082] In this embodiment, by adjusting the proportional relationship between the first number and the second number, the noise reduction strength of the first noise reduction network can be adjusted, so as to train a more accurate second noise reduction network that better meets user needs.

[0083] In one embodiment, a brightness channel texture map is determined based on a difference map between a first brightness channel map and a second brightness channel map, including: determining a difference map between the first brightness channel map and the second brightness channel map, downsampling and upsampling the difference map to obtain a first intermediate map; and determining a brightness channel texture map based on the difference map and the first intermediate map.

[0084] The first intermediate image is an image obtained after the electronic device sequentially downsamples and upsamples the difference image. The electronic device performs bilinear interpolation on the difference image to obtain a brightness channel image, specifically including: sequentially downsampling and upsampling the difference image, and then performing difference processing on the difference image and the first intermediate image, so as to extract high-frequency texture information erased by the noise reduction processing in the first brightness channel image corresponding to the first image, and obtain a brightness channel texture image.

[0085] In one embodiment, the electronic device first downsamples the difference image and then upsamples it to obtain the first intermediate image. In another embodiment, the electronic device first upsamples the difference image and then downsamples it to obtain the first intermediate image.

[0086] If the color space type of the first image and the second image is YUV, the electronic device can use the following formula to determine the brightness channel texture map:

[0087] blendY=noiseY-upsamle(downsample(noiseY));

[0088] Where noiseY = I inputY -I denoiseY . Among them, blendY is the brightness channel texture map, noiseY is the difference map, I inputY is the first brightness channel image of the first image, that is, the first image I input Y channel graph, I denoiseY is the second brightness channel image of the second image, that is, the second image I denoise The Y channel image of the noise channel is upsampled (downsample (noiseY)) as the first intermediate image.

[0089] In this embodiment, a difference map between the first brightness channel map and the second brightness channel map is determined, and the difference map is downsampled and upsampled to obtain a first intermediate map. Based on the difference map and the first intermediate map, high-frequency texture information that is erased by the noise reduction process in the first brightness channel map corresponding to the first image can be extracted to more accurately obtain the brightness channel texture map.

[0090] In one embodiment, a target image is generated based on a brightness channel texture map and a second image, including: obtaining correction parameters; generating a corrected brightness channel texture map based on the correction parameters and the brightness channel texture map; fusing the corrected brightness channel texture map with a second brightness channel map corresponding to the second image to obtain a target brightness channel map; and generating a target image based on the target brightness channel map and a second color channel map corresponding to the second image.

[0091] The correction parameter is a parameter for correcting the brightness channel texture map. The target brightness channel map is a channel map that fuses the corrected brightness channel texture map and the second brightness channel map and represents the brightness of the target image.

[0092] It is understandable that lens shadow phenomenon is widely seen in electronic devices, which gives the intuitive feeling that the corners of the image are too dark, such as Figure 3 As shown, it is necessary to correct and brighten it in the ISP (Image Signal Processor) process. Therefore, in this embodiment, the electronic device obtains the correction parameters, corrects the brightness channel texture map, and obtains a more accurate corrected brightness channel texture map, thereby generating a more accurate target brightness channel map and target image.

[0093] Specifically, the electronic device divides the brightness channel texture map by the correction parameter to obtain a corrected brightness channel texture map, and then superimposes the corrected brightness channel texture map and the second brightness channel map corresponding to the second image to obtain a target brightness channel map supplemented with high-frequency texture information. The electronic device can splice the target brightness channel map and the second color channel map corresponding to the second image into a target image in a target format. For example, the target format can be NV21 format, NV12 format, etc., but is not limited thereto.

[0094] The correction parameter is a matrix of a preset size, and the image size of the brightness channel texture map is obtained. If the preset size of the correction parameter is inconsistent with the image size, the preset size of the correction parameter is adjusted to the image size; if the preset size of the correction parameter is consistent with the image size, the step of dividing the brightness channel texture map by the correction parameter is performed. For example, if the correction parameter is a matrix of a preset size of 13*17, and the image size of the brightness channel texture map is 3072*4096, the preset size of the correction parameter matrix 13*17 is first interpolated to 3072*4096, which is consistent with the image size, and then the step of dividing the brightness channel texture map by the correction parameter is performed.

[0095] If the color space type of the first image and the second image is YUV, the electronic device can calculate the target brightness channel map using the following formula:

[0096]

[0097] Among them, I denoiseY_blend is the target brightness channel map, LSC is the correction parameter, blendY is the brightness channel texture map, I denoiseY The second brightness channel image of the second image, i.e., the second image I denoise Y channel graph of .

[0098] In this embodiment, the electronic device obtains correction parameters, corrects the brightness channel texture map, obtains a corrected brightness channel texture map, and fuses the corrected brightness channel texture map with the second brightness channel map corresponding to the second image to obtain a more accurate target brightness channel map. Based on the target brightness channel map and the second color channel map corresponding to the second image, a target image can be generated, which can avoid the shadow phenomenon of the lens. In addition, the method of generating a target image in this embodiment can also be applied to other pixel-level tasks or AI (Artificial Intelligence) algorithms to improve image quality. Among them, the pixel-level task refers to the output and input of the network are images of the same resolution, rather than categories represented by a single number or target detection boxes represented by an array.

[0099] In one embodiment, the above method also includes: obtaining a label image corresponding to the first image; determining a target loss value of the second image relative to the label image; training the first denoising network based on the target loss value until a training cutoff condition is met, thereby obtaining a trained third denoising network; wherein the first denoising network is used to perform denoising on the first image.

[0100] The target loss value is a loss value of the target image relative to the label image. The third denoising network is a network obtained by training the first denoising network based on the target loss value.

[0101] The electronic device obtains a target loss function, uses the target loss function to calculate a target loss value between the target image and the label image, and based on the target loss value, optimizes the network parameters by the gradient descent method, minimizes the target loss value, and obtains a more accurate third denoising network. The target loss function may be L1 (Least Abosulote Error), or other functions, such as L2 (Least Square Error), TV (Total Variation), etc., but is not limited thereto.

[0102] The training cut-off condition can be set as needed. In one embodiment, the training cut-off condition can be that the target loss value is less than a preset threshold. In another embodiment, the training cut-off condition can be that the number of training times reaches a preset number of times. In other embodiments, the training cut-off condition can also be that the training duration reaches a preset duration. Among them, the preset threshold, the preset number of times, and the preset duration can all be set as needed.

[0103] In this embodiment, the label image corresponding to the first image is obtained, and the target loss value of the second image relative to the label image can be determined. Then, the first denoising network is trained based on the target loss value until the training cutoff condition is met, and a more accurate third denoising network can be trained, and the third denoising network can perform more accurate denoising on the image to be processed later.

[0104] In one embodiment, obtaining the label image corresponding to the first image includes: shooting multiple frames of scene images for the same shooting scene; and averaging the multiple frames of scene images to obtain the label image corresponding to the first image.

[0105] Among them, the scene image is an image captured of the same scene and used to generate a label image corresponding to the target image. The scene image can be a short exposure image, that is, an image obtained by exposure with a shorter exposure time. The number of scene images captured by the electronic device for the same shooting scene can be set as needed. For example, the electronic device captures 120 frames of scene images for the same shooting scene, averages the 120 frames of scene images, and obtains the label image corresponding to the target image. For another example, the electronic device captures 150 frames of scene images for the same shooting scene, averages the 150 frames of scene images, and obtains the label image corresponding to the target image.

[0106] In particular, due to the vibration interference of the external environment, there is a slight displacement between the first frame of the scene image and the last frame of the scene image, which will cause image blur after averaging. Therefore, the number of scene images should not be too many. Through testing, it can be known that averaging 120 frames of scene images can achieve a good balance between noise level and blur. Of course, in other cases, the electronic device can also shoot 130 frames of scene images for the same shooting scene, and average the 130 frames of scene images to obtain a label image; it can also shoot 125 frames of scene images for the same shooting scene, and average the 125 frames of scene images to obtain a label image.

[0107] It can be understood that the noise in a single scene image conforms to the zero mean distribution, and averaging multiple frames of scene images can effectively eliminate the noise. In addition, the noise level in the label image is negatively correlated with the number of scene images. In other words, the more scene images there are, the lower the noise level of the label image.

[0108] Furthermore, the color space type of the target image and the scene image is YUV; multiple frames of scene images are averaged to obtain a label image corresponding to the first image, including: converting the color space type of each frame of the scene image to RGB to obtain an RGB type scene image; averaging multiple frames of RGB type scene images to obtain an RGB type label image; converting the color space type of the RGB type label image to YUV to obtain a YUV type label image.

[0109] It is understandable that if the electronic device averages multiple frames of scene images in the YUV domain, the resulting YUV-type label image will have a color cast problem. Therefore, in this embodiment, the electronic device first converts each frame of the YUV-type scene image into an RGB-type scene image, averages multiple frames of the RGB-type scene image in the RGB domain to obtain an RGB-type label image, and then converts the RGB-type label image into a YUV-type label image, thereby avoiding the color cast problem of the generated YUV-type label image and improving the accuracy of the generated label image, so that the target loss value between the target image and the label image can be calculated more accurately.

[0110] In one embodiment, Figure 4 As shown, the electronic device shoots the same shooting scene to obtain n YUV-type scene images, performs color space type conversion on each YUV-type scene image to obtain an RGB-type scene image, averages the n RGB-type scene images to obtain an RGB-type label image, and performs color space type conversion on the RGB-type label image to obtain a YUV-type label image.

[0111] In one embodiment, the target image and the label image have the same shape, which is (H, W, C). H, W, and C are respectively the height, width, and number of channels. The color space type of the label image is the same as the color space type of the target image. For example, if the color space type of the target image is YUV, the color space type of the label image is also YUV; if the color space type of the target image is RGB, the color space type of the label image is also RGB.

[0112] In one embodiment, performing average processing on multiple frames of scene images to obtain a label image corresponding to the first image includes: selecting a specified number of scene images from the multiple frames of scene images, performing average processing on the specified number of scene images, and obtaining the label image corresponding to the first image. The specified number can be set according to user needs. For example, the specified number is 10, 45, etc.

[0113] For example, the electronic device shoots 100 frames of scene images for the same shooting scene, and the specified number is 50. Then, 50 frames of scene images are selected from the 100 frames of scene images, and the 50 frames of scene images are averaged to obtain a label image corresponding to the first image.

[0114] In one embodiment, the color space types of the second image and the label image are both YUV, where Y represents the brightness channel of the image and UV represents the color channel of the image. Figure 5 As shown, determining the target loss value of the second image relative to the label image includes:

[0115] Step 502: Based on the second image and the label image, a first loss calculation is performed on the second brightness channel map and the second color channel map corresponding to the second image to obtain a first loss value of the second image; the first loss value is used to train the first denoising network to retain the low-frequency information of the color channel map when performing denoising processing.

[0116] YUV is a type of true-color color space. "Y" represents luminance (Luminance or Luma), which is the grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0117] The second brightness channel map is the Y channel map of the second image. The second color channel map is the U channel map and the V channel map of the second image. The first loss value is the loss value calculated for the second brightness channel map and the second color channel map of the loss image. The first loss value is used to train the first denoising network to retain the low-frequency information of the color channel map when performing denoising. The low-frequency information is the information of the slow color change in the image, that is, the information of the continuous gradient area.

[0118] Specifically, based on the second image and the label image, an L1 (Least Abosulote Error) loss function is used to perform L1 loss calculation on the second brightness channel map and the second color channel map corresponding to the second image, respectively, to obtain a first loss value of the second image.

[0119] In another embodiment, based on the loss image and the label image, an L2 (Least Square Error) loss function is used to perform L2 loss calculation on the second brightness channel image and the second color channel image corresponding to the second image to obtain a first loss value of the second image.

[0120] In another embodiment, based on the loss image and the label image, a TV (Total Variation) loss function is used to perform TV loss calculation on the second brightness channel image and the second color channel image corresponding to the second image to obtain a first loss value of the second image.

[0121] In one embodiment, based on the second image and the label image, a first loss calculation is performed on the second brightness channel map and the second color channel map corresponding to the second image, respectively, before obtaining the first loss value of the second image, it also includes: generating a second brightness channel map and a second color channel map respectively according to the second image, and generating a label brightness channel map and a label color channel map respectively according to the label image.

[0122] Take the generation of the second brightness channel image and the second color channel image as an example to illustrate:

[0123] In one embodiment, the electronic device may extract data of the same channel map from the second image to generate a second brightness channel map and a second color channel map. In another embodiment, the electronic device may split the second image to obtain a second brightness channel map and a second color channel map. In another embodiment, the electronic device may also interpolate a second brightness channel map representing brightness and a second color channel map representing chromaticity based on each pixel in the second image. In other embodiments, the electronic device may also generate the second brightness channel map and the second color channel map in other ways, which are not limited here.

[0124] It should be noted that the label brightness channel map and the label color channel map can also be generated in the same manner as above, which will not be described in detail here.

[0125] Step 504: Based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, a second loss calculation is performed on the second brightness channel to obtain a second loss value of the second brightness channel map.

[0126] The second loss value is a loss value calculated for the second brightness channel map of the second image.

[0127] Specifically, based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, the electronic device uses at least one loss function among the perceptual loss function, the low-pass filtering loss function, the traditional edge operator loss function and the multi-scale structural similarity loss function to perform a second loss calculation on the second brightness channel map to obtain a second loss value of the second brightness channel map.

[0128] In one embodiment, the electronic device uses any one of the loss functions including the perceptual loss function, the low-pass filtering loss function, the traditional edge operator loss function and the multi-scale structural similarity loss function to perform a second loss calculation on the second brightness channel image to obtain a second loss value of the second brightness channel image.

[0129] In another embodiment, the electronic device uses at least two loss functions selected from the group consisting of a perceptual loss function, a low-pass filter loss function, a traditional edge operator loss function, and a multi-scale structural similarity loss function to perform a second loss calculation on the second brightness channel image to obtain a second loss value of the second brightness channel image. The calculation order of the at least two loss functions can be set as needed.

[0130] For example, the electronic device sequentially uses the perceptual loss function, the low-pass filter loss function, the traditional edge operator loss function, and the multi-scale structural similarity loss function to perform a second loss calculation on the second brightness channel image to obtain a second loss value of the second brightness channel image. For another example, the electronic device sequentially uses the low-pass filter loss function, the perceptual loss function, and the multi-scale structural similarity loss function to perform a second loss calculation on the second brightness channel image to obtain a second loss value of the second brightness channel image.

[0131] Step 506: Determine a target loss value of the second image relative to the label image based on the first loss value and the second loss value.

[0132] The target loss value is the final loss value obtained after comparing the second image with the label image.

[0133] Specifically, the electronic device combines the first loss function and the second loss function to construct a target loss function; substitutes the first loss value and the second loss value into the target loss function to obtain the target loss value of the second image. The first loss function is used to perform the first loss calculation, and the second loss function is used to perform the second loss calculation. The combination of the first loss function and the second loss function can be addition, or multiplication by a weight factor before addition, which is not limited here.

[0134] It can be understood that for YUV type images, the brightness information (brightness information) represented by the Y channel map determines the details in the image, so it needs to be optimized, while the chrominance information represented by the U channel map and the V channel map only needs to retain low-frequency information, and no additional constraints are required. Therefore, in the above-mentioned image processing method, the first loss calculation is performed on the second brightness channel map and the second color channel map corresponding to the second image to obtain the first loss value of the second image, and the second loss calculation is performed on the second brightness channel map to obtain the second loss value of the second brightness channel map, so that the target loss value of the second image can be determined based on the first loss value and the second loss value. In other words, the target loss value not only has the second loss value for the second brightness channel map, the second loss value can be used to train the denoising network to more accurately reduce the brightness noise in the brightness channel map when performing denoising, but also has the first loss value calculated for the second brightness channel map and the second color channel map of the second image, the first loss value can be used to train the first denoising network to retain the low-frequency information of the color channel map when performing denoising, thereby removing the high-frequency noise (color noise) of the color channel map, so that a more accurate target loss value can be determined. Furthermore, a more accurate denoising network can be trained based on the more accurate target loss value, which can better preserve texture information while removing color noise in the image.

[0135] In one embodiment, based on the second image and the label image, a first loss calculation is performed on a second brightness channel map and a second color channel map corresponding to the second image, respectively, to obtain a first loss value of the second image, including: based on the second image and the label image, a first loss calculation is performed on a second brightness channel map and a second color channel map corresponding to the second image, respectively, to obtain a first brightness loss value of the second brightness channel map, and a first color loss value of the second color channel map; based on the first brightness loss value and the first color loss value, the first loss value of the second image is determined.

[0136] The first brightness channel loss value is a loss value obtained by performing a first loss calculation on a second brightness channel map of the second image. The first color loss value is a loss value obtained by performing a first loss calculation on a second color channel map of the second image.

[0137] Specifically, the electronic device uses an L1 loss function to perform L1 loss calculation on the second brightness channel map based on the second brightness channel map of the second image and the label brightness channel map of the label image to obtain a first brightness loss value; and uses an L1 loss function to perform L1 loss calculation on the second color channel map based on the second color channel map of the second image and the label color channel map of the label image to obtain a first color loss value.

[0138] The electronic device uses the following L1 loss function to perform L1 loss calculation on the Y channel image of the second image to obtain a first brightness loss value: Among them, L Y is the first brightness loss value, H represents the height of the Y channel image of the second image, W represents the width of the Y channel image of the second image, and I denoiseY is the Y channel image of the second image, I gtY is the Y channel map of the label image.

[0139] The electronic device uses the following L1 loss function to perform L1 loss calculation on the U channel image of the second image to obtain a first color loss value of the U channel: Among them, L U is the first color loss value of the U channel of the second image, H represents the height of the U channel map of the second image, W represents the width of the U channel map of the second image, and I denoiseU is the U channel map of the second image, I gtU is the U channel map of the label image.

[0140] The electronic device uses the following L1 loss function to perform L1 loss calculation on the V channel image of the second image to obtain a first color loss value of the V channel: Among them, L V is the first color loss value of the V channel image of the second image, H represents the height of the V channel image of the second image, W represents the width of the V channel image of the second image, and I denoiseV is the V channel image of the second image, I gtV is the V channel map of the label image.

[0141] It should be noted that in other embodiments, the electronic device may also use other loss functions such as the L2 loss function or the TV loss function to perform the first loss calculation, so as to obtain the first brightness loss value of the Y channel image of the second image, the first color loss value of the U channel image of the second image, and the first color loss value of the V channel image of the second image.

[0142] In this embodiment, based on the second image and the label image, the first loss calculation is performed on the second brightness channel map and the second color channel map corresponding to the second image, respectively, to obtain the first brightness loss value of the second brightness channel map and the first color loss value of the second color channel map, so that the first loss value of the second image can be accurately determined based on the first brightness loss value and the first color loss value.

[0143] In one embodiment, the electronic device may also perform a first loss calculation based on the U channel map of the second image and the U channel map of the label image to obtain a first color loss value of the U channel map of the second image, and perform a first loss calculation based on the V channel map of the second image and the V channel map of the label image to obtain a first color loss value of the V channel map of the second image. It is understandable that the first color loss value of the U channel map of the second image and the first color loss value of the V channel map of the second image are both loss values ​​for the channel map representing chromaticity (U channel map and V channel map), and the first color loss value of the U channel map of the second image and the first color loss value of the V channel map of the second image can be used to train the first denoising network to optimize chromaticity processing when performing denoising processing.

[0144] Similarly, the electronic device can also perform a first loss calculation and a second loss calculation based on the second brightness channel map of the second image and the label brightness channel map of the label image, respectively, to obtain a first loss value and a second loss value of the second brightness channel map. The second loss value is also a loss value calculated for the second brightness channel map. It can be understood that the first loss value and the second loss value are both loss values ​​for the second brightness channel map representing brightness, and the first loss value and the second loss value can be used to train the denoising network to optimize brightness processing when performing denoising processing.

[0145] In one embodiment, based on the first brightness loss value and the first color loss value, determining the first loss value of the second image includes: respectively obtaining a brightness loss weight corresponding to the first brightness loss value and a color loss weight corresponding to the first color loss value; wherein the brightness loss weight and the color loss weight are different; multiplying the first brightness loss value with the corresponding brightness loss weight, and multiplying the first color loss value with the corresponding color loss weight, and then adding the products obtained by the multiplication to obtain the first loss value of the second image.

[0146] The brightness loss weight is a weight factor corresponding to the first brightness loss value. The color loss weight is a weight factor corresponding to the first color loss value. The brightness loss weight and the color loss weight can be set according to the experience value obtained from actual debugging, or according to the actual scene. For example, different brightness loss weights and color channel loss weights can be set according to different scenes during the day and at night.

[0147] In one embodiment, the value of the brightness loss weight is higher than the value of the color loss weight, that is, the first brightness loss value weighted by the higher weight can be used to train the first denoising network to retain the high-frequency information of the brightness channel map when performing denoising. The high-frequency information is the information with fast frequency changes in the image, that is, the edge information in the image.

[0148] Furthermore, the brightness channel map includes a Y channel map, and the color channel map includes a U channel map and a V channel map. Then the brightness loss weight may be higher than the color loss weight of the U channel map, and also higher than the color loss weight of the V channel map.

[0149] For example, the brightness loss weight is set to 3, and the color loss weight of the U channel map and the color loss weight of the V channel map are both set to 1. For another example, the brightness loss weight is set to 3, the color loss weight of the U channel map is set to 2, and the color loss weight of the V channel map is set to 1.

[0150] It can be understood that the electronic device respectively obtains the brightness loss weight corresponding to the first brightness loss value and the color loss weight corresponding to the first color loss value, and the brightness loss weight and the color loss weight are different. Then, the weighted first brightness loss value can be used to train the first denoising network to retain the high-frequency information of the brightness channel map when performing denoising.

[0151] The first brightness channel image includes a Y channel image, and the first color channel image includes a U channel image and a V channel image. The electronic device calculates the first loss value of the second image using the following formula: L 1 =L Y +γ(L U +L V ). Among them, L 1 is the first loss value, L Y is the first brightness loss value of the Y channel map of the first image, L U is the first color loss value of the U channel map of the first image, L V is the first color loss value of the V channel map of the first image, γ is the brightness loss weight of the Y channel, and the color loss weights of the U channel map and the V channel map are both 1.

[0152] In one embodiment, based on a second brightness channel map corresponding to the second image and a label brightness channel map of the label image, a second loss calculation is performed on the second brightness channel to obtain a second loss value of the second brightness channel map, including: based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, perceptual loss calculation, low-pass filtering loss calculation, traditional edge operator loss calculation and multi-scale structural similarity loss calculation are performed on the second brightness channel map to obtain perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map; based on the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map, the second loss value of the second brightness channel map is determined.

[0153] Specifically, the electronic device uses a perceptual loss function to perform perceptual loss calculation based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image to obtain the perceptual loss of the second brightness channel map, uses a low-pass filtering loss function to perform low-pass filtering loss calculation to obtain the low-pass filtering loss of the second brightness channel map, uses a traditional edge operator loss function to perform traditional edge operator loss calculation to obtain the traditional edge operator loss, and uses a multi-scale structural similarity loss function to perform multi-scale structural similarity loss calculation to obtain the multi-scale structural similarity loss; obtains weight factors of the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss respectively, multiplies the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss by the corresponding weight factors respectively, and then adds the four products obtained by the multiplication to obtain the second loss value of the second brightness channel map.

[0154] The electronic device uses a perceptual loss function to perform perceptual loss calculation based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image to obtain the perceptual loss of the second brightness channel map. Specifically, the electronic device inputs the second brightness channel map corresponding to the second image and the label brightness channel map of the label image into a VGG-16 (Visual Geometry Group Network) network, extracts output features of a specified layer from the VGG-16 network, and performs perceptual loss calculation on the output features of the specified layer to obtain the perceptual loss of the second brightness channel map.

[0155] Among them, the VGG-16 network includes 13 convolutional layers and 3 fully connected layers. It is understandable that different layers in the VGG-16 network can extract features of different scales. The designated layer can be set as needed. In this embodiment, the designated layer can be selected from the 4th layer and the 7th layer.

[0156] The specified layers are layer 4 and layer 7. The characteristic shape of the output of layer i in the VGG-16 network is defined as (H i , W i , C i ), H i represents the height of the characteristic shape of the output of the i-th layer, W i Represents the width of the feature shape output by the i-th layer, C i The electronic device extracts the output features of the 4th and 7th layers from the VGG-16 network. The features of the label images output by the 4th and 7th layers are F4 and F7, respectively. gtY and F7 gtY , the features of the second image output by the 4th and 7th layers are F4 denoiseY and F7 denoiseY, get the perceptual loss parameter alpha = 0.8, then the perceptual loss Among them, L perceptual is the perceptual loss of the second brightness channel map, C 4 *H 4 *W 4 is the characteristic shape of the output of layer 4, C 7 *H 7 *W 7 It is the characteristic shape of the output of layer 7.

[0157] It can be understood that the perceptual loss only acts on the second brightness channel map, and more semantic information and high-frequency information can be extracted with the help of the VGG-16 network, so that the perceptual loss is used to train the first denoising network to retain the details in the image during denoising.

[0158] The electronic device calculates the low-pass filter loss by calling the low-pass filter loss function through a Gaussian filter with a specified kernel size based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, thereby obtaining the low-pass filter loss of the second brightness channel map. The specified kernel size can be set as needed. For example, the specified kernel size is 9.

[0159] It can be understood that the electronic device calculates the low-pass filter loss based on the second brightness channel image corresponding to the second image and the label brightness channel image of the label image to obtain the low-pass filter loss of the second brightness channel image. The low-pass filter loss can be used to train the first denoising network to extract low-frequency information in the image during denoising, and the low-frequency information can ensure the overall contour of the image. Figure 6 , which is a comparison diagram before and after low-pass filtering of an image in one embodiment.

[0160] Take the second brightness channel map as the Y channel map of the second image and the label brightness channel map as the Y channel map of the label image as an example for explanation. The low-pass filter loss of the Y channel map of the second image is calculated using the following formula:

[0161] Among them, L lowpass is the low-pass filtering loss of the Y channel image of the second image, H is the height of the Y channel image of the second image, W is the width of the Y channel image of the second image, and I denoiseY is the Y channel image of the second image, I gtY is the Y channel map of the label image.

[0162] The electronic device uses a four-directional Sobel operator to extract edge information of the second brightness channel image and the label brightness channel image, and can calculate the traditional edge operator loss of the second brightness channel image.

[0163] Take the second brightness channel map as the Y channel map of the second image and the label brightness channel map as the Y channel map of the label image as an example for explanation. The low traditional edge operator loss of the Y channel map of the second image is calculated using the following formula:

[0164] Among them, L sobel is the low traditional edge operator loss of the Y channel map of the second image, H is the height of the Y channel map of the second image, W is the width of the Y channel map of the second image, and I denoiseY is the Y channel image of the second image, I gtY is the Y channel map of the label image.

[0165] It can be understood that by calculating the traditional edge operator loss of the second brightness channel image, it can be used to train the first denoising network to constrain edge information when performing denoising processing.

[0166] Take the second brightness channel map as the Y channel map of the second image and the label brightness channel map as the Y channel map of the label image as an example for explanation. The multi-scale structural similarity loss of the Y channel map of the second image is calculated using the following formula:

[0167] L MS-SSIM =1-MS-SSIM(I gtY , I denoiseY ). Among them, L MS-SSIM is the multi-scale structural similarity loss of the Y channel map of the second image, I denoiseY is the Y channel image of the second image, I gtY is the Y channel map of the label image.

[0168] It can be understood that by calculating the multi-scale structural similarity loss of the second brightness channel image, it can be used to train the first denoising network to ensure the similarity of the spatial structure of the image when performing denoising processing.

[0169] Furthermore, the electronic device calculates the target loss value of the second image using the following formula:

[0170] L YUV =L 1 +0.01*L perceptual +0.1*L lowpass +0.1*L sobel +0.1*L MS-SSIM Among them, L YUV is the target loss value, L 1 is the first loss value, L perceptual is the perceptual loss, L lowpass is the low-pass filter loss, L sobel is the traditional edge operator loss, L MS-SSIM is the multi-scale structural similarity loss.

[0171] In this embodiment, based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, perceptual loss calculation, low-pass filtering loss calculation, traditional edge operator loss calculation and multi-scale structural similarity loss calculation are performed on the second brightness channel map respectively to obtain the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map, and then based on the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map, a second loss value of the second brightness channel map is determined, and the perceptual loss included in the second loss value can be used to train the first denoising network to retain the details in the image when performing denoising, the low-pass filtering loss included in the second loss value can be used to train the first denoising network to extract low-frequency information in the image when performing denoising, the traditional edge operator loss included in the second loss value can be used to train the first denoising network to constrain edge information when performing denoising, and the multi-scale structural similarity loss included in the second loss value can be used to train the first denoising network to ensure the similarity of the spatial structure of the image when performing denoising. That is to say, after training, the first denoising network can better retain the edge information, spatial structure and high- and low-frequency important information in the image on the basis of removing color noise and brightness noise, thereby improving the precision of image denoising and being able to denoise the image more accurately.

[0172] In other embodiments, the above-mentioned denoising network may be applied to image denoising tasks, image super-resolution tasks, and image deblurring tasks, but is not limited thereto.

[0173] In one embodiment, Figure 7 As shown, the electronic device obtains a first image, inputs the first image into a first denoising network, obtains a second image after denoising, and obtains a label image corresponding to the first image, and calculates the second image and the label image through a pre-constructed target loss function, and obtains a target loss value of the second image. Among them, the first denoising network can be a convolutional neural network. The target loss function can be applied to deep learning pixel-level tasks in the YUV domain, such as image super-resolution, image deblurring, etc. Among them, the pixel-level task refers to the output and input of the network are images of the same resolution, rather than categories represented by a single number or target detection boxes represented by an array.

[0174] In one embodiment, Figure 8 As shown, the first image is subjected to noise reduction processing to obtain a second image, including:

[0175] Step 802: Encode the first image to obtain a first feature map.

[0176] The first feature map is an image containing features obtained by encoding the first image. Image encoding, also known as image compression, refers to a technology that uses fewer bits to represent an image or information contained in the image while meeting a certain quality requirement (signal-to-noise ratio requirement or subjective evaluation score).

[0177] Specifically, the first image is encoded by an encoder to obtain a first feature map. The encoder may include multiple sub-encoders, and the first image is encoded by the multiple sub-encoders, and the image output by the last sub-encoder is used as the first feature map.

[0178] In one embodiment, the encoding process may include a convolution process and a downsampling process. In other embodiments, the encoding process includes a convolution process and a downsampling process, and may also include a cropping process, a merging process, and the like.

[0179] Step 804: Perform noise processing on the first feature map to obtain a noise-processed feature map.

[0180] The noise processing feature map refers to an image containing noise processing related features obtained after the first feature map is subjected to noise processing. The noise processing map may include a global pooling feature map and a non-local feature map.

[0181] The electronic device performs noise processing on the first feature graph to obtain a noise processing feature graph. The electronic device performs noise processing on the first feature graph, which may be noise reduction processing on the first feature graph, or may extract noise features of the first feature graph. The specific noise processing method may be set according to user needs and is not limited here.

[0182] Step 806: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0183] The concatenated feature map is a feature map obtained by concatenating the first feature map and the noise processing feature map. The second feature map is an image containing features obtained by decoding the concatenated feature map.

[0184] Specifically, the decoder decodes the series feature map to obtain the second feature map. The decoder may include multiple sub-decoders, the series feature map is decoded by the multiple sub-decoders, the series feature map is input into the first sub-decoder, and the image output by the last sub-decoder is used as the second feature map.

[0185] In one embodiment, the decoding process may include upsampling and convolution. In other embodiments, the decoding process may include upsampling and convolution, and may also include interpolation, concatenation, and the like.

[0186] Step 808: Generate a second image based on the first image and the second feature map.

[0187] It can be understood that the electronic device obtains the first image at the processor front end, and through a skip connection (skip connection) to the processor back end, the processor back end can obtain the first image and the second feature map, and generate a target image based on the first image and the second feature map.

[0188] In one embodiment, the electronic device may perform element-wise addition on the first image and the second feature map to obtain a target image. In another embodiment, the electronic device may also perform average processing on the first image and the second feature map to obtain a target image. In other embodiments, the electronic device may also use other methods to fuse the first image and the second feature map to generate a target image, which is not limited here.

[0189] In this embodiment, the acquired first image is encoded to obtain a first feature map, the first feature map is noise processed to obtain a noise processed feature map, and then the first feature map and the noise processed feature map are concatenated to obtain a concatenated feature map, and the concatenated feature map is decoded to obtain a second feature map after noise reduction processing. The first image is connected to the processor back end through a jump, and the first image contains more texture information and edge information. Based on the first image and the second feature map, a target image that has been subjected to noise reduction processing and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0190] It can also effectively remove a large amount of brightness noise and color noise in short-exposure images, and has a good noise reduction effect on noise with inconsistent intensities in different light and dark areas and different spatial positions in the image. It can also process input images with different noise intensities, and can effectively remove brightness noise and color noise in single-frame images or fused images. It does not rely on a specific input noise pattern and can be used as an independent noise reduction module. Furthermore, by changing the number of input channels of the first convolutional layer in the encoder module, it can also realize noise reduction processing of images of different color space types, such as noise reduction processing of YUV images, RGB images or Raw images, which can adapt to input images of different color space types.

[0191] In one embodiment, the color space type of the first image is YUV, where Y represents the brightness channel of the first image and UV represents the color channel of the first image. Fig. 9 As shown, before encoding the first image to obtain the first feature map, the process further includes:

[0192] Step 902: Based on the first image, obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution; the first resolution is twice the second resolution.

[0193] The first brightness channel map is a channel map representing the brightness of the first image, that is, a Y channel map. The first color channel map is a channel map representing the color of the first image, that is, a UV channel map.

[0194] The first resolution is the resolution of the first brightness channel image. The second resolution is the resolution of the first color channel image. In addition, the first resolution is twice the second resolution, that is, the YUV format of the first image is the NV21 format. The NV21 format is a two-plane mode, that is, Y and UV are divided into two planes, and the four Y components share two UV components, that is, the first resolution (H, W) of the first brightness channel image is twice the second resolution (H / 2, W / 2) of the first color channel image.

[0195] In one embodiment, the electronic device may extract a first brightness channel map representing brightness and a first color channel map representing color from the first image. In another embodiment, the electronic device may interpolate the first brightness channel map representing brightness and the first color channel map representing color based on each pixel in the first image. In other embodiments, the electronic device may also generate the first brightness channel map and the first color channel map in other ways, which are not limited here.

[0196] Step 904 , converting the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenating the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image.

[0197] The spatial dimension is converted to the channel dimension (Space to depth), that is, the pixels of the first brightness channel image in the spatial dimension are converted to the channel dimension, and the resolution of the first brightness channel image in the spatial dimension can be adjusted. The first channel image is an image obtained by connecting the first brightness channel image of the second resolution and the first color channel image of the second resolution in series.

[0198] The electronic device converts the spatial dimension of the first brightness channel image to the channel dimension, and can convert the first brightness channel image of the first resolution into the first brightness channel image of the second resolution, that is, the resolution of the first brightness channel image in the spatial dimension is reduced to half of the original, so that the first brightness channel image of the second resolution and the first color channel image of the second resolution have the same resolution, and the first brightness channel image of the second resolution and the first color channel image of the second resolution can be connected in series to obtain a series of first channel images.

[0199] The first image is encoded to obtain a first feature map, including:

[0200] Step 906: Encode the first channel map to obtain a first feature map.

[0201] In this embodiment, based on the first image, a first brightness channel image of a first resolution and a first color channel image of a second resolution are obtained, and the first resolution is twice the second resolution; the first brightness channel image is converted from a spatial dimension to a channel dimension, and a first brightness channel image with the same resolution as the first color channel image can be obtained, that is, the resolution of the first brightness channel image is reduced, and the size of the image processed by the network can be reduced, thereby speeding up the network to process the image. In addition, the first brightness channel image of the second resolution and the first color channel image of the second resolution are connected in series, so that the image information can be fully utilized, and the brightness noise of the first brightness channel image and the color noise of the first color channel image can be removed at the same time.

[0202] In one embodiment, Fig.10 As shown, after decoding the series feature map to obtain the second feature map, the process further includes:

[0203] Step 1002: Separate an intermediate brightness channel map with a third resolution and an intermediate color channel map with a third resolution from the second feature map.

[0204] The intermediate brightness channel map is a channel map representing brightness separated from the second feature map. The intermediate color channel map is a channel map representing color separated from the second feature map.

[0205] The third resolution is the resolution of the intermediate brightness channel map and the intermediate color channel map separated from the second feature map, that is, the resolution of the intermediate brightness channel map and the intermediate color channel map are the same.

[0206] Specifically, the second feature map includes an intermediate brightness channel map and an intermediate color channel map, and the intermediate brightness channel map and the intermediate color channel map are connected in series. The electronic device determines a series node between the intermediate brightness channel map and the intermediate color channel map from the second feature map, and separates the intermediate brightness channel map and the intermediate color channel map from the second feature map based on the series node.

[0207] Step 1004, converting the channel dimension of the intermediate brightness channel image of the third resolution into the space dimension to obtain an intermediate brightness channel image of the fourth resolution; the fourth resolution is twice the third resolution.

[0208] Channel dimension to space dimension (Depth to space) is to convert the pixels of the intermediate brightness channel image in the channel dimension to the space dimension, and the resolution of the intermediate brightness channel image in the space dimension can be adjusted. The fourth resolution is the resolution of the intermediate brightness channel image obtained after the channel dimension is converted to the space dimension.

[0209] The electronic device converts the channel dimension of the intermediate brightness channel image of the third resolution into the spatial dimension, and can convert the intermediate brightness channel image of the third resolution into an intermediate brightness channel image of the fourth resolution, that is, adjusts the resolution of the intermediate brightness channel image in the spatial dimension to twice the original.

[0210] Step 1006: Generate a second intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution.

[0211] The electronic device can generate a first intermediate image in NV21 format based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution. The format of the first intermediate image is the same as that of the first image, and the resolution of the brightness channel image is twice that of the color channel image, which ensures the consistency of the format between the output image and the input image, thereby improving the accuracy of image processing.

[0212] Generating a second image based on the first image and the second feature map includes:

[0213] Step 1008 : Generate a second image based on the first image and the second intermediate image.

[0214] In this embodiment, an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution are separated from the second feature map; the intermediate brightness channel map of the third resolution is converted from the channel dimension to the space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a first intermediate map is generated, that is, the format of the first intermediate map is the same as the format of the first image, and the resolution of the brightness channel map is twice the resolution of the color channel map, so that the target image can be accurately generated and output based on the first image and the first intermediate map with the same format, ensuring the consistency of the format between the output image and the input image, thereby improving the accuracy of image processing.

[0215] In one embodiment, Fig.11 As shown, an image processing method is provided, comprising the following steps:

[0216] Step 1102, obtain a first image, and obtain a corresponding first brightness channel map based on the first image; the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image.

[0217] Step 1104: Based on the first image, obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution; the first resolution is twice the second resolution.

[0218] Step 1106 , converting the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenating the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image.

[0219] Step 1108: Encode the first channel map to obtain a first feature map.

[0220] Step 1110: Perform noise processing on the first feature map to obtain a noise-processed feature map.

[0221] Step 1112: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0222] Step 1114: Generate a third intermediate map based on the first channel map and the second feature map.

[0223] The third intermediate image is an image generated based on the first channel image and the second feature image connected in series. Specifically, the electronic device connects the first channel image and the second feature image connected in series to obtain the third intermediate image.

[0224] After the electronic device acquires the first image at the processor front end and processes the first image to obtain a series-connected first channel map, it jumps to the processor back end, and the processor back end can acquire the first channel map and the second feature map, and generate a third intermediate map based on the first channel map and the second feature map.

[0225] It can be understood that the first brightness channel map of the first resolution and the first color channel map of the second resolution are obtained from the first image, the first brightness channel map is converted from the spatial dimension to the channel dimension to obtain the first brightness channel map of the second resolution, and then the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain the series-connected first channel map, and the series-connected first channel map has not been encoded and still retains more image texture information. Then, the series-connected first channel map is connected to the processor back end through a jump, and the processor back end can obtain the second intermediate map that has been denoised and retains more image texture information based on the first channel map and the second feature map.

[0226] Step 1116: Separate an intermediate brightness channel image with a third resolution and an intermediate color channel image with a third resolution from the third intermediate image.

[0227] Step 1118, convert the channel dimension of the intermediate brightness channel image of the third resolution to the space dimension to obtain the intermediate brightness channel image of the fourth resolution; the fourth resolution is twice the third resolution.

[0228] Step 1120: Generate a second intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution.

[0229] Step 1122: Generate a second image based on the first image and the second intermediate image, and acquire a corresponding second brightness channel image based on the second image.

[0230] Step 1124: Determine a brightness channel texture map according to the difference map between the first brightness channel map and the second brightness channel map.

[0231] Step 1126, generating a target image based on the brightness channel texture map and the second image.

[0232] In this embodiment, the electronic device connects the first image of the processor front end to the processor back end through a jump, and connects the first channel map of the processor front end to the processor back end through a jump. The two images at different node positions of the processor front end are connected to the corresponding processor back end through a jump. Different image texture information can be obtained at the processor back end, so that more image texture information can be retained while reducing the noise of the image, thereby improving the accuracy of image processing.

[0233] In one embodiment, encoding a first image to obtain a first feature map includes: performing convolution and downsampling processing on the first image through a first sub-encoder in an encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, performing convolution and downsampling processing on the shallow feature map output by the previous sub-encoder to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer; until a shallow feature map output by a last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

[0234] The number of sub-encoders included in the encoder can be set as needed. For example, the encoder includes 4 sub-encoders, 3 sub-encoders, or 6 sub-encoders. Downsampling refers to the extraction of a signal.

[0235] Each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer. In this embodiment, each sub-encoder includes a 3x3 convolution layer and a downsampling module, and each downsampling module includes a 3x3 convolution layer and an average pooling layer with a step size of 2. Fig.12 As shown, the downsampling module includes a 3x3 convolution layer and an average pooling layer. In other embodiments, the sub-encoder may further include multiple convolution layers and multiple downsampling modules, and each downsampling module may include multiple convolution layers and multiple average pooling layers.

[0236] In one embodiment, decoding the series feature map to obtain a second feature map includes: upsampling and convolution processing the series feature map through the first sub-decoder in the decoder to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, upsampling and convolution processing the high-level feature map output by the previous sub-decoder to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer.

[0237] Similarly, the number of sub-decoders included in the decoder can also be set as needed. For example, the decoder includes 3 sub-decoders, 6 sub-decoders or 2 sub-decoders. Upsampling is the inverse process of downsampling, also known as upsampling or interpolating.

[0238] Each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer. In this embodiment, each sub-decoder includes an upsampling module and a 3x3 convolution layer, and each upsampling module includes two parallel 1x1 convolution layers, a 2x upsampling layer, and a 3x3 convolution layer. Fig.13 As shown, the upsampling module includes two parallel 1x1 convolutional layers, a 2x upsampling layer and a 3x3 convolutional layer. In other embodiments, the sub-decoder may further include multiple upsampling modules and multiple convolutional layers, and each upsampling module may further include one or more convolutional layers and multiple upsampling layers.

[0239] It should be noted that the number of sub-encoders included in the encoder and the number of sub-decoders included in the decoder can be the same or different. For example, the encoder includes 4 sub-encoders and the decoder includes 3 sub-decoders. The number of convolutional layers included in the sub-encoder and the number of convolutional layers included in the sub-decoder can be the same or different. The number of convolutional layers included in the downsampling module and the number of convolutional layers included in the upsampling module can be the same or different.

[0240] In this embodiment, during the encoding process, the first sub-encoder in the encoder performs convolution and downsampling processing on the first image to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolution and downsampling processing to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer; until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map. . During the decoding process, the first sub-decoder in the decoder performs upsampling and convolution processing on the series feature map to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, the high-level feature map output by the previous sub-decoder is up-sampled and convolution processed to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map, the encoding and decoding processing of the image can be accurately realized, thereby processing the image more accurately.

[0241] In one embodiment, the method for each sub-decoder to obtain an output high-level feature map also includes: jump-connecting the shallow feature map output by each sub-encoder to the corresponding sub-decoder; inputting the concatenated feature map as a high-level feature map into the first sub-decoder, and for each sub-decoder, upsampling and convolving the obtained shallow feature map and the input high-level feature map to obtain an output high-level feature map.

[0242] The above-mentioned image processing method is applied to an image processing network of an electronic device, and the image processing network includes an encoder module and a decoder module, and may also include a noise processing module. Among them, the encoder module is used to encode the original input image to extract a first feature map; the noise processing module is used to perform noise processing on the first feature map to extract a noise processing feature map; the decoder module is used to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map. In this embodiment, the image processing network can specifically be a network with a UNet structure.

[0243] The shallow feature map is the feature map output by the convolution layer in the encoder, which is located at a shallower position in the network and contains more spatial detail information and less semantic information. The high-level feature map is the feature map output by the convolution layer in the decoder, which is located at a deeper position in the network and contains more semantic information but lacks spatial detail information.

[0244] In this embodiment, the shallow feature map output by each sub-encoder is jump-connected to the corresponding sub-decoder; the concatenated feature map is input into the first sub-decoder as a high-level feature map, and for each sub-decoder, the obtained shallow feature map and the input high-level feature map are up-sampled and convolved to obtain the output high-level feature map, which can accurately encode and decode the first image. In addition, the up-sampling module is used to replace the deconvolution layer in the traditional technology, which can avoid the checkerboard or aliasing effect and improve the accuracy of image processing.

[0245] In one embodiment, for each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map, including: for each sub-decoder, the obtained shallow feature map and the input high-level feature map are respectively 1x1 convolved to obtain a 1x1 convolved shallow feature map and a 1x1 convolved high-level feature map; the 1x1 convolved high-level feature map is upsampled to obtain an upsampled high-level feature map; the 1x1 convolved shallow feature map and the upsampled high-level feature map are concatenated, and the concatenated feature map is convolved to obtain an output high-level feature map.

[0246] It is understandable that the resolution of the shallow feature map is higher, while the resolution of the high-level feature map is lower. The electronic device performs 1x1 convolution processing on the shallow feature map obtained by the current sub-decoder and the high-level feature map output by the previous sub-decoder, respectively, which can unify the number of channels between the high-resolution shallow feature map and the low-resolution high-level feature map, and obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map.

[0247] The electronic device performs upsampling processing on the high-level feature map of the 1x1 convolution, which can unify the resolution between the high-resolution shallow feature map and the low-resolution high-level feature map, and obtain an upsampled high-level feature map that is consistent with the resolution of the shallow feature map.

[0248] Then, the electronic device can connect in series the shallow feature map of 1x1 convolution with the same number of channels and resolution and the upsampled high-level feature map, and then perform convolution processing on the feature map obtained in series to accurately output the high-level feature map of the current sub-decoder.

[0249] In one embodiment, noise processing is performed on the first feature map to obtain a noise-processed feature map, including: performing global average pooling processing on the first feature map to obtain a global pooling feature map; and / or performing non-local processing on the first feature map to obtain a non-local feature map; the first feature map and the noise-processed feature map are connected in series to obtain a series feature map, including: connecting at least one feature map of the global pooling feature map and the non-local feature map, and the first feature map in series to obtain a series feature map.

[0250] The global pooling feature map is a feature map obtained by performing global average pooling on the first feature map. The non-local feature map is a feature map obtained by performing non-local processing on the first feature map.

[0251] The electronic device performs global average pooling processing on the first feature map through the global average pooling module to obtain a global pooling feature map, which can comprehensively analyze the features of the entire map and accurately estimate the average noise level in the first feature map. Fig.14 As shown, the global average pooling module includes a global average pooling layer and an upsampling layer.

[0252] The electronic device performs non-local processing on the first feature map through a non-local module, that is, the weighted sum of features at other positions in the entire first feature map is used to represent the features at the current position. The weight is the similarity between feature vectors, which can make full use of similar features at other positions in the image, capture long-range dependencies, and find positions with similar features in the feature space, thereby significantly expanding the receptive field of the network and helping to improve the robustness of the features.

[0253] The receptive field is the area size of the pixel points on the feature map output by each layer of the convolutional neural network mapped on the input image. Fig.15 As shown, the non-local module includes two parallel 1x1 convolution layers and a logistic regression layer. The input data is processed by 1x1 convolution respectively through two parallel 1x1 convolution layers to obtain 1x1 convolution data respectively; one of the obtained input logistic regression layers is processed by logistic regression to obtain logistic regression data, and then the 1x1 convolution data and the logistic regression data are element-by-element multiplied to obtain multiplied data; the multiplied data and the input data are added element-by-element to obtain output data. In one embodiment, the electronic device can also adopt a simplified non-local processing method to reduce the amount of calculation.

[0254] In this embodiment, the electronic device performs global average pooling processing on the first feature map to obtain a global pooling feature map, and / or performs non-local processing on the first feature map to obtain a non-local feature map; the global pooling feature map and at least one feature map of the non-local feature map, and the first feature map are connected in series to obtain a series feature map, so that during the process of processing the series feature map, the average noise level in the first feature map can be estimated, and / or the robustness of the feature can be improved, so as to perform image processing more accurately.

[0255] In one embodiment, the above method also includes: performing residual learning on the first denoising network based on the first image and the second feature map until the first cutoff condition is met to obtain a fourth denoising network; the first denoising network is used to perform denoising processing on the first image.

[0256] The fourth denoising network is a network obtained after residual learning of the first denoising network. In this embodiment, the first denoising network can be a model of a UNet structure.

[0257] In mathematical statistics, residual refers to the difference between the actual observed value and the estimated value (fitted value). The electronic device performs residual learning on the first denoising network based on the residual between the first image and the second feature map, so that the first denoising network can learn the texture information in the first image until the first cutoff condition is met, thereby obtaining the second denoising network. Then, the second denoising network can be used to generate a denoised image when processing the image to be processed, while retaining more image texture information, reducing the training difficulty of the first denoising network and improving the fidelity of the image.

[0258] The first cutoff condition can be set as needed. In one embodiment, the first cutoff condition can be that the number of residual learning times reaches a preset number of times. In other embodiments, the first cutoff condition can also be that the residual learning duration reaches a preset duration. Among them, the preset number of times and the preset duration can be set as needed.

[0259] In one embodiment, Fig.16 The figure shows a lightweight first denoising network, which adopts the UNet structure. The lightweight first denoising network can improve the ability to remove brightness noise and color noise while ensuring fast speed and low power consumption, while maintaining good texture details and natural image. Among them, the lightweight first denoising network can be applied to YUV domain deep learning pixel-level tasks, such as image super-resolution, image deblurring, etc.

[0260] The electronic device acquires a first image, acquires a first brightness channel image of a first resolution and a first color channel image of a second resolution through a module for converting a spatial dimension to a channel dimension, then converts the spatial dimension of the first brightness channel image to a channel dimension to obtain a first brightness channel image of a second resolution, and concatenates the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image; the first resolution is twice the second resolution.

[0261] The encoder includes 4 sub-encoders. The first sub-encoder in the encoder performs convolution and downsampling processing on the first channel map to obtain a shallow feature map. For each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolved and downsampled (3x3conv+relu) to obtain a shallow feature map. Each sub-encoder includes a 3×3 convolution layer and a downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer. Until the shallow feature map output by the last sub-encoder is obtained, the shallow feature map output by the last sub-encoder is used as the first feature map. At the same time, the electronic device jumps the shallow feature map output by each sub-encoder to the corresponding sub-decoder.

[0262] The first feature map is subjected to noise processing by a noise processing module to obtain a noise processing feature map. Specifically, the electronic device performs global average pooling processing on the first feature map to obtain a global pooling feature map; and / or performs non-local processing on the first feature map to obtain a non-local feature map; and then at least one feature map of the global pooling feature map and the non-local feature map and the first feature map are connected in series to obtain a series feature map.

[0263] The decoder includes three sub-decoders. The electronic device inputs the series feature map as a high-level feature map into the first sub-decoder. Through the first sub-decoder, the series feature map and the obtained shallow feature map are upsampled and convolved to obtain an output high-level feature map; for each sub-decoder after the first sub-decoder, the obtained shallow feature map and the high-level feature map output by the previous sub-decoder are upsampled and convolved to obtain an output high-level feature map; wherein each sub-decoder includes an upsampling module and a 3×3 convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

[0264] The second feature map is upsampled by the upsampling module, and the size of the second feature map is adjusted to be consistent with the size of the first image to obtain the adjusted second feature map. The first channel map obtained by the processor front end is connected to the processor back end through a jump, and the adjusted second feature map and the first channel map are connected in series to obtain a third intermediate map.

[0265] The third intermediate image is convolved by the convolution layer to obtain a convolution-processed third intermediate image. In this embodiment, the convolution module includes two 3×3 convolution layers.

[0266] An intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution are separated from the third intermediate map after convolution processing through a channel dimension to space dimension conversion module; the intermediate brightness channel map of the third resolution is converted from the channel dimension to the space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a second intermediate map is generated.

[0267] The first image obtained by the processor front end is connected to the processor back end through a jump, and the first image and the second intermediate image are added element by element to obtain the target image.

[0268] In one embodiment, the color space type of the first image is YUV, where Y represents the brightness channel of the first image and UV represents the color channel of the first image. Fig.17 As shown, the first image is subjected to noise reduction processing to obtain a second image, including:

[0269] Step 1702: Based on the first image, obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution; the first resolution is twice the second resolution.

[0270] Step 1704 , convert the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenate the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a first channel image.

[0271] Step 1706: Encode the first channel map to obtain a first feature map.

[0272] Step 1708: Perform noise processing on the first feature map to obtain a noise-processed feature map.

[0273] Step 1710: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0274] Step 1712, generating a second image based on the first channel map and the second feature map.

[0275] In this embodiment, based on a first image whose color space type is YUV, a first brightness channel map of a first resolution and a first color channel map of a second resolution are obtained; the first resolution is twice the second resolution; the spatial dimension of the first brightness channel map is converted to the channel dimension to obtain a first brightness channel map of the second resolution, and the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain a first channel map; the first channel map is encoded to obtain a first feature map; the first feature map is noise processed to obtain a noise processed feature map; the first feature map and the noise processed feature map are connected in series to obtain a series feature map, and the series feature map is decoded to obtain a second feature map; the series first channel map of the processor front end is connected to the processor back end through a jump, and based on the first channel map and the second feature map, a target image that has been denoised and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0276] It should be understood that although Figure 1 , Figure 5 , Figures 7 to 11 as well as Fig.17 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 5 , Figures 7 to 11 as well as Fig.17 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0277] Fig.18 FIG. 1 is a structural block diagram of an image processing device according to an embodiment. Fig.18 As shown, an image processing device is provided, including: an acquisition module 1802, a noise reduction module 1804, a determination module 1806 and an image generation module 1808, wherein:

[0278] The acquisition module 1802 is used to acquire a first image and acquire a corresponding first brightness channel map based on the first image.

[0279] The noise reduction module 1804 is used to perform noise reduction processing on the first image to obtain a second image, and obtain a corresponding second brightness channel image based on the second image.

[0280] The determination module 1806 is configured to determine a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map.

[0281] The image generation module 1808 is used to generate a target image based on the brightness channel texture map and the second image.

[0282] The above-mentioned image processing device obtains a first image, and obtains a corresponding first brightness channel map based on the first image; performs noise reduction processing on the first image to obtain a second image after noise reduction, and obtains a corresponding second brightness channel map after noise reduction based on the second image. It can be understood that the first brightness channel map is a channel map representing brightness that has not been subjected to noise reduction processing, and contains more high-frequency texture information. Then, based on the difference map between the first brightness channel map and the second brightness channel map, a brightness channel texture map can be determined, and the brightness channel texture map represents the high-frequency texture information that is erased by the noise reduction processing in the first brightness channel map. Then, based on the brightness channel texture map and the second image, a target image that has been subjected to noise reduction processing and retains more high-frequency texture information can be generated, thereby improving the accuracy of image processing.

[0283] In one embodiment, the acquisition module 1802 is also used to acquire a first number of single-frame images and a second number of fused images; wherein the fused image is an image obtained by fusing multiple single-frame images; construct an input image set based on the first number of single-frame images and the second number of fused images; and acquire the first image from each image included in the input image set.

[0284] In one embodiment, the above-mentioned device also includes a training module, which is used to obtain a label image corresponding to the first image; based on the label image and the first image, a training pair is constructed; the training pair is used to train the first denoising network to obtain a trained second denoising network; wherein the first denoising network is used to perform denoising processing on the first image.

[0285] In one embodiment, the noise level of a single frame image is higher than the noise level of a fused image; the training module is also used to adjust the proportional relationship between the first number and the second number in the input image set; based on the input image set after the proportional relationship is adjusted, a trained second denoising network is obtained.

[0286] In one embodiment, the above-mentioned determination module 1806 is also used to determine a difference map between the first brightness channel map and the second brightness channel map, downsample and upsample the difference map to obtain a first intermediate map; based on the difference map and the first intermediate map, determine the brightness channel texture map.

[0287] In one embodiment, the above-mentioned image generation module 1808 is also used to obtain correction parameters; generate a corrected brightness channel texture map based on the correction parameters and the brightness channel texture map; fuse the corrected brightness channel texture map and the second brightness channel map corresponding to the second image to obtain a target brightness channel map; generate a target image based on the target brightness channel map and the second color channel map corresponding to the second image.

[0288] In one embodiment, the above-mentioned training module is also used to obtain a label image corresponding to the first image; determine a target loss value of the second image relative to the label image; train the first denoising network based on the target loss value until the training cutoff condition is met, and obtain a trained third denoising network; wherein the first denoising network is used to perform denoising processing on the first image.

[0289] In one embodiment, the training module is further used to capture multiple frames of scene images for the same shooting scene; and perform averaging processing on the multiple frames of scene images to obtain a label image corresponding to the first image.

[0290] In one embodiment, the color space type of the scene image is YUV; the above-mentioned training module is also used to convert the color space type of each frame of the scene image into RGB to obtain an RGB type scene image; average multiple frames of RGB type scene images to obtain an RGB type label image; convert the color space type of the RGB type label image into YUV to obtain a YUV type label image.

[0291] In one embodiment, the color space types of the second image and the label image are both YUV, Y represents the brightness channel of the image, and UV represents the color channel of the image; the above-mentioned training module is also used to perform a first loss calculation on the second brightness channel map and the second color channel map corresponding to the second image based on the second image and the label image, respectively, to obtain a first loss value of the second image; the first loss value is used to train the first denoising network to retain the low-frequency information of the color channel map when performing denoising; based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, a second loss calculation is performed on the second brightness channel to obtain a second loss value of the second brightness channel map; based on the first loss value and the second loss value, a target loss value of the second image relative to the label image is determined.

[0292] In one embodiment, the above-mentioned training module is also used to perform a first loss calculation on the second brightness channel map and the second color channel map corresponding to the second image based on the second image and the label image, respectively, to obtain a first brightness loss value of the second brightness channel map and a first color loss value of the second color channel map; based on the first brightness loss value and the first color loss value, determine the first loss value of the second image.

[0293] In one embodiment, the above-mentioned training module is also used to obtain the brightness loss weight corresponding to the first brightness loss value and the color loss weight corresponding to the first color loss value, respectively; wherein the brightness loss weight and the color loss weight are different; multiply the first brightness loss value with the corresponding brightness loss weight, and multiply the first color loss value with the corresponding color loss weight, and then add the products obtained by the multiplication to obtain the first loss value of the second image.

[0294] In one embodiment, the above-mentioned training module is also used to perform perceptual loss calculation, low-pass filtering loss calculation, traditional edge operator loss calculation and multi-scale structural similarity loss calculation on the second brightness channel map based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, so as to obtain the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map; based on the perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map, determine the second loss value of the second brightness channel map.

[0295] In one embodiment, the noise reduction module 1804 is also used to encode the first image to obtain a first feature map; perform noise processing on the first feature map to obtain a noise-processed feature map; concatenate the first feature map and the noise-processed feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map; and generate a second image based on the first image and the second feature map.

[0296] In one embodiment, the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image; the above-mentioned device also includes a conversion module, which is used to obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution based on the first image; the first resolution is twice the second resolution; the first brightness channel image is converted from a spatial dimension to a channel dimension to obtain a first brightness channel image of a second resolution, and the first brightness channel image of the second resolution and the first color channel image of the second resolution are connected in series to obtain a series of first channel images; the above-mentioned noise reduction module 1804 is also used to encode the first channel image to obtain a first feature image.

[0297] In one embodiment, the above-mentioned conversion module is also used to separate an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map; the intermediate brightness channel map of the third resolution is converted from the channel dimension to the spatial dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a second intermediate map is generated; the above-mentioned denoising module 1804 is also used to generate a second image based on the first image and the second intermediate map.

[0298] In one embodiment, the image generation module 1808 is further used to generate a third intermediate image based on the first channel image and the second feature image; the conversion module is further used to separate an intermediate brightness channel image of a third resolution and an intermediate color channel image of a third resolution from the third intermediate image.

[0299] In one embodiment, the denoising module 1804 is further used to perform convolution and down-sampling processing on the first image through the first sub-encoder in the encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, perform convolution and down-sampling processing on the shallow feature map output by the previous sub-encoder to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one down-sampling module, and each down-sampling module includes at least one convolution layer and at least one average pooling layer; until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

[0300] In one embodiment, the noise reduction module 1804 is also used to upsample and convolve the serial feature map through the first sub-decoder in the decoder to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, upsample and convolve the high-level feature map output by the previous sub-decoder to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

[0301] In one embodiment, the above-mentioned noise reduction module 1804 is also used to jump-connect the shallow feature map output by each sub-encoder to the corresponding sub-decoder; the series feature map is input into the first sub-decoder as a high-level feature map, and for each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map.

[0302] In one embodiment, the noise reduction module 1804 is also used to perform 1x1 convolution processing on the obtained shallow feature map and the input high-level feature map for each sub-decoder, respectively, to obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map; upsample the 1x1 convolution high-level feature map to obtain an upsampled high-level feature map; concatenate the 1x1 convolution shallow feature map and the upsampled high-level feature map, and then convolve the concatenated feature map to obtain an output high-level feature map.

[0303] In one embodiment, the denoising module 1804 is further used to perform global average pooling processing on the first feature map to obtain a global pooling feature map; and / or perform non-local processing on the first feature map to obtain a non-local feature map; and concatenate at least one feature map among the global pooling feature map and the non-local feature map, and the first feature map to obtain a concatenated feature map.

[0304] In one embodiment, the color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image; the above-mentioned noise reduction module 1804 is also used to obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution based on the first image; the first resolution is twice the second resolution; the spatial dimension of the first brightness channel image is converted to the channel dimension to obtain a first brightness channel image of a second resolution, and the first brightness channel image of the second resolution and the first color channel image of the second resolution are connected in series to obtain a first channel image; the first channel image is encoded to obtain a first feature image; the first feature image is noise processed to obtain a noise processed feature image; the first feature image and the noise processed feature image are connected in series to obtain a series feature image, and the series feature image is decoded to obtain a second feature image; and a second image is generated based on the first channel image and the second feature image.

[0305] The division of the modules in the above-mentioned image processing device is only for illustration. In other embodiments, the image processing device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image processing device.

[0306] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0307] Fig.19Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium.

[0308] The implementation of each module in the image processing device provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program module constituted by the computer program may be stored in a memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0309] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0310] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0311] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0312] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, and acquire a corresponding first brightness channel map based on the first image; Performing noise reduction processing on the first image to obtain a second image, and acquiring a corresponding second brightness channel image based on the second image; Determine a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map; The brightness channel texture map is an image of high-frequency texture information represented by the first brightness channel map corresponding to the first image; A target image is generated based on the brightness channel texture map and the second image.

2. The method according to claim 1, characterized in that The acquiring of the first image comprises: Acquire a first number of single-frame images and a second number of fused images; wherein the fused image is an image obtained by fusion of multiple single-frame images; Constructing an input image set based on a first number of single-frame images and a second number of fused images; A first image is obtained from each image included in the input image set.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining a label image corresponding to the first image; Based on the label image and the first image, a training pair is constructed; the training pair is used to train a first denoising network to obtain a trained second denoising network; wherein the first denoising network is used to perform denoising processing on the first image.

4. The method according to claim 3, characterized in that The noise level of the single frame image is higher than the noise level of the fused image; and the method further comprises: Adjusting the proportional relationship between the first number and the second number in the input image set; Based on the input image set after the proportional relationship is adjusted, a trained second denoising network is obtained.

5. The method according to claim 1, characterized in that The step of determining a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map comprises: Determine a difference map between the first brightness channel map and the second brightness channel map, and perform downsampling and upsampling on the difference map to obtain a first intermediate map; A brightness channel texture map is determined based on the difference map and the first intermediate map.

6. The method according to claim 1, characterized in that The step of generating a target image based on the brightness channel texture map and the second image comprises: Get correction parameters; generating a corrected luminance channel texture map based on the correction parameter and the luminance channel texture map; Fusing the corrected brightness channel texture map with the second brightness channel map corresponding to the second image to obtain a target brightness channel map; A target image is generated based on the target brightness channel map and a second color channel map corresponding to the second image.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining a label image corresponding to the first image; determining a target loss value of the second image relative to the label image; The first denoising network is trained based on the target loss value until a training cutoff condition is met, thereby obtaining a trained third denoising network; wherein the first denoising network is used to perform denoising processing on the first image.

8. The method according to claim 7, characterized in that The obtaining of the label image corresponding to the first image includes: Shooting multiple frames of scene images for the same shooting scene; The multiple frames of scene images are averaged to obtain a label image corresponding to the first image.

9. The method according to claim 8, characterized in that The color space type of the scene image is YUV; and the averaging process is performed on the multiple frames of scene images to obtain a label image corresponding to the first image, including: Converting the color space type of each frame of the scene image in the multiple frames of scene images into RGB to obtain a scene image of RGB type; Perform average processing on multiple frames of RGB scene images to obtain RGB label images; The color space type of the RGB type label image is converted into YUV to obtain a YUV type label image.

10. The method according to claim 7, characterized in that The color space types of the second image and the label image are both YUV, where Y represents the brightness channel of the image and UV represents the color channel of the image; The determining a target loss value of the second image relative to the label image comprises: Based on the second image and the label image, respectively, a first loss calculation is performed on a second brightness channel map and a second color channel map corresponding to the second image to obtain a first loss value of the second image; the first loss value is used to train the first denoising network to retain low-frequency information of the color channel map when performing denoising processing; Based on a second brightness channel map corresponding to the second image and the label brightness channel map of the label image, performing a second loss calculation on the second brightness channel to obtain a second loss value of the second brightness channel map; Based on the first loss value and the second loss value, a target loss value of the second image relative to the label image is determined.

11. The method according to claim 10, characterized in that The step of performing a first loss calculation on a second brightness channel map and a second color channel map corresponding to the second image based on the second image and the label image to obtain a first loss value of the second image includes: Based on the second image and the label image, respectively perform a first loss calculation on a second brightness channel map and a second color channel map corresponding to the second image to obtain a first brightness loss value of the second brightness channel map and a first color loss value of the second color channel map; Based on the first brightness loss value and the first color loss value, a first loss value of the second image is determined.

12. The method according to claim 11, characterized in that The determining, based on the first brightness loss value and the first color loss value, a first loss value of the second image comprises: Respectively obtaining a brightness loss weight corresponding to the first brightness loss value and a color loss weight corresponding to the first color loss value; wherein the brightness loss weight and the color loss weight are different; The first brightness loss value is multiplied by the corresponding brightness loss weight, and the first color loss value is multiplied by the corresponding color loss weight, and the products are added together to obtain a first loss value of the second image.

13. The method according to claim 10, characterized in that The performing a second loss calculation on the second brightness channel based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image to obtain a second loss value of the second brightness channel map includes: Based on the second brightness channel map corresponding to the second image and the label brightness channel map of the label image, performing perceptual loss calculation, low-pass filtering loss calculation, traditional edge operator loss calculation and multi-scale structural similarity loss calculation on the second brightness channel map, respectively, to obtain perceptual loss, low-pass filtering loss, traditional edge operator loss and multi-scale structural similarity loss of the second brightness channel map; Based on the perceptual loss, the low-pass filtering loss, the traditional edge operator loss and the multi-scale structural similarity loss of the second brightness channel map, a second loss value of the second brightness channel map is determined.

14. The method according to claim 1, characterized in that The performing noise reduction processing on the first image to obtain a second image includes: Performing encoding processing on the first image to obtain a first feature map; Performing noise processing on the first feature map to obtain a noise-processed feature map; Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map; A second image is generated based on the first image and the second feature map.

15. The method according to claim 14, characterized in that The color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image; Before encoding the first image to obtain the first feature map, the method further includes: Based on the first image, acquiring a first brightness channel image with a first resolution and a first color channel image with a second resolution, wherein the first resolution is twice the second resolution; Converting the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with a second resolution, and concatenating the first brightness channel map with the second resolution and the first color channel map with the second resolution to obtain a concatenated first channel map; The encoding process of the first image to obtain a first feature map includes: The first channel map is encoded to obtain a first feature map.

16. The method according to claim 15, characterized in that After the series feature graph is decoded to obtain the second feature graph, the method further includes: Separating an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map; Converting the intermediate brightness channel map of the third resolution from a channel dimension to a space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; Generate a second intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution; The generating a second image based on the first image and the second feature map comprises: A second image is generated based on the first image and the second intermediate image.

17. The method according to claim 16, characterized in that Before separating the intermediate brightness channel map of the third resolution and the intermediate color channel map of the third resolution from the second feature map, the method further includes: Generate a third intermediate map based on the first channel map and the second feature map; The step of separating an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map comprises: An intermediate brightness channel map with a third resolution and an intermediate color channel map with a third resolution are separated from the third intermediate map.

18. The method according to claim 14, characterized in that The encoding process of the first image to obtain a first feature map includes: The first image is convolved and downsampled by the first sub-encoder in the encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolved and downsampled to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer; Until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

19. The method according to claim 18, characterized in that The decoding process of the series-connected feature graph to obtain a second feature graph includes: The first sub-decoder in the decoder is used to upsample and convolve the series feature map to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, the high-level feature map output by the previous sub-decoder is upsampled and convoluted to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; Until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

20. The method according to claim 19, characterized in that The way each sub-decoder obtains the output high-level feature map also includes: Jump-connecting the shallow feature map output by each sub-encoder to the corresponding sub-decoder; The series feature map is input into the first sub-decoder as a high-level feature map. For each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map.

21. The method according to claim 20, characterized in that For each sub-decoder, upsampling and convolution processing are performed on the obtained shallow feature map and the input high-level feature map to obtain the output high-level feature map, including: For each sub-decoder, the obtained shallow feature map and the input high-level feature map are respectively subjected to 1x1 convolution processing to obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map; Performing upsampling processing on the high-level feature map of the 1x1 convolution to obtain an upsampled high-level feature map; The shallow feature map of the 1x1 convolution and the upsampled high-level feature map are connected in series, and then the concatenated feature map is convolved to obtain an output high-level feature map.

22. The method according to claim 14, characterized in that The performing noise processing on the first feature map to obtain a noise processed feature map includes: Performing global average pooling processing on the first feature map to obtain a global pooling feature map; and / or performing non-local processing on the first feature map to obtain a non-local feature map; The step of serially connecting the first feature map and the noise processing feature map to obtain a serial feature map includes: The global pooling feature map, at least one of the non-local feature maps, and the first feature map are connected in series to obtain a series feature map.

23. The method according to claim 1, characterized in that The color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image; The performing noise reduction processing on the first image to obtain a second image includes: Based on the first image, acquiring a first brightness channel image with a first resolution and a first color channel image with a second resolution, wherein the first resolution is twice the second resolution; Converting the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with a second resolution, and concatenating the first brightness channel map with the second resolution and the first color channel map with the second resolution to obtain a first channel map; Performing encoding processing on the first channel map to obtain a first feature map; Performing noise processing on the first feature map to obtain a noise-processed feature map; Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map; A second image is generated based on the first channel map and the second feature map.

24. An image processing device, characterized in that: include: An acquisition module, used to acquire a first image, and acquire a corresponding first brightness channel map based on the first image; a noise reduction module, configured to perform noise reduction processing on the first image to obtain a second image, and acquire a corresponding second brightness channel image based on the second image; A determination module, configured to determine a brightness channel texture map according to a difference map between the first brightness channel map and the second brightness channel map; The brightness channel texture map is an image of high-frequency texture information represented by the first brightness channel map corresponding to the first image; An image generation module is used to generate a target image based on the brightness channel texture map and the second image.

25. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the image processing method according to any one of claims 1 to 23.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 23 are implemented.

27. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the method enables the computer to execute the steps of implementing the method according to any one of claims 1 to 23.

Images