Image processing methods, apparatuses, electronic devices, and computer-readable storage media

Abstract

The present application relates to an image processing method, device, electronic device and storage medium. The method comprises: acquiring a first image; encoding the first image to obtain a first feature map; performing noise processing on the first feature map to obtain a noise processing feature map; concatenating the first feature map and the noise processing feature map to obtain a concatenated feature map, decoding the concatenated feature map to obtain a second feature map; and generating a target image based on the first image and the second feature map. The method can retain the texture information in the image while reducing noise.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, more and more electronic devices are equipped with image sensors, which can capture various types of images, such as YUV (Y represents brightness, UV represents chrominance) images, RGB (Red, Green, Blue) images, etc. Usually, a lot of noise will remain in the image, affecting the overall perception of the image, so it is necessary to reduce the noise of the image.

[0003] Traditional methods of image denoising include spatial domain filtering (such as bilateral filtering, non-local mean algorithm, etc.), frequency domain denoising (wavelet, discrete cosine transform, etc.), etc. However, after denoising an image using traditional image processing methods, the generated image cannot retain texture information, and there are problems such as over-smearing and loss of texture information. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can retain texture information in an image while reducing noise, thereby improving the accuracy of image processing.

[0005] An image processing method, comprising:

[0006] acquiring a first image;

[0007] Performing encoding processing on the first image to obtain a first feature map;

[0008] Performing noise processing on the first feature map to obtain a noise-processed feature map;

[0009] Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map;

[0010] A target image is generated based on the first image and the second feature map.

[0011] An image processing device, comprising:

[0012] An acquisition module, used to acquire a first image

[0013] An encoder module, used for encoding the first image to obtain a first feature map;

[0014] A noise processing module, used for performing noise processing on the first feature map to obtain a noise-processed feature map;

[0015] A decoder module, configured to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map;

[0016] An image generation module is used to generate a target image based on the first image and the second feature map.

[0017] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image processing method.

[0018] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0019] The above-mentioned image processing method, device, electronic device, and computer-readable storage medium encode the acquired first image to obtain a first feature map, perform noise processing on the first feature map to obtain a noise-processed feature map, and then concatenate the first feature map and the noise-processed feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map after noise reduction processing. The first image is connected to the processor back end through a jump, and the first image contains more texture information. Based on the first image and the second feature map, a target image that has been subjected to noise reduction processing and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0020] An image processing method, comprising:

[0021] Acquire a first image; the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image;

[0022] Based on the first image, acquiring a first brightness channel image with a first resolution and a first color channel image with a second resolution, wherein the first resolution is twice the second resolution;

[0023] Convert the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with the second resolution, and connect the first brightness channel map with the second resolution and the first color channel map with the second resolution in series to obtain a first channel map;

[0024] Performing encoding processing on the first channel map to obtain a first feature map;

[0025] Performing noise processing on the first feature map to obtain a noise-processed feature map;

[0026] Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map;

[0027] A target image is generated based on the first channel map and the second feature map.

[0028] An image processing device, comprising:

[0029] An acquisition module, used for acquiring a first image; the color space type of the first image is YUV, Y represents a brightness channel of the first image, and UV represents a color channel of the first image;

[0030] The acquisition module is further used to acquire, based on the first image, a first brightness channel image with a first resolution and a first color channel image with a second resolution; the first resolution is twice the second resolution;

[0031] a conversion module, configured to convert the spatial dimension of the first brightness channel map to the channel dimension to obtain a first brightness channel map of the second resolution, and connect the first brightness channel map of the second resolution and the first color channel map of the second resolution in series to obtain a first channel map;

[0032] An encoder module, used for encoding the first channel map to obtain a first feature map;

[0033] A noise processing module, used for performing noise processing on the first feature map to obtain a noise-processed feature map;

[0034] A decoder module, configured to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map;

[0035] An image generation module is used to generate a target image based on the first channel map and the second feature map.

[0036] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image processing method.

[0037] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0038] The above-mentioned image processing method, device, electronic device, and computer-readable storage medium obtain a first brightness channel map of a first resolution and a first color channel map of a second resolution based on a first image of a color space type of YUV; the first resolution is twice the second resolution; the spatial dimension of the first brightness channel map is converted to the channel dimension to obtain a first brightness channel map of a second resolution, and the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain a first channel map; the first channel map is encoded to obtain a first feature map; the first feature map is noise-processed to obtain a noise-processed feature map; the first feature map and the noise-processed feature map are connected in series to obtain a series feature map, and the series feature map is decoded to obtain a second feature map; the series first channel map of the processor front end is connected to the processor back end through a jump, and based on the first channel map and the second feature map, a target image that has been denoised and retains more texture information can be generated, thereby improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flowchart of an image processing method in one embodiment;

[0041] Figure 2 is a flowchart of an image processing method in another embodiment;

[0042] Figure 3 is a flowchart of an image processing method in another embodiment;

[0043] Figure 4 is a flowchart of an image processing method in another embodiment;

[0044] Figure 5 is a schematic diagram of a downsampling module in one embodiment;

[0045] Figure 6 is a schematic diagram of an upsampling module in one embodiment;

[0046] Figure 7 is a schematic diagram of a global average pooling module in one embodiment;

[0047] Figure 8 is a schematic diagram of a non-local module in one embodiment;

[0048] Fig. 9 is a flowchart of an image processing method in another embodiment;

[0049] Fig.10 is a flowchart of an image processing method in another embodiment;

[0050] Fig.11 is a structural block diagram of an image processing device in one embodiment;

[0051] Fig.12 is a structural block diagram of an image processing device in one embodiment;

[0052] Fig.13 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client. Both the first client and the second client are clients, but they are not the same client.

[0055] In one embodiment, Figure 1 As shown, an image processing method is provided, and this embodiment is illustrated by applying the method to an electronic device. Among them, the electronic device can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server can be implemented as an independent server or a server cluster composed of multiple servers. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step 102: Acquire a first image.

[0057] The first image refers to an image acquired by an electronic device for encoding processing. In one embodiment, the color space type of the first image is YUV, where Y represents the brightness (Luminance or Luma) channel of the first image, and UV represents the color (Chrominance or Chroma) channel of the first image. In another embodiment, the color space type of the first image may be RGB (Red, Green, Blue). In other embodiments, the color space type of the first image may also be CMY (Cyan, Magenta, Yellow) or HSV (Hue, Saturation, Value) and the like.

[0058] The electronic device acquires a first image, detects an original color space type of the first image, and if the original color space type is a non-target color space type, converts the original color space type of the first image into a target color space type; if the original color space type is the target color space type, performs encoding processing on the first image to obtain a first feature map.

[0059] The color space type refers to the type of the color system of the image. The color space type may include at least YUV, RGB, CMY, and HSV. The original color space type is the original color space type of the first image. The target color space type is the color space type expected by the first image.

[0060] If the original color space type is not the target color space type, the electronic device calls the color space conversion model to convert the original color space type into the target color space type.

[0061] For example, an electronic device obtains a first image, the original color space type of the first image is RGB, and the target color space type is YUV, that is, the original color space type RGB of the first image is a non-target color space type YUV, then the color space conversion model is called to convert the original color space type RGB of the first image into YUV.

[0062] For another example, the electronic device obtains a first image, the original color space type of the first image is YUV, and the target color space type is also YUV, then performs encoding processing on the first image to obtain a first feature map.

[0063] Step 104: encode the first image to obtain a first feature map.

[0064] The first feature map is an image containing features obtained by encoding the first image. Image encoding, also known as image compression, refers to a technology that uses fewer bits to represent an image or information contained in the image while meeting a certain quality requirement (signal-to-noise ratio requirement or subjective evaluation score).

[0065] Specifically, the first image is encoded by an encoder to obtain a first feature map. The encoder may include multiple sub-encoders, and the first image is encoded by the multiple sub-encoders, and the image output by the last sub-encoder is used as the first feature map.

[0066] In one embodiment, the encoding process may include a convolution process and a downsampling process. In other embodiments, the encoding process includes a convolution process and a downsampling process, and may also include a cropping process, a merging process, and the like.

[0067] Step 106: Perform noise processing on the first feature map to obtain a noise-processed feature map.

[0068] The noise processing feature map refers to an image containing noise processing related features obtained after the first feature map is subjected to noise processing. The noise processing map may include a global pooling feature map and a non-local feature map.

[0069] The electronic device performs noise processing on the first feature graph to obtain a noise processing feature graph. The electronic device performs noise processing on the first feature graph, which may be noise reduction processing on the first feature graph, or may extract noise features of the first feature graph. The specific noise processing method may be set according to user needs and is not limited here.

[0070] Step 108: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0071] The concatenated feature map is a feature map obtained by concatenating the first feature map and the noise processing feature map. The second feature map is an image containing features obtained by decoding the concatenated feature map.

[0072] Specifically, the decoder decodes the series feature map to obtain the second feature map. The decoder may include multiple sub-decoders, the series feature map is decoded by the multiple sub-decoders, the series feature map is input into the first sub-decoder, and the image output by the last sub-decoder is used as the second feature map.

[0073] In one embodiment, the decoding process may include upsampling and convolution. In other embodiments, the decoding process may include upsampling and convolution, and may also include interpolation, concatenation, and the like.

[0074] Step 110: Generate a target image based on the first image and the second feature map.

[0075] The target image is the final generated image. It is understandable that the electronic device obtains the first image at the processor front end, and through a skip connection to the processor back end, the processor back end can obtain the first image and the second feature map, and generate the target image based on the first image and the second feature map.

[0076] In one embodiment, the electronic device may perform element-wise addition on the first image and the second feature map to obtain a target image. In another embodiment, the electronic device may also perform average processing on the first image and the second feature map to obtain a target image. In other embodiments, the electronic device may also use other methods to fuse the first image and the second feature map to generate a target image, which is not limited here.

[0077] The above-mentioned image processing method encodes the acquired first image to obtain a first feature map, performs noise processing on the first feature map to obtain a noise-processed feature map, and then concatenates the first feature map and the noise-processed feature map to obtain a concatenated feature map. The concatenated feature map is decoded to obtain a second feature map after noise reduction processing. The first image is connected to the processor back end through a jump, and the first image contains more texture information and edge information. Based on the first image and the second feature map, a target image that has undergone noise reduction processing and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0078] Furthermore, the above image processing method can effectively remove a large amount of brightness noise and color noise in short-exposure images, and also has a good noise reduction effect on noise with inconsistent intensities in different light and dark areas and different spatial positions in the image.

[0079] The above image processing method can process input images with different noise intensities, and can effectively remove brightness noise and color noise in a single image or an image fused from multiple images. It does not rely on a specific input noise pattern and can be used as an independent noise reduction module. Furthermore, by changing the number of input channels of the first convolutional layer in the encoder module, it is also possible to implement noise reduction processing for images of different color space types, such as noise reduction processing for YUV images, RGB images, or Raw images, and can adapt to input images of different color space types.

[0080] In one embodiment, when an electronic device captures motion images, the above-mentioned image processing method is used to process the motion captured images to generate a target image, which can effectively remove noise in the short-exposure image while retaining more image texture information and edge information.

[0081] In one embodiment, the color space type of the first image is YUV, where Y represents the brightness channel of the first image and UV represents the color channel of the first image. Figure 2 As shown, after acquiring the first image, the method further includes:

[0082] Step 202: Based on the first image, obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution; the first resolution is twice the second resolution.

[0083] The first brightness channel map is a channel map representing the brightness of the first image, that is, a Y channel map. The first color channel map is a channel map representing the color of the first image, that is, a UV channel map.

[0084] The first resolution is the resolution of the first brightness channel image. The second resolution is the resolution of the first color channel image. In addition, the first resolution is twice the second resolution, that is, the YUV format of the first image is the NV21 format. The NV21 format is a two-plane mode, that is, Y and UV are divided into two planes, and the four Y components share two UV components, that is, the first resolution (H, W) of the first brightness channel image is twice the second resolution (H / 2, W / 2) of the first color channel image.

[0085] In one embodiment, the electronic device may extract a first brightness channel map representing brightness and a first color channel map representing color from the first image. In another embodiment, the electronic device may interpolate the first brightness channel map representing brightness and the first color channel map representing color based on each pixel in the first image. In other embodiments, the electronic device may also generate the first brightness channel map and the first color channel map in other ways, which are not limited here.

[0086] Step 204 , converting the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenating the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image.

[0087] The spatial dimension is converted to the channel dimension (Space to depth), that is, the pixels of the first brightness channel image in the spatial dimension are converted to the channel dimension, and the resolution of the first brightness channel image in the spatial dimension can be adjusted. The first channel image is an image obtained by connecting the first brightness channel image of the second resolution and the first color channel image of the second resolution in series.

[0088] The electronic device converts the spatial dimension of the first brightness channel image to the channel dimension, and can convert the first brightness channel image of the first resolution into the first brightness channel image of the second resolution, that is, the resolution of the first brightness channel image in the spatial dimension is reduced to half of the original, so that the first brightness channel image of the second resolution and the first color channel image of the second resolution have the same resolution, and the first brightness channel image of the second resolution and the first color channel image of the second resolution can be connected in series to obtain a series of first channel images.

[0089] The first image is encoded to obtain a first feature map, including:

[0090] Step 206: Encode the serially connected first channel graphs to obtain a first feature graph.

[0091] In this embodiment, based on the first image, a first brightness channel image of a first resolution and a first color channel image of a second resolution are obtained, and the first resolution is twice the second resolution; the first brightness channel image is converted from a spatial dimension to a channel dimension, and a first brightness channel image with the same resolution as the first color channel image can be obtained, that is, the resolution of the first brightness channel image is reduced, and the size of the image processed by the network can be reduced, thereby speeding up the network to process the image. In addition, the first brightness channel image of the second resolution and the first color channel image of the second resolution are connected in series, so that the image information can be fully utilized, and the brightness noise of the first brightness channel image and the color noise of the first color channel image can be removed at the same time.

[0092] In one embodiment, Figure 3 As shown, after decoding the series feature map to obtain the second feature map, the process further includes:

[0093] Step 302: Separate an intermediate brightness channel map with a third resolution and an intermediate color channel map with a third resolution from the second feature map.

[0094] The intermediate brightness channel map is a channel map representing brightness separated from the second feature map. The intermediate color channel map is a channel map representing color separated from the second feature map.

[0095] The third resolution is the resolution of the intermediate brightness channel map and the intermediate color channel map separated from the second feature map, that is, the resolution of the intermediate brightness channel map and the intermediate color channel map are the same.

[0096] Specifically, the second feature map includes an intermediate brightness channel map and an intermediate color channel map, and the intermediate brightness channel map and the intermediate color channel map are connected in series. The electronic device determines a series node between the intermediate brightness channel map and the intermediate color channel map from the second feature map, and separates the intermediate brightness channel map and the intermediate color channel map from the second feature map based on the series node.

[0097] Step 304, converting the channel dimension of the intermediate brightness channel image of the third resolution to the space dimension to obtain an intermediate brightness channel image of a fourth resolution; the fourth resolution is twice the third resolution.

[0098] Channel dimension to space dimension (Depth to space) is to convert the pixels of the intermediate brightness channel image in the channel dimension to the space dimension, and the resolution of the intermediate brightness channel image in the space dimension can be adjusted. The fourth resolution is the resolution of the intermediate brightness channel image obtained after the channel dimension is converted to the space dimension.

[0099] The electronic device converts the channel dimension of the intermediate brightness channel image of the third resolution into the spatial dimension, and can convert the intermediate brightness channel image of the third resolution into an intermediate brightness channel image of the fourth resolution, that is, adjusts the resolution of the intermediate brightness channel image in the spatial dimension to twice the original.

[0100] Step 306: Generate a first intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution.

[0101] The electronic device can generate a first intermediate image in NV21 format based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution. The format of the first intermediate image is the same as that of the first image, and the resolution of the brightness channel image is twice that of the color channel image, which ensures the consistency of the format between the output image and the input image, thereby improving the accuracy of image processing.

[0102] Generating a target image based on the first image and the second feature map includes:

[0103] Step 308: Generate a target image based on the first image and the first intermediate image.

[0104] In this embodiment, an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution are separated from the second feature map; the intermediate brightness channel map of the third resolution is converted from the channel dimension to the space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a first intermediate map is generated, that is, the format of the first intermediate map is the same as the format of the first image, and the resolution of the brightness channel map is twice the resolution of the color channel map, so that the target image can be accurately generated and output based on the first image and the first intermediate map with the same format, ensuring the consistency of the format between the output image and the input image, thereby improving the accuracy of image processing.

[0105] In one embodiment, Figure 4As shown, an image processing method is provided, comprising the following steps:

[0106] Step 402, obtaining a first image; the color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image.

[0107] Step 404: Based on the first image, a first brightness channel image of a first resolution and a first color channel image of a second resolution are acquired; the first resolution is twice the second resolution.

[0108] Step 406 , converting the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenating the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image.

[0109] Step 408: Encode the first channel map to obtain a first feature map.

[0110] Step 410: Perform noise processing on the first feature map to obtain a noise processed feature map.

[0111] Step 412: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0112] Step 414: Generate a second intermediate map based on the first channel map and the second feature map.

[0113] The second intermediate image is an image generated based on the first channel image and the second feature image connected in series. Specifically, the electronic device connects the first channel image and the second feature image connected in series to obtain the second intermediate image.

[0114] After the electronic device obtains the first image at the processor front end and processes the first feature map to obtain the serially connected first channel map, it is connected to the processor back end through a jump connection. The processor back end can then obtain the first channel map and the second feature map, and generate a second intermediate map based on the first channel map and the second feature map.

[0115] It can be understood that the first brightness channel map of the first resolution and the first color channel map of the second resolution are obtained from the first image, the first brightness channel map is converted from the spatial dimension to the channel dimension to obtain the first brightness channel map of the second resolution, and then the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain the series-connected first channel map, and the series-connected first channel map has not been encoded and still retains more image texture information. Then, the series-connected first channel map is connected to the processor back end through a jump, and the processor back end can obtain the second intermediate map that has been denoised and has more image texture information based on the first channel map and the second feature map.

[0116] Step 416: Separate an intermediate brightness channel image with a third resolution and an intermediate color channel image with a third resolution from the second intermediate image.

[0117] Step 418, converting the channel dimension of the intermediate brightness channel image of the third resolution to the space dimension to obtain an intermediate brightness channel image of the fourth resolution; the fourth resolution is twice the third resolution.

[0118] Step 420: Generate a first intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution.

[0119] Step 422: Generate a target image based on the first image and the first intermediate image.

[0120] In this embodiment, the electronic device connects the first image of the processor front end to the processor back end through a jump, and connects the first channel map in series of the processor front end to the processor back end through a jump. Two images at different node positions of the processor front end are connected to the corresponding processor back end through a jump. Different image texture information can be obtained at the processor back end, so that more image texture information can be retained while reducing the noise of the image, thereby improving the accuracy of image processing.

[0121] In one embodiment, encoding a first image to obtain a first feature map includes: performing convolution and downsampling processing on the first image through a first sub-encoder in an encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, performing convolution and downsampling processing on the shallow feature map output by the previous sub-encoder to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer; until a shallow feature map output by a last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

[0122] The number of sub-encoders included in the encoder can be set as needed. For example, the encoder includes 4 sub-encoders, 3 sub-encoders, or 6 sub-encoders. Downsampling refers to the extraction of a signal.

[0123] Each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer. In this embodiment, each sub-encoder includes a 3x3 convolution layer and a downsampling module, and each downsampling module includes a 3x3 convolution layer and an average pooling layer with a step size of 2. Figure 5 As shown, the downsampling module includes a 3x3 convolution layer and an average pooling layer. In other embodiments, the sub-encoder may further include multiple convolution layers and multiple downsampling modules, and each downsampling module may include multiple convolution layers and multiple average pooling layers.

[0124] In one embodiment, a first feature map and a noise-processed feature map are concatenated to obtain a concatenated feature map, and the concatenated feature map is decoded to obtain a second feature map, including: the first feature map and the noise-processed feature map are concatenated to obtain a concatenated feature map, and the concatenated feature map is upsampled and convolved through a first sub-decoder in a decoder to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, the high-level feature map output by the previous sub-decoder is upsampled and convolved to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until a high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

[0125] Similarly, the number of sub-decoders included in the decoder can also be set as needed. For example, the decoder includes 3 sub-decoders, 6 sub-decoders or 2 sub-decoders. Upsampling is the inverse process of downsampling, also known as upsampling or interpolating.

[0126] Each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer. In this embodiment, each sub-decoder includes an upsampling module and a 3x3 convolution layer, and each upsampling module includes two parallel 1x1 convolution layers, a 2x upsampling layer, and a 3x3 convolution layer. Figure 6As shown, the upsampling module includes two parallel 1x1 convolutional layers, a 2x upsampling layer and a 3x3 convolutional layer. In other embodiments, the sub-decoder may further include multiple upsampling modules and multiple convolutional layers, and each upsampling module may further include one or more convolutional layers and multiple upsampling layers.

[0127] It should be noted that the number of sub-encoders included in the encoder and the number of sub-decoders included in the decoder can be the same or different. For example, the encoder includes 4 sub-encoders and the decoder includes 3 sub-decoders. The number of convolutional layers included in the sub-encoder and the number of convolutional layers included in the sub-decoder can be the same or different. The number of convolutional layers included in the downsampling module and the number of convolutional layers included in the upsampling module can be the same or different.

[0128] In this embodiment, during the encoding process, the first image is convolved and down-sampled by the first sub-encoder in the encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolved and down-sampled to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one down-sampling module, and each down-sampling module includes at least one convolution layer and at least one average pooling layer; until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map. During the decoding process, the first sub-decoder in the decoder performs upsampling and convolution processing on the series feature map to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, the high-level feature map output by the previous sub-decoder is up-sampled and convolution processed to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map, the encoding and decoding processing of the image can be accurately realized, thereby processing the image more accurately.

[0129] In one embodiment, the method for each sub-decoder to obtain an output high-level feature map also includes: jump-connecting the shallow feature map output by each sub-encoder to the corresponding sub-decoder; inputting the concatenated feature map as a high-level feature map into the first sub-decoder, and for each sub-decoder, upsampling and convolving the obtained shallow feature map and the input high-level feature map to obtain an output high-level feature map.

[0130] The above-mentioned image processing method is applied to an image processing network of an electronic device, and the image processing network includes an encoder module and a decoder module, and may also include a noise processing module. Among them, the encoder module is used to encode the original input image to extract a first feature map; the noise processing module is used to perform noise processing on the first feature map to extract a noise processing feature map; the decoder module is used to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map. In this embodiment, the image processing network can specifically be a network with a UNet structure.

[0131] The shallow feature map is the feature map output by the convolution layer in the encoder, which is located at a shallower position in the network and contains more spatial detail information and less semantic information. The high-level feature map is the feature map output by the convolution layer in the decoder, which is located at a deeper position in the network and contains more semantic information but lacks spatial detail information.

[0132] In this embodiment, the shallow feature map output by each sub-encoder is jump-connected to the corresponding sub-decoder; the concatenated feature map is input into the first sub-decoder as a high-level feature map, and for each sub-decoder, the obtained shallow feature map and the input high-level feature map are up-sampled and convolved to obtain the output high-level feature map, which can accurately encode and decode the first image. In addition, the up-sampling module is used to replace the deconvolution layer in the traditional technology, which can avoid the checkerboard or aliasing effect and improve the accuracy of image processing.

[0133] In one embodiment, for each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map, including: for each sub-decoder, the obtained shallow feature map and the input high-level feature map are respectively 1x1 convolved to obtain a 1x1 convolved shallow feature map and a 1x1 convolved high-level feature map; the 1x1 convolved high-level feature map is upsampled to obtain an upsampled high-level feature map; the 1x1 convolved shallow feature map and the upsampled high-level feature map are concatenated, and the concatenated feature map is convolved to obtain an output high-level feature map.

[0134] It is understandable that the resolution of the shallow feature map is higher, while the resolution of the high-level feature map is lower. The electronic device performs 1x1 convolution processing on the shallow feature map obtained by the current sub-decoder and the high-level feature map output by the previous sub-decoder, respectively, which can unify the number of channels between the high-resolution shallow feature map and the low-resolution high-level feature map, and obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map.

[0135] The electronic device performs upsampling processing on the high-level feature map of the 1x1 convolution, which can unify the resolution between the high-resolution shallow feature map and the low-resolution high-level feature map, and obtain an upsampled high-level feature map that is consistent with the resolution of the shallow feature map.

[0136] Then, the electronic device can connect in series the shallow feature map of 1x1 convolution with the same number of channels and resolution and the upsampled high-level feature map, and then perform convolution processing on the feature map obtained in series to accurately output the high-level feature map of the current sub-decoder.

[0137] In one embodiment, noise processing is performed on the first feature map to obtain a noise-processed feature map, including: performing global average pooling processing on the first feature map to obtain a global pooling feature map; performing non-local processing on the first feature map to obtain a non-local feature map; and concatenating the first feature map and the noise-processed feature map to obtain a concatenated feature map, including: concatenating the first feature map, the global pooling feature map, and the non-local feature map to obtain a concatenated feature map.

[0138] The global pooling feature map is a feature map obtained by performing global average pooling on the first feature map. The non-local feature map is a feature map obtained by performing non-local processing on the first feature map.

[0139] The electronic device performs global average pooling processing on the first feature map through the global average pooling module to obtain a global pooling feature map, which can comprehensively analyze the features of the entire map and accurately estimate the average noise level in the first feature map. Figure 7 As shown, the global average pooling module includes a global average pooling layer and an upsampling layer.

[0140] The electronic device performs non-local processing on the first feature map through a non-local module, that is, the weighted sum of features at other positions in the entire first feature map is used to represent the features at the current position. The weight is the similarity between feature vectors, which can make full use of similar features at other positions in the image, capture long-range dependencies, and find positions with similar features in the feature space, thereby significantly expanding the receptive field of the network and helping to improve the robustness of the features.

[0141] The receptive field is the area size of the pixel points on the feature map output by each layer of the convolutional neural network mapped on the input image. Figure 8As shown, the non-local module includes two parallel 1x1 convolution layers and a logistic regression layer. The input data is processed by 1x1 convolution respectively through two parallel 1x1 convolution layers to obtain 1x1 convolution data respectively; one of the obtained input logistic regression layers is processed by logistic regression to obtain logistic regression data, and then the 1x1 convolution data and the logistic regression data are element-by-element multiplied to obtain multiplied data; the multiplied data and the input data are added element-by-element to obtain output data. In one embodiment, the electronic device can also adopt a simplified non-local processing method to reduce the amount of calculation.

[0142] In this embodiment, the electronic device performs global average pooling processing on the first feature map to obtain a global pooling feature map; performs non-local processing on the first feature map to obtain a non-local feature map, and concatenates the first feature map, the global pooling feature map, and the non-local feature map to obtain a concatenated feature map, so that in the process of processing the concatenated feature map, the average noise level in the first feature map can be estimated, and at the same time, the robustness of the features is improved, so that image processing can be performed more accurately.

[0143] In one embodiment, the above method also includes: performing residual learning on the first image processing model based on the first image and the second feature map until the first cutoff condition is met to obtain the second image processing model; wherein the first image processing model is used to process the first image to generate a target image.

[0144] The first image processing model is a model for processing the first image. The second image processing model is a model obtained after residual learning. In this embodiment, the first image processing model can be a model of UNet structure.

[0145] In mathematical statistics, residual refers to the difference between the actual observed value and the estimated value (fitted value). The electronic device performs residual learning on the first image processing model based on the residual between the first image and the second feature map, so that the first image processing model can learn the texture information in the first image until the first cutoff condition is met, thereby obtaining the second image processing model. Then, the second image processing model can be used to generate a denoised image when processing the image to be processed, while retaining more image texture information, reducing the difficulty of model training, and improving the fidelity of the image.

[0146] The first cutoff condition can be set as needed. In one embodiment, the first cutoff condition can be that the number of residual learning times reaches a preset number of times. In other embodiments, the first cutoff condition can also be that the residual learning duration reaches a preset duration. Among them, the preset number of times and the preset duration can be set as needed.

[0147] In one embodiment, the above method also includes: obtaining a label image corresponding to the target image; determining a target loss value of the target image relative to the label image based on the target image and the label image; training the first image processing model based on the target loss value until the second cutoff condition is met to obtain a third image processing model; wherein the first image processing model is used to process the first image to generate the target image.

[0148] The label image is a predefined image used to refer to the target image and calculate the target loss value. The target loss value is the loss value of the target image relative to the reference image.

[0149] In one embodiment, obtaining a label image corresponding to the target image specifically includes: shooting multiple frames of scene images for the same shooting scene; and averaging the multiple frames of scene images to obtain a label image corresponding to the target image.

[0150] The scene image is an image captured by the same scene and used to generate a label image corresponding to the target image. The number of scene images captured by the electronic device for the same shooting scene can be set as needed. For example, the electronic device captures 120 frames of scene images for the same shooting scene, averages the 120 frames of scene images, and obtains a label image corresponding to the target image.

[0151] It can be understood that the noise in a single scene image conforms to the zero mean distribution, and averaging multiple frames of scene images can effectively eliminate the noise. In addition, the noise level in the label image is negatively correlated with the number of scene images. In other words, the more scene images there are, the lower the noise level of the label image.

[0152] Furthermore, the color space type of the target image and the scene image is YUV; multiple frames of scene images are averaged to obtain a label image corresponding to the denoised image, including: converting the color space type of each frame of the scene image into RGB to obtain an RGB type scene image; averaging multiple frames of RGB type scene images to obtain an RGB type label image; converting the color space type of the RGB type label image into YUV to obtain a YUV type label image.

[0153] It is understandable that if the electronic device averages multiple frames of scene images in the YUV domain, the resulting YUV-type label image will have a color cast problem. Therefore, in this embodiment, the electronic device first converts each frame of the YUV-type scene image into an RGB-type scene image, averages multiple frames of the RGB-type scene image in the RGB domain to obtain an RGB-type label image, and then converts the RGB-type label image into a YUV-type label image, thereby avoiding the color cast problem of the generated YUV-type label image and improving the accuracy of the generated label image, so that the target loss value between the target image and the label image can be calculated more accurately.

[0154] In one embodiment, the target image and the label image have the same shape, which is (H, W, C). H, W, and C are respectively the height, width, and number of channels. The color space type of the reference image is the same as the color space type of the target image. For example, if the color space type of the target image is YUV, the color space type of the reference image is also YUV; if the color space type of the target image is RGB, the color space type of the reference image is also RGB.

[0155] The electronic device can use L1 (Least Abosulote Error) and SSIM (structural similarity) to calculate the target loss value between the target image and the label image, and train the first image processing model based on the target loss value until the second cutoff condition is met to obtain the third image processing model.

[0156] The second cutoff condition can be set as needed. In one embodiment, the second cutoff condition can be that the target loss value is less than a preset threshold. In another embodiment, the second cutoff condition can be that the number of training times reaches a preset number of times. In other embodiments, the second cutoff condition can also be that the training duration reaches a preset duration. Among them, the preset threshold, the preset number of times, and the preset duration can all be set as needed.

[0157] The third image processing model is a model obtained by training the first image processing model based on the target loss value. Specifically, the electronic device optimizes the network parameters by the gradient descent method based on the target loss value, minimizes the target loss value, and obtains the trained third image processing model until the second cutoff condition is met.

[0158] Then, the trained third image processing model can obtain a more accurate image that better meets user needs when processing the image to be processed.

[0159] In one embodiment, Fig. 9The figure shows a lightweight denoising network, which adopts a UNet structure. The electronic device obtains a first image, obtains a first brightness channel image of a first resolution and a first color channel image of a second resolution through a module for converting the spatial dimension to the channel dimension, then converts the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenates the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a concatenated first channel image; the first resolution is twice the second resolution.

[0160] The encoder includes 4 sub-encoders. The first sub-encoder in the encoder performs convolution and downsampling processing on the first channel map to obtain a shallow feature map. For each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolved and downsampled (3x3conv+relu) to obtain a shallow feature map. Each sub-encoder includes a 3×3 convolution layer and a downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer. Until the shallow feature map output by the last sub-encoder is obtained, the shallow feature map output by the last sub-encoder is used as the first feature map. At the same time, the electronic device jumps the shallow feature map output by each sub-encoder to the corresponding sub-decoder.

[0161] The first feature map is subjected to global average pooling processing through a noise processing module to obtain a global pooling feature map; the first feature map is subjected to non-local processing to obtain a non-local feature map; and the first feature map, the global pooling feature map and the non-local feature map are connected in series to obtain a series feature map.

[0162] The decoder includes three sub-decoders. The electronic device inputs the series feature map as a high-level feature map into the first sub-decoder. Through the first sub-decoder, the series feature map and the obtained shallow feature map are upsampled and convolved to obtain an output high-level feature map; for each sub-decoder after the first sub-decoder, the obtained shallow feature map and the high-level feature map output by the previous sub-decoder are upsampled and convolved to obtain an output high-level feature map; wherein each sub-decoder includes an upsampling module and a 3×3 convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

[0163] The second feature map is upsampled by the upsampling module, and the size of the second feature map is adjusted to be consistent with the size of the first image to obtain the adjusted second feature map. The first channel map obtained by the processor front end is connected to the processor back end through a jump, and the adjusted second feature map and the first channel map are connected in series to obtain the second intermediate map.

[0164] The second intermediate image is convolved by the convolution layer to obtain a convolution-processed second intermediate image. In this embodiment, the convolution module includes two 3×3 convolution layers.

[0165] An intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution are separated from the second intermediate map after convolution processing through a channel dimension to space dimension conversion module; the intermediate brightness channel map of the third resolution is converted from channel dimension to space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a first intermediate map is generated.

[0166] The first image obtained by the processor front end is connected to the processor back end through a jump, and the first image and the first intermediate image are added element by element to obtain a target image.

[0167] In one embodiment, Fig.10 As shown, an image processing method is provided, and this embodiment is illustrated by applying the method to an electronic device. Among them, the electronic device can be a terminal or a server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server can be implemented as an independent server or a server cluster composed of multiple servers. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0168] Step 1002, obtaining a first image; the color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image.

[0169] Step 1004: Based on the first image, obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution; the first resolution is twice the second resolution.

[0170] Step 1006: Convert the spatial dimension of the first brightness channel image to the channel dimension to obtain a first brightness channel image of a second resolution, and concatenate the first brightness channel image of the second resolution and the first color channel image of the second resolution to obtain a first channel image.

[0171] Step 1008: Encode the first channel map to obtain a first feature map.

[0172] Step 1010: Perform noise processing on the first feature map to obtain a noise-processed feature map.

[0173] Step 1012: Concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0174] Step 1014: Generate a target image based on the first channel map and the second feature map.

[0175] In this embodiment, based on a first image whose color space type is YUV, a first brightness channel map of a first resolution and a first color channel map of a second resolution are obtained; the first resolution is twice the second resolution; the spatial dimension of the first brightness channel map is converted to the channel dimension to obtain a first brightness channel map of the second resolution, and the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain a first channel map; the first channel map is encoded to obtain a first feature map; the first feature map is noise processed to obtain a noise processed feature map; the first feature map and the noise processed feature map are connected in series to obtain a series feature map, and the series feature map is decoded to obtain a second feature map; the series first channel map of the processor front end is connected to the processor back end through a jump, and based on the first channel map and the second feature map, a target image that has been denoised and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0176] It should be understood that although Figures 1 to 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 1 to 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0177] Fig.11 FIG. 1 is a structural block diagram of an image processing device according to an embodiment. Fig.11 As shown, an image processing device is provided, including: an acquisition module 1102, an encoder module 1104, a noise processing module 1106, a decoder module 1108 and an image generation module 1110, wherein:

[0178] The acquisition module 1102 is used to acquire a first image.

[0179] The encoder module 1104 is used to encode the first image to obtain a first feature map.

[0180] The noise processing module 1106 is used to perform noise processing on the first feature map to obtain a noise-processed feature map.

[0181] The decoder module 1108 is used to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0182] The image generation module 1110 is used to generate a target image based on the first image and the second feature map.

[0183] The above-mentioned image processing device encodes the acquired first image to obtain a first feature map, performs noise processing on the first feature map to obtain a noise-processed feature map, and then concatenates the first feature map and the noise-processed feature map to obtain a concatenated feature map, and decodes the concatenated feature map to obtain a second feature map after noise reduction processing. The first image is connected to the processor back end through a jump, and the first image contains more texture information. Based on the first image and the second feature map, a target image that has undergone noise reduction processing and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0184] In one embodiment, the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image; the above-mentioned device also includes a conversion module, which is used to obtain a first brightness channel image of a first resolution and a first color channel image of a second resolution based on the first image; the first resolution is twice the second resolution; the first brightness channel image is converted from a spatial dimension to a channel dimension to obtain a first brightness channel image of a second resolution, and the first brightness channel image of the second resolution and the first color channel image of the second resolution are connected in series to obtain a series of first channel images; the above-mentioned encoding module is also used to encode the first channel image to obtain a first feature image.

[0185] In one embodiment, the above-mentioned conversion module is also used to separate an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map; the intermediate brightness channel map of the third resolution is converted from the channel dimension to the spatial dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; based on the intermediate brightness channel map of the fourth resolution and the intermediate color channel map of the third resolution, a first intermediate map is generated; the above-mentioned image generation module 1110 is also used to generate a target image based on the first image and the first intermediate map.

[0186] In one embodiment, the image generation module 1110 is further used to generate a second intermediate image based on the first channel image and the second feature image; the conversion module is further used to separate an intermediate brightness channel image of a third resolution and an intermediate color channel image of a third resolution from the second intermediate image.

[0187] In one embodiment, the encoder module 1104 is further used to perform convolution and down-sampling processing on the first image through the first sub-encoder in the encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, perform convolution and down-sampling processing on the shallow feature map output by the previous sub-encoder to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one down-sampling module, and each down-sampling module includes at least one convolution layer and at least one average pooling layer; until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

[0188] In one embodiment, the decoder module 1108 is further used to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and upsample and convolve the concatenated feature map through the first sub-decoder in the decoder to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, upsample and convolve the high-level feature map output by the previous sub-decoder to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

[0189] In one embodiment, the decoder module 1108 is also used to jump-connect the shallow feature map output by each sub-encoder to the corresponding sub-decoder; the concatenated feature map is input into the first sub-decoder as a high-level feature map, and for each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map.

[0190] In one embodiment, the decoder module 1108 is further used to perform 1x1 convolution processing on the obtained shallow feature map and the input high-level feature map for each sub-decoder, respectively, to obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map; upsample the 1x1 convolution high-level feature map to obtain an upsampled high-level feature map; concatenate the 1x1 convolution shallow feature map and the upsampled high-level feature map, and then convolute the concatenated feature map to obtain an output high-level feature map.

[0191] In one embodiment, the noise processing module 1106 is also used to perform global average pooling processing on the first feature map to obtain a global pooling feature map; perform non-local processing on the first feature map to obtain a non-local feature map; the decoder module 1108 is also used to concatenate the first feature map, the global pooling feature map and the non-local feature map to obtain a concatenated feature map.

[0192] In one embodiment, the above-mentioned device also includes a training module, which is used to perform residual learning on the first image processing model based on the first image and the second feature map until the first cutoff condition is met to obtain the second image processing model; wherein the first image processing model is used to process the first image to generate a target image.

[0193] In one embodiment, the above-mentioned training module is also used to obtain a label image corresponding to the target image; based on the target image and the label image, determine the target loss value of the target image relative to the label image; train the first image processing model based on the target loss value until the second cutoff condition is met to obtain a third image processing model; wherein the first image processing model is used to process the first image to generate a target image.

[0194] Fig.12 FIG. 1 is a structural block diagram of an image processing device according to another embodiment. Fig.12 As shown, an image processing device is provided, including: an acquisition module 1202, a conversion module 1204, an encoder module 1206, a noise processing module 1208, a decoder module 1210 and an image generation module 1212, wherein:

[0195] The acquisition module 1202 is used to acquire a first image; the color space type of the first image is YUV, where Y represents the brightness channel of the first image, and UV represents the color channel of the first image.

[0196] The acquisition module 1202 is further configured to acquire, based on the first image, a first brightness channel image at a first resolution and a first color channel image at a second resolution; the first resolution is twice the second resolution.

[0197] The conversion module 1204 is used to convert the spatial dimension of the first brightness channel map to the channel dimension to obtain the first brightness channel map of the second resolution, and connect the first brightness channel map of the second resolution and the first color channel map of the second resolution in series to obtain the first channel map.

[0198] The encoder module 1206 is used to encode the first channel map to obtain a first feature map.

[0199] The noise processing module 1208 is used to perform noise processing on the first feature map to obtain a noise-processed feature map.

[0200] The decoder module 1210 is used to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map.

[0201] The image generation module 1212 is used to generate a target image based on the first channel map and the second feature map.

[0202] The above-mentioned image processing device obtains a first brightness channel map of a first resolution and a first color channel map of a second resolution based on a first image of a color space type of YUV; the first resolution is twice the second resolution; the spatial dimension of the first brightness channel map is converted to the channel dimension to obtain a first brightness channel map of a second resolution, and the first brightness channel map of the second resolution and the first color channel map of the second resolution are connected in series to obtain a first channel map; the first channel map is encoded to obtain a first feature map; the first feature map is noise processed to obtain a noise-processed feature map; the first feature map and the noise-processed feature map are connected in series to obtain a series feature map, and the series feature map is decoded to obtain a second feature map; the series first channel map of the processor front end is connected to the processor back end through a jump, and based on the first channel map and the second feature map, a target image that has been denoised and retains more texture information can be generated, thereby improving the accuracy of image processing.

[0203] The division of the modules in the above-mentioned image processing device is only for illustration. In other embodiments, the image processing device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image processing device.

[0204] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0205] Fig.13Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium.

[0206] The implementation of each module in the image processing device provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program module constituted by the computer program may be stored in a memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0207] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0208] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0209] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0210] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image; the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image; Based on the first image, acquiring a first brightness channel image with a first resolution and a first color channel image with a second resolution; The first resolution is twice the second resolution; Converting the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with a second resolution, and concatenating the first brightness channel map with the second resolution and the first color channel map with the second resolution to obtain a concatenated first channel map; Performing encoding processing on the first channel map to obtain a first feature map; Performing noise processing on the first feature map to obtain a noise-processed feature map; Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map; A target image is generated based on the first image and the second feature map.

2. The method according to claim 1, characterized in that After the decoding process is performed on the series feature graph to obtain the second feature graph, the method further includes: Separating an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map; Converting the intermediate brightness channel map of the third resolution from a channel dimension to a space dimension to obtain an intermediate brightness channel map of a fourth resolution; the fourth resolution is twice the third resolution; Generate a first intermediate image based on the intermediate brightness channel image of the fourth resolution and the intermediate color channel image of the third resolution; Generating a target image based on the first image and the second feature map includes: A target image is generated based on the first image and the first intermediate image.

3. The method according to claim 2, characterized in that Before separating the intermediate brightness channel map of the third resolution and the intermediate color channel map of the third resolution from the second feature map, the method further includes: Generate a second intermediate map based on the first channel map and the second feature map; The step of separating an intermediate brightness channel map of a third resolution and an intermediate color channel map of a third resolution from the second feature map comprises: An intermediate brightness channel map with a third resolution and an intermediate color channel map with a third resolution are separated from the second intermediate map.

4. The method according to claim 1, characterized in that: The encoding process of the first channel map to obtain a first feature map includes: The first channel map is convolved and downsampled by the first sub-encoder in the encoder to obtain a shallow feature map; for each sub-encoder after the first sub-encoder, the shallow feature map output by the previous sub-encoder is convolved and downsampled to obtain a shallow feature map; wherein each sub-encoder includes at least one convolution layer and at least one downsampling module, and each downsampling module includes at least one convolution layer and at least one average pooling layer; Until the shallow feature map output by the last sub-encoder is obtained, and the shallow feature map output by the last sub-encoder is used as the first feature map.

5. The method according to claim 4, characterized in that The step of concatenating the first feature map and the noise processing feature map to obtain a concatenated feature map, and decoding the concatenated feature map to obtain a second feature map comprises: The first feature map and the noise processing feature map are connected in series to obtain a series feature map, and the series feature map is upsampled and convolved through the first sub-decoder in the decoder to obtain a high-level feature map; for each sub-decoder after the first sub-decoder, the high-level feature map output by the previous sub-decoder is upsampled and convolved to obtain an output high-level feature map; wherein each sub-decoder includes at least one upsampling module and at least one convolution layer, and each upsampling module includes at least one convolution layer and at least one upsampling layer; Until the high-level feature map output by the last sub-decoder is obtained, and the high-level feature map output by the last sub-decoder is used as the second feature map.

6. The method according to claim 5, characterized in that The way each sub-decoder obtains the output high-level feature map also includes: Jump-connecting the shallow feature map output by each sub-encoder to the corresponding sub-decoder; The series feature map is input into the first sub-decoder as a high-level feature map. For each sub-decoder, the obtained shallow feature map and the input high-level feature map are upsampled and convolved to obtain an output high-level feature map.

7. The method according to claim 6, characterized in that For each sub-decoder, upsampling and convolution processing are performed on the obtained shallow feature map and the input high-level feature map to obtain the output high-level feature map, including: For each sub-decoder, the obtained shallow feature map and the input high-level feature map are respectively subjected to 1x1 convolution processing to obtain a 1x1 convolution shallow feature map and a 1x1 convolution high-level feature map; Performing upsampling processing on the high-level feature map of the 1x1 convolution to obtain an upsampled high-level feature map; The shallow feature map of the 1x1 convolution and the upsampled high-level feature map are connected in series, and then the concatenated feature map is convolved to obtain an output high-level feature map.

8. The method according to claim 1, characterized in that The performing noise processing on the first feature map to obtain a noise processed feature map includes: Performing global average pooling processing on the first feature map to obtain a global pooling feature map; Performing non-local processing on the first feature map to obtain a non-local feature map; The step of serially connecting the first feature map and the noise processing feature map to obtain a serial feature map includes: The first feature map, the global pooling feature map and the non-local feature map are connected in series to obtain a series feature map.

9. The method according to claim 1, characterized in that: The method further comprises: Based on the first image and the second feature map, residual learning is performed on the first image processing model until a first cutoff condition is met to obtain a second image processing model; wherein the first image processing model is used to process the first image to generate the target image.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Obtaining a label image corresponding to the target image; Based on the target image and the label image, determining a target loss value of the target image relative to the label image; The first image processing model is trained based on the target loss value until a second cutoff condition is met to obtain a third image processing model; wherein the first image processing model is used to process the first image to generate the target image.

11. An image processing method, characterized in that: include: Acquire a first image; the color space type of the first image is YUV, Y represents the brightness channel of the first image, and UV represents the color channel of the first image; Based on the first image, acquiring a first brightness channel image with a first resolution and a first color channel image with a second resolution; The first resolution is twice the second resolution; Converting the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with a second resolution, and concatenating the first brightness channel map with the second resolution and the first color channel map with the second resolution to obtain a first channel map; Performing encoding processing on the first channel map to obtain a first feature map; Performing noise processing on the first feature map to obtain a noise-processed feature map; Connecting the first feature map and the noise processing feature map in series to obtain a series feature map, and decoding the series feature map to obtain a second feature map; A target image is generated based on the first channel map and the second feature map.

12. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image; the color space type of the first image is YUV, Y represents a brightness channel of the first image, and UV represents a color channel of the first image; A conversion module, configured to obtain a first brightness channel image with a first resolution and a first color channel image with a second resolution based on the first image; The first resolution is twice the second resolution; Converting the first brightness channel map from a spatial dimension to a channel dimension to obtain a first brightness channel map with a second resolution, and concatenating the first brightness channel map with the second resolution and the first color channel map with the second resolution to obtain a concatenated first channel map; An encoder module, used for encoding the first channel map to obtain a first feature map; A noise processing module, used for performing noise processing on the first feature map to obtain a noise-processed feature map; A decoder module, configured to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map; An image generation module is used to generate a target image based on the first image and the second feature map.

13. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image; the color space type of the first image is YUV, Y represents a brightness channel of the first image, and UV represents a color channel of the first image; The acquisition module is further used to acquire a first brightness channel image with a first resolution and a first color channel image with a second resolution based on the first image; The first resolution is twice the second resolution; a conversion module, configured to convert the spatial dimension of the first brightness channel map to the channel dimension to obtain a first brightness channel map with a second resolution, and connect the first brightness channel map with the second resolution and the first color channel map with the second resolution in series to obtain a first channel map; An encoder module, used for encoding the first channel map to obtain a first feature map; A noise processing module, used for performing noise processing on the first feature map to obtain a noise-processed feature map; A decoder module, configured to concatenate the first feature map and the noise processing feature map to obtain a concatenated feature map, and decode the concatenated feature map to obtain a second feature map; An image generation module is used to generate a target image based on the first channel map and the second feature map.

14. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images