Image processing method, device, storage medium and program product

Abstract

The present application relates to the field of image processing technology, and discloses an image processing method, device, storage medium and program product. The image processing method includes: after the electronic device obtains the image P0 to be processed, it can first use a first noise reduction algorithm with a larger noise reduction strength to perform noise reduction processing on the image P0 to obtain an image P01, and the background area (for example, the sky area, as the first area) of the image P01 has a higher purity; and, it can also use a second noise reduction algorithm with a smaller noise reduction strength to perform noise reduction processing on the image P0 to obtain an image P02, and the main scene area of ​​the image P02 has a higher clarity. Next, the electronic device can merge the image content of the first area in the image P01 with the image content of the image P02 except the first area to obtain a target image P1. In this way, the obtained target image P1 not only ensures the noise reduction effect of the entire image, but also retains more image detail information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and program product. Background Art

[0002] When taking photos at night, due to the poor light conditions at night, the camera usually cannot collect more light to obtain more image signals, resulting in a large noise in the captured night scene image. In the process of shooting night scene images, due to the illumination of street lamps or other light sources on the main scene area (such as the shooting target), the light conditions in the main scene area are usually better than those in the sky area, resulting in the noise in the main scene area of ​​the night scene image being less than the noise in the sky area.

[0003] Among them, when processing night scene images, if an algorithm with a relatively high noise reduction strength is used to reduce the noise of the night scene images, although the sky area in the night scene images after noise reduction will be relatively pure and clear, the main scene area may be over-smeared due to excessive noise reduction. For example, when the sky area in the night scene images is relatively pure and clear, the skin texture of the characters in the night scene images may be smoothed out and lose the sense of reality due to excessive noise reduction. On the contrary, if an algorithm with a relatively low noise reduction strength is used to reduce the noise of the night scene images, although the main scene area in the night scene images after noise reduction is relatively clear and real, the sky area may have poor purity due to the relatively low noise reduction strength, so that the sky area presents a gray picture effect, and cannot present a deep and thorough picture effect.

[0004] Therefore, in the process of image processing of night scene images, how to reduce the noise in the sky area and thus improve the purity of the sky area without losing the clarity of the main scene area is a problem that needs to be solved urgently. Summary of the invention

[0005] To solve the above problems, the present application provides an image processing method, device, storage medium and program product.

[0006] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the method includes: acquiring a first image to be processed; performing denoising on the first image based on a first denoising algorithm to obtain a second image, and performing denoising on the first image based on a second denoising algorithm to obtain a third image, wherein the denoising intensity of the first denoising algorithm is greater than the denoising intensity of the second denoising algorithm; fusing the image content of a first area in the second image and the image content of the third image except the first area to obtain a fourth image, wherein the first area is an area obtained by removing an area overlapping with the third area from the second area, the second area is an area where the background in the second image obtained based on semantic segmentation is located, and the third area is an area where the main scene in the third image obtained based on an edge extraction algorithm is located.

[0007] In the present application, the first image may be the image P0 mentioned later; the second image may be the image P01 mentioned later; the third image may be the image P02 mentioned later; the fourth image may be the target image P1 mentioned later; the first area may be the background area B2 mentioned later; the second area may be the background area B1 mentioned later; the third area may be the main scene area F1 mentioned later.

[0008] It can be understood that in the present application, the area where the background of the first image is located can be used as the background area, and the area where the shooting target of the first image is located or the area outside the background area can be used as the main scene area. Among them, the first image mentioned in the present application can be any type of image, for example, a seascape image, a desert landscape image, an indoor landscape image, a scenic area image, a person image, a building image, an animal image, etc. Moreover, when the image type of the first image is different, the background area and the main scene area of ​​the first image can also be different. For example, if the first image is a night scene image, its background area can be a sky area, and the main scene area can be an image area outside the sky area such as a building area and a person area. For another example, if the first image is a seascape image, its background area can be a seawater area, and the main scene area can be a beach area. For another example, if the first image is a desert landscape image, its background area can be a desert area, and the main scene area can be a camel caravan area. It can be seen that the present application does not limit the type of the first image and the main scene area and background area of ​​the first image.

[0009] It can also be understood that in the present application, since the second image, the third image, and the fourth image are all obtained based on the first image, the first areas of the images correspond to each other. For example, the first area of ​​the first image, the first area of ​​the second image, the first area of ​​the third image, and the first area of ​​the fourth image have the same position and size. In other words, when the first image, the second image, the third image, and the fourth image are stacked and aligned, the first areas of the images are also aligned. Similarly, the correspondence between other areas of each image (for example, the second area, etc.) can refer to the correspondence between the first areas of each image. For example, when the first image, the second image, the third image, and the fourth image are stacked and aligned, the second areas of each image are also aligned, which will not be repeated here.

[0010] In some embodiments, the first noise reduction algorithm and the second noise reduction algorithm may be any noise reduction algorithm. Furthermore, the first noise reduction algorithm and the second noise reduction algorithm may be noise reduction algorithms of the same type. Among them, for the same noise reduction algorithm, the first noise reduction algorithm and the second noise reduction algorithm with different noise reduction strengths can be obtained by adjusting the noise reduction parameters in the noise reduction algorithm. For example, if the first noise reduction algorithm and the second noise reduction algorithm are both noise reduction algorithms of type A, the type of noise reduction algorithm has a noise reduction parameter A0, and the larger the noise reduction parameter A0, the greater the noise reduction strength. Then the electronic device can set the noise reduction parameter A0 in the first noise reduction algorithm to be greater than the noise reduction parameter A0 in the second noise reduction algorithm, so that the noise reduction strength of the first noise reduction algorithm is greater than the noise reduction strength of the second noise reduction algorithm. In addition, in other embodiments, the above-mentioned first noise reduction algorithm and the second noise reduction algorithm may also be noise reduction algorithms of different types, which is not limited in this application.

[0011] In other embodiments, the first image is denoised using a first denoising algorithm with a greater denoising strength, which can eliminate the noise in the background area to the greatest extent, so that the background area of ​​the second image has a higher degree of purity. Also, the electronic device uses a second denoising algorithm with a smaller denoising strength to denoise the first image, so that the main scene area of ​​the third image can retain more image details while completing the denoising.

[0012] In this way, through the above method, by fusing the first area of ​​the second image and the image content of the third image except the first area, the first area of ​​the fourth image can have a higher purity, and other areas can have more image details and higher clarity while having less noise.

[0013] In a possible implementation of the first aspect above, the above-mentioned fusion of the image content of the first area in the second image and the image content of the third image excluding the first area to obtain the fourth image includes: using the image content of the first area in the second image as the image content of the first area in the fourth image, and using the image content of the third image excluding the first area as the image content of the fourth image excluding the first area.

[0014] In some embodiments, the image content of the first area of ​​the second image and the image content of other areas in the third image may be directly merged. For example, the image content of the background area B2 (as the first area) in the night scene image P01 (as the second image) may be used as the image content of the background area B2 of the night scene image P1 (as the fourth image), and the image content of the night scene image P02 (as the third image) excluding the background area B2 may be used as the image content of the night scene image P1 excluding the background area B2.

[0015] In this way, the fourth image can have the characteristics of the first area of ​​the second image, and can also have the characteristics of the main scene area of ​​the third image.

[0016] In a possible implementation of the first aspect, the second image and the third image include M pixels respectively, where M is an integer greater than or equal to 2; the method of using the image content of the first area in the second image as the image content of the first area in the fourth image, and using the image content of the third image other than the first area as the image content of the fourth image other than the first area, includes: determining a weight map according to a mask corresponding to the first area of ​​the second image, the weight map including M weight values, the M weight values ​​corresponding one-to-one to the M pixels in the second image, and corresponding one-to-one to the M pixels in the third image; wherein the weight value of the pixel corresponding to the first area in the weight map is 1, and the weight value of the pixel outside the first area in the weight map is 0; the fourth image is obtained according to the second image, the third image and the weight map, wherein the pixel data of the i-th pixel (i is a positive integer less than or equal to M) in the fourth image = the pixel data of the i-th pixel in the second image × the i-th weight value in the weight map + the pixel data of the i-th pixel in the third image × (1-the i-th weight value in the weight map).

[0017] It is understood that the pixel data of a pixel may be the brightness value of the pixel, wherein the brightness value is generally data between 0 and 255 (e.g., white is 255 and black is 0). Alternatively, the pixel data may also include the chromaticity value, saturation value, etc. of the pixel, which is not limited in this application.

[0018] In some implementations, after determining the background mask corresponding to the first area of ​​the second image, the electronic device may normalize each data in the background mask so that each data in the background mask is converted to 0 or 1, thereby obtaining a weight map.

[0019] In some embodiments, when the second image and the third image are fused, there is a pixel point in the first area of ​​the second image whose pixel data is A1, and the corresponding weight value of the pixel data A1 in the weight map is 1, and the pixel data of the pixel point corresponding to the position of the pixel data A1 in the third image is A1´, then the pixel data of the pixel point corresponding to the pixel data A1 in the fourth image is A1×1+(1-1)×A1´. For another example, there is a pixel point in the main scene area of ​​the second image whose pixel data is A2, and the corresponding weight value of the pixel data A2 in the weight map is 0, and the pixel point corresponding to the position of the pixel data A2 in the third image is A2´, then the pixel data of the pixel point corresponding to the pixel data A2 in the fourth image can be A2×0+(1-0)×A2´. In this way, the fused fourth image can include both the characteristics of high purity of the background area of ​​the second image and the characteristics of high clarity of the main scene area of ​​the third image.

[0020] In a possible implementation of the first aspect, the second region is acquired in the following manner: determining a fifth region where the background in the second image is located based on semantic segmentation; and dilating the fifth region to obtain the second region.

[0021] In the present application, the fifth area may be the background area B1′ mentioned later.

[0022] In some embodiments, the electronic device can input the second image into a trained semantic segmentation algorithm model. The semantic segmentation algorithm model can identify the fifth region where the background in the second image is located based on the learned knowledge, and represent the recognition result through a binary image. The binary image can be used as the first initial mask corresponding to the fifth region of the second image. It can be understood that after the electronic device obtains the first initial mask corresponding to the fifth region of the second image, it can be said that the electronic device has identified the fifth region of the second image.

[0023] In some implementations, the electronic device may directly use the fifth region of the second image identified by the semantic segmentation algorithm as the second region of the second image.

[0024] In addition, in some other embodiments, the electronic device may also perform an expansion operation on the fifth region of the second image obtained based on the semantic segmentation algorithm to expand the identified background region, and use the expanded background region as the second region of the second image. In this way, the expanded second region can cover more background pixels in the second image.

[0025] In a possible implementation of the first aspect, the third region may be acquired in the following manner: determining a sixth region where a main scene in the third image is located based on an edge extraction algorithm; and performing a closing operation on the sixth region to obtain the third region.

[0026] In the present application, the sixth area may be the main scene area F1′ mentioned later.

[0027] In some embodiments, the electronic device can input the third image into an edge extraction algorithm to obtain a first edge mask corresponding to the main scene area of ​​the third image. It can be understood that after the electronic device obtains the first edge mask corresponding to the main scene area of ​​the third image, it can be said that the electronic device has identified the edge contour of the main scene area of ​​the third image, and can then identify the sixth area where the main scene of the third image is located. Among them, if a certain pixel point can touch the pixel points of the edge contour when it spreads to the surroundings, it can be determined that the pixel point is located within the edge contour. In addition, the pixel points located on the edge contour and the pixel points located within the edge contour can also be regarded as the pixel points in the main scene area. In this way, the sixth area where the main scene is located can be determined based on the edge contour of the main scene area.

[0028] In some implementations, the electronic device may directly use the sixth region where the main scene of the third image is located, which is identified by the edge extraction algorithm, as the third region of the third image.

[0029] In addition, in some other embodiments, the electronic device may also perform a closing operation on the sixth area where the main scene of the third image obtained based on the edge extraction algorithm is located, so as to connect the disconnected parts of the edge area to avoid mistaking them for non-edge parts, and use the main scene area obtained after the closing operation as the third area of ​​the third image. In this way, the edge contour after the connection can be complete and continuous, so that the main scene area closed by the connected edge contour can cover more main scene pixels, and avoid mistaking the broken parts of the edge contour of the original main scene area in the first edge mask for background pixels.

[0030] In a possible implementation of the first aspect above, the first image is a night scene image, the background in the second image is the sky, and the image content of the first area in the second image and the image content of the third image excluding the first area are fused to obtain the fourth image, including: when the contrast of the first area of ​​the second image meets the adjustment condition, adjusting the contrast of the first area of ​​the second image, and fusing the adjusted image content of the first area with the image content of the third image excluding the first area to obtain the fourth image.

[0031] In some embodiments, the electronic device can calculate the brightness information of the first area of ​​the second image, and then determine the contrast of the first area of ​​the second image based on the brightness information. When the contrast of the first area of ​​the second image meets the adjustment condition, the contrast can be adjusted so that the first area of ​​the second image is clearer after adjustment.

[0032] In a possible implementation of the first aspect above, the adjustment condition includes: an absolute value of a difference between a contrast of a first area of ​​the second image and a contrast threshold is greater than a preset difference threshold; and contrast adjustment is performed on the first area of ​​the second image, including: when the contrast of the first area of ​​the second image is less than the contrast threshold, increasing the contrast of the first area of ​​the second image; and when the contrast of the first area of ​​the second image is greater than the contrast threshold, reducing the contrast of the first area of ​​the second image.

[0033] In some embodiments, when the contrast of the first area of ​​the second image is large, the contrast can be reduced; or when the contrast of the second image is small, the contrast can be increased. It can be understood that since the contrast adjustment is performed in the second image, it is irrelevant to the main scene area of ​​the third image, and thus will not affect the image quality of the main scene area of ​​the third image, and further will not affect the image quality of the main scene area of ​​the fourth image.

[0034] Thus, the fourth image obtained by the above method not only ensures the noise reduction effect of the entire image, but also retains the image detail information, thereby improving the image quality. In addition, the electronic device can also adjust the contrast of the first area of ​​the second image without affecting the contrast of the third image, so that the first area and other areas in the fourth image have good clarity.

[0035] In a possible implementation of the first aspect above, when the first image includes a preset target, the image content of the first area in the second image and the image content of the third image except the first area are fused to obtain a fourth image, including: fusing the image content of the first area in the second image, the image content of the fourth area in the fifth image, and the image content of the third image except the first area and the fourth area to obtain the fourth image, wherein the fifth image is obtained by denoising the first image based on a third denoising algorithm, the fourth area is the area where the preset target in the fifth image is located, and the denoising intensity of the third denoising algorithm is less than the denoising intensity of the second denoising algorithm.

[0036] In some embodiments, the electronic device can not only fuse the second image and the third image, but also fuse multiple images. For example, if the first image is a person image, the electronic device usually focuses on the person area when shooting, so the noise of the person area in the first image will be less than the noise of other areas. Therefore, when processing the first image, the electronic device can make the background area (that is, the first area) of the second image have a higher purity after using the first noise reduction algorithm with a larger noise reduction intensity to perform noise reduction on the person image. Then, the electronic device can also use the third noise reduction algorithm with a smaller noise reduction intensity to perform noise reduction on the person image, so that the person area (that is, the fourth area) of the fifth image has a higher clarity. And, the electronic device can also use the second noise reduction algorithm to perform noise reduction on the person image, so that other areas of the third image (such as the building area) can retain more image details while having less noise. In this way, the electronic device can fuse the background area of ​​the second image, the person area of ​​the fifth image, and other areas of the third image, so that each area of ​​the fused fourth image can meet the noise reduction requirements.

[0037] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory being used to store computer program codes / instructions; when the computer program codes / instructions are executed by the processor, the electronic device executes the image processing method mentioned in the present application.

[0038] In a third aspect, the present application provides a readable storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the image processing method mentioned in the present application.

[0039] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method mentioned in the present application.

[0040] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 According to some embodiments of the present application, a schematic diagram of an application scenario of an edge extraction algorithm is shown;

[0042] Figure 2 According to some embodiments of the present application, a schematic diagram of an application scenario for obtaining a background mask is shown;

[0043] Figure 3A According to some embodiments of the present application, a schematic diagram of a stacking process scenario is shown;

[0044] Figure 3B According to some embodiments of the present application, a schematic diagram of a scenario in which a night scene image is denoised using algorithms with different denoising intensities is shown;

[0045] Figure 4 According to some embodiments of the present application, a flow chart of an image processing method is shown;

[0046] Figure 5A According to some embodiments of the present application, a schematic diagram of an application scenario of an image processing method is shown;

[0047] Figure 5B According to some embodiments of the present application, a schematic diagram of a process of obtaining a first area is shown;

[0048] Figure 6 According to some embodiments of the present application, a schematic diagram of an application scenario of an expansion operation is shown;

[0049] Figure 7 According to some embodiments of the present application, a schematic diagram of an application scenario of a closing operation is shown;

[0050] Figure 8 According to some embodiments of the present application, a schematic diagram of a process for processing a night scene image is shown;

[0051] Fig. 9 According to some embodiments of the present application, some processing flow diagrams of purity optimization modules are shown;

[0052] Fig.10 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0053] The illustrative embodiments of the present application include, but are not limited to, an image processing method, an apparatus, a storage medium, and a program product.

[0054] In order to more clearly illustrate the solutions of the embodiments of the present application, the related field terms mentioned in the present application are first explained below.

[0055] (1) Edge extraction algorithm: An image processing technique that detects and determines the edge contours of image content based on the changes in the color or brightness of each pixel in the image. For example, Figure 1 As shown, the electronic device may execute an edge extraction algorithm on the image 101 to obtain the edge contour shown in the image 102 .

[0056] Among them, common edge extraction algorithms may include various operator types such as Sobel operator, Laplacian operator, Canny operator and Prewitt operator. For example, the Sobel operator determines the edge position by calculating the spatial gradient of the brightness of each pixel in the image; the Laplacian operator determines the edge position by calculating the second-order derivative of the brightness of each pixel in the image. This application does not limit the operator type of the edge extraction algorithm.

[0057] (2) Edge Mask: After executing the edge extraction algorithm on the original image to be processed, if the edge detection result is represented by a binary image, then the binary image is the edge mask corresponding to the original image. In the edge mask, the edge part can be white (or other highlight colors), and the non-edge part can be black (or other low-brightness colors). For example, in the above Figure 1 In the figure, after the edge extraction algorithm is executed on the image 101 to be processed, the edge contour shown in the image 102 can be obtained. In which, the image 102 represents the edge contour detection result through a binary image, that is, the edge contour identified by the edge extraction algorithm is represented by white in the figure, and the rest is represented by black. Therefore, the image 102 is the edge mask corresponding to the image 101.

[0058] (3) Semantic segmentation: A computer vision processing task that can classify each pixel in an image to distinguish different objects or regions in the image. For example, a semantic segmentation algorithm model can be trained with a large amount of labeled training data (e.g., a large number of sky pictures) so that the semantic segmentation algorithm model can learn the feature representation of a specific object (e.g., the sky). Next, when the electronic device inputs a new image to be processed (e.g., a night scene image) into the trained semantic segmentation algorithm model, the semantic segmentation algorithm model can identify the target area of ​​the image to be processed based on the learned knowledge, for example, identifying the sky area of ​​a night scene image based on the learned sky feature representation.

[0059] (4) Background mask: After identifying the sky area in the image, if the recognition result is represented by a binary image, then the binary image is the background mask corresponding to the original image. Figure 2 As shown, after the electronic device identifies the background area (specifically the sky area) of the night scene image 201 through a semantic segmentation algorithm, if the identified background area is represented by black and other areas are represented by white, then the binary image 202 is the background mask corresponding to the night scene image 201.

[0060] Similarly, the electronic device can also obtain the regional masks corresponding to other regions of the image, such as human masks, building masks, etc. For example, after the electronic device identifies the human region of the image to be processed through the semantic segmentation algorithm, if the identified human region is represented by color A (such as white) and other regions are represented by color B (such as black), then the binary image is the human mask corresponding to the image to be processed. This application does not limit this.

[0061] (5) Weight map: It is an important parameter in image fusion, and the numerical range of each data in the weight map is usually 0-1. Among them, the weight map corresponding to the original image is the same size as the original image, and each numerical value in the weight map corresponds to each pixel of the original image one-to-one, which is used to represent the weight corresponding to the corresponding pixel. For example, if the original image has a total of M pixels, then the weight map corresponding to the original image has M weight data (each weight data is between 0-1), and the M weight data corresponds to the M pixels of the original image one-to-one.

[0062] Among them, the background mask (or the regional mask corresponding to other areas of the image) is a binary image, and the pixel data of each pixel of the binary image can be the brightness value of the pixel, wherein in the binary image, the brightness value is usually 0 or 255 (e.g., white is 255 and black is 0). In addition, the pixel data can also include the chromaticity value, saturation value, etc. of the pixel, which is not limited in this application. Therefore, the electronic device can perform filtering and normalization processing on the background mask (or the regional mask corresponding to other areas of the image) so as to convert the pixel data of each pixel of the background mask (or the regional mask corresponding to other areas of the image) into a weight map with a numerical range between 0-1.

[0063] (6) Fusion algorithm / network: A neural network that can be used to fuse multiple images into a new image through a weight map. For example, the fusion network can receive image A, image A1, and the weight map corresponding to image A as input data, and fuse image A and image A1 based on the weight map corresponding to image A to obtain a target image. Among them, if the pixel point A´ in image A, the pixel point A1´ in image A1, the weight value w in the weight map corresponding to image A, and the pixel point A2 in the fused target image are one-to-one corresponding, then the pixel data of pixel point A2 can be calculated as: weight w×pixel data of pixel point A´+(1-weight w)×pixel data of pixel point A1´. In addition, the fusion network can be any network, such as a U-Net network, a ResNet network, a transformer network, etc., and this application does not limit this.

[0064] Before inputting the image to be fused and the weight map into the fusion network, the multiple images to be fused and the weight map are usually stacked first, and then the stacked image data are input into the fusion network for fusion. Figure 3A As shown. Figure 3A For example, image a with an image format of H1×W1×3 (height×width×number of channels), image b with an image format of H1×W1×3, and a weight map with an image format of H1×W1×1 can be stacked into image data with an image format of H1×W1×7. The number of channels refers to the number of colors or feature information that each pixel can include. For example, Figure 3A In the above example, both image a and image b can be RGB color images with 3 channels, that is, they have three RGB color channels. The weight map can be a grayscale map with 1 channel, that is, it has one grayscale value channel. Next, the stacked image to be fused and the weight map can be combined. Figure 1 The data are input into the fusion network for fusion processing.

[0065] Combine the following Figure 3B The scene diagram shown in the figure describes the situation in which, when the noise of the night scene image is reduced, the main scene area may be over-noised or the background area (specifically, the sky area) may not be pure enough.

[0066] In the embodiment of the present application, an image may be composed of a background area and a main scene area. The area where the background of the image is located is the background area, and the area where the shooting object of the image is located or the area outside the background area may be the main scene area. Figure 3B , image 301 is a night scene image, the background area of ​​image 301 is the sky area, and the main scene area is the building area.

[0067] As mentioned above, in the night scene images captured by the camera, the noise in the background area (mainly the sky area) is usually greater than the noise in the main scene area. Therefore, in the process of processing the night scene images, if an algorithm with a stronger noise reduction strength is used to reduce the noise of the night scene images, an image with a higher purity of the background area but excessive noise reduction of the main scene area may be obtained; conversely, if an algorithm with a weaker noise reduction strength is used to reduce the noise of the night scene images, an image with a higher clarity of the main scene area but a lower purity of the background area may be obtained.

[0068] For example, Figure 3B As shown, when the mobile phone 10 performs image processing on the night scene image 301, the mobile phone 10 can use an algorithm with a relatively high noise reduction strength to perform noise reduction processing on the night scene image 301 (i.e., strong noise reduction processing), thereby obtaining a noise-reduced image 302. In addition, the mobile phone 10 can also use an algorithm with a relatively low noise reduction strength to perform noise reduction processing on the night scene image 301 (i.e., weak noise reduction processing), thereby obtaining a noise-reduced image 303. Among them, compared with the image 303 obtained by the weak noise reduction processing, the image 302 obtained by the strong noise reduction processing has a background area 3021 that is purer and deeper than the background area 3031, and the main scene area 3032 is clearer and more realistic than the main scene area 3022.

[0069] Therefore, when electronic devices process night scene images, how to reduce the noise in the background area and improve the purity of the background area without losing the clarity of the main scene area is a problem that needs to be solved urgently. Figure 3B In the present invention, how to process the night scene image 301 so that the processed image has the picture effect of the background area 3021 of the image 302 and the picture effect of the main scene area 3032 of the image 303 is a problem that needs to be solved urgently.

[0070] In order to solve the above problems, the present application proposes an image processing method. In the present application, after acquiring the first image to be processed, the electronic device can first use a first noise reduction algorithm with a larger noise reduction strength to perform noise reduction processing on the first image to obtain a second image, so that the background area of ​​the second image can have less noise; and the electronic device can use a second noise reduction algorithm with a smaller noise reduction strength to perform noise reduction processing on the first image to obtain a third image, so that the main scene area of ​​the third image can be clearer and more realistic. For example, the first noise reduction algorithm can be used to perform noise reduction processing on the image P0 (as the first image) to obtain the image P01 (as the second image), and the background area of ​​the image P01 (for example, the sky area, as the first area) is relatively pure; and the second noise reduction algorithm is used to perform noise reduction processing on the image P0 to obtain the image P02 (as the third image), and the main scene area of ​​the image P02 (that is, the area outside the first area) is relatively clear and realistic. Next, the electronic device can merge the image content of the background area in the second image with the image content of other areas in the third image to obtain a fourth image. For example, the image content of the first area of ​​the above-mentioned image P01 and the image content area of ​​the image P02 other than the first area are merged to obtain image P1 (as the fourth image), so that image P1 can have the high purity characteristic of the first area of ​​image P01, and can also have the real and clear characteristics of other areas of image P02.

[0071] exist Figure 3B In the example shown, image P0 may be a night scene image 301, the background area may be a sky area, and the main scene area may be a building area. However, the present application is not limited thereto. Image P0 may also be any other image. For example, in other examples, image P0 may be a seascape image, the background area may be a seawater area, and the main scene area may be a beach area; in other examples, image P0 may be a desert landscape image, the background area may be a desert area, and the main scene area may be a camel caravan area; in other examples, image P0 may be an indoor landscape image, the background area may be a wall area, and the main scene area may be an indoor furnishings area, etc.; in other examples, image P0 may also be a scenic area image, the background area may be a natural landscape area, and the main scene area may be a character area, etc., which will not be described one by one here.

[0072] It can also be understood that in the present application, since the second image, the third image, and the fourth image are all obtained based on the first image, the first areas of the images correspond to each other. For example, the first area of ​​the first image, the first area of ​​the second image, the first area of ​​the third image, and the first area of ​​the fourth image have the same position and size. In other words, when the first image, the second image, the third image, and the fourth image are stacked and aligned, the first areas of the images are also aligned.

[0073] Similarly, the correspondence between other areas of each image (for example, the second area, etc.) can refer to the correspondence between the first areas of each image. For example, when the first image, the second image, the third image, and the fourth image are stacked and aligned, the second areas of each image will also be aligned, which will not be elaborated here.

[0074] In this way, the fourth image obtained by the above-mentioned image processing method not only ensures the noise reduction effect of the image background area, but also retains the image detail information of the main scene area, thereby improving the image quality.

[0075] In other embodiments, the electronic device can not only fuse the second image and the third image, but also fuse multiple images. For example, if the first image includes a preset target with less noise, the electronic device can also perform noise reduction processing on the first image through a third noise reduction algorithm to obtain a fifth image, wherein the noise reduction intensity of the third noise reduction algorithm can be less than the noise reduction intensity of the second noise reduction algorithm, and the fourth area of ​​the fifth image is an area where the preset target (such as a person) is located. Next, the electronic device can fuse the image content of the first area in the second image, the image content of the fourth area in the fifth image, and the image content of the third image except the first area and the fourth area to obtain a fourth image.

[0076] Specifically, when fusing the second image, the third image, and the fifth image, the second image with an image format of H1×W1×3 (height×width×number of channels), the second weight map with a format of H1×W1×1 corresponding to the second image (for example, the weight value of the first area in the second weight image can be 1, and the weight values ​​of other areas can be 0), the third image with a format of H1×W1×3, the fifth image with a format of H1×W1×3, and the fifth weight map with a format of H1×W1×1 corresponding to the fifth image (for example, the weight value of the fourth area in the fifth weight image can be 1, and the weight values ​​of other areas can be 0) can be stacked to form image data with a format of H1×W1×11 and input the image data into the fusion network for fusion. In this way, the fusion network can calculate the pixel data of the i-th pixel of the fourth image by “the i-th weight value of the second weight map × the pixel data of the i-th pixel of the second image + the i-th weight value of the fifth weight map × the pixel data of the i-th pixel of the fifth image + the pixel data of the i-th pixel of the third image × (1-the i-th weight value of the second weight map-the i-th weight value of the fifth weight map)”. Among them, the pixel data of the pixel can be the brightness, hue value, saturation value, etc. of the pixel, which is not limited in this application. In this way, the electronic device can fuse the first area of ​​the second image, the fourth area of ​​the fifth image, and other areas of the third image, so that each area of ​​the fused fourth image can meet the noise reduction requirements.

[0077] For example, if the first image is a person image, the electronic device usually focuses on the person area when shooting, so the noise of the person area in the first image will be less than the noise of other areas. Therefore, when processing the first image, the electronic device can make the background area (that is, the first area) of the second image have a higher purity after using the first noise reduction algorithm with a larger noise reduction intensity to perform noise reduction on the person image. Then, the electronic device can also use the third noise reduction algorithm with a smaller noise reduction intensity to perform noise reduction on the person image, so that the person area (that is, the fourth area) of the fifth image has a higher clarity. And, the electronic device can also use the second noise reduction algorithm to perform noise reduction on the person image, so that other areas of the third image (such as the building area) can retain more image details while having less noise. In this way, the electronic device can fuse the background area of ​​the second image, the person area of ​​the fifth image, and other areas of the third image, so that each area of ​​the fused fourth image can meet the noise reduction requirements.

[0078] In other embodiments, after the electronic device determines the first area of ​​the second image, the electronic device may also obtain contrast information of the first area of ​​the second image, thereby determining whether the contrast of the first area of ​​the second image needs to be adjusted. For example, if the electronic device determines that the contrast of the first area of ​​the second image is small, the electronic device may increase the contrast of the first area; or, if the electronic device determines that the contrast of the first area of ​​the second image is large, the electronic device may reduce the contrast of the first area. In this way, after adjusting the contrast of the first area of ​​the second image, the electronic device may merge the image content of the adjusted first area with the image content of the third image except the first area, thereby obtaining a fourth image.

[0079] In this way, the target image obtained by the above method not only ensures the noise reduction effect of the entire image, but also retains the image detail information, thereby improving the image quality. In addition, the electronic device can also adjust the contrast of the first area of ​​the second image without affecting the contrast of the third image, so that the first area and other areas in the target image have good clarity.

[0080] It can be understood that the first image mentioned in the present application can be any type of image, for example, a night scene image, a person image, an animal image, a building image, a seascape image, a desert landscape image, an indoor landscape image, etc. Moreover, when the image type of the first image is different, the background area and the main scene area of ​​the first image can also be different.

[0081] For example, in some embodiments, the electronic device may use the first noise reduction algorithm to perform noise reduction processing on the character image P0 (as the first image) to obtain the character image P01 (as the second image), and the seascape area (as the first area) of the character image P01 is blue and pure; and use the second noise reduction algorithm to perform noise reduction processing on the character image P0 to obtain the character image P02 (as the third image), and the character area (that is, the area outside the first area) of the character image P02 has a high definition. Next, the electronic device may merge the seascape area of ​​the above-mentioned character image P01 with the character area of ​​the character image P02 to obtain the character image P1 (as the fourth image), so that the character image P1 may have the characteristics of the seascape area of ​​the character image P01, and may also have the characteristics of the character area of ​​the character image P02.

[0082] Similarly, if the first image is a night scene image, the background of the first image may be the sky area, and the main scene may be image content outside the sky area such as buildings and people. For another example, if the first image is a person image, the background of the first image may also be any non-person area such as buildings, sky, desert, etc., and the main scene may be the person area. For another example, if the first image is a building image, the background of the first image may be any non-building area such as the sky, sea view, etc., and the main scene may be the building area. It can be seen that the present application does not limit the type of the first image and the main scene area and background area of ​​the first image.

[0083] In order to facilitate the description of the image processing method provided by the present application, the following embodiments all take the first image as a night scene image and the background as a sky area as an example to describe the image processing method provided by the present application.

[0084] It can be understood that the above-mentioned image processing method of the present application can be applied to any electronic device. Among them, the electronic device includes but is not limited to a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The embodiments of the present application do not limit the specific form of the electronic device.

[0085] The following is based on Figure 4 The flowchart shown in FIG. 1 briefly introduces the image processing method mentioned in the embodiment of the present application. The image processing method can be applied to electronic devices, such as any electronic device such as the mobile phone 10 mentioned above. Figure 4 Specifically, the method is as follows:

[0086] S401: Acquire an image P0 to be processed (as the first image).

[0087] In some embodiments, the image P0 may be any type of image. For example, the image P0 may be Figure 5AThe night scene image 50 is shown. Alternatively, in other embodiments, the image P0 may also be other types of images, such as the seascape image, desert landscape image, indoor landscape image, scenic area image, person image, building image, animal image, etc., which are not limited in the present application.

[0088] In some embodiments, image P0 may be an image taken by an electronic device, an image stored in a gallery of the electronic device, an image sent by other devices, or an image searched by a user through an electronic device. This application does not limit this.

[0089] S402: performing denoising on image P0 based on a first denoising algorithm to obtain image P01 (as a second image), and performing denoising on image P0 based on a second denoising algorithm to obtain image P02 (as a third image), wherein the denoising intensity of the first denoising algorithm is greater than the denoising intensity of the second denoising algorithm.

[0090] In some embodiments, after the electronic device uses a first noise reduction algorithm with a relatively high noise reduction strength to perform noise reduction processing on the image P0, the background area of ​​the obtained image P01 has a relatively high purity. Figure 5A In the embodiment, the electronic device can execute the first noise reduction algorithm on the night scene image 50 to obtain the night scene image 501. The sky area (that is, the background area) of the night scene image 501 presents a deep and clear picture effect.

[0091] Furthermore, after the electronic device uses the second noise reduction algorithm with a weaker noise reduction strength to perform noise reduction processing on the image P0, the main scene area of ​​the obtained image P02 has less noise and higher clarity. Figure 5A In the embodiment, the electronic device may perform a second noise reduction algorithm on the night scene image 50 to obtain a night scene image 502. The building area (that is, the main scene area) of the night scene image 502 has a relatively high definition.

[0092] In some embodiments, the first noise reduction algorithm and the second noise reduction algorithm may be any noise reduction algorithm. Furthermore, the first noise reduction algorithm and the second noise reduction algorithm may be noise reduction algorithms of the same type. For the same type of noise reduction algorithm, the first noise reduction algorithm and the second noise reduction algorithm with different noise reduction strengths may be obtained by adjusting the parameter values ​​in the noise reduction algorithm.

[0093] For example, if the first noise reduction algorithm and the second noise reduction algorithm are both type A noise reduction algorithms, and the type A noise reduction algorithm has a noise reduction parameter A0, and the larger the noise reduction parameter A0 is, the greater the noise reduction strength is. The electronic device can set the noise reduction parameter A0 in the first noise reduction algorithm to be greater than the noise reduction parameter A0 in the second noise reduction algorithm, so that the noise reduction strength of the first noise reduction algorithm is greater than the noise reduction strength of the second noise reduction algorithm.

[0094] In addition, in some other embodiments, the first noise reduction algorithm and the second noise reduction algorithm may also be different types of noise reduction algorithms, which is not limited in the present application.

[0095] S403: Determine a background area B1 (as a second area) in the image P01 based on semantic segmentation.

[0096] In some embodiments, the electronic device can train the semantic segmentation algorithm model through a large amount of labeled training data (e.g., a large amount of sky pictures), so that the semantic segmentation algorithm model can learn the feature representation of the background area. Next, the electronic device can input the image P01 into the trained semantic segmentation algorithm model, and the semantic segmentation algorithm model can identify the background area B1 of the image P01 based on the learned knowledge. For example, Figure 5A In the embodiment, the electronic device can identify the background area (mainly the sky area) of the night scene image 501, that is, the black area of ​​the image 5011, through the semantic segmentation algorithm.

[0097] The specific process of obtaining the background area B1 can be referred to Figure 5B As shown, no further details are given here.

[0098] S404: Determine the main scene region F1 (as the third region) in the image P02 based on an edge extraction algorithm.

[0099] In some embodiments, the electronic device can input the image P02 into an edge extraction algorithm to extract the edge contour of the main scene area F1 of the image P02. If a certain pixel point can touch the pixel points of the edge contour when it spreads to the surroundings, it can be determined that the pixel point is located within the edge contour. In addition, the pixel points located on the edge contour and the pixel points located within the edge contour can also be used as the pixel points of the main scene area F1. In this way, the main scene area F1 can be determined based on the edge contour of the main scene area F1. For example, in Figure 5A In the embodiment, the electronic device can determine the edge contour of the main scene area of ​​the night scene image 502, that is, the white contour of the image 5021, by using an edge extraction algorithm. Next, the electronic device can use the pixel points located on the white contour and the pixel points located within the white contour (that is, the pixel points can all touch the white contour when spreading to the surroundings) as the pixel points within the main scene area of ​​the night scene image 502, thereby determining the main scene area of ​​the night scene image 502.

[0100] The specific process of obtaining the main scene area F1 can be referred to Figure 5B As shown, no further details are given here.

[0101] S405: A background area B2 (as the first area) is obtained by removing the area overlapping with the main scene area F1 from the background area B1.

[0102] In some embodiments, compared with the image recognition effect of the semantic segmentation algorithm, the edge extraction algorithm can more accurately extract the edge contour of the main scene area of ​​the third image. That is to say, the background area B1 obtained by the semantic segmentation algorithm will expand the true background area of ​​the image P01. Therefore, the part overlapping with the main scene area F1 of the image P02 can be removed from the background area B1 of the image P01 to obtain the background area B2. For example, if there is a pixel point A1 (e.g., the coordinate value is (x1, y1)) in the background area B1 of the image P01, then the corresponding pixel point A2 is found at the same position of the image P02 (the coordinate value is (x1, y1)). Among them, if the pixel point A2 is located in the main scene area F1 of the image P02, the electronic device can remove the pixel point A1 from the background area B1 of the image P01. In this way, by traversing each pixel point in the background area B1 of the image P01, the part overlapping with the main scene area F1 of the image P02 can be removed from the background area B1 of the image P01 to obtain the background area B2 of the image P01.

[0103] S406: Merge the image content in the background area B2 in the image P01 and the image content excluding the background area B2 in the image P02 to obtain the image P1 (as the fourth image).

[0104] In some embodiments, the fusion processing operation can be as follows: directly use the image content in the background area B2 in image P01 as the image content of the background area B2 of image P1, and use the image content in image P02 except the background area B2 as the image content in image P1 except the background area B2.

[0105] Specifically, before the electronic device obtains the image P1 through the fusion process, it can first determine the weight map required for the fusion operation based on the background mask corresponding to the background area B2 of the image P01. Figure 5A In the above, each data in the background mask 5013 corresponding to the background area B2 of the image 501 can be normalized so that each data in the background mask 5013 is converted into a weight value of 0 or 1, thereby obtaining a weight map 504. The M (M is a positive integer greater than 2) weight values ​​of the weight map correspond to the M pixels of the image P01; and the M weight values ​​of the weight map correspond to the M pixels of the image P02; and the M weight values ​​of the weight map correspond to the M pixels of the image P1.

[0106] In some embodiments, in the weight image, the weight values ​​corresponding to each pixel of the background area B2 in the image P01 in the weight map can be set to 1, indicating that the corresponding pixel of the image P1 is the corresponding pixel of the image P01; and the weight values ​​corresponding to each pixel of the image P01 except the background area B2 in the weight map can also be set to 0, indicating that the corresponding pixel of the image P1 is the corresponding pixel of the image P02. In this way, based on the weight values ​​in the weight map, the image content in the background area B2 in the image P01 and the image content in the image P02 except the background area B2 can be fused to obtain the image P1. Specifically, the calculation process of the pixel data of the i-th pixel (1≤i≤M and i is a positive integer) in the image P1 is: the pixel data of the i-th pixel in the image P01×the i-th weight value in the weight map+the pixel data of the i-th pixel in the image P02×(1-the i-th weight value in the weight map).

[0107] For example, when performing image fusion, there is pixel data A1 of a certain pixel point in the background area B2 of image P01, the corresponding weight value of the pixel data A1 in the weight map is 1, and the pixel data of the pixel point corresponding to the position of the pixel data A1 in image P02 is A1', then the pixel data of the pixel point corresponding to the pixel data A1 in image P1 is A1×1+(1-1)×A1'. For another example, there is pixel data A2 of a certain pixel point in the main scene area of ​​image P01, the corresponding weight value of the pixel data A2 in the weight map is 0, and the pixel data of the pixel point corresponding to the position of the pixel data A2 in image P02 is A2', then the pixel data of the pixel point corresponding to the pixel data A2 in image P1 can be A2×0+(1-0)×A2'. The pixel data can be the brightness value of the pixel, wherein the brightness value is generally data between 0 and 255 (e.g., white is 255 and black is 0). Alternatively, the pixel data can also include the chromaticity value, saturation value, etc. of the pixel, which is not limited in this application.

[0108] In some other embodiments, the fusion processing operation may also include the following contents: first, the contrast of the image content in the background area B2 in the image P01 is adjusted. For example, if the contrast is less than the contrast threshold, the contrast may be increased; or, if the contrast is greater than the contrast threshold, the contrast may be reduced. Then, the image content in the background area B2 in the image P01 with the adjusted contrast may be used as the image content of the background area B2 of the image P1, and the image content in the image P02 except the background area B2 may be used as the image content in the image P1 except the background area B2.

[0109] In this way, the fused image P1 can include both the deep and pure characteristics of the sky area of ​​the image P01 and the high definition characteristics of the main scene area of ​​the image P02. In addition, the electronic device can also adjust the contrast of the background area B2 of the image P01 without affecting the contrast of the image P02, so that the background area B2 and other areas in the image P1 have good clarity.

[0110] The following is based on Figure 5B The flow chart shown in the figure is Figure 4 The specific implementation process of S403 to S405 in the embodiment is described. The method can be applied to any electronic device, such as the mobile phone 10 mentioned above. Figure 5B Specifically, the method is as follows:

[0111] S501: Determine a background area B1 (as a second area) in the image P01 based on semantic segmentation.

[0112] It can be understood that after the electronic device obtains the mask corresponding to the background area B1 of the image P01, it means that the electronic device has recognized the background area B1 of the image P01. Therefore, in the embodiment of the present application, the background area B1 can be determined by obtaining the mask.

[0113] For example, in Figure 5A In the example, a mask 5011 of the night scene image 501 can be obtained by executing a semantic segmentation algorithm, wherein in the mask 5011, the background area (mainly the sky area) of the night scene image 501 can be represented by black, and the main scene area (mainly the building area) of the night scene image 501 can be represented by white. Thus, it can be seen that after obtaining the mask corresponding to the night scene image 501, the background area of ​​the night scene image 501 can be determined.

[0114] In some embodiments, the electronic device may directly use the background area B1 ′ of the image P01 identified by the semantic segmentation algorithm as the background area B1 of the image P01 .

[0115] In addition, in some other embodiments, the electronic device may also perform a dilation operation on the background area B1' (as the fifth area) of the image P01 obtained based on the semantic segmentation algorithm to expand the identified background area, and use the expanded background area as the background area B1 (as the second area) of the image P01. Specifically, the electronic device may perform a dilation operation on the first initial mask obtained by the semantic segmentation algorithm, thereby implementing a dilation operation on the background area B1' of the first initial mask. For example, Figure 5AIn the example, the background area B1' (ie, the black area) of the first initial mask 5011 can be expanded into the background area B1 of the second initial mask 5012 by dilation. At this time, the background area B1' identified by the electronic device is expanded into the background area B1.

[0116] The dilation operation refers to dilating the background area B1' in the first initial mask so that the background area B1 in the second initial mask obtained after the dilation is larger. For example, the background area B1' in the first initial mask is convolved with the kernel of the dilation operation (hereinafter referred to as the "dilation kernel") to dilate the background area B1'. The dilation kernel can have any shape and size, and has a reference point in the kernel. Furthermore, the dilation kernel can include multiple pixel points, and one of the pixel points can be determined as the reference point.

[0117] Specifically, when the expansion operation is performed on the first initial mask, the reference point in the expansion kernel can slide on the background area B1' of the first initial mask until all the pixels of the background area B1' of the first initial mask are traversed. In each slide, the position occupied by the expansion kernel in the first initial mask can be added to the background area B1 of the expanded second initial mask, thereby obtaining the second initial mask.

[0118] For example, Figure 6As shown, the first initial mask may be a background mask 601, the background area B1' (e.g., the sky area) of the first initial mask may be a black area in the background mask 601, and the expansion kernel of the current expansion operation may be an expansion kernel 602. Specifically, the expansion operation may be performed on the background mask 601 through the following two steps: step (a) and step (b). Step (a): Align the reference point of the expansion kernel 602 with the pixel point A in the background area B1' of the background mask 601, and then add the area occupied by the expansion kernel 602 (e.g., the three shadow areas in step (a)) to the background area B1, thereby obtaining the background mask 601'. At this point, the background area B1 in the background mask 601' has been expanded to three pixels. Step (b): Align the reference point of the expansion kernel 602 with the pixel point B of the original background area B1' in the background mask 601', and then add the area occupied by the expansion kernel 602 (such as the three oblique shaded areas in step (b)) to the background area B1, thereby obtaining the background mask 601''. At this point, the background area B1 in the background mask 601'' has been expanded to five pixels. Next, after executing steps (a) and (b), since all the pixel points of the background area B1' of the background mask 601 have been traversed (the original background area B1' only includes pixel points A and pixel point B), it means that the expansion operation has been performed on the background mask 601, and the background mask 601'' finally obtained is the second initial mask after the expansion operation is performed on the background mask 601.

[0119] It can be understood that for the sake of illustration, the above Figure 6 The background area B1' of the background mask 601 (an example of the first initial mask of the present application) in FIG. 1 only illustrates two pixels. In fact, the background area B1' may also include more pixels, which is not limited in the present application. If the background area B1' of the background mask 601 includes more pixels, the electronic device needs to repeatedly perform Figure 6 That is, the electronic device needs to align the reference point of the expansion kernel 602 with each pixel of the original background area B1' of the background mask 601, and then add the area occupied by the expansion kernel 602 to the background area B1, until all the pixels of the original background area B1' of the background mask 601 are traversed, and the expansion operation is completed to obtain the second initial mask.

[0120] In this way, by performing the dilation operation on the background area B1′ in the first initial mask, the background area B1 of the acquired second initial mask can cover more pixels, thereby preventing some pixels belonging to the background area from being mistaken for pixels in other areas.

[0121] S502: Determine the main scene region F1 (as the third region) in the image P02 based on an edge extraction algorithm.

[0122] Referring to S404 above, after determining the edge contour of the main scene area F1 in the image P02, the pixel points on the edge contour and the pixel points within the edge contour can be used as the pixel points of the main scene area F1, thereby determining the main scene area F1 of the image P02.

[0123] It can be understood that after the electronic device obtains the edge mask corresponding to the main scene area F1 of the image P02, it means that the electronic device has recognized the edge contour of the main scene area F1 of the image P02. Therefore, in the embodiment of the present application, the edge contour of the main scene area F1 can also be determined by obtaining the edge mask.

[0124] For example, in Figure 5A In the embodiment, the edge mask 5021 of the night scene image 502 can be obtained by executing the edge extraction algorithm, wherein the edge contour of the main scene area of ​​the night scene image 502 can be represented by white in the edge mask 5021. Thus, it can be seen that after the edge mask corresponding to the main scene area of ​​the night scene image 502 is obtained, the edge contour of the main scene area of ​​the night scene image 502 can be determined. Furthermore, after the edge contour of the main scene area is determined, the main scene area can be determined.

[0125] In some embodiments, the electronic device may directly use the main scene region F1 ′ of the image P02 identified by the edge extraction algorithm as the main scene region F1 of the image P02 .

[0126] In addition, in some other embodiments, the electronic device may also perform a closing operation on the main scene area F1' (as the sixth area) of the image P02 obtained based on the edge extraction algorithm to connect the disconnected parts of the edge contour to avoid mistaking it for a non-edge contour, and use the main scene area obtained after the closing operation as the main scene area F1 (as the third area) of the image P02. Specifically, the electronic device may perform a closing operation on the first edge mask obtained by the edge extraction algorithm to implement a closing operation on the edge contour of the main scene area F1' corresponding to the first edge mask, thereby obtaining the main scene area F1 after the closing operation. For example, in Figure 5A In the embodiment, the edge contour of the first edge mask 5021 can be closed to obtain the edge contour of the closed second edge mask 5022. Compared with the edge contour of the first edge mask 5021, the edge contour in the second edge mask 5022 is more continuous and closed, so that it is easier to determine the pixel points within the edge contour, and then it is easier to determine the main scene area F1.

[0127] The closing operation refers to connecting the broken parts of the edge contour of the main scene area F1' in the first edge mask so that the connected edge contour is complete and continuous, so that the main scene area F1 enclosed by the connected edge contour can cover more main scene pixels, avoiding mistaking the broken parts of the edge contour of the original main scene area F1' in the first edge mask for background pixels. The kernel of the closing operation (hereinafter described as the "closing operation kernel") can have any shape and size, and there is a reference point in the closing operation kernel. Furthermore, the closing operation kernel can include multiple pixel points, and one of the pixel points can be determined as the reference point.

[0128] Wherein, performing a closing operation on the edge contour of the first edge mask means that the edge contour of the first edge mask is firstly dilated and then eroded. Specifically, when performing a closing operation on the first edge mask, the edge contour of the first edge mask is firstly dilated by a closing operation core. For details, reference may be made to the dilation operation described in S501 above, which will not be described in detail here. Next, after performing a dilation operation on the first edge mask, an erosion operation is performed on the dilated edge mask by a closing operation core. For example, a reference point in a closing operation core may slide on the edge contour of the dilated edge mask until all pixel points of the edge contour of the dilated edge mask are traversed. Wherein, each time the reference point of the closing operation core is slid to the dilated edge contour, if the dilated edge contour cannot completely cover the closing operation core (i.e., a part of the closing operation core is located at the edge contour and a part is located at the non-edge contour), the position occupied by the reference point of the closing operation core may be added to the non-edge contour, thereby obtaining a second edge mask.

[0129] For example, refer to Figure 7 As shown, the first edge mask may be edge mask 701, and the closing operation kernel of the current closing operation may be closing operation kernel 702. The white area in edge mask 701 is the edge contour of the main scene area F1´ identified by the electronic device performing an edge extraction algorithm on image P02. Specifically, when performing a closing operation on edge mask 701, the following two processes, step 1 and step 2, may be included: Step 1: Perform an expansion operation on the edge contour of the main scene area F1´ in edge mask 701 (that is, the white pixel points in edge mask 701) through closing operation kernel 702 to obtain an expanded edge mask 703. The process of the expansion operation may refer to the expansion operation shown in S501 above, which will not be repeated here.

[0130] Next, after performing the dilation operation on the edge mask 701 to obtain the edge mask 703, the following step 2 can be continued to be performed on the edge mask 703: performing an erosion operation on the edge mask 703 obtained after dilation through the closed operation kernel 702. Specifically, the process of the erosion operation includes the following steps: Step (a): aligning the reference point of the closed operation kernel 702 with the pixel point A in the edge contour of the edge mask 703. Since a part of the closed operation kernel 702 is located at the edge contour of the edge mask 703 and the other part is located at the non-edge contour, the edge contour occupied by the closed operation kernel 702 (that is, the pixel point A) can be added to the non-edge contour (that is, added to the black area). Step (b): aligning the reference point of the closed operation kernel 702 with the pixel point B in the edge contour of the edge mask 703. Since a part of the closed operation kernel 702 is located at the edge contour of the edge mask 703 and the other part is located at the non-edge contour, the edge contour occupied by the closed operation kernel 702 (that is, the pixel point B) can be added to the non-edge contour (that is, added to the black area). Step (c): Align the reference point of the closing operation kernel 702 with the pixel point C in the edge contour of the edge mask 703. Since the closing operation kernel 702 can be completely located on the edge contour of the edge mask 703 at this time (that is, the closing operation kernel 702 can be completely in the white area), no operation is performed. Similarly, the electronic device can also align the reference point of the closing operation kernel 702 with other pixel points D, E, F, G, and H in the edge contour of the edge mask 703 respectively. Since the closing operation kernel 702 can be completely located in the edge contour of the edge mask 703 when the reference point is aligned with each pixel point from pixel points D to H respectively, no operation is performed. Therefore, after the corrosion operation is completed on the edge mask 703, the edge mask 704 (an example of the second edge mask of the present application) can be obtained. That is to say, after the closing operation is performed on the edge mask 701, the edge mask 704 can be obtained.

[0131] It can be understood that for the sake of illustration, the above Figure 7 The edge contour (white area) of the edge mask 701 (an example of the first edge mask of the present application) only illustrates 5 pixels. In fact, the edge contour may include more pixels, which is not limited in the present application. When the edge contour in the edge mask 701 includes more edge pixels, the dilation operation and the erosion operation may be performed on more edge pixels through the closing operation kernel 702, which is not described in detail in the present application.

[0132] In this way, by performing a closing operation on the first edge mask corresponding to the main scene region F1′ of the image P02, the disconnected parts of the edge contour in the first edge mask can be connected to avoid being mistaken for non-edge contours.

[0133] S503: A background area B2 (as the first area) is obtained by removing the area overlapping with the main scene area F1 from the background area B1.

[0134] Referring to S501 and S502 above, after the mask corresponding to the image P01 is determined, the background area B1 of the image P01 can be determined. Also, after the edge mask corresponding to the image P02 is determined, the edge contour of the main scene area F1 of the image P02 can be determined, and then the main scene area F1 of the image P02 can be determined. Therefore, in the present application, the portion overlapping with the main scene area F1 of the edge mask corresponding to the image P02 can be removed from the background area B1 of the mask corresponding to the image P01, so as to obtain the background area B2 after removing the portion overlapping with the main scene area F1 from the background area B1. For example, referring to Figure 5A As shown, the electronic device can remove the portion overlapping with the main scene area F1 of the edge mask 5022 from the background area B1 (that is, the black area) of the mask 5012 corresponding to the image 501, thereby obtaining a mask 5013.

[0135] Specifically, for example, if there is a pixel point A1 in the background area B1 of the mask 5012 corresponding to the night scene image 501 (e.g., the coordinate value is (x1, y1)), then the corresponding pixel point A1' is found at the same position of the edge mask 5022 in the night scene image 502 (the coordinate value is (x1, y1)). Among them, if the pixel point A1' is located on the edge contour of the edge mask 5022, the electronic device can remove the pixel point A1 in the background area B1 of the mask 5012. Alternatively, if the pixel points of the edge contour can be encountered when diffusing from the pixel point A1' to the surroundings, it means that the pixel point A1' is surrounded by the edge contour of the edge mask 5022, that is, the pixel point A1' is located in the main scene area F1 constructed by the edge contour, and the electronic device can also remove the pixel point A1 in the background area B1 of the mask 5012.

[0136] In this way, by traversing each pixel point in the background area B1 of the mask corresponding to the image P01, the part overlapping with the main scene area F1 can be removed from the background area B1 of the mask, and the target mask corresponding to the background area B2 of the image P01 can be obtained. At this time, it is equivalent to removing the part overlapping with the main scene area F1 of the image P02 from the background area B1 of the image P01, thereby obtaining the background area B2 of the image P01.

[0137] Combine the following Figure 8 and Fig. 9 The flowchart shown in FIG. 1 takes the image P0 as a night scene image as an example to describe the overall execution process of the image processing method provided by the present application. The image processing method can be applied to electronic devices, such as any electronic device such as the mobile phone 10 mentioned above. Figure 8Specifically, the method is as follows:

[0138] S801: Acquire an image P0 to be processed (as the first image).

[0139] In some embodiments, the background area in the image P0 may be a sky area. The electronic device may obtain the image P0 to be processed in a variety of ways. For example, the electronic device may directly capture the image P0 through a camera application; or the electronic device may receive the image P0 sent by other devices. This application does not limit this.

[0140] S802: Perform noise reduction processing on the image P0 using a first noise reduction algorithm.

[0141] In some embodiments, the first noise reduction algorithm may be a noise reduction algorithm with greater noise reduction strength. For example, if the first noise reduction algorithm has a parameter value A0, and the greater the parameter value A0, the higher the corresponding noise reduction strength, then the parameter value A0 in the first noise reduction algorithm may have a larger value.

[0142] S803: Obtain image P01 (as the second image).

[0143] In some embodiments, after the image P0 is subjected to noise reduction processing by using the first noise reduction algorithm with greater noise reduction strength in step S802, the background area of ​​the obtained image P01 has a higher degree of purity, presenting a deep and thorough picture effect.

[0144] S804: Execute a perception algorithm on the image P01.

[0145] The perception algorithm may be the semantic segmentation algorithm mentioned above. Specifically, the specific process of executing the semantic segmentation algorithm on the image P01 may refer to the above Figure 4 As described in S403, it will not be repeated here.

[0146] S805: Obtain a first initial mask.

[0147] In some embodiments, after the electronic device recognizes the background area B1' of the image P01 (that is, recognizes the sky area) through the semantic segmentation algorithm in the above S804, if the recognition result is represented by a binary image, then the binary image is the first initial mask corresponding to the background area B1' of the image P01. Figure 2 As shown, after the electronic device recognizes the background area (mainly the sky area) of the night scene image 201 through the semantic segmentation algorithm, if the recognized background area is represented by black and other areas are represented by white, then the binary image 202 is the background mask corresponding to the night scene image 201 (which can be used as the first initial mask).

[0148] S806: Perform noise reduction processing on the image P0 using a second noise reduction algorithm.

[0149] In some embodiments, the second noise reduction algorithm may be a noise reduction algorithm with weaker noise reduction strength. For example, if the second noise reduction algorithm has a parameter value A0, and the larger the parameter value A0 is, the higher the corresponding noise reduction strength is, then the parameter value A0 in the second noise reduction algorithm may have a smaller value.

[0150] S807: Obtain image P02 (as the third image).

[0151] In some embodiments, after the image P0 is subjected to noise reduction processing by using the second noise reduction algorithm with weaker noise reduction strength in the above S806, the main scene area of ​​the obtained image P02 has higher clarity.

[0152] S808: Execute edge extraction algorithm on image P02.

[0153] In some embodiments, the electronic device may extract the edge contour of the main scene area F1' of the image P02 by using an edge extraction algorithm. Figure 1 In the example, the electronic device may perform an edge extraction algorithm on the image 101 to obtain the edge contour shown in the image 102 .

[0154] S809: Obtain a first edge mask.

[0155] In some embodiments, after the electronic device extracts the edge contour of the main scene area F1' of the image P02 by the edge extraction algorithm in the above S808, if the recognition result is represented by a binary image, the binary image is the first edge mask corresponding to the main scene area F1' of the image P02. Figure 1 As shown, after the edge extraction algorithm is executed on the image 101 to be processed, the edge contour shown in the image 102 can be obtained. Among them, the image 102 represents the edge contour extraction result through a binary image, that is, the edge contour is represented by white and the rest is represented by black in the image 102. Therefore, the image 102 is the edge mask corresponding to the image 101.

[0156] It can be understood that the present application does not limit the execution order of the above S802 to S809. For example, the electronic device may first execute S802 to S805, and then execute S806 to S809. Alternatively, the electronic device may also first execute S806 to S809, and then execute S802 to S805. Alternatively, the electronic device may also first execute S802 and S803, then execute S806 and S807, and then execute S804 and S805, and finally execute S808 and S809. Alternatively, the electronic device may also first execute S806 and S807, then execute S802 to S805, and then execute S808 and S809. Alternatively, when the electronic device executes S802 to S805, it may also execute S806 to S809 in parallel. It should be understood that the present application does not limit the execution order of the above S802 to S809.

[0157] S810: Input the first edge mask, the first initial mask, the image P01 and the image P02 to a purity optimization module for processing.

[0158] The processing process of the first edge mask, the first initial mask, the image P01 and the image P02 by the purity optimization module can be referred to in the following text. Fig. 9 As shown, no further details are given here.

[0159] S811: Obtain a target image P1 (as a fourth image).

[0160] In some embodiments, the purity optimization module may output a fused target image P1, wherein the fused target image P1 has the picture effect of the background area of ​​the image P01 and the picture effect of the main scene area of ​​the image P02.

[0161] In this way, through the above method, not only the noise reduction effect of the entire image and the purity of the sky area are guaranteed, but also the image detail information of the main scene area is retained, thereby improving the image quality.

[0162] Combine the following Fig. 9 The flowchart shown in FIG. 1 describes the processing process after the electronic device inputs the first edge mask, the first initial mask, the image P01 and the image P02 into the purity optimization module. The image processing method can be applied to electronic devices, such as any electronic device such as the mobile phone 10 mentioned above. Fig. 9 Specifically, the method is as follows:

[0163] S901: Acquire a first edge mask.

[0164] In some embodiments, the electronic device may input the first edge mask obtained in S809 into the purity optimization module. Figure 5AAs shown, the electronic device can obtain the first edge mask 5021 corresponding to the main scene area F1′ of the image 502.

[0165] S902: Obtain a first initial mask.

[0166] In some embodiments, the electronic device may input the first initial mask obtained in S805 into the purity optimization module. Figure 5A As shown, the electronic device can obtain a first initial mask 5011 corresponding to the background area B1′ of the image 501.

[0167] S903: Acquire image P01.

[0168] In some embodiments, the electronic device may input the image P01 obtained in S803 into the purity optimization module. Figure 5A As shown, the electronic device may execute a first noise reduction algorithm on the image 50 to obtain the image 501 .

[0169] S904: Acquire image P02.

[0170] In some embodiments, the electronic device may input the image P02 obtained in S807 into the purity optimization module. Figure 5A As shown, the electronic device may execute a second noise reduction algorithm on the image 50 to obtain the image 502 .

[0171] S905: Perform a closing operation on the first edge mask.

[0172] In some embodiments, performing a closing operation on the first edge mask can connect broken portions of the edge contour in the first edge mask, and can also be used to fill holes or missing portions in the edge contour, thereby making the edge contour more complete and continuous. Figure 5A As shown, the electronic device can perform a closing operation on the first edge mask 5021 to obtain a second edge mask 5022. Compared with the first edge mask 5021, the second edge mask 5022 has no breaks or holes in its edge profile, so that it is more complete and continuous. The specific execution process of the closing operation can refer to the above Figure 5B The above is described in S502, which will not be repeated here.

[0173] S906: Obtain a second edge mask.

[0174] In some embodiments, after the electronic device performs a closing operation on the first edge mask in step S905 , the second edge mask can be obtained.

[0175] S907: Perform a dilation operation on the first initial mask.

[0176] In some embodiments, the dilation operation performed on the first initial mask can be used to expand the sky area (that is, the background area B1′) in the first initial mask, so that the entire sky area of ​​the image P01 can be identified. Figure 5A As shown, the electronic device can perform an expansion operation on the first initial mask 5011 to obtain a second initial mask 5012. Compared with the first initial mask 5011, the second initial mask 5012 has a larger background area (black area) and covers the entire background area of ​​the image P01. The specific execution process of the expansion operation can refer to the above Figure 5B The above is described in S501 and will not be repeated here.

[0177] S908: Obtain a second initial mask.

[0178] In some embodiments, after the electronic device performs the expansion operation on the first initial mask through the above S907, the second initial mask can be obtained.

[0179] It is understood that the present application does not limit the execution order of the above S905 to S908. For example, the electronic device may first execute S905 and S906, and then execute S907 and S908. Alternatively, the electronic device may first execute S907 and S908, and then execute S905 and S906. Alternatively, the electronic device may also execute S907 and S908 in parallel during the execution of S905 and S906. The present application does not limit this.

[0180] S909: Correct the second initial mask using the second edge mask to obtain a first mask.

[0181] In some embodiments, modifying the second initial mask by using the second edge mask means replacing the edge contour in the second initial mask with the edge contour in the second edge mask, so that the second initial mask is more accurate in dividing the background area. Figure 5A As shown, the electronic device can correct the second initial mask 5012 by the second edge mask 5022, thereby obtaining the first mask 5013. The edge contour in the first mask 5013 is similar to the edge contour in the second edge mask 5022, and the first mask 5013 is more accurate in dividing the sky area and the main scene area of ​​the image P01. The specific execution process of the correction operation can refer to the above Figure 5B The above is described in S503, which will not be repeated here.

[0182] S910: Based on the first mask, collect statistics on brightness information of the background area B2 of the image P01.

[0183] In some embodiments, after the electronic device acquires the first mask of the image P01, the electronic device can determine the background area B2 of the image P01, and thus can calculate the brightness information of the background area B2 of the image P01. Figure 5A As shown, after the electronic device obtains the first mask 5013 corresponding to the image P01, the electronic device can determine the background area B2 of the image P01 based on the background area B2 (e.g., black area) of the first mask 5013, thereby counting the brightness of each pixel in the background area B2 of the image P01.

[0184] S911: Adjust the contrast of the background area B2 of the image P01.

[0185] In some embodiments, after the electronic device has counted the brightness information of each pixel point of the background area B2 of the image P01, the electronic device can determine whether the contrast of the background area B2 of the image P01 meets the adjustment condition, and adjust the contrast if the adjustment condition is met. For example, if the absolute value of the difference between the contrast of the background area B2 of the image P01 and the preset contrast threshold is greater than the preset difference threshold, it can be said that the contrast meets the adjustment condition. Among them, if the contrast of the background area B2 of the image P01 is less than the contrast threshold, the electronic device can increase the contrast of the background area B2 of the image P01; or, if the contrast of the background area B2 of the image P01 is greater than the contrast threshold, the electronic device can reduce the contrast of the background area B2 of the image P01. Next, when the electronic device fuses the background area B2 of the image P01 and other areas in the image P02, it can fuse the background area B2 with adjusted contrast and other areas of the image P02 to obtain the image P1.

[0186] It can be understood that since the contrast adjustment is performed on the image P01, it is irrelevant to the main scene area of ​​the image P02, and thus will not affect the image quality of the main scene area of ​​the image P02, and further will not affect the image quality of the main scene area of ​​the target image P1.

[0187] S912: Obtain the image P01′ after contrast adjustment.

[0188] In some embodiments, after the electronic device performs contrast adjustment on the background area B2 of the image P01 through the above S911, an image P01′ after contrast adjustment is obtained. Figure 5A As shown, the electronic device can adjust the contrast of the background area B2 of the image 501 through the first mask 5013, so as to obtain the image 501'.

[0189] S913: Determine a weight map based on the first mask.

[0190] In some instances, since the first mask is a binary image, and the value range of each pixel of the binary image is usually 0 or 255 (e.g., white is 255 and black is 0). Therefore, the electronic device can perform filtering, normalization and other operations on the first mask to convert the first mask into a weight map (Map) in which the value range of each pixel is between 0-1. Among them, filtering can be used to smooth the edge contour in the first mask, so that the boundary in the first mask (e.g., the boundary between the background area and the main scene area) is clearer and continuous; normalization can be used to map each data in the first mask from the range of 0-255 to the range of 0-1. In this way, the electronic device can fuse the background area B2 of the image P01´ and the image area of ​​the image P02 other than the background area B2 based on the weight map to obtain the target image P1.

[0191] It is understood that the present application does not limit the execution order of the above S910 to S913. For example, the electronic device may first execute S910 to S912, and then execute S913. Alternatively, the electronic device may first execute S913, and then execute S910 to S912. Alternatively, the electronic device may also execute S913 in parallel during the process of executing S910 to S912. The present application does not limit this.

[0192] S914: stacking the image P01 ´, the image P02, and the weight map.

[0193] In some embodiments, the electronic device may stack the image P01', the image P02 and the weight map together. For example, the image P01' with an image format of H1×W1×3 (height×width×number of channels), the image P02 with an image format of H1×W1×3, and the weight map with an image format of H1×W1×1 may be stacked into an image with an image format of H1×W1×7. The number of channels refers to the number of colors that each pixel may include. For example, both the image P01' and the image P02 may be red, green, and blue (RGB) color images with a channel number of 3, that is, they have three RGB color channels. The weight map may be a grayscale image with a channel number of 1, that is, it has one grayscale value channel.

[0194] S915: Input the stacked image into the fusion network.

[0195] In some embodiments, after stacking the images, the electronic device can input the stacked images into the fusion network for fusion processing, thereby obtaining a fused target image. The specific fusion process can be referred to above Figure 4 As described in S406, it will not be repeated here.

[0196] S916: Obtain image P1 (as the fourth image).

[0197] In some embodiments, after the fusion network performs fusion processing on the background area B2 of the image P01′ and the area other than the background area B2 in the image P02, a fourth image can be obtained.

[0198] The specific execution process of S914 to S916 can be referred to above. Figure 4 As described in S406, it will not be repeated here.

[0199] Thus, the target image obtained by the above method not only ensures the noise reduction effect of the whole image, so that the background area of ​​the target image has a higher purity, but also retains the image detail information, thereby improving the image quality. In addition, since the contrast adjustment operation is performed in the background area of ​​the image P01, it has nothing to do with the main scene area of ​​the image P02. Therefore, the above method can also adjust the contrast of the background area without affecting the contrast of the main scene area, so that the target image has a better contrast.

[0200] The following takes image P0 as a night scene image as an example. Figure 5A The scene diagram shown in the figure is Figure 8 and Fig. 9 The process of image processing for night scene images is introduced as shown in the figure.

[0201] like Figure 5A As shown, the electronic device can perform a first noise reduction algorithm with a relatively high noise reduction intensity on the image 50 to be processed (an example of the first image of the present application), thereby obtaining an image 501 (an example of the second image of the present application), and the background area of ​​the image 501 is relatively pure and deep. Then, the electronic device can perform a semantic segmentation algorithm on the image 501, thereby obtaining a first initial mask 5011 corresponding to the background area B1´ of the image 501. Next, the electronic device can perform an expansion operation on the first initial mask 5011, thereby obtaining a second initial mask 5012. Among them, the black area used to represent the background area B1 in the second initial mask 5012 accounts for a large proportion, which can cover the entire sky area in the image 501.

[0202] In addition, the electronic device may also perform a second noise reduction algorithm with a weaker noise reduction strength on the image 50 to be processed, thereby obtaining an image 502 (an example of the third image of the present application), and the main scene area F1' of the image 502 is clearer. Then, the electronic device may perform an edge extraction algorithm on the image 502, thereby obtaining a first edge mask 5021 corresponding to the image 502. Next, the electronic device may perform a closing operation on the first edge mask 5021, thereby obtaining a second edge mask 5022. Among them, the edge contour of the main scene area F1 in the second edge mask 5022 is more continuous and complete.

[0203] Next, refer to Figure 5A As shown, the electronic device can correct the second initial mask 5012 through the second edge mask 5022, so that the edge contour in the second initial mask 5012 can be replaced by the edge contour in the second edge mask 5022, so as to obtain the first mask 5013 corresponding to the background area B2 of the image 501. Among them, the edge of the main scene area in the first mask 5013 is more regular and clear. Then, the electronic device can adjust the contrast of the background area B2 of the image 501 through the first mask 5013, so as to obtain the image 501' with adjusted contrast. In addition, the electronic device can convert the first mask 5013 into a weight map 504, wherein in the weight map 504, the weight value corresponding to the background area B2 of the image 501' is 1, and the weight value corresponding to the main scene area of ​​the image 502 is 0. Finally, the electronic device can stack and fuse the image 502, the image 501' and the weight map 504, so as to obtain the target image 51. In this way, the purity of the sky area of ​​the target image 51 can be higher and the clarity of the main scene area can also be higher, so as to have higher image quality.

[0204] It can be understood that the above-mentioned image processing method of the present application can be applied to any electronic device. Among them, the electronic device includes but is not limited to a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The embodiments of the present application do not limit the specific form of the electronic device.

[0205] In some embodiments, the present application provides a readable storage medium, on which instructions are stored, wherein when the instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0206] In some other embodiments, the present application further provides a computer program product, wherein the computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the image processing method mentioned in the present application.

[0207] In some other embodiments, the present application further provides an electronic device, the electronic device comprising a memory and a processor, wherein the memory is coupled to the processor, wherein the memory is used to store computer program codes / instructions, and when the computer program codes / instructions are executed by the processor, the electronic device can execute the image processing method mentioned in the present application.

[0208] Combine the following Fig.10 As shown, taking mobile phone 10 as an example, a hardware structure diagram of an electronic device according to an embodiment of the present application is exemplified.

[0209] like Fig.10As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, a magnetic sensor 180B, an acceleration sensor 180C, an infrared sensor 180D, a temperature sensor 180E, a touch sensor 180F, an ambient light sensor 180G, a light sensor 180H, etc.

[0210] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0211] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), a controller, a video codec, a digital signal processor (DSP), etc. Different processing units may be independent devices or integrated into one or more processors.

[0212] In an embodiment of the present application, the processor 110 may be used to execute the image processing method mentioned in the present application. For example, the processor 110 may execute a first denoising algorithm on the first image to be processed to obtain a second image, and execute a second denoising algorithm on the first image to obtain a third image. In addition, the processor 110 may also obtain a first mask corresponding to the second image, convert the first mask into a weight map, and process the second image, the third image, and the weight map to obtain a target fourth image.

[0213] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0214] The USB interface 130 can be used to connect a charger to charge the mobile phone 10, or to transfer data between the mobile phone 10 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0215] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0216] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the screen 194, the camera 193, and the wireless communication module 160. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0217] The wireless communication function of the mobile phone 10 can be realized by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, and a modulation and demodulation processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 10 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna of a wireless local area network. In some other embodiments, antenna 1 can be used in combination with a tuning switch.

[0218] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the mobile phone 10 .

[0219] The screen 194 is used to display images, videos, etc. The screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), etc. In some embodiments, the mobile phone 10 may include 1 or N screens 194, where N is a positive integer greater than 1.

[0220] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the mobile phone 10 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0221] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 10 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0222] The mobile phone 10 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0223] The gyro sensor 180A may be used to determine the motion posture of the mobile phone 10. In some embodiments, the angular velocity of the mobile phone 10 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180A.

[0224] The magnetic sensor 180B includes a Hall sensor and can be used to detect the strength of the surrounding magnetic field.

[0225] The acceleration sensor 180C can detect the magnitude of the acceleration of the mobile phone 10 in various directions (generally three axes). When the mobile phone 10 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0226] The infrared sensor 180D is used to detect, receive and analyze infrared radiation signals.

[0227] The temperature sensor 180E is used to detect temperature. In some embodiments, the mobile phone 10 uses the temperature detected by the temperature sensor 180E to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180E exceeds a threshold, the mobile phone 10 reduces the performance of a processor located near the temperature sensor 180E to reduce power consumption and implement thermal protection.

[0228] The touch sensor 180F is also called a "touch control device". The touch sensor 180F can be set on the screen 194, and the touch sensor 180F and the screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180F is used to detect touch operations acting on or near it.

[0229] The ambient light sensor 180G is used to sense the brightness of ambient light.

[0230] The light sensor 180H can be used to detect light intensity. It has high sensitivity and fast response speed. It can be used in various lighting environments and provide real-time feedback on light conditions.

[0231] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The mobile phone 10 may receive key input and generate key signal input related to user settings and function control of the mobile phone 10.

[0232] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0233] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0234] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 10 by inserting the SIM card interface 195 or removing the SIM card interface 195. The mobile phone 10 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0235] The embodiments disclosed in the present application may be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0236] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0237] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0238] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0239] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0240] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "include one" do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0241] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: Acquire a first image to be processed; Performing denoising processing on the first image based on a first denoising algorithm to obtain a second image, and performing denoising processing on the first image based on a second denoising algorithm to obtain a third image, wherein the denoising intensity of the first denoising algorithm is greater than the denoising intensity of the second denoising algorithm; The image content of the first area in the second image and the image content of the third image except the first area are merged to obtain a fourth image, wherein the first area is a region obtained by removing the region overlapping with the third area from the second area, the second area is a region where the background in the second image obtained based on semantic segmentation is located, and the third area is a region where the main scene in the third image obtained based on an edge extraction algorithm is located.

2. The method according to claim 1, characterized in that: The step of fusing the image content of the first area in the second image with the image content of the third image excluding the first area to obtain a fourth image includes: using the image content of the first area in the second image as the image content of the first area in the fourth image, and, The image content of the third image except the first area is used as the image content of the fourth image except the first area.

3. The method according to claim 2, characterized in that The second image and the third image each include M pixels, where M is an integer greater than 2; The using the image content of the first area in the second image as the image content of the first area in the fourth image, and using the image content of the third image except the first area as the image content of the fourth image except the first area, comprises: Determine a weight map according to the mask corresponding to the first area of ​​the second image, the weight map including M weight values, the M weight values ​​corresponding one-to-one to the M pixels in the second image, and corresponding one-to-one to the M pixels in the third image; wherein the weight values ​​of the pixels corresponding to the first area in the weight map are 1, and the weight values ​​of the pixels outside the first area in the weight map are 0; The fourth image is obtained according to the second image, the third image and the weight map, wherein the pixel data of the i-th pixel in the fourth image = the pixel data of the i-th pixel in the second image × the i-th weight value in the weight map + the pixel data of the i-th pixel in the third image × (1-the i-th weight value in the weight map), where i is a positive integer less than or equal to M.

4. The method according to claim 1, characterized in that The second area is obtained by: Determine a fifth region where the background in the second image is located based on the semantic segmentation; The fifth region is expanded to obtain the second region.

5. The method according to claim 1, characterized in that The third area is obtained by: Determine a sixth area where a main scene in the third image is located based on the edge extraction algorithm; A closing operation is performed on the sixth region to obtain the third region.

6. The method according to claim 1, characterized in that The first image is a night scene image, the background in the second image is the sky, and the image content of the first area in the second image is fused with the image content of the third image except the first area to obtain the fourth image, including: When the contrast of the first area of ​​the second image meets the adjustment condition, the contrast of the first area of ​​the second image is adjusted, and the image content of the adjusted first area is fused with the image content of the third image except the first area to obtain the fourth image.

7. The method according to claim 6, characterized in that The adjustment condition includes: the absolute value of the difference between the contrast of the first area of ​​the second image and the contrast threshold is greater than a preset difference threshold; and The step of adjusting the contrast of the first area of ​​the second image includes: When the contrast of the first area of ​​the second image is less than the contrast threshold, increasing the contrast of the first area of ​​the second image; When the contrast of the first area of ​​the second image is greater than the contrast threshold, the contrast of the first area of ​​the second image is reduced.

8. The method according to claim 1, characterized in that: In a case where the first image includes a preset target, fusing the image content of the first area in the second image with the image content of the third image excluding the first area to obtain the fourth image, comprises: The image content of the first area in the second image, the image content of the fourth area in the fifth image, and the image content of the third image except the first area and the fourth area are fused to obtain a fourth image, wherein the fifth image is obtained by denoising the first image based on a third denoising algorithm, the fourth area is an area where a preset target in the fifth image is located, and the denoising intensity of the third denoising algorithm is less than the denoising intensity of the second denoising algorithm.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is coupled to the processor; the memory is used to store computer program codes / instructions; when the computer program codes / instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method according to any one of claims 1 to 8.

11. A computer program product, characterized in that include: A computer instruction, when the computer instruction is executed on an electronic device, causes the electronic device to execute the image processing method according to any one of claims 1 to 8.

Images