Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing and digital imaging. The method comprises the following steps: determining a first image and a saliency map of the first image; performing Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image; and sharpening the first image according to the target residual information to obtain a target image. Therefore, according to the scheme, noise elimination and enhancement based on the saliency map can be realized, the target residual information of the first image is determined through Gaussian filtering, and the first image is sharpened by using the target residual information, so that the sharpening effect of the first image can be improved, high-frequency noise can be effectively suppressed, and the overall structure information of the image is reserved.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing and digital imaging technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] During image transmission, low-pass filters can cause overall blurring. Sharpening can enhance detail textures and edge contrast, thereby improving detail and clarity, and image quality. Currently, image sharpening can be performed using frequency domain sharpening and spatial domain sharpening. Summary of the Invention

[0003] The present disclosure provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product to at least address the problems of noise and color cast that are prone to occur during sharpening in the related art. The technical solutions of the present disclosure are as follows: According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising: determining a first image and a saliency map of the first image; performing Gaussian filtering on the first image based on the saliency map to obtain target residual information of pixels in the first image; and performing sharpening processing on the first image based on the target residual information to obtain a target image.

[0004] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, including: a determination module for determining a first image and a saliency map of the first image; a Gaussian filtering module for performing Gaussian filtering on the first image based on the saliency map to obtain target residual information of pixels in the first image; and a sharpening processing module for performing sharpening processing on the first image based on the target residual information to obtain a target image.

[0005] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method described in the first aspect of the embodiment of the present disclosure.

[0006] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method described in the first aspect of the embodiment of the present disclosure are implemented.

[0007] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the embodiment of the present disclosure when executed by a processor of an electronic device.

[0008] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: by determining a first image and a saliency map of the first image, and performing Gaussian filtering on the first image based on the saliency map, target residual information of pixels in the first image can be obtained. Furthermore, the first image can be sharpened based on the target residual information to obtain a target image. Thus, noise enhancement based on the saliency map can be achieved, and by determining the target residual information of the first image through Gaussian filtering and sharpening the first image using the target residual information, the sharpening effect of the first image can be improved, high-frequency noise can be effectively suppressed, and the overall structural information of the image can be preserved.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0011] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment; Figure 2 is a flowchart of an image processing method according to another exemplary embodiment; Figure 3 is a flowchart of an image processing method according to another exemplary embodiment; Figure 4 is a schematic diagram showing image saliency detection effects corresponding to different set thresholds according to an exemplary embodiment; Figure 5 is a schematic diagram of a Gaussian filter pyramid according to an exemplary embodiment; Figure 6a is a schematic diagram of a residual mapping curve according to an exemplary embodiment; Figure 6b is a schematic diagram showing a brightness mapping curve and a contrast mapping curve according to an exemplary embodiment; Figure 7 is a flowchart of processing an image according to an exemplary embodiment; Figure 8 is a block diagram of an image processing apparatus according to an exemplary embodiment; Figure 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0012] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," and the like in the description of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in sequences other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure.

[0014] The acquisition, storage, use, and processing of data in the technical solution disclosed herein are in compliance with the provisions of relevant laws and regulations.

[0015] The image processing method and apparatus according to the embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0016] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method of the embodiment of the present disclosure includes the following steps: S101: Determine a first image and a saliency map of the first image.

[0017] It should be noted that the image processing method of the embodiments of the present disclosure is executed by an electronic device, such as a mobile phone, laptop, desktop computer, vehicle-mounted terminal, smart home appliance, wearable device, server, etc. The image processing method of the embodiments of the present disclosure can be executed by the image processing device of the embodiments of the present disclosure, and the image processing of the embodiments of the present disclosure can be configured in any electronic device to execute the image processing method of the embodiments of the present disclosure.

[0018] In some embodiments, any image can be selected from an image library as the first image, and / or an image can be downloaded from the Internet as the first image, and / or an image can be captured by an image capture device as the first image, and / or an image can be created by a user through software or work as the first image. The first image is an RGB image.

[0019] It is understandable that the saliency map can reflect the sensitivity of the human eye to the image content, and the saliency map can be used to eliminate the residual value enhancement of the unimportant area in the first image to achieve the purpose of suppressing noise and other defects.

[0020] In some embodiments, a saliency map of the first image is determined based on the saliency values ​​by acquiring a grayscale image corresponding to the first image and calculating a saliency value corresponding to each pixel based on the sum of grayscale distances of each pixel in the grayscale image.

[0021] That is, for each pixel in the grayscale image, the saliency value corresponding to each pixel can be calculated by calculating the sum of the distances between all grayscale levels and the grayscale level of the current pixel.

[0022] S102: Perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image.

[0023] In some embodiments, target pixels in the first image that require Gaussian filtering can be determined from the saliency map, and Gaussian filtering can be performed on the target pixels to obtain target residual information of the pixels in the first image. By performing Gaussian filtering on the target pixels, the efficiency of Gaussian filtering can be improved.

[0024] In some embodiments, a saliency value of each pixel in the saliency map may be determined, and pixels whose saliency values ​​are greater than a set threshold are obtained as target pixels. In other words, if the saliency value of a pixel is greater than the set threshold, it can be determined that the pixel needs to be enhanced, and Gaussian filtering is performed on the pixel to obtain target residual information.

[0025] In some embodiments, a Gaussian filter is performed on the target pixel to obtain a residual value corresponding to the target pixel as the target residual information. Optionally, to improve the effect of sharpening the first image using the target residual information, the residual value obtained by the Gaussian filter can be used as the initial first residual information, and the first residual information can be corrected to obtain the corrected residual value as the target residual information.

[0026] In some embodiments, the first residual information can be corrected based on the residual gain corresponding to the first residual information to obtain target residual information. Alternatively, the first residual information can be corrected based on the chrominance component of the target pixel to obtain target residual information, thereby avoiding color cast after sharpening the target residual information. Alternatively, the first residual information can be corrected based on the discrete points of the target pixel to obtain target residual information, thereby eliminating abnormal pixel values.

[0027] For example, a gain mapping curve can be used to determine the residual gain corresponding to the first residual information based on the first residual information. For example, the residual gain can be determined by determining the difference between the target pixel and the center pixel in the chrominance component channel within the sliding window, and based on the difference. For example, residual adjustment information can be determined based on the discrete points of the target pixel, and the first residual information can be corrected based on the adjustment information.

[0028] In some embodiments, when performing Gaussian filtering on the first image, a convolution operation may be performed on the first image using a Gaussian filter pyramid, wherein the Gaussian filter pyramid includes at least two layers of Gaussian filtering, and each layer uses a sliding window of a different size for Gaussian filtering.

[0029] S103: Perform sharpening processing on the first image according to the target residual information to obtain a target image.

[0030] In some embodiments, the target image can be obtained by adding the residual value in the target residual information to the pixel value of the first image to perform sharpening processing on the first image.

[0031] In some embodiments, when Gaussian filtering is performed on the first image, a median filtering result of the first image may also be obtained. The first image may be sharpened based on the target residual information and the median filtering result to obtain the target image. Optionally, the median filtering result may be a median filtered image of the first image.

[0032] That is, the first image may be sharpened based on the target residual information, and noise suppression may be performed on the sharpened image according to the median filtering result, thereby obtaining a target image with good sharpening effect and noise suppression.

[0033] The image processing method provided by the embodiments of the present disclosure can obtain target residual information of pixels in the first image by determining a first image and a saliency map of the first image and performing Gaussian filtering on the first image based on the saliency map. Furthermore, the first image can be sharpened based on the target residual information to obtain a target image. Thus, noise enhancement based on the saliency map can be achieved. By determining the target residual information of the first image through Gaussian filtering and sharpening the first image using the target residual information, the sharpening effect of the first image can be improved, high-frequency noise can be effectively suppressed, and the overall structural information of the image can be preserved.

[0034] Figure 2 is a flowchart of an image processing method according to another exemplary embodiment. Figure 2 As shown, the image processing method of the embodiment of the present disclosure includes the following steps: S201: Determine a first image and a saliency map of the first image.

[0035] For determining the relevant content of the first image in step S201 , please refer to the above embodiment and will not be described in detail here.

[0036] In some embodiments, the saliency map of the first image can be determined based on a grayscale image of the first image. That is, the grayscale image of the first image is determined, and based on the grayscale image, the sum of grayscale distances corresponding to the pixels is determined. Optionally, the formula for determining the sum of grayscale distances corresponding to the pixels is as follows:

[0037] Among them, Sum(x,y) represents the grayscale distance and I(x,y) represents the grayscale value of the pixel at position (x,y). i represents the gray value of the i-th pixel, f n Represents the grayscale frequency, nϵ(0,255).

[0038] Furthermore, the mean grayscale distance of the pixels can be determined based on the sum of the grayscale distances corresponding to the pixels, and the saliency value of the pixels can be determined based on the mean grayscale distance and a set threshold to obtain a saliency map of the first image.

[0039] Optionally, the mean grayscale distance of pixels corresponding to different windows can be determined based on the sum of the grayscale distances of pixels in different windows. The calculation formula is as follows:

[0040] Among them, Mean_Sum(x,y) represents the mean grayscale distance, win x 、win y Indicates the window size.

[0041] Furthermore, the intermediate result of the significant value can be calculated based on the grayscale distance mean, and the significant value of the pixel can be determined based on the intermediate result and a set threshold. Optionally, the formula for calculating the intermediate result of the significant value is as follows:

[0042] Among them, Sx(x,y) represents the intermediate result, a, λ, x0, y min The parameters to be set.

[0043] Optionally, based on the intermediate result and the set threshold, the formula for determining the saliency value of the pixel is as follows:

[0044] Among them, Sal(x,y) represents the significance value, and T is the set threshold.

[0045] S202: Perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image.

[0046] For the relevant content of step S202, please refer to the above embodiment and will not be repeated here.

[0047] S203: Determine a median filtered image of the first image.

[0048] In some embodiments, a median filtered image of the first image can be determined by performing median filtering on the first image. The median filtered image of the first image can be obtained by determining a filtering window size, performing median filtering on pixels within the filtering window, determining a median pixel value, and replacing the center pixel value within the filtering window with the median value until all pixels of the first image are processed.

[0049] In some embodiments, when Gaussian filtering is performed on the first image, the Gaussian filter pyramid may output a filtered luminance component of the first image and a median filtered image of the first image.

[0050] S204 , performing sharpening processing on the first image according to the target residual information and the median filtered image to obtain a target image.

[0051] In some embodiments, the first image may be sharpened based on the target residual information, and the median filtered image may be fused with the sharpened first image to obtain a target image with good sharpening effect and noise suppression. In other words, the median filtered image is used for noise suppression.

[0052] In some embodiments, the first image may be sharpened and enhanced based on the target residual information to obtain a candidate image. For example, the residual value in the target residual information is added to the pixel value of the first image to perform sharpening and enhancement on the first image.

[0053] Furthermore, noise suppression can be performed on the candidate image based on the median filtered image to obtain the target image. For example, the median filtered image can be fused with the candidate image to achieve noise suppression on the candidate image. For example, the median filtered image can be fused with the candidate image using fusion methods such as weighted fusion, adaptive fusion, and multi-scale fusion.

[0054] The image processing method provided by the embodiments of the present disclosure determines a median filtered image of a first image and performs sharpening processing on the first image based on target residual information and the median filtered image to obtain a target image. This method can suppress noise in the target image while ensuring a high sharpening effect, thereby improving noise resistance.

[0055] Figure 3 is a flowchart of an image processing method according to another exemplary embodiment. Figure 3 As shown, the image processing method of the embodiment of the present disclosure includes the following steps: S301: Determine a first image and a saliency map of the first image.

[0056] For the relevant content of step S301, please refer to the above embodiment and will not be repeated here.

[0057] S302 : Determine target pixels in the first image that require Gaussian filtering according to saliency values ​​of pixels in the saliency map.

[0058] In some embodiments, a set threshold corresponding to a saliency value can be determined, and based on the magnitude of the saliency value and the set threshold, a target pixel requiring Gaussian filtering can be determined from the first image. Alternatively, the saliency value of each pixel can be compared with the set threshold, and a pixel can be determined as a target pixel in response to a saliency value of the pixel being greater than the set threshold.

[0059] It is understandable that the size of the threshold can be used to indicate the effect of suppressing defects in the image, that is, the size of the threshold can be used to indicate the effect of image saliency detection. The larger the threshold is, the more obvious the effect of suppressing defects is. Figure 4 Schematic diagram of image saliency detection effects corresponding to different set thresholds shown. Figure 4 The detection effects of setting the threshold Th to 0.55, 0.65 and 0.75 are shown respectively.

[0060] S303 , performing Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel.

[0061] In some embodiments, the target pixel may be Gaussian filtered using a Gaussian filter pyramid to obtain the first residual information, wherein the Gaussian filter pyramid includes at least two layers of Gaussian filters, and the sliding windows of each layer of Gaussian filters have different sizes.

[0062] In some embodiments, Gaussian filtering is performed on the target pixel to obtain a filtered brightness component corresponding to the target pixel, so that the initial first residual information of the target pixel can be determined based on the filtered brightness component of each layer in the Gaussian filter pyramid.

[0063] That is, a convolution operation can be performed layer by layer on a sliding window centered on the target pixel through a Gaussian filter pyramid of at least two layers to obtain a filtered luminance component of the target pixel at each layer, and first residual information can be obtained based on the filtered luminance component and the original luminance component of the target pixel. Alternatively, the first residual information can be obtained by calculating the difference between the filtered luminance component and the original luminance component.

[0064] It can be understood that the Gaussian filter pyramid includes at least two layers of Gaussian filtering. For the first layer in the Gaussian filter pyramid, the first residual information of the first layer corresponding to the target pixel is determined based on the filtered brightness component of the first layer and the original brightness component of the target pixel; for the i-th layer in the Gaussian filter pyramid, the first residual information of the i-th layer corresponding to the target pixel is determined based on the filtered brightness components of the i-1-th layer and the i-th layer, where i is an integer greater than or equal to 1.

[0065] In some embodiments, for the first layer in the Gaussian filter pyramid, a first difference between the filtered luminance component of the first layer and the original luminance component of the target pixel can be calculated as the first residual information of the first layer. For the i-th layer in the Gaussian filter pyramid, a second difference between the filtered luminance component of the i-1-th layer and the filtered luminance component of the i-th layer can be calculated as the first residual information of the i-th layer.

[0066] In some embodiments, in order to avoid amplifying noise and reduce strong edge interference, the difference value can be corrected based on a preset offset to limit the first residual information to a certain range, thereby avoiding excessive residual values ​​in the first residual information.

[0067] That is, a first difference between the original luminance component and the filtered luminance component of the first layer can be determined, and the first difference can be processed based on a preset first offset to obtain first residual information of the first layer. A second difference between the filtered luminance component of the i-1th layer and the filtered luminance component of the i-th layer can be determined, and the second difference can be processed based on a preset second offset to obtain first residual information of the i-th layer.

[0068] Optionally, the first difference and the first offset may be subtracted to obtain a first-layer residual value as the first residual information of the first layer. The second difference and the second offset may be subtracted to obtain an i-th layer residual value as the first residual information of the i-th layer.

[0069] like Figure 5 Schematic diagram of the Gaussian filter pyramid shown. Figure 5 The left side of the middle is an n-layer Gaussian filter pyramid, where the first layer of the Gaussian filter pyramid is The sliding window of The sliding window of ..., the nth layer is Sliding window of . Figure 5In the first and second layers of the medium-Gaussian filter pyramid, the sliding window with the target pixel as the center pixel is convolved layer by layer to obtain the filtered brightness component Lyr0_gs of the first layer and the filtered brightness component Lyr1_gs of the second layer. By determining the original brightness component Lyr_gs of the target pixel, and calculating the first difference between Lyr_gs and Lyr0_gs, and the second difference between Lyr0_gs and Lyr1_gs. The first difference is further processed based on the first offset to obtain the first residual information Lyr0_clp of the first layer, and the second difference is processed based on the second offset to obtain the first residual information Lyr1_clp of the second layer. Figure 5 In the nth layer of the medium-Gaussian filter pyramid, a second difference between the filtered luminance component of the n-1th layer and the filtered luminance component of the nth layer is determined, and the second difference is processed based on the second offset to obtain first residual information Lyrn_clp of the nth layer.

[0070] S304: Correct the first residual information to obtain target residual information.

[0071] In some embodiments, the first residual information includes a residual value, a residual gain corresponding to the residual value can be determined, and the first residual information can be corrected based on the residual gain to obtain target residual information. Alternatively, the first residual information of each layer can be corrected to determine a corresponding residual value, and the corrected residual values ​​of each layer can be added together to obtain a final residual value as the target residual information.

[0072] In some embodiments, Gaussian filtering is performed on the target pixel, and the contrast of the target pixel can be determined based on the filtered brightness component, and the residual gain corresponding to the contrast can be determined, and the first residual information can be corrected based on the residual gain to obtain the target residual information.

[0073] In some embodiments, a residual gain corresponding to the brightness may be determined according to the brightness component, and the first residual information may be corrected based on the residual gain to obtain target residual information.

[0074] That is, a plurality of residual gains may be determined according to the first residual information, and the first residual information may be corrected according to the plurality of residual gains to obtain target residual information.

[0075] In some embodiments, after the first residual information is corrected based on the residual gain, the corrected first residual information may be optimized according to the chrominance component of the target pixel to obtain target residual information.

[0076] In some embodiments, the corrected first residual information can be used as the second residual information, and the optimized second residual information can be used as the third residual information. In order to avoid excessive sharpness and suppress abnormal pixel values, the third residual information can also be adjusted based on the discrete points of the pixel to obtain the target residual information.

[0077] That is to say, when the first residual information is corrected, the contrast of the target pixel is determined, and the first residual information of the target pixel is adjusted according to the filtered luminance component and contrast to obtain the second residual information of the target pixel, and the second residual information is optimized based on the chrominance component of the target pixel to obtain the third residual information of the target pixel, and the residual adjustment information corresponding to the target pixel is determined, and the second residual information or the third residual information is adjusted according to the residual adjustment information to obtain the target residual information, thereby realizing the determination of the target residual information of the target pixel according to the second residual information.

[0078] In some embodiments, the maximum and minimum luminance components within the sliding window can be determined based on the filtered luminance components, and the contrast of the target pixel can be determined based on the maximum and minimum luminance components. For example, the contrast of the target pixel can be determined by calculating the difference between the maximum and minimum luminance components. For example, for the first-layer sliding window, the contrast is Lyr0_contrast, and for the second-layer sliding window, the contrast is Lyr1_contrast.

[0079] In some embodiments, after determining the contrast of the target pixel, multiple residual gains may be determined based on the first residual information, the filtered luminance component, and the contrast. Specifically, a first gain corresponding to the first residual information, a second gain corresponding to the filtered luminance component, and a third gain corresponding to the contrast may be determined.

[0080] Furthermore, the first residual information may be adjusted according to the first gain, the second gain, and the third gain to obtain second residual information of the target pixel. Optionally, the first residual information may be multiplied by the first gain, the second gain, and the third gain to obtain the second residual information of the target pixel.

[0081] For example, the first gain, the second gain, and the third gain may be determined by using a residual mapping curve, a brightness mapping curve, and a contrast mapping curve. Figure 6a The residual mapping curve shown, and Figure 6b Brightness mapping curve and contrast mapping curve shown. Figure 6aThe residual mapping curve 1 includes the first residual information of the first layer and the residual mapping curve 2 corresponding to the first residual information of the i-th layer. If the first residual information is the first residual information of the first layer, the first gain can be determined according to the residual mapping curve 1; if the first residual information is the first residual information of the i-th layer, the first gain can be determined according to the residual mapping curve 2.

[0082] It can be understood that the horizontal axis of the residual mapping curve is the residual value. The steeper the front end of the curve, the faster the smaller residual value can be mapped to a larger value. This part corresponds to the flat area and the area where noise exists. It is easy to amplify the noise while stretching some detailed textures. Therefore, the residual mapping curve is set to suppress the enhancement of noise. Similarly, the strong edge gain needs to be weakened to suppress aliasing.

[0083] The horizontal axis of the contrast mapping curve represents the contrast value within the window. The smaller the contrast, the more likely it is that the current window is located in a flat area or a weak texture area. Conversely, a higher contrast indicates that the window is noisy or located in a strong edge area, which should not be excessively enhanced. Therefore, the contrast mapping curve shows an overall downward trend.

[0084] The horizontal axis of the brightness gain curve represents the Gaussian filter value, which reflects the brightness and darkness changes of the image. To avoid pulling up the dark area noise, the front end of the curve rises as a whole. There is a certain correlation between the sharpness and brightness of the image in terms of visual effects. Areas with high brightness are prone to being too sharp, and it is necessary to suppress the enhancement of the bright field area.

[0085] In some embodiments, to avoid oversharpening caused by high visual enhancement, the second residual information can be optimized based on the chrominance component of the target pixel to obtain third residual information for the target pixel. Alternatively, a target difference between the target pixel and the center pixel in the chrominance component channel within the sliding window can be determined, and a fourth gain can be determined based on the target difference to optimize the second residual information based on the fourth gain to obtain the third residual information. The chrominance component channels can be the U channel and the V channel.

[0086] For example, a mapping table corresponding to different differences and different gains can be established in advance, and the mapping table can be queried through the target difference to determine the fourth gain corresponding to the target difference, so that the residual value in the second residual information can be multiplied by the fourth gain to obtain a new residual value as the third residual information.

[0087] Furthermore, the target residual information can be determined based on the third residual information. Optionally, the third residual information corresponding to each layer in the Gaussian filter pyramid can be determined, and the residual values ​​contained in the third residual information can be added together to obtain a final residual value as the target residual information.

[0088] In some embodiments, to avoid abnormal pixel values, a discrete point operation can be performed on the sliding window to exclude abnormal pixel values ​​within the sliding window. Optionally, the discrete point operation is performed on the sliding window to determine residual adjustment information corresponding to the target pixel, and the second residual information or the third residual information is adjusted based on the residual adjustment information to obtain target residual information.

[0089] In some embodiments, the number of inliers and outliers within a sliding window, as well as the difference between the pixel value within the sliding window and the center pixel value, can be determined, and then, based on this information, residual adjustment information corresponding to the target pixel can be determined. Inliers refer to pixels within a set range of difference from the center pixel; outliers refer to pixels outside the set range of difference from the center pixel.

[0090] For example, the step of determining residual adjustment information may include: 1. Initialize the inlier threshold inlier_th, outlier threshold outlier_th, inlier number threshold inlier_num_th and outlier number threshold outlier_num_th to 0, and initialize out_bit to 8; 2. Count the difference between each pixel value and the center pixel value in the sliding window. When the difference is less than the set inlier threshold inlier_th, the inlier number inlier_num is increased by 1. When the final inlier number inlier_num is greater than the set inlier number threshold inlier_num_th, update out_bit to the sum of the inlier number and the inlier number threshold; When the difference is greater than the set outlier threshold outlier_th, the number of outliers outlier_num increases by 1. When the final number of outliers outlier_num is greater than the set outlier number threshold outlier_num_th, out_bit is updated to the sum of the number of outliers and the outlier number threshold; 3. Use out_bit as residual adjustment information.

[0091] Among them, out_bit can be understood as a bit position, and the residual adjustment information is to adjust the bit position of the second residual information or the third residual information, so as to obtain the target residual information.

[0092] S305: Perform sharpening processing on the first image according to the target residual information to obtain a target image.

[0093] For the relevant content of step S305, please refer to the above embodiment and will not be repeated here.

[0094] The image processing method provided by the embodiment of the present disclosure determines the target pixel from the first image through a saliency map, and performs Gaussian filtering on the target pixel through at least two layers of Gaussian filter pyramid to obtain the initial first residual information of the target pixel, and determines the residual gain corresponding to the first residual information, so as to correct the first residual information according to the residual gain to obtain the second residual information. Furthermore, in order to avoid the problem of excessive sharpness of the residual information, the second residual information can be further optimized based on the chroma component of the target pixel to obtain the third residual information. Furthermore, a sliding window can be used to perform discrete point operations to determine the residual adjustment information, so as to adjust the third residual information according to the residual adjustment information to eliminate the problem of abnormal pixel values ​​and obtain the target residual information.

[0095] Figure 7 The flowchart for image processing is shown. Pixel saliency values ​​are calculated for a first image to obtain a saliency map for the first image. A Gaussian filter pyramid is then established. Target pixels in the saliency map whose pixel saliency values ​​are greater than a set threshold are input into the Gaussian filter pyramid for layer-by-layer convolution. This results in filtered luminance components for the target pixels at each layer. First residual information is then determined based on the filtered luminance components and the original luminance components of the target pixels.

[0096] A first gain corresponding to the first residual information, a second gain corresponding to the filtered luminance component, and a third gain corresponding to the contrast are determined, and the first residual information is corrected based on the first gain, the second gain, and the third gain to obtain target residual information. The first image is further sharpened based on the target residual information to obtain a candidate image, and noise suppression is performed on the candidate image based on the median filtered image to obtain a target image.

[0097] Optionally, when correcting the first residual information, the first residual information can be corrected based on the first gain, the second gain and the third gain to obtain the second residual information, and the second residual information can be optimized according to the chrominance component of the target pixel to obtain the third residual information, and the third residual information can be further adjusted based on the discrete points of the pixel to obtain the target residual information.

[0098] Figure 8 FIG. 1 is a block diagram of an image processing apparatus according to an exemplary embodiment. Figure 8 The image processing apparatus 800 according to an embodiment of the present disclosure includes: A determination module 801 is configured to determine a first image and a saliency map of the first image; A Gaussian filtering module 802 is configured to perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image; The sharpening processing module 803 is configured to perform sharpening processing on the first image according to the target residual information to obtain a target image.

[0099] In one embodiment of the present disclosure, the sharpening processing module 803 is further configured to: determine a median filtered image of the first image; and perform sharpening processing on the first image according to the target residual information and the median filtered image to obtain a target image.

[0100] In one embodiment of the present disclosure, the sharpening processing module 803 is further used to: perform sharpening enhancement processing on the first image according to the target residual information to obtain a candidate image; and perform noise suppression on the candidate image based on the median filtered image to obtain a target image.

[0101] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: determine the target pixel in the first image that requires Gaussian filtering based on the saliency value of the pixel in the saliency map; perform Gaussian filtering on the target pixel to obtain the initial first residual information of the target pixel; and correct the first residual information to obtain the target residual information.

[0102] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: in response to a saliency value of a pixel being greater than a set threshold, determine that the pixel is a target pixel.

[0103] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: perform a layer-by-layer convolution operation on a sliding window centered on a target pixel through at least two layers of a Gaussian filtering pyramid to obtain a filtered brightness component of the target pixel at each layer; and obtain first residual information based on the filtered brightness component and the original brightness component of the target pixel.

[0104] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: for the first layer in the Gaussian filtering pyramid, determine the first residual information of the first layer corresponding to the target pixel based on the filtered brightness component of the first layer and the original brightness component of the target pixel; for the i-th layer in the Gaussian filtering pyramid, determine the first residual information of the i-th layer corresponding to the target pixel based on the filtered brightness components of the i-1-th layer and the i-th layer, where i is an integer greater than or equal to 1.

[0105] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: determine a first difference between the original luminance component and the filtered luminance component of the first layer, and process the first difference based on a preset first offset to obtain first residual information of the first layer; determine a second difference between the filtered luminance component of the i-1th layer and the filtered luminance component of the i-th layer, and process the second difference based on a preset second offset to obtain first residual information of the i-th layer.

[0106] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: determine the contrast of the target pixel; adjust the first residual information of the target pixel according to the filtered brightness component and contrast to obtain the second residual information of the target pixel; and determine the target residual information of the target pixel based on the second residual information.

[0107] In one embodiment of the present disclosure, the Gaussian filtering module 802 is further used to: determine a first gain corresponding to the first residual information; determine a second gain corresponding to the filtered brightness component; determine a third gain corresponding to the contrast; and adjust the first residual information according to the first gain, the second gain, and the third gain to obtain second residual information of the target pixel.

[0108] In one embodiment of the present disclosure, the Gaussian filter module 802 is further configured to: determine a maximum brightness component and a minimum brightness component within the sliding window; and determine a contrast of the target pixel according to the maximum brightness component and the minimum brightness component.

[0109] In one embodiment of the present disclosure, the Gaussian filter module 802 is further configured to: optimize the second residual information based on the chrominance component of the target pixel to obtain third residual information of the target pixel; and determine target residual information based on the third residual information.

[0110] In one embodiment of the present disclosure, the Gaussian filter module 802 is further used to: perform discrete point operations on the sliding window to determine residual adjustment information corresponding to the target pixel; and adjust the second residual information or the third residual information according to the residual adjustment information to obtain target residual information.

[0111] In one embodiment of the present disclosure, the determination module 801 is further used to: determine a grayscale image of the first image; determine the sum of grayscale distances corresponding to pixels based on the grayscale image; determine the mean grayscale distance of the pixels based on the sum of the grayscale distances corresponding to the pixels; and determine the saliency value of the pixels based on the mean grayscale distance and a set threshold to obtain a saliency map of the first image.

[0112] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] The image processing device provided by the embodiments of the present disclosure can obtain target residual information of pixels in the first image by determining a first image and a saliency map of the first image and performing Gaussian filtering on the first image based on the saliency map. Furthermore, the first image can be sharpened based on the target residual information to obtain a target image. Thus, noise enhancement based on the saliency map can be achieved. By determining the target residual information of the first image through Gaussian filtering and sharpening the first image using the target residual information, the sharpening effect of the first image can be improved, high-frequency noise can be effectively suppressed, and the overall structural information of the image can be preserved.

[0114] Figure 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0115] like Figure 9 As shown, the electronic device 900 includes: The memory 901 and the processor 902 , and the bus 903 connecting different components (including the memory 901 and the processor 902 ), the memory 901 stores a computer program, and when the processor 902 executes the program, the image processing method described in the embodiment of the present disclosure is implemented.

[0116] Bus 903 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0117] The electronic device 900 typically includes a variety of electronic device-readable media, which can be any available media that can be accessed by the electronic device 900, including volatile and non-volatile media, removable and non-removable media.

[0118] The memory 901 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 904 and / or cache memory 905. The electronic device 900 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 906 may be used to read and write non-removable, non-volatile magnetic media ( Figure 9 Not shown, usually called a "hard drive"). Although Figure 9Although not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 903 via one or more data media interfaces. Memory 901 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present disclosure.

[0119] A program / utility 908 having a set (at least one) of program modules 907 may be stored, for example, in memory 901. Such program modules 907 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 907 generally implement the functions and / or methods described in the embodiments of the present disclosure.

[0120] The electronic device 900 may also communicate with one or more external devices 909 (e.g., a keyboard, a pointing device, a display 991, etc.), one or more devices that enable a user to interact with the electronic device 900, and / or any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 992. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 993. Figure 9 As shown, the network adapter 993 communicates with other modules of the electronic device 900 via the bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0121] The processor 902 executes various functional applications and data processing by running programs stored in the memory 901 .

[0122] It should be noted that the implementation process and technical principles of the electronic device of this embodiment can be found in the aforementioned explanation of the image processing method of the embodiment of the present disclosure, and will not be repeated here.

[0123] In order to implement the above embodiments, the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of the image processing method provided by the present disclosure when the program instructions are executed by a processor.

[0124] Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0125] In order to implement the above embodiments, the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned image processing method when executed by a processor of an electronic device.

[0126] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0127] It will be understood that the present disclosure is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. An image processing method, characterized in that: The method comprises: determining a first image and a saliency map of the first image; performing Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image; According to the target residual information, the first image is sharpened to obtain a target image; the sharpening process is performed on the first image according to the target residual information to obtain the target image, including: determining a median filtered image of the first image; The first image is sharpened according to the target residual information and the median filtered image to obtain the target image.

2. The method according to claim 1, characterized in that The step of performing sharpening processing on the first image according to the target residual information and the median filtered image to obtain the target image includes: Performing sharpening and enhancement processing on the first image according to the target residual information to obtain a candidate image; Based on the median filtered image, noise suppression is performed on the candidate image to obtain the target image.

3. The method according to claim 1 or 2, characterized in that The step of performing Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image includes: Determining target pixels requiring Gaussian filtering in the first image according to saliency values ​​of pixels in the saliency map; Performing Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel; The first residual information is corrected to obtain the target residual information.

4. The method according to claim 3, characterized in that The determining, according to the saliency values ​​of the pixels in the saliency map, target pixels in the first image that require Gaussian filtering includes: In response to the saliency value of the pixel being greater than a set threshold, the pixel is determined to be the target pixel.

5. The method according to claim 3, characterized in that The performing Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel includes: Performing a convolution operation layer by layer on a sliding window centered on the target pixel through at least two layers of a Gaussian filter pyramid to obtain a filtered brightness component of the target pixel at each layer; The first residual information is obtained according to the filtered brightness component and the original brightness component of the target pixel.

6. The method according to claim 5, characterized in that The obtaining the first residual information according to the filtered brightness component and the original brightness component of the target pixel includes: For a first layer in the Gaussian filter pyramid, determining first residual information of the first layer corresponding to the target pixel according to a filtered luminance component of the first layer and an original luminance component of the target pixel; For the i-th layer in the Gaussian filter pyramid, determine the first residual information of the i-th layer corresponding to the target pixel based on the filtered brightness components of the i-1-th layer and the i-th layer, where i is an integer greater than or equal to 1.

7. The method according to claim 6, characterized in that The method further comprises: Determine a first difference between the original luminance component and the filtered luminance component of the first layer, and process the first difference based on a preset first offset to obtain first residual information of the first layer; Determine a second difference between the filtered luminance component of the (i-1)th layer and the filtered luminance component of the i-th layer, and process the second difference based on a preset second offset to obtain first residual information of the i-th layer.

8. The method according to claim 5, characterized in that The correcting the first residual information to obtain the target residual information includes: determining a contrast ratio of the target pixel; Adjusting the first residual information of the target pixel according to the filtered brightness component and contrast to obtain second residual information of the target pixel; Determine target residual information of the target pixel according to the second residual information.

9. The method according to claim 8, characterized in that The adjusting the first residual information of the target pixel according to the filtered brightness component and contrast to obtain second residual information of the target pixel includes: determining a first gain corresponding to the first residual information; Determining a second gain corresponding to the filtered brightness component; determining a third gain corresponding to the contrast; The first residual information is adjusted according to the first gain, the second gain, and the third gain to obtain second residual information of the target pixel.

10. The method according to claim 8, characterized in that Determining the contrast of the target pixel includes: Determining a maximum brightness component and a minimum brightness component within the sliding window; The contrast of the target pixel is determined according to the maximum brightness component and the minimum brightness component.

11. The method according to claim 8, characterized in that The determining target residual information of the target pixel according to the second residual information includes: Optimizing the second residual information based on the chrominance component of the target pixel to obtain third residual information of the target pixel; The target residual information is determined according to the third residual information.

12. The method according to claim 11, characterized in that The method further comprises: Performing a discrete point operation on the sliding window to determine residual adjustment information corresponding to the target pixel; The second residual information or the third residual information is adjusted according to the residual adjustment information to obtain the target residual information.

13. The method according to any one of claims 1 to 2, characterized in that Determining a saliency map of the first image includes: determining a grayscale image of the first image; Determining the sum of grayscale distances corresponding to pixels according to the grayscale image; Determining a mean grayscale distance of the pixel according to the sum of the grayscale distances corresponding to the pixel; The saliency value of the pixel is determined according to the grayscale distance mean and a set threshold to obtain a saliency map of the first image.

14. An image processing device, characterized in that: The device comprises: a determination module, configured to determine a first image and a saliency map of the first image; a Gaussian filtering module, configured to perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image; a sharpening processing module, configured to perform sharpening processing on the first image according to the target residual information to obtain a target image; the sharpening processing module is further configured to: determining a median filtered image of the first image; The first image is sharpened according to the target residual information and the median filtered image to obtain the target image.

15. The device according to claim 14, characterized in that The Gaussian filtering module is further used to: Determining target pixels requiring Gaussian filtering in the first image according to saliency values ​​of pixels in the saliency map; Performing Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel; The first residual information is corrected to obtain the target residual information.

16. The device according to claim 15, characterized in that The Gaussian filtering module is further used to: Performing a convolution operation layer by layer on a sliding window centered on the target pixel through at least two layers of a Gaussian filter pyramid to obtain a filtered brightness component of the target pixel at each layer; The first residual information is obtained according to the filtered brightness component and the original brightness component of the target pixel.

17. The device according to claim 16, characterized in that The Gaussian filtering module is further used to: determining a contrast ratio of the target pixel; Adjusting the first residual information of the target pixel according to the filtered brightness component and contrast to obtain second residual information of the target pixel; Determine target residual information of the target pixel according to the second residual information.

18. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Implement the steps of the method according to any one of claims 1 to 13.

19. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images