Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image processing method and device, electronic equipment and storage medium, belonging to the field of image processing and digital imaging technology. The method comprises: determining a first image and a saliency map of the first image; performing Gaussian filtering on the first image according to the saliency map to obtain target residual information of pixels in the first image; and performing sharpening processing on the first image according to the target residual information to obtain a target image. Thus, the present scheme can exclude noise enhancement based on the saliency map, determine the target residual information of the first image through Gaussian filtering, and use the target residual information to sharpen the first image, which can improve the sharpening effect of the first image, effectively suppress high-frequency noise, and preserve the overall structural information of the image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, digital imaging technology, and particularly relates to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the transmission process of an image, the image is prone to overall blurring caused by a low-pass filter. Sharpening can enhance the detail texture and edge contrast, thereby improving the detail performance and clarity and improving the image quality. At present, an image can be sharpened through frequency domain sharpening and spatial domain sharpening. SUMMARY

[0003] The present disclosure provides an image processing method, device, electronic equipment, computer readable storage medium and computer program product to at least solve the problem of noise and color deviation in sharpening in the related art. The technical solutions of the present disclosure are as follows.

[0004] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including: determining a first image and a saliency map of the first image; performing Gaussian filtering on the first image according to the saliency map to obtain target residual information of a pixel in the first image; and performing sharpening processing on the first image according to the target residual information to obtain a target image.

[0005] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, including: a determination module configured to determine a first image and a saliency map of the first image; a Gaussian filtering module configured to perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of a pixel in the first image; and a sharpening processing module configured to perform sharpening processing on the first image according to the target residual information to obtain a target image.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic equipment is provided, including a processor; a memory for storing processor-executable instructions; and wherein the processor is configured to implement the steps of the method according to the first aspect of the present disclosure.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer readable storage medium is provided, having stored thereon computer program instructions which, when executed by a processor, implement the steps of the method according to the first aspect of the present disclosure.

[0008] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program which, when executed by a processor of an electronic equipment, implements the steps of the method according to the first aspect of the present disclosure.

[0009] The embodiments of the present disclosure provide at least the following beneficial effects: by determining the first image and the saliency map of the first image, and performing Gaussian filtering on the first image according to the saliency map, the target residual information of the pixels in the first image can be obtained. Further, the first image can be sharpened according to the target residual information to obtain a target image. Thus, the noise enhancement based on the saliency map can be excluded, the target residual information of the first image is determined through Gaussian filtering, and the first image is sharpened using the target residual information, which can improve the sharpening effect of the first image, effectively suppress high-frequency noise, and retain the overall structural information of the image.

[0010] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure and, do not limit the present disclosure.

[0012] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment;

[0013] Figure 2 is a flowchart of an image processing method according to another exemplary embodiment;

[0014] Figure 3 is a flowchart of an image processing method according to another exemplary embodiment;

[0015] Figure 4 is a schematic diagram of image saliency detection effects corresponding to different threshold settings according to an exemplary embodiment;

[0016] Figure 5 is a schematic diagram of a Gaussian filtering pyramid according to an exemplary embodiment;

[0017] Figure 6a is a schematic diagram of a residual mapping curve according to an exemplary embodiment;

[0018] Figure 6b is a schematic diagram of a luminance mapping curve and a contrast mapping curve according to an exemplary embodiment;

[0019] Figure 7 is a flowchart of an image processing method according to an exemplary embodiment;

[0020] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment;

[0021] Figure 9 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0022] In order for the ordinary person in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings.

[0023] It should be noted that the terms "first", "second" and the like in the specification and above drawings of the present disclosure are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure.

[0024] The acquisition, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with the relevant legal regulations.

[0025] The image processing method and device of the embodiments of the present disclosure will be described below with reference to the drawings.

[0026] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment, as shown in Figure 1 The image processing method of the embodiments of the present disclosure includes the following steps:

[0027] S101, determining a first image and a saliency map of the first image.

[0028] It should be noted that the execution subject of the image processing method of the embodiments of the present disclosure is an electronic device, such as a mobile phone, a notebook, a desktop computer, a vehicle terminal, a smart home appliance, a wearable device, a server, etc. The image processing method of the embodiments of the present disclosure can be executed by the image processing device of the embodiments of the present disclosure, and the image processing of the embodiments of the present disclosure can be configured in any electronic device to execute the image processing method of the embodiments of the present disclosure.

[0029] In some embodiments, any image can be selected from an image library as the first image, and / or an image can be downloaded from the network as the first image, and / or an image can be collected based on an image collection device as the first image, and / or an image created by a user through software or work as the first image. Wherein, the first image is an RGB image.

[0030] It can be understood that the saliency map can reflect the sensitivity of the human eye to the content of the image, and the residual value of the unimportant area in the first image can be enhanced by eliminating the saliency map to achieve the purpose of suppressing noise and other defects.

[0031] In some embodiments, the saliency map of the first image is determined based on a saliency value of each pixel in the saliency map. The saliency value of each pixel in the saliency map is calculated based on a distance sum of all gray levels in the gray image to the gray level of the pixel.

[0032] That is, for each pixel in the gray image, the saliency value of the pixel is calculated by calculating the distance sum of all gray levels to the gray level of the pixel.

[0033] In S102, the first image is Gaussian filtered according to the saliency map to obtain target residual information of the pixels in the first image.

[0034] In some embodiments, the target pixels in the first image that need to be Gaussian filtered can be determined from the saliency map, and the target pixels are Gaussian filtered to obtain the target residual information of the pixels in the first image. By Gaussian filtering the target pixels, the efficiency of Gaussian filtering can be improved.

[0035] In some embodiments, the saliency value of each pixel in the saliency map can be determined, and the pixels with a saliency value greater than a set threshold are obtained as target pixels. That is, if the saliency value of a pixel is greater than a set threshold, it can be determined that the pixel needs to be enhanced, and the pixel is Gaussian filtered to obtain the target residual information.

[0036] In some embodiments, Gaussian filtering the target pixels can obtain a residual value corresponding to the target pixel as the target residual information. Alternatively, in order to improve the effect of the target residual information on the sharpening of the first image, the residual value obtained by Gaussian filtering can be used as the initial first residual information, and the first residual information is modified to obtain the modified residual value as the target residual information.

[0037] In some embodiments, the first residual information can be modified according to a residual gain corresponding to the first residual information to obtain the target residual information. Alternatively, the first residual information can also be modified according to the chroma component of the target pixel to obtain the target residual information, so as to avoid the occurrence of color deviation after sharpening the target residual information. Alternatively, the first residual information can also be modified according to the discrete points of the target pixel to obtain the target residual information, so as to exclude the problem of abnormal pixel value.

[0038] For example, the gain mapping curve can be used to determine the residual gain corresponding to the first residual information according to the first residual information. For example, the difference between the target pixel and the center pixel in the chroma component channel within the sliding window can be determined, and the residual gain can be determined according to the difference. For example, the adjustment information of the residual is determined according to the discrete points of the target pixel, so as to modify the first residual information according to the adjustment information.

[0039] In some embodiments, when performing the Gaussian filtering on the first image, a convolution operation can be performed on the first image by a Gaussian filtering pyramid. The Gaussian filtering pyramid includes at least two layers of Gaussian filtering, and each layer uses a sliding window of different size to perform the Gaussian filtering.

[0040] S103, performing sharpening processing on the first image according to the target residual information to obtain a target image.

[0041] In some embodiments, the sharpening processing on the first image can be performed by adding the residual value in the target residual information to the pixel value of the first image to obtain the target image.

[0042] In some embodiments, when performing the Gaussian filtering on the first image, a median filtering result of the first image can also be obtained, and the target image can be obtained by performing the sharpening processing on the first image according to the target residual information and the median filtering result. Optionally, the median filtering result can be a median filtering image of the first image.

[0043] That is, the sharpening processing can be performed on the first image based on the target residual information, and the noise suppression can be performed on the image after the sharpening processing based on the median filtering result, so that the target image with good sharpening effect and noise suppression can be obtained.

[0044] The image processing method provided by the embodiments of the present disclosure can obtain the target residual information of the pixels in the first image by determining the first image and the saliency map of the first image, and performing the Gaussian filtering on the first image according to the saliency map. Further, the target image can be obtained by performing the sharpening processing on the first image according to the target residual information. Therefore, the noise enhancement based on the saliency map can be excluded, the target residual information of the first image can be determined by the Gaussian filtering, and the first image can be sharpened by using the target residual information, so that the sharpening effect of the first image can be improved, the high-frequency noise can be effectively suppressed, and the overall structural information of the image can be preserved.

[0045] Figure 2 is a flowchart of an image processing method according to another exemplary embodiment, as shown in Figure 2 The image processing method of the embodiments of the present disclosure includes the following steps:

[0046] S201, determining a first image and a saliency map of the first image.

[0047] The related content of the first image determined in step S201 can be referred to the above embodiments, which will not be described here.

[0048] In some embodiments, a saliency map of the first image can be determined based on the gray image of the first image. That is, by determining the gray image of the first image, and according to the gray image, the gray level distance sum of a pixel is determined. Optionally, the formula for determining the gray level distance sum of a pixel is as follows:

[0049]

[0050] wherein Sum(x, y) represents the gray level distance sum, I(x, y) represents the gray value of the pixel at position (x, y), and I i represents the gray value of the i-th pixel, f n represents the gray level frequency, and n e (0, 255).

[0051] Further, according to the gray level distance sum of a pixel, the gray level distance mean of the pixel can be determined, and according to the gray level distance mean and a set threshold value, the saliency value of the pixel can be determined to obtain the saliency map of the first image.

[0052] Optionally, according to the gray level distance sum of the pixels in different windows, the gray level distance mean of the pixels corresponding to different windows can be determined, and the calculation formula is as follows:

[0053]

[0054] wherein Mean_Sum(x, y) represents the gray level distance mean, win x , win y represents the representative window size.

[0055] Further, the intermediate result of the saliency value can be calculated according to the gray level distance mean, and according to the intermediate result and a set threshold value, the saliency value of the pixel can be determined. Optionally, the formula for calculating the intermediate result of the saliency value is as follows:

[0056]

[0057] wherein Sx(x, y) represents the intermediate result, a, λ, x0, y min are set parameters.

[0058] Optionally, according to the intermediate result and a set threshold value, the formula for determining the saliency value of the pixel is as follows:

[0059]

[0060] wherein Sal(x, y) represents the saliency value, and T is a set threshold value.

[0061] S202, according to the saliency map, Gaussian filtering is performed on the first image to obtain the target residual error information of the pixels in the first image.

[0062] For details regarding step S202, please refer to the above embodiments, which will not be repeated here.

[0063] S203, determine the median-filtered image of the first image.

[0064] In some embodiments, a median-filtered image of the first image can be determined by performing median filtering on the first image. This can be achieved by determining the size of the filtering window, performing median filtering on the pixels within the filtering window to determine the median value of each pixel, and replacing the center pixel value within the filtering window with this median value, until all pixels of the first image have been processed.

[0065] In some embodiments, when Gaussian filtering is applied to the first image, the Gaussian filtering pyramid can output the filtered luminance component of the first image and the median filtered image of the first image.

[0066] S204. Based on the target residual information and the median filtered image, the first image is sharpened to obtain the target image.

[0067] In some embodiments, the first image can be sharpened based on the target residual information, and then the median-filtered image can be fused with the sharpened first image to obtain a target image with good sharpening effect and noise suppression. That is, the median-filtered image is used for noise suppression.

[0068] In some embodiments, the first image can be sharpened and enhanced based on the target residual information to obtain a candidate image. For example, the residual value in the target residual information can be added to the pixel value of the first image to achieve sharpening and enhancement of the first image.

[0069] Furthermore, noise suppression can be applied to candidate images based on the median-filtered image to obtain the target image. For example, the median-filtered image can be fused with the candidate image to suppress noise in the candidate image. For instance, fusion methods such as weighted fusion, adaptive fusion, and multi-scale fusion can be used to fuse the median-filtered image with the candidate image.

[0070] The image processing method provided in the embodiments of this disclosure determines a median-filtered image of a first image and sharpens the first image based on the target residual information and the median-filtered image to obtain a target image. Therefore, while ensuring a high level of sharpening effect on the target image, noise suppression of the target image can be achieved, thereby improving its resistance to noise.

[0071] Figure 3 This is a flowchart illustrating an image processing method according to another exemplary embodiment, such as...Figure 3 As shown, the image processing method of the embodiments of the present disclosure includes the following steps:

[0072] S301, determining a first image and a saliency map of the first image.

[0073] The related content of step S301 can be referred to the above embodiments, which will not be described here.

[0074] S302, determining a target pixel in the first image that needs to be Gaussian filtered according to a saliency value of a pixel in the saliency map.

[0075] In some embodiments, the target pixel in the first image that needs to be Gaussian filtered can be determined by determining a set threshold corresponding to the saliency value, and according to the size of the saliency value and the set threshold. Optionally, by comparing the saliency value of each pixel with the size of the set threshold, in response to the saliency value of the pixel being greater than the set threshold, the pixel is determined as the target pixel.

[0076] It can be understood that the size of the set threshold can be used to indicate the suppression effect on the defects in the image, that is, the size of the set threshold can be used to indicate the image saliency detection effect. Among them, the greater the set threshold, the more obvious the suppression effect on the defects. For example, Figure 4 The schematic diagram of the image saliency detection effect corresponding to different set thresholds. Figure 4 The detection effects of the set thresholds Th of 0.55, 0.65 and 0.75 are respectively shown in the middle.

[0077] S303, Gaussian filtering the target pixel to obtain initial first residual information of the target pixel.

[0078] In some embodiments, the target pixel can be Gaussian filtered by a Gaussian filtering pyramid to obtain the first residual information. Among them, the Gaussian filtering pyramid includes at least two layers of Gaussian filtering, and the size of the sliding window of each layer of Gaussian filtering is different.

[0079] In some embodiments, Gaussian filtering the target pixel can obtain a filtered luminance component corresponding to the target pixel, so that the initial first residual information of the target pixel can be determined according to the filtered luminance component of each layer in the Gaussian filtering pyramid.

[0080] That is, the sliding window centered on the target pixel can be subjected to layer-by-layer convolution operation by the Gaussian filtering pyramid with at least two layers, to obtain the filtered luminance component of the target pixel at each layer, and the first residual information can be obtained according to the filtered luminance component and the original luminance component of the target pixel. Optionally, the first residual information can be obtained by calculating the difference between the filtered luminance component and the original luminance component.

[0081] It can be understood that at least two layers of Gaussian filtering are included in the Gaussian filtering pyramid. For the first layer in the Gaussian filtering pyramid, first residual information corresponding to the target pixel in the first layer is determined according to the filtered luminance component of the first layer and the original luminance component of the target pixel. For the i-th layer in the Gaussian filtering pyramid, first residual information corresponding to the target pixel in the i-th layer is determined according to the filtered luminance components of the (i-1)-th layer and the i-th layer, where i is an integer greater than or equal to 1.

[0082] In some embodiments, for the first layer in the Gaussian filtering pyramid, a first difference value between the filtered luminance component of the first layer and the original luminance component of the target pixel can be calculated as the first residual information of the first layer. For the i-th layer in the Gaussian filtering pyramid, a second difference value between the filtered luminance component of the (i-1)-th layer and the filtered luminance component of the i-th layer can be calculated as the first residual information of the i-th layer.

[0083] In some embodiments, in order to avoid amplifying noise and reducing strong edge interference, the difference value can be corrected based on a preset offset to limit the first residual information within a certain range, thereby avoiding the residual value in the first residual information being too large.

[0084] That is, the first difference value between the original luminance component and the filtered luminance component of the first layer can be determined, and the first difference value can be processed based on a preset first offset to obtain the first residual information of the first layer. The second difference value between the filtered luminance component of the (i-1)-th layer and the filtered luminance component of the i-th layer can be determined, and the second difference value can be processed based on a preset second offset to obtain the first residual information of the i-th layer.

[0085] Optionally, the first difference value and the first offset can be subtracted to obtain a residual value of the first layer as the first residual information of the first layer. The second difference value and the second offset can be subtracted to obtain a residual value of the i-th layer as the first residual information of the i-th layer.

[0086] As shown in the schematic diagram of the Gaussian filtering pyramid. Figure 5 Figure 5 The left side in the middle is an n-layer Gaussian filtering pyramid, where the first layer in the Gaussian filtering pyramid is a sliding window of , the second layer is a sliding window of , and the n-th layer is a sliding window of . For the first layer in the Gaussian filtering pyramid, Figure 5 ​The first layer and the second layer in the Gaussian filtering pyramid are convoluted layer by layer to obtain the filtered luminance component Lyr0 gs of the first layer and the filtered luminance component Lyr1 gs of the second layer. The original luminance component Lyr gs of the target pixel is determined, and the first difference between Lyr gs and Lyr0 gs and the second difference between Lyr0 gs and Lyr1 gs are calculated. The first difference is processed based on the first offset to obtain the first residual information Lyr0 clp of the first layer, and the second difference is processed based on the second offset to obtain the first residual information Lyr1 clp of the second layer. For the n-th layer in the Gaussian filtering pyramid, the second difference between the filtered luminance component of the (n-1)-th layer and the filtered luminance component of the n-th layer is determined, and the second difference is processed based on the second offset to obtain the first residual information Lyrn clp of the n-th layer. Figure 5 The first layer and the second layer in the Gaussian filtering pyramid are convoluted layer by layer to obtain the filtered luminance component Lyr0 gs of the first layer and the filtered luminance component Lyr1 gs of the second layer. The original luminance component Lyr gs of the target pixel is determined, and the first difference between Lyr gs and Lyr0 gs and the second difference between Lyr0 gs and Lyr1 gs are calculated. The first difference is processed based on the first offset to obtain the first residual information Lyr0 clp of the first layer, and the second difference is processed based on the second offset to obtain the first residual information Lyr1 clp of the second layer. For the n-th layer in the Gaussian filtering pyramid, the second difference between the filtered luminance component of the (n-1)-th layer and the filtered luminance component of the n-th layer is determined, and the second difference is processed based on the second offset to obtain the first residual information Lyrn clp of the n-th layer.

[0087] In some embodiments, the first residual information includes a residual value, a residual gain corresponding to the residual value can be determined, and the first residual information is corrected based on the residual gain to obtain the target residual information. Alternatively, the first residual information of each layer can be corrected to determine a corresponding residual value, and the final residual value is obtained by adding the corrected residual value of each layer as the target residual information.

[0088] In some embodiments, the target pixel is filtered by Gaussian filtering, the contrast of the target pixel can be determined according to the filtered luminance component, and the residual gain corresponding to the contrast is determined, and the first residual information is corrected based on the residual gain to obtain the target residual information.

[0089] In some embodiments, the residual gain corresponding to the luminance can also be determined according to the luminance component, and the first residual information is corrected based on the residual gain to obtain the target residual information.

[0090] In some embodiments, the residual gain corresponding to the luminance can also be determined according to the luminance component, and the first residual information is corrected based on the residual gain to obtain the target residual information.

[0091] That is, a plurality of residual gains can be determined according to the first residual information, and the first residual information is corrected according to the plurality of residual gains to obtain the target residual information.

[0092] In some embodiments, after the first residual information is corrected based on the residual gain, the corrected first residual information can also be optimized according to the chrominance component of the target pixel to obtain the target residual information.

[0093] In some embodiments, the first modified residual information can be taken as the second residual information, and the second optimized residual information can be taken as the third residual information. In order to avoid over-sharpening and abnormal pixel value, the third residual information can be adjusted based on the discrete points of the pixel to obtain the target residual information.

[0094] That is, when the first residual information is modified, the contrast of the target pixel is determined, and the first residual information of the target pixel is adjusted according to the filtered luminance component and the contrast to obtain the second residual information of the target pixel. The second residual information is optimized based on the chrominance component of the target pixel to obtain the third residual information of the target pixel. The residual adjustment information corresponding to the target pixel is determined, and the second residual information or the third residual information is adjusted according to the residual adjustment information to obtain the target residual information. Thus, the target residual information of the target pixel is determined according to the second residual information.

[0095] In some embodiments, the maximum luminance component and the minimum luminance component in the sliding window can be determined based on the filtered luminance component, and the contrast of the target pixel can be determined according to the maximum luminance component and the minimum luminance component. For example, the difference between the maximum luminance component and the minimum luminance component is calculated to determine the contrast of the target pixel. For example, the contrast of the first layer sliding window is Lyr0_contrast, and the contrast of the second layer sliding window is Lyr1_contrast.

[0096] In some embodiments, after the contrast of the target pixel is determined, a plurality of residual gains can be determined based on the first residual information, the filtered luminance component, and the contrast. That is, the first gain corresponding to the first residual information can be determined, the second gain corresponding to the filtered luminance component can be determined, and the third gain corresponding to the contrast can be determined.

[0097] Further, the first residual information can be adjusted according to the first gain, the second gain, and the third gain to obtain the second residual information of the target pixel. Alternatively, the first gain, the second gain, and the third gain can be multiplied by the first residual information to obtain the second residual information of the target pixel.

[0098] For example, the first gain, the second gain, and the third gain can be determined by a residual mapping curve, a luminance mapping curve, and a contrast mapping curve. For example, the residual mapping curve is shown in FIG. 6A, the luminance mapping curve is shown in FIG. 6B, and the contrast mapping curve is shown in FIG. 6C. Figure 6a Figure 6b For example, the residual mapping curve is shown in FIG. 6A, the luminance mapping curve is shown in FIG. 6B, and the contrast mapping curve is shown in FIG. 6C. Figure 6a ​The first residual information corresponding to the first residual mapping curve 1 of the first layer and the second residual mapping curve 2 of the i-th layer are included, and if the first residual information is the first residual information of the first layer, the first gain can be determined according to the first residual mapping curve 1; if the first residual information is the first residual information of the i-th layer, the first gain can be determined according to the second residual mapping curve 2.

[0099] It can be understood that the horizontal coordinate of the residual mapping curve is the residual value, the steeper the front end of the curve, the smaller the residual value can be quickly mapped to a larger value, and this part corresponds to the flat area and the area where noise exists, and the noise is easily amplified while the details of the stretched part are textured, so the residual mapping curve is set to suppress the enhancement of the noise, and for the same reason, the strong edge gain needs to be weakened to suppress the jaggies.

[0100] The horizontal coordinate of the contrast mapping curve represents the contrast value in the window, and the smaller the contrast, the more it indicates that the current window is located in the flat area or weak texture area, and vice versa, which indicates that the window exists noise or is located in the strong edge area, and this part should not be over-enhanced, so the overall contrast mapping curve presents a downward trend.

[0101] The horizontal coordinate of the brightness gain curve represents the Gaussian filtering value, reflecting the light and dark changes of the image, to avoid lifting the noise in the dark area, the front end of the curve rises as a whole, and the sharpness and brightness of the image are related in visual effect, and the over-sharpness is prone to occur in the bright area, and the bright area needs to be suppressed.

[0102] In some embodiments, to avoid over-sharpness caused by high degree of visual enhancement, the second residual information can be optimized based on the chroma component of the target pixel to obtain third residual information of the target pixel. Optionally, a target difference value between the target pixel and the center pixel point in the chroma component channel in the sliding window can be determined, and a fourth gain can be determined through the target difference value, so as to optimize the second residual information according to the fourth gain to obtain the third residual information. The chroma component channel can be a U channel and a V channel.

[0103] For example, a mapping table corresponding to different difference values and different gains can be established in advance, and the fourth gain corresponding to the target difference value can be determined by querying the mapping table through the target difference value, so that the residual value in the second residual information can be multiplied by the fourth gain to obtain a new residual value as the third residual information.

[0104] Further, the target residual information can be determined according to the third residual information. Optionally, the third residual information corresponding to each layer in the Gaussian filtering pyramid can be determined, and the residual values contained in the third residual information can be added to obtain the final residual value as the target residual information.

[0105] In some embodiments, in order to avoid the problem of abnormal pixel value, the sliding window can be operated on discrete points to exclude abnormal pixel values in the sliding window. Alternatively, by operating the sliding window on discrete points, the residual adjustment information corresponding to the target pixel is determined, and the second residual information or the third residual information is adjusted according to the residual adjustment information to obtain the target residual information.

[0106] In some embodiments, the number of inner points and the number of outer points in the sliding window, and the difference between the pixel values in the sliding window and the center pixel value can be determined, and then according to the above information, the residual adjustment information corresponding to the target pixel can be determined. The inner point refers to a pixel point within the set range of the difference from the center pixel point; the outer point refers to a pixel point outside the set range of the difference from the center pixel point.

[0107] For example, the step of determining the residual adjustment information can include:

[0108] 1. Initialize the inlier_th, outlier_th, inlier_num_th and outlier_num_th to 0, and initialize the out_bit to 8;

[0109] 2. Count the difference between each pixel value in the sliding window and the center pixel value, and when the difference is less than the set inlier_th, the inlier_num is increased by 1, and when the final inlier_num is greater than the set inlier_num_th, the out_bit is updated to the sum of the inlier_num and the inlier_num_th;

[0110] When the difference is greater than the set outlier_th, the outlier_num is increased by 1, and when the final outlier_num is greater than the set outlier_num_th, the out_bit is updated to the sum of the outlier_num and the outlier_num_th;

[0111] 3. The out_bit is used as the residual adjustment information.

[0112] Wherein, the out_bit can be understood as a bit, and the residual adjustment information is an adjustment to the bit of the second residual information or the third residual information, so as to obtain the target residual information.

[0113] S305, according to the target residual information, the first image is sharpened to obtain the target image.

[0114] The related content of step S305 can be referred to the above embodiments, which will not be repeated here.

[0115] The image processing method provided by the embodiments of the present disclosure determines a target pixel from a first image through a saliency map, and performs Gaussian filtering on the target pixel through a Gaussian filtering pyramid with at least two layers to obtain initial first residual information of the target pixel, and determines residual gain corresponding to the first residual information, so as to modify the first residual information according to the residual gain to obtain second residual information. Further, in order to avoid the problem of over-sharpness of the residual information, the second residual information can be further optimized based on the chroma component of the target pixel to obtain third residual information. Further, a sliding window can be used for discrete point operation to determine residual adjustment information, so as to adjust the third residual information according to the residual adjustment information to eliminate the problem of abnormal pixel value and obtain target residual information.

[0116] Figure 7 A flowchart of processing an image is shown. The first image is subjected to pixel saliency value calculation to obtain a saliency map of the first image, and a Gaussian filtering pyramid is established, so that the target pixel with a pixel saliency value greater than a set threshold in the saliency map is input into the Gaussian filtering pyramid for layer-by-layer convolution operation to obtain the filtered luminance component of the target pixel at each layer. Further, the first residual information is determined according to the filtered luminance component and the original luminance component of the target pixel.

[0117] The first gain corresponding to the first residual information, the second gain corresponding to the filtered luminance component, and the third gain corresponding to the contrast are determined, and the first residual information is modified according to the first gain, the second gain and the third gain to obtain target residual information. Further, the first image is subjected to sharpening processing according to the target residual information to obtain a candidate image, and the candidate image is subjected to noise suppression based on a median filtering image to obtain a target image.

[0118] Optionally, when the first residual information is modified, the first residual information can be modified based on the first gain, the second gain and the third gain to obtain second residual information, and the second residual information can be optimized according to the chroma component of the target pixel to obtain third residual information, and the third residual information can be further adjusted based on the discrete points of the pixels to obtain the target residual information.

[0119] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment. Referring to Figure 8 , the image processing device 800 of the embodiments of the present disclosure comprises:

[0120] A determination module 801 is configured to determine a first image and a saliency map of the first image.

[0121] A Gaussian filtering module 802 is configured to perform Gaussian filtering on the first image according to the saliency map to obtain target residual information of the pixels in the first image.

[0122] The sharpening processing module 803 is configured to perform sharpening processing on the first image according to the target residual information, to obtain a target image.

[0123] In an embodiment of the present disclosure, the sharpening processing module 803 is further configured to determine a median filtering image of the first image, and perform sharpening processing on the first image according to the target residual information and the median filtering image, to obtain the target image.

[0124] In an embodiment of the present disclosure, the sharpening processing module 803 is further configured to perform sharpening enhancement processing on the first image according to the target residual information, to obtain a candidate image, and perform noise suppression on the candidate image based on the median filtering image, to obtain the target image.

[0125] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to determine a target pixel in the first image that needs to be Gaussian filtered according to the saliency value of the pixel in the saliency map, perform Gaussian filtering on the target pixel, to obtain initial first residual information of the target pixel, and correct the first residual information, to obtain the target residual information.

[0126] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to determine the pixel as the target pixel in response to the saliency value of the pixel being greater than a set threshold.

[0127] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to perform a layer-by-layer convolution operation on a sliding window centered on the target pixel by at least two layers of Gaussian filtering pyramids, to obtain a filtered luminance component of the target pixel at each layer, and obtain the first residual information according to the filtered luminance component and an original luminance component of the target pixel.

[0128] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to, for a first layer in the Gaussian filtering pyramids, determine the first residual information of the first layer corresponding to the target pixel according to the filtered luminance component of the first layer and the original luminance component of the target pixel, and for an i-th layer in the Gaussian filtering pyramids, determine the first residual information of the i-th layer corresponding to the target pixel according to the filtered luminance components of the i-1-th layer and the i-th layer, where i is an integer greater than or equal to 1.

[0129] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to determine a first difference value between the original luminance component and the filtered luminance component of the first layer, and process the first difference value based on a preset first offset, to obtain the first residual information of the first layer, and determine a second difference value between the filtered luminance component of the i-1-th layer and the filtered luminance component of the i-th layer, and process the second difference value based on a preset second offset, to obtain the first residual information of the i-th layer.

[0130] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: determine a contrast of the target pixel; adjust the first residual information of the target pixel according to the filtered luminance component and the contrast to obtain second residual information of the target pixel; and determine the target residual information of the target pixel according to the second residual information.

[0131] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: determine a first gain corresponding to the first residual information; determine a second gain corresponding to the filtered luminance component; determine a third gain corresponding to the contrast; and adjust the first residual information according to the first gain, the second gain and the third gain to obtain the second residual information of the target pixel.

[0132] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: determine a maximum luminance component and a minimum luminance component in the sliding window; and determine the contrast of the target pixel according to the maximum luminance component and the minimum luminance component.

[0133] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: optimize the second residual information based on a chrominance component of the target pixel to obtain third residual information of the target pixel; and determine the target residual information according to the third residual information.

[0134] In an embodiment of the present disclosure, the Gaussian filtering module 802 is further configured to: perform a discrete point operation on the sliding window to determine residual adjustment information corresponding to the target pixel; and adjust the second residual information or the third residual information according to the residual adjustment information to obtain the target residual information.

[0135] In an embodiment of the present disclosure, the determining module 801 is further configured to: determine a grayscale image of the first image; determine a sum of gray level distances corresponding to the pixel according to the grayscale image; determine a mean value of the sum of gray level distances corresponding to the pixel; and determine a saliency value of the pixel according to the mean value of the sum of gray level distances and a set threshold to obtain a saliency map of the first image.

[0136] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in details in the embodiments of the method, and thus will not be described in details here.

[0137] The image processing apparatus provided by the embodiments of the present disclosure can obtain target residual information of pixels in the first image by determining the first image and a saliency map of the first image, and performing Gaussian filtering on the first image according to the saliency map. Further, the first image can be sharpened according to the target residual information to obtain a target image. Thus, the saliency map can be used to exclude noise enhancement, the target residual information of the first image can be determined by Gaussian filtering, and the first image can be sharpened using the target residual information, so that the sharpening effect of the first image can be improved, high-frequency noise can be effectively suppressed, and overall structural information of the image can be preserved.

[0138] Figure 9 is a block diagram of an electronic device according to an example embodiment.

[0139] As shown in Figure 9 , the electronic device 900 includes:

[0140] The memory 901 and the processor 902, the bus 903 connecting different components (including the memory 901 and the processor 902), the memory 901 stores a computer program, and the processor 902 executes the program to realize the image processing method described in the embodiments of the present disclosure.

[0141] The bus 903 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus, and peripheral component interconnect (PCI) bus.

[0142] The electronic device 900 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 900, including volatile and nonvolatile media, removable and non-removable media.

[0143] The memory 901 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 904 and / or cache memory 905. The electronic device 900 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 906 can be used for reading from and writing to non-removable, non-volatile magnetic media (e.g. Figure 9 (not shown), commonly referred to as a "hard disk drive"). Although Figure 9As shown in FIG. 9, the electronic device 900 can include a bus 903 which is a communication medium that interconnects the components of the electronic device 900 and permits the components to communicate with each other. The electronic device 900 also includes one or more processing units (CPU's) 902, a main memory 901 and a system memory 908. As will be appreciated by those skilled in the art, the main memory 901 and the system memory 908 can store program codes and / or data accessible to the processing unit(s) 902. In various embodiments, both the main memory 901 and the system memory 908 can include one or more types of memory used to store data and instructions for execution by the processing unit(s) 902, including high speed random access memory, solid state drives and non-volatile memory such as magnetic disks or optical disks. The main memory 901 can also include a file system 905 that stores one or more files, such as a file 906. The file 906 can include one or more program modules 907, such as an operating system, one or more application programs, other program modules and program data, and can include implementation of a network environment, alone or in combination. The program modules 907 generally carry out the functions and / or methodologies described in the embodiments of the present disclosure.

[0144] The program / utility 908 having a set (at least one) of program modules 907, can be stored in, for example, the memory 901 and comprises, but is not limited to, for example, an operating system, one or more application programs, other program modules and program data, and can include implementation of a network environment, alone or in combination. The program modules 907 generally carry out the functions and / or methodologies described in the embodiments of the present disclosure.

[0145] The electronic device 900 can also communicate with one or more external devices 909 such as a keyboard or pointing device, a display 991, etc.; other devices such as a storage device or an external effects device (not shown); and / or one or more devices that enable a user to interact with the electronic device 900; and / or one or more devices that enable the electronic device 900 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 992. Still yet, the electronic device 900 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via a network adapter 993. As depicted, the network adapter 993 communicates with the other components of the electronic device 900 via the bus 903. It should be appreciated that although the network adapter 993 is depicted as a single component, the network adapter 993 can comprise two or more components that operate together to facilitate communication between the electronic device 900 and one or more other computing devices. Figure 9 As depicted, the network adapter 993 communicates with the other components of the electronic device 900 via the bus 903. It should be appreciated that although the network adapter 993 is depicted as a single component, the network adapter 993 can comprise two or more components that operate together to facilitate communication between the electronic device 900 and one or more other computing devices.

[0146] The processor 902 performs functions of various program codes stored in the memory 901.

[0147] It should be noted that the implementation process and technical principles of the electronic device of the present embodiment are described above in the description of the image processing method of the embodiments of the present disclosure, and will not be described here.

[0148] In order to implement the above-mentioned embodiments, the present disclosure further provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the steps of the image processing method provided by the present disclosure.

[0149] Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk and an optical data storage device, etc.

[0150] In order to realize the above-mentioned embodiments, the present disclosure further provides a computer program product comprising a computer program which, when executed by a processor of an electronic device, implements the image processing method as previously described.

[0151] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that can be incorporated into the above detailed description and making use of the general principles of the present disclosure, which are defined in the following claims.

[0152] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope.

Claims

1. An image processing method, characterized by, The method comprises: determining a first image and a saliency map of the first image, the saliency map being used to determine whether each pixel in the first image needs to be enhanced; determining a saliency value of each pixel in the saliency map and obtaining a target pixel with a saliency value greater than a set threshold value; performing Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel; correcting the first residual information to obtain target residual information; performing sharpening processing on the first image according to the target residual information to obtain a target image; the sharpening processing on the first image according to the target residual information to obtain a target image comprises: determining a median filter image of the first image; performing sharpening processing on the first image according to the target residual information and the median filter image to obtain the target image.

2. The method of claim 1, wherein, the sharpening processing on the first image according to the target residual information and the median filter image to obtain the target image comprises: performing sharpening enhancement processing on the first image according to the target residual information to obtain a candidate image; performing noise suppression on the candidate image based on the median filter image to obtain the target image.

3. The method of claim 1, wherein, the Gaussian filtering on the target pixel to obtain the initial first residual information of the target pixel comprises: performing layer-by-layer convolution operation on a sliding window with the target pixel as a center pixel through at least two layers of Gaussian filtering pyramids to obtain a filtered luminance component of the target pixel at each layer; obtaining the first residual information according to the filtered luminance component and an original luminance component of the target pixel.

4. The method of claim 3, wherein, the obtaining of the first residual information according to the filtered luminance component and the original luminance component of the target pixel comprises: for a first layer in the Gaussian filtering pyramids, determining the first residual information of the first layer corresponding to the target pixel according to the filtered luminance component of the first layer and the original luminance component of the target pixel; for an i-th layer in the Gaussian filtering pyramids, determining the first residual information of the i-th layer corresponding to the target pixel according to filtered luminance components of an (i-1)-th layer and the i-th layer, wherein i is an integer greater than or equal to 1.

5. The method of claim 4, wherein, the method further comprises: determining a first difference value between the original luminance component and the filtered luminance component of the first layer and processing the first difference value based on a preset first offset to obtain the first residual information of the first layer; determining a second difference value between the filtered luminance component of the (i-1)-th layer and the filtered luminance component of the i-th layer and processing the second difference value based on a preset second offset to obtain the first residual information of the i-th layer.

6. The method of claim 3, wherein, the correction of the first residual information to obtain the target residual information comprises: determining a contrast of the target pixel; adjusting the first residual information of the target pixel according to the filtered luminance component and the contrast to obtain second residual information of the target pixel; determining the target residual information of the target pixel according to the second residual information.

7. The method of claim 6, wherein, The adjusting the first residual information of the target pixel according to the filtered luminance component and the contrast, to obtain second residual information of the target pixel, comprises: determining a first gain corresponding to the first residual information; determining a second gain corresponding to the filtered luminance component; determining a third gain corresponding to the contrast; adjusting the first residual information according to the first gain, the second gain and the third gain, to obtain the second residual information of the target pixel.

8. The method of claim 6, wherein, The determining the contrast of the target pixel comprises: determining a maximum luminance component and a minimum luminance component in the sliding window; determining the contrast of the target pixel according to the maximum luminance component and the minimum luminance component.

9. The method of claim 6, wherein, The determining the target residual information of the target pixel according to the second residual information comprises: optimizing the second residual information based on the chrominance component of the target pixel, to obtain third residual information of the target pixel; determining the target residual information according to the third residual information.

10. The method of claim 9, wherein, The method further comprises: performing discrete point operation on the sliding window to determine residual adjustment information corresponding to the target pixel; adjusting the second residual information or the third residual information according to the residual adjustment information, to obtain the target residual information.

11. The method of any one of claims 1-2, wherein, The determining the saliency map of the first image comprises: determining a grayscale image of the first image; determining a pixel corresponding grayscale level distance sum according to the grayscale image; determining a grayscale level distance mean value of the pixel according to the pixel corresponding grayscale level distance sum; determining a saliency value of the pixel according to the grayscale level distance mean value and a set threshold, to obtain the saliency map of the first image.

12. An image processing apparatus characterized by comprising: The device comprises: a determining module, configured to determine a first image and a saliency map of the first image, the saliency map being used to determine whether each pixel in the first image needs to be enhanced; a Gaussian filtering module, configured to determine a saliency value of each pixel in the saliency map, and acquire a pixel with a saliency value greater than a set threshold as a target pixel; perform Gaussian filtering on the target pixel to obtain initial first residual information of the target pixel; and correct the first residual information to obtain target residual information; a sharpening processing module, configured to perform sharpening processing on the first image according to the target residual information, to obtain a target image; the sharpening processing module is further configured to: determine a median filter image of the first image; perform sharpening processing on the first image according to the target residual information and the median filter image, to obtain the target image.

13. The apparatus of claim 12, wherein, The Gaussian filtering module is further configured to: perform layer-by-layer convolution operation on a sliding window with the target pixel as a center pixel through at least two layers of Gaussian filtering pyramid, to obtain a filtered luminance component of the target pixel at each layer; obtain the first residual information according to the filtered luminance component and an original luminance component of the target pixel.

14. The apparatus of claim 13, wherein, The Gaussian filtering module is further configured to: determine a contrast of the target pixel; adjusting the first residual information of the target pixel according to the filtered luminance component and the contrast, to obtain second residual information of the target pixel; determining target residual information of the target pixel according to the second residual information.

15. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement the steps of the method of any one of claims 1-11.

16. A computer-readable storage medium having stored thereon computer program instructions, wherein, the program instructions, when executed by the processor, implement the steps of the method of any one of claims 1-11.

Images