Image enhancement method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides an image enhancement method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the mask information and the scene classification information of the to-be-enhanced image are determined, the clipping amplitude limit of each image pixel in the to-be-enhanced image is determined according to the mask information, and the redistribution ratio is determined according to the scene classification information; a brightness channel image of a to-be-enhanced image is divided into a plurality of image blocks, and contrast-limited adaptive histogram equalization processing is performed on the image blocks according to clipping limiting and a redistribution ratio to obtain a plurality of target image blocks, so that the plurality of target image blocks can be spliced into a target brightness channel image. And the target image is generated according to the target brightness channel image and the chrominance channel image of the to-be-enhanced image, so that the foreground and background masks in the to-be-enhanced image and scene classification can be effectively combined for image enhancement, the natural degree of image enhancement is improved, and the image enhancement effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image enhancement method, device, equipment, storage medium and product. Background Art

[0002] With the development of computer technology and Internet technology, the application of video platforms is becoming more and more extensive. Given the diversity of scenes, problems such as overexposure or underexposure of images are more common. The images cannot accurately reflect the colors of the real world, resulting in unnatural contrast in the images and a decrease in image quality in downstream image processing.

[0003] In order to obtain better contrast and image quality, the collected images are usually enhanced. Currently, the commonly used image enhancement processing scheme is generally based on the global image and uniformly enhances the contrast of the image. However, this image enhancement processing scheme is relatively direct and is prone to unnatural image enhancement effects, resulting in poor image enhancement effects. Summary of the invention

[0004] The embodiments of the present application provide an image enhancement method, device, equipment, storage medium and product to solve the technical problems of unnatural image enhancement effect and poor image enhancement effect in the image enhancement processing scheme in the related art, and can improve the naturalness of image enhancement, improve the fineness of image enhancement, and improve the image enhancement effect.

[0005] In a first aspect, an embodiment of the present application provides an image enhancement method, comprising:

[0006] Acquire an image to be enhanced, and determine mask information and scene classification information of the image to be enhanced, wherein the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced;

[0007] Determining a cropping limit of each image pixel in the image to be enhanced according to the mask information, wherein the cropping limit of the foreground pixel is smaller than the cropping limit of the background pixel;

[0008] determining a reallocation ratio according to the scene classification information;

[0009] Dividing the brightness channel image of the image to be enhanced into a plurality of image blocks, and performing contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks;

[0010] A plurality of the target image blocks are spliced ​​into a target brightness channel image, and a target image is generated according to the target brightness channel image and the chroma channel image of the image to be enhanced.

[0011] In a second aspect, an embodiment of the present application provides an image enhancement device, including an image analysis module, a limit determination module, a ratio determination module, an equalization processing module and an image merging module, wherein:

[0012] The image analysis module is configured to obtain the image to be enhanced, determine mask information and scene classification information of the image to be enhanced, and the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced;

[0013] The clipping determination module is configured to determine the clipping clipping of each image pixel in the image to be enhanced according to the mask information, wherein the clipping clipping of the foreground pixel is smaller than the clipping clipping of the background pixel;

[0014] The ratio determination module is configured to determine the reallocation ratio according to the scene classification information;

[0015] The equalization processing module is configured to divide the brightness channel image of the image to be enhanced into a plurality of image blocks, and perform contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks;

[0016] The image merging module is configured to splice the plurality of target image blocks into a target brightness channel image, and generate a target image according to the target brightness channel image and the chroma channel image of the image to be enhanced.

[0017] In a third aspect, an embodiment of the present application provides an image enhancement device, including: a memory and one or more processors;

[0018] The memory is used to store one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image enhancement method as described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to perform the image enhancement method as described in the first aspect when executed by a computer processor.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the image enhancement method as described in the first aspect.

[0022] The embodiment of the present application determines the mask information and scene classification information of the image to be enhanced, determines the cropping limit of each image pixel in the image to be enhanced according to the mask information, and determines the redistribution ratio according to the scene classification information, divides the brightness channel image of the image to be enhanced into multiple image blocks, and performs contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain multiple target image blocks, can splice the multiple target image blocks into a target brightness channel image, and generate a target image according to the target brightness channel image and the chroma channel image of the image to be enhanced, can effectively combine the foreground and background masks and scene classification in the image to be enhanced to perform image enhancement, image enhancement can effectively adapt to the attention of the human eye in different scenes, can improve the naturalness of image enhancement, improve the refinement of image enhancement, and improve the image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0024] Figure 2 is a flow chart of another image enhancement method provided in an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of cropping and redistributing the number of pixels in the brightness histogram;

[0026] Figure 4 It is a schematic diagram of the process of generating a target brightness histogram;

[0027] Figure 5 is a structural schematic diagram of an image enhancement device provided in an embodiment of the present application;

[0028] Figure 6 It is a structural schematic diagram of an image enhancement device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0030] The image enhancement method provided in the present application can be applied to image enhancement in real-time broadcasting scenarios (such as live video broadcasting, video calls), and aims to perform image enhancement in combination with foreground and background masks and scene classification in the image to be enhanced, adapt to the attention of the human eye in different scenarios, improve the naturalness of image enhancement, improve the refinement of image enhancement, and improve the image enhancement effect.

[0031] In existing image processing solutions, due to the diversity of scenes and different video acquisition devices, problems such as overexposure or underexposure of images are common, which usually results in the image failing to accurately reflect the colors of the real world, resulting in unnatural contrast in the image and reduced image quality in downstream image processing. Contrast enhancement processing is generally performed on the captured image. Image contrast enhancement technology is generally divided into global enhancement processing or local enhancement processing. However, both of these image enhancement methods will significantly affect the image brightness and introduce artifacts into the image, resulting in problems such as color distortion, increased noise, and unnatural image enhancement effects. It is difficult to maintain accurate skin color and facial color during broadcasting, and the image enhancement effect is poor. Based on this, an image enhancement method of an embodiment of the present application is provided to solve the technical problems of unnatural image enhancement effects and poor image enhancement effects in existing image enhancement processing solutions.

[0032] Figure 1 A flowchart of an image enhancement method provided in an embodiment of the present application is given. The image enhancement method provided in an embodiment of the present application can be executed by an image enhancement device, which can be implemented by hardware and / or software and integrated in the image enhancement equipment.

[0033] The following description is made by taking an image enhancement device executing an image enhancement method as an example. Figure 1 , the image enhancement method comprises:

[0034] S110: Acquire the image to be enhanced, and determine mask information and scene classification information of the image to be enhanced, where the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced.

[0035] In one embodiment, the image to be enhanced provided by the present solution may be a video frame in a video stream acquired in real time during a real-time broadcast process (eg, live video), may be a video frame in a complete video, or may be a single image.

[0036] Exemplarily, an image to be enhanced that needs to be enhanced is obtained, and the mask information (mask) and scene classification information of the image to be enhanced are determined. The mask information can be used to indicate whether each pixel (pixel point) in the image to be enhanced is a foreground pixel or a background pixel. For example, the pixels corresponding to portraits, animals, vehicles, objects (commodities), etc. in the image to be enhanced are foreground pixels, and the pixels other than portraits, animals, vehicles, objects (commodities), etc. are background pixels. The scene classification information can be used to indicate the scene type of the image to be enhanced or the video or video stream corresponding to the enhanced image. Optionally, the scene type may include static scenes, indoors, outdoors, inside vehicles, games, performance streaming media, close-ups, text overlays, human motion, overexposure, underexposure, dark scenes, etc.

[0037] In one embodiment, the image to be enhanced can be segmented by the trained image segmentation model and the foreground mask and / or background mask of the image to be enhanced can be extracted according to the image segmentation result, wherein the size of the foreground mask and / or background mask is consistent with the size of the image to be enhanced, and the corresponding pixel can be determined as a foreground pixel or a background pixel according to the mask value of each pixel on the foreground mask and / or background mask. For example, the mask value range is [0,1], and the pixels with a mask value of a first set value (for example, 0) or below the first set value are background pixels, and the pixels with a mask value greater than the first set value are foreground pixels. For example, the image to be enhanced I t Input to the image segmentation model to extract the foreground mask

[0038] m x,y =m(x,y)

[0039] Among them, (x, y) is the coordinate of the pixel on the image to be enhanced, and the mask value m x,y Used to classify pixels, if m x,y = 0, the pixel can be classified as a background pixel. x,y >0, the pixel can be classified as a foreground pixel, m x,y The value of varies continuously between 0 and 1, and the segmentation value (mask value) at the pixel (x, y) follows the indicator function

[0040] In one embodiment, the scene classification information of the image or video to be enhanced can be determined by a trained scene classification model, and the scene classification information of the image or video to be enhanced can be quickly determined by the scene classification model.

[0041] S120: Determine the cropping limit of each image pixel in the image to be enhanced according to the mask information, wherein the cropping limit of the foreground pixel is smaller than the cropping limit of the background pixel.

[0042] In one embodiment, different clipping limits (clipLimit) can be pre-configured for foreground pixels and background pixels, and the first correspondence between the foreground pixels and background pixels and the clipping limits is recorded. The clipping limits are used to limit the number of pixels in each brightness range (bins) on the brightness histogram, and a larger clipping limit can achieve a greater image contrast enhancement effect.

[0043] Among them, the smaller the cropping limit, the smaller the change in image contrast, and the human eye is more sensitive to changes in the foreground. This scheme sets the cropping limit of foreground pixels to be smaller than the cropping limit of background pixels, so that the adjustment amplitude of the foreground is smaller than the adjustment amplitude of the background, reducing excessive enhancement of the foreground, ensuring the naturalness of image enhancement, and improving the image enhancement effect.

[0044] S130: Determine a reallocation ratio according to the scene classification information.

[0045] Exemplarily, after the mask information of the image to be enhanced is determined, the cropping limit of each image pixel in the image to be enhanced may be determined according to the mask value of each pixel in the mask information and the first corresponding relationship:

[0046] clipLimit = {CL FG ,CL BG}

[0047] Among them, CL FG is the crop limit corresponding to the foreground pixel, CL BG is the cropping limit corresponding to the background pixels.

[0048] In one embodiment, different reallocation ratios (binIncrRatio) may be pre-configured for different scene classifications, and a second correspondence between scene classification information and the reallocation ratio may be recorded, wherein the reallocation ratio is used to configure the distribution of the number of pixels on the reallocation histogram.

[0049] In one embodiment, after determining the scene classification information of the image to be enhanced, the scene classification information SC of the image to be enhanced may be used to identify the scene classification information of the image to be enhanced. t And the second correspondence determines the redistribution ratio:

[0050] binIncrRatio SC =ratio(SC t )

[0051] S140: Divide the brightness channel image of the image to be enhanced into a plurality of image blocks, and perform contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks.

[0052] Exemplarily, based on a preset image size and / or image division number, the brightness channel image (eg, Y channel image) of the image to be enhanced is divided into a plurality of image tiles. The image tiles G i , i = {1, 2, 3, ..., N}, N is the number of image blocks. This solution performs image enhancement processing on each image block. In one embodiment, the image to be enhanced can be converted into an image format including a brightness channel and a chrominance channel (such as a YUV format, an HSV format, etc.), and then a brightness channel image is extracted from the brightness channel of the image to be enhanced, and the brightness channel image is divided into a plurality of image blocks.

[0053] For each image block, a contrast limited adaptive histogram equalization (CLAHE) process is performed on the image block according to the above-determined cropping limit and redistribution ratio to obtain a target image block corresponding to each image block.

[0054] In one embodiment, the image blocks can be subjected to contrast-limited adaptive histogram equalization processing based on foreground enhancement and background enhancement according to the cropping limits and redistribution ratios corresponding to the foreground pixels and the background pixels, respectively, to obtain a foreground enhanced image block and a background enhanced image block corresponding to each image block, respectively, and then the pixels corresponding to the background in the background enhanced image block and the pixels corresponding to the foreground in the foreground enhanced image block are spliced ​​to obtain a target image block in which the foreground and the background are enhanced respectively.

[0055] For example, for each image block, the number of excess pixels in each brightness range in the brightness histogram of the cropped image block corresponding to the background pixels is cropped, and the excess pixels are redistributed to each brightness range according to the redistribution ratio, so as to obtain a background equalized brightness histogram based on the background cropped clipping and clipping for equalization processing, and then the brightness values ​​of each pixel on the image block are remapped according to the background equalized brightness histogram to obtain a background enhanced image block based on the background cropped clipping and clipping for enhancement processing. And the number of excess pixels in each brightness range in the brightness histogram of the cropped image block corresponding to the foreground pixels is cropped, and the excess pixels are redistributed to each brightness range according to the redistribution ratio, so as to obtain a foreground equalized brightness histogram based on the foreground cropped clipping and clipping for equalization processing, and then the brightness values ​​of each pixel on the image block are remapped according to the foreground equalized brightness histogram to obtain a foreground enhanced image block based on the foreground cropped clipping and clipping for enhancement processing. Then, the pixels corresponding to the background in the background enhanced image block and the pixels corresponding to the foreground in the foreground enhanced image block are spliced ​​to obtain a target image block in which the foreground and background are enhanced respectively.

[0056] S150: splicing a plurality of target image blocks into a target brightness channel image, and generating a target image according to the target brightness channel image and the chrominance channel image of the image to be enhanced.

[0057] Exemplarily, after obtaining the target image blocks corresponding to each image block, each target image block is spliced ​​into a target brightness channel image, and the target brightness channel image is added to the chrominance channel image (U channel image and V channel image) of the image to be enhanced to generate the target image. The target image can be determined by the following formula:

[0058]

[0059] Among them, Y adj is the target brightness channel image, U t is the U channel image, V t It is the V channel image.

[0060] In one embodiment, after obtaining the target brightness channel image, or in the process of splicing multiple target image blocks into the target brightness channel image, each target image block in the target brightness channel image can be interpolated (e.g., bilinear interpolation) to effectively maintain the continuity of the brightness values ​​between adjacent target image blocks and reduce the boundary effect of the target brightness channel image.

[0061] In a possible embodiment, the image enhancement method provided by the present solution may further filter the target brightness channel image after splicing multiple target image blocks into a target brightness channel image. Accordingly, the image enhancement method provided by the present solution splices multiple target image blocks into a target brightness channel image, which may be splicing multiple filtered target image blocks into a target brightness channel image.

[0062] Exemplarily, after obtaining the target brightness channel image, the target brightness channel image may be filtered (e.g., Gaussian filtering). Correspondingly, when multiple target image blocks are spliced ​​into the target brightness channel image, multiple filtered target image blocks may be spliced ​​into the target brightness channel image to maintain the continuity of the brightness values ​​between adjacent target image blocks and reduce the boundary effect of the target brightness channel image. Optionally, the target brightness channel image may be filtered using a Gaussian filter to generate a blurred brightness channel image, and the target brightness channel image may be subtracted from the blurred brightness channel image to obtain the filtered target image block: Y blur_adj =Y adj -Y blur , where Y blur_adj is the target image block after filtering, Y adj is the target image block before filtering, Y blur is the blurred brightness channel image. Correspondingly, the target image can be determined by the following formula:

[0063]

[0064] This scheme sharpens the edge of the target brightness histogram through filtering processing and improves the contrast of the target brightness histogram, which allows the enhancement intensity to be reduced in the contrast-limited adaptive histogram equalization processing of image blocks, helps to reduce the appearance of image artifacts, and at the same time achieves an ideal contrast enhancement effect.

[0065] In the above, by determining the mask information and scene classification information of the image to be enhanced, determining the cropping limit of each image pixel in the image to be enhanced according to the mask information, and determining the redistribution ratio according to the scene classification information, the brightness channel image of the image to be enhanced is divided into multiple image blocks, and the image blocks are subjected to contrast-limited adaptive histogram equalization processing according to the cropping limit and the redistribution ratio to obtain multiple target image blocks. The multiple target image blocks can be spliced ​​into a target brightness channel image, and a target image can be generated according to the target brightness channel image and the chroma channel image of the image to be enhanced. The foreground and background masks and scene classification in the image to be enhanced can be effectively combined for image enhancement. Image enhancement can effectively adapt to the attention of the human eye in different scenes, improve the naturalness of image enhancement, improve the refinement of image enhancement, and improve the image enhancement effect.

[0066] Based on the above embodiments, Figure 2 A flowchart of another image enhancement method provided in an embodiment of the present application is given, which is a specific implementation of the above-mentioned image enhancement method. Figure 2 , the image enhancement method comprises:

[0067] S210: Acquire the image to be enhanced, and determine mask information and scene classification information of the image to be enhanced, where the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced.

[0068] S220: Determine the cropping limit of each image pixel in the image to be enhanced according to the mask information, wherein the cropping limit of the foreground pixel is smaller than the cropping limit of the background pixel.

[0069] S230: Determine a reallocation ratio according to the scene classification information.

[0070] In one embodiment, the reallocation ratio can be set to a value greater than 1 or less than 1 to configure the amount of the cropped pixel number allocated to the head and tail of the brightness histogram. When the cropped pixel number is redistributed on the brightness histogram, if the reallocation ratio is greater than 1, the amount of the pixel number allocated on the tail of the brightness histogram is larger, and if the reallocation ratio is less than 1, the amount of the pixel number allocated on the head of the brightness histogram is larger.

[0071] like Figure 3 A schematic diagram of cropping and redistributing the number of pixels of a brightness histogram is provided, wherein S is the number of cropped pixels cropped based on the cropping limit for the original brightness histogram, and when the reallocation ratio is greater than 1, the distribution of the number of cropped pixels after redistribution in the target brightness histogram is shown in A1 and A2, and it can be seen that the amount of the reallocated number of cropped pixels allocated to the tail of the target brightness histogram is larger, and when the reallocation ratio is less than 1, the distribution of the number of cropped pixels after redistribution in the target brightness histogram is shown in B1 and B2, and it can be seen that the amount of the reallocated number of cropped pixels allocated to the head of the target brightness histogram is larger.

[0072] S240: Divide the brightness channel image of the image to be enhanced into a plurality of image blocks, and determine an original brightness histogram of each image block.

[0073] Exemplarily, a brightness channel image is extracted from the image to be enhanced, and the brightness channel image is divided into multiple image blocks, and the original brightness histogram of each image block is determined. The brightness histogram provided by this solution records multiple brightness ranges (for example, the preset brightness value range of 0 to 255 is divided into multiple brightness ranges (or brightness buckets), and the brightness histogram records the number of pixels corresponding to each brightness range in the image (image block)).

[0074] S250: performing contrast-limited adaptive histogram equalization processing on the original brightness histogram according to the clipping, limiting and redistribution ratio to obtain a plurality of target brightness histograms.

[0075] Exemplarily, for each original brightness histogram, a contrast-limited adaptive histogram equalization process is performed on the original brightness histogram according to the clipping, limiting, and redistribution ratio to obtain a plurality of target brightness histograms.

[0076] For example, the number of pixels in each brightness range in the original brightness histogram can be cropped according to the cropping limit, and the cropped number of pixels can be distributed to each brightness range of the cropped original brightness histogram according to the redistribution ratio to achieve redistribution of the cropped number of pixels, and obtain a target brightness histogram corresponding to each original brightness histogram. Compared with the original brightness histogram, the target brightness histogram has a more uniform distribution of the number of pixels in different brightness ranges, and the contrast of the target image block obtained by brightness mapping the image block based on the target brightness histogram is enhanced.

[0077] In one possible embodiment, Figure 4 As shown in the schematic diagram of a target brightness histogram generation process, the image enhancement method provided by this scheme performs contrast-limited adaptive histogram equalization processing on the original brightness histogram according to clipping, limiting, and redistribution ratio to obtain multiple target brightness histograms, including:

[0078] S251: Determine a cropping threshold according to the cropping limit and the number of image blocks.

[0079] Exemplarily, for each image block, the cropping threshold is determined based on the cropping limit and the number of image blocks. The number of pixels exceeding the cropping threshold in each brightness range is the excess number of pixels. For example, when the cropping threshold is 200 and the number of pixels in a brightness range is 210, the corresponding excess number of pixels is 10. When the number of pixels in a brightness range is 180, there is no excess number of pixels in the brightness range or the excess number of pixels is 0.

[0080] Optionally, the cropping threshold can be determined by dividing the product of the cropping limit and the number of image blocks by a preset brightness level (e.g., 256), and determining the cropping threshold based on the division result. Alternatively, the cropping threshold can be determined by taking the division result of the cropping limit and a set coefficient as the cropping threshold to accurately determine the cropping threshold, thereby improving the accuracy of pixel number cropping processing on the original brightness histogram and ensuring the image enhancement effect.

[0081] In one embodiment, when determining the clipping threshold according to the division result, the present solution may determine the maximum value between the division result and the preset clipping threshold (for example, the preset clipping threshold is set to 1), and determine the maximum value as the clipping threshold, or may determine the clipping threshold by adding the division result and the preset threshold, so as to accurately determine the clipping threshold and ensure the image enhancement effect. Optionally, the clipping threshold provided by the present solution may be determined by the following formula:

[0082] tileSize=(width=It.width / grid_width,height

[0083] =It.height / grid_height)

[0084] tileSizeTotal=tileSize.width*tileSize.height

[0085] threshold′=clipLimit*tileSizeTotal / k

[0086] threshold=max(threshold,a)

[0087] Among them, threshold′ is the cropping threshold determined by the division result of the product of the cropping limit clipLimit and the number of image blocks tileSizeTotal and the preset brightness level k, threshold is the cropping threshold determined according to the maximum value of the division result and the preset cropping threshold a, tileSize.width is the number of image blocks divided in the width direction of the brightness channel image, tileSize.height is the number of image blocks divided in the height direction of the brightness channel image, tileSize is the image block size information, which includes tileSize.width and tileSize.height, tileSize.width=It.width / grid_width, tileSize.height=It.height / grid_height, It.width and It.height are the width and height of the brightness channel image or the image to be enhanced, grid_width and grid_height are the width and height of the image block.

[0088] S252: performing pixel number cropping processing on each original brightness histogram according to the cropping threshold, and determining the number of cropped pixels exceeding the cropping threshold in each original brightness histogram according to the pixel number cropping processing result.

[0089] Exemplarily, after determining the clipping threshold, for each original brightness histogram, according to the clipping threshold and the number of pixels in each brightness range in the original brightness histogram, the number of pixels in each brightness range is clipped. For example, for each brightness range in the original brightness histogram, the number of pixels in the corresponding brightness range is determined; if the number of pixels in the brightness range is less than or equal to the clipping threshold, the number of pixels in the brightness range is not clipped; if the number of pixels in the brightness range is greater than the clipping threshold, the number of pixels in the brightness range that exceeds the clipping threshold is clipped, and the number of pixels remaining in the clipped brightness range is equal to the clipping threshold.

[0090] In one embodiment, after completing the pixel number cropping process for each brightness range, the number of cropped pixels in the original brightness histogram that exceeds the cropping threshold is determined based on the pixel number cropping process result, that is, the number of cropped pixels in each brightness range is added together to obtain the number of cropped pixels in the original brightness histogram that exceeds the cropping threshold.

[0091] S253: Determine first incremental information according to the number of cropped pixels and the number of brightness ranges, and determine second incremental information according to the first incremental information and the reallocation ratio.

[0092] Exemplarily, for each original brightness histogram, after determining the number of cropped pixels in the original brightness histogram that exceed the cropping threshold, the first incremental information is determined based on the number of cropped pixels and the preset number of gray levels (the number of bins). For example, the ratio of the number of cropped pixels to the number of brightness ranges is calculated, and the ratio is determined as the first incremental information. The first incremental information is accurately calculated to ensure the image enhancement effect.

[0093] In one embodiment, the second incremental information can be determined based on the first incremental information and the redistribution ratio determined above. For example, the first incremental information is multiplied by the redistribution ratio, and the multiplication result is determined as the second incremental information. The second incremental information is accurately calculated to ensure the image enhancement effect.

[0094] S254: Add the original number of pixels in each brightness range in the original brightness histogram to the second incremental information, determine the target number of pixels in each brightness range in the original brightness histogram according to the addition result, and determine the target brightness histogram according to the target number of pixels in each brightness range.

[0095] Exemplarily, for each original brightness histogram, the original number of pixels in each brightness range in the original brightness histogram is added to the second incremental information respectively, and the target number of pixels in each brightness range in the original brightness histogram can be determined based on the addition result, so as to realize the redistribution of the number of cropped pixels, and determine the target brightness histogram based on the target number of pixels in each brightness range. At this time, the number of pixels in each brightness range in the target brightness histogram is updated to the number of pixels after cropping and redistribution, and the distribution of the number of pixels in each brightness range of the target brightness histogram is more balanced than the distribution of the number of pixels in each brightness range of the original brightness histogram. For example, the target number of pixels of the target brightness histogram provided by this scheme can be determined by the following formula:

[0096] avgBinIncr=avgBinIncr′×binIncrRatio

[0097] binValues=binValues′+avgBinIncr

[0098] Among them, avgBinIncr′ is the first incremental information, binIncrRatio is the redistribution ratio, avgBinIncr is the second incremental information, binValues′ is the number of original pixels in the original brightness histogram in the corresponding brightness range, and binValues ​​is the number of target pixels in the target brightness histogram in the corresponding brightness range.

[0099] This solution determines a cropping threshold according to the cropping limit and the number of image blocks, performs pixel number cropping processing on each original brightness histogram according to the cropping threshold, and determines the number of cropped pixels in each original brightness histogram that exceeds the cropping threshold according to the pixel number cropping processing result, determines the first incremental information according to the number of cropped pixels and the number of brightness ranges, and determines the second incremental information according to the first incremental information and the redistribution ratio, and adds the original number of pixels in each brightness range in the original brightness histogram to the second incremental information, determines the target number of pixels in each brightness range in the original brightness histogram according to the addition result, and determines the target brightness histogram according to the target number of pixels in each brightness range, accurately performs contrast-limited adaptive histogram equalization processing on the original brightness histogram, and can effectively combine the foreground and background masks and scene classification in the image to be enhanced to perform image enhancement, improve the naturalness of image enhancement, and improve the image enhancement effect.

[0100] S260: Perform brightness mapping processing on the image blocks according to the multiple target brightness histograms to obtain multiple target image blocks.

[0101] Exemplarily, after obtaining the target brightness histograms corresponding to the original brightness histograms, brightness mapping processing may be performed on the corresponding image blocks according to the target brightness histograms to obtain the target image blocks corresponding to the image blocks.

[0102] In one embodiment, when brightness mapping is performed on an image block according to a target brightness histogram, a cumulative distribution function (CDF) may be determined according to the target brightness histogram. The cumulative distribution function may be obtained by calculating the cumulative number of pixels in each brightness range in the target brightness histogram, which represents the ratio of the number of pixels from 0 to the current brightness range to the total number of pixels in the target brightness histogram. Optionally, the value range of the CDF may be between 0 and 1. The original brightness value of each pixel in the image block is mapped to a new brightness value according to the cumulative distribution function to obtain the target image block. The mapping relationship of mapping the original brightness value of each pixel in the image block to the new brightness value according to the cumulative distribution function may be that the original brightness value is converted into a cumulative distribution function value through a cumulative distribution function, and then the cumulative distribution function value is mapped to the new brightness value through linear scaling. This process realizes the redistribution of the brightness value, thereby enhancing the contrast of the image and realizing the enhancement processing of the image.

[0103] Optionally, a new table can be created as a CDF table to store CDF values. The CDF table can contain two columns: one is the brightness range of the target brightness histogram, and the other is the corresponding CDF value. For each brightness range in the target brightness histogram, the cumulative sum of the number of pixels of all previous brightness ranges is calculated. This cumulative sum is the CDF value of the brightness range. Optionally, the CDF value can be divided by the number of pixels in the target brightness histogram to normalize the CDF value. The CDF table records the normalized CDF value. For each pixel in the image block, its brightness value will be used as the input index of the CDF. By looking up the corresponding brightness value in the CDF table, the new brightness value of the pixel after equalization can be determined.

[0104] For example, for each pixel in the image block, its original brightness value will be used as the input index of the CDF. By looking up the corresponding brightness value in the CDF table, the new brightness value of the pixel after equalization can be determined, where the new brightness value can be obtained by normalizing the CDF value to a range of 0 to 255, such as multiplying each value of the CDF by 255 and then rounding to the nearest integer to obtain a new brightness value. Finally, the brightness value of each pixel in the image block is replaced with the new brightness value obtained by CDF mapping, thereby completing the redistribution of the brightness value and obtaining the target image block corresponding to the image block.

[0105] In one embodiment, the original brightness histogram of each image block can be subjected to contrast-limited adaptive histogram equalization processing according to the cropping limits and redistribution ratios corresponding to the foreground pixels and the background pixels, respectively, to obtain a plurality of target brightness histograms, and the image blocks are subjected to brightness mapping processing according to the plurality of target brightness histograms to obtain a foreground enhanced image block and a background enhanced image block corresponding to each image block, and then the pixels corresponding to the background in the background enhanced image block and the pixels corresponding to the foreground in the foreground enhanced image block are spliced ​​to obtain target image blocks in which the foreground and the background are enhanced respectively.

[0106] For example, for each grid, different contrast enhancement strengths are applied to the image blocks, so that the image blocks Based on the segmentation mask limit clipLimit the ratio binIncrRatio SC Based on SC t , we use the CLAHE contrast enhancement algorithm

[0107] According to the scene classification information SC t , as well as the clip limit clipLimit and the redistribution ratio binIncrRatio Sc For each image block The original brightness histogram is processed by contrast-limited adaptive histogram equalization:

[0108]

[0109] We can get:

[0110]

[0111] in, For image blocks The corresponding target image block, CL FG is the crop limit corresponding to the foreground pixel, CL BG is the cropping limit corresponding to the background pixel, I (0,∞) (m x,y ) is the mask information mapped to 0 and 1 (for example, when the mask information is 0, the mapping value is 0, and when the mask information is greater than 0, the mapping value is 1).

[0112] This scheme performs contrast-limited adaptive histogram equalization processing on the original brightness histogram of each image block according to the cropping limit and the redistribution ratio to obtain multiple target brightness histograms, and performs brightness mapping processing on the image block according to the multiple target brightness histograms to obtain multiple target image blocks, accurately realizing the contrast enhancement processing of the image to be enhanced, combining the mask information and the scene category to adjust the contrast of the image so that it can adapt to various scenes and human attention, and adjust the contrast according to the different sensitivities of different scenes and foreground and background, and pay more attention to the enhancement of contour details, effectively reducing the unnatural effects and artifacts of image enhancement, improving the clarity and contrast of the image, while retaining the main dynamic range of the image color space, and improving the image enhancement effect.

[0113] S270: splicing the multiple target image blocks into a target brightness channel image, and generating a target image according to the target brightness channel image and the chrominance channel image of the image to be enhanced.

[0114] In the above, by determining the mask information and scene classification information of the image to be enhanced, determining the cropping limit of each image pixel in the image to be enhanced according to the mask information, and determining the redistribution ratio according to the scene classification information, the brightness channel image of the image to be enhanced is divided into multiple image blocks, and the image blocks are subjected to contrast-limited adaptive histogram equalization processing according to the cropping limit and the redistribution ratio to obtain multiple target image blocks. The multiple target image blocks can be spliced ​​into a target brightness channel image, and a target image can be generated according to the target brightness channel image and the chroma channel image of the image to be enhanced. The foreground and background masks and scene classification in the image to be enhanced can be effectively combined for image enhancement. Image enhancement can effectively adapt to the attention of the human eye in different scenes, improve the naturalness of image enhancement, improve the refinement of image enhancement, and improve the image enhancement effect. At the same time, by performing contrast-limited adaptive histogram equalization processing on the original brightness histogram of each image block according to the cropping limit and the redistribution ratio, multiple target brightness histograms are obtained, and brightness mapping processing is performed on the image blocks according to the multiple target brightness histograms to obtain multiple target image blocks, so as to accurately realize the contrast enhancement processing of the image to be enhanced, and adjust the contrast of the image in combination with the mask information and the scene category, so that it can adapt to various scenes and human attention, and adjust the contrast according to the different sensitivities of different scenes and foreground and background, and pay more attention to the enhancement of contour details, effectively reduce the unnatural effects and artifacts of image enhancement, improve the clarity and contrast of the image, and at the same time retain the main dynamic range of the image color space to improve the image enhancement effect.

[0115] Figure 5 Schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application. Figure 5The image enhancement device includes an image analysis module 51, a limit determination module 52, a ratio determination module 53, an equalization processing module 54 and an image merging module 55.

[0116] Among them, the image analysis module 51 is configured to obtain the image to be enhanced, determine the mask information and scene classification information of the image to be enhanced, and the mask information is used to indicate the foreground pixels and background pixels in the image to be enhanced; the limit determination module 52 is configured to determine the cropping limit of each image pixel in the image to be enhanced according to the mask information, wherein the cropping limit of the foreground pixel is smaller than the cropping limit of the background pixel; the ratio determination module 53 is configured to determine the reallocation ratio according to the scene classification information; the equalization processing module 54 is configured to divide the brightness channel image of the image to be enhanced into multiple image blocks, and perform contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the reallocation ratio to obtain multiple target image blocks; the image merging module 55 is configured to splice the multiple target image blocks into a target brightness channel image, and generate a target image according to the target brightness channel image and the chrominance channel image of the image to be enhanced.

[0117] In the above, by determining the mask information and scene classification information of the image to be enhanced, determining the cropping limit of each image pixel in the image to be enhanced according to the mask information, and determining the redistribution ratio according to the scene classification information, the brightness channel image of the image to be enhanced is divided into multiple image blocks, and the image blocks are subjected to contrast-limited adaptive histogram equalization processing according to the cropping limit and the redistribution ratio to obtain multiple target image blocks. The multiple target image blocks can be spliced ​​into a target brightness channel image, and a target image can be generated according to the target brightness channel image and the chroma channel image of the image to be enhanced. The foreground and background masks and scene classification in the image to be enhanced can be effectively combined for image enhancement. Image enhancement can effectively adapt to the attention of the human eye in different scenes, improve the naturalness of image enhancement, improve the refinement of image enhancement, and improve the image enhancement effect.

[0118] In a possible embodiment, the equalization processing module 54 performs contrast-limited adaptive histogram equalization processing on the image block according to the cropping limit and the redistribution ratio to obtain multiple target image blocks, and is configured as follows:

[0119] Determine the original brightness histogram of each image block;

[0120] Performing contrast-limited adaptive histogram equalization processing on the original brightness histogram according to the clipping limit and the redistribution ratio to obtain multiple target brightness histograms;

[0121] The image blocks are subjected to brightness mapping processing according to the multiple target brightness histograms to obtain multiple target image blocks.

[0122] In a possible embodiment, the equalization processing module 54 performs contrast-limited adaptive histogram equalization processing on the original brightness histogram according to the clipping, limiting, and redistribution ratio to obtain multiple target brightness histograms, which are configured as follows:

[0123] Determine a cropping threshold according to the cropping limit and the number of image blocks;

[0124] Performing pixel number cropping processing on each original brightness histogram according to the cropping threshold, and determining the number of cropped pixels exceeding the cropping threshold in each original brightness histogram according to the pixel number cropping processing result;

[0125] determining first incremental information according to the number of cropped pixels and the number of brightness ranges, and determining second incremental information according to the first incremental information and the redistribution ratio;

[0126] The original number of pixels in each brightness range in the original brightness histogram is added to the second incremental information, the target number of pixels in each brightness range in the original brightness histogram is determined according to the addition result, and the target brightness histogram is determined according to the target number of pixels in each brightness range.

[0127] In a possible embodiment, the equalization processing module 54 determines the cropping threshold according to the cropping limit and the number of image blocks, and is configured as follows:

[0128] The product of the cropping limit and the number of image blocks is divided by the preset brightness level, and the cropping threshold is determined according to the division result.

[0129] In a possible embodiment, the equalization processing module 54 determines the clipping threshold according to the division result, and is configured as follows:

[0130] The maximum value between the division result and the preset clipping threshold is determined, and the maximum value is determined as the clipping threshold.

[0131] In a possible embodiment, the equalization processing module 54 determines the first incremental information according to the number of cropped pixels and the number of brightness ranges, and determines the second incremental information according to the first incremental information and the redistribution ratio, and is configured as follows:

[0132] Calculating a ratio of the number of cropped pixels to the number of brightness ranges, and determining the ratio as first incremental information;

[0133] The first increment information is multiplied by the reallocation ratio, and the multiplication result is determined as the second increment information.

[0134] In a possible embodiment, the image enhancement device further includes a filtering processing module, and the filtering processing module is configured to perform filtering processing on the target brightness channel image;

[0135] Correspondingly, the image merging module 55 splices the multiple target image blocks into a target brightness channel image, and is configured to splice the multiple filtered target image blocks into a target brightness channel image.

[0136] It is worth noting that in the embodiment of the above-mentioned image enhancement device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0137] The embodiment of the present application also provides an image enhancement device, which can integrate the image enhancement device provided in the embodiment of the present application. Figure 6 Schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application. Figure 6 The image enhancement device is configured as follows: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when one or more programs are executed by one or more processors 61, the one or more processors 61 implement the image enhancement method provided in the above embodiment. The above-mentioned image enhancement device, equipment, and computer can be used to execute the image enhancement method provided in any of the above-mentioned embodiments, and have corresponding functions and beneficial effects.

[0138] The embodiment of the present application also provides a non-volatile storage medium storing computer executable instructions, which are used to execute the image enhancement method provided in the above embodiment when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the image enhancement method provided above, can also execute the related operations in the image enhancement method provided in any embodiment of the present application. The image enhancement device, equipment and storage medium provided in the above embodiments can execute the image enhancement method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the image enhancement method provided in any embodiment of the present application.

[0139] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the image enhancement method provided in each embodiment of the present application.

Claims

1. An image enhancement method, characterized in that: include: Acquire an image to be enhanced, and determine mask information and scene classification information of the image to be enhanced, wherein the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced; Determining a cropping limit of each image pixel in the image to be enhanced according to the mask information, wherein the cropping limit of the foreground pixel is smaller than the cropping limit of the background pixel; determining a reallocation ratio according to the scene classification information; Dividing the brightness channel image of the image to be enhanced into a plurality of image blocks, and performing contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks; A plurality of the target image blocks are spliced ​​into a target brightness channel image, and a target image is generated according to the target brightness channel image and the chroma channel image of the image to be enhanced.

2. The image enhancement method according to claim 1, characterized in that: The step of performing contrast-limited adaptive histogram equalization processing on the image block according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks includes: Determining an original brightness histogram of each of the image blocks; Performing contrast-limited adaptive histogram equalization processing on the original brightness histogram according to the clipping limit and the redistribution ratio to obtain a plurality of target brightness histograms; The image block is subjected to brightness mapping processing according to the multiple target brightness histograms to obtain multiple target image blocks.

3. The image enhancement method according to claim 2, characterized in that: The step of performing contrast-limited adaptive histogram equalization processing on the original brightness histogram according to the clipping limit and the redistribution ratio to obtain a plurality of target brightness histograms includes: Determining a cropping threshold according to the cropping limit and the number of the image blocks; Performing pixel quantity cropping processing on each of the original brightness histograms according to the cropping threshold, and determining the number of cropped pixels exceeding the cropping threshold in each of the original brightness histograms according to the pixel quantity cropping processing result; determining first incremental information according to the number of cropped pixels and the number of brightness ranges, and determining second incremental information according to the first incremental information and the reallocation ratio; The original number of pixels in each brightness range in the original brightness histogram is added to the second incremental information, the target number of pixels in each brightness range in the original brightness histogram is determined according to the addition result, and the target brightness histogram is determined according to the target number of pixels in each brightness range.

4. The image enhancement method according to claim 3, characterized in that: The determining the cropping threshold according to the cropping limit and the number of the image blocks includes: The product of the cropping limit and the number of the image blocks is divided by a preset brightness level, and a cropping threshold is determined according to the division result.

5. The image enhancement method according to claim 4, characterized in that: The step of determining the clipping threshold according to the division result includes: The maximum value between the division result and the preset clipping threshold is determined, and the maximum value is determined as the clipping threshold.

6. The image enhancement method according to claim 3, characterized in that: The determining of the first incremental information according to the number of cropped pixels and the number of brightness ranges, and determining the second incremental information according to the first incremental information and the reallocation ratio, comprises: Calculating a ratio of the number of cropped pixels to the number of brightness ranges, and determining the ratio as first incremental information; The first increment information is multiplied by the reallocation ratio, and the multiplication result is determined as the second increment information.

7. The image enhancement method according to any one of claims 1 to 6, characterized in that: After the target image blocks are stitched together into a target brightness channel image, the method further includes: Performing filtering processing on the target brightness channel image; Correspondingly, the step of splicing the plurality of target image blocks into a target brightness channel image includes: The target image blocks after the multiple filtering processes are spliced ​​into a target brightness channel image.

8. An image enhancement device, characterized in that: It includes an image analysis module, a limit determination module, a ratio determination module, an equalization processing module and an image merging module, wherein: The image analysis module is configured to obtain the image to be enhanced, determine mask information and scene classification information of the image to be enhanced, and the mask information is used to indicate foreground pixels and background pixels in the image to be enhanced; The clipping determination module is configured to determine the clipping clipping of each image pixel in the image to be enhanced according to the mask information, wherein the clipping clipping of the foreground pixel is smaller than the clipping clipping of the background pixel; The ratio determination module is configured to determine the reallocation ratio according to the scene classification information; The equalization processing module is configured to divide the brightness channel image of the image to be enhanced into a plurality of image blocks, and perform contrast-limited adaptive histogram equalization processing on the image blocks according to the cropping limit and the redistribution ratio to obtain a plurality of target image blocks; The image merging module is configured to splice the plurality of target image blocks into a target brightness channel image, and generate a target image according to the target brightness channel image and the chroma channel image of the image to be enhanced.

9. An image enhancement device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image enhancement method as described in any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to perform the image enhancement method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image enhancement method according to any one of claims 1 to 7 is implemented.

Images