Image processing chip, method, device and storage medium

Abstract

The invention discloses an image processing chip, method and device and a storage medium, and belongs to the technical field of image display. The chip includes an image processor configured to: acquire a first grayscale image based on a first image, the first grayscale image indicating a brightness degree of the first image; according to the brightness range of the first grayscale image, the local contrast of the first grayscale image is enhanced to obtain a second grayscale image, the first grayscale image comprises a plurality of grayscale blocks, and the local contrast indicates the brightness difference of the grayscale blocks in the first grayscale image; and performing color restoration on the second grayscale image based on the color of the first image to obtain a second image. The local contrast is accurately enhanced for the brightness range, so that the local tone effect of the second image obtained through adjustment is better, and the accuracy is higher.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image display technology, and in particular to an image processing chip, method, device and storage medium. Background Art

[0002] In the field of image display technology, there are differences between the images captured by the image capture device and the images observed by the human eye. Therefore, in the process of displaying the image, the image will be corrected to reduce the difference between the displayed image and the image observed by the human eye. Summary of the invention

[0003] The embodiments of the present application provide an image processing chip, method, device and storage medium, which can be used to correct the contrast of an image. The technical solution is as follows:

[0004] In one aspect, an image processing chip is provided, the chip comprising an image processor, and the image processor is configured to:

[0005] Acquire a first grayscale image based on the first image, wherein the first grayscale image indicates brightness of the first image;

[0006] enhancing a local contrast of the first grayscale image according to a brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates a brightness difference of the grayscale blocks in the first grayscale image;

[0007] The second grayscale image is color restored based on the color of the first image to obtain a second image.

[0008] In a possible implementation manner, when the image processor enhances the local contrast of the first grayscale image according to the brightness range of the first grayscale image to obtain the second grayscale image, the image processor is configured as follows:

[0009] Obtain a base layer pixel value and a detail layer pixel value corresponding to any grayscale block, wherein the base layer pixel value is the base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value is the detail layer data of the grayscale value of the central pixel point of any grayscale block;

[0010] According to the brightness range, adjusting the base pixel value to obtain a first pixel value;

[0011] A second pixel value corresponding to a central pixel point of any grayscale block is determined according to the first pixel value and the detail layer pixel value, and a second grayscale image is determined based on the second pixel values ​​of each grayscale block.

[0012] In a possible implementation manner, when the image processor determines the second pixel value corresponding to the central pixel point of any grayscale block according to the first pixel value and the detail layer pixel value, the image processor is configured as follows:

[0013] The detail layer pixel value is increased, the first pixel value is decreased, and the second pixel value is determined based on the increased detail layer pixel value and the decreased first pixel value.

[0014] In one possible implementation, the brightness range is at least one of a local brightness range and a global brightness range, the local brightness range includes the maximum and minimum values ​​of the grayscale values ​​of each pixel in any grayscale block, and the global brightness range includes the maximum and minimum values ​​of the grayscale values ​​of each pixel in the first grayscale image.

[0015] In a possible implementation manner, the global brightness range of the first grayscale image is the same as the global brightness range of an adjacent grayscale image, and the adjacent grayscale image is a grayscale image corresponding to an adjacent frame image of the first image.

[0016] In a possible implementation manner, when the image processor performs color restoration on the second grayscale image based on the color of the first image to obtain the second image, the image processor is configured as follows:

[0017] Determine the number of pixels of each gray value according to the gray value of each pixel in the second gray image, and obtain distribution information of the second gray image;

[0018] determining a contrast enhancement curve of the second grayscale image according to the distribution information;

[0019] adjusting the global contrast of the second grayscale image according to the contrast enhancement curve to obtain a third grayscale image, wherein the global contrast indicates a brightness difference in the second grayscale image;

[0020] The third grayscale image is color restored based on the color of the first image to obtain the second image.

[0021] In a possible implementation manner, when determining the contrast enhancement curve of the second grayscale image according to the distribution information, the image processor is configured as follows:

[0022] Acquire distribution information of a fourth grayscale image, where the fourth grayscale image indicates brightness of a third image, where the third image is a previous frame image of the first image;

[0023] It is determined whether a scene change occurs in the first image relative to the third image according to the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and a contrast enhancement curve of the second grayscale image is determined according to the determination result.

[0024] In a possible implementation manner, when determining the contrast enhancement curve of the second grayscale image according to the judgment result, the image processor is configured as follows:

[0025] When the judgment result indicates that the first image has no scene change relative to the third image, determining the contrast enhancement curve of the fourth grayscale image as the contrast enhancement curve of the second grayscale image;

[0026] Alternatively, when the judgment result indicates that a scene change occurs in the first image relative to the third image, a contrast enhancement curve of the second grayscale image is calculated according to the distribution information of the second grayscale image.

[0027] In a possible implementation manner, when determining the contrast enhancement curve of the second grayscale image according to the distribution information, the image processor is configured as follows:

[0028] The number of pixel points of each gray value is balanced based on the number threshold, and the difference in the number of pixel points of each gray value after the balance is smaller than the difference in the number of pixel points of each gray value in the distribution information;

[0029] A first enhancement curve of the second grayscale image is determined according to the number of each pixel after the equalization, and the contrast enhancement curve is determined based on the first enhancement curve.

[0030] In a possible implementation, when the image processor balances the number of pixels of each gray value based on the number threshold, it is configured as follows:

[0031] For a first gray value in which the number of pixel points is greater than a quantity threshold, adjusting the number of pixel points of the first gray value from the first quantity to the quantity threshold;

[0032] The number of pixel points of the second grayscale value is adjusted according to a second number, the second number is obtained by adding the difference between the first number of each first grayscale value and the number threshold, and the second grayscale value is different from the first grayscale value.

[0033] In a possible implementation manner, when determining the contrast enhancement curve based on the first enhancement curve, the image processor is configured to:

[0034] When the first image is a dark light scene, adjusting the first enhancement curve using a second enhancement curve to obtain the contrast enhancement curve, wherein the second enhancement curve is used to enhance the global contrast of the image in the dark light scene;

[0035] In a possible implementation manner, when determining the contrast enhancement curve based on the first enhancement curve, the image processor is configured to:

[0036] The first enhancement curve is adjusted using a smoothing coefficient to obtain the contrast enhancement curve, wherein the smoothing coefficient is used to reduce a sudden change between the first image and adjacent frame images of the first image.

[0037] On the other hand, the present application provides an image processing method, the method comprising:

[0038] Acquire a first grayscale image based on the first image, wherein the first grayscale image indicates brightness of the first image;

[0039] enhancing a local contrast of the first grayscale image according to a brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates a brightness difference of the grayscale blocks in the first grayscale image;

[0040] The second grayscale image is color restored based on the color of the first image to obtain a second image.

[0041] In a possible implementation, enhancing the local contrast of the first grayscale image according to the brightness range of the first grayscale image to obtain the second grayscale image includes:

[0042] Obtain a base layer pixel value and a detail layer pixel value corresponding to any grayscale block, wherein the base layer pixel value is the base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value is the detail layer data of the grayscale value of the central pixel point of any grayscale block;

[0043] According to the brightness range, adjusting the base pixel value to obtain a first pixel value;

[0044] A second pixel value corresponding to a central pixel point of any grayscale block is determined according to the first pixel value and the detail layer pixel value, and a second grayscale image is determined based on the second pixel values ​​of each grayscale block.

[0045] In a possible implementation manner, determining the second pixel value corresponding to the central pixel point of any grayscale block according to the first pixel value and the detail layer pixel value includes:

[0046] The detail layer pixel value is increased, the first pixel value is decreased, and the second pixel value is determined based on the increased detail layer pixel value and the decreased first pixel value.

[0047] In one possible implementation, the brightness range includes at least one of a local brightness range or a global brightness range, the local brightness range includes the maximum and minimum values ​​of the grayscale values ​​of each pixel in any grayscale block, and the global brightness range includes the maximum and minimum values ​​of the grayscale values ​​of each pixel in the first grayscale image.

[0048] In a possible implementation manner, the global brightness range of the first grayscale image is the same as the global brightness range of an adjacent grayscale image, and the adjacent grayscale image is a grayscale image corresponding to an adjacent frame image of the first image.

[0049] In a possible implementation, performing color restoration on the second grayscale image based on the color of the first image to obtain the second image includes:

[0050] Determine the number of pixels of each gray value according to the gray value of each pixel in the second gray image, and obtain distribution information of the second gray image;

[0051] determining a contrast enhancement curve of the second grayscale image according to the distribution information;

[0052] enhancing the global contrast of the second grayscale image according to the contrast enhancement curve to obtain a third grayscale image, wherein the global contrast indicates a brightness difference in the second grayscale image;

[0053] The third grayscale image is color restored based on the color of the first image to obtain the second image.

[0054] In a possible implementation manner, determining a contrast enhancement curve of the second grayscale image according to the distribution information includes:

[0055] Acquire distribution information of a fourth grayscale image, where the fourth grayscale image indicates brightness of a third image, where the third image is a previous frame image of the first image;

[0056] It is determined whether a scene change occurs in the first image relative to the third image according to the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and a contrast enhancement curve of the second grayscale image is determined according to the determination result.

[0057] In a possible implementation manner, determining the contrast enhancement curve of the second grayscale image according to the judgment result includes:

[0058] When the judgment result indicates that the first image has no scene change relative to the third image, determining the contrast enhancement curve of the fourth grayscale image as the contrast enhancement curve of the second grayscale image;

[0059] Alternatively, when the judgment result indicates that a scene change occurs in the first image relative to the third image, a contrast enhancement curve of the second grayscale image is calculated according to the distribution information of the second grayscale image.

[0060] In a possible implementation manner, determining a contrast enhancement curve of the second grayscale image according to the distribution information includes:

[0061] The number of pixel points of each gray value is balanced based on the number threshold, and the difference in the number of pixel points of each gray value after the balance is smaller than the difference in the number of pixel points of each gray value in the distribution information;

[0062] A first enhancement curve of the second grayscale image is determined according to the number of each pixel after the equalization, and the contrast enhancement curve is determined based on the first enhancement curve.

[0063] In a possible implementation, balancing the number of pixels of each gray value based on the number threshold includes:

[0064] For a first gray value in which the number of pixel points is greater than a quantity threshold, adjusting the number of pixel points of the first gray value from the first quantity to the quantity threshold;

[0065] adjusting the number of pixel points of the second grayscale value according to a second number, wherein the second number is obtained by adding the difference between the first number of each first grayscale value and the number threshold, and the second grayscale value is different from the first grayscale value;

[0066] In a possible implementation manner, determining the contrast enhancement curve based on the first enhancement curve includes:

[0067] In the case where the first image is a dark light scene, the first enhancement curve is adjusted using a second enhancement curve to obtain the contrast enhancement curve, and the second enhancement curve is used to enhance the global contrast of the image in the dark light scene.

[0068] In a possible implementation manner, determining the contrast enhancement curve based on the first enhancement curve includes:

[0069] The first enhancement curve is adjusted using a smoothing coefficient to obtain the contrast enhancement curve, wherein the smoothing coefficient is used to reduce a sudden change between the first image and adjacent frame images of the first image.

[0070] On the other hand, an electronic device is provided, comprising any one of the above-mentioned image processing chips.

[0071] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by an image processor so that the computer implements any of the above-mentioned image processing methods.

[0072] On the other hand, a computer program product or a computer program is also provided, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. An image processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the image processor executes the computer instructions, so that the electronic device performs any of the above-mentioned image processing methods.

[0073] The technical solution provided by this application brings at least the following beneficial effects:

[0074] In the process of enhancing the local contrast, the brightness range of the first grayscale image is referred to to achieve accurate enhancement of the local contrast for the brightness range, so that the local hue effect of the second image obtained by color adjustment is better and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present application;

[0077] Figure 2 is a schematic diagram of adjusting a mapping curve provided in an embodiment of the present application;

[0078] Figure 3 It is a distance mapping schematic diagram provided in an embodiment of the present application;

[0079] Figure 4 is a schematic diagram of a process for determining a first enhancement curve provided in an embodiment of the present application;

[0080] Figure 5 is a schematic diagram of a histogram provided in an embodiment of the present application;

[0081] Figure 6 This is a glare schematic diagram provided in an embodiment of the present application;

[0082] Figure 7 is a schematic diagram of adjusting a grayscale image provided in an embodiment of the present application;

[0083] Figure 8 is a structural schematic diagram of an image processing chip provided in an embodiment of the present application;

[0084] Fig. 9 It is a schematic diagram of the structure of another image processing chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0086] In the field of image display technology, there is a high dynamic scene with high brightness, high color saturation and high contrast. When using a general image sensor to shoot a high dynamic scene, it is often only possible to choose to properly expose the low-brightness area, causing the highlight area to be overexposed and the details of the highlight area to be lost, or to choose to properly expose the highlight area, causing the low-brightness area to be underexposed and the details of the low-brightness area to be difficult to recognize. Based on this, a high dynamic range image sensor or HDR (High Dynamic Range Imaging) synthesis technology can be used to obtain a high dynamic range image to record the details of the high dynamic range scene.

[0087] However, the high dynamic range image displayed on some displays cannot match the real scene perceived by the human eye, such as 8-bit or 10-bit displays. Therefore, tone mapping technology is used to correct the brightness of the high dynamic range image so that the corrected high dynamic range image matches the perceived real scene, thereby reducing the difference between the high dynamic range image displayed on the display and the scene observed by the human eye.

[0088] Among them, tone mapping technology is an image processing technology that approximates the display of high dynamic range images on limited dynamic range media. The scene brightness is transformed to a displayable range through large contrast attenuation while maintaining information such as image details and color, so that the scene after tone mapping matches the perception of the real scene.

[0089] In some cases, tone mapping technology is divided into GTM (Global Tone Mapping) and LTM (Local Tone Mapping). Global tone mapping maps the same grayscale value in the image to the same mapping value through a mapping curve to improve the global brightness and contrast. Local tone mapping is based on the local feature information of the image and maps the same grayscale value to different mapping values ​​according to the different positions of the original pixels in the image.

[0090] In the related art, during the local tone mapping process, the image is decomposed into a base layer and a detail layer, and a fixed contrast curve is used to map the base layer brightness and the detail layer brightness. Then, a new grayscale image is determined based on the mapped base layer brightness and the detail layer brightness, and the new grayscale image is color restored to obtain an enhanced image. However, the contrast curve referenced by the above local tone mapping is fixed, and the curve cannot adapt to the real dynamic range of the image, resulting in unsatisfactory results of the local tone mapping.

[0091] The present application embodiment provides an image processing method, the flowchart of which is as follows: Figure 1 As shown, it includes steps 101 to 103.

[0092] In step 101, a first grayscale image is acquired based on a first image, where the first grayscale image indicates the brightness of the first image.

[0093] In a possible implementation, the image processing method provided in the embodiment of the present application is applied to a processing device, and the processing device can be any device with processing functions, such as a terminal or a server. Optionally, an image processing chip is configured in the processing device, and the processing device processes the first image through the image processing chip to obtain a second image. Alternatively, the processing device loads and executes a computer program through a processor to implement the image processing method provided in the embodiment of the present application, that is, the process of the processing device executing the image processing method described below can be a processing process on the software side or a processing process on the hardware side.

[0094] Exemplarily, the processing device obtains a first image to be displayed, extracts a grayscale image indicating brightness from the first image, and obtains a first grayscale image. The first image may be a captured image. For example, an image capture device is configured in the processing device, and the image capture device is used to capture an image to obtain the first image. The image capture device may be referred to as an image sensor in some cases. Taking the processing device as a mobile phone as an example, the image capture device may be a camera on the mobile phone.

[0095] Optionally, the first image may also be an image obtained by accessing a storage space, and the accessed storage space may be a storage space of a processing device or a storage space of other devices. The first image may also be an image obtained by searching on the Internet.

[0096] The embodiment of the present application does not limit the image type of the first image, which may be a static image, that is, the image format of the first image is jpg (Joint Photographic Experts Group). The first image may also be one frame of a multi-frame image, and the multi-frame image may be a dynamic image belonging to gif (Graphics Interchange Format) or a video.

[0097] Regardless of the method and type of the first image, a grayscale image of the first image can be extracted and recorded as the first grayscale image. In the case where the first image includes multiple pixels, the grayscale value of each pixel can be determined to obtain a grayscale image including multiple grayscale values. The grayscale value is the brightness value of the brightness channel after the RGB (Red Green Blue) three channels are normalized. Optionally, the maximum value on the RGB channel of each pixel can be used as the grayscale value of each pixel, that is, the grayscale value of the pixel is determined using Formula 1.

[0098] I(x,y)=MAX(HDR r (x,y),HDR g (x,y),HDR b (x,y)) (Formula 1)

[0099] I in formula 1 indicates the gray value, (x, y) is the coordinate of the pixel point on the first image, which is used to identify the pixel point, HDR r (x,y) indicates the brightness of the pixel on the R channel, HDR g (x,y) indicates the brightness of the pixel on the G channel, HDR b (x,y) indicates the brightness of the pixel on the B channel, and MAX indicates the value from HDR r (x,y),HDR g (x,y) and HDR b Select the maximum brightness among (x,y).

[0100] Exemplarily, the grayscale value of a pixel point can also be calculated by other methods, for example, by weighting the brightness value of any pixel point on the RGB channel to obtain the grayscale value of any pixel point. The processing device can directly determine the first grayscale image according to the calculated grayscale value, generate a grayscale image according to the grayscale value of each pixel point, and use the generated grayscale image as the first grayscale image. The calculated grayscale value can also be processed to obtain the first grayscale image.

[0101] In a possible implementation, after the grayscale value of each pixel is calculated, the grayscale value may be compressed, and the compression formula used is, for example, Formula 2.

[0102]

[0103] In formula 2, I(x,y) indicates the gray value of the pixel, and t indicates the compression degree of the logarithmic curve. The larger t is, the stronger the compression degree of the logarithmic curve is, and the more obvious the brightness improvement of the dark part of the compressed gray image is. On the contrary, the smaller t is, the lower the compression degree of the logarithmic curve is, and the less obvious the brightness improvement of the dark part of the compressed gray image is, and the brightness is darker. log(t+1) is the normalization parameter, L in Indicates the compressed grayscale value.

[0104] After compressing the grayscale value of each pixel using Formula 2, the processing device can use the compressed grayscale value to determine the first grayscale image, and the brightness value of any pixel on the first grayscale image is the compressed grayscale value.

[0105] In step 102, the local contrast of the first grayscale image is enhanced according to the brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates the brightness difference of the grayscale blocks in the first grayscale image.

[0106] In a possible implementation, if the first image is a high dynamic range image, that is, an image obtained by shooting a high dynamic scene, directly displaying the first image will result in a brightness difference between the displayed first image and the high dynamic scene viewed by human eyes. For example, when the image acquisition device used is a limited dynamic range medium that can only approximately display a high dynamic range image, if a suitable exposure is selected for the dark area when shooting the first image, it is easy to overexpose the highlight area and lose the details of the highlight area. If a suitable exposure is selected for the highlight area when shooting the first image, the low-brightness area will be underexposed and the details of the low-brightness area will be difficult to recognize.

[0107] Therefore, it is necessary to compress the dynamic range of the image while retaining the details through tone mapping technology, so as to transform the scene brightness to a displayable range through a large contrast attenuation, and maintain information such as image details and colors, so that the scene after tone mapping matches the perception of the real scene. Among them, the dynamic range DR of the image is also the brightness range of the image, which is determined by the maximum and minimum brightness values.

[0108] For example

[0109] Optionally, the color adjustment of the first image can be achieved by enhancing the local contrast of the first grayscale image. The process of enhancing the local contrast may be referred to as LTM (Local Tone Mapping) in some cases, which is used to adjust the grayscale value according to the local feature information of the image, that is, the position of the pixel in the first image. In the process of mapping using LTM, even if the grayscale values ​​of two pixels in the first grayscale image are the same, the grayscale values ​​obtained by mapping are different due to the different positions of the two pixels.

[0110] In one possible implementation, the local contrast of a first grayscale image is enhanced according to the positions of pixels to obtain a second grayscale image, the process comprising: obtaining a base layer pixel value and a detail layer pixel value corresponding to any grayscale block, the base layer pixel value being base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value being detail layer data of the grayscale value of the central pixel point of any grayscale block; adjusting the base layer pixel value according to the brightness range to obtain a first pixel value; determining a second pixel value corresponding to the central pixel point of any grayscale block according to the first pixel value and the detail layer pixel value, and determining a second grayscale image according to the second pixel values ​​of each grayscale block.

[0111] Optionally, after acquiring the first grayscale image, a sliding window may be used to divide the first grayscale image into grayscale blocks, and the number of grayscale blocks is the same as the number of pixels in the first grayscale image. The embodiment of the present application does not limit the size of the sliding window, and it may be any size set based on experience, such as 5×5, 7×7, or other sizes. In addition, the sliding window may also be other shapes. The processing device uses a sliding window to scan each pixel position in the first grayscale image in turn, and the scanning order is, for example, from top to bottom and from left to right.

[0112] Next, the process of dividing and obtaining multiple grayscale blocks is described by taking a sliding window size of 7×7 as an example. For each pixel in the first grayscale image, a 7×7 window area is cut with each pixel as the center to obtain a grayscale block. For pixels located at the image boundary, when the window area is taken with the pixel as the center, the window area taken will exceed the image area. Therefore, the window area can be filled, for example, by mirroring.

[0113] Optionally, in the case where the processing device performs layered processing on the base layer and the detail layer in the process of enhancing the local contrast, the processing device may further perform layered processing using a filter after acquiring the grayscale block to obtain a base layer image at a low frequency and a detail layer image at a high frequency. Continuing with the example of taking a window area of ​​size 7×7 with each pixel as the center in the above embodiment, after determining the window area, a filter may be further used in the window area for filtering processing, and the filter used may be an edge-preserving filter, such as a guided filter.

[0114] In the case where the guided filter includes two image input ports, grayscale blocks can be input to the two image input ports, and one grayscale block is used as the input image L in , and the other is used as a guide image. The guide filter refers to the guide image, filters the input image to obtain the base layer image B at the low frequency, and then subtracts the input image from the base layer image B to obtain the detail layer image D=L at the high frequency. in -B.

[0115] Exemplarily, the processing device may also filter the first grayscale image first, and then use a sliding window to cut the filtered base layer image and detail layer image to obtain detail layer blocks and base layer blocks corresponding to each grayscale block. Regardless of the order in which the layering and cutting operations are performed, the grayscale value of the central pixel point of any grayscale block in the base layer can be used as the base layer pixel value, and the grayscale value of the central pixel point of any grayscale block in the detail layer can be used as the detail layer pixel value. Among them, the base layer is the basic layer in the image, which contains the basic structure and main information of the image, and the detail layer contains the detailed information in the image, such as texture, edge, etc.

[0116] After determining the base layer pixel value and detail layer pixel value corresponding to the grayscale block, the base layer pixel value may be adjusted according to the brightness range to obtain the first pixel value, wherein the brightness range includes at least one of a local brightness range and a global brightness range, the local brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel point in any grayscale block, and the global brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel point in the first grayscale image.

[0117] The maximum value of a grayscale block in a local brightness range may in some cases be referred to as a local maximum value Max local , the minimum value can be called the local minimum value Min in some cases local After the grayscale block is divided, the processing device may take the maximum value among the multiple grayscale values ​​in the grayscale block as the local maximum value Max local , take the minimum value among multiple gray values ​​in the gray block as the local minimum value Min local .

[0118] The maximum value in the global brightness range may in some cases be referred to as the global maximum value Max global , the minimum value in the global brightness range can be called the global minimum value Min in some cases global After acquiring the first grayscale image, the processing device may use the maximum value of the grayscale values ​​in the first grayscale image as the global maximum value Max global , take the minimum grayscale value in the first grayscale image as the global minimum value Min global .

[0119] Optionally, the global maximum and global minimum values ​​can also be determined based on adjacent frame images. For example, the global brightness range of the first grayscale image is the same as the global brightness range of the adjacent grayscale images, wherein the adjacent grayscale images are grayscale images corresponding to adjacent frame images of the first image. In this case, the global maximum and global minimum values ​​of each grayscale image can be determined based on the image dynamic range actually generated by the automatic exposure module of the image acquisition device during the image capture process. By controlling the values ​​of the global maximum and global minimum values ​​of adjacent frame images to be the same, it is ensured that the base images of adjacent frame images are displayed within the same brightness range to avoid image flickering.

[0120] For different brightness ranges, the base pixel value may be adjusted in the following two ways, including but not limited to, to obtain the first pixel value.

[0121] Adjustment method 1: adjust the base pixel value according to the local brightness range to obtain the first pixel value. The adjustment process is shown in formula 3.

[0122]

[0123] B in Formula 3 local Indicates the base pixel value after local adjustment, Min local Indicates the local minimum value of the grayscale block, Max local Indicates the local maximum of the grayscale block, f b(*) is a mapping function used to map the base layer, which belongs to the global mapping curve and is used to map the local maximum and local minimum values ​​to the range of 0 to 1 to improve the brightness and contrast of the grayscale block. Referring to Formula 3, when mapping the base layer pixel values, the local maximum and local minimum values ​​of the grayscale block are also referenced, so that local image information is considered on the basis of mapping with reference to the global mapping curve to ensure the local contrast of the grayscale block after mapping.

[0124] Adjustment method 2: adjust the base pixel value according to the global brightness range to obtain the first pixel value. The adjustment process is shown in formula 4.

[0125]

[0126] B in Formula 4 global Indicates the base pixel value after global adjustment, Min global Indicates the global minimum value of the first grayscale image, Max global Indicates the global maximum of the first grayscale image, f b (*) is a mapping function for mapping the base layer. The grayscale value is corrected in the process of mapping the base layer pixel value by using the global maximum and global minimum of the first grayscale image. Even if the first image is a frame image in a video, and the scene of each frame in the video changes, resulting in a change in the global brightness range of the image, the global mapping curve can be adjusted to ensure that the global mapping curve can meet the mapping requirements of each frame image and ensure the mapping accuracy.

[0127] Optionally, you can choose to perform adjustment method 1 and change B local As the first pixel value, you can also choose to perform adjustment method 2, and change B globaI As the first pixel value, the adjustment method 1 and the adjustment method 2 may be performed in combination, that is, after executing formula 3 and formula 4, formula 5 is further performed to obtain the first pixel value.

[0128] B fusion =fusion w ×B local +(1-fusion w )×B global (Formula 5)

[0129] In formula 5, B fusion To integrate B local and B global Gray value, fusion w Indicates the weight referenced during fusion, fusion w It can be a fixed value set based on experience, or a weight calculated based on the brightness value or local statistics.

[0130] According to formula 5, local and B global Perform weighted summation to obtain a new grayscale value. Due to the small amount of statistical information in the local operation, B local The mapping information is greatly affected by local extreme values, but it can effectively improve local contrast and brightness. The global statistical information is rich, but it cannot take local contrast and brightness into account. Therefore, the weighted summation can be performed through formula 5 to balance the effects of the two, and the first pixel value that can improve both local contrast and brightness and overall contrast and brightness can be obtained.

[0131] Since the mapping curve changes with the local and global maximum and minimum values, the above adjustment process can be understood as dynamically adjusting f according to the image content. b (*).like Figure 2 The figure shows a modified f b (*) Mapping curve. Figure 2 Use double (double-precision floating point) type values ​​to represent grayscale values. Figure 2 The horizontal axis indicates the grayscale value before mapping, and the vertical axis indicates the grayscale value after mapping. Figure 2 (1) shows the original f b (*) Mapping curve, Figure 2 (2) shows the f of the local correction using adjustment method 1. b (*) Mapping curve, Figure 2 (3) shows the global correction of f using adjustment method 2. b (*) Mapping curve.

[0132] In addition, since the noise of the first grayscale image is located in the detail layer, the above adjustment methods are all aimed at the base pixel values ​​located in the base layer, that is, the noise of the first grayscale image will not be amplified due to the above mapping stretching, which effectively controls the noise and ensures the accuracy of the first pixel value obtained by mapping.

[0133] After determining the first pixel value of each grayscale block, the second pixel value corresponding to the central pixel point of any grayscale block can be determined according to the first pixel value and the detail layer pixel value, and then the second grayscale image is obtained based on the second pixel value of each grayscale block. Exemplarily, the detail layer pixel value is increased, the first pixel value is decreased, and the second pixel value is determined based on the increased detail layer pixel value and the decreased first pixel value. The adjustment process can be referred to in Formula 6.

[0134] T=k1×B+k2×f d (D)(Formula 6)

[0135] T in Formula 6 indicates the adjustment to obtain the second pixel point, f d(*) is a mapping function for mapping the detail layer pixel value located in the detail layer, B is the first pixel value in the above embodiment, if the first pixel value is obtained based on the adjustment method 1, B corresponds to the above B local If the first pixel value is obtained based on the adjustment method 2, B corresponds to the above B global If the first pixel is obtained by combining adjustment method 1 and adjustment method 2, B corresponds to the above B fusion D corresponds to the detail layer pixel in the above embodiment. k1∈(0,1), k2≥1.

[0136] Through formula 6, the local details of the central pixel of any grayscale block are enhanced by k2 times and then superimposed on the base layer B weakened by k1 times to highlight the contrast of local details and the change of details to increase the visual impact. After local tone mapping by layer decomposition and reorganization enhancement, the local difference of T is amplified, that is, the local contrast is increased.

[0137] The processing device may perform the operation of step 102 on each grayscale block in the first grayscale image to obtain a second pixel value of each grayscale block, thereby acquiring a second grayscale image composed of a plurality of second pixel values.

[0138] In step 103, the second grayscale image is color restored based on the color of the first image to obtain a second image.

[0139] In one possible case, the second grayscale image may have a perceived difference in the global brightness range of the image. For example, in step 102, when increasing the pixel value of the detail layer, the global brightness range of the image is not referenced, and the perceived difference in the global brightness range of the detail layer of the second grayscale image thus obtained may cause the second grayscale image to appear gray or blurry.

[0140] In addition, if the first image is a frame in a video, in video processing, if the actual dynamic range of each frame cannot be perceived, the brightness range of the mapping results of adjacent frames will be different, and the problem of video flickering may occur, affecting the effect of video image processing. Secondly, after layer decomposition is used for reorganization and enhancement, although the local contrast is increased, the global contrast of the image is low, the image is blurred, and the viewing experience is poor. Therefore, after the local contrast enhancement of the first grayscale image, the global contrast is further enhanced based on the dynamic range of the image, that is, the global brightness range. For example, the grayscale value of the pixel in the second grayscale image is adjusted.

[0141] Exemplarily, the number of pixels of each grayscale value is determined according to the grayscale value of each pixel in the second grayscale image to obtain distribution information of the second grayscale image; a contrast enhancement curve of the second grayscale image is determined according to the distribution information; the global contrast of the second grayscale image is enhanced according to the contrast enhancement curve to obtain a third grayscale image, wherein the global contrast indicates the brightness difference in the second grayscale image.

[0142] In a possible implementation, the distribution information of the second grayscale image can be calculated using a histogram. When the distribution information is calculated using a histogram, the value interval corresponds to the grayscale value, and the data point corresponds to the pixel point on the second grayscale image. The distribution information can be calculated using the histogram according to Formula 7.

[0143] Hist=CalcHist(T(x,y))(Formula 7)

[0144] Among them, Hist indicates distribution information, CalcHist(*) represents the calculation of image histogram, T(x, y) indicates the grayscale value of the pixel point on the second grayscale image, and using Formula 7, the number of pixel points at any grayscale value can be determined according to the grayscale value of each pixel point, thereby obtaining the distribution information of the second grayscale image.

[0145] For the case where the grayscale value of the second grayscale image is in the range of 0 to 255 and there are 256 grayscale values ​​in total, the number of bins (buckets) of the image histogram calculated by Formula 7 can be 256 or other numbers. Other numbers are, for example, 64, or other numbers that are divisible by 256. By reducing the number of bins of the image histogram, the calculation overhead of the histogram is reduced, and the processing efficiency of subsequent global enhancement using the histogram is improved.

[0146] Optionally, other methods may be used to count the distribution information of the second grayscale image, for example, the number of pixels corresponding to each grayscale value may be counted in the form of a table. Regardless of the method used to obtain the distribution information of the second grayscale image, a contrast enhancement curve for global enhancement may be determined based on the distribution information, and the determination process includes but is not limited to the following two methods.

[0147] Determination process 1: obtaining distribution information of a fourth grayscale image, the fourth grayscale image indicating the brightness of a third image, the third image being a previous frame image of the first image; judging whether a scene change occurs in the first image relative to the second image based on the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and determining a contrast enhancement curve of the second grayscale image based on the judgment result.

[0148] In a possible implementation, there is a third image in the previous frame of the first image, for example, the first image is an image in a video, and the first image and the third image are adjacent frame images in the video. The processing device can determine the contrast enhancement curve of the second grayscale image of the first image based on the fourth grayscale image of the third image, and the fourth grayscale image is a grayscale image that has not been globally adjusted. The process of obtaining the distribution information of the fourth grayscale image of the third image is similar to the process of obtaining the distribution information of the second grayscale image of the first image, and the relevant description can be referred to, and the description will not be repeated here.

[0149] Optionally, the error between the second grayscale image and the fourth grayscale image can be calculated using the distribution information of the second grayscale image and the distribution information of the fourth grayscale image to reflect the similarity between the second grayscale image and the fourth grayscale image. The calculated error is, for example, covariance, which is used to represent the overall error of two variables. If the change trends of the two variables are consistent, that is, one is greater than its expected value and the other is also greater than its expected value, the covariance between the two variables is positive. If the change trends of the two variables are inconsistent, that is, one is greater than its expected value and the other is not greater than its expected value, the covariance between the two variables is negative.

[0150] In one possible case, the first covariance between the distribution information of the second grayscale image and the distribution information of the fourth grayscale image can be calculated, and the second covariance between the distribution information of the second grayscale image and the distribution information of the fourth grayscale image can be calculated. For example, the frequency is first calculated based on the distribution information using Formula 8, and then the first covariance and the second covariance are calculated based on the frequency.

[0151]

[0152] Where p[i] indicates the frequency, Hist[i] indicates the number of pixels of any bin value in the distribution information, i is any integer indicating a bin in the distribution information, and NUM indicates the total number of pixels in the grayscale image. Formula 8 can be used to determine the proportion of each bin in the total number of pixels in the grayscale image, that is, the frequency.

[0153] After the frequency of the second grayscale image and the frequency of the fourth grayscale image are calculated using the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, the first covariance can be calculated based on the frequency using Formula 9 and Formula 10.

[0154]

[0155]

[0156] In Formula 9, k indicates the number of bins in the distribution information, which can be 256 or 64.n [i] indicates the frequency of the second grayscale image, p p [i] indicates the frequency of the fourth grayscale image, corrp n _p p Indicates the calculated correlation coefficient. In formula 10, covp n _p p indicating a first covariance between the frequency of the fourth grayscale image and the frequency of the second grayscale image, indicates the mean value calculated based on the frequency of the second grayscale image, Indicates the mean value calculated based on the frequency of the fourth grayscale image.

[0157] Optionally, Formula 11 and Formula 12 may be used to calculate the frequency of the second grayscale image and the covariance between the frequencies of the second grayscale images.

[0158]

[0159]

[0160] In formula 11, corrp n indicates the correlation coefficient, k indicates the number of bins in the distribution information, and p n [i] indicates the frequency of the second grayscale image. In formula 12, varp n the frequency of the second grayscale image and the second covariance between the frequencies of the second grayscale image, Indicates the mean value calculated based on the frequency of the second grayscale image.

[0161] After the first covariance and the second covariance are calculated using Formula 9 to Formula 12, Formula 13 may be further used to calculate the distance between the first covariance and the second covariance to determine the similarity between the second grayscale image and the fourth grayscale image based on the distance.

[0162] diff=ABS(covp n _p p -varp n ) (Formula 13)

[0163] Among them, diff indicates the distance between the first covariance and the second covariance, covp n _p p Indicates the first covariance, varp n Indicates the second covariance. ABS is the absolute value function, which is used to calculate the absolute value.

[0164] Since the distance between the first covariance and the second covariance can reflect the similarity between the second grayscale image and the fourth grayscale image, the scene change can be judged based on the distance, and the distance and the similarity threshold are compared. The similarity threshold is an arbitrary value set based on experience, and the similarity threshold is, for example, 0 or other non-negative numbers.

[0165] For example, when the distance is not greater than the similarity threshold, it is determined that the second grayscale image and the fourth grayscale image have a high similarity, and since the second grayscale image is a normalized grayscale image of the first image, and the fourth grayscale image is a normalized grayscale image of the third image. If the second grayscale image and the fourth grayscale image have a high similarity and the proportions of each grayscale value are close, it can be said that the image contents of the first image and the third image are also similar, that is, there is no scene change in the first image relative to the third image.

[0166] For the case where the similarity threshold is 0, the above judgment process can also be expressed as judging whether the first covariance is equal to the second covariance. If covp n _p n =varp n , that is, when the first covariance is equal to the second covariance, it can be determined that there is no scene change in the first image relative to the third image.

[0167] In addition to directly comparing the distance and similarity threshold, you can also map the distance to between 0 and 1, and then determine the size of the mapped distance and similarity threshold. Figure 3 A schematic diagram of distance mapping provided in an embodiment of the present application, Figure 3 The horizontal axis indicates the distance before mapping, and the vertical axis indicates the distance after mapping, which can be called the weight of the distance in some cases. Since the value of the distance is not fixed, it can be large or small, and the difference between different values ​​is large, by mapping the distance to between 0 and 1, it can better reflect the size relationship between the first covariance and the second covariance, thereby better reflecting the similarity between the second grayscale image and the fourth grayscale image, so as to accurately judge the result.

[0168] Regardless of the method used to determine the judgment result, the contrast enhancement curve of the second grayscale image can be determined based on the judgment result. For example, when the judgment result indicates that there is no scene change in the first image relative to the third image, the contrast enhancement curve of the fourth grayscale image is determined as the contrast enhancement curve of the second grayscale image; or, when the judgment result indicates that there is a scene change in the first image relative to the third image, the contrast enhancement curve of the second grayscale image is calculated based on the distribution information of the second grayscale image.

[0169] If the first image does not have a scene change relative to the third image, the second grayscale image and the fourth grayscale image have similar global brightness ranges and similar numbers of pixels at each grayscale value, and the process of using the contrast enhancement curve to perform global contrast enhancement is to adjust the number of pixels at each grayscale value so that the number of pixels at each grayscale value is more evenly distributed. For example, a mathematical method is used to readjust the brightness distribution of pixels so that the adjusted histogram has the largest dynamic range and the number of pixels contained in each bin / bucket is close.

[0170] Therefore, the contrast enhancement curve of the fourth grayscale image can be reused, and the contrast enhancement curve used to adjust the fourth grayscale image can be used as the contrast enhancement curve of the second grayscale image. The calculation process of the contrast enhancement curve of the fourth grayscale image is similar to the calculation process of the contrast enhancement curve of the second grayscale image, and the relevant description of the determination process 1 or the determination process 2 can be referred to, and will not be repeated here.

[0171] Based on similar reasons, when there is a scene change in the first image relative to the third image, it can also be determined that the similarity between the second grayscale image and the fourth grayscale image is low. In this case, since the contrast enhancement curve of the fourth grayscale image cannot be reused, the contrast enhancement curve of the second grayscale image will be recalculated based on the distribution information of the second grayscale image. For example, the contrast enhancement curve is calculated by executing the second determination process.

[0172] Determine process two: balance the number of pixels of each gray value based on a quantity threshold, so that the difference in the number of pixels of each gray value after equalization is smaller than the difference in the number of pixels of each gray value in the distribution information; determine a first enhancement curve of the second gray image according to the number of pixels after equalization, and determine a contrast enhancement curve based on the first enhancement curve.

[0173] Optionally, the quantity threshold may be any value set based on experience and implementation environment. For example, the implementation environment may be the total number of pixels in the second grayscale image. The processing device may determine the quantity threshold using Formula 14.

[0174] ClipLimit = clip_ratio * NUM (Formula 14)

[0175] Among them, ClipLimit is a quantity threshold, which can be called a clipping value in some cases, clip_ratio is a preset contrast parameter, which is an arbitrary integer in the range of 0 to 1, and NUM is the total number of pixels of the second grayscale image. After determining the quantity threshold, the number of pixels of each grayscale value of the second grayscale image can be balanced based on the quantity threshold. The balancing process includes but is not limited to: for a first grayscale value whose number of pixels is greater than the quantity threshold, adjusting the number of pixels of the first grayscale value from the first number to the quantity threshold; adjusting the number of pixels of the second grayscale value according to the second number, the second number is obtained by adding the difference between the first number of each first grayscale value and the quantity threshold, and the second grayscale value is different from the first grayscale value.

[0176] Optionally, the processing device first determines the second number according to the number threshold and the number of pixels of the first gray value in the distribution information. Figure 4 A schematic diagram of a process for determining a first enhancement curve provided in an embodiment of the present application is shown in FIG. Figure 4 The horizontal axis of (1)-(4) is the gray value, and the vertical axis is the number of pixels. Figure 4 The horizontal axis of (5) is the grayscale value before mapping, and the vertical axis is the grayscale value after mapping. Figure 4 (1) indicates the distribution information before adjustment, that is, the histogram determined based on the grayscale values ​​of each pixel in the second grayscale image, Figure 4 The dotted line in (2) indicates a quantity threshold, and the second quantity indicates the partial sum of all bins in the histogram that are above the quantity threshold, which may be referred to as totalExcess in some cases.

[0177] After determining the second number, the second number can be evenly distributed to each grayscale value. The evenly distributed grayscale value can be the second grayscale value, that is, the number of pixels is not less than the grayscale value of the number threshold. Then the overall rising height of the histogram L = totalExcess / bin num Among them, bin num is the number of bins in the histogram. The above process can be summarized as follows: for bin values ​​in the histogram whose amplitude is higher than ClipLimit, directly set it to ClipLimit, that is, use ClipLimit to clip the histogram. If the amplitude is between Upper and ClipLimit, fill the amplitude of the bin value to ClipLimit, upper = ClipLimit-L. If the amplitude is lower than Upper, directly fill L pixels.

[0178] Through the above operation, the number of pixels used for filling will be less than totalExcess, that is, there are some remaining pixels that have not been allocated. The remaining pixels can be found in Figure 4The remaining number in (3) indicates the distribution of the number of pixels not allocated in the second grayscale image. In this case, the processing device can continue to evenly distribute the remaining number to the grayscale values ​​whose amplitude is still less than ClipLimit, and obtain Figure 4 The histogram shown in (4) indicates the number of pixels of each gray value after equalization. Optionally, the processing device can also add L directly to all bins without continuing to allocate them one by one. The above process can be called reducing the excess and making up for the deficiency, or peak shaving and valley filling, which evenly distributes the excess probability to other pixels so that the brightness gain is relatively evenly distributed to all pixels.

[0179] Regardless of the method used to balance the number of pixels of each grayscale value, the contrast enhancement curve can be determined based on the number of equalized pixels. For example, by using Formula 15, the cumulative distribution function of the contrast enhancement curve is obtained by accumulating the proportion of the number of pixels in each bin in the histogram in the image.

[0180]

[0181] Among them, C1 k is the contrast enhancement curve, k indicates bin, the value range of k in formula 15 is 0 to 255, j is the identifier of the bin, and n j is the number of pixels in the bin. Using formula 15 Figure 4 By calculating the histogram of (4), we can get Figure 4 The first enhancement curve shown in (5).

[0182] Optionally, the processing device may choose to directly execute determination process two to calculate the first enhancement curve based on the distribution information of the second grayscale image, or may first execute determination process one, and based on the judgment result in determination process one, selectively execute determination process two, so as to avoid the need to perform enhancement curve calculation operations on each frame of grayscale image, thereby reducing computational overhead.

[0183] In addition, after the first enhancement curve is calculated, the first enhancement curve can be directly used as the contrast enhancement curve, or the first enhancement curve can be adjusted to obtain the contrast enhancement curve. For example, the contrast enhancement curve can be obtained by adjusting including but not limited to the following adjustment process.

[0184] Adjustment process 1: When the first image is a dark light scene, the first enhancement curve is adjusted using the second enhancement curve, where the second enhancement curve is used to enhance the global contrast of the image in the dark light scene.

[0185] Among them, dark light scenes include but are not limited to backlight or night scenes. The histogram of dark light scenes is quite special. In an image in a backlight scene, there are highlight areas and darker backlight areas. The histogram of this image shows a trend of being sharp at both ends and less in the middle, such as Figure 5 As shown, Figure 5 The horizontal axis indicates the grayscale value (in this case, the grayscale value is represented by a double value), and the vertical axis indicates the number of pixels. In this case, although adaptive enhancement is performed based on the histogram of the image, since the contrast parameters used for cropping are the same, the contrast parameters that meet the requirements for ordinary scenes cannot meet the requirements for backlit scenes.

[0186] For night scene images, the number of pixels in the dark part of the histogram is relatively concentrated. During the cropping process, these concentrated pixels will be divided into larger pixel value bin areas. This operation will cause the glare and highlight overflow of some highlights or light source areas in the picture to become larger in some special processing scenarios, such as the scene of the on-board ISP (Image Signal Processor), which will affect the image quality during driving. Figure 6 shown. Figure 6 The left picture is the first grayscale image before adjustment. Figure 6 The right picture is the grayscale image obtained by directly adjusting the first enhancement curve. Figure 6 The highlight overflow is larger in the right image than in the left image.

[0187] Therefore, it is possible to first determine whether the first image is in a dark light scene based on the distribution information of the second grayscale image, and if it is in a dark light scene, the first enhancement curve is adjusted using the second enhancement curve. For example, formulas 16 to 18 may be used to determine whether the first image is in a dark light scene.

[0188] dark pixel =a×NUM (Formula 16)

[0189]

[0190] Among them, dark pixel is the black threshold, a is the pixel threshold set based on experience, indicating how many proportions of pixels in the image are black pixels, that is, it is judged as a dark light scene, and NUM is the total number of pixels in the second grayscale image. tmp Indicates the number of black pixels in the second grayscale image. Hist[i] is the number of pixels corresponding to the grayscale value indicated by the histogram. i indicates the grayscale value. dark bin _num is the grayscale value of the black pixel. For grayscale values ​​ranging from 0 to 255, dark bin_num can be set to 10 based on experience, that is, all pixels with grayscale values ​​not greater than 10 are considered black pixels. ratio is the ratio of black pixels, and MINQ is used to output the minimum value, that is, When it is less than 1, exist When it is not less than 1, ratio=1.

[0191] After the ratio is calculated, when the ratio is less than 1, it can be determined that the first image is not in a dark light scene, and when the ratio is not less than 1, it can be determined that the first image is in a dark light scene. For the first image in a dark light scene, a second enhancement curve in a dark light scene can also be obtained. The determination process of the second enhancement curve is similar to that of the first enhancement curve, and can be determined using the contrast parameter clip_ratio_dark preset under dark conditions. For a detailed description, please refer to the above embodiment, which will not be repeated here. Afterwards, the first enhancement curve is adjusted using Formula 19.

[0192] C2=C dark ×ratio+(1-ratio)×C1 (Formula 19)

[0193] In formula 19, C2 is the adjusted enhancement curve, C dark is the second enhancement curve, and C1 is the first enhancement curve. In Formula 19, ratio is also referred to in the process of adjusting the first enhancement curve. This is because there will be a certain number of black pixels in the image under normal scenes. Therefore, in the process of adjusting the first enhancement curve, ratio is also used for protection to avoid misjudging the normal scene as a dark light scene, which will lead to incorrect adjustment of the first enhancement curve.

[0194] Adjustment process 2: using a smoothing coefficient to adjust the first enhancement curve, where the smoothing coefficient is used to reduce abrupt changes between the first image and adjacent frame images of the first image.

[0195] For example, for adjacent frame images in a video, an enhancement curve is obtained by cropping with the same contrast parameter. In this case, even if there may be large differences in the histograms of adjacent frame images, the enhancement curve formed after cropping can basically ensure that the brightness of the previous and next frames will not change too much, so as to avoid the problem of video flickering and insufficient contrast. However, a smoothing coefficient smooth can be added on this basis to smooth the enhancement curve of adjacent frame images. By smoothing the enhancement curve, the brightness and contrast of adjacent frame images adjusted respectively by the enhancement curve can be smoothed, thereby further avoiding the sudden change of the brightness and contrast of adjacent frame images. The process of smoothing the enhancement curve using the smoothing coefficient can be seen in Formula 20.

[0196] Cs =smooth r ×C x +(1-smooth r )×pre_C x (Formula 20)

[0197] Among them, C s is the smoothed enhancement curve, smooth r is the smoothing coefficient, C x is the enhancement curve of the second grayscale image, pre_C x is the enhancement curve of the grayscale image of the adjacent frame image of the first image. In addition, in addition to smoothing the enhancement curve of the second grayscale image using the smoothing coefficient, the processing device can also use the smoothing coefficient to smooth the enhancement curve of the grayscale image of the previous frame image, that is, the smoothed enhancement curve is used as the enhancement curve of the fourth grayscale image of the third image. The above process is, for example, pre_C x =C s Through simple temporal smoothing, sudden changes in brightness and contrast can be further avoided.

[0198] Optionally, only the first adjustment process may be performed, and C2 may be used as the contrast enhancement curve of the second grayscale image. Alternatively, only the second adjustment process may be performed, that is, C in Formula 20 is x The first enhancement curve C1 is the smoothed enhancement curve C s As the contrast enhancement curve of the second grayscale image. You can also choose to perform adjustment process one and adjustment process two in combination, that is, C in formula 20 x The processing device can convert C calculated using Formula 20 into C2 calculated using Formula 19. s as a contrast enhancement curve.

[0199] In addition, the contrast enhancement curve calculated by the determination process 2 is temporarily taken as an example to illustrate the process of adjusting the contrast enhancement curve by the adjustment process 1 and the adjustment process 2. However, in actual applications, the contrast enhancement curve calculated by the determination process 1 can also perform the adjustment process 1 and the adjustment process 2. For example, if the contrast enhancement curve of the fourth grayscale image does not perform the above adjustment process, the processing device can adjust the contrast enhancement curve of the fourth grayscale image and use the adjusted contrast enhancement curve as the contrast enhancement curve of the second grayscale image and the fourth grayscale image.

[0200] Regardless of the method used to calculate the contrast enhancement curve of the second grayscale image, the second grayscale image can be globally contrast enhanced based on the contrast enhancement curve to obtain a third grayscale image. Figure 7 , Figure 7(1) is the histogram of the second grayscale image, the horizontal axis indicates the number of pixels, and the vertical axis indicates the grayscale value (in this case, the grayscale value is represented by a value of type unit8 (unsigned 8-bit integer type)). Figure 7 (2) is a contrast enhancement curve determined based on the histogram, the horizontal axis indicates the grayscale value before mapping, and the vertical axis indicates the grayscale value after mapping. Figure 7 (3) is a histogram of a third grayscale image obtained by enhancing the second grayscale image based on the contrast enhancement curve, wherein the horizontal axis indicates the grayscale value and the vertical axis indicates the number of pixels. The above process of enhancing the second grayscale image by the contrast enhancement curve can be called contrast limited adaptive histogram equalization (CLAHE).

[0201] After the third grayscale image is determined, the color of the first image may be used to restore the third grayscale image to obtain the second image. For example, formula 21 may be used to restore the color.

[0202]

[0203] Among them, T e is the third grayscale image, c is the color channel of the image, r is the color correction coefficient, r∈(0,1), the larger the value, the heavier the color correction, and vice versa, the closer it is to the grayscale image, src c is the first image, I is the grayscale value of the first image, indicating the brightness of the first image, and may be a brightness value calculated using Formula 1.

[0204] By using Formula 21, the third grayscale image is color restored to obtain a second image that is a color image. The brightness display of the second image is closer to that of the human eye, which effectively improves the human-computer interaction experience. In addition, the first image adjusted in Formula 21 is a non-grayscale image. If the first image is a grayscale image, the process of color restoration of the third grayscale image based on the color of the first image can be understood as directly using the third grayscale image obtained by local and global enhancement as the second image to be displayed based on the fact that the first image is a grayscale image.

[0205] In one possible case, the processing device may display the second image after processing the second image. If the processing device has a display function, for example, the processing device includes a display screen, the processing device may display the second image through the display screen. If the processing device does not have a display function, the processing device may send the second image to a display device, and the display device may display the second image. The display device may be, for example, a terminal that establishes a communication connection with the processing device.

[0206] In summary, the image processing method provided by the embodiment of the present application, in the process of enhancing the local contrast, will refer to the brightness range of the first grayscale image to achieve accurate enhancement of the local contrast for the brightness range, so that the local hue of the second image obtained by color adjustment should be better and more accurate. Secondly, an effective global contrast enhancement method is proposed, which can adaptively generate a global contrast enhancement curve based on the global statistical information of the image, that is, the distribution information, to enhance the global contrast of the image. In addition, the present application can also correct the adaptively generated global contrast enhancement curve based on the distribution information to enhance the dark light scene.

[0207] See also Figure 8 , an embodiment of the present application provides an image processing chip, the chip includes an image processor 901. The image processor 901 is, for example, an ISP (Image Signal Processor), which is a hardware or processing unit for processing image signals. Optionally, the image processor 901 is used to obtain a first grayscale image based on a first image, and the first grayscale image indicates the brightness of the first image.

[0208] In one possible implementation, see Fig. 9 The image processing chip is connected to the image sensor, and the image processing chip can receive the first image captured by the image sensor. Alternatively, the image processing chip includes a memory, and the memory stores the image, and the image processor 901 can read the image stored in the memory as the first image to be processed.

[0209] After acquiring the first image, the image processing chip can use the image processor 901 to process the first image to obtain a first grayscale image. For the process of the image processor 901 processing the first image to obtain the first grayscale image, refer to the above Figure 1 The process of the processing device processing the first image to obtain the first grayscale image in the illustrated embodiment will not be repeated here.

[0210] In a possible case, the image processor 901 also enhances the local contrast of the first grayscale image according to the brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates the brightness difference of the grayscale blocks in the first grayscale image. For example, the base layer pixel value and detail layer pixel value corresponding to any grayscale block are obtained, the base layer pixel value is the base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value is the detail layer data of the grayscale value of the central pixel point of any grayscale block; the base layer pixel value is adjusted according to the brightness range to obtain the first pixel value; the second pixel value corresponding to the central pixel point of any grayscale block is determined according to the first pixel value and the detail layer pixel value, and the second grayscale image is determined based on the second pixel values ​​of each grayscale block.

[0211] For the process of obtaining base layer pixel values ​​and detail layer pixel values, please refer to the above Figure 1 The process of obtaining the base layer pixel value and the detail layer pixel value in step 102 of the illustrated embodiment will not be repeated here. The embodiment of the present application does not limit the brightness range referenced by the image processor 901 in the process of adjusting the base layer pixel value, including at least one of a local brightness range or a global brightness range, the local brightness range includes the maximum value and the maximum value of the grayscale values ​​of each pixel point in any grayscale block, and the global brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel point in the first grayscale image. Optionally, the global brightness range of the first grayscale image is the same as the global brightness range of the adjacent grayscale image, and the adjacent grayscale image refers to the grayscale image of the adjacent frame image of the first image.

[0212] The maximum value and the minimum value of any grayscale block may be referred to as the local maximum value and the local minimum value. The process of the image processor 901 adjusting the base pixel value to obtain the first pixel value by using the local maximum value and the local minimum value may be referred to in the above description. Figure 1 The maximum value and the minimum value in the first grayscale image can be called the global maximum value and the global minimum value. The process of the image processor 901 adjusting the base pixel value to obtain the first pixel value by using the global maximum value and the global minimum value can be referred to in the above Figure 1 The following is a description of the second adjustment method in step 102 of the embodiment shown.

[0213] Exemplarily, the process in which the image processor 901 adjusts the base pixel value to obtain the first pixel value by using the local maximum value, the local minimum value, the global maximum value and the global minimum value is the same as the above Figure 1 The process of comprehensively executing adjustment method 1 and adjustment method 2 to adjust the base pixel value to obtain the first pixel value in step 102 of the illustrated embodiment is similar, and reference may be made to the relevant description, which will not be repeated here.

[0214] Regardless of the manner in which the image processor 901 adjusts the base layer pixel value to obtain the first pixel value, the second pixel value may be determined based on the first pixel value and the detail layer pixel value to obtain a second grayscale image including the second pixel value. For example, the image processor 901 may increase the detail layer pixel value, decrease the first pixel value, and determine the second pixel value based on the increased detail layer pixel value and the decreased first pixel value. For detailed description, see the above Figure 1 The process of determining the second pixel value in step 102 of the illustrated embodiment will not be described in detail here.

[0215] In a possible implementation, after acquiring the second grayscale image, the image processor 901 may perform color restoration on the second grayscale image based on the color of the first image to obtain the second image. For example, the number of pixels of each grayscale value is determined according to the grayscale value of each pixel in the second grayscale image to obtain the distribution information of the second grayscale image; the contrast enhancement curve of the second grayscale image is determined according to the distribution information; the global contrast of the second grayscale image is enhanced according to the contrast enhancement curve to obtain the third grayscale image, and the global contrast indicates the brightness difference in the second grayscale image; the third grayscale image is color restored based on the color of the first image to obtain the second image.

[0216] Optionally, the image processor 901 determines the distribution information of the second grayscale image and the above Figure 1 The process of determining the distribution information of the second grayscale image in step 103 is similar to that in the above-mentioned step, and the image processor 901 determines the contrast enhancement curve of the second grayscale image according to the distribution information. Figure 1 The process of determining the distribution information of the second grayscale image in step 103 is similar, including but not limited to the following two determination methods.

[0217] Determination method 1: obtain distribution information of a fourth grayscale image, the fourth grayscale image indicates the brightness of the third image, and the third image is a previous frame image of the first image; determine whether a scene change occurs in the first image relative to the third image based on the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and determine the contrast enhancement curve of the second grayscale image based on the judgment result.

[0218] Determination method two: equalize the number of pixel points of each grayscale value based on a quantity threshold, and the difference in the number of pixel points of each grayscale value after equalization is smaller than the difference in the number of pixel points of each grayscale value in the distribution information; determine the first enhancement curve of the second grayscale image according to the number of each pixel point after equalization, and determine the contrast enhancement curve based on the first enhancement curve.

[0219] Optionally, determine the method 1 and the above Figure 1 The description of the determination process 1 in step 103 is similar to that of the determination process 1. Figure 1 The description of the second determination process in step 103 is similar, and reference may be made to the relevant description, which will not be repeated here.

[0220] In a possible implementation, after acquiring the first enhancement curve, the image processor 901 may directly use the first enhancement curve as the contrast enhancement curve of the second grayscale image, or may adjust the first enhancement curve and use the adjusted first enhancement curve as the contrast enhancement curve. In this case, the image processor 901 may perform at least one of the following adjustments: when the first image is a dark light scene, the first enhancement curve is adjusted using the second enhancement curve, and the second enhancement curve is used to enhance the global contrast of the image in the dark light scene; or, the first enhancement curve is adjusted using a smoothing coefficient, and the smoothing coefficient is used to reduce the mutation between the first image and the adjacent frame images of the first image.

[0221] Exemplarily, the process of the image processor 901 adjusting the first enhancement curve is similar to the above Figure 1 The adjustment process 1 and the adjustment process 2 in step 103 are similar, and reference may be made to the relevant description, which will not be repeated here.

[0222] Regardless of the method by which the image processor 901 obtains the contrast enhancement curve, the contrast enhancement curve can be used to adjust the second grayscale image to obtain a third grayscale image, and then the third grayscale image is restored to color to obtain a second image that is closer to human eye observation. Figure 1 In step 103 of the embodiment shown in the figure, the process of adjusting the second grayscale image is similar, and the process of color restoration of the third grayscale image by the image processor 901 is similar to the above-mentioned process. Figure 1 In step 103 of the illustrated embodiment, the process of color restoration of the third grayscale image is similar, and reference may be made to the relevant description, which will not be repeated here.

[0223] In the process of enhancing the local contrast, the above chip will refer to the brightness range of the first grayscale image to accurately enhance the local contrast for the brightness range. The local hue of the second image obtained by color adjustment should have better effect and higher accuracy.

[0224] In an exemplary embodiment, an electronic device is also provided. The electronic device is configured Figure 8 or Fig. 9The image processing chip shown, the electronic device can implement the image processing methods provided by the above-mentioned various method embodiments. The electronic device is, for example, a terminal or a server. Exemplarily, the terminal can be any electronic product that can interact with the user through one or more methods such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction or a handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a laptop computer, a desktop computer, a smart car machine, a smart TV, a smart speaker or a player, etc. The terminal may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and other names. The electronic device can also be a server, which can be a server or a server cluster composed of multiple servers, or the electronic device can be any other device that can implement the image processing method described above.

[0225] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by an image processor of a computer so that the computer implements any one of the above-mentioned image processing methods.

[0226] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0227] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. An image processor of a computer device reads the computer instructions from the computer-readable storage medium, and the image processor executes the computer instructions, so that the computer device performs any of the above-mentioned image processing methods.

[0228] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the first grayscale images involved in this application are all obtained with full authorization.

[0229] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0230] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing chip, characterized in that: The chip includes an image processor, and the image processor is configured to: Acquire a first grayscale image based on the first image, wherein the first grayscale image indicates brightness of the first image; enhancing a local contrast of the first grayscale image according to a brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates a brightness difference of the grayscale blocks in the first grayscale image; The second grayscale image is color restored based on the color of the first image to obtain a second image.

2. The chip according to claim 1, characterized in that: When the image processor enhances the local contrast of the first grayscale image according to the brightness range of the first grayscale image to obtain the second grayscale image, the image processor is configured to: Obtain a base layer pixel value and a detail layer pixel value corresponding to any grayscale block, wherein the base layer pixel value is the base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value is the detail layer data of the grayscale value of the central pixel point of any grayscale block; According to the brightness range, adjusting the base pixel value to obtain a first pixel value; The second pixel value corresponding to the central pixel point of any grayscale block is determined according to the first pixel value and the detail layer pixel value, and the second grayscale image is determined based on the second pixel value corresponding to the central pixel point of each grayscale block.

3. The chip according to claim 2, characterized in that: When the image processor determines the second pixel value corresponding to the central pixel of any grayscale block according to the first pixel value and the detail layer pixel value, the image processor is configured as follows: The detail layer pixel value is increased, the first pixel value is decreased, and the second pixel value is determined based on the increased detail layer pixel value and the decreased first pixel value.

4. The chip according to any one of claims 1 to 3, characterized in that: The brightness range includes at least one of a local brightness range and a global brightness range, the local brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel in any grayscale block, and the global brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel in the first grayscale image.

5. The chip according to claim 4, characterized in that: The global brightness range of the first grayscale image is the same as the global brightness range of adjacent grayscale images, and the adjacent grayscale images are grayscale images corresponding to adjacent frame images of the first image.

6. The chip according to any one of claims 1 to 3, characterized in that: When the image processor performs color restoration on the second grayscale image based on the color of the first image to obtain the second image, the image processor is configured to: Determine the number of pixels of each gray value according to the gray value of each pixel in the second gray image, and obtain distribution information of the second gray image; determining a contrast enhancement curve of the second grayscale image according to the distribution information; enhancing the global contrast of the second grayscale image according to the contrast enhancement curve to obtain a third grayscale image, wherein the global contrast indicates a brightness difference in the second grayscale image; The third grayscale image is color restored based on the color of the first image to obtain the second image.

7. The chip according to claim 6, characterized in that: When determining the contrast enhancement curve of the second grayscale image according to the distribution information, the image processor is configured to: Acquire distribution information of a fourth grayscale image, where the fourth grayscale image indicates brightness of a third image, where the third image is a previous frame image of the first image; It is determined whether a scene change occurs in the first image relative to the third image according to the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and a contrast enhancement curve of the second grayscale image is determined according to the determination result.

8. An image processing method, characterized in that: The method comprises: Acquire a first grayscale image based on the first image, wherein the first grayscale image indicates brightness of the first image; enhancing a local contrast of the first grayscale image according to a brightness range of the first grayscale image to obtain a second grayscale image, wherein the first grayscale image includes a plurality of grayscale blocks, and the local contrast indicates a brightness difference of the grayscale blocks in the first grayscale image; The second grayscale image is color restored based on the color of the first image to obtain a second image.

9. The method according to claim 8, characterized in that The step of enhancing the local contrast of the first grayscale image according to the brightness range of the first grayscale image to obtain a second grayscale image includes: Obtain a base layer pixel value and a detail layer pixel value corresponding to any grayscale block, wherein the base layer pixel value is the base layer data of the grayscale value of the central pixel point of any grayscale block, and the detail layer pixel value is the detail layer data of the grayscale value of the central pixel point of any grayscale block; According to the brightness range, adjusting the base pixel value to obtain a first pixel value; The second pixel value corresponding to the central pixel point of any grayscale block is determined according to the first pixel value and the detail layer pixel value, and the second grayscale image is determined based on the second pixel value corresponding to the central pixel point of each grayscale block.

10. The method according to claim 9, characterized in that The determining, according to the first pixel value and the detail layer pixel value, a second pixel value corresponding to the central pixel point of any grayscale block comprises: The detail layer pixel value is increased, the first pixel value is decreased, and the second pixel value is determined based on the increased detail layer pixel value and the decreased first pixel value.

11. The method according to any one of claims 8 to 10, characterized in that: The brightness range includes at least one of a local brightness range and a global brightness range, the local brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel in any grayscale block, and the global brightness range includes the maximum value and the minimum value of the grayscale values ​​of each pixel in the first grayscale image.

12. The method according to claim 11, characterized in that The global brightness range of the first grayscale image is the same as the global brightness range of adjacent grayscale images, and the adjacent grayscale images are grayscale images corresponding to adjacent frame images of the first image.

13. The method according to any one of claims 8 to 10, characterized in that: The step of performing color restoration on the second grayscale image based on the color of the first image to obtain a second image includes: Determine the number of pixels of each gray value according to the gray value of each pixel in the second gray image, and obtain distribution information of the second gray image; determining a contrast enhancement curve of the second grayscale image according to the distribution information; enhancing the global contrast of the second grayscale image according to the contrast enhancement curve to obtain a third grayscale image, wherein the global contrast indicates a brightness difference in the second grayscale image; The third grayscale image is color restored based on the color of the first image to obtain the second image.

14. The method according to claim 13, characterized in that The step of determining a contrast enhancement curve of the second grayscale image according to the distribution information comprises: Acquire distribution information of a fourth grayscale image, where the fourth grayscale image indicates brightness of a third image, where the third image is a previous frame image of the first image; It is determined whether a scene change occurs in the first image relative to the third image according to the distribution information of the second grayscale image and the distribution information of the fourth grayscale image, and a contrast enhancement curve of the second grayscale image is determined according to the determination result.

15. An electronic device, characterized in that: The electronic device comprises the image processing chip as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the image processor so that the computer implements the image processing method according to any one of claims 8 to 14.

Images