Image enhancement methods, apparatuses, field-programmable gate arrays, and devices

Abstract

The present disclosure relates to an image enhancement method, apparatus, field programmable logic gate array and device. Obtain a brightness domain representation of an image, the brightness domain representation is used to characterize the pixel value distribution of the image in the brightness domain; calculate a stretching coefficient based on the brightness domain representation; and update the pixel value of the image in the brightness domain based on the stretching coefficient. Thus, the overall brightness of the image can be adjusted without affecting the color of the image (i.e., without distortion) to achieve contrast enhancement, and the amount of calculation can be reduced to a certain extent, which is hardware-friendly.

Description

Technical Field

[0001] The present disclosure relates to the field of image enhancement technology, and in particular to an image enhancement method, device, field programmable logic gate array and equipment suitable for enhancing images with low contrast and inappropriate exposure. Background Art

[0002] Image enhancement technology is applicable to scenes that require image quality enhancement and the preprocessing part of various image processing and analysis modules. This technology improves image quality and adapts to special needs by emphasizing or changing local or overall features of an image. Therefore, image enhancement technology has developed and progressed rapidly in recent years.

[0003] Histogram equalization is a commonly used technique in image enhancement. The histogram of an image is a graphical representation of the intensity and probability of occurrence of intensity values ​​in a given image. Image histogram equalization is a method of equalizing the probability distribution of intensity values ​​in an image. Due to the limitations of the camera device or the characteristics of the scene, the captured image pixels will be concentrated in certain parts of the pixel value range, resulting in a dim image with no sense of hierarchy and unable to truly reproduce the scene information, that is, the image contrast is low. The histogram of an image describes the distribution of the pixel values ​​of the image in the value range and shows the characteristics of the distribution of the pixel values ​​of the image. The problem of low image contrast mentioned above can be expressed by constructing a histogram of the image pixel values. At the same time, adjusting the histogram of the image can change the contrast and other related features of the image.

[0004] The current color image enhancement methods using histogram equalization are mainly the following:

[0005] 1. Generate a mapping relationship based on the cumulative histogram of image color values ​​to enhance image contrast. In order to protect the overall brightness or other purposes, this method needs to divide the sub-regions on the cumulative histogram and calculate the cumulative density function separately. There are many hyperparameters in the process of dividing the sub-regions, which is difficult to manually select and difficult to adapt to different scenes.

[0006] 2. Use a local adaptive scheme to generate digital images showing improved exposure and tone uniformity by adopting histogram equalization on a local basis, that is, the stretching and compression of pixel brightness levels adapt to the local distribution of pixels in the image. This method has a good adjustment effect on local areas, but it is easy to produce discontinuous edges between different areas, and because it emphasizes local effects, the overall effect is not good.

[0007] 3. Performing histogram equalization on RGB channels in the RGB domain color space will change the hue, so a hue preservation module needs to be added, such as using wavelet transform to preserve the wavelet detail coefficients of the image and reconstructing the result after histogram equalization. This type of method has high computational complexity and introduces the problem of hue distortion.

[0008] It can be seen that the existing histogram equalization implementation scheme has problems of image distortion or excessive calculation, so there is a need for an image enhancement method that has better effect, is hardware-friendly and can realize real-time processing. Summary of the invention

[0009] A technical problem to be solved by the present disclosure is to provide an image enhancement method with good effect, adaptability to different scenes, hardware-friendly and capable of achieving real-time processing.

[0010] According to a first aspect of the present disclosure, there is provided an image enhancement method, comprising: obtaining a brightness domain representation of an image, the brightness domain representation being used to characterize the pixel value distribution of the image in the brightness domain; calculating a stretching coefficient based on the brightness domain representation; and updating the pixel values ​​of the image in the brightness domain based on the stretching coefficient.

[0011] Optionally, the step of calculating the stretching coefficient based on the brightness domain representation includes: calculating the brightness domain pixel mean based on the brightness domain representation; calculating the stretching coefficient based on the brightness domain pixel mean and the brightness expected value, wherein the size of the stretching coefficient is negatively correlated with the brightness domain pixel mean and positively correlated with the brightness expected value.

[0012] Optionally, the stretch factor k is calculated based on the following formula,

[0013] or

[0014]

[0015] Among them, b is the mean value of pixels in the brightness domain, y is the expected brightness value, and p(x) represents the proportion of pixels with a pixel value of x in the image.

[0016] Optionally, the method further includes: limiting the value of the stretching coefficient to a first numerical range.

[0017] Optionally, the method further includes: calculating a first numerical interval based on a value range restriction parameter and a brightness domain pixel mean.

[0018] Optionally, the first numerical interval is [(1-μ)*b, μ+(1-μ)*b], where μ is a value range limitation parameter and b is a brightness domain pixel mean.

[0019] Optionally, the step of updating the pixel value of the image in the brightness domain based on the stretching coefficient includes: adjusting the pixel value based on the pixel value, the brightness domain pixel mean and the stretching coefficient.

[0020] Optionally, the pixel value x is adjusted based on the following formula i ,

[0021]

[0022] Among them, y i is the adjusted pixel value, k is the stretching coefficient, b is the mean pixel value in the brightness domain, x min is the minimum pixel value of the image in the brightness domain, x max is the maximum pixel value of the image in the brightness domain.

[0023] Optionally, the method further includes: quantizing operations involved in the execution of the method into fixed-point operations, and executing the method by a field programmable logic gate array.

[0024] Optionally, the field programmable gate array uses a multi-port BRAM memory, and the field programmable gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are assigned to different sub-depths. The field programmable gate array uses the pixel value of the read image in the brightness domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes it back to the original address. After reading all pixels, the field programmable gate array adds the values ​​stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of pixel values ​​to obtain the brightness domain representation.

[0025] According to a second aspect of the present disclosure, an image enhancement device is also provided, including: an acquisition module for acquiring a brightness domain representation of an image, wherein the brightness domain representation is used to characterize the pixel value distribution of the image in the brightness domain; a calculation module for calculating a stretching coefficient based on the pixel value distribution; and an update module for updating the pixel values ​​of the image in the brightness domain based on the stretching coefficient.

[0026] According to a third aspect of the present disclosure, a field programmable logic gate array is also provided for executing the method described in the first aspect above, wherein the operations involved in the method are fixed-point operations obtained through quantization.

[0027] Optionally, the field programmable gate array uses a multi-port BRAM memory, and the field programmable gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are assigned to different sub-depths. The field programmable gate array uses the pixel value of the read image in the brightness domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes it back to the original address. After reading all pixels, the field programmable gate array adds the values ​​stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of pixel values ​​to obtain the brightness domain representation.

[0028] According to a fourth aspect of the present disclosure, an imaging device is also provided, comprising: an imaging device and a field programmable logic gate array, wherein the field programmable logic gate array is used to execute the method described in the first aspect above to enhance the image captured by the imaging device, wherein the operations involved in the method are fixed-point operations obtained by quantization.

[0029] According to a fifth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the method described in the first aspect above.

[0030] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising an executable code, which, when executed by a processor of an electronic device, causes the processor to execute the method described in the first aspect above.

[0031] According to a seventh aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0032] Therefore, the present invention obtains the brightness domain representation of the image and uses the stretching coefficient calculated based on the brightness domain representation to adjust the pixel value of the image in the brightness domain, so that contrast enhancement can be achieved without affecting the image color (i.e., there will be no distortion), and the amount of calculation can be reduced to a certain extent, which is hardware-friendly. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0034] Figure 1A schematic flowchart of an image enhancement method according to an embodiment of the present disclosure is shown.

[0035] Figure 2 A visualization of the pixel value adjustment formula is shown.

[0036] Figure 3 A schematic diagram of the overall processing flow of the hardware is shown.

[0037] Figure 4 A schematic diagram of data stream transmission and BRAM instantiation is shown.

[0038] Figure 5A An overall darker image is shown.

[0039] Figure 5B In order to utilize this disclosure Figure 5A The image shown is the image after contrast enhancement.

[0040] Figure 6 A schematic structural diagram of an image enhancement device according to an embodiment of the present disclosure is shown.

[0041] Figure 7 A schematic structural diagram of an imaging device according to an embodiment of the present disclosure is shown.

[0042] Figure 8 A schematic diagram of the structure of a computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] In response to the problems of low image contrast and inappropriate exposure (the image is relatively dark or too bright), the present disclosure proposes that the components of the image in the brightness domain (i.e., the brightness component or the brightness domain component) can be obtained, and the brightness component can be adjusted (stretched) while considering the brightness level of the image. This is to adjust the overall brightness of the image without affecting the image color (i.e., without distortion), thereby achieving contrast enhancement and reducing the amount of calculation to a certain extent, which is hardware-friendly.

[0045] Figure 1 A schematic flowchart of an image enhancement method according to an embodiment of the present disclosure is shown.

[0046] See also Figure 1, in step S110, a brightness domain representation of the image is obtained.

[0047] The brightness domain representation is used to characterize the pixel value distribution of the image in the brightness domain (i.e., the brightness channel). Pixel value distribution, that is, the distribution of pixel values, can be the number of pixels for each pixel value, or the pixel ratio for each pixel value.

[0048] The pixel value of an image in the brightness domain reflects the brightness value of the image pixel.

[0049] The brightness domain representation may be a component of an image in the brightness domain, ie, a luminance component or a brightness domain component of an image.

[0050] The brightness domain representation may also be a statistical result obtained by performing statistics on the brightness component of the image.

[0051] For example, the brightness domain representation may be a histogram obtained by statistically analyzing the brightness component of the image. The histogram marks the number of each pixel value in the brightness domain of the image, or the proportion of each pixel value in the total number of pixels.

[0052] The image refers to an image to be enhanced, and specifically may be an image with low contrast and improper exposure, such as a relatively dark or overly bright image.

[0053] The image can be an RGB image or an image in other color spaces (such as HSV, YUV, YCbCr). There is no brightness component in the RGB color space, so if the image is in a color space such as RGB without a brightness component, the image needs to be converted to another color space with a brightness component to obtain the brightness domain representation of the image.

[0054] For example, HSV (hue, saturation, value) is a color space created by AR Smith in 1978 based on the intuitive characteristics of color, also known as the Hexcone Model, in which the V channel is the brightness channel, so the RGB image can be converted to the HSV color space.

[0055] For another example, in the YUV color space, "Y" represents brightness (Luminance or Luma), which is the grayscale value; while "U" and "V" represent chrominance (Chrominance or Chroma), where the Y channel is the brightness channel, and "brightness" is obtained by linearly transforming the RGB input signal, so the RGB image can also be converted to the YUV color space.

[0056] In step S120 , a stretch factor is calculated based on the luminance domain representation.

[0057] The stretch factor is used to enhance the contrast of the image to solve the problems of low contrast and inappropriate exposure. The main reason for the low contrast of the image is that the image pixels are concentrated in certain parts of the pixel value range, which makes the image dark or too bright, without a sense of hierarchy, and unable to truly reproduce the scene information, that is, the image contrast is low.

[0058] The "stretching" in the stretching coefficient mentioned in the present disclosure refers to stretching the originally concentrated pixel values ​​of the image in the brightness domain to a wider value range of the pixel value domain (such as 0 to 255, or 0 to 1 after normalization), thereby improving the contrast of the image by balancing the pixel value distribution.

[0059] In other words, the stretch factor is used to adjust the pixel value distribution of the image in the brightness domain. The "stretch" in the stretch factor is a stretch in a broad sense, which means stretching the pixel value distribution represented by the brightness domain representation of the image to enhance the contrast, rather than limiting the stretching of each pixel value. In other words, when the stretch factor is used to adjust a specific pixel value, it can include not only increasing the pixel value (stretching) but also decreasing the pixel value (scaling).

[0060] When contrast enhancement is performed on an image based on a stretching factor, the original brightness of the image needs to be considered and protected to a certain extent to avoid brightness distortion. To this end, the present disclosure proposes that the stretching factor can be calculated based on the brightness domain representation, so that the value of the stretching factor is highly correlated with the image brightness level, thereby protecting the brightness.

[0061] As an example, the brightness domain pixel mean can be calculated based on the brightness domain representation, and the stretching coefficient can be calculated based on the brightness domain pixel mean and the brightness expected value. The magnitude of the stretching coefficient is negatively correlated with the brightness domain pixel mean and positively correlated with the brightness expected value. The brightness domain pixel mean is used to characterize the average value of all pixel values ​​of the image in the brightness domain. The brightness expected value is used to characterize the expected brightness mean after adjusting the pixel values ​​of the image in the brightness domain. The stretching coefficient can be used to make the brightness domain pixel mean of the updated image closer to the brightness expected value than before the update.

[0062] The expected brightness value is generally set to 0.5 (the result after numerical normalization), which is half of the image pixel value range. From a visual perspective, if the mean value of the brightness domain pixels is close to this value, the overall visual effect of the image is good. The expected brightness value generally does not change, that is, it does not need to be changed as the image changes. However, the expected brightness value can still be designed as an adjustable parameter. In practical applications, if you want the image to be darker or brighter, you can achieve this by adjusting the expected brightness value.

[0063] Optionally, the stretching coefficient may be calculated based on the difference between the brightness domain pixel mean and the brightness expected value. The stretching coefficient may be designed such that when the brightness domain pixel mean is lower than the brightness expected value, the value of the stretching coefficient is greater than the value of the stretching coefficient when the brightness domain pixel mean is higher than the brightness expected value, and the value of the stretching coefficient is negatively correlated with the brightness domain pixel mean.

[0064] For example, when the mean value of the brightness domain pixels is lower than the expected brightness value, the stretching coefficient is larger, and the lower the mean value of the brightness domain pixels is, the larger the stretching coefficient is; conversely, when the image pixel mean is higher than the expected value, the stretching coefficient is smaller, and the higher the mean value of the brightness domain pixels is, the smaller the stretching coefficient is.

[0065] For example, the stretch coefficient k can be calculated based on the following formula:

[0066]

[0067] b is the mean value of the brightness domain pixels, y is the expected brightness value, and p(x) represents the proportion of pixels with a pixel value of x in the image. The brightness domain pixel value in the formula is the normalized result, with the highest pixel value being 1 and the lowest pixel value being 0.

[0068] (1-b)y represents the product of the difference between the highest pixel value and the mean value of the pixels in the brightness domain and the expected value of the brightness; Represents the pixel average value of all pixels whose pixel values ​​are greater than the pixel mean value in the brightness domain; It represents the product of the percentage of pixels greater than the mean value of pixels in the brightness domain and the mean value of pixels in the brightness domain; Represents the pixel average value of all pixels whose pixel values ​​are less than the pixel mean value in the brightness domain; Represents the mean value of pixels in the brightness domain.

[0069] For ease of understanding, the calculation formula (1) for the stretch coefficient k can be simplified to

[0070]

[0071] in -delta is the integral of (xb)p(x) between b and 1, that is, the average of the pixel values ​​of all pixels above the mean value of the brightness domain pixels.

[0072] The brightness domain pixel mean b and the brightness expected value y in the above formula are both floating point numbers between 0 and 1, which are dimensionless units. Therefore, the stretching coefficient k finally calculated is a dimensionless unit.

[0073] It can be seen from formula (2) that the size of the stretch coefficient k is calculated based on the ratio of the brightness domain pixel mean b to the brightness expected value y, and the brightness domain pixel mean b. When the brightness domain pixel mean b is lower than the brightness expected value y, the k value is larger, and the lower the brightness domain pixel mean b, the larger the k value; conversely, when the brightness domain pixel mean b is higher than the brightness expected value y, the k value is smaller, and the higher the brightness domain pixel mean b, the smaller the k value; when the brightness domain pixel mean b is equal to the brightness expected value y, the k value is equal to the brightness expected value y.

[0074] The calculation principle of the stretch factor k can be summarized as follows: the stretch factor k is calculated based on the difference between the image brightness level and the expected brightness level (i.e., the expected brightness value), and the stretch factor k is highly correlated with the image brightness level, playing a role in protecting the brightness. In contrast, the current mainstream contrast enhancement algorithm does not consider the original image brightness level.

[0075] Considering that a stretch coefficient that is too large or too small will cause an image to change too obviously, resulting in partial overexposure or dimness, the present disclosure proposes that the value of the stretch coefficient can be limited to a first numerical range, which can be regarded as a reasonable value range. The first numerical range can be a pre-set range that is irrelevant to the image, or it can be a range determined based on the brightness domain representation of the image.

[0076] As an example, the first value interval can be calculated based on the value range restriction parameter and the brightness domain pixel mean. The value range restriction parameter is used to limit the value range of the stretch coefficient. In practical applications, the value of the value range restriction parameter can be adjusted according to the algorithm effect. For example, if the image processed by the algorithm appears too bright, the value of the value range restriction parameter can be appropriately reduced to suppress the stretching effect.

[0077] As mentioned above, the range limit parameter limits the range of the stretch coefficient. In actual testing, this value is 0.25. This value generally does not change. However, this value is still adjustable, and in some extreme cases, this value affects the actual stretching effect. If the scene is too dark or too bright, you can increase this value appropriately to enhance the stretching effect; conversely, if the output image is too dark or bright, you can reduce this value to weaken the stretching effect.

[0078] The first numerical interval can be [(1-μ)*b, μ+(1-μ)*b], where μ is the value range limit parameter and b is the pixel mean in the brightness domain. Thus, the limit range of k can be calculated based on the original input image mean b and the value range limit parameter μ and k can be updated.

[0079]

[0080] When (1-μ)*b<k<μ+(1-μ)*b, the value of k is equal to the value calculated according to the calculation formula (1) or (2) of the stretch coefficient k.

[0081] In step S130, the pixel values ​​of the image in the brightness domain are updated (ie, adjusted) based on the stretch factor.

[0082] Based on the stretching coefficient, at least part (e.g., all) of the pixel values ​​of the image in the brightness domain can be adjusted. For pixel values ​​of different values, a nonlinear adjustment method can be used to adjust the pixel values ​​of the image in the brightness domain to uniformly stretch the pixel values ​​in the pixel value domain. As an example, the pixel value can be adjusted based on the pixel value, the mean value of the pixel in the brightness domain, and the stretching coefficient. For example, the pixel value x can be adjusted based on the following formula: i

[0083]

[0084] Among them, y i is the adjusted pixel value, k is the stretching coefficient, b is the mean pixel value in the brightness domain, x min is the minimum pixel value of the image in the brightness domain, x max is the maximum pixel value of the image in the brightness domain. The brightness domain pixel value in the formula is the normalized result, the highest pixel value is 1, and the lowest pixel value is 0.

[0085] The function of the pixel value adjustment formula (4) is to uniformly stretch the image pixel value to the range of [0,1]. When the pixel value is less than the image pixel value mean, the formula compares the input pixel value with the mean b and distributes it to [0,k); otherwise, it is stretched to the range of [k,1]. Figure 2 The visualization of this formula is shown in Figure 2. Figure 2 As shown, the horizontal axis is the pixel value before stretching, and the vertical axis is the pixel value after stretching. The entire stretching process is nonlinear stretching, where k72_b11 indicates that the stretching coefficient k is 72 and the brightness domain pixel mean b is 11.

[0086] So far, combined with Figure 1 The image enhancement method disclosed in the present invention is exemplified.

[0087] The present disclosure also provides a hardware adaptation quantization solution based on the above method.

[0088] The present disclosure can quantize the operations involved in the execution of the method (such as floating-point operations) into fixed-point operations, and hand over the method to hardware for execution. For example, the method quantized into fixed-point operations can be handed over to a field programmable gate array (FPGA) for execution.

[0089] FPGA is a semi-customized circuit in integrated circuits, which consists of a series of configurable logic units (CLB, Configurable Logic Block) and programmable connections. It can be quickly compiled and burned into a combination of transistor circuits through hardware description language, so that the algorithm can be quickly deployed and accelerated on the hardware platform. It has the characteristics of extremely high computing energy consumption ratio, rapid iteration, customization, etc., and is particularly suitable for current artificial intelligence and data center related applications.

[0090] FPGA can be used to implement the above combination Figure 1 The method shown utilizes the abundant configurable resources such as DSP, BRAM, CLB, etc. inside the FPGA to process data streams in parallel, and can achieve real-time enhancement of video images of types such as 4K and FPS60.

[0091] When implementing the above method using FPGA, the various parameters involved in the method can be limited to non-negative integers within the pixel value range (such as 0 to 255), and the integration process can be changed to an equivalent multiplication and accumulation to quantize the floating-point operations in the method into fixed-point operations. The pixel value range refers to the pixel value range of the image to be enhanced.

[0092] Taking the method of enhancing an image using the above-mentioned stretching coefficient k calculation formula (1), stretching coefficient k value restriction formula (3) and pixel value adjustment formula (4) as an example, the hardware adaptation quantization scheme based on this method is as follows.

[0093] 1. Give the expected value y of the image pixel value after histogram equalization and the limit parameter μ of the expected value range.

[0094] 2. Count the number of pixel points corresponding to each pixel value to obtain the pixel value distribution histogram of the image, where the horizontal axis value range of the histogram is limited to 0 to 255. If the image pixel width is higher than 8 bits, the horizontal axis value range of the histogram can be increased. For example, if the input pixel value range is 0 to 4095, the horizontal axis value range of the histogram is also 0 to 4095;

[0095] 3. Calculate the pixel value mean b according to the pixel value of the input image, and take the high 8 bits b_;

[0096] 4. Calculate the cumulative distribution function using the cumulative multiplication method according to the following formula;

[0097]

[0098]

[0099]

[0100] 5. Substitute the result of the above cumulative distribution function into the following formula to calculate the histogram stretch coefficient k, and then convert the value into a fixed point after calculating the division;

[0101]

[0102] 6. Calculate the limit range of k according to the original input image mean b and the value range limit parameter μ and update k;

[0103]

[0104] 7. Calculate a new pixel value based on the input pixel value x, the stretch coefficient k and the image mean b;

[0105]

[0106] Figure 3 A schematic diagram of the overall processing flow of the hardware is shown.

[0107] like Figure 3 As shown in FIG. 1 , the hardware processing flow mainly includes three steps: histogram statistics, k value calculation, and adjusting the pixel value to obtain a new pixel value. Taking FPGA as an example, the specific implementation of the above three steps is as follows.

[0108] Step P1: Perform histogram statistics on image data frame by frame

[0109] Histogram statistics is to count the number of occurrences of each pixel value in the brightness domain of a frame of image.

[0110] FPGAs can use multi-port (eg, dual-port) BRAM memories.

[0111] FPGA can divide the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are assigned to different sub-depths to avoid the failure to update the BRAM in time when the same data appears continuously.

[0112] The FPGA uses the pixel value of the read image in the brightness domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes it back to the original address.

[0113] After reading all the pixels, the FPGA adds the values ​​stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of pixel values, so as to obtain the histogram statistical result.

[0114] The specific implementation steps of FPGA statistical histogram are as follows:

[0115] ① Take the high 8-bit data and get the pixel range of 0-255;

[0116] ② Instantiate a BRAM with a depth of 512, where addresses 0-255 are used to store the number of pixels in even columns, and addresses 256-511 are used to store the number of pixels in odd columns;

[0117] ③ When taking the pixel value of an even column, the obtained pixel value is used as the address to read the BRAM address 0-255; when taking the pixel value of an odd column, the obtained pixel value is shifted left by 1 bit and used as the address to read the BRAM address 256-511;

[0118] ④ Add 1 to the read data and write it back to the corresponding address;

[0119] ⑤ After storing a whole frame of data, read out the data of 0-255 and 256-511 in the BRAM at the same time, add the data of address 0 plus the data of address 256, and add them in sequence to get the number of each pixel value of 0-255;

[0120] ⑥Data stream transmission and BRAM instantiation such as Figure 4 shown.

[0121] FPGA uses the method of cross-storing odd and even columns into a true dual-port RAM with a depth of 512 and 256 addresses before and after, avoiding the use of multiplier cross-clock domain operations, and also reducing the use of BRAM blocks, saving resources.

[0122] like Figure 4 As shown, the number of columns of pixel values ​​in the image data can start from the 0th column, and the pixel value of the 0th column read is 0. At this time, the pixel value 0 read can be used as the address to read in the BRAM address 0-255, that is, read address 0, and then add 1 to the value in address 0 (the initial value is 0) and rewrite it to address 0. The pixel value of the 1st column is read as 18, which belongs to the odd column. The pixel value 18 read can be shifted left by one bit, and the value 273 obtained by adding 17 and 256 is used as the address to read in the BRAM address 256-511, that is, read address 273, and add 1 to the value in address 273 (the initial value is 0) and rewrite it to address 273. And so on, the traversal of all pixel values ​​in the image data is completed.

[0123] Step P2: Calculate the stretching coefficient k based on the histogram statistics.

[0124] The specific steps for FPGA to calculate the stretching coefficient k are as follows:

[0125] ① While the histogram is being counted, the input pixels are accumulated to get the actual pixel sum, which is divided by the number of pixels in each frame to get the pixel mean b, which is also the brightness domain pixel mean mentioned above.

[0126] ② Taking b as the boundary, the data at BRAM address 0-b is multiplied by 0-b and accumulated to get sum0; the data at BRAM address b-255 is accumulated to get sump; the data at BRAM address b-255 is multiplied by b-255 and accumulated to get sum1.

[0127] ③ According to formula (5), the divisor and dividend are obtained separately and sent to the divider to obtain the k value;

[0128] ④ According to formula (6), use the mean value b obtained in step ① in step P2 to calculate the limit range of k and update k;

[0129] Step P3: Calculate the new pixel value of each pixel in the current frame after stretching according to the formula

[0130] The specific steps for FPGA to calculate new pixel values ​​are as follows:

[0131] ① While counting the histogram, the maximum pixel xmax and the minimum pixel xmin are obtained by comparing the input pixel values ​​and stored;

[0132] ② According to formula (7), calculate the new pixel value.

[0133] Figure 5A It is an overall darker image.

[0134] against Figure 5A For the image shown, according to the above scheme, the stretching coefficient k is calculated to be 84, the image pixel mean b is 27, the image pixel value expected value y is set to 127, and the value range limit parameter μ is 64.

[0135] According to the above scheme Figure 5A The output result obtained by enhancing the image shown is as follows Figure 5B shown.

[0136] It can be seen that for images with low contrast and overall dimness, the present disclosure can be used to achieve contrast enhancement without causing image distortion. In addition, for images with low contrast and overall overbrightness, the present disclosure can also be used to achieve contrast enhancement without causing image distortion.

[0137] In summary, the present disclosure provides an image enhancement algorithm based on histogram equalization that works in the brightness domain, which can protect the image brightness while solving the problems of low contrast and inappropriate exposure (such as dark or overly bright images). In addition, the present disclosure has also improved and optimized the hardware features, so that the algorithm can be processed in real time on the hardware platform, especially in the application scenarios of high resolution and high frame rate, it can still ensure a good histogram equalization effect.

[0138] The image enhancement method disclosed in the present invention may also be implemented as an image enhancement device. Figure 6 FIG. 1 is a schematic diagram showing the structure of an image enhancement device according to an embodiment of the present disclosure. The functional units of the image enhancement device can be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present disclosure. It can be understood by those skilled in the art that Figure 6 The functional units described may be combined or divided into sub-units to implement the principles of the above invention. Therefore, the description herein may support any possible combination, division, or further limitation of the functional units described herein.

[0139] The following is a brief description of the functional units that the image enhancement device may have and the operations that each functional unit may perform. For the details involved, please refer to the relevant description above and will not be repeated here.

[0140] See also Figure 6 The image enhancement device 600 includes an acquisition module 610 , a calculation module 620 and an update module 630 .

[0141] The acquisition module 610 is used to acquire a brightness domain representation of an image, where the brightness domain representation is used to characterize the pixel value distribution of the image in the brightness domain.

[0142] The calculation module 620 is used to calculate the stretching coefficient based on the pixel value distribution.

[0143] As an example, the calculation module 620 may calculate the brightness domain pixel mean based on the brightness domain representation, and calculate the stretching coefficient based on the brightness domain pixel mean and the brightness expected value, wherein the magnitude of the stretching coefficient is negatively correlated with the brightness domain pixel mean and positively correlated with the brightness expected value. For example, the calculation module may calculate the stretching coefficient using the formula (1) mentioned above.

[0144] The image enhancement device 600 may further include a limiting module, which is used to limit the value of the stretching coefficient to a first numerical range.

[0145] The image enhancement device 600 may further include a second calculation module, which is used to calculate a first numerical interval based on the value range restriction parameter and the brightness domain pixel mean. For the first numerical interval, please refer to the above related description.

[0146] The updating module 630 is used to update the pixel values ​​of the image in the brightness domain based on the stretching coefficient.

[0147] The updating module 630 may adjust the pixel value based on the pixel value, the brightness domain pixel mean value and the stretching coefficient. For example, the updating module 630 may adjust the pixel value using the formula (3) mentioned above.

[0148] The present disclosure also provides a field programmable logic gate array, which can quantize the operations involved in the execution process of the above-mentioned method into fixed-point operations and execute the quantized method.

[0149] The present disclosure also provides an imaging device. Figure 7 FIG. 2 shows a schematic diagram of the structure of an imaging device according to an embodiment of the present disclosure. Figure 7 As shown, the imaging device 700 may include an imaging device 710 and a field programmable logic gate array 720. The imaging device 710 is used to take images. The field programmable logic gate array is used to perform the above combined Figure 1 The method shown is used to enhance the image captured by the imaging device, wherein the operation involved in the method is a fixed-point operation obtained by quantization. The specific execution process of the field programmable logic gate array can refer to the relevant description above.

[0150] Figure 8 A schematic diagram of the structure of a computing device that can be used to implement the above-mentioned image enhancement method according to an embodiment of the present disclosure is shown.

[0151] See also Figure 8 , the computing device 800 includes a memory 810 and a processor 820 .

[0152] The processor 820 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 820 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 820 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0153] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a disk or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0154] The memory 810 stores executable codes, and when the executable codes are processed by the processor 820 , the processor 820 can execute the image enhancement method mentioned above.

[0155] The image enhancement method, apparatus, field programmable logic gate array, and device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0156] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0157] Alternatively, the present disclosure may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present disclosure.

[0158] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0159] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image enhancement method, comprising: Acquire a brightness domain representation of an image, wherein the brightness domain representation is used to characterize pixel value distribution of the image in the brightness domain; calculating a stretch factor based on the luminance domain representation; as well as The pixel values ​​of the image in the brightness domain are updated based on the stretching coefficient to stretch the pixel values ​​of the image that are originally relatively concentrated in the brightness domain to a wider value range of the pixel value domain, wherein: The step of calculating the stretching coefficient based on the brightness domain representation includes: calculating the brightness domain pixel mean based on the brightness domain representation; calculating the stretching coefficient based on the brightness domain pixel mean and the brightness expected value, wherein the magnitude of the stretching coefficient is negatively correlated with the brightness domain pixel mean and positively correlated with the brightness expected value. The step of updating the pixel value of the image in the brightness domain based on the stretching coefficient includes: adjusting the pixel value based on the pixel value, the brightness domain pixel mean and the stretching coefficient.

2. The method according to claim 1, wherein: The stretch coefficient is calculated based on the following formula k , ,or ， in, b is the brightness domain pixel mean, y is the expected brightness value, p(x) Indicates the proportion of pixels with pixel value x in the image.

3. The method according to claim 1, further comprising: The value of the stretching coefficient is limited to a first numerical range.

4. The method according to claim 3, further comprising: The first numerical interval is calculated based on the value range restriction parameter and the brightness domain pixel mean.

5. The method according to claim 4, wherein: The first numerical interval is ,in, To limit the value range of the parameter, is the mean value of pixels in brightness domain.

6. The method according to claim 1, wherein: Adjust the pixel value based on the following formula , in, is the adjusted pixel value, k is the stretch coefficient, is the pixel mean in brightness domain, is the minimum pixel value of the image in the brightness domain, is the maximum pixel value of the image in the brightness domain.

7. The method according to any one of claims 1 to 6, further comprising: The operations involved in the execution of the method are quantized into fixed-point operations, and the method is executed by a field programmable logic gate array.

8. The method according to claim 7, wherein: The field programmable logic gate array uses a multi-port BRAM memory. The field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are allocated to different sub-depths, The field programmable logic gate array uses the pixel value of the read image in the brightness domain as an address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes it back to the original address, After reading all pixels, the field programmable logic gate array adds the values ​​stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of the pixel value to obtain the brightness domain representation.

9. An image enhancement device, comprising: An acquisition module, used for acquiring a brightness domain representation of an image, wherein the brightness domain representation is used for characterizing a pixel value distribution of the image in the brightness domain; A calculation module, used for calculating a stretching coefficient based on the pixel value distribution; as well as an updating module, configured to update the pixel values ​​of the image in the brightness domain based on the stretching coefficient, so as to stretch the pixel values ​​of the image that are originally relatively concentrated in the brightness domain to a wider value range of the pixel value domain, The calculation module calculates the brightness domain pixel mean based on the brightness domain representation, and calculates the stretching coefficient based on the brightness domain pixel mean and the brightness expected value, wherein the size of the stretching coefficient is negatively correlated with the brightness domain pixel mean and positively correlated with the brightness expected value. The updating module adjusts the pixel value based on the pixel value, the brightness domain pixel mean and the stretching coefficient.

10. A field programmable logic gate array, for executing the method according to any one of claims 1 to 8, wherein: The operations involved in the method are fixed-point operations obtained through quantization.

11. The field programmable gate array according to claim 10, wherein: The field programmable logic gate array uses a multi-port BRAM memory. The field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are allocated to different sub-depths, The field programmable logic gate array uses the pixel value of the read image in the brightness domain as an address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes it back to the original address, After reading all pixels, the field programmable logic gate array adds the values ​​stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of the pixel value to obtain the brightness domain representation.

12. An imaging device comprising: Imaging device and field programmable gate array, The field programmable logic gate array is used to execute the method described in any one of claims 1 to 8 to enhance the image captured by the imaging device, wherein the operations involved in the method are quantized fixed-point operations.

13. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.

14. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method according to any one of claims 1 to 6.

15. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 6.

Images