Image processing method and device, computer device, chip and medium

Abstract

The application relates to an image processing method and device, computer equipment, a chip and a medium. The method comprises the following steps: acquiring an image to be processed; performing brightness enhancement processing on each pixel point in the image to be processed according to the brightness value of the pixel point, so as to obtain a brightness-enhanced image; acquiring the brightness gradient data of each pixel point in the brightness-enhanced image; and performing black-and-white edge protection processing on the brightness-enhanced image according to the brightness gradient data of each pixel point. The application can improve the image quality.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, chip, and medium. Background Technology

[0002] In recent years, with the improvement of shooting technology in electronic devices, people's requirements for the quality of captured images have also been increasing. Therefore, it is necessary to further process the raw images captured by electronic devices.

[0003] Typically, original images contain significant noise, and removing this noise reduces image sharpness. To enhance image edges and details and improve sharpness, a sharpening module is usually used for enhancement. However, after enhancement, black and white borders can easily appear, affecting image quality. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer equipment, chip, and medium that can improve image quality in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides an image processing method, comprising: acquiring an image to be processed; performing brightness enhancement processing on corresponding pixels based on the brightness values ​​of each pixel in the image to be processed to obtain a brightness-enhanced image; acquiring brightness gradient data of each pixel in the brightness-enhanced image; and performing black-and-white edge protection processing on the brightness-enhanced image based on the brightness gradient data of each pixel.

[0006] In one embodiment, the black-and-white edge protection processing of the brightness-enhanced image based on the brightness gradient data of each pixel includes: for each pixel in the brightness-enhanced image, obtaining the brightness boundary value corresponding to the image block where the pixel is located; and in response to the brightness value of the pixel exceeding the corresponding brightness boundary value, adjusting the brightness of the pixel based on the brightness gradient data of the pixel.

[0007] In one embodiment, adjusting the brightness of a pixel based on its brightness gradient data includes: determining a black-and-white edge protection coefficient for the pixel based on the pixel's brightness gradient data; and determining the adjusted brightness value of the pixel based on the black-and-white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0008] In one embodiment, determining the adjusted brightness value of a pixel based on the black-and-white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value includes: determining a brightness control value based on the black-and-white edge protection coefficient and the degree to which the brightness value exceeds the corresponding brightness boundary value; and determining the adjusted brightness value based on the brightness control value and the brightness boundary value.

[0009] In one embodiment, performing brightness enhancement processing on corresponding pixels based on the brightness values ​​of each pixel in the image to be processed includes: determining an initial brightness enhancement value for the corresponding pixel based on the brightness values ​​of each pixel in the image to be processed; obtaining the semantic category to which each pixel in the image to be processed belongs; and performing brightness enhancement processing on the corresponding pixel based on the initial brightness enhancement value and the semantic category to which each pixel in the image to be processed belongs.

[0010] In one embodiment, brightness enhancement processing is performed on corresponding pixels based on their initial brightness enhancement values ​​and semantic categories in the image to be processed. This includes: extracting a target semantic region from the image to be processed based on the semantic category of each pixel, and obtaining a preset sharpening requirement for the target semantic region; adjusting the initial brightness enhancement values ​​of each pixel in the target semantic region according to the preset sharpening requirement to obtain a target brightness enhancement value for the corresponding pixel; performing brightness enhancement processing on the corresponding pixel based on the target brightness enhancement values ​​of each pixel in the target semantic region; and performing brightness enhancement processing on the corresponding pixel based on the initial brightness enhancement values ​​of each pixel in a non-target semantic region in the image to be processed.

[0011] Secondly, this application provides an image processing apparatus, comprising: an image acquisition module for acquiring an image to be processed; an enhancement processing module for performing brightness enhancement processing on corresponding pixels according to the brightness values ​​of each pixel in the image to be processed, thereby obtaining a brightness-enhanced image; a gradient acquisition module for acquiring brightness gradient data of each pixel in the brightness-enhanced image; and a protection processing module for performing black-and-white edge protection processing on the brightness-enhanced image according to the brightness gradient data of each pixel.

[0012] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method provided in the first aspect.

[0013] Fourthly, this application also provides a chip, including a processor and a communication interface, wherein the processor is configured to cause the chip to perform the steps of the method provided in the first aspect.

[0014] Fifthly, this application also provides a chip module, including a communication module, a power module, a storage module, and a chip, wherein: the power module is used to provide electrical energy to the chip module; the storage module is used to store data and instructions; the communication module is used for internal communication within the chip module, or for communication between the chip module and external devices; and the chip is used to perform the steps of the method provided in the first aspect above.

[0015] In a sixth aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method provided in the first aspect.

[0016] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method provided in the first aspect.

[0017] The aforementioned image processing method, apparatus, computer equipment, chip, and medium enhance the brightness of corresponding pixels based on their brightness values ​​in the image to be processed. Therefore, this embodiment achieves sharpening processing through brightness enhancement, thereby improving image clarity. Furthermore, by applying black-and-white edge protection processing to the brightness-enhanced image based on the brightness gradient data of each pixel, halos can be reduced, thus improving the black-and-white edge blurring problem caused by halos. Thus, this embodiment improves both image clarity and the black-and-white edge blurring problem, enhancing overall image quality. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an image processing method in one embodiment;

[0020] Figure 2A This is a flowchart illustrating the steps for determining brightness gradient data in one embodiment;

[0021] Figure 2B This is a schematic diagram of relevant pixels in the horizontal direction in an image block in one embodiment;

[0022] Figure 2C This is a schematic diagram of relevant pixels in the vertical direction in an image block in one embodiment;

[0023] Figure 2D This is a schematic diagram of relevant pixels in an image block at a 45° angle in one embodiment;

[0024] Figure 2E This is a schematic diagram of relevant pixels in an image block at a 135° angle in one embodiment;

[0025] Figure 3 This is a flowchart illustrating the black and white border protection process in one embodiment.

[0026] Figure 4A This is a flowchart illustrating the brightness adjustment steps in one embodiment;

[0027] Figure 4B This is a schematic diagram illustrating the functional relationship between the black and white edge protection coefficients in one embodiment;

[0028] Figure 5A This is a flowchart illustrating the steps for determining the adjusted brightness value in one embodiment;

[0029] Figure 5B This is a schematic diagram illustrating the relationship between brightness values ​​and brightness boundary values ​​in one embodiment;

[0030] Figure 6 This is a flowchart illustrating the brightness enhancement process steps in one embodiment;

[0031] Figure 7 This is a flowchart illustrating the steps for determining the initial brightness enhancement value in one embodiment;

[0032] Figure 8 This is a flowchart illustrating the brightness enhancement process steps in one embodiment;

[0033] Figure 9 This is a flowchart illustrating the brightness smoothing step in one embodiment;

[0034] Figure 10 This is a structural block diagram of an image processing device in one embodiment;

[0035] Figure 11 This is an internal structural diagram of a computer device in one embodiment;

[0036] Figure 12 This is an internal structure diagram of a chip module in one embodiment. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0039] In one exemplary embodiment, an image processing method is provided, see [link to example]. Figure 1 The method includes:

[0040] S110, Obtain the image to be processed.

[0041] The image to be processed is the original image captured by the electronic device after denoising. The image to be processed has low clarity and needs to be sharpened.

[0042] S120: Based on the brightness value of each pixel in the image to be processed, perform brightness enhancement processing on the corresponding pixel to obtain a brightness-enhanced image.

[0043] In this context, the luminance value is the Y component in the YCbCr color space of the digital signal, where the Y component represents luminance, Cb represents the blue chromaticity component, and Cr represents the red chromaticity component. Alternatively, the luminance value can be the Y component in the YUV color space of the analog signal, where the Y component represents luminance, U represents the blue chromaticity component, and V represents the red chromaticity component.

[0044] As can be seen, in S120, based on the brightness value of each pixel in the image to be processed, brightness enhancement processing is performed on that pixel. This brightness enhancement achieves sharpening processing, improving image clarity.

[0045] S130, acquire the brightness gradient data of each pixel in the brightness-enhanced image.

[0046] Each brightness gradient data point can be understood as the combined difference between the brightness values ​​of at least two pixels in an image block centered on that pixel.

[0047] The method for determining the brightness gradient data can be selected as needed, and no specific method is specified here.

[0048] S140 performs black and white edge protection processing on the brightness enhancement image based on the brightness gradient data of each pixel.

[0049] The black-and-white edge problem can be understood as the problem that after brightness enhancement processing, some pixels are too bright, resulting in halos and thus blurring the black-and-white edges of the image.

[0050] Among them, black and white edge protection processing can be understood as the process of improving the blurring of black and white edges caused by halo.

[0051] Among them, since the brightness gradient data can reflect the difference in brightness, the brightness of the pixels in the brightness-enhanced image is adjusted based on the brightness gradient data of the pixels to reduce halos and thus improve the problem of blurred black and white edges.

[0052] The image processing method described above enhances the brightness of each pixel in the image by adjusting its brightness value. This embodiment achieves sharpening through brightness enhancement, thereby improving image clarity. Furthermore, by applying black-and-white edge protection to the enhanced image based on the brightness gradient data of each pixel, halos are reduced, thus mitigating the blurring of black-and-white edges caused by halos. Therefore, this embodiment improves both image clarity and black-and-white edge blurring, enhancing overall image quality.

[0053] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the brightness gradient data determination step in S130 is refined.

[0054] See Figure 2A The steps for determining the refined brightness gradient data include:

[0055] S210. Determine the image block in the image to be processed with the pixel as the center pixel.

[0056] For example, the size of the image block is 5×5.

[0057] S220. Calculate the brightness gradient data of at least two pixels in the image block in different directions.

[0058] Different directions can include horizontal, vertical, 45° and 135° directions.

[0059] For example, see Figure 2B Based on the brightness values ​​of the 5 pixels in the second row and the 5 pixels in the fourth row, and the corresponding first gradient weight matrix in the horizontal direction, calculate the first brightness gradient data gradLayer0H in the horizontal direction. This can be achieved using the following formula:

[0060]

[0061] In the formula, i is the row number of the pixel, j is the column number of the pixel, p(i,j) is the brightness value of pixel (i,j), filterH is the first gradient weight matrix, and filterH(i,j) is the gradient weight of pixel (i,j) in the first gradient weight matrix. That is, the brightness values ​​of the 10 pixels in the second and fourth rows are multiplied by their corresponding gradient weights, summed, and then divided by the number of pixels (10) to obtain the first brightness gradient data gradLayer0H in the horizontal direction.

[0062] For example, see Figure 2CBased on the brightness values ​​of the 5 pixels in the second column, the 5 pixels in the fourth column, and the corresponding second gradient weight matrix in the vertical direction, the second brightness gradient data gradLayer0V in the vertical direction is calculated. This can be achieved using the following formula:

[0063]

[0064] In the formula, i is the row number of the pixel, j is the column number of the pixel, p(i,j) is the brightness value of pixel (i,j), filterV is the second gradient weight matrix, and filterV(i,j) is the gradient weight of pixel (i,j) in the second gradient weight matrix. That is, the brightness values ​​of the 10 pixels in the second and fourth columns are multiplied by their corresponding gradient weights, summed, and then divided by the number of pixels (10) to obtain the second brightness gradient data gradLayer0V in the vertical direction.

[0065] For example, see Figure 2D Based on the brightness values ​​of six pixels along the 45° direction and the corresponding third gradient weight matrix, the third brightness gradient data gradLayer0D45 along the 45° direction is calculated. This can be achieved using the following formula:

[0066]

[0067] In the formula, i is the row number of the pixel, j is the column number of the pixel, p(i,j) is the brightness value of pixel (i,j), filterD45 is the third gradient weight matrix, and filterD45(i,j) is the gradient weight of pixel (i,j) in the third gradient weight matrix. That is, the brightness values ​​of the 6 pixels along the 45° direction are multiplied by their corresponding gradient weights, summed, and then divided by the number of pixels, 6, to obtain the third brightness gradient data gradLayer0D45 corresponding to the 45° direction.

[0068] For example, see Figure 2E Based on the brightness values ​​of the six pixels along the 135° direction and the fourth gradient weight matrix corresponding to the 135° direction, the fourth brightness gradient data gradLayer0D135 corresponding to the 135° direction is calculated. This can be achieved using the following formula:

[0069]

[0070] In the formula, i is the row number of the pixel, j is the column number of the pixel, p(i,j) is the brightness value of pixel (i,j), filterD135 is the fourth gradient weight matrix, and filterD135(i,j) is the gradient weight of pixel (i,j) in the fourth gradient weight matrix. That is, the brightness values ​​of the 6 pixels along the 135° direction are multiplied by their corresponding gradient weights, summed, and then divided by the number of pixels, 6, to obtain the fourth brightness gradient data gradLayer0D135 corresponding to the 135° direction.

[0071] S230. Determine the brightness gradient data of the pixel based on the brightness gradient data in different directions.

[0072] For example, the brightness gradient data of a pixel can be calculated using the following formula. :

[0073]

[0074] In this embodiment, the brightness gradient data of a pixel is determined based on the brightness gradient data of at least two pixels in the image block where the pixel is located in different directions. Since the brightness gradient data in different directions are taken into account, the accuracy of the brightness gradient data of the pixel can be improved.

[0075] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the black and white edge protection processing steps in S140 are refined.

[0076] See Figure 3 The detailed steps for protecting black and white edges include:

[0077] S310: For each pixel in the brightness-enhanced image, obtain the brightness boundary value corresponding to the image block where the pixel is located.

[0078] The brightness boundary value corresponding to the image block where the pixel is located can be understood as the maximum and / or minimum brightness value among the brightness values ​​of each pixel in the image block centered on that pixel.

[0079] S320, in response to a pixel's brightness value exceeding the corresponding brightness boundary value, adjusts the brightness of the pixel based on the pixel's brightness gradient data.

[0080] In this context, a pixel's brightness value exceeding the corresponding brightness boundary value can be understood as either the pixel's brightness value being greater than the maximum brightness value mentioned above, or the pixel's brightness value being less than the minimum brightness value mentioned above.

[0081] Understandably, if the brightness value of a pixel does not exceed its corresponding brightness boundary value, the possibility of a halo effect is relatively small, so no brightness adjustment is performed. Since it is not necessary to adjust the brightness of all pixels, the computational workload related to brightness adjustment can be reduced.

[0082] In this embodiment, halos are more likely to appear when the brightness value of a pixel exceeds the corresponding brightness boundary value. Therefore, the brightness value of the pixel is adjusted to reduce halos, thereby improving the problem of blurred black and white edges.

[0083] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the brightness adjustment step in S320 is refined.

[0084] See Figure 4A The detailed brightness adjustment steps include:

[0085] S410 determines the black and white edge protection coefficient of a pixel based on the pixel's brightness gradient data.

[0086] Among them, the black and white edge protection coefficient can be understood as a weighting coefficient in the black and white edge protection process, which is used to adjust the degree of black and white edge protection.

[0087] The functional relationship between the black-and-white edge protection coefficient and the brightness gradient data can be found in [reference needed]. Figure 4B ,from Figure 4B As can be seen, when the pixel's brightness gradient data is less than the first preset value, there is a positive correlation between the brightness gradient data and the black-and-white edge protection coefficient; that is, the larger the brightness gradient data, the larger the black-and-white edge protection coefficient. When the pixel's brightness gradient data is greater than the second preset value but less than or equal to the third preset value, there is a negative-positive correlation between the brightness gradient data and the black-and-white edge protection coefficient; that is, the larger the brightness gradient data, the smaller the black-and-white edge protection coefficient. When the brightness gradient data is greater than or equal to the first preset value but less than or equal to the second preset value, the black-and-white edge protection coefficient is a fixed value. From the above functional relationships, the black-and-white edge protection coefficient corresponding to the brightness gradient data of each pixel can be found.

[0088] It can be seen that, based on the above Figure 4B For pixels in flat regions with low gradients and pixels in strong edge regions with high gradients, the black-and-white edge protection coefficient is smaller to provide a smaller degree of black-and-white edge protection. For pixels in weak texture regions with intermediate gradients, the possibility of halos is higher, so the black-and-white edge protection coefficient is larger to provide a larger degree of black-and-white edge protection.

[0089] Understandably, for each pixel, the black and white edge protection coefficient can be determined based on the pixel's brightness gradient data.

[0090] S420 determines the adjusted brightness value of a pixel based on the black and white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0091] That is, for each pixel, the adjusted brightness value of the pixel is determined based on the black and white edge protection coefficient of the pixel, the brightness boundary value corresponding to the image block to which the pixel is located, and the degree to which the brightness value of the pixel exceeds the corresponding brightness boundary value.

[0092] The brightness value in the degree to which the brightness value exceeds the corresponding brightness boundary value refers to the brightness value of the corresponding pixel in the brightness-enhanced image, not the brightness value of the corresponding pixel in the image to be processed.

[0093] Specifically, the degree to which a pixel's brightness value exceeds a corresponding brightness boundary value is defined as follows: For example, if the brightness value of a pixel is greater than the maximum brightness value of the image block in which the pixel is located, then the difference between the pixel's brightness value and the maximum brightness value of the image block is taken as the degree to which the pixel's brightness value exceeds the corresponding brightness boundary value. Conversely, if the brightness value of a pixel is less than the minimum brightness value of the image block in which the pixel is located, then the difference between the minimum brightness value of the image block and the pixel's brightness value is taken as the degree to which the pixel's brightness value exceeds the corresponding brightness boundary value.

[0094] In this embodiment, the black-and-white edge protection coefficient of a pixel is determined based on the pixel's brightness gradient data. This means that the degree of black-and-white edge protection is adjusted based on the brightness gradient data, allowing for flexible adjustment of the protection level. Furthermore, by considering the black-and-white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value, the adjusted brightness value can be accurately determined to improve the black-and-white edge problem.

[0095] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the step of determining the adjusted brightness value in S420 is refined.

[0096] See Figure 5A The detailed steps for determining the adjusted brightness value include:

[0097] S510 determines the brightness control value based on the black and white edge protection coefficient and the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0098] The brightness adjustment value can be understood as the adjusted value representing the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0099] The brightness control value can be obtained by multiplying the black and white edge protection coefficient by the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0100] For example, see Figure 5B When the luminance value is greater than the maximum luminance value among the luminance boundary values, the degree to which the luminance value exceeds the corresponding luminance boundary value can be expressed as:

[0101]

[0102] P1 = shootWeight × Overshoot

[0103] In the formula, Overshoot represents the degree to which the brightness value exceeds the corresponding maximum brightness value, blockMax represents the maximum brightness value, offset+PixelIn represents the brightness value of a pixel in the brightness-enhanced image, PixelIn represents the brightness value of the corresponding pixel in the image to be processed, offset represents the brightness enhancement value used when enhancing the brightness of the pixel in S120, shootWeight represents the black and white edge protection coefficient, and P1 represents the brightness adjustment value when the brightness value exceeds the corresponding maximum brightness value.

[0104] For example, see Figure 5B When the luminance value is less than the minimum luminance value among the luminance boundary values, the degree to which the luminance value exceeds the corresponding luminance boundary value can be expressed as:

[0105]

[0106] P2 = shootWeight × Undershoot

[0107] Wherein, Undershoot is the degree to which the brightness value is less than the corresponding minimum brightness value, blockMin is the minimum brightness value, and P2 is the brightness adjustment value when the brightness value is less than the corresponding minimum brightness value.

[0108] As can be seen, based on the above black and white edge protection coefficients, the degree of black and white edge protection can be adjusted.

[0109] S520 determines the adjusted brightness value based on the brightness control value and the brightness boundary value.

[0110] For example, when the brightness value is greater than the corresponding maximum brightness value, the formula for calculating the adjusted brightness value is as follows:

[0111]

[0112] In the formula, PixelOut1 is the adjusted brightness value when the brightness value is greater than the corresponding maximum brightness value.

[0113] For example, when the brightness value is less than the minimum brightness value among the brightness boundary values, the formula for calculating the adjusted brightness value is as follows:

[0114]

[0115] In the formula, PixelOut2 is the adjusted brightness value when the brightness value is less than the corresponding minimum brightness value.

[0116] That is, after performing the above steps on a pixel in the brightness enhancement image to obtain the adjusted brightness value, the adjusted brightness value is used to replace the brightness value in the brightness enhancement image, thereby achieving brightness control.

[0117] In this embodiment, the brightness control value is determined based on the black and white edge protection coefficient and the degree to which the brightness value exceeds the corresponding brightness boundary value. This enables the control of the degree to which the brightness value exceeds the corresponding brightness boundary value based on the black and white edge protection coefficient, thereby achieving accurate adjustment of the degree of black and white edge protection.

[0118] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the brightness enhancement processing steps in S120 are refined.

[0119] See Figure 6 The refined brightness enhancement processing steps include:

[0120] S610, determine the initial brightness enhancement value of the corresponding pixel based on the brightness value of each pixel in the image to be processed.

[0121] One approach is to use an image pyramid to determine the initial brightness enhancement value for each pixel. Of course, other methods can also be used to determine the initial brightness enhancement value, and this is not a limitation here.

[0122] S620, obtain the semantic category of each pixel in the image to be processed.

[0123] The semantic category can be understood as information that represents the category to which the semantic content of a pixel belongs.

[0124] For example, if an image includes both white cloud and blue sky regions, these regions can be extracted using semantic segmentation techniques. If a pixel is located in a white cloud region, its semantic category is "white cloud"; if it is located in a blue sky region, its semantic category is "blue sky". Thus, in this way, the semantic category of each pixel in the image to be processed can be determined.

[0125] In real-world scenarios, pixels with different semantic categories may have different enhancement requirements.

[0126] S630, based on the initial brightness enhancement value and semantic category of each pixel in the image to be processed, perform brightness enhancement processing on the corresponding pixel.

[0127] In this embodiment, when performing brightness enhancement processing on pixels, the initial brightness enhancement value and the semantic category to which they belong can be taken into account, which can improve the rationality of the brightness enhancement processing.

[0128] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the initial brightness enhancement value determination step in S610 is refined.

[0129] See Figure 7 The steps for determining the initial brightness enhancement value include:

[0130] S710 generates an N-layer pyramid image corresponding to the image block for each pixel.

[0131] Understandably, a pyramid image is a way of representing images at multiple scales. A pyramid image is a collection of images arranged in a pyramid shape with progressively decreasing resolution, all derived from the same original image. It is obtained by downsampling sequentially until a certain termination condition is reached.

[0132] Where N is a positive integer greater than 1.

[0133] The generation process of the i-th pyramid image is as follows:

[0134]

[0135] In the formula, P k For the image of the k-th pyramid, P k-1 Let be the (k-1)th layer pyramid image, where DownSample() is the downsampling function and Gauss() is the Gaussian filtering function.

[0136] For example, N is 3. For pixel a, P0 is an image patch centered at pixel a in the image to be processed, with a size of 9×9. After Gaussian filtering and downsampling, P1 is an image patch of size 5×5, centered at pixel a. P2 is an image patch centered at pixel a, with a size of 3×3. It can be seen that the resolution of P0, P1, and P2 decreases sequentially, with P0 containing the most high-frequency information (i.e., edge, detail, noise, etc.).

[0137] S720, determine the high-frequency information of each pyramid image in the N-layer pyramid image.

[0138] For example, the high-frequency information of P0 can be obtained by the following calculation formula:

[0139]

[0140] In the formula, Mlm() is the high-frequency information of P0, and is the nonlocal mean filter function.

[0141] For example, the high-frequency information of P1 can be obtained by the following calculation formula:

[0142]

[0143] In the formula, This refers to the high-frequency information of P1.

[0144] For example, the high-frequency information of P2 can be obtained by the following calculation formula:

[0145]

[0146] In the formula, This refers to high-frequency information for P2.

[0147] Among them, since the low-resolution layers P1 and P2 are less affected by noise, the difference between the image pixel value and its Gaussian filter can be directly calculated to obtain the high-frequency components.

[0148] S730 determines the initial brightness enhancement value corresponding to each pixel based on the high-frequency information of each pyramid image layer.

[0149] For example, the initial brightness enhancement value offset for pixel a is calculated using the following formula:

[0150]

[0151] In the formula, offset is the initial brightness enhancement value.

[0152] In this embodiment, for each pixel, the initial brightness enhancement value is determined based on the high-frequency information in the N-layer pyramid image corresponding to the image block containing that pixel. It is evident that this embodiment fully utilizes the different resolutions of the pyramid image, enabling better sharpening of different frequency details in the image based on the initial brightness enhancement value.

[0153] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the brightness enhancement processing steps in S630 are refined.

[0154] See Figure 8 The refined brightness enhancement processing steps include:

[0155] S810: Based on the semantic category of each pixel in the image to be processed, extract the target semantic region from the image to be processed, and obtain the preset sharpening requirements of the target semantic region.

[0156] The target semantic region can be understood as the semantic region that the user wants to control individually.

[0157] Understandably, semantic segmentation technology identifies the semantic category of each pixel in the image to be processed and labels each pixel with a corresponding semantic category tag. These semantic category tags allow for rapid segmentation of the image into different semantic regions. These semantic regions do not overlap and correspond to different semantic tags. For one or more semantic regions within each semantic region, individual adjustments can be made to achieve targeted sharpening and thus targeted improvement in clarity. The semantic region to be individually adjusted is designated as the target semantic region.

[0158] For example, an image may include a blue sky region, a white cloud region, and a person region. The person region can be used as the target semantic region.

[0159] The preset sharpening requirements for the target semantic region may include a brightness enhancement factor, or brightness enhancement sub-factors corresponding to different pyramid layers.

[0160] S820 adjusts the initial brightness enhancement value of each pixel in the target semantic region according to the preset sharpening requirements to obtain the target brightness enhancement value of the corresponding pixel.

[0161] For example, when the preset sharpening requirement includes a brightness enhancement factor, the target brightness enhancement value for each pixel in the target semantic region is calculated using the following formula:

[0162] offset' = offset × β

[0163] In the formula, offset' is the target brightness enhancement value of a pixel, offset is the initial brightness enhancement value of the corresponding pixel, and β is the brightness enhancement factor in the preset sharpening requirements corresponding to the target semantic region.

[0164] For example, when the preset sharpening requirements include brightness enhancement sub-factors corresponding to different pyramid image layers, the target brightness enhancement value of each pixel in the target semantic region is calculated using the following formula:

[0165] offset'=offset0×β0+offset1×β1+offset2×β2

[0166] In the formula, β0 is the brightness enhancement factor corresponding to P0, β1 is the brightness enhancement factor corresponding to P1, and β2 is the brightness enhancement factor corresponding to P2.

[0167] S830 performs brightness enhancement processing on the corresponding pixels based on the target brightness enhancement value of each pixel in the target semantic region.

[0168] That is, the brightness value of each pixel in the target semantic region is added to the target brightness enhancement value of that pixel to obtain the enhanced brightness value.

[0169] S840: Based on the initial brightness enhancement value of each pixel in the non-target semantic region of the image to be processed, perform brightness enhancement processing on the corresponding pixels.

[0170] That is, the brightness value of each pixel in the target semantic region is added to the initial brightness enhancement value of that pixel to obtain the enhanced brightness value.

[0171] In this embodiment, the brightness of pixels in non-target semantic regions is enhanced based on the initial brightness enhancement value; and the brightness of pixels in the target semantic region is enhanced based on the target brightness enhancement value. It is evident that this embodiment achieves both general sharpening and targeted sharpening processing.

[0172] Based on the technical solutions provided in the above embodiments, an optional embodiment is provided, in which the image processing method is refined to include a brightness smoothing step before S610.

[0173] See Figure 9 The brightness smoothing step includes:

[0174] S910, for each pixel in the image to be processed, determine the average brightness of at least two pixels in different directions of the image block centered on that pixel.

[0175] Different directions can include horizontal, vertical, 45° and 135° directions.

[0176] For example, see Figure 2B The average brightness value of the 5 pixels in the third row is calculated to obtain the average brightness value in the horizontal direction.

[0177] For example, see Figure 2C The average brightness value of the 5 pixels in the third column is calculated to obtain the average brightness value in the vertical direction.

[0178] For example, see Figure 2D The average brightness value of the five pixels along the diagonal at a 45° angle is calculated to obtain the average brightness value along the 45° angle.

[0179] For example, see Figure 2E The average brightness value of the five pixels along the diagonal at 135° is calculated to obtain the average brightness value along the 135° direction.

[0180] The average brightness value corresponding to different directions can be calculated using the following formula:

[0181]

[0182] In the formula, i is the direction number, and the value of i is 0, 1, 2, 3, which represent the horizontal direction, vertical direction, 45° direction and 135° direction respectively; avg_direction(i) represents the average brightness value corresponding to the i-th direction, and pixel'(j,i) represents the brightness value of the j-th pixel in the i-th direction.

[0183] S920: For each direction, determine the brightness gradient data for that direction based on the average brightness value corresponding to that direction and the brightness values ​​of at least two pixels in that direction.

[0184] Among them, the brightness gradient data in one direction can represent the difference in brightness between pixels in that direction.

[0185] For example, the brightness gradient data in different directions can be calculated using the following formula:

[0186]

[0187] In the formula, diff(i) represents the brightness gradient data in the i-th direction.

[0188] S930: Select the minimum brightness gradient data from the brightness gradient data in each direction. In response to the minimum brightness gradient data being less than a preset threshold, take the direction corresponding to the minimum brightness gradient data as the target direction.

[0189] For example, the formula for selecting the minimum brightness gradient data is:

[0190]

[0191] In the formula, mindiff represents the minimum brightness gradient data.

[0192] As can be seen, when mindiff is less than the preset threshold, the center pixel of the image block is considered to have directionality, so the direction corresponding to the minimum brightness gradient data is taken as the target direction.

[0193] S940, determine the smoothed brightness value of the center pixel of the image block from the brightness values ​​of at least two pixels in the target direction, and use the smoothed brightness value to replace the brightness value of the center pixel.

[0194] For example, the average value of the brightness values ​​of the four pixels other than the center pixel out of the five pixels in the target direction is calculated, and the calculated average value is used as the smoothed brightness value. This smoothed brightness value replaces the original brightness value of the center pixel, thus achieving the brightness smoothing process of the center pixel.

[0195] In this embodiment, the smoothed brightness value of the center pixel of the image block is determined based on the brightness values ​​of at least two pixels in the target direction, achieving a smoothing effect along its direction. Furthermore, smoothing is performed before generating the pyramid image to avoid abrupt brightness changes and improve the accuracy of the initial brightness enhancement value.

[0196] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0197] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. This apparatus can be applied to or integrated into a chip or chip module, for example. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0198] In one exemplary embodiment, see Figure 10 An image processing apparatus is provided, comprising: an image acquisition module 1010, an enhancement processing module 1020, a gradient acquisition module 1030, and a protection processing module 1040, wherein:

[0199] Image acquisition module 1010 is used to acquire the image to be processed;

[0200] The enhancement processing module 1020 is used to perform brightness enhancement processing on the corresponding pixels according to the brightness value of each pixel in the image to be processed, so as to obtain a brightness enhanced image;

[0201] Gradient acquisition module 1030 is used to acquire brightness gradient data of each pixel in the brightness enhancement image;

[0202] The protection processing module 1040 is used to perform black and white edge protection processing on the brightness enhancement image based on the brightness gradient data of each pixel.

[0203] In one embodiment, the protection processing module includes: a first acquisition submodule, configured to acquire the brightness boundary value corresponding to the image block where the pixel is located for each pixel in the brightness-enhanced image; and a first adjustment submodule, configured to adjust the brightness of the pixel according to the brightness gradient data of the pixel in response to the pixel's brightness value exceeding the corresponding brightness boundary value.

[0204] In one embodiment, the first adjustment submodule includes: a first determining unit, configured to determine the black-and-white edge protection coefficient of a pixel based on the pixel's brightness gradient data; and a second determining unit, configured to determine the adjusted brightness value of a pixel based on the black-and-white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value.

[0205] In one embodiment, the second determining unit is specifically used to: determine a brightness adjustment value based on the black-and-white edge protection coefficient and the degree to which the brightness value exceeds the corresponding brightness boundary value; and determine the adjusted brightness value based on the brightness adjustment value and the brightness boundary value.

[0206] In one embodiment, the enhancement processing module includes: a first determining submodule, configured to determine an initial brightness enhancement value for a corresponding pixel based on the brightness value of each pixel in the image to be processed; a second obtaining submodule, configured to obtain the semantic category to which each pixel in the image to be processed belongs; and a first enhancement submodule, configured to perform brightness enhancement processing on the corresponding pixel based on the initial brightness enhancement value and the semantic category to which each pixel in the image to be processed belongs.

[0207] In one embodiment, the first enhancement submodule is specifically configured to: extract a target semantic region from the image to be processed according to the semantic category to which each pixel belongs, and obtain a preset sharpening requirement for the target semantic region; adjust the initial brightness enhancement value of each pixel in the target semantic region according to the preset sharpening requirement to obtain the target brightness enhancement value of the corresponding pixel; perform brightness enhancement processing on the corresponding pixel according to the target brightness enhancement value of each pixel in the target semantic region; and perform brightness enhancement processing on the corresponding pixel according to the initial brightness enhancement value of each pixel in the non-target semantic region of the image to be processed.

[0208] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, all of their modules / units can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0209] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0210] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0211] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image processing method described above.

[0212] Based on the same inventive concept, this application also provides a chip, including a processor and a communication interface; the communication interface is used to receive or send data; the processor is configured to cause the chip to perform the steps of the above-described image processing method.

[0213] It is understood that the chip involved in the embodiments of this application may be a field-programmable gate array (FPGA), may be an application-specific integrated circuit (ASIC), may be a system on chip (SoC), may be a central processor unit (CPU), may be a network processor (NP), may be a digital signal processor (DSP), may be a microcontroller unit (MCU), may be a programmable logic device (PLD), or other integrated chips, etc.

[0214] Based on the same inventive concept, this application also provides a chip module, such as... Figure 12 As shown, the chip module includes a communication module, a power module, a storage module, and a chip. Specifically: the power module provides power to the chip module; the storage module stores data and instructions; the communication module enables internal communication within the chip module or communication between the chip module and external devices; and the chip corresponds to the chip in the aforementioned chip embodiment. The implementation of this chip module can be found in the relevant content of the aforementioned chip embodiment, and will not be repeated here.

[0215] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method described above.

[0216] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image processing method described above.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; Based on the brightness value of each pixel in the image to be processed, the corresponding pixel is subjected to brightness enhancement processing to obtain a brightness enhanced image; Obtain the brightness gradient data of each pixel in the brightness-enhanced image; Based on the brightness gradient data of each pixel, the brightness-enhanced image is subjected to black and white edge protection processing.

2. The method according to claim 1, characterized in that, The black-and-white edge protection processing of the brightness-enhanced image based on the brightness gradient data of each pixel includes: For each pixel in the brightness-enhanced image, obtain the brightness boundary value corresponding to the image block where the pixel is located; In response to the brightness value of the pixel exceeding the corresponding brightness boundary value, the brightness of the pixel is adjusted according to the brightness gradient data of the pixel.

3. The method according to claim 2, characterized in that, The step of adjusting the brightness of the pixel based on the pixel's brightness gradient data includes: The black and white edge protection coefficient of the pixel is determined based on the brightness gradient data of the pixel. The adjusted brightness value of the pixel is determined based on the black and white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value.

4. The method according to claim 3, characterized in that, The step of determining the adjusted brightness value of the pixel based on the black-and-white edge protection coefficient, the brightness boundary value, and the degree to which the brightness value exceeds the corresponding brightness boundary value includes: The brightness adjustment value is determined based on the black and white edge protection coefficient and the degree to which the brightness value exceeds the corresponding brightness boundary value; The adjusted brightness value is determined based on the brightness control value and the brightness boundary value.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing brightness enhancement processing on corresponding pixels based on the brightness values ​​of each pixel in the image to be processed includes: Based on the brightness value of each pixel in the image to be processed, determine the initial brightness enhancement value of the corresponding pixel; Obtain the semantic category of each pixel in the image to be processed; Based on the initial brightness enhancement value and semantic category of each pixel in the image to be processed, brightness enhancement processing is performed on the corresponding pixels.

6. The method according to claim 5, characterized in that, The step of performing brightness enhancement processing on corresponding pixels based on their initial brightness enhancement values ​​and semantic categories in the image to be processed includes: Based on the semantic category of each pixel in the image to be processed, the target semantic region is extracted from the image to be processed, and the preset sharpening requirements of the target semantic region are obtained. According to the preset sharpening requirements, the initial brightness enhancement value of each pixel in the target semantic region is adjusted to obtain the target brightness enhancement value of the corresponding pixel. Based on the target brightness enhancement value of each pixel in the target semantic region, perform brightness enhancement processing on the corresponding pixels; Based on the initial brightness enhancement value of each pixel in the non-target semantic region of the image to be processed, brightness enhancement processing is performed on the corresponding pixels.

7. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire the image to be processed. The enhancement processing module is used to perform brightness enhancement processing on the corresponding pixels according to the brightness value of each pixel in the image to be processed, so as to obtain a brightness enhanced image; The gradient acquisition module is used to acquire the brightness gradient data of each pixel in the brightness enhancement image; The protection processing module is used to perform black and white edge protection processing on the brightness enhancement image based on the brightness gradient data of each pixel.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A chip, characterized in that, The device includes a processor and a communication interface, wherein the processor is configured to cause the chip to perform the steps of the method described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images