Image processing method and device, computer readable storage medium, and terminal

Abstract

An image processing method, apparatus, computer-readable storage medium, and terminal are disclosed. The method includes: determining an image to be processed and the brightness value of each pixel; selecting low brightness values ​​and high brightness values ​​from among the brightness values; determining four control points corresponding to each pixel in the image to be processed, wherein the x-coordinate and y-coordinate of the first control point are both low brightness values, the x-coordinate and y-coordinate of the second control point are both high brightness values, and the slopes of the lines between the first and third control points and between the first and fourth control points are both greater than the slope of the line between the first and second control points; fitting a tone mapping curve corresponding to the pixel based on the four control points, and performing tone mapping on the brightness value of the pixel using the tone mapping curve. Using the above scheme to perform tone mapping on the original image helps to retain more detail information in the original image when subsequently reducing the bit width of the tone-mapped image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image processing method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] In the field of image processing technology, high-bit-width images (e.g., high dynamic range multi-exposure fusion images) typically contain more brightness and detail information, but correspondingly occupy more bits. In many applications, it is necessary to reduce the bit-width of the input high-bit-width image (e.g., by using shift operations to reduce the bit-width) to obtain a low-bit-width image, in order to reduce the amount of computational data or adapt to low-bit-width image processing systems.

[0003] However, in existing technologies, reducing the bit width of high-bit-width images often leads to the loss of original image details, especially in dark areas. Since the brightness values ​​of pixels in dark areas are already very small, reducing the bit width results in even more severe loss of image information in these areas. Therefore, researching how to reduce the bit width of an image while preserving as much of its original detail as possible, so that the reduced-bit-width image still maintains high contrast, dynamic range, and color fidelity, is of great significance for improving image processing performance. Summary of the Invention

[0004] The technical problem solved by the embodiments of the present invention is: how to reduce the bit width of the image while preserving as much of the original detail information in the image as possible, so that the image after the bit width reduction can still maintain a high image quality.

[0005] To address the aforementioned technical problems, this invention provides an image processing method comprising the following steps: determining an image to be processed and the brightness value of each pixel; selecting two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, denoted as a low brightness value and a high brightness value, wherein the high brightness value is greater than the low brightness value; for each pixel in the image to be processed, determining four control points corresponding to that pixel, denoted as a first control point, a second control point, a third control point, and a fourth control point, wherein the horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal and vertical coordinates of the second control point are... The vertical and horizontal axes represent the high brightness values. The slopes of the lines between the first and third control points, and between the first and fourth control points, are both greater than the slope of the line between the first and second control points. Curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel. The first and second control points serve as the endpoints of the tone mapping curve, while the third and fourth control points control the curvature of the tone mapping curve. The tone mapping result of the pixel is determined based on its brightness value and the tone mapping curve.

[0006] Optionally, for each pixel in the image to be processed, determining the third and fourth control points corresponding to that pixel includes: calculating the average brightness value of each pixel within a preset area centered on that pixel; determining the local average brightness value of the pixel based on the average value, the low brightness value, and the high brightness value; determining the difference between the high brightness value and the local average brightness value, denoted as a first difference, and determining the difference between the high brightness value and the low brightness value, denoted as a second difference; determining the ratio of the first difference and the second difference, and the adjustment coefficient corresponding to that pixel; selecting a first initial point and a second initial point respectively, such that the slope of the straight line between the first control point and the first initial point, and the slope of the straight line between the first control point and the second initial point are both greater than the slope of the straight line between the first control point and the second control point; and adjusting the coordinates of the first initial point and the second initial point respectively using the adjustment coefficient corresponding to that pixel to obtain the third and fourth control points corresponding to that pixel.

[0007] Optionally, determining the local average brightness of the pixel based on the average value, the low brightness value, and the high brightness value includes: if the average value is greater than or equal to the high brightness value, then the high brightness value is used as the local average brightness of the pixel; if the average value is less than or equal to the low brightness value, then the low brightness value is used as the local average brightness of the pixel; if the average value is greater than the low brightness value and less than the high brightness value, then the average value is used as the local average brightness of the pixel.

[0008] Optionally, the step of using the adjustment coefficient corresponding to the pixel to adjust the coordinates of the first initial point and the second initial point respectively to obtain the third control point and the fourth control point corresponding to the pixel includes: calculating the product of the adjustment coefficient corresponding to the pixel and the x-coordinate and y-coordinate of the first initial point respectively, as the x-coordinate and y-coordinate of the third control point; and calculating the product of the adjustment coefficient corresponding to the pixel and the x-coordinate and y-coordinate of the second initial point respectively, as the x-coordinate and y-coordinate of the fourth control point.

[0009] Optionally, the slope of the straight line between the first control point and the third control point is greater than the slope of the straight line between the first control point and the fourth control point.

[0010] Optionally, two brightness values ​​are selected from the brightness values ​​of each pixel in the image to be processed, and are respectively denoted as a low brightness value and a high brightness value. This includes: determining the Nth brightness value in ascending order from the minimum brightness value among the brightness values ​​of each pixel in the image to be processed, as the low brightness value, and determining the Mth brightness value in descending order from the maximum brightness value, as the high brightness value, where M and N are both positive integers, and M and N are both less than 1 / 2 of the total number of pixels in the image to be processed.

[0011] Optionally, determining the Nth brightness value sorted from the minimum brightness value in ascending order as the low brightness value, and determining the Mth brightness value sorted from the maximum brightness value in descending order as the high brightness value, includes: using multiple brightness value intervals to perform histogram statistics on the brightness values ​​of each pixel in the image to be processed, and obtaining histogram statistics results; starting from the minimum brightness value interval, accumulating the number of pixels in ascending order of brightness value until the total number of accumulated pixels reaches N, and determining the brightness value of the Nth pixel as the low brightness value; starting from the maximum brightness value interval, accumulating the number of pixels in descending order of brightness value until the total number of accumulated pixels reaches M, and determining the brightness value of the Mth pixel as the high brightness value.

[0012] Optionally, the plurality of brightness value intervals are divided based on the bit width of the image to be processed; wherein, the smaller the brightness value of a brightness value interval, the smaller the width of the brightness value interval, and the larger the brightness value of a brightness value interval, the larger the width of the brightness value interval.

[0013] Optionally, curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel, including: using the Bézier curve fitting formula, curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel.

[0014] Optionally, before selecting two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, the method further includes: for each pixel in the image to be processed, determining the R channel value, G channel value, and B channel value of the pixel based on the brightness values ​​of each pixel in a preset area surrounding the pixel as the center point; and updating the brightness value of the pixel to the maximum value among the R channel value, G channel value, and B channel value.

[0015] Optionally, based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel, the tone mapping result of the pixel is determined, including: using the brightness value of the pixel as the horizontal axis value, determining the corresponding vertical axis value on the tone mapping curve corresponding to the pixel as the initial tone mapping result of the pixel; determining the difference between the brightness value of the pixel and the high brightness value, and then using the difference to adjust the initial tone mapping result to obtain the tone mapping result of the pixel.

[0016] Optionally, the initial tone mapping result can be adjusted using the following formula:

[0017] RAW OUT =RAW1+(RAW0-Y) max )×ratio;

[0018] Among them, RAW OUT RAW1 represents the tone mapping result of this pixel, RAW0 represents the brightness value of this pixel, and Y represents the initial tone mapping result of this pixel. max The value represents the high brightness, and ratio represents the preset adjustment coefficient.

[0019] Optionally, determining the image to be processed includes: acquiring image blocks to be fused, the image blocks including one or more reference blocks and a set of fused blocks matching each reference block, each set of fused blocks including one or more fused blocks, wherein the reference blocks originate from a reference frame, and each fused block in the set of fused blocks originates from a fused frame with a different exposure duration than the reference frame; sequentially for each reference block, using the reference block as a reference, determining the fusion weight of the reference block and its matching fused blocks according to the exposure duration and brightness, and performing fusion according to the fusion weight to obtain the fusion result of each reference block; generating a fused image based on the fusion result of each reference block, the fused image serving as the image to be processed.

[0020] Optionally, determining the fusion weights of the reference block and its matching block to be fused based on the exposure time and brightness includes: determining the fusion weights of the center points of each block in the reference block and its matching block to be fused, using pixels in the reference frame corresponding to the reference block as units, based on the exposure time and brightness; and performing fusion based on the fusion weights to obtain the fusion result of each reference block includes: calculating the fusion value of the center point of the reference block based on the fusion weights of the center points of each block; and using the fusion value of the center point of the reference block as the fusion result of the reference block.

[0021] Optionally, the fusion weights may include any one or more of the following: exposure weights, brightness weights, saturation weights, and difference weights.

[0022] Optionally, determining the exposure weight of the center point of each block includes: determining the exposure weight of the block based on the exposure duration and gain of the original image corresponding to each block; and using the exposure weight of the block as the exposure weight of the center point of the block.

[0023] Optionally, determining the brightness weight of the center point of each block includes: calculating the weighted average of the brightness values ​​of all pixels in each color channel in the block to obtain the brightness value of each color channel; determining the overexposure weight based on the maximum brightness value of each color channel; determining the underexposure weight based on the minimum brightness value of each color channel; determining the smaller value between the overexposure weight and the underexposure weight, and normalizing the smaller value to obtain the brightness weight of the center point of the block.

[0024] Optionally, determining the saturation weight of the center point of each block includes: determining the R channel value, G channel value, and B channel value of the center point of the block based on the brightness value of each pixel in the block; determining the maximum and minimum values ​​among the R channel value, G channel value, and B channel value of the center point of the block, and determining the average value of the maximum and minimum values; and using the difference between the maximum value and the average value as the saturation weight of the center point of the block.

[0025] Optionally, determining the difference weight of the center point of each block includes: taking a weighted average of the differences of all pixels in each color channel in the block to obtain the difference value of each color channel based on the pixel; determining the maximum difference value of each color channel and normalizing the maximum difference value to obtain the difference weight of the center point of the block.

[0026] Optionally, the method further includes: acquiring multiple original images corresponding to the same scene but with different exposure durations; selecting one image from the multiple original images as a reference frame, and other image frames outside the reference frame as frames to be fused; mirroring and tracing the edges of each original image according to a set block size to obtain a traced image corresponding to the original image; partitioning the traced image according to the block size to obtain each block of the traced image; using the block corresponding to the reference frame as the reference block, and the block corresponding to each frame to be fused as the fused block; aligning and registering the fused blocks with the reference block as a reference to obtain a set of fused blocks matching each reference block.

[0027] This invention also provides an image processing apparatus, comprising: a to-be-processed image determination module, configured to determine the to-be-processed image and the brightness value of each pixel; a high / low brightness value determination module, configured to select two brightness values ​​from the brightness values ​​of each pixel in the to-be-processed image, denoted as a low brightness value and a high brightness value, wherein the high brightness value is greater than the low brightness value; and a control point determination module, configured to determine four control points corresponding to each pixel in the to-be-processed image, denoted as a first control point, a second control point, a third control point, and a fourth control point, wherein the horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal coordinate of the second control point is... The vertical axis and the horizontal axis are both the high brightness value. The slopes of the straight lines between the first control point and the third control point, and between the first control point and the fourth control point, are both greater than the slope of the straight line between the first control point and the second control point. The tone mapping curve determination module is used to perform curve fitting based on the four control points to obtain the tone mapping curve corresponding to the pixel. The first control point and the second control point are respectively used as the endpoints of the tone mapping curve, and the third control point and the fourth control point are used to control the curvature of the tone mapping curve. The tone mapping module is used to determine the tone mapping result of the pixel based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel.

[0028] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, executes the steps of the above-described image processing method.

[0029] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the above-described image processing method when running the computer program.

[0030] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0031] In this embodiment of the invention, by selecting lower and higher brightness values ​​from the brightness values ​​of each pixel in the image to be processed, a first control point and a second control point are respectively determined as the two endpoints of the tone mapping curve. These two endpoints can be used to limit the minimum and maximum brightness values ​​after tone mapping, preventing the dark areas of the tone-mapped image from being too dark or the bright areas from being too bright. Furthermore, since the third and fourth control points are used to control the curvature (or shape) of the tone mapping curve, this embodiment sets the third and fourth control points to satisfy the following: the slopes of the lines between the first and third control points, and the slopes of the lines between the first and fourth control points, are both greater than the slope of the line between the first and second control points. This allows for the fitting of a tone mapping curve with the expected curvature / shape change trend. After using the fitted tone mapping curve to perform tone mapping on the image to be processed, it is possible to increase the brightness values ​​of pixels in darker areas of the image and appropriately decrease the brightness values ​​of pixels in brighter areas, while avoiding grayscale inversion in the tone-mapped image compared to the image to be processed. Therefore, by adopting this implementation scheme, the image to be processed is tone-mapped to obtain a tone-mapped image. Since the brightness value of the dark area of ​​the image has been appropriately increased, the loss of image detail information can be minimized when the bit width reduction operation is subsequently performed on the tone-mapped image. That is, the detail information, high dynamic range, and high contrast of the image to be processed are preserved.

[0032] Furthermore, in determining the third and fourth control points, the following condition can be set: the slope of the straight line between the first and third control points is greater than the slope of the straight line between the first and fourth control points. This is more conducive to accurately improving the brightness of overly dark areas and accurately reducing the brightness of overly bright areas, further improving the tone mapping effect.

[0033] Furthermore, in this embodiment of the invention, for each pixel, the coordinates of its corresponding third and fourth control points can be determined by adjusting coefficients based on the low brightness value, high brightness value, and the local average brightness of the pixel. However, in practical applications, since the local average brightness of different pixels may differ, the adjustment coefficients corresponding to different pixels may also differ, and the third / fourth control points corresponding to different pixels may also differ. Therefore, the tone mapping curves corresponding to different pixels may also differ. Compared to using the entire image as a unit and determining the adjustment coefficients using the overall average brightness of the image to be processed (e.g., the average brightness of all pixels in the image), this embodiment determines the corresponding adjustment coefficients and third and fourth control points for each pixel based on its local average brightness. This allows each pixel to have a tone mapping curve adapted to its local average brightness. Thus, more refined tone mapping can be achieved, improving the accuracy of tone mapping. Attached Figure Description

[0034] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0035] Figure 2 yes Figure 1 A flowchart of a specific implementation of step S13;

[0036] Figure 3 This is a schematic diagram of a tone mapping curve obtained based on fitting four control points in an embodiment of the present invention;

[0037] Figure 4 yes Figure 1 A flowchart of a specific implementation of step S15;

[0038] Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention. Detailed Implementation

[0039] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Reference Figure 1 , Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention. The method may include steps S11 to S15:

[0041] Step S11: Determine the image to be processed and the brightness value of each pixel;

[0042] Step S12: Select two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, and record them as low brightness value and high brightness value respectively, wherein the high brightness value is greater than the low brightness value;

[0043] Step S13: For each pixel of the image to be processed, determine four control points corresponding to that pixel, denoted as the first control point, the second control point, the third control point, and the fourth control point. The horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal and vertical coordinates of the second control point are both the high brightness value. The slopes of the lines between the first and third control points and between the first and fourth control points are both greater than the slope of the line between the first and second control points.

[0044] Step S14: Perform curve fitting based on the four control points to obtain the tone mapping curve corresponding to the pixel, wherein the first control point and the second control point are respectively used as the endpoints of the tone mapping curve, and the third control point and the fourth control point are used to control the curvature of the tone mapping curve.

[0045] Step S15: Determine the tone mapping result of the pixel based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel.

[0046] Furthermore, after determining the tone mapping results of all pixels in the image to be processed, the tone mapping results of all pixels are used to generate the tone-mapped image.

[0047] It is understandable that, in practice, the above method can be implemented using software programs that run within the processor integrated into the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software.

[0048] In the specific implementation of step S11, the image to be processed can be a high-bit-width RWA image (i.e., the original image), such as a High Dynamic Range (HDR) image, a multi-exposure fusion image, etc. Here, a RAW image refers to an unprocessed image directly output by a digital camera. Typically, each pixel in a RAW image has a single channel. Different channels are distributed in an array on the RAW image.

[0049] In a specific implementation of step S12, it may include: determining the Nth brightness value in ascending order of brightness from the minimum brightness value among the brightness values ​​of each pixel in the image to be processed, as the low brightness value, and determining the Mth brightness value in descending order of brightness from the maximum brightness value, as the high brightness value, wherein M and N are both positive integers, and M and N are both less than 1 / 2 of the total number of pixels in the image to be processed.

[0050] Furthermore, the brightness value that is Nth in order of increasing brightness from the minimum brightness value (i.e., the low brightness value Y) can be determined in the following manner. min And the Mth brightness value (i.e., the high brightness value Y) sorted from the maximum brightness value in descending order. max The following steps are taken: A histogram is plotted on the brightness values ​​of each pixel in the image to be processed, using multiple brightness value intervals. Starting from the interval with the smallest brightness value, the number of pixels is accumulated in ascending order of brightness value until the total number of accumulated pixels reaches N. The brightness value of the Nth pixel is then determined as the low brightness value Y. min Starting from the range with the highest brightness value, the number of pixels is accumulated interval by interval in descending order of brightness value until the total number of accumulated pixels reaches M. Then, the brightness value of the Mth pixel is determined as the high brightness value Y. max .

[0051] As a non-limiting embodiment, M and N can be selected from 2% to 5% of the total number of pixels in the image to be processed.

[0052] The multiple brightness value intervals are divided based on the bit width of the image to be processed; the smaller the brightness value of a brightness value interval, the smaller the width of that interval; conversely, the larger the brightness value of a brightness value interval, the larger the width of that interval. Non-limitingly, if the bit width of the image to be processed is X bits, it can be divided into 2... X / 2 A non-linear brightness value range.

[0053] In specific implementation, in addition to using the above method to determine the low brightness value Y min and high brightness value Y max Alternatively, other appropriate methods can be used. For example, first determine the global mean of the image to be processed (such as the average brightness value of all pixels), then select a relatively small brightness value between the minimum brightness value of each pixel and the global mean as the low brightness value; and select a relatively large brightness value between the global mean and the maximum brightness value of each pixel as the high brightness value.

[0054] Furthermore, before performing step S12, the method may further include: for each pixel of the image to be processed, determining the R channel value, G channel value, and B channel value of the pixel based on the brightness values ​​of each pixel in a preset area surrounding the pixel as the center point; and updating the brightness value of the pixel to the maximum value among the R channel value, G channel value, and B channel value.

[0055] The preset area surrounding the pixel can specifically refer to an image block centered on that pixel. Each pixel in the image to be processed belongs to an image block and serves as the center point of that block. For example, the size of the image block can be set to a×a, where a represents the number of pixels. After setting the block size to a×a, the edges of each original image are mirrored, so that any pixel on any edge of the original image can serve as the center point of an a×a block in the corresponding mirrored image. Without limitation, a can be selected from 3, 5, 7, or other appropriate values.

[0056] For more details on determining the image block, please refer to the description of steps S201 to S205 below, which will not be repeated here.

[0057] As mentioned earlier, for RAW images, since each pixel typically has only a single color channel, the R channel value, G channel value, and B channel value of the current pixel can be determined based on the brightness values ​​of each pixel in the preset area (i.e., the image block) surrounding the current pixel.

[0058] Taking the 3×3 image block shown below as an example (where the small rectangle represents a pixel, and the English letters in the small rectangle represent a single color channel of the pixel; in this example, the color channel of the center point of the image block is the R channel). Specifically, the color channel of the center point of the image block is the R channel, and the R channel value, G channel value, and B channel value of the center point of the image block can be determined using the following formula (1):

[0059] B G B G R G B G B

[0060] R i,j =RAW i,j ;

[0061]

[0062]

[0063] Where (i, j) represents the position of the pixel, i represents the x-coordinate of the pixel, and j represents the y-coordinate of the pixel. i,j R represents the brightness value of the pixel at position (i, j), that is, the brightness value of the center point of the image block. i,j G represents the red channel value at the center of an image patch. i,j B represents the green channel value at the center of an image block. i,j This represents the blue channel value at the center of the image block.

[0064] After determining the R channel, R channel value, G channel value, and B channel value of the current pixel using the above method, the brightness value of the pixel is updated to the maximum value among the R channel value, G channel value, and B channel value. Furthermore, in step S12, the low brightness value and high brightness value can be selected from the updated brightness values ​​of each pixel in the image to be processed.

[0065] In the specific implementation of step S13, for each pixel, the low brightness value Y can be selected as the reference value. min High brightness value Y max Determine the four control points corresponding to this pixel. The x and y coordinates of the first control point A corresponding to this pixel are both the low brightness value, i.e., the coordinates of the first control point A are (Y...). min Y min The x-coordinate and y-coordinate of the second control point B are both the high brightness value, that is, the coordinates of the second control point B are (Y-Y-Z). max Y max The third control point C and the fourth control point D satisfy the following objective conditions:

[0066] The slopes k1 and k2 of the straight lines between the first control point A and the third control point C are both greater than the slope k0 of the straight line between the first control point A and the second control point B.

[0067] As can be seen from the above, for the same frame of the image to be processed, the low brightness value Y min and the high brightness value Y max Since the first control point A is fixed, the second control point B corresponding to each pixel in the image to be processed in this frame is the same and remains unchanged. The third control point C / fourth control point D corresponding to each pixel can be the same or different, provided that the above target conditions are met.

[0068] For a specific implementation scheme for determining the third control point C and the fourth control point D of each pixel, see below. Figure 2 Description of each step.

[0069] Reference Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation of step S13 is provided. In step S13, for each pixel of the image to be processed, the third control point and the fourth control point corresponding to the pixel are determined. Specifically, it may include steps S21 to S25.

[0070] In step S21, the average value of the brightness values ​​of each pixel in the preset area surrounding the pixel is calculated, and the local average brightness value of the pixel is determined based on the average value, the low brightness value, and the high brightness value.

[0071] Further, determining the local average brightness of the pixel based on the average value, the low brightness value, and the high brightness value includes: if the average value is greater than or equal to the high brightness value, then the high brightness value is used as the local average brightness of the pixel; if the average value is less than or equal to the low brightness value, then the low brightness value is used as the local average brightness of the pixel; if the average value is greater than the low brightness value and less than the high brightness value, then the average value is used as the local average brightness of the pixel.

[0072] Compared to directly using the average brightness value of each pixel within the preset area as the local average brightness value, the above-described scheme in this embodiment of the invention ensures that the local average brightness value of each pixel is within the range of low and high brightness values. Furthermore, it limits the values ​​of the subsequently calculated adjustment coefficients to the range [0, 1], facilitating more precise adjustment of the positions of the third and fourth control points and accurate adjustment of the curvature of the tone mapping curve, resulting in a better tone mapping effect.

[0073] In step S22, the difference between the high brightness value and the local average brightness value is determined and recorded as the first difference, and the difference between the high brightness value and the low brightness value is determined and recorded as the second difference.

[0074] In step S23, the ratio of the first difference to the second difference is determined, which is the adjustment coefficient corresponding to the pixel.

[0075] Specifically, the adjustment coefficient for each pixel can be determined using the following formula (2):

[0076] strength=(Y max -Y avg ) / (Y max -Y min );

[0077] Where strength represents the adjustment coefficient of the current pixel, Y max Indicates the high brightness value, Y min Indicates the low brightness value, Y avg This represents the local average brightness of the current pixel.

[0078] In step S24, a first initial point and a second initial point are selected respectively, such that the slope of the straight line between the first control point and the first initial point, and the slope of the straight line between the first control point and the second initial point are both greater than the slope of the straight line between the first control point and the second control point.

[0079] In step S25, the adjustment coefficient corresponding to the pixel is used to adjust the coordinates of the first initial point and the second initial point respectively, so as to obtain the third control point and the fourth control point corresponding to the pixel.

[0080] In step S25 above, the scheme of adjusting the coordinates of the first initial point and the second initial point respectively using the adjustment coefficient corresponding to each pixel can be adaptively set according to the actual needs of tone mapping of the image to be processed in the actual application scenario. For example, the adjustment scheme can be set to obtain the third control point and the fourth control point that meet the requirements based on the brightness increase requirement of the area with small brightness value (e.g., underexposed area or dark area) and the brightness decrease requirement of the area with large brightness value (e.g., bright area or overexposed area).

[0081] For example, in practical applications, the coordinates of the first and second initial points can be adjusted using, but not limited to, any of the following methods.

[0082] Method 1: Calculate the product of the adjustment coefficient corresponding to the pixel point and the x-coordinate and y-coordinate of the first initial point, and use it as the x-coordinate and y-coordinate of the third control point; calculate the product of the adjustment coefficient corresponding to the pixel point and the x-coordinate and y-coordinate of the second initial point, and use it as the x-coordinate and y-coordinate of the fourth control point.

[0083] Method 2: Calculate the sum of the adjustment coefficient corresponding to the pixel and the x and y coordinates of the first initial point, respectively, and use them as the x and y coordinates of the third control point; calculate the sum of the adjustment coefficient corresponding to the pixel and the x and y coordinates of the second initial point, respectively, and use them as the x and y coordinates of the fourth control point.

[0084] Method 3: Assign corresponding weights to the first and second initial points, denoted as the first weight and the second weight, respectively. Calculate the product of the adjustment coefficient corresponding to the pixel, the first weight, and the x and y coordinates of the first initial point, and use this product as the x and y coordinates of the third control point. Calculate the product of the adjustment coefficient corresponding to the pixel, the second weight, and the x and y coordinates of the second initial point, and use this product as the x and y coordinates of the fourth control point. The values ​​of the first and second weights can be appropriately set according to actual needs and are not limited here.

[0085] As mentioned above, the adjustment coefficient is limited to the interval [0, 1]. Therefore, by adopting the coordinate adjustment scheme of the first method, the x-coordinate and y-coordinate of the first and second initial points can be reduced proportionally. By adopting the coordinate adjustment scheme of the second method, the x-coordinate and y-coordinate of the first and second initial points can be increased by the same amount. By adopting the coordinate adjustment scheme of the third method, more flexible adjustment can be achieved according to the numerical settings of the first and second weights. It can achieve both proportional increase / decrease of the x-coordinate and y-coordinate of the first and second initial points and non-proportional increase / decrease.

[0086] It should be noted that, in order to ensure that the third and fourth control points corresponding to each pixel meet the target conditions, in addition to using... Figure 2 The solutions provided in each step determine the third and fourth control points, and other appropriate solutions may also be adopted.

[0087] For example, in one implementation, a consistent preset adjustment coefficient (and) can be directly applied to each pixel. Figure 2 The difference in the scheme shown is that, instead of determining the corresponding adjustment coefficient based on the local mean of the pixel, the first and second initial points are adjusted to obtain the third and fourth control points corresponding to the pixel.

[0088] For example, in another implementation, for each pixel, the adjustment coefficient can be disregarded, and the first and second initial points that satisfy the above slope conditions, determined manually or automatically by computer, can be directly used as the third and fourth control points.

[0089] Continue to refer to Figure 1 In the specific implementation of step S14, curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel. The first and second control points are respectively used as the endpoints of the tone mapping curve, and the third and fourth control points are used to control the curvature of the tone mapping curve.

[0090] In one specific implementation, the following Bézier curve fitting formula (e.g., the third-order Bézier curve fitting formula, abbreviated as formula (3)) can be used to perform curve fitting based on the four control points to obtain the tone mapping curve corresponding to the pixel.

[0091] B(t) = p0(bitmax - t) 3 +3p1t(bitmax-t) 2 +3p2t 2 (1-t)+p3t 3 ;

[0092] Where B(t) represents the tone mapping result of the pixel, p0, p1, p2, and p3 represent the coordinates of the first control point, the second control point, the third control point, and the fourth control point, respectively, bitmax represents the maximum bit width of the image to be processed, and t represents the brightness value of the pixel (for example, it can be the original brightness value or the updated brightness value).

[0093] It should be noted that, in addition to Bézier curves, other polynomial fitting curves or algorithms that can achieve the same or similar functions can also be used to obtain the tone mapping curve. For example, B-spline curves.

[0094] Reference Figure 3 , Figure 3 This is a schematic diagram of a tone mapping curve obtained based on fitting four control points in an embodiment of the present invention. The x-axis of the coordinate system to which the tone mapping curve belongs represents the brightness value of a pixel, and the y-axis represents the tone mapping result of the pixel (i.e., the brightness value after tone mapping).

[0095] Specifically, the x-coordinate and y-coordinate of the first control point A are both the low brightness value, i.e., A(Y) min Y min The x-coordinate and y-coordinate of the second control point B are both the high brightness value, i.e., B(Y). max Y max The third control point C and the fourth control point D satisfy the following objective conditions: the slope k1 of the straight line between the first control point A and the third control point C, and the slope k2 of the straight line between the first control point A and the fourth control point D are both greater than the slope k0 of the straight line between the first control point A and the second control point B.

[0096] Without limitation, the first control point A can be set as the origin of the coordinate system. In this case, the third control point C and the fourth control point D both satisfy the condition that the ordinate is greater than the abscissa.

[0097] Understandably, for the same frame of the image to be processed, with the first control point A and the second control point B fixed, the curvature and bending shape of the tone mapping curve fitted for each pixel are controlled by the third control point C and the fourth control point D. Specifically, the larger the ordinate value of the third control point C, the greater the increase in brightness value of pixels in low-brightness areas (i.e., dark or underexposed areas) of the image; conversely, the smaller the ordinate value of the fourth control point, the greater the decrease in brightness value of pixels in high-brightness areas (i.e., bright or overexposed areas) of the image. In other words, by controlling the positional changes of the third control point C and the fourth control point D, the curvature and bending shape of the tone mapping curve corresponding to the current pixel can be controlled accordingly.

[0098] In this embodiment of the invention, by selecting lower and higher brightness values ​​from the brightness values ​​of each pixel in the image to be processed, a first control point A and a second control point B are respectively determined as the two endpoints of the tone mapping curve. These two endpoints can be used to limit the minimum and maximum brightness values ​​after tone mapping, preventing the dark areas of the image from being too dark or the bright areas from being too bright. Furthermore, since the third control point C and the fourth control point D are used to control the curvature (or shape) of the tone mapping curve, this embodiment satisfies the aforementioned target condition (i.e., k1 and k2 are both greater than k0) by setting the third control point C and the fourth control point D. Figure 3 As can be seen from the curvature / shape change trend of the tone mapping curve, using this tone mapping curve to perform tone mapping on the image to be processed can increase the brightness value of pixels in darker areas of the image and appropriately reduce the brightness value of pixels in brighter areas, while avoiding the grayscale inversion problem in the tone-mapped image compared to the image to be processed.

[0099] Therefore, by adopting this implementation scheme, the image to be processed is tone-mapped to obtain a tone-mapped image. Since the brightness value of the dark area of ​​the image has been appropriately increased, the loss of image detail information can be minimized when the bit width reduction operation is subsequently performed on the tone-mapped image. That is, the detail information, high dynamic range, and high contrast of the image to be processed are preserved.

[0100] Furthermore, in determining the third control point C and the fourth control point D, the following condition can be set: the slope k1 of the straight line between the first control point A and the third control point C is greater than the slope k2 of the straight line between the first control point A and the fourth control point D. For example, the slope k1 of the straight line between the first control point A and the first initial point should be as close as possible to or equal to the slope of the y-axis, and the slope k2 of the straight line between the first control point A and the second initial point should be as close as possible to or equal to the slope k0 of the straight line between the first control point A and the second control point B. This is more conducive to accurately improving the brightness of overly dark areas of the image and accurately reducing the brightness of overly bright areas, further improving the tone mapping effect.

[0101] Furthermore, such as Figure 2 In one specific implementation described in the steps shown, for each pixel, the coordinates of its corresponding third control point C and fourth control point D can be determined by adjusting the adjustment coefficients based on the low brightness value, high brightness value, and the local average brightness of the pixel. However, in actual applications, since the local average brightness of different pixels may be different, the adjustment coefficients corresponding to different pixels may also be different, and the third control point C / fourth control point D corresponding to different pixels may also be different. Therefore, the tone mapping curves corresponding to different pixels may also be different.

[0102] Specifically, according to the formula for calculating the adjustment coefficient strength of each pixel, the larger the local average brightness of a pixel, the smaller the adjustment coefficient will be. Correspondingly, this means that the coordinate change rate of the third control point C and the fourth control point D is lower, and the curvature change of the corresponding tone mapping curve will be relatively smoother. Conversely, the smaller the local average brightness of a pixel, the larger the adjustment coefficient will be. Correspondingly, this means that the coordinate change rate of the third control point C and the fourth control point D is faster, and the curvature change of the corresponding tone mapping curve will be relatively greater.

[0103] Therefore, compared to determining the adjustment coefficient by using the entire image as a unit and the overall brightness mean of the image to be processed (e.g., the average brightness of all pixels in the image), this embodiment determines the corresponding adjustment coefficient, third control point C, and fourth control point D for each pixel based on its local mean. This allows each pixel to have a tone mapping curve adapted to its local brightness mean. Thus, more refined tone mapping is achieved, improving the accuracy of tone mapping.

[0104] Continue to refer to Figure 1 In the specific implementation of step S15, the tone mapping result of the pixel is determined based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel.

[0105] Reference Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation of step S15. In this embodiment, step S15 may include steps S41 to S42.

[0106] In step S41, the brightness value of the pixel is used as the horizontal coordinate value, and the corresponding vertical coordinate value is determined on the tone mapping curve corresponding to the pixel, which is used as the initial tone mapping result of the pixel.

[0107] In practice, tone mapping can be performed on the original or updated brightness value of the pixel, depending on the actual needs of the scenario.

[0108] In step S42, the difference between the brightness value of the pixel and the high brightness value is determined, and then the initial tone mapping result is adjusted using the difference to obtain the tone mapping result of the pixel.

[0109] Furthermore, the initial tone mapping result can be adjusted using the following formula (4):

[0110] RAW OUT =RAW1+(RAW0-Y) max )×ratio;

[0111] Among them, RAW OUT RAW1 represents the tone mapping result of this pixel, RAW0 represents the brightness value of this pixel, and Y represents the initial tone mapping result of this pixel. max The value represents the high brightness, and ratio represents the preset adjustment coefficient.

[0112] It should be noted that, in another conventional implementation, the brightness value of the pixel can be used as the horizontal axis value, and the corresponding vertical axis value can be determined on the tone mapping curve corresponding to the pixel, and the vertical axis value can be used as the final tone mapping result of the pixel.

[0113] Compared to the conventional tone mapping schemes described above, in this embodiment of the invention, by employing... Figure 4 The proposed scheme further adjusts the initial tone mapping result obtained directly from the tone mapping curve, thereby adjusting the image saturation and obtaining a processed image with higher contrast and color fidelity.

[0114] Furthermore, in one specific embodiment, the image to be processed may be a multi-exposure fusion image, and the method of determining the image to be processed may include steps S101 to S103.

[0115] In step S101, an image block to be fused is obtained. The image block includes one or more reference blocks and a set of blocks to be fused that match each reference block. Each set of blocks to be fused includes one or more blocks to be fused. The reference blocks are derived from a reference frame, and each block to be fused in the set of blocks to be fused is derived from a frame to be fused that has a different exposure duration than the reference frame.

[0116] It should be noted that the reference frame and the frame to be fused are both RAW images corresponding to the same scene but with different exposure durations.

[0117] In step S102, for each reference block, the fusion weight of the reference block and the matching block to be fused is determined based on the exposure time and brightness, and fusion is performed according to the fusion weight to obtain the fusion result of each reference block.

[0118] Further, step S102 may include: determining the fusion weight of the center points of each block in the reference block and its matching block to be fused, using pixels in the reference frame corresponding to the reference block as units, based on the exposure time and brightness; calculating the fusion value of the center point of the reference block based on the fusion weight of each block center point; and using the fusion value of the center point of the reference block as the fusion result of the reference block. That is, for each reference block in the reference frame, matching and fusion are performed by traversing from the first pixel in the upper left corner of the original image, i.e., the reference frame.

[0119] In step S103, a fused image is generated based on the fusion results of each reference block, and the fused image is used as the image to be processed.

[0120] Considering that the alignment and registration of some scene blocks may have certain deviations, especially for moving scenes, there may be significant alignment deviations. In order to make the solution more robust and obtain better fusion results in any scene, the image processing method in this embodiment of the invention can calculate the fusion weight based on the local statistical information of each block.

[0121] In specific implementation, the fusion weights include, but are not limited to, any one or more of the following: exposure weight, brightness weight, saturation weight, and difference weight.

[0122] The calculation of each of the above fusion weights is explained in detail below.

[0123] 1) Exposure weight calculation

[0124] The exposure weight of the center point of each block can be calculated as follows: the exposure weight of the block is determined based on the exposure duration and gain of the original image corresponding to each block, and the exposure weight of the block is used as the exposure weight of the center point of the block.

[0125] Considering the difference in bit width between the fused image and the original image, the bit width of the fused image can be used as the target bit width. The exposure weight refers to the weight required to adjust blocks with different exposure durations to the same brightness and the same target bit width.

[0126] Specifically, taking three original images as an example, they are divided into long exposure frames, medium exposure frames, and low exposure frames according to the exposure time.

[0127] Assuming the maximum exposure ratio is EXP max Insufficient long and short exposure ratio EXP max When the fused output needs to be multiplied by the analog gain to make the bit width of the fused image reach the target bit width, then:

[0128]

[0129]

[0130] Where, ratio l_s The ratio represents the exposure ratio between long-exposure frames and short-exposure frames. m_s Gain is the exposure ratio between medium exposure frames and short exposure frames. l and expo l These are the analog gain and exposure time of the long exposure frame, respectively. s and expo s These are the simulated gain and exposure time of the short exposure frame, respectively, with gain′ representing the calculated gain.

[0131] The exposure weights of long exposure frames, medium exposure frames, and short exposure frames are related to the exposure time and analog gain; the greater the exposure time and analog gain, the greater the weight.

[0132] Assuming a short exposure frame is used as the baseline, the exposure weight of each frame can be calculated:

[0133] w exp0 =EXP max (7)

[0134]

[0135]

[0136] w exp0 w expref w exp1 These are the exposure weights for short exposure frames, medium exposure frames, and long exposure frames, respectively.

[0137] The exposure weight of a short exposure frame is defined as EXP. max This indicates that the brightness of the three frames is aligned with that of the short exposure frame, that is, the short exposure frame is first increased to the target bit width (output frame bit width).

[0138] Of course, medium exposure frames or long exposure frames can also be used as brightness references to determine the exposure weight of each frame.

[0139] After determining the exposure weights of each original image (i.e., the reference frame) and each frame to be fused, the exposure weights of each reference block and each block to be fused can be obtained.

[0140] 2) Calculation of brightness weight

[0141] Specifically, for each block, the weighted average of the brightness values ​​of all pixels in each color channel in the block can be calculated to obtain the brightness value of that color channel; the overexposure weight is determined based on the maximum brightness value of each color channel; the underexposure weight is determined based on the minimum brightness value of each color channel; then the smaller value between the overexposure weight and the underexposure weight is determined, and the smaller value is normalized to obtain the brightness weight of the center point of the block.

[0142] For example, assuming the block size is 3x3, the weighted average of the brightness values ​​of all pixels in the four color channels of the block is as follows:

[0143]

[0144]

[0145]

[0146]

[0147] Where (cx, cy) represents the position of a pixel, cx can represent the horizontal coordinate of the pixel, cy can represent the vertical coordinate of the pixel, v represents the brightness value of the pixel at position (cx, cy), and the superscript numbers 0, 1, 2, 3 represent the four color channels of the RAW image.

[0148] The weighted average of the brightness values ​​of all pixels in each color channel calculated above is used as the brightness value of the corresponding color channel. Then, the maximum brightness value of the four color channels is used to calculate the overexposure weight, and the minimum brightness value of the four color channels is used to calculate the underexposure weight. Therefore:

[0149] Overexposure weight is:

[0150]

[0151] Underexposure weight is:

[0152]

[0153] Where maskP represents overexposure weight, maskN represents underexposure weight, and th max It is an adjustable brightness threshold used to determine whether the current brightness is overexposed. v represents the average brightness value of the four color channels. max This represents the maximum brightness value among the four color channels.

[0154] The smaller value between the overexposure weight and the underexposure weight is selected, and this smaller value is normalized to obtain the brightness weight of the center point of the block. That is, the brightness weight of the center point of the block is:

[0155] mask=min(maskP,maskN) (13)

[0156] w mask =exp(-(|mask|*sigma) mask ))(14)

[0157] Where mask represents the smaller of the overexposure weight and the underexposure weight, w mask This represents the brightness weight of the block center point obtained after normalizing this smaller value, sigma. mask This is the first configurable parameter.

[0158] Specifically, sigma mask You can refer to the brightness information of the current scene to configure it. The configuration principle can be that the higher the brightness value of the current scene, the higher the sigma. mask The smaller the value, the larger the value.

[0159] (3) Calculation of difference weights

[0160] Specifically, a weighted average is performed on the differences of all pixels in each color channel within the block to obtain the difference value of each color channel based on the pixel; the maximum difference value of each color channel is determined, and the maximum difference value is normalized to obtain the difference weight of the center point of the block.

[0161] Assuming the size of the block is 5x5, then for the center point of the block, the difference between the block and the reference block needs to be calculated within a 5x5 area of ​​the center point.

[0162] It should be noted that when calculating the difference, each block needs to be multiplied by the exposure ratio to bring it to the same brightness. Then, the difference of all pixels in each color channel is weighted and averaged to obtain the difference value of each color channel based on the pixel. Finally, the maximum value of the difference values ​​of the four color channels is taken.

[0163] Specifically, as shown in the following formula (15):

[0164] d pix =max{diff0,diff1,diff2,diff3}

[0165] diff0 = (I 22 *4+(I 02 +I 20 +I 24 +I 42 )*2+(I 00 +I 04 +I 40 +I 44 )) / 16

[0166] diff1=((I 21 +I 23 )*2+(I 01 +I 03 +I 41 +I 43 )) / 8

[0167] diff2=((I 12 +I 32 )*2+(I 10 +I 30 +I 14 +I 34 )) / 8

[0168] diff3 = (I 11 +I 13 +I 31 +I 33 ) / 4

[0169] Where, d pix The maximum value among the four color channel differences is denoted by diff, which represents the difference value of a single color channel based on a pixel. The subscripts 0, 1, 2, and 3 of diff represent the color channel numbers, respectively. I = ref_pix - cur_pix represents the difference between the area to be fused and the reference area corresponding to the pixel pix. The subscript number of I represents the coordinates of the pixel pix within a 5x5 range.

[0170] After calculating the difference values ​​of each color channel, the maximum difference value among the four color channels is taken, and this maximum difference value is normalized to obtain the difference weight of the block center point:

[0171] w diff =exp(-(|d pix |*sigma pix (16)

[0172] Among them, w diff The difference weight of the block center point is represented by sigma. pix This is the second configurable parameter. Specifically, sigma pix You can refer to the brightness information of the current scene to configure it. The configuration principle can be that the higher the brightness value of the current scene, the higher the sigma. pix The smaller the value, the larger the value.

[0173] (4) Saturation weight calculation

[0174] Specifically, based on the brightness values ​​of each pixel in the block, the R-channel value, G-channel value, and B-channel value of the center point of the block are determined; the maximum and minimum values ​​among the R-channel value, G-channel value, and B-channel value of the center point of the block are determined, and the average value of the maximum and minimum values ​​is determined; the difference between the maximum value and the average value is used as the saturation weight of the center point of the block.

[0175] Regarding the R-channel value R for determining the center point of the block i,j G channel value G i,j and the B channel value B i,j For specific solutions, please refer to the description of formula (1) and related content in the previous text, which will not be repeated here.

[0176] The maximum value (rgb_max) and minimum value (rgb_min) of the three channels are determined using the following formula, and the saturation weight W_satu is calculated:

[0177] rgb_max = max(R i,j G i,j B i,j (17)

[0178] rgb_min=min(R i,j G i,j B i,j (18)

[0179] W _satu =rgb max -(rgb max +rgb_min) / 2 (19)

[0180] Accordingly, after determining the weights of the center points of the reference block and each matching block to be fused, the reference block and each matching block to be fused are fused according to the corresponding weights of the center points. Taking three original images (medium exposure frame, short exposure frame, and long exposure frame) as an example, the fusion formula for each reference block is as follows:

[0181]

[0182] Where the subscripts ref, 0, and 1 represent medium exposure frames, short exposure frames, and long exposure frames, respectively, v represents the center point brightness value, and w di It is the difference weight between medium exposure frames and short exposure frames, w diff1 It represents the difference weights between medium-exposure frames and long-exposure frames, and combine is the fusion result of the current reference block.

[0183] For the block sequence to be merged, the center point of each reference block is sequentially calculated with the center point of the block to be merged that matches it according to the above formula (20) to obtain the final fusion result of the reference block.

[0184] It should be noted that in practical applications, when performing fusion calculations on different blocks, one or more of the above-mentioned exposure weights, brightness weights, saturation weights, and difference weights can be considered. Moreover, the calculation of various weights is not limited to the above-mentioned calculation methods, and other calculation methods can also be used. For example, when only considering brightness weights and difference weights for fusion, it is necessary to consider the differences caused by the different exposure times of each image. In other words, the exposure time information needs to be fused into the difference weight.

[0185] The image processing method provided in this embodiment of the invention, for an image block to be fused, sequentially for each reference block, using the reference block as a reference, determines the fusion weight of the reference block and the matching fusion block according to the exposure time and brightness, and performs fusion according to the fusion weight to obtain the fusion result of each reference block; generates a fused image based on the fusion result of each reference block, thereby realizing the fusion of multiple frames of images with different exposure times corresponding to the same scene, and obtaining better image quality.

[0186] Furthermore, in performing multi-exposure fusion, the solution of this embodiment introduces the exposure weight, brightness weight, saturation weight and difference weight of each image to perform local fusion weight judgment, which can obtain a fusion result with high dynamic range, and at the same time effectively prevent the occurrence of fusion ghosting.

[0187] It should be noted that, in this embodiment of the invention, the image blocks to be fused are obtained by processing multiple frames of original images taken for the same scene with different exposure times. Specifically, one frame is selected from the original images as a reference frame, and the other image frames are selected as frames to be fused. The reference frame is divided into multiple reference blocks, and correspondingly, each frame to be fused is divided into multiple frames to be fused of the same size. Then, after alignment and registration, the region to be fused corresponding to each reference block is determined.

[0188] Existing technologies can be used to segment the original image. For example, in a non-limiting embodiment, each frame of the original image can be divided into uniform squares, and the size of the blocks can be set according to the original size of the image.

[0189] Since the image processing method of this embodiment of the invention fuses different blocks based on pixel units, in order to facilitate the uniformity of the calculation method for the fusion weight of the center point of each block, reduce the complexity of subsequent calculations and the accuracy of each weight, in another non-limiting embodiment provided by this invention, the original image can be first subjected to edge-rolling processing, and then the blocks can be divided according to the edge-rolled image.

[0190] In specific implementation, the method of dividing the original image into blocks may include the following steps S201 to S205.

[0191] In step S201, multiple frames of original images with different exposure durations corresponding to the same scene are acquired.

[0192] For example, the images could be three frames with different exposure times obtained through continuous shooting, and could be images captured by a standard image sensor or an HDR image sensor; this embodiment of the invention does not limit the scope. For instance, it could be acquiring three low dynamic range RAW images with different exposure times.

[0193] In step S202, one frame is selected from the multiple original images as a reference frame, and the other image frames outside the reference frame are selected as frames to be fused.

[0194] For example, three images with different exposure durations are given: a short exposure image, a medium exposure image, and a long exposure image. The medium exposure image is selected as the reference frame, and the short exposure image and the long exposure image are used as the frames to be fused. Of course, either the short exposure image or the long exposure image can be selected as the frame to be fused; this embodiment of the invention does not limit this choice.

[0195] For example, the frame with the best imaging quality can be selected from the multiple original images as a reference frame. The image with the best imaging quality can be selected based on the current shooting scene. For example, if the shooting scene is during the day, a short-exposure image can be selected as the reference frame; or if the shooting scene is at night, a short-exposure image can be selected as the reference frame.

[0196] In step S203, each original image is mirrored and its edges are traced according to the set block size to obtain the traced image corresponding to the original image.

[0197] When expanding the edges of the original image, the size of the blocks to be divided must first be determined. For example, the block size can be set to a×a, where a represents the number of pixels.

[0198] Because the fusion of blocks in this invention is calculated based on the fusion weights of the center point, the value of 'a' can preferably be an odd number. Accordingly, after setting the size of the block to a×a, the edges of each original image are mirrored, so that any pixel on any edge of the original image can be used as the center point of an a×a block in the corresponding mirrored image.

[0199] For example, if the block size is 3×3, then each side of the original image needs to be expanded by 1 unit; if the block size is 7×7, then each side of the original image needs to be expanded by 3 units.

[0200] In step S204, the edge-tracing image is divided into blocks according to the block size to obtain each block of the edge-tracing image.

[0201] In step S205, the block corresponding to the reference frame is used as the reference block, and the region corresponding to each frame to be fused is used as the fused block. Based on the reference block, the image to be fused is aligned and registered with the reference image to obtain a set of fused blocks that match each reference block.

[0202] Alignment and registration methods are based on blocks and can employ traditional ORB feature-based methods, optical flow methods, multi-scale alignment algorithms, and other algorithms.

[0203] After alignment and registration, for each parameter block in the reference frame, a matching region can be found in the frame to be fused, that is, each reference region corresponds to a set of matching blocks to be fused.

[0204] The image processing method provided in this embodiment of the invention can perform fusion processing on images captured in various different scenarios to obtain better image effects.

[0205] Furthermore, for certain shooting scenarios, such as motion scenes, when a medium-exposure frame is used as the reference frame, motion ghosting occurs in the overexposed area of ​​the reference frame. As can be seen from the above formulas (13) and (14), an underexposed frame will be used in this area. Therefore, in another non-limiting embodiment of the present invention, the above... Figure 2 The fused image obtained in the illustrated embodiment is adjusted. Specifically, the fused image can be adjusted using the reference frame according to the shooting scene. For example, if the scene corresponding to the reference frame is a moving scene, the fused image is supplemented using the reference frame.

[0206] Specifically, the backfill can be performed using the following formula:

[0207]

[0208] w add=exp(-(|add) diff |*sigma add ))

[0209] combine22 = combine*w add +v ref *(1-w add ) (twenty one)

[0210] Among them, v ref It is the brightness value of the center point of the reference frame, add diff The fusion result is multiplied by the exposure ratio of the medium and short exposures, and then the difference value is taken with the reference frame. `combine22` is the final fusion result. add Sigma represents the intermediate coefficient. add This indicates the third configurable parameter.

[0211] Specifically, sigma add You can refer to the brightness information of the current scene to configure it. The configuration principle can be that the higher the brightness value of the current scene, the higher the sigma. add The smaller the value, the larger the value.

[0212] Since the brightness of the three frames is aligned with the short exposure frame at the beginning of the fusion process, it is not difficult to find that the overall brightness of the first fusion result is consistent with that of the short exposure frame.

[0213] By adding back the fused image as described above, the problem of ghosting in moving images can be effectively reduced.

[0214] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention. The image processing device may include:

[0215] The image to be processed determination module 51 is used to determine the image to be processed and the brightness value of each pixel.

[0216] The high and low brightness value determination module 52 is used to select two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, which are respectively denoted as low brightness value and high brightness value, wherein the high brightness value is greater than the low brightness value;

[0217] The control point determination module 53 is used to determine four control points corresponding to each pixel of the image to be processed, denoted as the first control point, the second control point, the third control point, and the fourth control point. The horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal and vertical coordinates of the second control point are both the high brightness value. The slopes of the straight lines between the first and third control points and between the first and fourth control points are both greater than the slope of the straight line between the first and second control points.

[0218] The tone mapping curve determination module 55 is used to perform curve fitting based on the four control points to obtain the tone mapping curve corresponding to the pixel point, wherein the first control point and the second control point are respectively used as the endpoints of the tone mapping curve, and the third control point and the fourth control point are used to control the curvature of the tone mapping curve.

[0219] The tone mapping module 55 is used to determine the tone mapping result of the pixel based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel.

[0220] In specific implementations, the aforementioned image processing device may correspond to a chip with image processing capabilities; or to a chip module with image processing capabilities in a terminal; or to a terminal itself.

[0221] For details regarding the principles, implementation, and beneficial effects of this image processing device, please refer to the preceding text. Figures 1 to 4 The descriptions of the image processing methods shown are not repeated here.

[0222] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the above-described... Figures 1 to 4 The steps of the image processing method are shown. The computer-readable storage medium may include non-volatile or non-transitory memory, and may also include optical discs, hard disk drives, solid-state drives, etc.

[0223] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0224] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0225] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor runs the computer program, it performs the above-described... Figures 1 to 4The steps of the image processing method are shown. The terminal may include, but are not limited to, mobile phones, computers, tablets, and other terminal devices, and may also be servers, cloud platforms, etc.

[0226] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0229] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0230] In the embodiments of this application, "multiple" refers to two or more.

[0231] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0232] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0233] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. An image processing method, characterized in that, include: Determine the image to be processed and the brightness value of each pixel; Two brightness values ​​are selected from the brightness values ​​of each pixel in the image to be processed, and are denoted as low brightness value and high brightness value, respectively, wherein the high brightness value is greater than the low brightness value; For each pixel in the image to be processed, four control points corresponding to that pixel are determined and denoted as the first control point, the second control point, the third control point, and the fourth control point. The horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal and vertical coordinates of the second control point are both the high brightness value. The slopes of the lines between the first and third control points and between the first and fourth control points are both greater than the slope of the line between the first and second control points. Curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel. The first and second control points are respectively used as the endpoints of the tone mapping curve, and the third and fourth control points are used to control the curvature of the tone mapping curve. The tone mapping result of the pixel is determined based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel.

2. The method according to claim 1, characterized in that, For each pixel in the image to be processed, determine the corresponding third control point and fourth control point, including: Calculate the average brightness value of each pixel within a preset area centered on the pixel, and determine the local average brightness value of the pixel based on the average value, the low brightness value, and the high brightness value; determine the difference between the high brightness value and the local average brightness value, denoted as the first difference value, and determine the difference between the high brightness value and the low brightness value, denoted as the second difference value; Determine the ratio of the first difference to the second difference, and the adjustment coefficient corresponding to that pixel. Select a first initial point and a second initial point respectively, such that the slope of the straight line between the first control point and the first initial point, and the slope of the straight line between the first control point and the second initial point are both greater than the slope of the straight line between the first control point and the second control point. Using the adjustment coefficient corresponding to the pixel, the coordinates of the first initial point and the second initial point are adjusted respectively to obtain the third control point and the fourth control point corresponding to the pixel.

3. The method according to claim 2, characterized in that, Determining the local average brightness of the pixel based on the average value, the low brightness value, and the high brightness value includes: If the average value is greater than or equal to the high brightness value, then the high brightness value is taken as the local average brightness value of the pixel. If the average value is less than or equal to the low brightness value, then the low brightness value is taken as the local average brightness value of the pixel. If the average value is greater than the low brightness value and less than the high brightness value, then the average value is taken as the local average brightness value of the pixel.

4. The method according to claim 2, characterized in that, The step of using the adjustment coefficient corresponding to the pixel to adjust the coordinates of the first initial point and the second initial point respectively to obtain the third control point and the fourth control point corresponding to the pixel includes: Calculate the product of the adjustment coefficient corresponding to the pixel point and the x-coordinate and y-coordinate of the first initial point, and use it as the x-coordinate and y-coordinate of the third control point; The product of the adjustment coefficient corresponding to the pixel and the x-coordinate and y-coordinate of the second initial point is calculated respectively, and used as the x-coordinate and y-coordinate of the fourth control point.

5. The method according to any one of claims 1 to 4, characterized in that, The slope of the straight line between the first control point and the third control point is greater than the slope of the straight line between the first control point and the fourth control point.

6. The method according to claim 1, characterized in that, Two brightness values ​​are selected from the brightness values ​​of each pixel in the image to be processed, and are denoted as low brightness value and high brightness value, respectively, including: Among the brightness values ​​of each pixel in the image to be processed, the brightness value that is Nth in order of increasing brightness from the minimum brightness value is determined as the low brightness value, and the brightness value that is Mth in order of decreasing brightness from the maximum brightness value is determined as the high brightness value. Where M and N are both positive integers, and both M and N are less than 1 / 2 of the total number of pixels in the image to be processed.

7. The method according to claim 6, characterized in that, The step of determining the Nth brightness value sorted from the minimum brightness value in ascending order as the low brightness value, and determining the Mth brightness value sorted from the maximum brightness value in descending order as the high brightness value, includes: By using multiple brightness value ranges, histogram statistics are performed on the brightness values ​​of each pixel in the image to be processed to obtain histogram statistics results; Starting from the smallest brightness value range, the number of pixels is accumulated in ascending order of brightness value until the total number of accumulated pixels reaches N. Then, the brightness value of the Nth pixel is determined as the low brightness value. Starting from the range with the highest brightness value, the number of pixels is accumulated in descending order of brightness value until the total number of accumulated pixels reaches M. Then, the brightness value of the Mth pixel is determined as the high brightness value.

8. The method according to claim 7, characterized in that, The multiple brightness value ranges are divided based on the bit width of the image to be processed; Among them, the smaller the brightness value in a brightness value range, the smaller the width of that brightness value range; and the larger the brightness value in a brightness value range, the larger the width of that brightness value range.

9. The method according to claim 1, characterized in that, Based on the four control points, curve fitting is performed to obtain the tone mapping curve corresponding to the pixel, including: Using the Bezier curve fitting formula, curve fitting is performed based on the four control points to obtain the tone mapping curve corresponding to the pixel.

10. The method according to claim 1, characterized in that, Before selecting two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, the method further includes: For each pixel in the image to be processed, the R channel value, G channel value, and B channel value of the pixel are determined based on the brightness values ​​of each pixel in a preset area surrounding the pixel as the center point. Update the brightness value of the pixel to the maximum value among the R channel value, G channel value, and B channel value.

11. The method according to claim 1 or 10, characterized in that, Based on the brightness value of the pixel and the tone mapping curve corresponding to the pixel, the tone mapping result of the pixel is determined, including: Using the brightness value of the pixel as the horizontal axis value, determine the corresponding vertical axis value on the tone mapping curve corresponding to the pixel, and use it as the initial tone mapping result of the pixel. The difference between the brightness value of the pixel and the high brightness value is determined, and then the initial tone mapping result is adjusted using the difference to obtain the tone mapping result of the pixel.

12. The method according to claim 11, characterized in that, The initial tone mapping result is adjusted using the following formula: RAW OUT = RAW1 + (RAW0 - Y max ) x ratio; wherein RAW OUT represents the tone mapping result of the pixel point, RAW1 represents the initial tone mapping result of the pixel point, RAW0 represents the luminance value of the pixel point, Y max represents the high luminance value, and ratio represents a preset adjustment coefficient.

13. The method according to claim 1, characterized in that, The process of determining the image to be processed includes: Obtain image blocks to be merged, the image blocks including one or more reference blocks, and a set of merge blocks matching each reference block, each set of merge blocks including one or more merge blocks, wherein the reference blocks are derived from reference frames, and each merge block in the set of merge blocks is derived from a merge frame with a different exposure duration than the reference frame; For each reference block, the fusion weights of the reference block and its matching fusion block are determined based on the exposure time and brightness, and fusion is performed according to the fusion weights to obtain the fusion result of each reference block. A fused image is generated based on the fusion results of each reference block, and the fused image is used as the image to be processed.

14. The method according to claim 13, characterized in that, Its features are, The step of determining the fusion weight of the reference block and the matching block to be fused based on the exposure time and brightness includes: Using the pixels in the reference frame corresponding to the reference block as the unit, the fusion weight of the center point of each block in the reference block and the block to be fused that matches it is determined according to the exposure time and brightness. The process of fusing according to the fusion weights to obtain the fusion results for each reference block includes: The fusion value of the reference block center point is calculated based on the fusion weight of each block center point; The fusion value of the center point of the reference block is used as the fusion result of the reference block.

15. The method according to claim 14, characterized in that, The fusion weights include any one or more of the following: exposure weight, brightness weight, saturation weight, and difference weight.

16. The method according to claim 15, characterized in that, Determining the exposure weight of each block's center point includes: The exposure weight of each block is determined based on the exposure duration and gain of the original image corresponding to each block. The exposure weight of the block is used as the exposure weight of the block's center point.

17. The method according to claim 15, characterized in that, Determining the brightness weight of the center point of each block includes: Calculate the weighted average of the brightness values ​​of all pixels in each color channel within the block to obtain the brightness value of each color channel; The overexposure weight is determined based on the maximum brightness value of each color channel; The underexposure weight is determined based on the minimum brightness value of each color channel; The smaller of the overexposure weight and the underexposure weight is determined, and the smaller value is normalized to obtain the brightness weight of the center point of the block.

18. The method according to claim 15, characterized in that, Determining the saturation weights of the center points of each block includes: Based on the brightness values ​​of each pixel in the block, determine the R channel value, G channel value, and B channel value of the center point of the block; Determine the maximum and minimum values ​​among the R-channel, G-channel, and B-channel values ​​at the center point of the block, and determine the average of these maximum and minimum values; The difference between the maximum value and the average value is used as the saturation weight of the block center point.

19. The method according to claim 15, characterized in that, Determining the difference weights for the center points of each block includes: The weighted average of the differences of all pixels in each color channel in the block is used to obtain the difference value of each color channel based on the pixel; The maximum difference value of each color channel is determined, and the maximum difference value is normalized to obtain the difference weight of the center point of the block.

20. The method according to any one of claims 13 to 19, characterized in that, The method further includes: acquiring multiple original images corresponding to the same scene but with different exposure durations; One frame is selected from the multiple original images as a reference frame, and the other image frames outside the reference frame are the frames to be fused. Based on the set block size, each original image is mirrored and its edges are traced to obtain the traced image corresponding to the original image; The edge-tracing image is divided into blocks according to the block size to obtain each block of the edge-tracing image; the block corresponding to the reference frame is taken as the reference block, and the block corresponding to each frame to be fused is taken as the fused block. Based on the reference block, the fused blocks are aligned and registered with the reference block to obtain a set of fused blocks that match each reference block.

21. An image processing apparatus, characterized in that, include: The image to be processed determination module is used to determine the image to be processed and the brightness value of each pixel. The high and low brightness value determination module is used to select two brightness values ​​from the brightness values ​​of each pixel in the image to be processed, which are respectively denoted as low brightness value and high brightness value, wherein the high brightness value is greater than the low brightness value; The control point determination module is used to determine four control points corresponding to each pixel of the image to be processed, denoted as the first control point, the second control point, the third control point, and the fourth control point. The horizontal and vertical coordinates of the first control point are both the low brightness value, and the horizontal and vertical coordinates of the second control point are both the high brightness value. The slopes of the lines between the first and third control points and between the first and fourth control points are both greater than the slope of the line between the first and second control points. The tone mapping curve determination module is used to perform curve fitting based on the four control points to obtain the tone mapping curve corresponding to the pixel point, wherein the first control point and the second control point are respectively used as the endpoints of the tone mapping curve, and the third control point and the fourth control point are used to control the curvature of the tone mapping curve. The tone mapping module is used to determine the tone mapping result of a pixel based on its brightness value and the tone mapping curve corresponding to that pixel.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the image processing method according to any one of claims 1 to 20.

23. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the image processing method according to any one of claims 1 to 20.

Images