Multi-exposure image processing method, device and noise reduction circuit

Abstract

The embodiments of the present application disclose a multi-exposure image processing method, device and noise reduction circuit. When the method processes the multi-exposure images of HDR photography, the size of the pixel window used for performing noise reduction processing on at least two exposure images is determined according to the exposure time, and the noise reduction processing is performed on the at least two exposure images according to the determined pixel window size, thereby meeting the noise reduction processing requirements of different exposure images and effectively suppressing image noise. On the basis of ensuring the noise reduction effect, it saves processing resources, reduces the noise reduction cost, and optimizes the HDR noise reduction scheme.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a multi-exposure image processing method, device and noise reduction circuit. Background Art

[0002] There are many ways to achieve High Dynamic Range (HDR) photography. The most mainstream method is Digital Overlap (DOL), which is to take long, medium, and short exposures of the same scene, collect three images, and then fuse the three images to get a high dynamic range image. When fusing the three images, it is necessary to consider the impact of the noise in the three images on the fusion results, and generally the images need to be denoised. When implementing noise reduction, you can choose to do it before fusion or after fusion. Therefore, the noise reduction problem of multi-exposure images is a problem that requires long-term research and solution. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application hope to provide a multi-exposure image processing method, device and noise reduction circuit.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, a multi-exposure image processing method is provided, comprising:

[0006] Acquire at least two exposure images of the target scene captured at different exposure times;

[0007] Based on the exposure times of the at least two exposure images, respectively determining the sizes of pixel windows for performing noise reduction processing on the at least two exposure images;

[0008] performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window;

[0009] The at least two exposure images after the noise reduction process are fused to obtain a high dynamic range image of the target scene.

[0010] In a second aspect, a multi-exposure image processing device is provided, comprising:

[0011] An acquisition module is used to acquire at least two exposure images of a target scene captured at different exposure times;

[0012] A noise reduction module, configured to determine, based on the exposure time of the at least two exposure images, the sizes of pixel windows used to perform noise reduction processing on the at least two exposure images; and perform noise reduction processing on the at least two exposure images based on the determined sizes of the pixel windows;

[0013] The fusion module is used to fuse the at least two exposure images after noise reduction processing to obtain a high dynamic range image of the target scene.

[0014] In a third aspect, a noise reduction circuit is provided, the noise reduction circuit comprising:

[0015] A pixel window construction circuit, used for loading a pixel window of the exposure image according to the size of the pixel window for performing noise reduction processing on the exposure image;

[0016] An odd-numbered row extraction circuit, used for extracting odd-numbered row pixels from the pixel window loaded by the window construction circuit;

[0017] An even-numbered row extraction circuit, used for extracting even-numbered row pixels from the pixel window loaded by the window construction circuit;

[0018] A pixel matrix construction circuit, used for constructing a pixel matrix according to the odd-numbered rows of pixels or the even-numbered rows of pixels;

[0019] The noise reduction operation circuit is used to perform noise reduction operation on the pixel matrix and output noise-reduced pixel points.

[0020] In a fourth aspect, a multi-exposure image processing chip is provided, wherein the chip includes the noise reduction circuit described in the third aspect.

[0021] In a fifth aspect, an electronic device is provided, comprising: an image acquisition device and the aforementioned multi-exposure image processing chip.

[0022] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0023] In this way, when processing multi-exposure images of HDR photography, the sizes of the pixel windows used to perform noise reduction processing on at least two exposure images are determined according to the exposure time, and the noise reduction processing is performed on the at least two exposure images according to the determined pixel window sizes, thereby meeting the noise reduction processing requirements of different exposure images and effectively suppressing image noise. On the basis of ensuring the noise reduction effect, processing resources are saved, the noise reduction cost is reduced, and the HDR noise reduction scheme is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1Schematic diagram of the process of multi-exposure image processing method in an embodiment of the present application;

[0025] Figure 2 Schematic diagram of the noise reduction process in the embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of a first size of a pixel window in an embodiment of the present application;

[0027] Figure 4 This is a first schematic diagram of the Gr dot pixel matrix in an embodiment of the present application;

[0028] Figure 5 This is a first schematic diagram of the Gb point pixel matrix in an embodiment of the present application;

[0029] Figure 6 This is a first schematic diagram of the R-point pixel matrix in an embodiment of the present application;

[0030] Figure 7 This is a first schematic diagram of the pixel matrix at point B in an embodiment of the present application;

[0031] Figure 8 This is a schematic diagram of the second size of the pixel window in the embodiment of the present application;

[0032] Fig. 9 This is a second schematic diagram of the Gr dot pixel matrix in the embodiment of the present application;

[0033] Fig.10 This is a second schematic diagram of the Gb point pixel matrix in an embodiment of the present application;

[0034] Fig.11 This is a second schematic diagram of the R-point pixel matrix in an embodiment of the present application;

[0035] Fig.12 This is a second schematic diagram of the pixel matrix at point B in an embodiment of the present application;

[0036] Fig.13 This is a schematic diagram of the structure of a multi-exposure image processing device in an embodiment of the present application;

[0037] Fig.14 Schematic diagram of the structure of the noise reduction circuit in the embodiment of the present application;

[0038] Fig.15 This is a schematic diagram of the first component structure of the noise reduction operation circuit in an embodiment of the present application;

[0039] Fig.16 This is a schematic diagram of a second component structure of the noise reduction operation circuit in an embodiment of the present application;

[0040] Fig.17Schematic diagram of the composition structure of the multi-exposure image processing chip in the embodiment of the present application;

[0041] Fig.18 It is a schematic diagram of the composition structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0043] The following is a detailed example of a multi-exposure image processing method provided in an embodiment of the present application. Figure 1 FIG. 1 is a flow chart of a multi-exposure image processing method in an embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0044] Step 101: acquiring at least two exposure images of a target scene captured at different exposure times;

[0045] Step 102: Based on the exposure time of the at least two exposure images, respectively determine the size of the pixel window used to perform noise reduction processing on the at least two exposure images;

[0046] Exemplarily, the size of the pixel window includes width and height, and determining the size of the pixel window for performing noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images includes: making at least one of the width and the height of the pixel window proportional to the exposure time.

[0047] That is, the longer the exposure time, the larger the size of the pixel window. Exemplarily, the pixel window size for the first exposure image is configured as the first width*the first height. The pixel window size for the second exposure image is configured as the second width*the second height. When the first exposure time is less than the second exposure time, the first width is less than the second width, and / or the first height is less than the second height.

[0048] Step 103: performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window;

[0049] In some embodiments, performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window includes: performing noise reduction processing on pixel points in a central window within the pixel window based on the determined size of the pixel window.

[0050] Here, the size of the center window is 2 (width) x 2 (height), and the center window includes 4 pixel points RGrGbB, which are the points that need to be denoised in the current pixel window. Further, the method also includes: after the denoising of the four Bayer format pixel points in the center window is completed, the pixel window is moved horizontally by two pixels to obtain a new pixel window; and the new pixel window is subjected to denoising until the denoising of all pixel points of the exposure image is completed.

[0051] In HDR photography, considering that the noise of exposure images taken at different exposure times is different, and different exposure images contribute differently to the HDR image, the embodiment of the present application configures pixel windows of different sizes for different exposure images, uses a larger pixel window for noise reduction operations for images with greater noise interference, and uses a smaller pixel window for noise reduction operations for images with less noise interference. Different processing resources are allocated to different exposure images, and on the basis of ensuring the noise reduction effect, reasonable allocation of processing resources is achieved, thereby improving the utilization rate of processing resources and reducing the noise reduction cost.

[0052] Step 104: performing fusion processing on the at least two exposure images after the noise reduction processing to obtain a high dynamic range image of the target scene.

[0053] In some embodiments, the at least two exposure images include a first exposure image captured at a first exposure time and a second exposure image captured at a second exposure time, and the method further includes: acquiring a third exposure image captured at a third exposure time of the target scene, wherein the third exposure time is greater than the first exposure time and greater than the second exposure time; and fusing the first exposure image after noise reduction processing, the second exposure image after noise reduction processing, and the third exposure image without noise reduction processing to obtain a high dynamic range image of the target scene.

[0054] That is to say, for multiple exposure images in HDR photography, noise reduction operation can be performed on each exposure image, and then the exposure images after noise reduction operation can be fused; or noise reduction operation can be performed on part of the exposure images in the multiple exposure images, while noise reduction operation is not performed on the other part of the exposure images, and then the exposure images after noise reduction operation and the exposure images without noise reduction operation can be fused.

[0055] For example, three exposure times are often used in HDR photography to obtain three exposure images, a short exposure image, a medium exposure image, and a long exposure image, which correspond to a first exposure time, a second exposure time, and a third exposure time, respectively, wherein the first exposure time is less than the second exposure time, and the second exposure time is less than the third exposure time. The denoising method provided in the embodiment of the present application can be used to perform denoising operations on the short exposure image and the medium exposure image, and no denoising operation is performed on the long exposure image, and the denoising operation is directly fused with the denoised images of the other two images to obtain an HDR image.

[0056] By adopting the above technical solution, when processing multi-exposure images of HDR photography, the sizes of pixel windows used to perform noise reduction processing on at least two exposure images are determined according to the exposure time, and noise reduction processing is performed on at least two exposure images according to the determined sizes of pixel windows, thereby meeting the noise reduction processing requirements of different exposure images and effectively suppressing image noise. On the basis of ensuring the noise reduction effect, processing resources are saved, noise reduction costs are reduced, and the HDR noise reduction solution is optimized.

[0057] Based on the above embodiment, taking the exposure image as a Bayer format image as an example, the noise reduction operation is further illustrated. For example, Figure 2 As shown, performing noise reduction processing on the at least two exposure images based on the pixel window includes:

[0058] Step 201: for a first pixel point among four Bayer format pixel points in a central window of the pixel window, construct a first pixel matrix using all the first pixel points in the pixel window;

[0059] Here, the exposure image is a Bayer format image. The so-called Bayer format is a pixel arrangement method named after the inventor Bayer. First, an image is divided into many 2x2 blocks. Each 2x2 block contains a blue block (Blue, which can be abbreviated as B), a red block (Red, which can be abbreviated as R) and two green blocks (Green, which can be abbreviated as G), of which the two green blocks can only be located in diagonal positions. Generally, in order to distinguish the two green blocks, the green block in the same row as the blue block is called Gb, and the green block in the same row as the red block is called Gr. Gb and Gr can be considered to be different colors. All possible Bayer formats in the 2x2 block include four modes: RGrGbB, BGbGrR, GrRBGb and GbBRGr. The central window of this application is a 2x2 block at the center of the pixel window. The four Bayer format pixels are illustrated using RGrGbB as an example, and the other three formats are also applicable.

[0060] The first pixel point is a portion of pixel points in the central window. Exemplarily, the first pixel point is a green point Gr point or a green point Gb point among the four Bayer format pixel points.

[0061] For example, Figure 3 As shown, the noise reduction operation of the mid-exposure image of the HDR camera is performed with a pixel window of 14 (width) x 10 (height) in Bayer format. The four Bayer format pixel points RGrGbB in the center window are the points that need to be denoised in the current pixel window, and other non-center points can be used as noise reduction reference points to participate in the noise reduction operation.

[0062] When the noise reduction operation is performed on the green point Gr in the center window, it is performed on the first pixel matrix composed of 7 (width) x 5 (height) Gr points centered on Gr, such as Figure 4 shown.

[0063] When the noise reduction operation is performed on the green point Gb in the center window, it is performed on the first pixel matrix composed of 7 (width) x 5 (height) Gb points centered on Gb, such as Figure 5 shown.

[0064] Step 202: performing the noise reduction operation on the first pixel matrix to obtain a first pixel point after noise reduction in the central window;

[0065] Exemplarily, the noise reduction operation may include taking an average value of points in the pixel matrix as the first pixel point after noise reduction, or filtering the points in the pixel matrix and taking an average value as the first pixel point after noise reduction to ensure the noise reduction effect.

[0066] In some embodiments, the noise reduction operation includes: first taking the absolute value of each pixel point in the pixel matrix, and then taking the square root to obtain the square root result sqrt_abs_cen of the center point of the pixel matrix and the square root result sqrt_abs_ref of the non-center point of the pixel matrix; subtracting the center point sqrt_abs_cen from each non-center point sqrt_abs_ref and taking the absolute value to obtain the difference ABSDIFF; determining the target non-center point and the number of target non-center points whose ABSDIFF is less than a first threshold; accumulating all the target non-center points and dividing them by the number of target non-center points to obtain the noise reduction operation result.

[0067] Exemplarily, the first threshold may be a square root result sqrt_abs_cen of the center point, or a threshold obtained by calibrating sqrt_abs_cen according to configuration parameters.

[0068] In some embodiments, the method further includes: obtaining configured weights and offsets; multiplying the square root result sqrt_abs_cen of the center point by the weight to obtain a product; and adding the product and the offset to obtain the first threshold.

[0069] For example, Figure 4 The steps for denoising the Gr points are as follows:

[0070] Step 1: For each Gr point in the 7x5 pixel matrix, take the absolute value first, and then take the square root. For the convenience of presentation, the square root result of the center Gr point is marked as sqrt_abs_cenGr, and the square root results of other Gr points are sqrt_abs_Gr;

[0071] Step 2: Calculate a threshold thresh_cenGr based on the sqrt_abs_cenGr value of the center Gr point. This threshold will be involved in subsequent calculations. The value of thresh_cenGr is equal to sqrt_abs_cenGr*reg_weight+reg_offset, where reg_weight and reg_offset are parameters configured in the register;

[0072] Step 3: Subtract the sqrt_abs_cenGr of the center Gr point from the 34 sqrt_abs_Gr values ​​to get 34 differences, and then take the absolute value of these 34 differences. The calculation result is named ABSDIFF. We get a total of 34 ABSDIFF values;

[0073] Step 4: Compare the 34 ABSDIFF values ​​with the threshold thresh_cenGr respectively, pick out the points whose ABSDIFF values ​​are less than the threshold thresh_cenGr, and add up these points to get a cumulative sum.

[0074] Step 5: Divide the accumulated sum by the number of pixels involved in the accumulation to get the average value of the pixels involved in the accumulation. Use this average value to replace the center Gr point to complete the noise reduction.

[0075] The noise reduction process is the same as that of Gr points. Figure 5 The steps for noise reduction operation of Gb points are as follows:

[0076] Step 1: For each Gb point in the 7x5 pixel matrix, take the absolute value first, and then take the square root. For the convenience of presentation, the square root result of the center Gb point is marked as sqrt_abs_cenGb, and the square root results of other Gb points are marked as sqrt_abs_Gb;

[0077] Step 2: Calculate a threshold thresh_cenGb based on the sqrt_abs_cenGb value of the center point Gb. This threshold will be involved in subsequent calculations. The value of thresh_cenGb is equal to sqrt_abs_cenGb*reg_weight+reg_offset, where reg_weight and reg_offset are parameters configured in the register.

[0078] Step 3: Subtract the sqrt_abs_cenGb of the center Gb point from the 34 sqrt_abs_Gb values ​​to get 34 differences, then take the absolute value of these 34 differences, and the calculation result is named ABSDIFF. We get a total of 34 ABSDIFF values;

[0079] Step 4: Compare the 34 ABSDIFF values ​​with the threshold thresh_cenGb respectively, pick out the points whose ABSDIFF values ​​are less than the threshold thresh_cenGb, and add up these points to get a cumulative sum.

[0080] Step 5: Divide the accumulated sum by the number of pixels involved in the accumulation to get the average value of the pixels involved in the accumulation. Use this average value to replace the center point Gb to complete the noise reduction.

[0081] Step 203: for a second pixel point among the four Bayer format pixel points in the central window of the pixel window, use all the second pixel points in the pixel window and the pixel points adjacent in the row to form a pixel pair;

[0082] The second pixel point is another part of the pixel points in the central window. Exemplarily, the second pixel point is a red point R or a blue point B among the four Bayer format pixel points.

[0083] The noise reduction of the red point among the four points in the center of the pixel window is based on all the pixels in the row where the red point is located, with a total of 14 (width) x 5 (height) points, such as Figure 6 As shown, R and Gr in adjacent rows can be considered as a pair, and there are a total of 7x5 such "pixel pairs".

[0084] The noise reduction of the blue point among the four points in the center of the pixel window is based on all the pixels in the row where the blue point is located, with a total of 14 (width) x 5 (height) points, such as Figure 7 As shown, B and Gb in adjacent rows can be considered as a pair, and there are a total of 7x5 such "pixel pairs".

[0085] Step 204: sum each pixel pair and construct a second pixel matrix using the sum value; find the difference between each pixel pair and construct a third pixel matrix using the difference value;

[0086] Step 205: performing the noise reduction operation on the second pixel matrix and the third pixel matrix to obtain a sum value and a difference value after noise reduction;

[0087] Here, the second pixel matrix and the third pixel matrix are subjected to noise reduction processing respectively. In some embodiments, the noise reduction operation includes: taking the absolute value of each pixel point in the pixel matrix first, and then taking the square root, to obtain the square root result sqrt_abs_cen of the center point of the pixel matrix and the square root result sqrt_abs_ref of the non-center point of the pixel matrix; subtracting the center point sqrt_abs_cen from each non-center point sqrt_abs_ref and taking the absolute value to obtain the difference ABSDIFF; determining the target non-center point and the number of target non-center points whose ABSDIFF is less than the first threshold; accumulating all the target non-center points and dividing by the number of the target non-center points to obtain the noise reduction operation result.

[0088] For example, Figure 6 The noise reduction steps at point R are as follows:

[0089] Step 1: Since R and Gr always appear in pairs, sum up 35 pairs of R and Gr to get 35 (R+Gr) values. At this time, (R+Gr) is regarded as a pixel, so there are 35 such pixels in total, which is exactly the same as the pixel matrix of Gr and Gb denoising. Using the same denoising operation, we can get the denoised (R+Gr);

[0090] Step 2: Since R and Gr always appear in pairs, the differences of 35 pairs of R and Gr are calculated to obtain 35 (R-Gr) values. At this time, (R-Gr) is regarded as a pixel, so there are 35 such pixels in total, which is exactly the same as the pixel matrix of Gr and Gb denoising. Using the same denoising operation, the denoised (R-Gr) can be obtained;

[0091] Step 3: Average the two values ​​after noise reduction (R+Gr) obtained in step 1 and (R-Gr) obtained in step 2, and replace the central point R with the calculated result to complete the noise reduction.

[0092] right Figure 7 The noise reduction steps at point B are as follows:

[0093] Step 1: Since B and Gb always appear in pairs, sum up 35 pairs of B and Gb respectively to get 35 (B+Gb) values. At this time, (B+Gb) is regarded as one pixel, so there are 35 such pixels in total, which is exactly the same as the color matrix of Gr and Gb denoising. Using the same algorithm, we can get the denoised (B+Gb);

[0094] Step 2: Since B and Gb always appear in pairs, the differences of 35 pairs of B and Gb are calculated to obtain 35 (B-Gb) values. At this time, (B-Gb) is regarded as a pixel, so there are 35 such pixels in total, which is exactly the same as the color matrix of Gr and Gb denoising. Using the same algorithm, the denoised (B-Gb) can be obtained;

[0095] Step 3: Average the denoised (B+Gb) obtained in step 1 and the denoised (B-Gb) obtained in step 2 again, and replace the central point B with the calculated result to complete the denoising.

[0096] It should be noted that the noise reduction processing of the four points can be performed simultaneously or in sequence.

[0097] Step 206: averaging the sum and difference after noise reduction to obtain the second pixel point after noise reduction in the central window.

[0098] In some embodiments, the method further includes: after the noise reduction of the four Bayer format pixels in the center window is completed, the pixel window is moved two pixels in the horizontal direction to obtain a new pixel window; and the new pixel window is subjected to noise reduction processing until the noise reduction of all the pixels of the exposure image is completed. Starting from the leftmost side of the exposure image, the above operation is repeated until the noise reduction of the four rightmost points in the horizontal direction is completed, and the 14x10 pixel window is translated downward by two pixels and returned to the leftmost side of the image to start the noise reduction of a new row. Repeating the above noise reduction operation can complete the noise reduction of all the pixels of the entire frame image.

[0099] Considering that the contribution of short exposure images to HDR images is smaller than that of medium exposure images, a simpler noise reduction processing strategy can be used for short exposure images, that is, by reducing the pixel window size, reducing the amount of calculation, simplifying the noise reduction circuit, and thus reducing the noise reduction cost. Exemplarily, when the first exposure image is a short exposure image, the pixel window size can be 6 (width) x 10 (height). When the second exposure image is a medium exposure image, the pixel window size can be 14 (width) x 10 (height).

[0100] For example, Figure 8 As shown, the denoising operation for the short exposure image of the HDR camera is performed with a pixel window of 6 (width) x 10 (height) in Bayer format. The four Bayer format pixel points RGrGbB in the center window are the points that need to be denoised in the current pixel window, and other non-center points can be used as denoising reference points to participate in the denoising operation.

[0101] The noise reduction of the green point Gr among the four points in the center of the short exposure image pixel window is performed with a pixel matrix consisting of 3 (width) x 5 (height) Gr points centered on Gr, as shown in Fig. 9 The steps of the noise reduction operation on the Gr point 3x5 pixel matrix can refer to the noise reduction operation of the Gr point in the exposure image.

[0102] The noise reduction of the green point Gb is performed on a pixel matrix consisting of 3 (width) x 5 (height) Gr points centered on Gb, such as Fig.10 The steps of the noise reduction operation on the 3x5 pixel matrix of the Gb point can refer to the noise reduction operation of the Gb point in the middle exposure image.

[0103] The noise reduction of the red point among the four points in the center of the short exposure image pixel window is based on all the pixels in the row where the red point is located, with a total of 6 (width) x 5 (height) points, such as Fig.11 As shown, the adjacent R and Gr in the row can be regarded as a pair, and there are 3x5 such "pixel pairs" in total. The steps of the 3x5 pixel matrix noise reduction operation can refer to the noise reduction operation of the R point in the exposure image.

[0104] The noise reduction of the blue point among the four points in the center of the short exposure image pixel window is based on all the pixels in the row where the blue point is located, with a total of 6 (width) x 5 (height) points, such as Fig.12 As shown, the B and Gb in adjacent rows can be considered as a pair, and there are 3x5 such "pixel pairs" in total. The steps of the denoising operation of the 3x5 pixel matrix can refer to the denoising operation of point B in the exposed image.

[0105] Similarly, the noise reduction of the four points in the center of the short exposure pixel window is completed. Next, the 6x10 pixel window is horizontally shifted to the right by two pixels in the short exposure image to construct a new 6x10 pixel window. The noise reduction method described above is used to start noise reduction of the four points in the center of the new 6x10 pixel window. Repeat the above operation until the noise reduction of the four points on the far right is completed. The 6x10 pixel window is shifted down by two pixels and returned to the far left of the image to start noise reduction of a new row. Repeat the above noise reduction operation to complete the noise reduction of all pixels in the entire frame image.

[0106] It should be noted that if it is necessary to perform noise reduction processing on the long exposure image, the same noise reduction calculation method as that for the short exposure image and the medium exposure image can be used. It is only necessary to configure the corresponding pixel window size according to the noise reduction requirements of the long exposure image.

[0107] By adopting the above technical solution, when processing multi-exposure images of HDR photography, the sizes of pixel windows used to perform noise reduction processing on at least two exposure images are determined according to the exposure time, and noise reduction processing is performed on at least two exposure images according to the determined sizes of pixel windows, thereby meeting the noise reduction processing requirements of different exposure images and effectively suppressing image noise. On the basis of ensuring the noise reduction effect, processing resources are saved, noise reduction costs are reduced, and the HDR noise reduction solution is optimized.

[0108] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a multi-exposure image processing device, such as Fig.13 As shown, the multi-exposure image processing device includes:

[0109] An acquisition module 131 is used to acquire at least two exposure images of a target scene captured at different exposure times;

[0110] A noise reduction module 132 is used to determine the size of a pixel window for performing noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images; and perform noise reduction processing on the at least two exposure images based on the determined size of the pixel window;

[0111] The fusion module 133 is used to fuse the at least two exposure images after the noise reduction process to obtain a high dynamic range image of the target scene.

[0112] Exemplarily, in some embodiments, the noise reduction module 132 includes:

[0113] A pixel window construction circuit 141, used for loading a pixel window of an exposure image according to a size of the pixel window;

[0114] An odd-row extraction circuit 142, used to extract odd-row pixels from the pixel window loaded by the window construction circuit;

[0115] An even-numbered row extraction circuit 143, used for extracting even-numbered row pixels from the pixel window loaded by the window construction circuit;

[0116] A pixel matrix construction circuit 144, configured to construct a pixel matrix according to the odd-numbered rows of pixels or the even-numbered rows of pixels;

[0117] The noise reduction operation circuit 145 is used to perform noise reduction operation on the pixel matrix and output noise-reduced pixel points.

[0118] A pixel matrix construction submodule, for constructing a first pixel matrix using all the first pixel points in the pixel window for a first pixel point among four Bayer format pixel points in a central window of the pixel window;

[0119] A denoising operation submodule, configured to perform the denoising operation on the first pixel matrix to obtain a first pixel point of the central window after denoising;

[0120] The pixel matrix construction submodule is further used to form a pixel pair using all the second pixel points in the pixel window and the pixel points adjacent in a row for the second pixel point in the four Bayer format pixels in the central window of the pixel window; sum each pixel pair and construct a second pixel matrix using the sum value; and differ each pixel pair and construct a third pixel matrix using the difference value;

[0121] The denoising operation submodule is used to perform the denoising operation on the second pixel matrix and the third pixel matrix to obtain the sum and difference after denoising; and to average the sum and difference after denoising to obtain the second pixel point after denoising of the central window.

[0122] In some embodiments, the first pixel point is a green dot Gr or a green dot Gb among the four Bayer format pixel points, and the second pixel point is a red dot R or a blue dot B among the four Bayer format pixel points.

[0123] In some embodiments, the noise reduction operation submodule is specifically used to first take the absolute value of each pixel point in the pixel matrix, and then take the square root to obtain the square root result sqrt_abs_cen of the center point of the pixel matrix and the square root result sqrt_abs_ref of the non-center point of the pixel matrix; subtract the center point sqrt_abs_cen from each non-center point sqrt_abs_ref and take the absolute value to obtain the difference ABSDIFF; determine the target non-center point and the number of target non-center points whose ABSDIFF is less than the first threshold; accumulate all the target non-center points and divide them by the number of target non-center points to obtain the noise reduction operation result.

[0124] Exemplarily, in some embodiments, the noise reduction operation submodule is also used to obtain the configured weight and offset; multiply the square root result sqrt_abs_cen of the center point by the weight to obtain the product; add the product and the offset to obtain the first threshold.

[0125] Exemplarily, in some embodiments, the noise reduction module 132 is also used to complete the noise reduction of four Bayer format pixels in the center window, move the pixel window by two pixels in the horizontal direction to obtain a new pixel window; and perform noise reduction on the new pixel window until the noise reduction of all pixels of the exposed image is completed.

[0126] Exemplarily, in some embodiments, the size of the pixel window includes width and height, and the noise reduction module 132 is further configured to make at least one of the width and height of the pixel window proportional to the exposure time.

[0127] Exemplarily, in some embodiments, the at least two exposure images include a first exposure image captured at a first exposure time and a second exposure image captured at a second exposure time, and the acquisition module 131 is used to acquire a third exposure image captured at a third exposure time of the target scene, wherein the third exposure time is greater than the first exposure time and greater than the second exposure time;

[0128] The fusion module 133 is used to fuse the first exposure image after the noise reduction processing, the second exposure image after the noise reduction processing, and the third exposure image without the noise reduction processing to obtain a high dynamic range image of the target scene.

[0129] Based on the exposure image noise reduction method in the above-mentioned multi-exposure image processing method, the embodiment of the present application further provides a noise reduction circuit, exemplarily, as Fig.14 As shown, the noise reduction circuit includes:

[0130] A pixel window construction circuit 141 is used to load the pixel window of the exposure image according to the size of the pixel window for performing noise reduction processing on the exposure image;

[0131] An odd-row extraction circuit 142, used to extract odd-row pixels from the pixel window loaded by the window construction circuit;

[0132] An even-numbered row extraction circuit 143, used for extracting even-numbered row pixels from the pixel window loaded by the window construction circuit;

[0133] A pixel matrix construction circuit 144, configured to construct a pixel matrix according to the odd-numbered rows of pixels or the even-numbered rows of pixels;

[0134] The noise reduction operation circuit 145 is used to perform noise reduction operation on the pixel matrix and output noise-reduced pixel points.

[0135] In some embodiments, the noise reduction circuit further includes a row buffer 146 for caching pixels of H-1 rows of pixel windows, where H is the height of the pixel window. When the Hth row of pixels is acquired, the pixel window construction circuit 141 loads the complete pixel window.

[0136] In some embodiments, the pixel matrix construction circuit 144 includes: a Gr dot pixel matrix construction circuit, an R dot pixel matrix construction circuit, a B dot pixel matrix construction circuit, and a Gb dot pixel matrix construction circuit.

[0137] like Figure 3 As shown, R point and Gr point are located in even rows, Gb point and B point are located in odd rows, and the Gr point pixel matrix construction circuit and the R point pixel matrix construction circuit construct the Gr point pixel matrix (i.e., the first pixel matrix) and the R point pixel matrix according to the even row pixels output by the even row extraction circuit 143.

[0138] The B point pixel matrix construction circuit and the Gb point pixel matrix construction circuit construct the Gb point pixel matrix (ie, the first pixel matrix) and the B point pixel matrix according to the odd row pixels output by the odd row extraction circuit 142 .

[0139] Correspondingly, the noise reduction operation circuit 145, the Gr point noise reduction operation circuit, the R point noise reduction operation circuit, the B point noise reduction operation circuit and the Gb point noise reduction operation circuit perform noise reduction operations on the four points respectively.

[0140] In some embodiments, the noise reduction circuit further includes an output circuit 147 for arranging the output order of the four points after noise reduction and outputting the same.

[0141] When the pixel window size is 14 (width) x10 (height), the noise reduction circuit includes: a 9-row buffer and a 14x10 pixel window construction circuit. The 9-row buffer is used to buffer the first 9 rows of pixels. When the 10th row of pixels is received, the pixel window construction circuit constructs a 14x10 pixel window. The odd-row extraction circuit is used to extract the 14x5 pixel matrix, the Gr point pixel matrix construction circuit extracts the 7x5 pixel matrix, and the R point pixel matrix construction circuit directly obtains the 14x5 pixel matrix; the even-row extraction circuit is used to extract the 14x5 pixel matrix, the B point pixel matrix construction circuit extracts the 14x5 pixel matrix, and the Gb point pixel matrix construction circuit directly obtains the 3x5 pixel matrix.

[0142] When the pixel window size is 6 (width) x10 (height), the noise reduction circuit includes: 9-row buffer and 6x10 pixel window construction circuit. The odd-row extraction circuit is used to extract the 6x5 pixel matrix, the Gr point pixel matrix construction circuit is used to extract the 3x5 pixel matrix, and the R point pixel matrix construction circuit is used to directly obtain the 6x5 pixel matrix; the even-row extraction circuit is used to extract the 6x5 pixel matrix, the B point pixel matrix construction circuit is used to extract the 6x5 pixel matrix, and the Gb point pixel matrix construction circuit is used to directly obtain the 3x5 pixel matrix.

[0143] For example, Fig.15 As shown, the noise reduction operation circuit 145 includes:

[0144] An absolute value taking subcircuit 151, used for taking an absolute value of each point in the pixel matrix;

[0145] The square root subcircuit 152 is used to square the absolute value of each point in the pixel matrix to obtain a square root result sqrt_abs_cen of the center point of the pixel matrix and a square root result sqrt_abs_ref of the non-center point of the pixel matrix;

[0146] The absolute value subcircuit 153 is used for subtracting the central point sqrt_abs_cen from each non-central point sqrt_abs_ref and taking the absolute value to obtain the difference ABSDIFF;

[0147] A comparison subcircuit 154, configured to compare the ABSDIFF with a preset threshold value, and determine the target non-center points and the number of target non-center points whose ABSDIFF is less than the first threshold value;

[0148] The first averaging subcircuit 155 is used to accumulate all the target non-center points and divide the accumulated sum by the number of the target non-center points to obtain the pixel point after noise reduction.

[0149] In some embodiments, the noise reduction operation circuit 145 further includes: a threshold configuration subcircuit 156, configured to configure the first threshold according to the square root result sqrt_abs_cen of the center point. Exemplarily, the first threshold is equal to sqrt_abs_cenGr*reg_weight+reg_offset, where reg_weight and reg_offset are weights and offsets configured by registers.

[0150] For example, Fig.16 As shown, when performing noise reduction operation on the second pixel point, the noise reduction operation circuit 145 further includes: a pixel pair summing subcircuit 157 and a pixel pair difference subcircuit 158,

[0151] The pixel pair summing subcircuit 157 is used to sum the pixel pairs with adjacent rows in the pixel matrix output by the pixel matrix construction circuit to obtain a second pixel matrix, and input the sum to the absolute value subcircuit;

[0152] The pixel pair difference subcircuit 158 ​​is used to find the difference of the pixel pairs with adjacent rows in the pixel matrix output by the pixel matrix construction circuit to obtain a third pixel matrix and input it to the absolute value subcircuit.

[0153] Since R and Gr always appear in pairs, 35 pairs of R and Gr are summed up to get 35 (R+Gr) values. At this time, (R+Gr) is regarded as a pixel, so there are 35 such pixels in total, which is exactly the same as the pixel matrix of Gr and Gb denoising. Using the same denoising operation, we can get the denoised (R+Gr).

[0154] Since R and Gr always appear in pairs, the differences of 35 pairs of R and Gr are calculated to obtain 35 (R-Gr) values. At this time, (R-Gr) is regarded as a pixel, so there are 35 such pixels in total, which is exactly the same as the pixel matrix of Gr and Gb denoising. Using the same denoising operation, the denoised (R-Gr) can be obtained.

[0155] Middle and Fig.15 The method is the same as the above, except that a second averaging subcircuit 159 is added at the end to average the sum and difference output by the first averaging subcircuit to obtain the second pixel point of the central window after noise reduction.

[0156] Exemplarily, the Gr noise reduction operation circuit and the Gb noise reduction operation circuit can be Fig.15 The noise reduction operation circuit shown, the R noise reduction operation circuit and the B noise reduction operation circuit can be Fig.16 The noise reduction operation circuit shown.

[0157] In practical applications, the noise reduction module in the multi-exposure image processing device can be implemented by the above-mentioned noise reduction circuit.

[0158] The present application also provides a multi-exposure image processing chip, such as Fig.17 As shown, the chip 170 includes any one of the noise reduction circuits 1701 in the above embodiments.

[0159] Optionally, the multi-exposure image processing chip 170 may further include an input interface 1702. The input interface 1702 communicates with other devices or chips, and specifically, may acquire information or data sent by other devices or chips.

[0160] Optionally, the multi-exposure image processing chip 170 may further include an output interface 1703. The output interface 1703 communicates with other devices or chips, and specifically, may output information or data to other devices or chips.

[0161] Optionally, the multi-exposure image processing chip 170 may be applied to the electronic device in the embodiment of the present application.

[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0163] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0164] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0165] An embodiment of the present application also provides an electronic device. The electronic device described in the present application has a shooting function and can shoot exposure images with different exposure times. The electronic device may include, for example, a mobile phone, a tablet computer, a laptop computer, a PDA, a personal digital assistant (PDA), a portable media player (PMP), a wearable device, a camera, a smart car, etc.

[0166] like Fig.18 As shown, the electronic device 180 includes: an image acquisition device 1801 and the aforementioned multi-exposure image processing chip 1802 .

[0167] Exemplarily, the image acquisition device 1801 is used to acquire at least two exposure images of a target scene captured at at least two exposure times;

[0168] The multi-exposure image processing chip 1802 is any multi-exposure image processing chip in the above embodiments, and is used to perform noise reduction processing on images with different exposures by adopting different noise reduction processing strategies.

[0169] Optionally, the multi-exposure image processing chip 1803 is also used to fuse the exposure images after noise reduction.

[0170] Of course, in actual application, as shown in 18, the various components in the electronic device 180 are coupled together through the bus system 1803. It can be understood that the bus system 1803 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1803 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.18 Various buses are labeled as bus system 1803.

[0171] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor to complete the steps of the aforementioned method.

[0172] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0173] Optionally, the computer program product can be applied to the processor in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the processor in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0174] The embodiment of the present application also provides a computer program.

[0175] Optionally, the computer program can be applied to the processor in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the processor in the various methods in the embodiments of the present application. For the sake of brevity, they are not described here.

[0176] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0177] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0178] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0179] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0180] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0181] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0182] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A multi-exposure image processing method, characterized in that: The method comprises: Acquire at least two exposure images of the target scene captured at different exposure times; Based on the exposure times of the at least two exposure images, respectively determining the sizes of pixel windows for performing noise reduction processing on the at least two exposure images; performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window; Performing fusion processing on the at least two exposure images after noise reduction processing to obtain a high dynamic range image of the target scene; The at least two exposure images are Bayer format images, and the noise reduction processing performed on the at least two exposure images based on the pixel window includes: for a second pixel point among the four Bayer format pixel points in the central window of the pixel window, using all the second pixel points in the pixel window and the pixel points adjacent to the pixel point in the row to form a pixel pair; summing each pixel pair, and constructing a second pixel matrix using the sum value; finding the difference for each pixel pair, and constructing a third pixel matrix using the difference value; performing noise reduction operation on the second pixel matrix and the third pixel matrix to obtain the sum value and difference value after noise reduction; averaging the sum value and difference value after noise reduction to obtain the second pixel point of the central window after noise reduction.

2. The multi-exposure image processing method according to claim 1, wherein: The performing noise reduction processing on the at least two exposure images based on the pixel window also includes: For a first pixel point among four Bayer format pixel points in a central window of the pixel window, construct a first pixel matrix using all the first pixel points in the pixel window; A noise reduction operation is performed on the first pixel matrix to obtain a first pixel point of the central window after noise reduction.

3. The multi-exposure image processing method according to claim 2, wherein: The first pixel point is a green point Gr or a green point Gb among the four Bayer format pixel points. The second pixel point is a red point R or a blue point B among the four Bayer format pixel points.

4. The multi-exposure image processing method according to claim 2 or 3, characterized in that: The noise reduction operation includes: Take the absolute value of each pixel in the pixel matrix first, then take the square root, and get the square root result sqrt_abs_cen of the center point of the pixel matrix and the square root result sqrt_abs_ref of the non-center point of the pixel matrix; Subtract the center point sqrt_abs_cen from each non-center point sqrt_abs_ref and take the absolute value to get the difference ABSDIFF; Determine the target non-center points and the number of target non-center points whose ABSDIFF is less than a first threshold; All target non-center points are accumulated and divided by the number of target non-center points to obtain the noise reduction calculation result.

5. The multi-exposure image processing method according to claim 4, characterized in that: The method further comprises: Get the configured weights and offsets; Multiply the square root result sqrt_abs_cen of the center point by the weight to obtain a product; The product and the offset are added to obtain the first threshold.

6. The multi-exposure image processing method according to claim 2 or 3, characterized in that: The method further comprises: After the noise reduction of the four Bayer format pixels in the center window is completed, the pixel window is moved two pixels in the horizontal direction to obtain a new pixel window; The new pixel window is subjected to noise reduction processing until noise reduction of all pixel points of the exposure image is completed.

7. The multi-exposure image processing method according to any one of claims 1 to 3, characterized in that: The size of the pixel window includes a width and a height, and the determining, based on the exposure time of the at least two exposure images, the size of the pixel window for performing noise reduction processing on the at least two exposure images includes: At least one of a width and a height of the pixel window is made proportional to the exposure time.

8. The multi-exposure image processing method according to any one of claims 1 to 3, characterized in that: The at least two exposure images include a first exposure image captured at a first exposure time and a second exposure image captured at a second exposure time, and the method further includes: Acquire a third exposure image of the target scene captured at a third exposure time, wherein the third exposure time is greater than the first exposure time and greater than the second exposure time; The first exposure image after the noise reduction processing, the second exposure image after the noise reduction processing, and the third exposure image without the noise reduction processing are fused to obtain a high dynamic range image of the target scene.

9. A multi-exposure image processing device, characterized in that: include: An acquisition module is used to acquire at least two exposure images of a target scene captured at different exposure times; A noise reduction module, configured to determine, based on the exposure times of the at least two exposure images, the sizes of pixel windows for performing noise reduction processing on the at least two exposure images; Performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window; A fusion module, used for fusing the at least two exposure images after noise reduction processing to obtain a high dynamic range image of the target scene; The at least two exposure images are Bayer format images, and the noise reduction module is specifically used to form a pixel pair using all the second pixel points in the pixel window and the pixel points adjacent in a row for a second pixel point in the four Bayer format pixels in the central window of the pixel window; sum each pixel pair, and construct a second pixel matrix using the sum value; Find the difference of each pixel pair and construct a third pixel matrix using the difference values; Performing a noise reduction operation on the second pixel matrix and the third pixel matrix to obtain a sum value and a difference value after noise reduction; The sum and difference values ​​after noise reduction are averaged to obtain the second pixel point after noise reduction in the central window.

10. A noise reduction circuit, characterized in that: The noise reduction circuit comprises: A pixel window construction circuit, used for loading a pixel window of the exposure image according to the size of the pixel window for performing noise reduction processing on the exposure image; An odd-numbered row extraction circuit, used for extracting odd-numbered row pixels from the pixel window loaded by the pixel window construction circuit; An even-numbered row extraction circuit, used for extracting even-numbered row pixels from the pixel window loaded by the pixel window construction circuit; A pixel matrix construction circuit, used for constructing a pixel matrix according to the odd-numbered rows of pixels or the even-numbered rows of pixels; A noise reduction operation circuit, used for performing noise reduction operation on the pixel matrix and outputting noise-reduced pixel points; When performing noise reduction operation on the second pixel point, the noise reduction operation circuit includes: an absolute value subcircuit, a pixel pair summing subcircuit and a pixel pair difference subcircuit, the absolute value subcircuit is used to take the absolute value of each point in the pixel matrix; the pixel pair summing subcircuit is used to sum the pixel pairs in the pixel matrix output by the pixel matrix construction circuit, to obtain a second pixel matrix, and input it to the absolute value subcircuit; the pixel pair difference subcircuit is used to differ the pixel pairs in the pixel matrix output by the pixel matrix construction circuit, to obtain a third pixel matrix, and input it to the absolute value subcircuit.

11. The noise reduction circuit according to claim 10, characterized in that: The noise reduction operation circuit comprises: A square root subcircuit, used for taking the square root of the absolute value of each point in the pixel matrix to obtain a square root result sqrt_abs_cen of the center point of the pixel matrix and a square root result sqrt_abs_ref of a non-center point of the pixel matrix; The absolute value subcircuit is used to subtract the center point sqrt_abs_cen from each non-center point sqrt_abs_ref and take the absolute value to obtain the difference ABSDIFF; A comparison subcircuit, configured to compare the ABSDIFF with a preset threshold value, and determine the target non-center point and the number of the target non-center points whose ABSDIFF is less than the first threshold value; The first averaging subcircuit is used to accumulate all the target non-center points and divide the accumulated sum by the number of the target non-center points to obtain the pixel points after noise reduction.

12. The noise reduction circuit according to claim 11, characterized in that: The noise reduction operation circuit also includes: The threshold configuration subcircuit is used to configure the first threshold according to the square root result sqrt_abs_cen of the center point.

13. The noise reduction circuit according to claim 11, characterized in that: The noise reduction operation circuit further includes: a second averaging subcircuit, which is used to average the sum and difference output by the first averaging subcircuit to obtain a second pixel point after noise reduction in the central window of the pixel window.

14. A multi-exposure image processing chip, characterized in that: The chip comprises the noise reduction circuit as described in any one of claims 10-13.

15. An electronic device, characterized in that: The electronic device comprises: an image acquisition device and the multi-exposure image processing chip as claimed in claim 14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images