Imaging device, imaging method, and electronic apparatus

Abstract

The present disclosure relates to an imaging device, an imaging method, and an electronic apparatus that make it possible to acquire a higher-quality HDR composite image. The present disclosure relates to an imaging device, an imaging method, and an electronic apparatus that make it possible to acquire a higher-quality HDR composite image. An HDR composite processing unit performs HDR composition processing for calculating composite data in which long exposure pixel data and short exposure pixel data are composited, thereby generating an HDR composite image in which a dynamic range is enlarged. A motion region HDR composite correction processing unit performs, on the composite data in a region in which a subject's motion is present in an HDR composite image, a motion region correction process for calculating composite correction data corrected by using the short exposure pixel data, thereby generating an HDR composite correction image in which deterioration in image quality due to the subject's motion is suppressed. The present technology can be applied to, for example, an imaging device that acquires an HDR composite image.

Description

Imaging device, imaging method, and electronic device

[0001] The present disclosure relates to an imaging device, an imaging method, and an electronic device, and more particularly to an imaging device, an imaging method, and an electronic device that are capable of acquiring an HDR composite image with higher image quality.

[0002] Conventionally, a technology has been developed to obtain a high dynamic range (HDR) composite image with a wide dynamic range and minimal blown-out highlights or crushed shadows by combining pixel data obtained from pixels set to long-time exposure and pixel data obtained from pixels set to short-time exposure.

[0003] For example, Patent Document 1 discloses an imaging device that efficiently encodes a RAW image using a first image representing high-frequency components for short exposure, a second image representing low-frequency components for short exposure, a third image representing low-frequency components for long exposure, and a fourth image representing high-frequency components for long exposure. Patent Document 2 also discloses an imaging device that generates a high dynamic range image by correcting blur of long-exposure pixels based on the amount of misalignment between multiple images.

[0004] JP 2021-184575 A JP 2015-142342 A

[0005] However, with conventional HDR composite images, when capturing moving subjects, artifacts, motion blur, color bleeding, and other issues can occur, resulting in a decrease in image quality. Therefore, there has been a demand for a way to suppress these issues and obtain HDR composite images with higher image quality.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to acquire an HDR composite image with higher image quality.

[0007] An imaging device according to one aspect of the present disclosure includes an HDR synthesis processing unit that generates an HDR synthetic image with an expanded dynamic range by performing HDR synthesis processing to calculate synthetic data that combines long-exposure pixel data and short-exposure pixel data, and a motion area HDR synthesis correction processing unit that generates an HDR synthetic correction image in which degradation in image quality due to subject movement is suppressed by performing motion area correction processing to calculate synthetic correction data that corrects the synthetic data in areas of the HDR synthetic image where the subject is moving, using the short-exposure pixel data.

[0008] An imaging method according to one aspect of the present disclosure includes generating an HDR composite image with an expanded dynamic range by performing HDR synthesis processing to calculate synthetic data obtained by synthesizing long-exposure pixel data and short-exposure pixel data, and generating an HDR composite corrected image in which degradation in image quality due to movement of the subject is suppressed by performing motion area correction processing to calculate synthetic corrected data obtained by correcting the synthetic data in an area of ​​the HDR composite image where the subject is moving, using the short-exposure pixel data.

[0009] An electronic device according to one aspect of the present disclosure includes an imaging device having an HDR synthesis processing unit that generates an HDR synthetic image with an expanded dynamic range by performing HDR synthesis processing to calculate synthetic data that combines long-exposure pixel data and short-exposure pixel data, and a motion area HDR synthesis correction processing unit that generates an HDR synthetic correction image in which degradation in image quality due to subject movement is suppressed by performing motion area correction processing to calculate synthetic correction data that corrects the synthetic data in areas of the HDR synthetic image where the subject is moving, using the short-exposure pixel data.

[0010] In one aspect of the present disclosure, an HDR composite image with an expanded dynamic range is generated by performing HDR synthesis processing to calculate synthetic data that combines long-exposure pixel data and short-exposure pixel data, and a motion area correction processing to calculate synthetic correction data that corrects synthetic data in areas of the HDR composite image where there is subject movement using short-exposure pixel data, thereby generating an HDR composite correction image that suppresses image quality degradation due to subject movement.

[0011] 1 is a block diagram illustrating an example configuration of an embodiment of an imaging device to which the present technology is applied. FIG. 1 is a diagram illustrating settings of long exposure and short exposure. FIG. 1 is a diagram illustrating settings of long exposure and short exposure. FIG. 2 is a diagram illustrating division of an image. FIG. 2 is a block diagram illustrating an example configuration of an exposure ratio division unit. FIG. 3 is a block diagram illustrating an example configuration of a first luminance change amount calculation unit. FIG. 4 is a block diagram illustrating an example configuration of a second luminance change amount calculation unit. FIG. 5 is a diagram illustrating enlargement processing and filter processing. FIG. 6 is a block diagram illustrating an example configuration of an HDR blending processing unit. FIG. 7 is a block diagram illustrating an example configuration of a blending ratio calculation processing unit. FIG. 8 is a diagram illustrating calculation of a blending ratio. FIG. 9 is a diagram illustrating blended data generation processing. FIG. 10 is a diagram illustrating an HDR image when the first blending ratio or the third blending ratio is used. FIG. 11 is a block diagram illustrating blended data when the first blending ratio or the third blending ratio is used. FIG. 12 is a block diagram illustrating an example configuration of a motion region HDR blending correction processing unit. FIG. 13 is a diagram illustrating data enlargement processing. FIG. 14 is a diagram illustrating calculation of a correction ratio. FIG. 15 is a flowchart illustrating imaging processing. FIG. 16 is a flowchart illustrating first luminance change amount calculation processing. FIG. 17 is a flowchart illustrating second luminance change amount calculation processing. FIG. 18 is a flowchart illustrating HDR blending processing. FIG. 19 is a flowchart illustrating motion region HDR blending correction processing.

[0012] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0013] <Configuration Example of Imaging Apparatus> FIG. 1 is a block diagram showing a configuration example of an embodiment of an imaging apparatus to which the present technology is applied.

[0014] As shown in FIG. 1 , the imaging device 11 includes an image sensor 12, an exposure setting unit 13, an image dividing unit 14, an exposure ratio division unit 15, a first brightness change amount calculation unit 16, a second brightness change amount calculation unit 17, a maximum brightness change amount selection unit 18, an enlargement and filtering processing unit 19, an HDR synthesis processing unit 20, and a motion region HDR synthesis correction processing unit 21.

[0015] The image sensor 12 is configured with a plurality of pixels arranged in a matrix in an imaging area, each of which performs photoelectric conversion according to a set exposure time, and outputs pixel data indicating a pixel value corresponding to the charge obtained by photoelectric conversion in each pixel.

[0016] The exposure setting unit 13 sets a long-term exposure for two pixels arranged in one diagonal direction out of the four pixels arranged in a 2x2 array in the imaging area of ​​the image sensor 12, and sets a short-term exposure for two pixels arranged in the other diagonal direction.

[0017] 2, for four pixels in a 2×2 array, the exposure setting unit 13 sets a long-time exposure with the same exposure time for the pixels in the upper right first quadrant and the pixels in the lower left third quadrant, and sets a short-time exposure with the same exposure time for the pixels in the upper left second quadrant and the pixels in the lower right fourth quadrant, so that the pixels in the first quadrant output first pixel data L1 for the long-time exposure, the pixels in the second quadrant output first pixel data S1 for the short-time exposure, the pixels in the third quadrant output second pixel data L2 for the long-time exposure, and the pixels in the fourth quadrant output second pixel data S2 for the short-time exposure.

[0018] 3 , the exposure setting unit 13 sets the exposure time so that the pixels in the first quadrant that output the first pixel data L1 for long-time exposure and the pixels in the third quadrant that output the second pixel data L2 for long-time exposure start and end exposure at the same timing. Meanwhile, the exposure setting unit 13 sets the exposure time so that the pixels in the second quadrant that output the first pixel data S1 for short-time exposure and the pixels in the fourth quadrant that output the second pixel data S2 for short-time exposure start and end exposure at different timings. The exposure setting unit 13 matches the start timing of the short-time exposure for the pixels in the second quadrant with the start timing of the long-time exposure for the pixels in the first and third quadrants, and matches the end timing of the short-time exposure for the pixels in the fourth quadrant with the end timing of the long-time exposure for the pixels in the first and third quadrants.

[0019] The image division unit 14 divides the image of all pixel data output from the image sensor 12 into four parts: first pixel data L1 of long exposure output from pixels in the first quadrant, first pixel data S1 of short exposure output from pixels in the second quadrant, second pixel data L2 of long exposure output from pixels in the third quadrant, and second pixel data S2 of short exposure output from pixels in the fourth quadrant.

[0020] For example, the left side of Fig. 4 shows pixel data of an image input from the image sensor 12 to the image division unit 14. The image division unit 14 divides the image of that pixel data into four images for each quadrant, as shown on the right side of Fig. 4, and outputs first pixel data L1 of a long-time exposure, first pixel data S1 of a short-time exposure, second pixel data L2 of a long-time exposure, and second pixel data S2 of a short-time exposure. Therefore, the image sizes of the images output from the image division unit 14 are each ¼ of the image sizes of the images input to the image division unit 14.

[0021] The image division unit 14 supplies the long-time exposure first pixel data L1 and the long-time exposure second pixel data L2 to the exposure ratio division unit 15. The image division unit 14 supplies the short-time exposure first pixel data S1 to the first luminance change amount calculation unit 16 and the HDR blending processing unit 20. The image division unit 14 supplies the short-time exposure second pixel data S2 to the first luminance change amount calculation unit 16, the second luminance change amount calculation unit 17, the HDR blending processing unit 20, and the motion region HDR blending correction processing unit 21.

[0022] The exposure ratio division unit 15 divides the first long-exposure pixel data L1 and the second long-exposure pixel data L2 supplied from the image division unit 14 by an exposure ratio (long-exposure / short-exposure), which is the ratio of the exposure time of the long-exposure to the exposure time of the short-exposure, for each identical coordinate in the image divided by the image division unit 14 into quarters of the image size. Then, the exposure ratio division unit 15 supplies the first long-exposure pixel data L'1 and the second long-exposure pixel data L'2 divided by the exposure ratio to the second luminance change amount calculation unit 17 and the HDR blending processing unit 20, respectively.

[0023] That is, the images in the first and third quadrants, for which long-time exposure was set, and the images in the second and fourth quadrants, for which short-time exposure was set, have different brightness due to the different exposure times. Since differences in brightness between these images can cause problems in subsequent processing, it is necessary to make the brightness of the images in the first through fourth quadrants uniform by dividing by the exposure ratio to darken the images in the first and third quadrants.

[0024] The first luminance change amount calculation unit 16 calculates a first luminance change amount d1 indicating a relative change in luminance between the short-exposure first pixel data S1 and the short-exposure second pixel data S2 supplied from the image division unit 14 for each identical coordinate in the image divided into quarter image sizes by the image division unit 14, and supplies the first luminance change amount d1 to a maximum luminance change amount selection unit 18. The calculation of the first luminance change amount d1 by the first luminance change amount calculation unit 16 will be described in detail later with reference to FIG.

[0025] The second luminance change amount calculation unit 17 calculates a second luminance change amount d2 for each identical coordinate in the image divided into quarter image sizes by the image division unit 14, using the long-exposure first pixel data L'1 and the long-exposure second pixel data L'2 divided by the exposure ratio supplied from the exposure ratio division unit 15, and the short-exposure second pixel data S2 supplied from the image division unit 14, and supplies the calculated second luminance change amount d2 to the maximum luminance change amount selection unit 18. The second luminance change amount d2 indicates a relative luminance change between the average value of the long-exposure first pixel data L'1 and the long-exposure second pixel data L'2 divided by the exposure ratio, and the short-exposure second pixel data S2. The calculation of the second luminance change amount d2 by the second luminance change amount calculation unit 17 will be described in detail later with reference to FIG. 7 .

[0026] The maximum luminance change amount selection unit 18 compares the first luminance change amount d1 supplied from the first luminance change amount calculation unit 16 with the second luminance change amount d2 supplied from the second luminance change amount calculation unit 17, and selects the larger one. Then, the maximum luminance change amount selection unit 18 supplies the selected one of the first luminance change amount d1 and the second luminance change amount d2 to the enlargement and filtering processing unit 19 as a third luminance change amount d3.

[0027] The enlargement and filtering processing unit 19 performs an enlargement process on the third luminance change amount d3 supplied from the maximum luminance change amount selection unit 18, and performs a filter process to smooth the enlarged third luminance change amount d3. Then, the enlargement and filtering processing unit 19 supplies the enlarged and filtered third luminance change amount d3 as a fourth luminance change amount d4 to the motion region HDR synthesis correction processing unit 21. The enlargement process and the filter process performed by the enlargement and filtering processing unit 19 will be described in detail later with reference to FIG. 8.

[0028] The HDR blending processor 20 performs HDR blending processing using the long-exposure first pixel data L'1 and long-exposure second pixel data L'2 divided by the exposure ratio supplied from the exposure ratio division unit 15, and the short-exposure first pixel data S1 and short-exposure second pixel data S2 supplied from the image division unit 14. In the HDR blending processing, the long-exposure first pixel data L'1 and long-exposure second pixel data L'2 divided by the exposure ratio are combined with the short-exposure first pixel data S1 and short-exposure second pixel data S2 to generate an HDR blended image with an expanded dynamic range. The HDR blending processor 20 then supplies the blended data HDR obtained as a result of the HDR blending processing to the motion region HDR blending correction processor 21. The HDR blending processing by the HDR blending processor 20 will be described in detail with reference to FIGS. 9 to 14, which will be described later.

[0029] The motion region HDR synthesis correction processor 21 performs a motion region HDR synthesis correction process using the short-exposure second pixel data S2 supplied from the image divider 14, the third luminance change amount d3 supplied from the enlargement and filtering processor 19, and the synthesis data HDR supplied from the HDR synthesis processor 20. In the motion region HDR synthesis correction process, a region with a large third luminance change amount d3 is regarded as a region in the HDR synthesis image where subject movement occurs, and the synthesis data HDR for that region is corrected using the short-exposure second pixel data S2, thereby generating a high-quality HDR synthesis image in which artifacts, motion blur, color bleeding, and the like are suppressed. The motion region HDR synthesis correction processor 21 then outputs the synthesis correction data HDR′ obtained as a result of the motion region HDR synthesis correction process as a high-quality HDR synthesis image captured by the imaging device 11. The motion region HDR synthesis correction process performed by the motion region HDR synthesis correction processor 21 will be described in detail with reference to FIGS. 15 to 17 , which will be described later.

[0030] FIG. 5 is a block diagram showing an example of the configuration of the exposure ratio division unit 15. As shown in FIG.

[0031] 5, the exposure ratio division unit 15 is configured to include a first division processing unit 31 and a second division processing unit 32. The exposure ratio division unit 15 is supplied with first pixel data L1 of long exposure and second pixel data L2 of long exposure from the image division unit 14. The exposure ratio division unit 15 is supplied with an exposure ratio based on the long exposure and short exposure set for the image sensor 12 from the exposure setting unit 13.

[0032] The first division processing unit 31 divides the first pixel data L1 of the long exposure time by the exposure ratio, and outputs the first pixel data L'1 of the long exposure time divided by the exposure ratio.

[0033] The second division processing unit 32 divides the long-time exposure second pixel data L2 by the exposure ratio, and outputs the long-time exposure second pixel data L'2 divided by the exposure ratio.

[0034] FIG. 6 is a block diagram showing an example of the configuration of the first luminance change amount calculation unit 16. As shown in FIG.

[0035] 6, the first luminance change amount calculation unit 16 is configured to include a comparison processing unit 41, an arithmetic processing unit 42, a correction processing unit 43, and an amplification processing unit 44. The first luminance change amount calculation unit 16 receives first short-time exposure pixel data S1 and second short-time exposure pixel data S2 from the image division unit 14. The first luminance change amount calculation unit 16 receives external parameters including a first coring cor1, a first bias bias1, and a first gain gain1.

[0036] The comparison processing unit 41 compares the magnitude relationship between the first pixel data S1 of short-time exposure and the second pixel data S2 of short-time exposure, and notifies the calculation processing unit 42 of the comparison result (S1≧S2 or S1<S2).

[0037] When the comparison result notified by the comparison processing unit 41 is S1≧S2 (the first pixel data S1 of the short-time exposure is equal to or greater than the second pixel data S2 of the short-time exposure), the calculation processing unit 42 performs calculation according to the following formula (1): On the other hand, when the comparison result notified by the comparison processing unit 41 is S1<S2 (the first pixel data S1 of the short-time exposure is less than the second pixel data S2 of the short-time exposure), the calculation processing unit 42 performs calculation according to the following formula (2):

[0038]

[0039] The calculation processing unit 42 then supplies the calculation result of the equation (1) or the equation (2) to the correction processing unit 43 .

[0040] If the value of the calculation result supplied from the calculation processing unit 42 is negative, the correction processing unit 43 corrects the calculation result to zero and supplies it to the amplification processing unit 44. If the value of the calculation result supplied from the calculation processing unit 42 is not negative, the correction processing unit 43 supplies the calculation result to the amplification processing unit 44 as is.

[0041] The amplification processing unit 44 amplifies the calculation result supplied from the correction processing unit 43 by a first gain gain1, and outputs the amplified calculation result as a first luminance change amount d1.

[0042] FIG. 7 is a block diagram showing an example of the configuration of the second luminance change amount calculation unit 17. As shown in FIG.

[0043] 7, the second luminance change amount calculation unit 17 includes an average calculation processing unit 51, a comparison processing unit 52, an arithmetic processing unit 53, a correction processing unit 54, and an amplification processing unit 55. The second luminance change amount calculation unit 17 receives the long-exposure first pixel data L'1 and the long-exposure second pixel data L'2 divided by the exposure ratio from the exposure ratio division unit 15, and receives the short-exposure second pixel data S2 from the image division unit 14. The second luminance change amount calculation unit 17 receives external parameters including a first saturation threshold S-th1, a second coring cor2, a second bias bias2, a second gain gain2, a third coring cor3, a third bias bias3, and a third gain gain3.

[0044] The average value calculation processing unit 51 calculates the average value avg of the first pixel data L'1 of the long-time exposure divided by the exposure ratio and the second pixel data L'2 of the long-time exposure divided by the exposure ratio, and supplies it to the comparison processing unit 52.

[0045] The comparison processing unit 52 compares the magnitude relationship between the average value avg supplied from the average calculation processing unit 51 and the first saturation threshold S-th1. If the comparison result between the average value avg and the first saturation threshold S-th1 is avg<S-th1 (the average value avg is less than the first saturation threshold S-th1), the comparison processing unit 52 compares the magnitude relationship between the average value avg and the second pixel data S2 of the short exposure and notifies the calculation processing unit 53 of the comparison result (avg<S-th1 and avg≧S2 or avg<S2). If the comparison result between the average value avg and the first saturation threshold S-th1 is avg≧S-th1 (the average value avg is equal to or greater than the first saturation threshold S-th1), the comparison processing unit 52 compares the magnitude relationship between the average value avg and the second pixel data S2 of the short exposure and notifies the calculation processing unit 53 of the comparison result (avg≧S-th1 and avg≧S2 or avg<S2).

[0046] If the comparison result notified by the comparison processing unit 52 is avg<S-th1 and avg≧S2 (the average value avg is less than the first saturation threshold S-th1 and the average value avg is equal to or greater than the second pixel data S2 of the short-time exposure), the calculation processing unit 53 performs a calculation according to the following equation (3). If the comparison result notified by the comparison processing unit 52 is avg<S-th1 and avg<S2 (the average value avg is less than the first saturation threshold S-th1 and the average value avg is less than the second pixel data S2 of the short-time exposure), the calculation processing unit 53 performs a calculation according to the following equation (4). If the comparison result notified by the comparison processing unit 52 is avg≧S-th1 and avg≧S2 (the average value avg is equal to or greater than the first saturation threshold S-th1 and the average value avg is equal to or greater than the second pixel data S2 of the short-time exposure), the calculation processing unit 53 performs a calculation according to the following equation (5). If the comparison result notified from the comparison processing unit 52 is avg≧S-th1 and avg<S2 (the average value avg is greater than or equal to the first saturation threshold S-th1 and the average value avg is less than the second pixel data S2 of short-time exposure), the calculation processing unit 53 performs calculation according to the following equation (6).

[0047]

[0048] Then, the calculation processing unit 53 supplies the calculation result of any one of the equations (3) to (6) to the correction processing unit 54 .

[0049] If the value of the calculation result supplied from the calculation processing unit 53 is negative, the correction processing unit 54 corrects the calculation result to zero and supplies it to the amplification processing unit 55. Note that if the value of the calculation result supplied from the calculation processing unit 53 is not negative, the calculation processing unit 53 supplies the calculation result to the amplification processing unit 55 as is.

[0050] The amplification processing unit 55 amplifies the calculation result supplied from the correction processing unit 54 by a second gain gain2 or a third gain gain3, and outputs the amplified calculation result as a second luminance change amount d2. Here, when the calculation result obtained by the calculation of the above-mentioned equation (3) or equation (4) is supplied from the correction processing unit 54 (when avg<S-th1), the amplification processing unit 55 amplifies the calculation result by the second gain gain2. On the other hand, when the calculation result obtained by the calculation of the above-mentioned equation (5) or equation (6) is supplied from the correction processing unit 54 (when avg≧S-th1), the amplification processing unit 55 amplifies the calculation result by a third gain gain3.

[0051] The enlargement processing and filtering processing by the enlargement and filtering processing unit 19 will be described with reference to FIG.

[0052] As described above, the image of all pixel data output from the image sensor 12 is divided into four images each having a quarter image size by the image dividing unit 14. Therefore, the image of the third luminance change amount d3 supplied to the enlargement and filtering processing unit 19 also has a quarter image size, as shown in A of Fig. 8 .

[0053] Therefore, the enlargement and filtering processing unit 19 performs enlargement processing on the image of the third luminance change amount d3, which has been reduced to 1 / 4 image size, to enlarge one pixel to 2×2 size, as shown in B of Fig. 8, thereby enlarging the image to 1 / 1 image size, the same as the image output from the image sensor 12. Furthermore, the enlargement and filtering processing unit 19 performs filtering processing, such as 3×3 averaging convolution processing, on the image of the third luminance change amount d3, which has been reduced to 1 / 1 image size, to smooth the image as shown in C of Fig. 8, and outputs it as an image of the fourth luminance change amount d4, which has 1 / 1 image size.

[0054] The HDR blending process performed in the HDR blending processor 20 will be described with reference to FIGS. 9 to 14.

[0055] FIG. 9 is a block diagram showing an example of the configuration of the HDR blending processing unit 20.

[0056] 9 , the HDR merging processing unit 20 is configured to include a merging ratio calculation processing unit 61 and a merging data generation processing unit 62. The HDR merging processing unit 20 receives the long-exposure first pixel data L′1 and the long-exposure second pixel data L′2 divided by the exposure ratio from the exposure ratio division unit 15, and receives the short-exposure first pixel data S1 and the short-exposure second pixel data S2 from the image division unit 14. The HDR merging processing unit 20 receives a first merging threshold C-th1 and a second merging threshold C-th2 as external parameters.

[0057] The composite ratio calculation processing unit 61 calculates a composite ratio α used to generate composite data HDR based on the average value of the long-exposure first pixel data L'1 and the long-exposure second pixel data L'2 divided by the exposure ratio, and supplies it to the composite data generation processing unit 62.

[0058] The composite data generation processing unit 62 generates composite data HDR by performing composite data generation processing in accordance with the composite ratio α supplied from the composite ratio calculation processing unit 61 on the first pixel data L'1 of long-time exposure divided by the exposure ratio, the second pixel data L'2 of long-time exposure divided by the exposure ratio, the first pixel data S1 of short-time exposure, and the second pixel data S2 of short-time exposure.

[0059] FIG. 10 is a block diagram showing an example of the configuration of the synthesis ratio calculation processing unit 61.

[0060] As shown in FIG. 10, the synthesis ratio calculation processing unit 61 includes an average value calculation processing unit 71 and a comparison processing unit 72 .

[0061] The average value calculation processing unit 71 calculates the average value avg of the first pixel data L'1 of the long-time exposure divided by the exposure ratio and the second pixel data L'2 of the long-time exposure divided by the exposure ratio, and supplies it to the comparison processing unit 72.

[0062] The comparison processing unit 72 compares the average value avg supplied from the average calculation processing unit 71 with the first combination threshold C-th1 input as an external parameter, and if the comparison result is avg≧C-th1 (the average value avg is equal to or greater than the first combination threshold C-th1), selects the first combination ratio α1 (=100%). The comparison processing unit 72 compares the average value avg supplied from the average calculation processing unit 71 with the second combination threshold C-th2 input as an external parameter, and if the comparison result is avg<C-th2 (the average value avg is less than the second combination threshold C-th2), selects the third combination ratio α3 (=0%). If the comparison results of the two above-mentioned comparisons are C-th2≦avg<C-th1 (the average value avg is equal to or greater than the second combination threshold C-th2 and less than the first combination threshold C-th1), the comparison processing unit 72 calculates the second combination ratio α2 according to the following equation (7):

[0063]

[0064] 11 , the comparison processing unit 72 selects the first combination ratio α1 (=100%) when avg≧C-th1, selects the third combination ratio α3 (=0%) when avg<C-th2, and calculates the current second combination ratio α2 from the current average value avg when C-th2≦avg<C-th1. The comparison processing unit 72 then supplies one of the first combination ratio α1, the second combination ratio α2, or the third combination ratio α3 to the combined data generation processing unit 62.

[0065] The composite data generation processing unit 62 performs the composite data generation process on an image such as that shown in the upper part of Fig. 12 . That is, the composite data generation processing unit 62 performs the composite data generation process on an image arranged at 1 / 1 image size such that pixel data at the same coordinates form a 2 x 2 array in the first long-exposure pixel data L'1, the second long-exposure pixel data L'2, the first short-exposure pixel data S1, and the second short-exposure pixel data S2, each of which has a 1 / 4 image size and is divided by the exposure ratio. Then, as a result of the composite data generation processing by the composite data generation processing unit 62, an HDR image (RAW data corresponding to HDR) made up of the composite data HDR as shown in the lower part of Fig. 12 is output from the composite data generation processing unit 62.

[0066] For example, when the comparison processing unit 72 supplies the composite data generation processing unit 62 with a first composite ratio α1 (=100%), the composite data generation processing unit 62 generates an HDR image as shown in A of FIG. 13 by the composite data generation processing. That is, for each 2×2 array of composite data HDR, the first pixel data L′1 (first quadrant) of the long-time exposure divided by the exposure ratio is overwritten on the second pixel data S2 of the short-time exposure, and the second pixel data L′2 (third quadrant) of the long-time exposure divided by the exposure ratio is overwritten on the first pixel data S1 of the short-time exposure. In this case, the first pixel data S1 of the short-time exposure and the second pixel data S2 of the short-time exposure are used as they are.

[0067] Furthermore, when the comparison processing unit 72 supplies the composite data generation processing unit 62 with a third composite ratio α3 (=0%), the composite data generation processing unit 62 generates an HDR image as shown in B of FIG. 13 by the composite data generation process. That is, for each 2×2 array of composite data HDR, the first pixel data S1 (second quadrant) of the short-time exposure is overwritten with the first pixel data L′1 of the long-time exposure divided by the exposure ratio, and the second pixel data S2 (fourth quadrant) of the short-time exposure is overwritten with the second pixel data L′2 of the long-time exposure divided by the exposure ratio. In this case, the first pixel data L′1 of the long-time exposure and the second pixel data L′2 of the long-time exposure divided by the exposure ratio are used as is.

[0068] Furthermore, when the second combination ratio α2 is supplied to the combined data generation processing unit 62 from the comparison processing unit 72, the combined data generation processing unit 62 calculates combined data HDR1 in the first upper right quadrant of the combined data HDR in the 2×2 array according to the following equation (8): For example, in the HDR image shown in the lower part of Fig. 12 , when calculating combined data HDR(1,0) in the first upper right quadrant of the combined data HDR in the 2×2 array at the top left, the short-exposure second pixel data S2(0,0) and the long-exposure first pixel data L'1(0,0) divided by the exposure ratio shown in the upper part of Fig. 12 are used in the calculation of the following equation (8).

[0069]

[0070] Note that, when the second combination ratio α2 in this equation (8) is set to the first combination ratio α1 (=100%), the combination data HDR(1,0) in the first upper right quadrant is equivalent to being overwritten with the short-exposure second pixel data S2(0,0), as shown in A of Fig. 14. Similarly, when the second combination ratio α2 in this equation (8) is set to the third combination ratio α3 (=0%), the combination data HDR(1,0) in the first upper right quadrant is equivalent to being overwritten with the long-exposure first pixel data L'1(0,0) divided by the exposure ratio, as shown in A of Fig. 14.

[0071] Furthermore, when the second combination ratio α2 is supplied to the combined data generation processing unit 62 from the comparison processing unit 72, the combined data generation processing unit 62 calculates combined data HDR2 in the upper left second quadrant of the 2×2 array combined data HDR according to the following equation (9): For example, in the HDR image shown in the lower part of Fig. 12 , when calculating combined data HDR(0,0) in the upper left second quadrant of the 2×2 array combined data HDR at the upper left end, the short-exposure first pixel data S1(0,0) and the long-exposure first pixel data L'1(0,0) divided by the exposure ratio shown in the upper part of Fig. 12 are used in the calculation of the following equation (9).

[0072]

[0073] Note that, when the second combination ratio α2 in this equation (9) is set to the first combination ratio α1 (=100%), the combination data HDR(0,0) in the upper left second quadrant is equivalent to being overwritten with the first pixel data S1(0,0) of short exposure, as shown in B of Fig. 14. Similarly, when the second combination ratio α2 in this equation (9) is set to the third combination ratio α3 (=0%), the combination data HDR(0,0) in the upper left second quadrant is equivalent to being overwritten with the first pixel data L'1(0,0) of long exposure divided by the exposure ratio, as shown in B of Fig. 14.

[0074] Furthermore, when the comparison processing unit 72 supplies the combined data generation processing unit 62 with the second combined ratio α2, the combined data generation processing unit 62 calculates combined data HDR3 in the lower left third quadrant of the combined data HDR in the 2×2 array according to the following equation (10): For example, in the HDR image shown in the lower part of Fig. 12 , when calculating combined data HDR(0,1) in the lower left third quadrant of the combined data HDR in the 2×2 array at the upper left corner, the short-exposure first pixel data S1(0,0) and the long-exposure second pixel data L'2(0,0) divided by the exposure ratio shown in the upper part of Fig. 12 are used in the calculation of the following equation (10).

[0075]

[0076] Note that, when the second combination ratio α2 in this equation (10) is set to the first combination ratio α1 (=100%), the combination data HDR(0,1) in the lower left third quadrant is equivalent to being overwritten with the first pixel data S1(0,0) of short exposure, as shown in C of Fig. 14. Similarly, when the second combination ratio α2 in this equation (10) is set to the third combination ratio α3 (=0%), the combination data HDR(0,1) in the lower left third quadrant is equivalent to being overwritten with the second pixel data L'2(0,0) of long exposure divided by the exposure ratio, as shown in C of Fig. 14.

[0077] Furthermore, when the second combination ratio α2 is supplied to the combined data generation processing unit 62 from the comparison processing unit 72, the combined data generation processing unit 62 calculates combined data HDR4 in the lower right fourth quadrant of the 2×2 array combined data HDR according to the following equation (11): For example, in the HDR image shown in the lower part of Fig. 12 , when calculating combined data HDR(1,1) in the lower right fourth quadrant of the 2×2 array combined data HDR at the upper left, the short-exposure second pixel data S2(0,0) and the long-exposure second pixel data L'2(0,0) divided by the exposure ratio shown in the upper part of Fig. 12 are used in the calculation of the following equation (11).

[0078]

[0079] Note that, when the second combination ratio α2 in this equation (11) is set to the first combination ratio α1 (=100%), the combination data HDR(1,1) in the lower right fourth quadrant is equivalent to being overwritten with the short-exposure second pixel data S2(0,0), as shown in D of Fig. 14. Similarly, when the second combination ratio α2 in this equation (11) is set to the third combination ratio α3 (=0%), the combination data HDR(1,1) in the lower right fourth quadrant is equivalent to being overwritten with the long-exposure second pixel data L'2(0,0) divided by the exposure ratio, as shown in D of Fig. 14.

[0080] The moving region HDR synthesis correction processing performed in the moving region HDR synthesis correction processing unit 21 will be described with reference to FIGS. 15 to 17.

[0081] FIG. 15 is a block diagram showing an example of the configuration of the motion region HDR synthesis correction processing unit 21.

[0082] 15 , the motion region HDR synthesis correction processing unit 21 is configured to include a data enlargement processing unit 81, a correction ratio calculation processing unit 82, and a correction data generation processing unit 83. The motion region HDR synthesis correction processing unit 21 receives the synthesis data HDR from the HDR synthesis processing unit 20, the short-exposure second pixel data S2 from the image division unit 14, and the fourth luminance change amount d4 from the enlargement and filtering processing unit 19. The motion region HDR synthesis correction processing unit 21 receives a first motion threshold M-th1 and a second motion threshold M-th2 as external parameters.

[0083] The data enlargement processing unit 81 applies a data enlargement process to the image of the short-exposure second pixel data S2, which is 1 / 4 the image size supplied from the image division unit 14, to enlarge one pixel to a 2x2 size, and supplies the enlarged short-exposure second pixel data S'2 to the correction data generation processing unit 83.

[0084] That is, as shown in the upper part of Fig. 16, the data enlargement processing unit 81 places the short-exposure second pixel data S2, which has a 1 / 4 image size, at the pixel position of the fourth quadrant, which is the lower right corner of the 2 x 2 array at the 1 / 1 image size. Then, as shown in the middle part of Fig. 16, the data enlargement processing unit 81 copies the second pixel data S2 placed at the pixel position of the fourth quadrant to pixels in the first, second, and third quadrants. Furthermore, as shown in the lower part of Fig. 16, the data enlargement processing unit 81 changes the coordinates of the short-exposure second pixel data S2, which remain at the 1 / 4 image size, to coordinates of the 1 / 1 image size, thereby obtaining enlarged short-exposure second pixel data S'2.

[0085] The correction ratio calculation processing unit 82 compares the fourth luminance change amount d4 supplied from the enlargement and filtering processing unit 19 with the first motion threshold M-th1 input as an external parameter, and if the comparison result is d4≧M-th1 (the fourth luminance change amount d4 is equal to or greater than the first motion threshold M-th1), selects the first correction ratio β1 (=100%).The correction ratio calculation processing unit 82 compares the fourth luminance change amount d4 supplied from the enlargement and filtering processing unit 19 with the second motion threshold M-th2 input as an external parameter, and if the comparison result is d4<M-th2 (the fourth luminance change amount d4 is less than the second motion threshold M-th2), selects the third correction ratio β3 (=0%). If the results of the two comparisons described above are M-th2≦d4<M-th1 (the fourth brightness change amount d4 is greater than or equal to the second motion threshold M-th2 and less than the first motion threshold M-th1), the correction ratio calculation processing unit 82 calculates the second correction ratio β2 according to the following equation (12).

[0086]

[0087] 17 , the correction ratio calculation processing unit 82 selects the first correction ratio β1 (=100%) when d4≧M-th1, selects the third correction ratio β3 (=0%) when d4<M-th2, and calculates the current second correction ratio β2 from the current fourth luminance change amount d4 when M-th2≦d4<M-th1. Then, the correction ratio calculation processing unit 82 supplies one of the first correction ratio β1, the second correction ratio β2, or the third correction ratio β3 to the correction data generation processing unit 83.

[0088] In addition, when a correction upper limit value as shown in Figure 17 is set, the correction ratio calculation processing unit 82 can replace the first correction ratio β1 or the second correction ratio β2 value with the correction upper limit value when the value exceeds the correction upper limit value.

[0089] The correction data generation processing unit 83 corrects the combined data HDR supplied from the HDR combination processing unit 20 with the expanded short-exposure second pixel data S'2 supplied from the data expansion processing unit 81, in accordance with the correction ratio β supplied from the correction ratio calculation processing unit 82, to generate combined corrected data HDR'. That is, the correction data generation processing unit 83 calculates the combined corrected data HDR' in accordance with the following equation (13).

[0090]

[0091] When the correction ratio β is replaced with the upper correction limit value, the correction ratio β in this equation (13) is replaced with the upper correction limit value to calculate the corrected composite data HDR′. When the third correction ratio β3 (=0%) is used, the corrected composite data HDR is output as the corrected composite data HDR′ (HDR=HDR′).

[0092] The imaging device 11 is configured as described above, and can determine an area in the HDR composite image where the third luminance change amount d3 is equal to or greater than the second motion threshold M-th2 as an area where the subject is moving, and correct the composite data HDR for that area with the short-exposure second pixel data S2. This makes it possible to suppress artifacts, motion blur, color bleeding, and the like that occur due to subject movement, and obtain an HDR composite image with higher image quality.

[0093] <Processing Example of Imaging Processing> The imaging processing executed in the imaging device 11 will be described with reference to FIGS.

[0094] FIG. 18 is a flowchart illustrating the imaging process.

[0095] In step S11, the exposure setting unit 13 sets a long-time exposure for the pixels in the upper right first quadrant and the lower left third quadrant of the four pixels arranged in a 2x2 array in the imaging area of ​​the image sensor 12, and sets a short-time exposure for the pixels in the upper left second quadrant and the lower right fourth quadrant.

[0096] In step S12, the image division unit 14 divides the image of all pixel data output from the image sensor 12 into four pieces: first pixel data L1 of long-time exposure output from pixels in the first quadrant, first pixel data S1 of short-time exposure output from pixels in the second quadrant, second pixel data L2 of long-time exposure output from pixels in the third quadrant, and second pixel data S2 of short-time exposure output from pixels in the fourth quadrant. The image division unit 14 supplies the first pixel data L1 of long-time exposure and the second pixel data L2 of long-time exposure to the exposure ratio division unit 15. The image division unit 14 supplies the first pixel data S1 of short-time exposure to the first luminance change amount calculation unit 16 and the HDR synthesis processing unit 20. The image dividing unit 14 supplies the short-exposure second pixel data S2 to the first luminance change amount calculation unit 16, the second luminance change amount calculation unit 17, the HDR synthesis processing unit 20, and the motion region HDR synthesis correction processing unit 21.

[0097] In step S13, the exposure ratio division unit 15 divides the first long-time exposure pixel data L1 and the second long-time exposure pixel data L2 supplied from the image division unit 14 in step S12 by the exposure ratio.

[0098] In step S14, the first luminance change amount calculation unit 16 executes the first luminance change amount calculation process (processing of the flowchart in FIG. 19) to calculate the first luminance change amount d1.

[0099] In step S15, the second luminance change amount calculation unit 17 executes the second luminance change amount calculation process (processing of the flowchart in FIG. 20) to calculate the second luminance change amount d2.

[0100] In step S16, the maximum luminance change amount selection unit 18 compares the first luminance change amount d1 calculated in the first luminance change amount calculation process in step S14 with the second luminance change amount d2 calculated in the second luminance change amount calculation process in step S15, and selects the larger one. Then, the maximum luminance change amount selection unit 18 supplies the selected one of the first luminance change amount d1 and the second luminance change amount d2 to the enlargement and filtering processing unit 19 as a third luminance change amount d3.

[0101] 8, the enlargement and filtering processing unit 19 performs enlargement processing and filtering processing on the third luminance change amount d3 supplied from the maximum luminance change amount selection unit 18 in step S16. Then, the enlargement and filtering processing unit 19 supplies the third luminance change amount d3 that has been subjected to the enlargement processing and filtering processing to the motion region HDR synthesis correction processing unit 21 as a fourth luminance change amount d4.

[0102] In step S18, the HDR synthesis processing unit 20 performs HDR synthesis processing (processing of the flowchart in Figure 21) using the long-exposure first pixel data L'1 and the long-exposure second pixel data L'2 divided by the exposure ratio supplied from the exposure ratio division unit 15 in step S13, and the short-exposure first pixel data S1 and the short-exposure second pixel data S2 supplied from the image division unit 14 in step S12.

[0103] In step S19, the motion region HDR synthesis correction processing unit 21 performs motion region HDR synthesis correction processing (the processing of the flowchart in FIG. 22 ) using the short-exposure second pixel data S2 supplied from the image dividing unit 14 in step S12, the third luminance change amount d3 supplied from the enlargement and filtering processing unit 19 in step S17, and the synthesis data HDR generated in the HDR synthesis processing in step S18. Then, the motion region HDR synthesis correction processing unit 21 outputs the synthesis correction data HDR′ obtained as a result of executing the motion region HDR synthesis correction processing as a high-quality HDR synthesized image captured by the imaging device 11, and then the imaging processing ends.

[0104] FIG. 19 is a flowchart illustrating the first luminance change amount calculation process executed in step S14 of FIG.

[0105] In step S21, the comparison processing unit 41 compares the magnitude relationship between the first pixel data S1 of short-time exposure and the second pixel data S2 of short-time exposure supplied from the image dividing unit 14 in step S12 of FIG.

[0106] In step S22, the comparison processing unit 41 determines whether the comparison result in step S21 is S1≧S2 (the first pixel data S1 of the short-time exposure is greater than or equal to the second pixel data S2 of the short-time exposure) or S1<S2 (the first pixel data S1 of the short-time exposure is less than the second pixel data S2 of the short-time exposure).

[0107] In step S22, if the comparison processing unit 41 determines that the comparison result in step S21 is S1≧S2, the process proceeds to step S23. In step S23, the comparison processing unit 41 notifies the calculation processing unit 42 of the comparison result in step S21 (S1≧S2), and the calculation processing unit 42 performs calculation according to the above-mentioned formula (1).

[0108] On the other hand, if the comparison processing unit 41 determines in step S22 that the comparison result in step S21 is S1<S2, the process proceeds to step S24. In step S24, the comparison processing unit 41 notifies the calculation processing unit 42 of the comparison result in step S21 (S1<S2), and the calculation processing unit 42 performs calculation according to the above-mentioned formula (2).

[0109] After the processing of step S23 or S24, the processing proceeds to step S25. In step S25, if the value of the calculation result by the calculation processing unit 42 in step S23 or S24 is negative, the correction processing unit 43 corrects the calculation result to zero and supplies it to the amplification processing unit 44. Note that if the value of the calculation result supplied from the calculation processing unit 42 is not negative, the processing of step S25 is skipped, and the calculation result is supplied to the amplification processing unit 44 as is.

[0110] In step S26, the amplification processing unit 44 amplifies the calculation result supplied from the correction processing unit 43 in step S25 by a first gain gain1, and outputs the amplified calculation result as a first luminance change amount d1. After the processing of step S26, the first luminance change amount calculation processing is terminated.

[0111] FIG. 20 is a flowchart illustrating the second luminance change amount calculation process executed in step S15 of FIG.

[0112] In step S31, the average value calculation processing unit 51 calculates the average value avg of the first pixel data L'1 of the long-time exposure divided by the exposure ratio supplied from the exposure ratio division unit 15 in step S13 of Figure 18 and the second pixel data L'2 of the long-time exposure divided by the exposure ratio, and supplies it to the comparison processing unit 52.

[0113] In step S32, the comparison processing unit 52 compares the average value avg supplied from the average value calculation processing unit 51 in step S31 with the first saturation threshold value S-th1 input as an external parameter.

[0114] In step S33, the comparison processing unit 52 determines whether the comparison result in step S32 is avg < S-th1 (the average value avg is less than the first saturation threshold S-th1) or avg ≧ S-th1 (the average value avg is greater than or equal to the first saturation threshold S-th1).

[0115] In step S33, if the comparison processing unit 52 determines that the comparison result in step S32 is avg<S-th1, the process proceeds to step S34.

[0116] In step S34, the comparison processing unit 52 compares the magnitude relationship between the average value avg supplied from the average value calculation processing unit 51 in step S31 and the second pixel data S2 of the short-time exposure supplied from the image division unit 14 in step S12 of Figure 18.

[0117] In step S35, if the comparison processing unit 52 determines that the comparison result in step S34 is avg≧S2, the process proceeds to step S36. In step S36, the comparison processing unit 41 notifies the calculation processing unit 53 of the comparison results in steps S33 and S35 (avg<S-th1 and avg≧S2), and the calculation processing unit 53 performs calculation according to the above-mentioned formula (3).

[0118] On the other hand, if the comparison processing unit 52 determines in step S35 that the comparison result in step S34 is avg<S2, the process proceeds to step S37. In step S37, the comparison processing unit 41 notifies the calculation processing unit 53 of the comparison results in steps S33 and S35 (avg<S-th1 and avg<S2), and the calculation processing unit 53 performs calculation according to the above-mentioned formula (4).

[0119] After the processing of step S36 or S37, the processing proceeds to step S38. In step S38, if the value of the calculation result by the calculation processing unit 53 in step S36 or S37 is negative, the correction processing unit 54 corrects the calculation result to zero and supplies it to the amplification processing unit 55. Note that if the value of the calculation result supplied from the calculation processing unit 53 is not negative, the processing of step S38 is skipped, and the calculation result is supplied to the amplification processing unit 55 as is.

[0120] In step S39, the amplification processing unit 55 amplifies the calculation result supplied from the correction processing unit 54 in step S38 by a second gain gain2, and outputs the amplified calculation result as a second luminance change amount d2.

[0121] On the other hand, if the comparison processing unit 52 determines in step S33 that the comparison result in step S32 is avg≧S-th1, the process proceeds to step S40.

[0122] In step S40, the comparison processing unit 52 compares the average value avg supplied from the average value calculation processing unit 51 in step S31 with the second pixel data S2 of the short-time exposure supplied from the image division unit 14 in step S12 of Figure 18.

[0123] In step S41, if the comparison processing unit 52 determines that the comparison result in step S40 is avg≧S2, the process proceeds to step S42. In step S42, the comparison processing unit 41 notifies the calculation processing unit 53 of the comparison results in steps S33 and S41 (avg≧S-th1 and avg≧S2), and the calculation processing unit 53 performs calculation according to the above-mentioned formula (5).

[0124] On the other hand, if the comparison processing unit 52 determines in step S41 that the comparison result in step S40 is avg<S2, the process proceeds to step S43. In step S43, the comparison processing unit 41 notifies the calculation processing unit 53 of the comparison results in steps S33 and S35 (avg≧S-th1 and avg<S2), and the calculation processing unit 53 performs calculation according to the above-mentioned formula (6).

[0125] After the processing of step S42 or S43, the processing proceeds to step S44. In step S44, if the value of the calculation result by the calculation processing unit 53 in step S42 or S43 is negative, the correction processing unit 54 corrects the calculation result to zero and supplies it to the amplification processing unit 55. Note that if the value of the calculation result supplied from the calculation processing unit 53 is not negative, the processing of step S44 is skipped, and the calculation result is supplied to the amplification processing unit 55 as is.

[0126] In step S45, the amplification processing unit 55 amplifies the calculation result supplied from the correction processing unit 54 in step S44 by a third gain gain3, and outputs the amplified calculation result as the second luminance change amount d2. After the processing of step S39 or S45, the second luminance change amount calculation processing ends.

[0127] FIG. 21 is a flowchart illustrating the HDR combining process executed in step S18 of FIG.

[0128] In step S51, the average value calculation processing unit 71 calculates the average value avg of the first pixel data L'1 of the long-time exposure divided by the exposure ratio supplied from the exposure ratio division unit 15 in step S13 of Figure 18 and the second pixel data L'2 of the long-time exposure divided by the exposure ratio, and supplies it to the comparison processing unit 72.

[0129] In step S52, the comparison processing unit 72 compares the average value avg supplied from the average value calculation processing unit 71 in step S51 with the first composite threshold value C-th1 input as an external parameter.

[0130] In step S53, the comparison processing unit 72 determines whether the comparison result in step S52 is avg≧C-th1 (the average value avg is greater than or equal to the first composite threshold C-th1) or avg<C-th1 (the average value avg is less than the first composite threshold C-th1).

[0131] In step S53, if the comparison processing unit 72 determines that the comparison result in step S52 is avg≧C-th1, the process proceeds to step S54. In step S54, the comparison processing unit 72 selects the first synthesis ratio α1 (=100%) and supplies it to the synthesis data generation processing unit 62.

[0132] On the other hand, if the comparison processing unit 72 determines in step S53 that the comparison result in step S52 is avg<C-th1, the process proceeds to step S55.

[0133] In step S55, the comparison processing unit 72 compares the average value avg supplied from the average value calculation processing unit 71 in step S51 with the second composite threshold value C-th2 input as an external parameter.

[0134] In step S56, the comparison processing unit 72 determines whether the comparison result in step S55 is avg≧C-th2 (the average value avg is greater than or equal to the second composite threshold C-th2) or avg<C-th2 (the average value avg is less than the second composite threshold C-th2).

[0135] In step S56, if the comparison processing unit 72 determines that the comparison result in step S55 is avg<C-th2, the process proceeds to step S57. In step S57, the comparison processing unit 72 selects the third synthesis ratio α3 (=0%) and supplies it to the synthesis data generation processing unit 62.

[0136] On the other hand, if the comparison processing unit 72 determines in step S56 that the comparison result in step S55 is avg≧C-th2, the process proceeds to step S58. In step S58, the comparison processing unit 72 calculates the second synthesis ratio α2 in accordance with the above-mentioned equation (7), and supplies it to the synthesis data generation processing unit 62.

[0137] After the processing of step S54, step S57, or step S58, the processing proceeds to step S59.

[0138] In step S59, the composite data generation processing unit 62 performs composite data generation processing using the first composite ratio α1 supplied from the comparison processing unit 72 in step S54, the third composite ratio α3 supplied from the comparison processing unit 72 in step S57, or the second composite ratio α2 supplied from the comparison processing unit 72 in step S58. That is, the composite data generation processing unit 62 composites, in accordance with the composite ratio α, the first long-exposure pixel data L′1 and the second long-exposure pixel data L′2 divided by the exposure ratio supplied from the exposure ratio division unit 15 in step S13 of FIG. 18 , and the first short-exposure pixel data S1 and the second short-exposure pixel data S2 supplied from the image division unit 14 in step S12 of FIG. 18 . Then, the composite data generation processing unit 62 supplies the composite data HDR generated by performing the composite data generation processing to the motion region HDR composite correction processing unit 21, and then the HDR composite processing ends.

[0139] FIG. 22 is a flowchart illustrating the motion region HDR synthesis correction process executed in step S19 of FIG.

[0140] In step S61, the data enlargement processing unit 81 performs data enlargement processing to enlarge the image of the short-exposure second pixel data S2, which has a quarter image size and is supplied from the image dividing unit 14, to a size of 2 × 2. Then, the data enlargement processing unit 81 supplies the enlarged short-exposure second pixel data S'2 obtained by the data enlargement processing to the correction data generation processing unit 83.

[0141] In step S62, the correction ratio calculation processing unit 82 compares the fourth brightness change amount d4 supplied from the enlargement and filtering processing unit 19 in step S17 of Figure 18 with the first motion threshold value M-th1 input as an external parameter.

[0142] In step S63, the correction ratio calculation processing unit 82 determines whether the comparison result in step S62 is d4≧M-th1 (the fourth brightness change amount d4 is greater than or equal to the first motion threshold M-th1) or d4<M-th1 (the fourth brightness change amount d4 is less than the first motion threshold M-th1).

[0143] In step S63, if the correction ratio calculation processing unit 82 determines that the comparison result in step S62 is d4≧M-th1, the process proceeds to step S64. In step S64, the correction ratio calculation processing unit 82 selects the first correction ratio β1 (=100%) and supplies it to the correction data generation processing unit 83.

[0144] On the other hand, if the correction ratio calculation processing unit 82 determines in step S63 that the comparison result in step S62 is d4<M-th1, the process proceeds to step S65.

[0145] In step S65, the correction ratio calculation processing unit 82 compares the magnitude relationship between the fourth brightness change amount d4 supplied from the enlargement and filtering processing unit 19 in step S17 of Figure 18 and the second motion threshold value M-th2 input as an external parameter.

[0146] In step S66, the correction ratio calculation processing unit 82 determines whether the comparison result in step S65 is d4≧M-th2 (the fourth brightness change amount d4 is greater than or equal to the second motion threshold M-th2) or d4<M-th2 (the fourth brightness change amount d4 is less than the second motion threshold M-th2).

[0147] In step S66, if the correction ratio calculation processing unit 82 determines that the comparison result in step S65 is d4<M-th2, the process proceeds to step S67. In step S67, the correction ratio calculation processing unit 82 selects the third correction ratio β3 (=0%) and supplies it to the correction data generation processing unit 83.

[0148] On the other hand, if the correction ratio calculation processing unit 82 determines in step S66 that the comparison result in step S65 is d4≧M-th2, the process proceeds to step S68. In step S68, the correction ratio calculation processing unit 82 calculates the second correction ratio β2 in accordance with the above-mentioned equation (12), and supplies it to the correction data generation processing unit 83.

[0149] After the processing of step S64, step S67, or step S68, the processing proceeds to step S69.

[0150] In step S69, the correction data generation processing unit 83 performs correction data generation processing using the first correction ratio β1 supplied from the correction ratio calculation processing unit 82 in step S64, the third correction ratio β3 supplied from the correction ratio calculation processing unit 82 in step S67, or the second correction ratio β2 supplied from the correction ratio calculation processing unit 82 in step S68. That is, the correction data generation processing unit 83 corrects the combined data HDR supplied from the combined data generation processing unit 62 in step S59 of FIG. 21 with the expanded short-exposure second pixel data S'2 supplied from the data expansion processing unit 81 in step S61, in accordance with the correction ratio β. Then, after the correction data generation processing unit 83 performs the correction data generation processing and outputs the generated combined corrected data HDR', the motion region HDR combination correction processing ends.

[0151] By performing the above-described imaging process in the imaging device 11, it is possible to obtain an HDR composite image with higher image quality.

[0152] <Configuration Example of Electronic Device> The imaging device 11 as described above can be applied to various electronic devices, such as imaging systems such as digital still cameras and digital video cameras, mobile phones with imaging functions, or other devices with imaging functions.

[0153] FIG. 23 is a block diagram showing an example of the configuration of an imaging system installed in an electronic device.

[0154] As shown in FIG. 23, the imaging system 101 comprises an optical system 102, an imaging element 103, a signal processing circuit 104, a monitor 105, and a memory 106, and is capable of capturing still images and moving images.

[0155] The optical system 102 is configured to have one or more lenses, and guides image light (incident light) from a subject to the image sensor 103 , forming an image on the light receiving surface (sensor section) of the image sensor 103 .

[0156] The imaging device 11 described above is applied as the imaging element 103 (for example, a configuration in which the imaging device 11 is built into the imaging element 103, or a configuration in which the imaging device 11 is provided in front of the signal processing circuit 104). Electrons are accumulated in the imaging element 103 for a certain period of time in accordance with an image formed on the light receiving surface via the optical system 102. A signal corresponding to the electrons accumulated in the imaging element 103 is then supplied to the signal processing circuit 104.

[0157] The signal processing circuit 104 performs various types of signal processing on the pixel signals output from the image sensor 103. The image (image data) obtained by the signal processing performed by the signal processing circuit 104 is supplied to a monitor 105 for display, or supplied to a memory 106 for storage (recording).

[0158] In the imaging system 101 configured in this manner, by applying the imaging device 11 described above, for example, it is possible to capture images with higher image quality.

[0159] <Example of Use of Image Sensor> FIG. 24 is a diagram showing an example of use of the image sensor (imaging device) described above.

[0160] The image sensor described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0161] ・Devices for taking images for viewing purposes, such as digital cameras and mobile devices with camera functions. ・Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping, and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. ・Devices for home appliances such as TVs, refrigerators, and air conditioners that take images of user gestures and operate the device according to those gestures. ・Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. ・Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. ・Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. ・Devices for sports purposes, such as action cameras and wearable cameras for sports, etc. ・Devices for agricultural purposes, such as cameras to monitor the condition of fields and crops.

[0162] <Examples of Combinations of Configurations> The present technology can also be configured as follows: (1) An imaging device including: an HDR (High Dynamic Range) combining processing unit that generates an HDR combined image with an expanded dynamic range by performing HDR combining processing to calculate combined data obtained by combining pixel data of a long exposure time and pixel data of a short exposure time; and a motion area HDR combining correction processing unit that generates an HDR combined image with reduced degradation in image quality due to movement of the subject by performing motion area correction processing to calculate combined corrected data obtained by correcting the combined data in a region of the HDR combined image with pixel data of the short exposure time. (2) The imaging device according to (1), further comprising: an image dividing unit that divides an image of all pixel data output from an image sensor in which, of four pixels in a 2×2 array arranged in an imaging area, long-term exposure is set for two of the pixels arranged in one diagonal direction and short-term exposure is set for two of the pixels arranged in the other diagonal direction, into four pieces of long-term exposure first pixel data, short-term exposure first pixel data, long-term exposure second pixel data, and short-term exposure second pixel data; and an exposure ratio division unit that divides the first long-term exposure pixel data and the second long-term exposure pixel data by an exposure ratio that is a ratio of an exposure time of the long-term exposure to an exposure time of the short-term exposure, wherein the HDR blending processing unit performs the HDR blending process using the long-term exposure first pixel data divided by the exposure ratio, the long-term exposure second pixel data divided by the exposure ratio, the first short-term exposure pixel data, and the second short-term exposure pixel data.(3) The imaging device according to (2), wherein the HDR blending processor comprises: a blending ratio calculation processor that calculates a blending ratio used to generate the HDR blended image based on an average value of the first long-exposure pixel data divided by the exposure ratio and the second long-exposure pixel data divided by the exposure ratio, and a blending data generation processor that performs blending data generation processing on the first long-exposure pixel data divided by the exposure ratio, the second long-exposure pixel data divided by the exposure ratio, the first short-exposure pixel data, and the second short-exposure pixel data in accordance with the blending ratio. (4) The imaging device according to (3), further comprising: a first luminance change amount calculation unit that calculates a first luminance change amount indicating a relative luminance change between the first short-exposure pixel data and the second short-exposure pixel data. (5) The imaging device according to (4), further comprising: a second luminance change amount calculation unit that calculates a second luminance change amount indicating a relative luminance change between the second pixel data of the short exposure and an average value of the first pixel data of the long exposure time divided by the exposure ratio and the second pixel data of the long exposure time divided by the exposure ratio. (6) The imaging device according to (5), further comprising: a maximum luminance change amount selection unit that compares the first luminance change amount and the second luminance change amount, selects one with a larger value, and sets the selected one as a third luminance change amount. (7) The imaging device according to (6), further comprising: an enlargement and filtering processing unit that performs an enlargement process on the third luminance change amount, and performs a filter process to smooth the enlarged third luminance change amount, and sets the third luminance change amount obtained by performing a filter process to smooth the enlarged third luminance change amount as a fourth luminance change amount. (8) The imaging device according to (7), wherein the motion region HDR synthesis correction processing unit includes: a data enlargement processing unit that enlarges the second pixel data of the short exposure to an image size of an image output from the image sensor; a correction ratio calculation processing unit that calculates a correction ratio used to generate the HDR synthesis correction image based on the fourth luminance change amount; and a correction data generation processing unit that corrects the HDR synthesis image with the enlarged second pixel data of the short exposure in accordance with the correction ratio.(9) An imaging method including: an imaging device performing HDR (High Dynamic Range) combining processing to calculate combined data obtained by combining pixel data of a long exposure time and pixel data of a short exposure time, thereby generating an HDR composite image with an expanded dynamic range; and performing motion area correction processing to calculate combined corrected data obtained by correcting the combined data in an area where a subject is moving in the HDR composite image with the pixel data of the short exposure time, thereby generating an HDR composite corrected image in which image quality degradation due to the movement of the subject is suppressed. (10) An electronic device including an imaging device having: an HDR combining processing unit performing HDR (High Dynamic Range) combining processing to calculate combined data obtained by combining pixel data of a long exposure time and pixel data of a short exposure time, thereby generating an HDR composite corrected image in which image quality degradation due to the movement of the subject is suppressed, and a motion area HDR combining correction processing unit performing motion area correction processing to calculate combined corrected data obtained by correcting the combined data in an area where a subject is moving in the HDR composite image with the pixel data of the short exposure time, thereby generating an HDR composite corrected image in which image quality degradation due to the movement of the subject is suppressed.

[0163] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0164] REFERENCE SIGNS LIST 11 Imaging device, 12 Image sensor, 13 Exposure setting unit, 14 Image division unit, 15 Exposure ratio division unit, 16 First brightness change amount calculation unit, 17 Second brightness change amount calculation unit, 18 Maximum brightness change amount selection unit, 19 Enlargement and filter processing unit, 20 HDR compositing processing unit, 21 Motion area HDR compositing correction processing unit, 31 First division processing unit, 32 Second division processing unit, 41 Comparison processing unit, 42 Arithmetic processing unit, 43 Correction processing unit, 44 Amplification processing unit, 51 Average value calculation processing unit, 52 Comparison processing unit, 53 Arithmetic processing unit, 54 Correction processing unit, 55 Amplification processing unit, 61 Compositing ratio calculation processing unit, 62 Compositing data generation processing unit, 71 Average value calculation processing unit, 72 Comparison processing unit, 81 Data enlargement processing unit, 82 Correction ratio calculation processing unit, 83 Correction data generation processing unit

Claims

1. An imaging device comprising: an HDR (High Dynamic Range) synthesis processing unit that generates an HDR synthetic image with an expanded dynamic range by performing HDR synthesis processing to calculate synthetic data that synthesizes long-exposure pixel data and short-exposure pixel data; and a motion area HDR synthesis correction processing unit that generates an HDR synthetic corrected image in which degradation in image quality due to subject movement is suppressed by performing motion area correction processing to calculate synthetic correction data that corrects the synthetic data in areas of the HDR synthetic image where the subject is moving, using the short-exposure pixel data.

2. The imaging device according to claim 1, further comprising: an image division unit that divides an image of all pixel data output from an image sensor in which, of four pixels in a 2 x 2 array arranged in an imaging area, long exposure is set for two of the pixels arranged in one diagonal direction and short exposure is set for two of the pixels arranged in the other diagonal direction, into four pieces of long exposure first pixel data, short exposure first pixel data, long exposure second pixel data, and short exposure second pixel data; and an exposure ratio division unit that divides the first long exposure pixel data and the second long exposure pixel data by an exposure ratio that is a ratio of the exposure time of the long exposure to the exposure time of the short exposure, wherein the HDR synthesis processing unit performs the HDR synthesis processing using the first long exposure pixel data divided by the exposure ratio, the second long exposure pixel data divided by the exposure ratio, the first short exposure pixel data, and the second short exposure pixel data.

3. The imaging device according to claim 2, wherein the HDR synthesis processing unit has: a synthesis ratio calculation processing unit that calculates a synthesis ratio used to generate the HDR synthetic image based on an average value of the first long exposure pixel data divided by the exposure ratio and the second long exposure pixel data divided by the exposure ratio; and a synthesis data generation processing unit that performs synthesis data generation processing in accordance with the synthesis ratio on the first long exposure pixel data divided by the exposure ratio, the second long exposure pixel data divided by the exposure ratio, the first short exposure pixel data, and the second short exposure pixel data.

4. The imaging device of claim 2, further comprising a first luminance change amount calculation unit that calculates a first luminance change amount indicating a relative luminance change between the first pixel data of the short-time exposure and the second pixel data of the short-time exposure.

5. The imaging device of claim 4, further comprising a second luminance change amount calculation unit that calculates a second luminance change amount indicating a relative luminance change between an average value of the first pixel data of the long exposure divided by the exposure ratio and the second pixel data of the long exposure divided by the exposure ratio, and the second pixel data of the short exposure.

6. The imaging device according to claim 5, further comprising a maximum luminance change amount selection unit which compares the first luminance change amount and the second luminance change amount to select the one with the larger value, and sets the selected one as a third luminance change amount.

7. The imaging device of claim 6, further comprising an enlargement and filtering processing unit that performs an enlargement process on the third amount of luminance change, and performs a filter process on the third amount of luminance change that has been enlarged to smooth the enlarged amount of the third amount of luminance change, thereby obtaining a fourth amount of luminance change.

8. The imaging device described in claim 7, wherein the motion area HDR synthesis correction processing unit has: a data enlargement processing unit that enlarges the second pixel data of the short exposure to an image size of an image output from the image sensor; a correction ratio calculation processing unit that calculates a correction ratio used to generate the HDR synthesis corrected image based on the fourth luminance change amount; and a correction data generation processing unit that corrects the HDR synthesis image with the enlarged second pixel data of the short exposure in accordance with the correction ratio.

9. An imaging method including the steps of: an imaging device performing HDR (High Dynamic Range) synthesis processing to calculate synthetic data by synthesizing long-exposure pixel data and short-exposure pixel data, thereby generating an HDR synthetic image with an expanded dynamic range; and performing motion area correction processing to calculate synthetic correction data by correcting the synthetic data in areas of the HDR synthetic image where there is subject movement with the short-exposure pixel data, thereby generating an HDR synthetic correction image in which degradation in image quality due to the movement of the subject is suppressed.

10. An electronic device equipped with an imaging device having: an HDR (High Dynamic Range) synthesis processing unit that generates an HDR synthetic image with an expanded dynamic range by performing HDR synthesis processing to calculate synthetic data by synthesizing long-exposure pixel data and short-exposure pixel data; and a motion area HDR synthesis correction processing unit that generates an HDR synthetic corrected image in which degradation of image quality due to subject movement is suppressed by performing motion area correction processing to calculate synthetic correction data by correcting the synthetic data in areas of the HDR synthetic image where the subject is moving, using the short-exposure pixel data.

Images