Image processing device and control method thereof, and imaging device

Abstract

To ensure a wide dynamic range while suppressing discomfort caused by a mobile body.SOLUTION: Two images applied with different gains by exposure at a proper exposure amount and two images applied with different gains by exposure at an underexposure amount are acquired. A mobile body detection is performed between the images acquired on each of the different exposures (between a target image after correcting an image location and a reference image). On the basis of a determination result thereof, images to be used for image composition are selected from a plurality of images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method thereof, and an imaging device. [Background technology]

[0002] Conventionally, a technique has been known in image processing devices, such as imaging devices, that captures multiple images, including images with less blown-out highlights and images with less crushed shadows, by varying exposure conditions and then compositing these images to achieve a wide dynamic range. This technique is called high dynamic range (HDR) compositing. However, HDR compositing is considered unsuitable for capturing moving subjects because it generates an unnatural composite image in which the brightness varies depending on the position of the moving subject due to misalignment in the capture timing between the multiple images.

[0003] On the other hand, there is a known image sensor (DGO) that has two column circuits for the output signal from the unit pixel, and separate gain amplifiers within the column circuits, allowing it to output images with different gains. This image sensor can output multiple images with different gains (e.g., high-gain and low-gain images) with a single exposure. Compared to combining two images obtained by a DGO with combining two images obtained by time-shared exposure, the DGO has the advantage of not requiring alignment processing and being robust against the presence of moving objects. This makes the DGO well-suited for HDR compositing, which produces images with an expanded dynamic range.

[0004] Incidentally, image sensors have floating diffusion (FD) elements that act as capacitors to store electric charge. The amount of charge that can be stored in these FD elements changes depending on the settings. Increasing the capacitance allows more light to be handled, so larger capacitances are used at low sensitivity levels. However, increasing the capacitance has the disadvantage of increasing noise.

[0005] Patent Document 1 discloses a technique for generating natural-looking images by replacing areas corresponding to moving objects in multiple images captured by time-division exposure with multiple weighted images. However, with this technique, the replaced moving object areas are synthesized as afterimages, making it impossible to sufficiently eliminate the sense of incongruity.

[0006] Patent Document 2 discloses a technique for generating an HDR image with a wide dynamic range by combining multiple images captured with different exposure times and amplified with different gains. This technique eliminates the sense of incongruity that can occur when moving objects are captured at different times. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-240031 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-56940 Summary of the Invention [Problem to be solved by the invention]

[0008] However, to obtain a dynamic range equivalent to that obtained by time-division exposure, it is necessary to use a capacitor with a larger charge capacity. Increasing the charge capacity of the capacitor results in more noise than an HDR image captured and combined using time-division exposure. Therefore, when achieving a wide dynamic range, it is not easy to suppress the unnaturalness caused by moving objects.

[0009] The present invention aims to ensure a wide dynamic range while suppressing the sense of incongruity caused by a moving object. [Means for solving the problem]

[0010] In order to achieve the above object, the present invention comprises an acquisition means for acquiring a plurality of images amplified with a different gain for each exposure at a different exposure amount, a determination means for determining the presence or absence of a moving object between the images acquired for each exposure at a different exposure amount, and a selection means for selecting an image to be used for image synthesis from the acquired plurality of images based on the determination result by the determination means, wherein the acquisition means acquires a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, and acquires a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount, and the selection means selects either the first image and the second image or the third image and the fourth image when it is determined that the moving object is present between the first image and the third image, and selects at least one of the first image and the second image and at least one of the third image and the fourth image when it is determined that the moving object is not present between the first image and the third image. and when it is determined that the first exposure amount is greater than the second exposure amount, the second gain is greater than the first gain, and the moving object is not present between the first image and the third image, the first image, the second image, and the third image are selected. It is characterized by: [Effects of the Invention]

[0011] According to the present invention, it is possible to ensure a wide dynamic range while suppressing the sense of incongruity caused by a moving object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an image processing device. [Figure 2] FIG. 2 is a block diagram of an imaging element. [Figure 3] This is a diagram showing one column with a column AMP section. [Figure 4] 10 is a flowchart showing an image synthesis process. [Figure 5] FIG. 10 shows an example of multiple images amplified with different gains for exposures with different exposure amounts. [Figure 6] 1A and 1B are diagrams illustrating an image and a detection result in moving object detection. [Figure 7] FIG. 10 is a diagram showing a synthesis ratio table. [Figure 8] FIG. 10 shows examples of multiple images amplified with different gains for each of three exposures. [Figure 9] FIG. 10 is a diagram showing the detection result of a moving object between images. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] 1 is a block diagram of an image processing device according to an embodiment of the present invention, which is an image capturing device 100 such as a digital single-lens reflex camera, a digital still camera, or a digital video camera.

[0015] The optical lens 101 is an optical element that takes in light from a subject and forms an image of the light on the image sensor 102. The image sensor 102 receives incident light from the optical lens 101, converts it into an electrical signal, and outputs it. The image sensor 102 may be a CCD image sensor (Charge Coupled Device) or a CMOS image sensor.

[0016] Generally, there are two types of image sensors: one that directly outputs an analog signal, and one that performs internal AD (analog-to-digital) conversion processing and outputs digital data using LVDS (low voltage differential signaling) or the like. The latter type is used for the image sensor 102. Note that if an image sensor that does not perform internal AD conversion is used, the image sensor 102 also includes an analog front end that performs analog-to-digital conversion.

[0017] The image acquisition unit 103 has blocks that capture the video signal output from the image sensor 102 and perform various processes. The image acquisition unit 103 performs processes such as removing fixed pattern noise from the image sensor 102 and clamping the black level. The image acquisition unit 103 also has the role of separating the video signal into an image signal to be used for recording and an evaluation signal for controlling the image sensor 102.

[0018] The image processing unit 104 performs various image processing such as white balance adjustment, color interpolation, filtering, and composition processing on the video signal acquired from the image acquisition unit 103. The image processing unit 104 also performs compression processing on the video signal acquired from the image acquisition unit 103 using a standard such as JPEG. The signal recording unit 105 records the video signal received from the image processing unit 104 in a storage device or storage medium (neither is shown). The exposure control unit 106 calculates an optimal exposure amount from the video signal received from the image acquisition unit 103, determines the operation of the image sensor control unit 107, and transmits the control to the image sensor control unit 107.

[0019] The imaging device 100 has a CPU, a ROM, and a RAM (none of which are shown). The functions of the image acquisition unit 103, the image processing unit 104, the signal recording unit 105, the exposure control unit 106, and the image sensor control unit 107 are realized by the cooperation of the CPU, ROM, RAM, etc.

[0020] 2 is a block diagram of the image sensor 102. A timing pulse control unit 201 controls the operation of the image sensor 102 by supplying an operating clock (CLK) to each block of the image sensor 102 and by supplying timing signals to each block. A vertical scanning circuit 202 performs timing control to sequentially read out pixel signal voltages generated by a two-dimensionally arranged pixel unit 203 during one frame. Generally, video signals are read out row by row from the top row to the bottom row during one frame.

[0021] The pixel unit 203 is a photoelectric conversion element that performs photoelectric conversion according to the amount of incident light and outputs it as a voltage. The pixel unit 203 converts the captured light into an electric charge and accumulates the electric charge in a floating diffusion (FD). The capacity of the FD is variable, and can be changed between "large" and "small," for example. The signal-to-noise ratio can be improved by changing the capacity according to the ISO sensitivity. Basically, the capacity is set to large at low ISO sensitivity and small at high ISO sensitivity.

[0022] When outputting images with two different gains (described later), the capacitance for accumulating charge is the same for these two gains. The capacitance size is not limited to two levels, large and small, and may be set to three or more levels.

[0023] The column AMP 204 is used to electrically amplify the signal read out from the pixel unit 203. By amplifying the signal level of the pixel signal in the column AMP 204, the pixel signal level becomes relatively large with respect to the noise output from the subsequent column ADC 205, thereby equivalently improving the S / N ratio.

[0024] Furthermore, the timing pulse control unit 201 can change the gain of the column AMP 204. The image sensor 102 has two input memories in the column AMP 204 for generating HDR images, and can output two types of gain by changing the gain of the column AMP 204 (column AMP gain). By having two input memories, a signal at a certain time read from the FD can be multiplied by two gains and output. Therefore, although the amount of data increases, two simultaneous images with different gains can be obtained. In this way, the image sensor 102 is an example of a sensor that outputs multiple images amplified with different gains by a single exposure. Note that in this embodiment, the number of different gains is set to two and two images are output, but the number of gains, i.e., the number of images simultaneously output by a single exposure, is not limited to two.

[0025] The column ADC 205 performs analog-to-digital conversion on the signal output from the column AMP 204. The digitized signal is sequentially read out by the horizontal transfer circuit 206. The output of the horizontal transfer circuit 206 is input to the signal processing circuit 207. The signal processing circuit 207 is a circuit that performs digital signal processing. In addition to adding a certain amount of offset value through digital processing, the signal processing circuit 207 can easily perform gain calculations by performing shift calculations and multiplications. Note that the signal processing circuit 207 may have intentionally light-shielded pixel regions in the pixel unit 203 and perform a digital black level clamping operation using this.

[0026] The output of the signal processing circuit 207 is passed to an external output circuit 208. The external output circuit 208 has a serializer function and converts the multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. The external output circuit 208 also converts this serial signal into, for example, an LVDS signal, and outputs it for exchanging image information with an external device.

[0027] Next, a description will be given of the operation of the image sensor 102 when generating an HDR image. As described above, the image sensor 102 can change and output column AMP gains in order to generate an HDR image.

[0028] 3 is a diagram illustrating one column of the column AMP204. The column AMP204 includes switch elements 301, 302, and 307, input capacitors 303 and 304, an operational amplifier 305, and feedback capacitors 306 and 308.

[0029] An input capacitor and a feedback capacitor are connected to the operational amplifier 305. That is, a reference voltage is applied to the positive terminal of the operational amplifier 305, and feedback capacitors 306 and 308 are connected to the negative (-) terminal. Signals read from the pixel unit 203 are input to the input capacitors 303 and 304 via switch elements 301 and 302. The connection of the feedback capacitor 308 is controlled by a switch element 307.

[0030] Because the column AMP204 uses capacitance, the amplification factor of the amplifier is the input capacitance / feedback capacitance. Because the column AMP204 has two input capacitances, it can output two images with different gains applied. For example, with the switch element 301 connected and the switch elements 302 and 307 disconnected, it is possible to multiply the gain of the input capacitance 303 and the feedback capacitance 306 and output the image to the column ADC205. Also, with the switch element 301 disconnected and the switch elements 302 and 307 connected, it is possible to multiply the gain of the input capacitance 304, the feedback capacitance 306, and the feedback capacitance 308 and output the image.

[0031] 4 is a flowchart showing the image compositing process. This process selects images to be subjected to HDR compositing and performs HDR compositing. This process is realized by the CPU of the imaging device 100, which loads a program stored in ROM into RAM and executes it. This process starts when an instruction to start HDR compositing is issued by the user operating an operation unit (not shown).

[0032] 5 is a diagram showing an example of a plurality of images amplified with different gains for each exposure with a different exposure amount (exposure time). Image 501 (first image) is an appropriate image captured with an exposure amount (first exposure amount) calculated by exposure control unit 106 to provide an appropriate exposure. Image 502 (second image) is an overexposed image read out with a gain (second gain) higher than the gain (first gain) used when image 501 was read out (amplified). Images 501 and 502 have the same exposure amount. Note that the number of times images are captured with different exposure amounts may be three or more.

[0033] Image 503 (third image) is an underexposed image captured with an exposure (second exposure) that is less than the exposure (shorter exposure time) used to acquire image 501. Image 504 (fourth image) is an image read out with a higher gain than image 503 so that the brightness level is equivalent to that of image 501. The readout gains for images 503 and 504 are the same as those for images 501 and 502, which are the first and second gains, respectively. As a result, the brightness level of image 504 is equivalent to that of image 501, which is the proper image.

[0034] In step S401, the image sensor control unit 107 captures an image by a single exposure with the first exposure amount calculated by the exposure control unit 106. As a result, the image acquisition unit 103 as an acquisition unit acquires images 501 and 502 (FIG. 5) from the image sensor 102, which are two images with different gains applied.

[0035] In step S402, the image sensor control unit 107 captures an image by a single exposure with a second exposure amount that is less than the exposure amount used in step S401 using the exposure control unit 106. As a result, the image acquisition unit 103 as an acquisition unit acquires images 503 and 504 (FIG. 5) from the image sensor 102, which are two images with different gains applied.

[0036] In step S403, the image processing unit 104 develops the images acquired in steps S401 and S402. In step S404, the image processing unit 104 calculates the amount of misalignment between images captured and developed with different exposure amounts. An example of a method for calculating the amount of misalignment is described below.

[0037] First, the image processing unit 104 sets a plurality of blocks in a reference image (e.g., image 501). The image processing unit 104 preferably sets the blocks so that the sizes are the same for each block. Next, the image processing unit 104 sets search ranges in a target image (e.g., image 503) for which misalignment is to be corrected, at the same positions as the respective blocks in the reference image.

[0038] Next, the image processing unit 104 calculates a corresponding point in each search range of the target image that minimizes the sum of absolute differences (hereinafter referred to as SAD) in brightness between the target image and the block of the reference image.

[0039] The image processing unit 104 calculates the positional deviation as a vector from the center of the block in the reference image and the corresponding point described above. In calculating the corresponding point described above, the image processing unit 104 may use the sum of squared differences (hereinafter referred to as SSD) or the like in addition to SAD. Alternatively, the image processing unit 104 may use normalized cross correlation (hereinafter referred to as NCC) or the like.

[0040] After calculating the motion vectors for all blocks, the image processing unit 104 calculates affine coefficients using equation (1). The affine coefficients are matrices used in affine transformation, which combines linear transformation and translation.

[0041]

number

[0042] In equation (1), (x, y) represent the coordinates of the image before correction, and (x', y') represent the coordinates of the image after correction. Matrix A represents affine coefficients, calculated using the motion vectors obtained from each block. Note that a, b, d, and e in matrix A represent rotational movement parameters, c represents the horizontal movement amount, and f represents the vertical movement amount.

[0043] In step S405, the image processing unit 104 performs affine transformation based on the affine coefficient calculated in step S404, and corrects the image position of the target image by the amount of displacement.

[0044] In step S406, the image processing unit 104 performs moving object detection between the reference image and the comparison image after image position correction. In this case, as an example, an image acquired with a first exposure amount and a first gain may be used as the reference image, and an image acquired with a second exposure amount and a second gain after position correction may be used as the comparison image.

[0045] 6(a) to 6(e) are diagrams showing images and detection results in moving object detection. A reference image 601 shown in FIG. 6(a) is the correct image (e.g., image 501) acquired in step S401. A corrected image 602 shown in FIG. 6(b) is a comparison image in which the image position has been corrected for an image in which a moving object exists in the reference image 601. In the corrected image 602, an object exists that has moved slightly to the right relative to the reference image 601. A detection result 603 shown in FIG. 6(c) is a diagram showing the result of moving object detection performed between the reference image 601 and the corrected image 602.

[0046] The corrected image 604 shown in Fig. 6(d) is a comparison image in which the image position is corrected for an image in which no moving object exists in the reference image 601. The detection result 605 shown in Fig. 6(e) is a diagram showing the result of moving object detection between the reference image 601 and the corrected image 604.

[0047] The image processing unit 104 detects a moving object region from the reference image and the comparison image. The method for detecting this moving object region is not limited, but one possible method is to take the difference between the two images as follows. That is, the image processing unit 104 determines the presence or absence of a moving object based on the difference between the image signal of the reference image and the image signal of the comparison image. The difference Diff is calculated using color and brightness signals according to Equation 2.

[0048]

number

[0049] Here, Y represents the luminance signal, and U and V represent the color signals. Therefore, the difference Diff means color difference. By this method, detection results 603 and 605 are obtained.

[0050] In step S407, the image processing unit 104, which serves as a determination unit, determines whether or not a moving object exists between the reference image and the image to be compared, based on the moving object detection result executed in step S406. For example, it is determined from detection result 603 that a moving object exists between reference image 601 and corrected image 602. On the other hand, it is determined from detection result 605 that a moving object does not exist between reference image 601 and corrected image 604. If the image processing unit 104 determines that a moving object exists between the reference image and the image to be compared, it proceeds to step S408, and if it determines that a moving object does not exist, it proceeds to step S409.

[0051] The presence of a moving object is determined based on the moving object detection result, but a threshold value may also be used for this determination. That is, the image processing unit 104 may determine that a moving object is present when the difference Diff between the color signal of the reference image and the color signal of the comparison image exceeds a threshold value.

[0052] In step S408, the image processing unit 104, which functions as a selection unit, selects two images read out with different gains in a single exposure as images to be used for HDR compositing. The two images referred to here are, for example, images 501 and 502 acquired in step S401, but may also be images 503 and 504 acquired in step S402. In other words, the image processing unit 104 selects either images 501 and 502 acquired with appropriate exposure, or images 503 and 504 acquired with underexposure. Which image to select may be determined based on a compositing ratio (compositing proportion), as will be described next.

[0053] FIG. 7 is a diagram showing a blending ratio table. The horizontal axis of FIG. 7 represents reference luminance, and the vertical axis represents blending ratio. In this embodiment, a method is adopted in which the application ratio of a blending area is determined from the luminance distribution on the reference image. Image processing unit 104 sets the blending ratio of each image from the luminance value of the reference image based on the blending ratio table. For example, assume that image 501 is the reference image and its luminance value is Y0. At luminance value Y0, an appropriate image has a high blending ratio, so image processing unit 104 selects image 501 and image 502. Any of images 501 to 504 may be used as the reference image.

[0054] In this way, when it is determined that a moving object is present, the image acquired at which exposure level is selected as the image to be used for HDR compositing is determined according to the compositing ratio determined based on the brightness level of the reference image. This allows the selection of an image with a high proportion of HDR compositing depending on the scene. Note that the method for acquiring the brightness distribution is not limited to that obtained from the reference image, and it may also be acquired from a histogram acquired before shooting.

[0055] In step S409, the image processing unit 104, which functions as a selection unit, selects images captured with different exposures as images to be used for HDR compositing. The image processing unit 104 selects at least one of images 501 and 502 acquired with appropriate exposure and at least one of images 503 and 504 acquired with underexposure. Note that, because images 501 and 504 have equivalent brightness levels, one of these may be excluded from the selection. For example, the image processing unit 104 may select images 501 and 502 acquired in step S401 and image 503 acquired in step S402. Note that when excluding one of the images with equivalent brightness levels from the selection, the image that is more suitable as a compositing target, such as one with less noise depending on the gain distribution during exposure and readout, may be included in the selection of images.

[0056] In this way, an image to be used for image synthesis is selected from the multiple images output from the image sensor 102 based on the result of determining whether or not there is a moving object.

[0057] In step S410, the image processing unit 104, which serves as a synthesis unit, performs HDR image synthesis using the images selected in step S408 or step S409, and ends the processing shown in Fig. 4. In this way, by switching the selection of images to be used for HDR synthesis to images that do not cause unnatural depictions due to moving objects, it is possible to prevent unnatural depictions due to moving objects.

[0058] Next, a case where there are three or more types of exposure amounts will be considered with reference to FIGS.

[0059] Fig. 8 shows examples of multiple images amplified with different gains for each of three different exposures. Fig. 8 shows an example of images captured with three different exposures: optimal exposure (first exposure), underexposure (second exposure), and overexposure (third exposure), and two images read out at each exposure using low gain (first gain) and high gain (second gain).

[0060] Image 701 (first image) is an appropriate image captured with an appropriate exposure amount and read out with low gain. Image 702 (second image) is an image captured with an appropriate exposure amount and read out with high gain. Image 703 (third image) is an image captured with underexposure and read out with low gain. Image 704 (fourth image) is an image captured with underexposure and read out with high gain.

[0061] Image 705 (fifth image) is an image captured with overexposure and read out with low gain. Image 706 (sixth image) is an image captured with overexposure and read out with high gain. Images 701 and 704 have the same brightness level. Images 702 and 705 have the same brightness level.

[0062] When comparing two images read out using different gains, the image read out using the higher gain may have increased noise due to amplification. Therefore, in scenes where moving objects do not cause unnatural depictions, it is best to select an overexposed image with increased exposure by extending the exposure time, rather than an image read out using high gain.

[0063] When generating an image with a wide dynamic range using HDR compositing, one method is to use an overexposed image to represent the crushed shadow areas of an image captured with a correct exposure, and an underexposed image to represent the blown-out highlight areas. However, when HDR compositing two images read with different gains, there are areas where the gradation cannot be interpolated. For example, in HDR compositing images 701 and 702, the images to be combined do not contain an underexposed image, so the blown-out highlight areas cannot be properly represented. On the other hand, in HDR compositing images 703 and 704, the images to be combined do not contain an overexposed image, so the blackout areas cannot be properly represented. Therefore, it is desirable to select the images to be used for compositing by taking these factors into consideration comprehensively.

[0064] 9(a) to 9(c) are diagrams showing the results of detecting a moving object between images. Fig. 9 explains the relationship between the results of detecting a moving object and the selection of images to be used for HDR compositing. Take the example of a case where multiple images equivalent to images 701 to 706 shown in Fig. 8 have been acquired.

[0065] 9(a) to 9(c) show only images 701 to 706 that correspond to images 701, 703, and 705 read out at low gain. Also shown are moving object detection results (91 to 96) between the appropriate exposure image (701) and the underexposed images (703A and 703B), and between the underexposed image and the overexposed image (705A and 705B). The detection results 91 to 96 are shown in the same manner as the detection results 603 and 605 shown in FIG. 6.

[0066] Among the images equivalent to image 703, images 703A and 703B have the same exposure and gain conditions but different object positions. Among the images equivalent to image 705, images 705A and 705B have the same exposure and gain conditions but different object positions.

[0067] In the example shown in Figure 9(a), it is determined that there is no moving object between the properly exposed image 701 and the underexposed image 703A (detection result 91), and it is determined that there is a moving object between the underexposed image 703A and the overexposed image 705A (detection result 92).

[0068] Since it is determined that there is a moving object between images 703A and 705A, if the underexposed images (703A, 704) and the overexposed images (705A, 706) are selected for HDR compositing, there is a risk of the moving object appearing unnaturally. On the other hand, since it is determined that there is no moving object between images 701 and 703A, even if the properly exposed images (701, 702) and the underexposed images (703A, 704) are selected, the moving object will not appear unnaturally. Therefore, the image processing unit 104 selects images 701, 702, and 703A for HDR compositing. Note that the image processing unit 104 may also select images 701, 702, 703A, and 704.

[0069] In the example shown in Figure 9(b), it is determined that there is a moving object between the properly exposed image 701 and the underexposed image 703B (detection result 93), and it is determined that there is no moving object between the underexposed image 703B and the overexposed image 705A (detection result 94).

[0070] Because it was determined that there was a moving object between images 701 and 703B, if the properly exposed images (701, 702) and the underexposed images (703B, 704) were selected for HDR compositing, there is a risk that the moving object would result in an unnatural depiction. On the other hand, because it was determined that there was no moving object between images 703B and 705A, even if the underexposed images (703B, 704) and the overexposed images (705A, 706) were selected, the depiction would not be unnatural.

[0071] Therefore, the image processing unit 104 selects images 703B, 704, and 705A for HDR compositing. If image 706 were added, the range expansion would be unbalanced between the underexposure and overexposure sides, so it is excluded from the selection. However, image 706 may also be added to the selection. Note that in the example of FIG. 9(b), if underexposure is considered to be the first exposure amount and overexposure is considered to be the second exposure amount, image 703B would be the first image, image 704 would be the second image, image 705A would be the third image, and image 706 would be the fourth image.

[0072] In the example shown in Figure 9(c), it is determined that there is no moving object between the properly exposed image 701 and the underexposed image 703A (detection result 95), and it is determined that there is no moving object between the underexposed image 703A and the overexposed image 705B (detection result 96).

[0073] In this case, even if the properly exposed images (701, 702), the underexposed images (703A, 704), and the overexposed images (705B, 706) are selected, the image will not be unnatural. However, it is desirable to avoid an increase in noise due to amplification. Therefore, the image processing unit 104 does not select images 702, 704, and 706, but selects images 701, 703A, and 705B. This reduces noise. Alternatively, all of the images 701 to 706 may be selected.

[0074] In this way, even when obtaining an appropriate image, an underexposed image, and an overexposed image with different exposures to perform compositing over a wide dynamic range, the images to be used for compositing can be appropriately selected based on the detection results of the moving object.

[0075] According to this embodiment, images to be used for image synthesis are selected based on the determination result of the presence or absence of a moving object between images acquired for each exposure at different exposure amounts. First, a first image amplified with a first gain and a second image amplified with a second gain are acquired by exposure at a first exposure amount, and a third image amplified with the first gain and a fourth image amplified with the second gain are acquired by exposure at a second exposure amount. Then, if it is determined that a moving object is present between the first and third images, either the first and second images or the third and fourth images are selected. This reduces the sense of incongruity caused by the moving object. On the other hand, if it is determined that no moving object is present between the first and third images, at least one of the first and second images and at least one of the third and fourth images are selected. This reduces the sense of incongruity caused by the moving object and ensures a wide dynamic range.

[0076] In particular, since the system automatically switches between selecting an image acquired with the same exposure but different gains and selecting an image acquired with different exposures, it is possible to provide an image with a wide dynamic range and little discomfort without the user being aware of it.

[0077] Furthermore, when images are acquired with three or more exposures, even if it is determined that a moving object is present, it is possible to select images so that the range expansion is balanced between the underexposure and overexposure sides (Figure 9(b)).

[0078] Furthermore, even if images are acquired with three or more exposures, if it is determined that there is no moving object, it is possible to select an image with lower noise (Figure 9(c)).

[0079] The present invention can also be applied to a mobile device with a built-in imaging element, a network camera capable of capturing images, etc. An imaging device to which the present invention is applied may also be called an image processing device.

[0080] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to these specific embodiments, and various forms within the scope of the invention that do not deviate from the gist of the invention are also included in the present invention.

[0081] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a non-transitory storage medium, and having one or more processors in the computer of the system or device read and execute the program. The above program and the storage medium storing the program constitute the present invention. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. [Explanation of symbols]

[0082] 100 Imaging device 102 Image sensor 103 Image acquisition unit 104 Image processing section 107 Image sensor control unit

Claims

1. an acquisition means for acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination means for determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection means for selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result by the determination means, the acquiring means acquires a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, and acquires a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount; The selection means When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; An image processing device characterized by selecting the first image, the second image, and the third image when the first exposure amount is greater than the second exposure amount, the second gain is greater than the first gain, and it is determined that there is no moving object between the first image and the third image.

2. An acquisition means for acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination means for determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection means for selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result by the determination means, the acquiring means acquires a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, and acquires a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount; The selection means When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; An image processing device characterized by selecting the first image, the second image, and the third image when the second exposure amount is greater than the first exposure amount, the second gain is greater than the first gain, and it is determined that there is no moving object between the first image and the third image.

3. An acquisition means for acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination means for determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection means for selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result by the determination means, the acquiring means acquires a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, acquires a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount, and acquires a fifth image amplified with the first gain and a sixth image amplified with the second gain by exposure at a third exposure amount different from the first and second exposure amounts, The selection means When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; An image processing device characterized by selecting the first image, the third image, and the fifth image when the first exposure amount is greater than the second exposure amount, the third exposure amount is greater than the first exposure amount, the second gain is greater than the first gain, it is determined that there is no moving object between the first image and the second image, and it is determined that there is no moving object between the second image and the third image.

4. The image processing device according to any one of claims 1 to 3, characterized in that when it is determined that the moving object is present between the first image and the third image, the selection means determines whether to select the first image and the second image, or the third image and the fourth image, depending on a synthesis ratio determined based on the brightness level of one of the first, second, third, and fourth images.

5. The image processing device according to any one of claims 1 to 4, characterized in that the determination means determines the presence or absence of the moving object between the reference image and the comparison image based on the difference between the image signal of the reference image and the image signal of the comparison image among the plurality of acquired images.

6. The image processing device according to claim 5, characterized in that the determination means determines that the moving object is present between the reference image and the comparison image when the difference between the color signal of the reference image and the color signal of the comparison image exceeds a threshold value.

7. 7. The image processing apparatus according to claim 1, wherein the acquisition means acquires the plurality of images from a sensor that outputs images amplified with different gains by a single exposure.

8. 8. The image processing apparatus according to claim 1, further comprising a combining unit for combining the images selected by the selecting unit.

9. acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination step of determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection step of selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result of the determination step, the acquiring step includes acquiring a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, and acquiring a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount; The selecting step includes: When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; A control method for an image processing device, characterized in that when the first exposure amount is greater than the second exposure amount, the second gain is greater than the first gain, and it is determined that there is no moving object between the first image and the third image, the first image, the second image, and the third image are selected.

10. An acquisition step of acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination step of determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection step of selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result of the determination step, the acquiring step includes acquiring a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, and acquiring a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount; The selecting step includes: When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; A control method for an image processing device, characterized in that when the second exposure amount is greater than the first exposure amount, the second gain is greater than the first gain, and it is determined that there is no moving object between the first image and the third image, the first image, the second image, and the third image are selected.

11. An acquisition step of acquiring a plurality of images amplified with different gains for each exposure at different exposure doses; a determination step of determining whether or not a moving object exists between images acquired for each exposure with a different exposure amount; a selection step of selecting an image to be used for image synthesis from the plurality of acquired images based on a determination result of the determination step, the acquiring step includes acquiring a first image amplified with a first gain and a second image amplified with a second gain by exposure at a first exposure amount, acquiring a third image amplified with the first gain and a fourth image amplified with the second gain by exposure at a second exposure amount different from the first exposure amount, and acquiring a fifth image amplified with the first gain and a sixth image amplified with the second gain by exposure at a third exposure amount different from the first and second exposure amounts; The selecting step includes: When it is determined that the moving object is present between the first image and the third image, selecting either the first image and the second image or the third image and the fourth image; when it is determined that the moving object is not present between the first image and the third image, selecting at least one of the first image and the second image and at least one of the third image and the fourth image; A control method for an image processing device, characterized in that when the first exposure amount is greater than the second exposure amount, the third exposure amount is greater than the first exposure amount, the second gain is greater than the first gain, it is determined that there is no moving object between the first image and the second image, and it is determined that there is no moving object between the second image and the third image, the first image, the third image, and the fifth image are selected.

12. An image processing device according to any one of claims 1 to 8; and a sensor that outputs an image amplified with different gains by a single exposure.

Images