Imaging device and imaging method

Abstract

The present invention provides an imaging device that can acquire images with a wider dynamic range while suppressing flicker. [Solution] The system comprises an imaging unit 13 that receives light from a subject and converts it into an electrical signal for each pixel, and alternately takes long-exposure and short-exposure images; a transmittance adjustment unit 12 provided between the subject and the imaging unit 13, which controls the transmittance of light for each pixel of the imaging unit 13; an analysis unit 15 that determines a control pixel, which is a pixel whose transmittance is controlled in the transmittance adjustment unit 12, based on the brightness of each pixel in at least one of the long-exposure and short-exposure images taken by the imaging unit 13; a control unit 16 that controls the transmittance of the control pixel in the transmittance adjustment unit 12 based on the control pixel determined by the analysis unit 15; and a synthesis unit 14 that synthesizes the long-exposure and short-exposure images taken by the imaging unit 13 to generate a composite image.

Description

Technical Field

[0001] The present disclosure relates to an imaging apparatus and an imaging method.

Background Art

[0002] In general, an image is captured when an imaging element included in an imaging apparatus receives light from a subject and converts the light into an electric signal. As a technique for expanding the dynamic range of an image captured by such an imaging apparatus, there is known High Dynamic-range Rendering composition (hereinafter referred to as HDR composition), which is a technique of capturing and compositing images with different exposure times that are the durations for which the imaging element receives light during imaging, that is, a technique of capturing and compositing a long-exposure image and a short-exposure image. By compositing the long-exposure image and the short-exposure image through HDR composition, a composite image with reduced blown-out highlights and blocked-up shadows can be obtained. However, when the exposure time is short, that is, when capturing a short-exposure image, flicker may occur due to a light source or the like within the imaging range, so flicker occurs in the short-exposure image, and the image quality of the composite image obtained by HDR composition may deteriorate.

[0003] As a method of suppressing the occurrence of flicker in a short-exposure image, for example, there is a method of setting the exposure time to an integral multiple of 1 / 2 of the flicker occurrence period. However, in this case, since the exposure time is increased in accordance with the flicker occurrence period, blown-out highlights are likely to occur, and the image quality of the composite image obtained by HDR composition deteriorates. Accordingly, as a technique for expanding the dynamic range of an image while suppressing the occurrence of flicker, for example, Patent Document 1 discloses an imaging apparatus in which a liquid crystal ND filter capable of adjusting light transmittance is provided between a subject and an imaging element, and the light transmittance is changed between when a long-exposure image is captured and when a short-exposure image is captured to adjust the amount of light received by the imaging element.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-197613 [Overview of the project] [Problems that the invention aims to solve]

[0005] In the above imaging device, since long-exposure and short-exposure images are captured using a single image sensor, it is necessary to change the transmittance of the liquid crystal ND filter for each shot, and the change in the transmittance of the liquid crystal ND filter must be completed before the next shot begins. To explain with a specific example, when capturing moving images, if a frame rate of 30fps is used to continuously generate composite images by HDR synthesis, the shooting cycle is 16.7 milliseconds. Assuming that the light source causing flicker is illumination with a power frequency of 50Hz, in order to suppress the occurrence of flicker in short-exposure images, the exposure time for capturing short-exposure images must be at least 10 milliseconds.

[0006] In this case, since long-exposure and short-exposure images are captured alternately according to the shooting cycle, the time from the end of long-exposure image capture to the start of short-exposure image capture is a maximum of 6.7 milliseconds. On the other hand, since the time required for the transmittance of a liquid crystal ND filter to change is generally about 10 milliseconds, if the transmittance of the liquid crystal ND filter is changed starting after the end of long-exposure image capture, the transmittance of the liquid crystal ND filter will not have finished changing by the time the short-exposure image capture begins. Thus, when the transmittance of the liquid crystal ND filter is changed for each long-exposure and short-exposure image capture, shooting starts while the transmittance of the liquid crystal ND filter is still changing, which presents a problem in that the dynamic range of the captured image cannot be sufficiently expanded.

[0007] This disclosure is made to solve the above-mentioned problems and provides an imaging device that can acquire images with a wider dynamic range while suppressing flicker. [Means for solving the problem]

[0008] The imaging device according to this disclosure is an imaging device that generates a composite image by combining a long-exposure image taken with a first exposure time and a short-exposure image taken with a second exposure time shorter than the first exposure time, and comprises: an imaging unit that receives light from a subject and converts it into an electrical signal for each pixel, and alternately takes long-exposure images and short-exposure images; a transmittance adjustment unit provided between the subject and the imaging unit, which controls the transmittance of light for each pixel of the imaging unit; an analysis unit that determines a control pixel, which is a pixel whose transmittance is controlled in the transmittance adjustment unit, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image taken by the imaging unit; a control unit that controls the transmittance of the control pixel in the transmittance adjustment unit based on the control pixel determined by the analysis unit; and a synthesis unit that generates a composite image by combining the long-exposure image and the short-exposure image taken by the imaging unit. [Effects of the Invention]

[0009] The imaging device according to this disclosure determines a control pixel based on the brightness of each pixel in at least one of a long-exposure image and a short-exposure image captured by the imaging unit, and controls the transmittance of the control pixel in the transmittance adjustment unit. Therefore, the amount of light received by the imaging unit can be adjusted by controlling the transmittance of the transmittance adjustment unit in accordance with the brightness and darkness in the short-exposure image or the long-exposure image. Consequently, by controlling the transmittance of the transmittance adjustment unit even while capturing a short-exposure image or a long-exposure image, time can be secured for the transmittance of the transmittance adjustment unit to change. This allows imaging to be performed using a transmittance adjustment unit whose transmittance is controlled in accordance with the brightness and darkness in the short-exposure image or the long-exposure image, thereby enabling the acquisition of an image with a wider dynamic range. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration of the imaging system according to Embodiment 1. [Figure 2] This figure shows an example of the configuration of the imaging device according to Embodiment 1. [Figure 3]This is a diagram showing the order in which a long-exposure image and a short-exposure image are captured by the imaging unit according to Embodiment 1. [Figure 4] This is a schematic diagram showing the first pixel and the control pixel according to Embodiment 1. [Figure 5] This is a configuration example of a computer system that implements the imaging apparatus according to Embodiment 1. [Figure 6] This is a flowchart showing the operation of the imaging apparatus according to Embodiment 1. [Figure 7] This is a flowchart showing the operation of the imaging apparatus according to Embodiment 1. [Figure 8] This is a diagram showing a configuration example of an imaging apparatus according to Embodiment 2. [Figure 9] This is a schematic diagram showing the rotational movement amount according to Embodiment 2. [Figure 10] This is a schematic diagram showing the first pixel and the third pixel according to Embodiment 2. [Figure 11] This is a flowchart showing the operation of the imaging apparatus according to Embodiment 2. [Figure 12] This is a diagram showing a configuration example of an imaging apparatus according to Embodiment 3. [Figure 13] This is a schematic diagram showing the subject movement amount according to Embodiment 3. [Figure 14] This is a schematic diagram showing the first pixel and the fourth pixel according to Embodiment 3. [Figure 15] This is a flowchart showing the operation of the imaging apparatus according to Embodiment 3. [Mode for Carrying Out the Invention]

[0011] Hereinafter, the embodiments will be described in detail based on the drawings. Note that the embodiments described below are illustrative. In addition, the respective embodiments can be implemented in appropriate combinations.

[0012] Embodiment 1. The imaging apparatus 10 according to Embodiment 1 will be described with reference to FIGS. 1 to 7.

[0013] FIG. 1 shows a configuration example of an imaging system 1 including an imaging apparatus 10 according to Embodiment 1. The imaging system 1 includes the imaging apparatus 10, a HUB 2, a decoder 3, a display apparatus 4, an operation terminal 5, and a recording apparatus 6. The imaging apparatus 10 captures an image, compresses the captured image, and outputs the compressed image as image data. Here, the image captured by the imaging apparatus 10 is a still image or a moving image. Details of the imaging apparatus 10 will be described later with reference to FIG. 2.

[0014] The HUB 2 is connected to the imaging apparatus 10, the decoder 3, the operation terminal 5, and the recording apparatus 6 via a Local Area Network cable (hereinafter referred to as a LAN cable), and transmits the image data output by the imaging apparatus 10 to the decoder 3, the operation terminal 5, and the recording apparatus 6.

[0015] The decoder 3 acquires the image data output by the imaging apparatus 10 from the HUB 2, decodes the image data, and outputs the decoded image data to the display apparatus 4. The display apparatus 4 displays the image captured by the imaging apparatus 10, which is the image data decoded by the decoder 3. The recording apparatus 6 acquires and stores the image data output by the imaging apparatus 10 from the HUB 2. The image data stored by the recording apparatus 6 is output to the display apparatus 4 via the HUB 2 and the decoder 3, and can be displayed on the display apparatus 4.

[0016] The operation terminal 5 receives input of operation instructions for the imaging system 1. Examples of the operation instructions for the imaging system 1 include imaging by the imaging apparatus 10, display of an image on the display apparatus 4, recording of image data to the recording apparatus 6, and reading of image data from the recording apparatus 6.

[0017] Note that although an example using communication via a LAN cable as the communication means between the imaging apparatus 10, the display apparatus 4, the operation terminal 5, and the recording apparatus 6 has been described, the present invention is not limited to this, and any means may be used as long as it can transmit image data between the imaging apparatus 10, the display apparatus 4, the operation terminal 5, and the recording apparatus 6. Further, the imaging system 1 is not limited to the above-described configuration, and any system may be used as long as it can display or record an image captured by the imaging apparatus 10.

[0018] Figure 2 shows an example of the configuration of the imaging device 10 according to Embodiment 1. The imaging device 10 generates a composite image by combining a long-exposure image taken with a first exposure time and a short-exposure image taken with a second exposure time shorter than the first exposure time. The imaging device 10 includes a lens 11, a transmittance adjustment unit 12, an imaging unit 13, a synthesis unit 14, an analysis unit 15, and a control unit 16.

[0019] The lens 11 focuses light from the subject onto the imaging unit 13. The lens 11 is provided with an aperture to adjust the amount of light focused from the lens 11 onto the imaging unit 13. The transmittance adjustment unit 12 is provided between the lens 11, into which light from the subject enters, and the imaging unit 13, and controls the transmittance of light for each pixel of the imaging unit 13, which will be described later. Here, the statement that the transmittance of the transmittance adjustment unit 12 is controlled for each pixel includes cases where the transmittance of some pixels in the transmittance adjustment unit 12 changes, while the transmittance of other pixels remains unchanged. The transmittance adjustment unit 12 may also be controlled for each group of pixels consisting of multiple pixels. The transmittance adjustment unit 12 is composed of, for example, liquid crystal material arranged for each pixel, but is not limited to this as long as it can control the transmittance of light for each pixel or each group of pixels. The control of the transmittance adjustment unit 12 is performed by the control unit 16, which will be described later.

[0020] The imaging unit 13 receives light from a subject focused by the lens 11, converts it into an electrical signal for each pixel, and alternately captures long-exposure images and short-exposure images. The imaging unit 13 includes, for example, an image sensor that receives light from a subject and converts it into an electrical signal, and an image processing unit that generates an image based on the electrical signal converted by the image sensor.

[0021] The synthesis unit 14 generates a composite image by combining the long-exposure image and the short-exposure image captured by the imaging unit 13. Based on the long-exposure image and the short-exposure image, the synthesis unit 14 determines the first composite pixels, which are pixels from the short-exposure image used in generating the composite image, and the second composite pixels, which are pixels from the long-exposure image used in generating the composite image, and generates a composite image based on the determined first composite pixels and second composite pixels. Here, the first composite pixels are bright areas that include pixels where overexposure may occur in the short-exposure image, and the second composite pixels are dark areas that include pixels where underexposure may occur in the long-exposure image.

[0022] As an example of how the compositing unit 14 determines the first and second composite pixels, the compositing unit 14 selects pixels in the short-exposure image whose individual tonal values ​​are equal to or greater than a preset first compositing threshold as the first composite pixels, and pixels in the long-exposure image whose individual tonal values ​​are equal to or greater than a preset second compositing threshold as the second composite pixels. By combining the long-exposure image and the short-exposure image based on the first and second composite pixels determined in this way, a composite image with reduced overexposure and underexposure can be generated.

[0023] Furthermore, the method by which the compositing unit 14 determines the first composite pixel and the second composite pixel is not limited to the method described above. Any method that can combine the long-exposure image and the short-exposure image based on the determined first and second composite pixels to produce a composite image with reduced overexposure and underexposure is acceptable. For example, the compositing unit 14 may determine the first composite pixel based on the brightness value of each pixel in the short-exposure image, rather than the grayscale value of each pixel in the short-exposure image. Also, the method by which the compositing unit 14 combines the long-exposure image and the short-exposure image to produce a composite image is not limited to the method described above. Various existing techniques used in HDR compositing may be utilized.

[0024] The sequence of events in which the imaging unit 13 captures long-exposure and short-exposure images and the synthesis unit 14 generates a composite image will be explained using Figure 3. Figure 3 shows the sequence of events in which the imaging unit 13 captures long-exposure and short-exposure images. In Figure 3, the long-exposure intervals in which long-exposure images are captured during the first exposure time are indicated by X1, X2, and X3, and the short-exposure intervals in which short-exposure images are captured during the second exposure time are indicated by Y1, Y2, and Y3. The points in time when long-exposure images are generated are indicated by x1, x2, and x3, and the points in time when short-exposure images are generated are indicated by y1, y2, and y3.

[0025] When generating composite images continuously, long-exposure and short-exposure images are captured periodically and alternately. Examples of cases where composite images are generated continuously include capturing still images or capturing moving images in succession.

[0026] The compositing unit 14 generates a composite image by combining long-exposure and short-exposure images that are captured alternately. When a short-exposure image is generated at y1 in Figure 3, the compositing unit 14 generates a composite image by combining the long-exposure image generated at x1 with the short-exposure image generated at y1. Similarly, when a short-exposure image is generated at y2, the compositing unit 14 generates a composite image by combining the long-exposure image generated at x2 with the short-exposure image generated at y2. In this way, a composite image is continuously generated by combining the captured long-exposure and short-exposure images each time a short-exposure image is generated.

[0027] Returning to Figure 2, the analysis unit 15 calculates a measured brightness value indicating the brightness of the short-exposure image based on the short-exposure image, and determines the first transmittance based on the measured brightness value and a preset brightness target value. Here, the value indicating the brightness of each pixel in the short-exposure image can be, for example, the grayscale value or luminance value for each pixel in the short-exposure image.

[0028] Furthermore, the analysis unit 15 calculates image information indicating the first pixel, which is the pixel used in the short-exposure image generation when generating the composite image, based on the long-exposure image and the short-exposure image, and determines the control pixel based on the calculated image information. Here, the first pixel indicates a bright area in the short-exposure image that may contain pixels where overexposure occurs, and the first pixel calculated by the analysis unit 15 coincides with the first composite pixel determined by the composite unit 14.

[0029] The determination of the first transmittance and control pixel by the analysis unit 15, and the control of the transmittance adjustment unit 12 by the control unit 16 (described later), are performed each time a short-exposure image is generated. Specifically, in Figure 3, based on the long-exposure image generated in x1 and the short-exposure image generated in y1, the transmittance of the transmittance adjustment unit 12 is controlled in accordance with the capture of the short-exposure image in the next short-exposure section Y2, from the time y1 when the short-exposure section Y1 ends until the next short-exposure section Y2 begins. Furthermore, based on the long-exposure image generated in x2 and the short-exposure image generated in y2, the transmittance of the transmittance adjustment unit 12 is controlled in accordance with the capture of the short-exposure image in the next short-exposure section Y3, from the time y2 when the short-exposure section Y2 ends until the next short-exposure section Y3 begins.

[0030] Therefore, when the imaging device 10 is continuously generating composite images, the analysis unit 15 determines the first transmittance and control pixels when the short exposure interval ends. Also, at the time the analysis unit 15 determines the first transmittance and control pixels, the transmittance of the transmittance adjustment unit 12 is controlled to match the capture of the short exposure image in the immediately preceding short exposure interval.

[0031] The method by which the analysis unit 15 determines the first transmittance and the method by which the analysis unit 15 determines the control pixel, based on the long-exposure image generated at x2 and the short-exposure image generated at y2 shown in Figure 3, will be described. Here, the first pixel determined by the analysis unit 15 based on the long-exposure image generated at x1 and the short-exposure image generated at y1 is referred to as the previous first pixel, and the first transmittance is referred to as the previous first transmittance.

[0032] The method by which the analysis unit 15 determines the first transmittance will now be described. The analysis unit 15 calculates the measured brightness value by integrating the grayscale values ​​of each pixel in the short-exposure image, and determines the first transmittance based on the measured brightness value and a preset brightness target value. If the measured brightness value is greater than the brightness target value, the analysis unit 15 determines the first transmittance to be lower than the previous first transmittance. Also, if the measured brightness value is less than the brightness target value, the analysis unit 15 determines the first transmittance to be higher than the previous first transmittance.

[0033] The method for calculating the measured brightness is not limited to the method described above; any value indicating the brightness of a short-exposure image can be used as the measured brightness. For example, the measured brightness can be calculated by integrating the luminance values ​​of each pixel in a short-exposure image. Alternatively, statistical quantities such as the average, maximum, minimum, or median of the grayscale values ​​for each pixel in a short-exposure image can also be used as the measured brightness.

[0034] Thus, once the first transmittance is determined based on the brightness of each pixel in the short-exposure image, the control unit 16, described later, controls the transmittance of the transmittance adjustment unit 12 so that the transmittance of the first pixel in the transmittance adjustment unit 12 becomes the first transmittance. Therefore, compared to the case where the transmittance of the first pixel in the transmittance adjustment unit 12 is controlled to a predetermined value, the transmittance of the transmittance adjustment unit 12 is controlled to be suitable for the brightness of the short-exposure image, making it possible to obtain short-exposure images and composite images with a wider dynamic range.

[0035] Furthermore, as another method for calculating the measured brightness value, for example, the brightness value may be calculated by integrating the tonal values ​​of each pixel in a portion of a short-exposure image. Specifically, the brightness value may be calculated by integrating the tonal values ​​of each pixel in a short-exposure image that are equal to or greater than a predetermined value. In this way, by calculating the brightness value using values ​​indicating the brightness of each pixel in a short-exposure image that are equal to or greater than a predetermined value, it is possible to calculate the brightness value while ignoring dark areas in the short-exposure image where black crushing may occur due to backlighting, etc.

[0036] Therefore, when the analysis unit 15 determines the first transmittance based on the measured brightness value calculated while ignoring dark areas in the short-exposure image where black crushing may occur due to backlighting, etc., the control unit 16, described later, controls the transmittance of the transmittance adjustment unit 12 so that the transmittance of the first pixel in the transmittance adjustment unit 12 becomes the first transmittance. As a result, the transmittance of the first pixel in the transmittance adjustment unit 12 is controlled to be suitable for the brightness other than the dark areas in the short-exposure image, so that a short-exposure image and a composite image with a wider dynamic range can be obtained.

[0037] Another specific example is to calculate the first pixel before calculating the measured brightness value, and then calculate the measured brightness value based on the brightness of the first pixel in the short-exposure image. In this case, the measured brightness value is calculated based on the brightness of the first pixel in the short-exposure image, and the first transmittance is determined based on the measured brightness value and the brightness target value. Once the analysis unit 15 determines the first transmittance based on the brightness of the first pixel in this way, the control unit 16, which will be described later, controls the transmittance of the transmittance adjustment unit 12 so that the transmittance of the first pixel in the transmittance adjustment unit 12 becomes the first transmittance.

[0038] Therefore, since the transmittance of the first pixel in the transmittance adjustment unit 12 is controlled to be suitable for the brightness of the first pixel in the short-exposure image, it is possible to obtain short-exposure images and composite images with a wider dynamic range.

[0039] Furthermore, the method for determining the first transmittance is not limited to the method described above. After the control pixels of the transmittance adjustment unit 12 are controlled based on the determined first transmittance, the first transmittance should be determined such that the measured brightness value in the captured short-exposure image approaches the brightness target value. In addition, the first transmittance may be a value that is set in advance to be lower than the transmittance specified value described later.

[0040] Next, the method by which the analysis unit 15 determines the control pixel will be described. Based on the long-exposure image and the short-exposure image, the analysis unit 15 calculates image information indicating the first pixel, which is the pixel used in the short-exposure image for generating the composite image, and determines the control pixel based on the calculated image information.

[0041] The method for calculating the first pixel based on the long-exposure and short-exposure images can be determined according to the method used by the synthesis unit 14 to determine the first composite pixel. Therefore, similar to the synthesis unit 14, the analysis unit 15 designates a pixel as the first pixel if the grayscale value of each pixel in the short-exposure image is equal to or greater than a preset first synthesis threshold.

[0042] One method for determining the control pixel based on the image information indicating the calculated first pixel is to determine it based on the first transmittance, the first pixel, the previous first pixel, and the previous first transmittance. Here, the transmittance of the control pixel in the transmittance adjustment unit 12 is controlled such that the transmittance of the pixel corresponding to the first pixel in the transmittance adjustment unit 12 becomes the first transmittance, and the transmittance of the pixels that do not correspond to the first pixel in the transmittance adjustment unit 12 becomes a preset transmittance value. Furthermore, at the time the analysis unit 15 determines the control pixel, the transmittance adjustment unit 12 is in a state where the transmittance of the pixel corresponding to the previous first pixel in the transmittance adjustment unit 12 is the previous first transmittance, and the transmittance of the pixels that do not correspond to the previous first pixel in the transmittance adjustment unit 12 is the preset transmittance value.

[0043] Here, the specified transmittance value is defined as the maximum transmittance value that can be set in the transmittance adjustment unit 12. However, the specified transmittance value is not limited to this; it is sufficient if it is greater than the minimum transmittance value that can be set in the transmittance adjustment unit 12.

[0044] Figure 4 schematically shows the first pixel, the preceding first pixel, and the control pixel in the captured image. The area A enclosed by the solid line represents the first pixel, and the area B enclosed by the dashed line represents the preceding first pixel.

[0045] The transmittance adjustment unit 12 controls the transmittance of the pixel corresponding to the first pixel to be the first transmittance, and the transmittance of the pixels not corresponding to the first pixel to be a preset transmittance value. Therefore, in Figure 4, the transmittance adjustment unit 12 controls the transmittance to be the first transmittance within the range of A, which is the first pixel, and controls the transmittance to be the preset transmittance value outside the range of A. On the other hand, at the time when the analysis unit 15 determines the control pixel, the transmittance adjustment unit 12 determines that the transmittance within the range of B is the immediately preceding first transmittance, and the transmittance outside the range of B is the preset transmittance value.

[0046] Therefore, in range C, which is within range A and outside range B, the transmittance adjustment unit 12 is controlled so that the transmittance changes from the specified transmittance value to the first transmittance. Also, in range D, which is within range A and within range B, the transmittance adjustment unit 12 is controlled so that the transmittance changes from the previous first transmittance to the first transmittance. Also, in range E, which is outside range A and within range B, the transmittance adjustment unit 12 is controlled so that the transmittance changes from the previous first transmittance to the specified transmittance value. Also, in range A and outside range B, the transmittance adjustment unit 12 is controlled so that the transmittance remains at the specified transmittance value.

[0047] If the immediately preceding first transmittance is different from the first transmittance, the analysis unit 15 determines the pixels included in ranges C, D, and E as control pixels. Also, if the immediately preceding first transmittance is the same as the first transmittance, the transmittance of the transmittance adjustment unit 12 in range D will remain unchanged from the immediately preceding first transmittance, so the analysis unit 15 determines the pixels included in ranges C and E as control pixels.

[0048] The method for determining the control pixel is not limited to the method described above; the control pixel can be determined based on the brightness of each pixel in the short-exposure image captured by the imaging unit 13. For example, the control pixel is determined such that the transmittance of the transmittance adjustment unit 12 corresponding to the first pixel, which is a bright area in the captured image that may contain pixels where overexposure may occur, can be made lower than the transmittance of other areas.

[0049] Furthermore, in the example described above, the first pixel calculated by the analysis unit 15 was assumed to be the same as the first composite pixel determined by the synthesis unit 14. However, the first pixel only needs to represent a bright area in the captured image that may contain pixels where overexposure occurs. For example, the first pixel may be calculated as any pixel whose brightness in a short-exposure image is equal to or greater than a preset first threshold. Here, the first threshold does not have to be the same as the first composite threshold used by the synthesis unit 14. For example, the first threshold may be set to the minimum brightness value at which overexposure occurs in a short-exposure image, thereby calculating the first pixel as any pixel at which overexposure occurs in a short-exposure image.

[0050] Furthermore, while a method for determining control pixels to lower the transmittance of the transmittance adjustment unit 12 in bright areas containing pixels where overexposure may occur in short-exposure images has been described, control pixels may also be determined so as to increase the transmittance of the transmittance adjustment unit 12 in dark areas containing pixels where underexposure may occur in short-exposure images. In this case, for example, pixels whose brightness in the short-exposure image is below a preset second threshold are calculated as the second pixels. Here, the second pixels only need to represent dark areas containing pixels where underexposure may occur in short-exposure images, and the transmittance adjustment unit 12 is controlled so that the transmittance of the pixels corresponding to the second pixels is higher than the transmittance of the pixels that do not correspond to the second pixels.

[0051] The control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 based on the control pixels and first transmittance determined by the analysis unit 15. Here, the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 such that the transmittance of the pixel corresponding to the first pixel in the transmittance adjustment unit 12 becomes the first transmittance, and the transmittance of the pixels in the transmittance adjustment unit 12 that do not correspond to the first pixel becomes a preset transmittance value.

[0052] Since consecutive short-exposure images are captured in short intervals, the brightness and darkness in consecutively captured short-exposure images generally coincide. Here, brightness and darkness in a short-exposure image refers to the brightness distribution of each pixel in the short-exposure image. Therefore, by controlling the transmittance of the transmittance adjustment unit 12 in accordance with the brightness and darkness of the generated short-exposure image before the start of the next short-exposure interval, the amount of light received by the imaging unit 13 in the next short-exposure interval can be appropriately adjusted, preventing overexposure and other issues from occurring in the short-exposure image and expanding the dynamic range.

[0053] Specifically, the brightness and darkness in the short-exposure image generated in y2, as shown in Figure 3, and the brightness and darkness in the short-exposure image generated in y3 are roughly the same. Therefore, by controlling the transmittance of the transmittance adjustment unit 12 in accordance with the brightness and darkness of the short-exposure image generated in y2 before the start of the next short-exposure section Y3, the amount of light received by the imaging unit 13 in the next short-exposure section Y3 can be appropriately adjusted, preventing overexposure and other issues from occurring in the short-exposure image generated in y3, and expanding the dynamic range.

[0054] Furthermore, since long-exposure and short-exposure images are captured consecutively at short intervals, and the ranges captured by the imaging unit 13 for both long-exposure and short-exposure images generally coincide, bright areas in short-exposure images remain bright in long-exposure images as well. Additionally, because long-exposure images have a longer exposure time than short-exposure images, long-exposure images are generally brighter than short-exposure images.

[0055] Therefore, even if, while capturing a long-exposure image, the transmittance of the first pixel in the transmittance adjustment unit 12 corresponding to a bright area containing pixels that may be overexposed in a short-exposure image is lower than the specified transmittance value, and the amount of light received by the imaging unit 13 at the first pixel is small, there is no risk of underexposure or other issues occurring at the first pixel in the captured long-exposure image. In other words, even if the transmittance of the first pixel in the transmittance adjustment unit 12 is controlled while capturing a long-exposure image, sufficient exposure is ensured at the first pixel in the long-exposure image.

[0056] The sequence in which the imaging unit 13 captures long-exposure and short-exposure images and the control unit 16 controls the transmittance of the transmittance adjustment unit 12 will be described. The imaging unit 13 alternately captures and generates long-exposure and short-exposure images. Each time a short-exposure image is generated, the analysis unit 15 determines the control pixel and the first transmittance based on the generated short-exposure image and the long-exposure image generated immediately before it. The control unit 16 then controls the transmittance of the transmittance adjustment unit 12 based on the determined control pixel and the first transmittance.

[0057] This will be explained using Figure 3. When a short-exposure image is generated at y2, the analysis unit 15 determines the control pixel and the first transmittance based on the long-exposure image generated at x2 and the short-exposure image generated at y2. Then, the control unit 16 controls the transmittance of the transmittance adjustment unit 12 based on the control pixel and the first transmittance determined by the analysis unit 15. Here, the determination of the control pixel and the first transmittance by the analysis unit 15 and the control of the transmittance of the control pixel in the transmittance adjustment unit 12 by the control unit 16 are performed between the generation of the short-exposure image at y2 and the start of the next short-exposure interval Y3.

[0058] Thus, between the end of capturing a short-exposure image in the imaging unit 13 and the start of capturing the next short-exposure image, the analysis unit 15 determines the control pixel and the first transmittance, and the control unit 16 controls the transmittance of the control pixel in the transmittance adjustment unit 12. Alternatively, the imaging unit 13 may output the start and end times of the short-exposure sections Y1, Y2, and Y3 to the control unit 16, and the control unit 16 may perform the determination of the control pixel and the first transmittance by the analysis unit 15 and the control of the transmittance of the transmittance adjustment unit 12 based on the start and end times of the short-exposure sections Y1, Y2, and Y3.

[0059] As a specific example, let's consider the case where the imaging device 10 captures video at a frame rate of 30fps. In this case, 30 composite images are continuously generated per second. Since the composite images are generated by combining long-exposure images and short-exposure images captured alternately, long-exposure images and short-exposure images are generated alternately approximately every 16.7 milliseconds. Therefore, in Figure 3, the interval between adjacent points in time among x1, y1, x2, y2, x3, and y3 is approximately 16.7 milliseconds.

[0060] Furthermore, in order to suppress the occurrence of flicker in short-exposure images, the second exposure time for capturing the short-exposure image is set based on the flicker period, which is the period of blinking of the light source that causes the flicker. One method of setting the second exposure time based on the flicker period is to set the second exposure time to an integer multiple of half the flicker period. Specifically, if the power supply frequency of the light source that causes the flicker is 50Hz and the flicker period is 1 / 50th of a second, the second exposure time will be set to a minimum of 1 / 100th of a second (10 milliseconds).

[0061] Therefore, when shooting video at a frame rate of 30fps while suppressing flicker, for example, the first exposure time in the long exposure sections X1, X2, X3 should be approximately 16.7 milliseconds, and the second exposure time in the short exposure sections Y1, Y2, Y3 should be 10 milliseconds. In other words, the first exposure time in the long exposure sections X1, X2, X3 is set based on the period in which long exposure images and short exposure images are alternately captured, and the second exposure time in the short exposure sections Y1, Y2, Y3 is set based on the flicker period.

[0062] When controlling the transmittance of the transmittance adjustment unit 12 based on the long-exposure image generated in x2 and the short-exposure image generated in y2, the control unit 16 operates during the control period, which is from the time the short-exposure image is generated in y2 until the next short-exposure interval Y3 begins. The long-exposure image and the short-exposure image are generated alternately at approximately 16.7 millisecond intervals, and since the first exposure time is approximately 16.7 milliseconds and the second exposure time is 10 milliseconds, the control period is approximately 23.4 milliseconds. When a liquid crystal material is used as the transmittance adjustment unit 12, the time required for the transmittance of the transmittance adjustment unit 12 to change is approximately 10 milliseconds.

[0063] Therefore, since the control period is longer than the time required for the transmittance of the transmittance adjustment unit 12 to change, the transmittance of the transmittance adjustment unit 12 can be controlled to an appropriate transmittance before the short exposure section Y3 begins. Consequently, the short exposure image captured in the short exposure section Y3 is captured using the transmittance adjustment unit 12, whose transmittance is controlled according to the brightness of the short exposure image captured in the short exposure section Y2. As a result, the occurrence of flicker in the short exposure image is suppressed, and the occurrence of overexposure in the short exposure image is suppressed, thereby expanding the dynamic range, and the dynamic range of the composite image generated by combining the long exposure image and the short exposure image can be expanded.

[0064] In this way, by controlling the transmittance of the control pixels in the transmittance adjustment unit 12 in accordance with the brightness and darkness in the short-exposure image, and adjusting the amount of light received by the imaging unit 13, it is possible to suppress the occurrence of black crushing and other issues in the first pixel of the captured long-exposure image, even if the transmittance of the first pixel in the transmittance adjustment unit 12 is controlled while the long-exposure image is being captured. Therefore, by controlling the transmittance of the control pixels in the transmittance adjustment unit 12 even while the long-exposure image is being captured, time can be secured for the transmittance of the transmittance adjustment unit 12 to change. As a result, it is possible to capture a short-exposure image using a transmittance adjustment unit 12 whose transmittance is controlled in accordance with the brightness and darkness in the short-exposure image.

[0065] Some or all of the synthesis unit 14, analysis unit 15, and control unit 16 of the imaging device 10 described above are performed by a computer system, for example, as shown in Figure 5, through the execution of a computer program that describes the processing in each of these units. As shown in Figure 5, this computer system includes a processor 101, memory 102, display 103, and input interface (I / F) 104.

[0066] In Figure 5, the processor 101 is, for example, a CPU (Central Processing Unit) and executes a program that describes the processing in the synthesis unit 14, the analysis unit 15, and the control unit 16, respectively. The memory 102 includes various storage media such as RAM (Random Access Memory) and ROM (Read Only Memory), as well as storage devices such as a hard disk, and stores the program that the processor 101 should execute, necessary data obtained during the processing, etc. The memory 102 is also used as a temporary storage area for the program. The processor 101 and the memory 102 constitute, for example, a processing circuit. The processing circuit may be one circuit or multiple circuits. The display 103 is, for example, an LCD (Liquid Crystal Display) and displays various screens to the user of the computer system. The input interface 104 consists of, for example, a keyboard, buttons, a mouse, etc., and is used by the user of the computer system to input various information. The display 103 may be a touch panel integrated with the input interface 104. Furthermore, Figure 5 is just one example, and the configuration of the computer system that implements some or all of the synthesis unit 14, analysis unit 15, and control unit 16 of the imaging device 10 is not limited to the example in Figure 5.

[0067] Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system with the above configuration, for example, the program is installed in memory 102 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). When the program is executed, the program read from memory 102 is stored in the main memory area of ​​memory 102. In this state, the processor 101 performs processing as the synthesis unit 14, analysis unit 15, and control unit 16 of this embodiment, respectively, according to the program stored in memory 102.

[0068] In the above description, a program describing the processing in the synthesis unit 14, the analysis unit 15, and the control unit 16 is provided using a CD-ROM or DVD-ROM as the recording medium. However, the system is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, a program provided via a transmission medium such as the Internet through the communication unit may also be used.

[0069] Next, the flow of the imaging method using the imaging device 10 according to Embodiment 1 will be explained with reference to Figure 6. Figure 6 is a flowchart showing the flow of the imaging method according to Embodiment 1.

[0070] In step S01, the imaging unit 13 receives light from the subject, converts it into an electrical signal for each pixel, and takes a long-exposure image. Steps S11 to S13 are performed in parallel with step S01.

[0071] In step S11, the analysis unit 15 calculates a measured brightness value indicating the brightness of the short-exposure image based on the short-exposure image, and determines the first transmittance based on the measured brightness value and a preset brightness target value. The analysis unit 15 may also determine a preset value as the first transmittance.

[0072] In step S12, the analysis unit 15 calculates image information indicating the first pixel, which is the pixel used in generating the composite image, based on the long-exposure image and short-exposure image captured by the imaging unit 13 immediately before, and determines the control pixel based on the calculated image information. Note that the method for determining the control pixel is not limited to the method described above; the control pixel can be determined based on the brightness of each pixel in the short-exposure image captured by the imaging unit 13 immediately before. Note that the order of steps S11 and S12 is not limited to the order shown in Figure 6, and can be performed in any order. Also, steps S11 and S12 may be performed in parallel.

[0073] In step S13, the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 based on the first transmittance determined by the analysis unit 15 in step S11 and the first pixel determined by the analysis unit 15 in step S12. For example, the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 such that the transmittance of the pixel corresponding to the first pixel in the transmittance adjustment unit 12 becomes the first transmittance, and the transmittance of the pixels in the transmittance adjustment unit 12 that do not correspond to the first pixel becomes a preset transmittance value.

[0074] In step S02, the imaging unit 13 receives light from the subject, converts it into an electrical signal for each pixel, and captures a short-exposure image. In step S03, the synthesis unit 14 combines the long-exposure image captured in step S01 and the short-exposure image captured in step S02 to generate a composite image.

[0075] As described above, one embodiment of Embodiment 1 is to control the transmittance of the control pixels in the transmittance adjustment unit 12 to match the brightness of the short-exposure image while a short-exposure image is not being captured. Another embodiment is to control the transmittance of the control pixels in the transmittance adjustment unit 12 to match the brightness of the long-exposure image while a long-exposure image is not being captured.

[0076] Another embodiment relating to Embodiment 1 will now be described. The analysis unit 15 calculates a value indicating the brightness of the long-exposure image based on the long-exposure image, and determines the second transmittance based on the calculated value indicating the brightness of the long-exposure image. The analysis unit 15 may also determine a preset value as the second transmittance.

[0077] Furthermore, the analysis unit 15 calculates image information indicating a second pixel, which is a pixel used in the long-exposure image generation, based on the long-exposure image and short-exposure image captured by the imaging unit 13. Based on the calculated image information indicating the second pixel, the transmittance adjustment unit 12 determines a control pixel, which is a pixel whose transmittance is controlled. Here, the second pixel refers to a dark area in the long-exposure image that may contain pixels where black clipping may occur.

[0078] The control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 between the end of capturing a long-exposure image in the imaging unit 13 and the start of capturing the next long-exposure image. The control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 such that the transmittance of the pixel corresponding to the second pixel in the transmittance adjustment unit 12 becomes the second transmittance, and the transmittance of the pixels in the transmittance adjustment unit 12 that do not correspond to the second pixel becomes a preset transmittance value. Here, the transmittance value is defined as the minimum transmittance value that can be set in the transmittance adjustment unit 12. Since the second pixel indicates a dark area that may result in black crushing in a long-exposure image, the analysis unit 15 determines that the second transmittance is higher than the transmittance value. Note that the transmittance value is not limited to the value described above, and can be any value smaller than the maximum transmittance value that can be set in the transmittance adjustment unit 12.

[0079] Since long-exposure and short-exposure images are captured consecutively, dark areas in the long-exposure image remain dark in the short-exposure image as well. Furthermore, because the exposure time for short-exposure images is shorter than that of long-exposure images, short-exposure images are generally darker. Therefore, even if the transmittance of the second pixel corresponding to a dark area containing pixels that may be underexposed in the long-exposure image is higher than the specified transmittance value while capturing a short-exposure image, there is no risk of overexposure or other issues occurring in the second pixel of the captured short-exposure image. In other words, even if the transmittance of the second pixel is controlled in the transmittance adjustment unit 12 while capturing a short-exposure image, the exposure amount for the second pixel of the short-exposure image is appropriately ensured.

[0080] Therefore, by controlling the transmittance of the control pixels in the transmittance adjustment unit 12 even while capturing short-exposure images, time can be secured for the transmittance of the transmittance adjustment unit 12 to change. Furthermore, since the transmittance of pixels in the transmittance adjustment unit 12 that do not correspond to the second pixel is controlled to a specified transmittance value lower than the second transmittance, the occurrence of flicker in short-exposure images can be suppressed, and the occurrence of overexposure in short-exposure images can also be suppressed. As a result, long-exposure images can be captured using the transmittance adjustment unit 12 whose transmittance is controlled according to the brightness of the short-exposure image, and a composite image with a wider dynamic range can be obtained while suppressing the occurrence of flicker.

[0081] Thus, the transmittance of the transmittance adjustment unit 12 may be controlled so that each pixel has an appropriate transmittance when capturing a short-exposure image, or it may be controlled so that each pixel has an appropriate transmittance when capturing a long-exposure image.

[0082] Therefore, the imaging unit 13 determines the control pixel based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image it captures, and controls the transmittance of the control pixel in the transmittance adjustment unit 12. This allows the amount of light received by the imaging unit 13 to be adjusted by controlling the transmittance of the transmittance adjustment unit 12 in accordance with the brightness of the long-exposure image or the short-exposure image. In this case, by controlling the transmittance of the transmittance adjustment unit 12 even while capturing the short-exposure image or the long-exposure image, time can be secured for the transmittance of the transmittance adjustment unit 12 to change. As a result, it is possible to capture images using the transmittance adjustment unit 12 whose transmittance is controlled in accordance with the brightness of the long-exposure image or the short-exposure image, thereby acquiring images with a wider dynamic range.

[0083] The analysis unit 15 may calculate image information indicating at least one of a first pixel whose brightness is equal to or greater than a preset first threshold, and a second pixel whose brightness is equal to or less than a preset second threshold, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image, and determine the control pixel based on the calculated image information.

[0084] Furthermore, the analysis unit 15 may calculate image information indicating at least one of the pixels that use the short-exposure image in generating the composite image, and the pixels that use the long-exposure image in generating the composite image, based on the long-exposure image and the short-exposure image, and determine the control pixels based on the calculated image information.

[0085] Next, the flow of the imaging method using the imaging device 10 according to another embodiment of Embodiment 1 will be explained with reference to Figure 7. Figure 7 is a flowchart showing the flow of the imaging method according to another embodiment of Embodiment 1.

[0086] In step S21, the imaging unit 13 receives light from the subject, converts it into an electrical signal for each pixel, and takes a short-exposure image. Steps S31 to S33 are performed in parallel with step S21.

[0087] In step S31, the analysis unit 15 calculates a value indicating the brightness of the long-exposure image based on the long-exposure image, and determines the second transmittance based on the calculated value indicating the brightness of the long-exposure image. The analysis unit 15 may also determine the second transmittance as a preset value.

[0088] In step S32, based on the long-exposure and short-exposure images captured by the imaging unit 13, image information indicating a second pixel, which is the pixel used in the long-exposure image for generating the composite image, is calculated. Based on the calculated image information indicating the second pixel, the transmittance adjustment unit 12 determines a control pixel, which is the pixel whose transmittance is controlled. Note that the method for determining the control pixel is not limited to the method described above; the control pixel can be determined based on the brightness of each pixel in the long-exposure image captured by the imaging unit 13 immediately before. Note that the order of steps S31 and S32 is not limited to the order shown in Figure 7, and can be performed in any order. Also, steps S31 and S32 may be performed in parallel.

[0089] In step S33, the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 based on the second transmittance determined in step S31 and the second pixel determined in step S32. For example, the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 such that the transmittance of the pixel corresponding to the second pixel in the transmittance adjustment unit 12 becomes the second transmittance, and the transmittance of the pixels that do not correspond to the second pixel in the transmittance adjustment unit 12 becomes a preset transmittance value.

[0090] In step S22, the imaging unit 13 receives light from the subject, converts it into an electrical signal for each pixel, and captures a long-exposure image. In step S23, the synthesis unit 14 combines the long-exposure image captured in step S21 and the short-exposure image captured in step S22 to generate a composite image.

[0091] As described above, the imaging device 10 according to Embodiment 1 is an imaging device 10 that generates a composite image by combining a long-exposure image taken with a first exposure time and a short-exposure image taken with a second exposure time shorter than the first exposure time, and comprises: an imaging unit 13 that receives light from a subject and converts it into an electrical signal for each pixel, and alternately takes long-exposure images and short-exposure images; a transmittance adjustment unit 12 provided between the subject and the imaging unit 13, which controls the transmittance of light for each pixel of the imaging unit 13; an analysis unit 15 that determines a control pixel, which is a pixel whose transmittance is controlled in the transmittance adjustment unit 12, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image taken by the imaging unit 13; a control unit 16 that controls the transmittance of the control pixel in the transmittance adjustment unit 12 based on the determination of the control pixel by the analysis unit 15; and a synthesis unit 14 that generates a composite image by combining the long-exposure image and the short-exposure image taken by the imaging unit 13.

[0092] With this configuration, the transmittance of the transmittance adjustment unit 12 can be controlled in accordance with the brightness and darkness in short-exposure or long-exposure images to adjust the amount of light received by the imaging unit 13. Therefore, by controlling the transmittance of the transmittance adjustment unit 12 even while capturing short-exposure or long-exposure images, time can be secured for the transmittance of the transmittance adjustment unit 12 to change. As a result, it is possible to capture images using the transmittance adjustment unit 12 whose transmittance is controlled in accordance with the brightness and darkness in short-exposure or long-exposure images, thereby acquiring images with a wider dynamic range.

[0093] Furthermore, in Embodiment 1, the analysis unit 15 calculates a measured brightness value indicating the brightness of a short-exposure image based on the short-exposure image, determines a first transmittance based on the measured brightness value and a preset brightness target value, and the control unit 16 controls the transmittance of the control pixels in the transmittance adjustment unit 12 such that the transmittance of the pixel corresponding to the first pixel in the transmittance adjustment unit 12 becomes the first transmittance, and the transmittance of the pixels in the transmittance adjustment unit 12 that do not correspond to the first pixel becomes a preset transmittance specified value. With this configuration, the transmittance of the control pixels in the transmittance adjustment unit 12 is controlled using the first transmittance determined based on the brightness of each pixel in the short-exposure image. Therefore, since the transmittance of the transmittance adjustment unit 12 is controlled to be suitable for the brightness in the short-exposure image, an image with a wider dynamic range can be obtained.

[0094] Furthermore, in Embodiment 1, the analysis unit 15 calculates a measured brightness value based on the brightness of the first pixel in the short-exposure image, and determines a first transmittance based on the measured brightness value and the brightness target value. With this configuration, the transmittance of the control pixel in the transmittance adjustment unit 12 is controlled using the first transmittance determined based on the brightness of the first pixel, which corresponds to a bright area in the short-exposure image that includes pixels where overexposure may occur, and whose transmittance in the transmittance adjustment unit 12 is set to be lower than the specified transmittance value. Therefore, since the transmittance of the transmittance adjustment unit 12 is controlled to be suitable for the brightness of the first pixel in the short-exposure image, an image with a wider dynamic range can be obtained.

[0095] Embodiment 2. The imaging device 10a according to Embodiment 2 will be described with reference to Figures 8 to 11. In Embodiment 2, the same reference numerals are used for the same components as in Embodiment 1, and the differences from Embodiment 1 will be mainly described.

[0096] Figure 8 shows an example configuration of the imaging device 10a according to Embodiment 2. In Embodiment 2, the imaging device 10a includes a rotating unit 17, a rotation angle detection unit 18, and a rotational movement amount calculation unit 19. The rotating unit 17 rotates integrally with the imaging unit 13 and the transmittance adjustment unit 12. An example of the rotating unit 17 is a turntable that rotates automatically based on input instructions.

[0097] The rotation angle detection unit 18 detects the rotation angle θ, which is the angle at which the rotating unit 17 rotates. Here, the rotation angle θ represents, for example, the angle at which the rotating unit 17 rotates between the time a short-exposure image used by the analysis unit 15a to determine the control pixel is captured and the time the next short-exposure image is captured.

[0098] The rotation angle detection unit 18 may detect the rotation angle θ by measuring the angle at which the rotating unit 17 rotates, or it may calculate the rotation angle θ based on the rotation speed of the rotating unit 17 and the period during which short-exposure images are captured. Furthermore, if the rotating unit 17 rotates based on an input instruction, the rotation angle detection unit 18 may detect the rotation angle θ based on the instruction input to the rotating unit 17.

[0099] The rotational displacement calculation unit 19 calculates a rotational displacement amount based on the rotation angle θ, which indicates the direction and distance the subject moves due to the rotation of the rotation unit 17 in the short-exposure image. The method for calculating the rotational displacement amount will be described later. The analysis unit 15a calculates image information indicating the first pixel, which is a bright area containing pixels where overexposure may occur in the short-exposure image, based on the brightness of each pixel in the short-exposure image, and determines the control pixel based on the image information and the rotational displacement amount.

[0100] In Embodiment 1, based on the brightness of each pixel in the short-exposure image captured in the short-exposure section Y2 shown in Figure 3, the control pixel was determined so that the transmittance of the first pixel of the transmittance adjustment unit 12 in the short-exposure section Y3 becomes the first transmittance. However, if the rotating unit 17 moves quickly in conjunction with the imaging unit 13 between the end of the short-exposure section Y2 and the start of the short-exposure section Y3, the range captured by the imaging unit 13 will shift. As a result, a discrepancy may occur between the bright areas in the short-exposure image captured in the short-exposure section Y2 and the bright areas in the short-exposure image captured in the short-exposure section Y3, potentially preventing the transmittance of each pixel in the transmittance adjustment unit 12 from being properly controlled.

[0101] Therefore, in Embodiment 2, the rotation angle detection unit 18 detects the rotation angle θ of the rotating unit 17, the rotational movement amount calculation unit 19 calculates the rotational movement amount using this rotation angle θ, and the analysis unit 15a determines the control pixel using the rotational movement amount. This ensures that even if the range captured by the imaging unit 13 moves between the end of capturing one short-exposure image and the start of capturing the next short-exposure image, the transmittance of each pixel in the transmittance adjustment unit 12 can be appropriately controlled.

[0102] The method for calculating the amount of rotational movement and the method for determining the control pixel using the amount of rotational movement will be explained. Figure 9(a) is a view from above of the imaging device 10a showing how the rotating part 17 rotates horizontally integrally with the lens 11 and the imaging part 13. Figure 9(b) schematically shows the amount of rotational movement in a short-exposure image.

[0103] In Figure 9(a), the rotating part 17 rotates clockwise by an angle θ from the direction indicated by the dashed line F to the direction indicated by the dashed line G. Here, the state indicated by the dashed line F is the state at the start of the short exposure section Y2 shown in Figure 3, and the state indicated by the dashed line G is the state at the start of the short exposure section Y3 shown in Figure 3. Also, the rectangles f and g shown in Figure 9(b) represent pixels f and g, respectively, in the short exposure image.

[0104] As shown in Figure 9(a), when the rotating part 17 rotates clockwise in the horizontal direction, the subject appears to move to the left in the short-exposure image shown in Figure 9(b). That is, the subject displayed in pixel f when the rotating part 17 is facing the direction indicated by the dashed line F will be displayed in pixel g when the rotating part 17 is facing the direction indicated by the dashed line G. In Figure 9(b), the distance the subject has moved from pixel f to pixel g is indicated by the distance l.

[0105] The rotating part 17 rotates clockwise by an angle θ from the state where it is facing the direction indicated by the dashed line F to the state where it is facing the direction indicated by the dashed line G. Therefore, the distance traveled l is calculated from the rotation angle θ, the focal length of the lens 11, and the pixel size. Here, the focal length of the lens 11 is l f , the pixel size is l p Therefore, the displacement l, expressed in pixels, is given by the following equation (1). l=l f ×tanθ / l p (1)

[0106] Therefore, regarding the direction and distance of the rotational movement, in a short-exposure image, the direction in which the subject moves due to the rotation of the rotating part 17 is to the left, and the movement distance l is calculated according to equation (1). Although the case in which the rotating part 17 rotates in the horizontal direction has been described, the direction in which the rotating part 17 rotates is not limited to this. For example, even when the rotating part 17 rotates in a direction oblique to the horizontal direction, the amount of rotational movement can be calculated in the same way.

[0107] Furthermore, the method for calculating the amount of rotational movement is not limited to the method described above; it is sufficient to calculate the amount of rotational movement, which indicates the direction and distance of movement of the subject due to the rotation of the rotating unit 17, in the image captured by the imaging unit 13.

[0108] Next, the control pixel is determined based on the calculated rotational displacement. The analysis unit 15a calculates image information indicating the first pixel, which is a bright area containing pixels where overexposure may occur in the short-exposure image, based on the brightness of each pixel in the short-exposure image. The pixel located a distance of the rotational displacement from the first pixel is designated as the third pixel, and the control pixel is determined based on the third pixel. Figure 10 is a schematic diagram showing the first and third pixels in the short-exposure image. In Figure 10, the area H shown by the solid line indicates the first pixel, and the area I shown by the dashed line indicates the third pixel.

[0109] The first pixel indicated by range H in Figure 10 is the first pixel calculated based on the short-exposure image taken in the short-exposure section Y2 shown in Figure 3, with the rotating unit 17 facing the direction indicated by the dashed line F shown in Figure 9(a). When the rotating unit 17 rotates clockwise in the horizontal direction as shown in Figure 9(a), the subject moves to the left by a distance l in the short-exposure image, as shown in Figure 9(b). Therefore, in the short-exposure section Y3 shown in Figure 3, where the transmittance of the transmittance adjustment unit 12 is controlled before taking the short-exposure image, the bright areas in the short-exposure image are pixels that are moved to the left by a distance l from the first pixel.

[0110] Therefore, by designating a pixel located a distance l to the left of the first pixel as the third pixel, the bright area containing pixels where overexposure may occur in the short-exposure image captured in the short-exposure section Y3 coincides with the third pixel. After determining the third pixel, the control pixel is determined based on the third pixel, similar to the method used in Embodiment 1 to determine the control pixel based on the first pixel. This allows the transmittance of the transmittance adjustment unit 12 to be controlled to match the bright area containing pixels where overexposure may occur in the short-exposure image captured in the short-exposure section Y3 shown in Figure 3. Thus, even if the range captured by the imaging unit 13 changes due to the rotation of the rotating unit 17, and the position of the bright area in the short-exposure image changes, the transmittance of each pixel in the transmittance adjustment unit 12 can be appropriately controlled, and an image with a wider dynamic range can be obtained.

[0111] Furthermore, the method for determining the control pixel based on the calculated rotational displacement is not limited to the method described above; any method that corrects the control pixel calculated by the analysis unit 15a based on the rotational displacement is acceptable.

[0112] Although the case in which the transmittance of the control pixels in the transmittance adjustment unit 12 is controlled according to the brightness and darkness in a short-exposure image has been described, similarly, in the case in which the transmittance of the control pixels in the transmittance adjustment unit 12 is controlled according to the brightness and darkness in a long-exposure image, as in other embodiments according to Embodiment 1, the control pixels may also be determined based on the amount of rotational movement.

[0113] Next, the flow of the imaging method using the imaging device 10a according to Embodiment 2 will be explained with reference to Figure 11. Figure 11 is a flowchart showing the flow of the imaging method according to Embodiment 2. The flowchart shown in Figure 11 is the same as the flowchart shown in Figure 6, but with step S41 added between steps S11 and S12, and step S12 changed to step S42. Therefore, steps S41 and S42 will be explained below.

[0114] In step S41, the rotational movement amount calculation unit 19 calculates the rotational movement amount, which indicates the direction and distance the subject moves due to the rotation of the rotating unit 17 in the short-exposure image, based on the rotation angle θ detected by the rotation angle detection unit 18.

[0115] In step S42, the analysis unit 15a calculates image information indicating the first pixel based on the brightness of each pixel in the short-exposure image, and determines the control pixel based on the image information and the amount of rotational movement.

[0116] Note that the order of steps S11, S41, and S42 is not limited to the order shown in Figure 11, and can be performed in any order. Also, steps S11, S41, and S42 may be performed in parallel.

[0117] As described above, the imaging device 10a according to Embodiment 2 includes a rotating unit 17 that rotates integrally with the imaging unit 13 and the transmittance adjustment unit 12, a rotating angle detection unit 18 that detects the rotation angle θ of the rotating unit 17, and a rotational movement amount calculation unit 19 that calculates a rotational movement amount indicating the direction and distance of movement of the subject due to the rotation of the rotating unit 17 in at least one of the short-exposure image and the long-exposure image based on the rotation angle θ, and the analysis unit 15a determines the control pixel based on the image information and the rotational movement amount. With this configuration, the analysis unit 15a determines the control pixel based on the image information and the rotational movement amount due to the rotation of the rotating unit 17, and controls the transmittance of each pixel of the transmittance adjustment unit 12 in accordance with the range captured by the imaging unit 13 that moves due to the rotation of the rotating unit 17. As a result, even if the brightness and darkness in at least one of the short-exposure image and the long-exposure image change due to the rotation of the rotating unit 17, the transmittance of each pixel of the transmittance adjustment unit 12 can be appropriately controlled, and an image with a wider dynamic range can be obtained.

[0118] Embodiment 3. The imaging device 10b according to Embodiment 3 will be described with reference to Figures 12 to 15. In Embodiment 3, the same reference numerals are used for components that are the same as those in Embodiments 1 and 2, and the differences from Embodiments 1 and 2 will be mainly described.

[0119] Figure 12 shows an example of the configuration of the imaging device 10b according to Embodiment 3. In Embodiment 3, the imaging device 10b includes a subject movement amount calculation unit 20. The subject movement amount calculation unit 20 calculates a subject movement amount, which indicates the direction and distance of movement of the subject in the composite image, based on the continuously generated composite image. The method for calculating the subject movement amount will be described later. The analysis unit 15b calculates image information indicating a first pixel, which is a bright area containing pixels where overexposure may occur in the short-exposure image, based on the brightness of each pixel in the short-exposure image. The pixel located a distance from the first pixel by the subject movement amount is designated as the fourth pixel, and a control pixel is determined based on the fourth pixel.

[0120] In Embodiment 1, the control pixel was determined so that the transmittance of the first pixel of the transmittance adjustment unit 12 in the short exposure section Y3 becomes the first transmittance, in accordance with the brightness of the short exposure image captured in the short exposure section Y2 shown in Figure 3. However, the bright area within the range captured by the imaging unit 13 may shift due to the movement of the subject between the end of the short exposure section Y2 and the start of the short exposure section Y3. Therefore, a discrepancy may occur between the bright area in the short exposure image captured in the short exposure section Y2 and the bright area in the short exposure image captured in the short exposure section Y3, potentially preventing the transmittance of each pixel in the transmittance adjustment unit 12 from being properly controlled.

[0121] Therefore, in Embodiment 3, the subject movement amount calculation unit 20 calculates the subject movement amount, which indicates the direction and distance of movement of the subject in the composite image, and the analysis unit 15a determines the control pixels using the subject movement amount. This ensures that even if a bright area containing pixels where overexposure may occur moves within the range captured by the imaging unit 13 between the end of capturing a short-exposure image and the start of capturing the next short-exposure image, the transmittance of each pixel in the transmittance adjustment unit 12 can be appropriately controlled.

[0122] This section explains how to calculate the amount of subject movement and how to determine the control pixel using the amount of subject movement. Here, a subject that moves within the range captured by the imaging unit 13 is called a moving subject, and we will explain the case where the moving subject moves to the left in the composite image captured between the start of the long exposure section X1 shown in Figure 3 and the end of the short exposure section Y3 y3.

[0123] As shown in Figure 3, composite image z1 is the composite image generated from the long-exposure image generated at x1 and the short-exposure image generated at y1, and composite image z2 is the composite image generated from the long-exposure image generated at x2 and the short-exposure image generated at y2.

[0124] Figure 13 schematically shows the amount of subject movement in the composite image. In Figure 13, point j represents a portion of the moving subject in composite image z1, and point k represents a portion of the moving subject in composite image z2. Here, points j and k represent the same portion of the moving subject. Furthermore, composite image z1 is obtained at time y1 shown in Figure 3, and composite image z2 is obtained at time y2 shown in Figure 3.

[0125] Thus, between time point y1 and time point y2 as shown in Figure 3, the moving subject moves leftward by a distance l' in pixels, as shown in Figure 13. In this case, regarding the direction and distance of the subject's movement, the direction of movement of the moving subject in the composite image is leftward, and the distance the subject has moved in the composite image is l'. Although we have explained the case where the moving subject moves leftward, the direction of movement of the moving subject is not limited to this. For example, even when the moving subject moves diagonally, the amount of subject movement can be calculated using the same method.

[0126] The method for calculating the amount of subject movement is not limited to the method described above; it is sufficient to calculate the amount of subject movement, which indicates the direction and distance of movement of a moving subject in the image captured by the imaging unit 13. For example, the direction and distance of movement of the subject in a short-exposure image may be calculated based on a series of short-exposure images, or the direction and distance of movement of the subject in a long-exposure image may be calculated based on a series of long-exposure images. Alternatively, for example, the direction and distance of movement of the subject in an image captured by the imaging unit 13 may be calculated based on a series of long-exposure and short-exposure images.

[0127] Furthermore, when the imaging device 10b outputs the composite images it continuously generates as a video, they are compressed. The motion vector calculated during this compression may be used as the subject movement amount. Here, the motion vector, like the subject movement amount, represents the change in the continuous composite images.

[0128] Next, the control pixel is determined based on the calculated subject movement amount. The analysis unit 15b calculates image information indicating the first pixel, which is a bright area containing pixels where overexposure may occur in the short-exposure image, based on the brightness of each pixel in the short-exposure image. The pixel located a distance from the first pixel by the subject movement amount is designated as the fourth pixel, and the control pixel is determined based on the fourth pixel. Figure 14 is a schematic diagram showing the first and fourth pixels in the short-exposure image. In Figure 14, the area J shown by the solid line indicates the first pixel, and the area K shown by the dashed line indicates the fourth pixel.

[0129] The first pixel shown in range J in Figure 14 is the first pixel calculated based on the short-exposure image used to generate the composite image z2, where a part of the moving subject is at point j in Figure 13. Between time points y1 and y2 in Figure 3, the moving subject moves leftward by a distance l' as shown in Figure 13. Therefore, it is assumed that the moving subject also moves leftward by a distance l' between time points y2 and y3 in Figure 3. Consequently, the position of the moving subject in the short-exposure image taken in short-exposure section Y3 is a distance l' to the left of the position of the moving subject in the short-exposure image taken in short-exposure section Y2. If the moving subject is bright, the first pixel, which is the bright part in the short-exposure image, will similarly be at the position shown in range K, which is a distance l' to the left.

[0130] Therefore, by designating the fourth pixel as the pixel located a distance l' to the left of the first pixel, the third pixel coincides with the bright area containing pixels where overexposure may occur in the short-exposure image captured in the short-exposure section Y3. Similar to the method of determining the control pixel based on the first pixel in Embodiment 1, by determining the control pixel based on the fourth pixel, the transmittance of the transmittance adjustment unit 12 can be controlled to match the bright area containing pixels where overexposure may occur in the short-exposure image captured in the short-exposure section Y3 shown in Figure 3. Thus, even if the subject moves within the range captured by the imaging unit 13 and the position of the bright area in the short-exposure image changes, the transmittance of each pixel in the transmittance adjustment unit 12 can be appropriately controlled, and an image with a wider dynamic range can be obtained.

[0131] Furthermore, the method for determining the control pixels based on the calculated subject movement amount is not limited to the method described above; any method that corrects the control pixels calculated by the analysis unit 15b based on the subject movement amount is acceptable.

[0132] Although the case in which the transmittance of the control pixels in the transmittance adjustment unit 12 is controlled according to the brightness and darkness in a short-exposure image has been described, similarly, in the case in which the transmittance of the control pixels in the transmittance adjustment unit 12 is controlled according to the brightness and darkness in a long-exposure image, as in other embodiments according to Embodiment 1, the control pixels may also be determined based on the amount of subject movement.

[0133] Next, the flow of the imaging method using the imaging device 10b according to Embodiment 3 will be explained with reference to Figure 15. Figure 15 is a flowchart showing the flow of the imaging method according to Embodiment 3. The flowchart shown in Figure 15 is the same as the flowchart shown in Figure 6, but with step S51 added between steps S11 and S12, and step S12 changed to step S52. Therefore, steps S51 and S52 will be mainly explained below.

[0134] In step S51, the rotational movement amount calculation unit 19 calculates the subject movement amount, which indicates the direction and distance of movement of the subject in the composite image, as calculated by the subject movement amount calculation unit 20.

[0135] In step S52, the analysis unit 15a calculates image information indicating the first pixel based on the brightness of each pixel in the short-exposure image, and determines the control pixel based on the image information and the amount of subject movement.

[0136] Note that the order of steps S11, S51, and S52 is not limited to the order shown in Figure 15, and can be performed in any order. Also, steps S11, S51, and S52 may be performed in parallel.

[0137] As described above, the imaging device 10b according to Embodiment 3 includes a subject movement amount calculation unit 20 that calculates a subject movement amount indicating the direction and distance of movement of the subject in the composite image based on a continuously generated composite image, and an analysis unit 15b that determines a control pixel based on the image information and the subject movement amount. With this configuration, the control pixel is determined based on the image information calculated by the analysis unit 15b and the subject movement amount due to the movement of the subject, and the transmittance of each pixel of the transmittance adjustment unit 12 is controlled in accordance with the change in brightness due to the movement of the subject in the range captured by the imaging unit 13. As a result, even when the brightness changes in at least one of the short-exposure image and the long-exposure image due to the movement of the subject, the transmittance of each pixel of the transmittance adjustment unit 12 can be appropriately controlled, and an image with a wider dynamic range can be obtained.

[0138] While this disclosure describes various exemplary embodiments and examples, the various features, aspects, and functions described in one or more embodiments are not limited to the application of a particular embodiment, but are applicable individually or in various combinations to the embodiments. Accordingly, countless variations not illustrated are conceivable within the scope of the art disclosed in this specification. These include, for example, modifying, adding or omitting at least one component, or even extracting at least one component and combining it with components of other embodiments. [Explanation of Symbols]

[0139] 1 Imaging system, 2 HUB, 3 Decoder, 4 Display device, 5 Operation terminal, 6 Recording device, 10, 10a, 10b Imaging device, 11 Lens, 12 Transmittance adjustment unit, 13 Imaging unit, 14 Compositing unit, 15, 15a, 15b Analysis unit, 16 Control unit, 17 Rotation unit, 18 Rotation angle detection unit, 19 Rotation movement amount calculation unit, 20 Subject movement amount calculation unit, 101 Processor, 102 Memory, 103 Display, 104 Input interface

Claims

1. An imaging device that generates a composite image by combining a long-exposure image taken with a first exposure time and a short-exposure image taken with a second exposure time shorter than the first exposure time, An imaging unit that receives light from a subject, converts it into an electrical signal for each pixel, and alternately takes long-exposure images and short-exposure images, A transmittance adjustment unit is provided between the subject and the imaging unit, and controls the transmittance of light for each pixel of the imaging unit. An analysis unit determines a control pixel, which is the pixel whose transmittance is controlled in the transmittance adjustment unit, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image captured by the imaging unit. A control unit controls the transmittance of the control pixel in the transmittance adjustment unit based on the control pixel determined by the analysis unit, An imaging apparatus comprising: an imaging unit; and a synthesis unit that generates a composite image by combining the long-exposure image and the short-exposure image captured by the imaging unit.

2. The imaging apparatus according to claim 1, wherein the analysis unit calculates image information indicating at least one of the first pixel, which is a pixel whose brightness is equal to or greater than a preset first threshold, and the second pixel, which is a pixel whose brightness is equal to or less than a preset second threshold, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image, and determines the control pixel based on the calculated image information.

3. The imaging apparatus according to claim 2, wherein the analysis unit calculates image information indicating at least one of the first pixel which is the pixel used in the short exposure image in generating the composite image, and the second pixel which is the pixel used in the long exposure image in generating the composite image, based on the long exposure image and the short exposure image, and determines the control pixel based on the calculated image information.

4. The analysis unit calculates the image information representing the first pixel, and determines the control pixel based on the first pixel. The imaging apparatus according to claim 2 or 3, wherein the control unit controls the transmittance of the control pixel in the transmittance adjustment unit between the time the imaging unit finishes capturing the short-exposure image and the time the next short-exposure image is captured.

5. The analysis unit calculates a measured brightness value indicating the brightness of the short-exposure image based on the short-exposure image, and determines a first transmittance based on the measured brightness value and a preset brightness target value. The imaging apparatus according to claim 4, wherein the control unit controls the transmittance of the control pixels in the transmittance adjustment unit such that the transmittance of the pixel corresponding to the first pixel in the transmittance adjustment unit becomes the first transmittance, and the transmittance of the pixels not corresponding to the first pixel in the transmittance adjustment unit becomes a preset transmittance value.

6. The imaging apparatus according to claim 5, wherein the analysis unit calculates the measured brightness value based on the brightness of the first pixel in the short-exposure image, and determines the first transmittance based on the measured brightness value and the brightness target value.

7. A rotating part that rotates integrally with the imaging unit and the transmittance adjustment unit, A rotation angle detection unit detects the rotation angle, which is the angle at which the rotating part rotates. The system includes a rotational movement amount calculation unit that calculates a rotational movement amount indicating the direction and distance the subject moves due to the rotation of the rotating unit in at least one of the short-exposure image and the long-exposure image, based on the rotation angle, The imaging apparatus according to claim 2 or 3, wherein the analysis unit determines the control pixel based on the image information and the amount of rotational movement.

8. The imaging apparatus according to claim 7, wherein the analysis unit calculates the image information indicating the first pixel, sets the pixel located a distance from the first pixel by the amount of rotational movement as the third pixel, and determines the control pixel based on the third pixel.

9. The system includes a subject movement amount calculation unit that calculates a subject movement amount indicating the direction and distance of movement of the subject in the composite image, based on the composite image generated in succession. The imaging apparatus according to claim 2 or 3, wherein the analysis unit determines the control pixel based on the image information and the amount of subject movement.

10. The imaging apparatus according to claim 9, wherein the analysis unit calculates the image information indicating the first pixel, sets the pixel located a distance from the first pixel by the amount of subject movement as the fourth pixel, and determines the control pixel based on the fourth pixel.

11. The imaging apparatus according to any one of claims 1 to 3, wherein the transmittance adjustment unit is composed of liquid crystal material arranged for each pixel.

12. An imaging method for generating a composite image by combining a long-exposure image taken with a first exposure time and a short-exposure image taken with a second exposure time shorter than the first exposure time, The imaging unit receives light from the subject and converts it into an electrical signal for each pixel, and alternately performs the steps of capturing the long-exposure image and the short-exposure image, A step of determining, in a transmittance adjustment unit provided between the subject and the imaging unit, in which the transmittance of light is controlled for each pixel of the imaging unit, a control pixel which is the pixel whose transmittance is controlled, based on the brightness of each pixel in at least one of the long-exposure image and the short-exposure image captured by the imaging unit; The steps include controlling the transmittance of the control pixel in the transmittance adjustment unit, An imaging method comprising the step of generating a composite image by combining the long-exposure image and the short-exposure image.

Images