Imaging device

Abstract

To generate a high-quality HDR composite image adapted to the imaging environment. [Solution] The system includes a video input unit that captures an image of a subject and generates an image signal of the subject, a video signal processing unit that generates a captured image of the subject based on the image signal, and a control unit that detects motion information of the subject based on the image signal, and based on the motion information, causes the video input unit to capture images of the subject multiple times with different exposure amounts, and causes the video signal processing unit to generate an HDR composite image of the subject based on the multiple image signals with different exposure amounts.

Description

[Technical Field]

[0001] The present invention relates to an imaging device, an imaging control method, and an imaging program, and more particularly to an imaging device, an imaging control method, and an imaging program that generate an HDR (High Dynamic Range) composite image by combining images captured by multiple cameras with different exposure amounts. [Background technology]

[0002] Digital cameras, which convert a subject image into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) and record the converted electrical signal in a memory, have become extremely popular. Recently, information terminal devices equipped with digital cameras, such as mobile phones, smartphones, and tablet terminals, have also become widely used.

[0003] Image sensors used in digital cameras and other devices have a very narrow dynamic range compared to film, which can cause so-called blocked-up shadows and blown-out highlights, resulting in a significant degradation of image quality, depending on the shooting conditions.To overcome this drawback, attention is being paid to HDR compositing, which captures multiple images with different exposures and then combines them by extracting dark areas from images with longer exposures and bright areas from images with shorter exposures, thereby achieving a wide dynamic range with reduced blocked-up highlights and blackouts.

[0004] For example, Patent Document 1 discloses an image generation device that has two image sensors, one of which captures a low-exposure image at high resolution and the other of which captures a high-exposure image at low resolution with a short exposure time, thereby creating an HDR composite image with reduced noise caused by camera shake and the like. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-336561 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 aims to reduce noise caused by camera shake and the like by shortening the exposure time when capturing an image with a high exposure amount, but does not consider an HDR compositing method for obtaining a more preferable image quality according to the imaging environment, such as the subject's movement, brightness, camera shake, etc. Furthermore, it does not mention how to effectively utilize two imaging elements.

[0007] Therefore, an object of the present invention is to provide an imaging device, an imaging control method, and a program that generate a high-quality HDR composite image that is adapted to the imaging environment. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] An imaging device according to a representative embodiment of the present invention includes a video input unit that images a subject and generates an image signal of the subject; a video signal processing unit that generates an image of the subject based on the image signal; and a control unit that detects motion information of the subject based on the image signal, and based on the motion information, causes the video input unit to image the subject multiple times with different exposure amounts, and causes the video signal processing unit to generate an HDR composite image of the subject based on the multiple image signals with different exposure amounts. [Effects of the Invention]

[0010] The effects obtained by the representative inventions disclosed in this application will be briefly explained as follows.

[0011] That is, according to the representative embodiment of the present invention, it is possible to provide an imaging device, an imaging control method, and a program that generate a high-quality HDR composite image that is adapted to the imaging environment. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an external view showing an example of an imaging device 100 according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment of the present invention. [Figure 3] 1 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment of the present invention. [Figure 4] 1 is a block diagram showing an example of the configuration of a video signal processing unit and a video input unit according to a first embodiment of the present invention. [Figure 5] FIG. 2 is a flowchart illustrating the imaging method according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a flowchart showing an example of processing in a motion information detecting step, an imaging step, a captured image generating step, etc. according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a flowchart showing an example of processing in a motion information detecting step, an imaging step, a captured image generating step, etc. according to the first embodiment of the present invention. [Figure 8] FIG. 3 is a timing chart showing imaging according to the first embodiment of the present invention. [Figure 9] FIG. 2 is a diagram illustrating a schematic diagram of HDR merging processing according to the first embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating a schematic diagram of HDR merging processing according to the first embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating a schematic diagram of HDR merging processing according to the first embodiment of the present invention. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of an imaging device according to a second embodiment of the present invention. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of an imaging device according to a second embodiment of the present invention. [Figure 14] FIG. 10 is a flowchart illustrating an imaging method according to a second embodiment of the present invention. [Figure 15] FIG. 10 is a flowchart illustrating an imaging method according to a second embodiment of the present invention. [Figure 16]FIG. 10 is a diagram schematically illustrating HDR merging processing according to a second embodiment of the present invention. [Figure 17] FIG. 10 is a diagram schematically illustrating HDR merging processing according to a second embodiment of the present invention. [Figure 18] FIG. 10 is a diagram schematically illustrating HDR merging processing according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below are examples for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. Furthermore, the present invention may be configured by appropriately combining parts of each embodiment described below. (Embodiment 1) Fig. 1 is a diagram showing an example of the appearance of an imaging device according to Embodiment 1 of the present invention, Fig. 1(A) is a plan view of the imaging device, and Fig. 1(B) is a rear view of imaging device 100. Fig. 2 is a block diagram showing an example of the configuration of the imaging device according to Embodiment 1 of the present invention. Fig. 3 is a diagram showing an example of the software configuration of the imaging device according to Embodiment 1 of the present invention.

[0014] 1 illustrates an imaging device mounted on a smartphone, the imaging device may also be mounted on information processing devices such as mobile phones, PDAs (Personal Digital Assistants) such as tablet terminals, notebook PCs (Personal Computers), etc. The imaging device may also be configured as a digital still camera.

[0015] As shown in Figures 1 and 2, the imaging device 100 includes a main control unit (control unit) 101, a system bus 102, a memory unit 104, a storage unit 110, a video processing unit 120, an audio processing unit 130, an operation unit 140, a communication processing unit 150, a sensor unit 160, an extension interface unit 170, a display unit 121, etc.

[0016] The system bus 102 is a data communication path that connects the various components of the imaging device 100. Data is input and output between the main control unit 101 and the various components within the imaging device 100 via the system bus 102.

[0017] The storage unit 110 is composed of a non-volatile memory such as a flash ROM (Read Only Memory), an SSD (Solid State Drive), or an HDD (Hard Disc Drive), and stores information even when power is not supplied to the imaging device 100.

[0018] As shown in FIG. 3, the storage unit 110 includes a basic operation program memory area 110a that stores a basic operation program for executing the basic operations of the imaging device 100, a camera function program memory area 110b that stores a camera function program for realizing the camera function, a voice recognition program memory area 110c that stores a voice recognition program for realizing the voice recognition function, a timing notification program memory area 110d that stores a timing notification program for realizing the timing notification function, an other program memory area 110e that stores other programs, and a various information / data memory area 110f that stores various information such as operation setting values ​​of the imaging device 100 and user information.

[0019] The storage unit 110 also stores captured images captured by the imaging device 100, moving images, HDR composite images (described later), thumbnail images of these, etc., in, for example, a miscellaneous information / data storage area 110f. The storage unit 110 also stores new application programs downloaded from application servers on the Internet in, for example, an other program storage area 110e.

[0020] The storage unit 110 also stores a first motion threshold value that serves as a reference for determining the motion of a subject, which will be described later, and a second motion threshold value that is greater than the first motion threshold value. These motion threshold values ​​are set appropriately, for example, through experiments or the like.

[0021] In the memory unit 104, various programs stored in each unit of the storage unit 110 are loaded. When the various programs are executed by the main control unit 101, various execution units that realize the functions of the respective programs are configured in the memory unit 104. For example, when a basic operation program stored in the basic operation program storage area 110a is loaded in the memory unit 104, a basic operation execution unit 104a that realizes the basic operation of the imaging device 100 is configured in the memory unit 104. Furthermore, when a camera function program stored in the camera function program storage area 110b is loaded in the memory unit 104, a camera function execution unit 104b that realizes the camera function is configured in the memory unit 104. Furthermore, when a voice recognition function program stored in the voice recognition function program storage area 110c is loaded in the memory unit 104, a voice recognition function execution unit 104c that realizes the voice recognition function is configured in the memory unit 104. Furthermore, when the timing notification function program stored in the timing notification function program storage area 110d is expanded in the memory unit 104, a timing notification function execution unit 104d that realizes the timing notification function is configured in the memory unit 104. The memory unit 104 also has a temporary storage area 104e in which data is temporarily stored as needed.

[0022] Each of the basic operation execution unit 104a, camera function execution unit 104b, voice recognition function execution unit 104c, timing notification function execution unit 104d, etc. may be configured as hardware having the same functions as these units. Also, the memory unit 104 may be configured integrally with the main control unit 101.

[0023] 1(A), the display unit 121 is provided on the surface 100a of the imaging device 100 on which the third video input unit 125 is provided. The display unit 121 is a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel, and displays an HDR composite image (described later) processed by the video signal processing unit 122, captured images before composite, moving images, thumbnail images, etc.

[0024] The audio processing unit 130 is composed of an audio output unit 131, an audio signal processing unit 132, and an audio input unit 133. The audio output unit 131 is a speaker and is provided around the display unit 121 on the surface 100a of the imaging device 100, as shown in FIG. 1A, for example. The audio output unit 131 emits audio based on an audio signal processed by the audio signal processing unit 132. The audio input unit 133 is a microphone and is provided on a side of the imaging device 100, on the opposite side of the display unit 121 from the audio output unit 131, as shown in FIGS. 1A and 1B, for example. The audio input unit 133 receives audio input from the user of the imaging device 100, and converts the input audio into an audio signal. The audio input unit 133 outputs the input audio signal to the audio signal processing unit 132. The audio input unit 133 may be configured separately from the imaging device 100, and in this case, the audio input unit 133 and the imaging device 100 may be connected by wired communication or wireless communication.

[0025] The operation unit 140 is an instruction input unit that inputs operation instructions to the imaging device 100. In this embodiment, as shown in FIGS. 1A and 1B, for example, the operation unit 140 is configured with a touch panel 140a arranged on top of the display unit 121, operation keys 140b provided on the side of the imaging device 100, and operation keys 140c provided near the display unit 121 on the surface 100a of the imaging device 100. However, it is not necessary to provide all of these; it is sufficient to provide any one of them. The touch panel 140a may be configured integrally with the display unit 121. The operation unit 140 may also be configured with a keyboard (not shown) connected to the extended interface unit 170 (described later). The operation unit 140 may also be configured with a separate information terminal device connected via wired or wireless communication.

[0026] The communication processing unit 150 is configured with, for example, a LAN (Local Area Network) communication unit 151, a mobile telephone network communication unit 152, and a short-range wireless communication unit 153. The LAN communication unit 151 transmits and receives data via wireless communication connected via, for example, an Internet wireless communication access point. The mobile telephone network communication unit 152 is connected to a base station of the mobile telephone network and transmits and receives telephone communication (calls) and data via wireless communication via the base station. The short-range wireless communication unit 153 transmits and receives data to and from a reader / writer compatible with short-range wireless communication. The LAN communication unit 151, the mobile telephone network communication unit 152, and the short-range wireless communication unit 153 each include various devices such as an encoding circuit, a decoding circuit, and an antenna (not shown). The communication processing unit 150 may also include an infrared communication unit or the like that performs infrared communication.

[0027] The sensor unit 160 is a group of sensors that detect the state of the imaging device 100. In this embodiment, the sensor unit 160 is configured with, for example, a GPS (Global Positioning System) receiving unit 161, an acceleration sensor 162, a gyro sensor 163, a geomagnetic sensor 164, a light amount sensor 165, and a proximity sensor 166. Note that the sensor unit 160 may also include sensors other than these.

[0028] The GPS receiver 161 receives GPS signals transmitted from multiple satellites using GPS. The GPS signals received by the GPS receiver 161 are output to, for example, the main controller 101, which then detects the position of the imaging device 100 based on the GPS signals.

[0029] The acceleration sensor 162 measures the magnitude and direction of acceleration (e.g., gravitational acceleration) acting on the image capture device 100. The measured values ​​of the magnitude and direction of acceleration measured by the acceleration sensor unit 133 are output to the main control unit 101 as acceleration information, and the main control unit 101 detects the acceleration acting on the image capture device 100 based on the acceleration information.

[0030] The gyro sensor 163 measures the angular velocity of the imaging device 100 that occurs when the user moves the imaging device 100. The angular velocity measured by the gyro sensor unit 163 is output to the main control unit 101 as, for example, angular velocity information. The main control unit 101 detects the angular velocity of the imaging device 100 based on the angular velocity information.

[0031] The geomagnetic sensor 164 measures the magnitude and direction of the geomagnetic field acting on the imaging device 100. The measured values ​​of the magnitude and direction of the geomagnetic field measured by the geomagnetic sensor 164 are output as geomagnetic field information to the main control unit 101. The main control unit 101 detects the geomagnetic field acting on the imaging device 100 based on the geomagnetic field information.

[0032] The light intensity sensor 165 measures the brightness around the imaging device 100. For example, the light intensity sensor 165 measures the amount of light around a subject when capturing an image of the subject. The measured value of the amount of light measured by the light intensity sensor 165 is output to the main control unit 101 as light intensity information. The main control unit 101 detects the amount of light around the imaging device 100 based on the light intensity information.

[0033] The proximity sensor 166 measures the proximity status of the imaging device 100 to surrounding objects. The proximity sensor 166 measures, for example, the distance and direction of the imaging device 100 to surrounding objects. The measurement value of the proximity status detected by the proximity sensor 166 is output to the main control unit 101 as proximity status information. The main control unit 101 detects the proximity status of the imaging device 100 to surrounding objects based on the proximity status information.

[0034] The expansion interface unit 170 is a group of interfaces for expanding the functions of the imaging device 100. The expansion interface unit 170 includes, for example, a video / audio interface 171, a USB (Universal Serial Bus) interface 172, and a memory interface 173.

[0035] The video / audio interface 171 is connected to an external video / audio output device and accepts input of video signals / audio signals output from the video / audio output device. The video / audio interface 171 also outputs video signals / audio signals to the video / audio input device. The USB interface 172 is connected to a USB device such as a keyboard and inputs / outputs information to / from the USB device. The memory interface 173 is connected to a memory medium such as a memory card and inputs / outputs data to / from the memory medium.

[0036] Next, the video input unit 120 and the video signal processing unit 122 will be described. Fig. 4 is a block diagram showing an example of the configuration of the video input unit and the video signal processing unit according to the first embodiment of the present invention. Note that the third video input unit 125 is omitted in Fig. 4.

[0037] 2 and 4, the image input unit 120 includes a first image input unit 123, a second image input unit 124, a third image input unit 125, and an exposure control unit 126. As shown in FIG. 1(B), the first image input unit 123 and the second image input unit 124 are provided side by side on the back surface 100b of the imaging device 100. As shown in FIG. 1(A), the third image input unit 125 is provided near the display unit 121 on the front surface 100a of the imaging device 100.

[0038] 1(B), the first video input unit 123 and the second video input unit 124 are provided on the rear surface 100b of the imaging device 100, but may be provided on the front surface 100a of the imaging device 100, for example. Also, the first video input unit 123 and the second video input unit 124 may be configured as an integrated unit.

[0039] The first video input unit 123 includes a first imaging optical system 123a and a first imaging element 123b, as shown in Fig. 4. The second video input unit 124 includes a second imaging optical system 124a and a second imaging element 124b, as shown in Fig. 4.

[0040] The first imaging optical system 123a and the second imaging optical system 124a are configured with, for example, a plurality of lenses that collect incident light from a subject, an aperture (such as an iris aperture), a mechanical or electronic shutter, etc. The first imaging element 123b and the second imaging element 124b are configured with, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc.

[0041] The first image input unit 123 and the second image input unit 124 capture an image of a subject and generate an image signal of the subject. Specifically, in each pixel of the first image sensor 123b, light from the first image sensor optical system 123a is stored as an electric charge, thereby converting the input light into an electrical signal. Similarly, in each pixel of the second image sensor 124b, light from the second image sensor optical system 124a is stored as an electric charge, thereby converting the input light into an electrical signal. In this way, the first image sensor 123 and the second image sensor 124 generate an electrical signal as an image signal.

[0042] In this embodiment, the first imaging element 123b is an imaging element with a first resolution (high resolution), and the second imaging element 124b is an imaging element with a second resolution (low resolution) that is lower in resolution than the first imaging element 123b.

[0043] Based on instructions from the main control unit 101, the exposure control unit 126 specifies the aperture and shutter speed of the first imaging optical system 123a and the second imaging optical system 124a, and controls the amount of exposure input to the first imaging element 123b and the second imaging element 124b.

[0044] The video signal processing unit 122 generates a captured image of the subject based on the image signal. As shown in Fig. 4, the video signal processing unit 122 includes a first image processing unit 122a, a second image processing unit 122b, an image synthesis unit 122c, and an image output unit 124d.

[0045] The first image processing unit 122a is connected to the first imaging element 123b and converts the electrical signal (image signal) output from the first imaging element 123b into digital image data having a gradation width of a predetermined number of bits. The first image processing unit 122a then performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The first image processing unit 122a outputs the generated intermediate image data to the image synthesis unit 122c. The first image processing unit 122a also outputs the captured image before synthesis to the storage unit 110.

[0046] The second image processing unit 122b is connected to the second imaging element 124b and converts the electrical signal (image signal) output from the second imaging element 124b into digital image data having a gradation width of a predetermined number of bits. The second image processing unit 122b then performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The second image processing unit 122b outputs the generated intermediate image data to the image synthesis unit 122c. The second image processing unit 122b also outputs the captured image before synthesis to the storage unit 110.

[0047] The image synthesis unit 122c synthesizes the intermediate image data input from the first image processing unit 122a and the second image processing unit 122b to generate an HDR synthesized image as a captured image. The image synthesis unit 122c outputs the generated HDR synthesized image to the storage unit 110.

[0048] The image output unit 122d outputs the HDR composite image generated by the image composition unit 122c, the captured image before composition, the moving image, thumbnail images thereof, and the like to the display unit 121 for display.

[0049] 1(B), for example, flash unit 129 is provided adjacent to first image input unit 123 and second image input unit 124 on rear surface 100b of imaging device 100. Flash unit 129 irradiates a subject with flash light when first image input unit 123 and second image input unit 124 capture an image, for example.

[0050] The main control unit (control unit) 101 is configured with a computer such as a microprocessor unit. The main control unit 101 executes each of the execution units 100a to 100d configured in the memory unit 104 to realize the functions of each program and operate each component of the imaging device 100.

[0051] The main control unit 101 detects motion information of the subject based on the image signal generated by the video input unit 120. Based on the motion information, the main control unit 101 causes the video input unit 120 to capture multiple images of the subject using different exposure amounts. Specifically, the main control unit 101 compares the motion information with a first motion threshold and a second motion threshold, and based on the comparison result, switches to each imaging mode described below to capture an image of the subject. Based on the image signal or the captured image, the main control unit 101 generates exposure information such as exposure amount, shutter speed, and aperture for the first video input unit 123 and the second video input unit 124. The main control unit 101 outputs this information to the video input unit 120. The main control unit 101 also causes the video signal processing unit 122 to generate an HDR composite image of the subject based on multiple image signals with different exposure amounts.

[0052] Next, an imaging method according to the present embodiment will be described. In the present embodiment, for example, various programs such as the camera function program 110b shown in Fig. 3 are loaded into the memory unit 104, and the main control unit 101 executes each execution unit such as the camera function execution unit 104b, thereby performing operations related to imaging.

[0053] Fig. 5 is a flowchart illustrating the imaging method according to Embodiment 1 of the present invention. As shown in Fig. 5, imaging involves performing a video input step S10, a motion information detection step S20, an imaging step S30, and an HDR composite image generation step S40.

[0054] When an instruction to capture an image is given from operation unit 140, an operation related to capturing an image of the subject is started. In image input step S10, image input unit 120 captures an image of the subject and generates an image signal of the subject. First image input unit 123 and second image input unit 124 capture images of the subject, and first image sensor 123b and second image sensor 124b generate electrical signals as image signals. First image sensor 123b outputs the generated electrical signals to first image processing unit 122a of video signal processing unit 122, and second image sensor 124b outputs the generated electrical signals to second image processing unit 122b of video signal processing unit 122. Note that although the case where both first image input unit 123 and second image input unit 124 capture an image of the subject has been described here, it is also possible for only one of the image input units to capture an image of the subject. Then, the process proceeds to motion information detection step S20.

[0055] 6 and 7 are flowcharts showing examples of processing in the motion information detecting step, imaging step, captured image generating step, etc. according to the first embodiment of the present invention. In the motion information detecting step S20, the main control unit 101 detects motion information of the subject based on the image signal. Specifically, the first image processing unit 122a of the video signal processing unit 122 outputs the electrical signal output from the first imaging element 123b to the main control unit 101. The second image processing unit 122b of the video signal processing unit 122 outputs the electrical signal output from the second imaging element 124b to the main control unit 101. Note that the first image processing unit 122a and the second image processing unit 122b may generate captured images based on the electrical signals and output the generated captured images to the main control unit 101.

[0056] The main control unit 101 detects motion information (e.g., a motion vector) of the subject based on these input electrical signals (image signals). The main control unit 101 detects the motion vector of the subject by using a well-known block matching method or the like for these image signals. Note that the main control unit 101 may also detect the motion information of the subject based on the input captured image. Then, the process proceeds to the imaging step S30.

[0057] In the imaging step S30, the video input unit 120 images the subject multiple times with different exposure amounts based on the motion information detected in the motion information detection step S20. In the imaging step S30, step S102 is first performed. In step S102, the motion of the subject is determined based on the motion information. Specifically, the main control unit 101 determines the motion of the subject by comparing the motion vector detected in the motion information detection step S20 with a first motion threshold and a second motion threshold. In more detail, the main control unit 101 reads the first motion threshold and the second motion threshold stored in the storage unit 110 into the memory 104 and compares the motion vector with the first motion threshold and the second motion threshold.

[0058] [Still image capture mode] If the main control unit 101 determines in step S102 that the motion vector is less than the first motion threshold, step S113 is performed. That is, the main control unit 101 determines that the subject is barely moving, and switches to a still image capture mode.

[0059] FIG. 8 is a diagram showing a timing chart for imaging according to the first embodiment of the present invention. FIG. 9 is a diagram showing a schematic diagram of HDR compositing processing according to the first embodiment of the present invention. In step S113, the first video input unit 123 successively captures images of the subject with different exposure amounts, as shown in FIG. 8(A), for example. Specifically, the main control unit 101 determines the exposure amount (for example, shutter speed, aperture value, etc.) for each image capture based on the electrical signal (image signal) output from the first image capture element 123b or the second image capture element 124b. For example, as shown in FIG. 8(A), the main control unit 101 determines the exposure amount for the first image capture to be an exposure amount L that prevents dark areas on the low-illuminance side from being crushed black. A0 Then, set the exposure for the second capture to L to prevent the bright areas on the high-illumination side from being blown out. A0 Less exposure L A1 Set to (L A0 >L A1The main control unit 101 outputs information relating to the determined exposure amount as exposure information to the image input unit 120. In the image input unit 120, the first image input unit 123 determines the exposure amount L A0 The subject is photographed with exposure L A1 Capture the subject with.

[0060] Each time an image of a subject is captured, the first image sensor 123b generates an electrical signal relating to the captured image, for example, as shown in Fig. 9. Each time an image of a subject is captured, the first image sensor 123b generates an electrical signal relating to the captured image, for example, as shown in Fig. 9. A0 Electrical signal (A0), exposure amount L A1 The generated electrical signal (A0, A1) is output to the first image processing unit 122a of the video signal processing unit 122. Note that although the case where the subject is imaged twice with different exposure amounts is taken here, the subject may also be imaged three or more times, for example.

[0061] On the other hand, the second video input unit 124 captures, for example, a moving image (Bm) of a subject. The second video input unit 124 may also capture the subject with different depths of field, for example, by changing the aperture value. The second imaging element 124b outputs an image signal relating to the moving image or the like to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to an HDR composite image generation step S40.

[0062] In the HDR composite image generating step S40, step S114 is performed. In step S114, an HDR composite image of a first resolution is generated based on the multiple image signals (A0, A1) with different exposures generated in step S113. Specifically, the first image processing unit 122a of the video signal processing unit 122 converts each of the electrical signals (A0, A1) input from the first imaging element 123b into digital image data having a gradation width of a predetermined number of bits. Then, the first image processing unit 122a performs image processing on each digital image data to generate each intermediate image data suitable for HDR image synthesis. Then, the first image processing unit 122a outputs each intermediate image data to the image synthesis unit 122c. Furthermore, the first image processing unit 122a generates each captured image before synthesis based on the electrical signals (A0, A1). Furthermore, the first image processing unit 122a generates thumbnail images corresponding to each captured image before synthesis.

[0063] The second image processing unit 122b generates a moving image based on the image signal input from the second video input unit 124, or generates captured images with different depths of field. The second image processing unit 122b also generates thumbnail images corresponding to the generated moving images and captured images.

[0064] The image synthesis unit 122c generates an HDR synthesis image (captured image) (A0+A1) with a first resolution (high resolution) as shown in Fig. 9 by weighting the intermediate image data input from the first image processing unit 122a. The HDR synthesis image (A0+A1) generated in this way eliminates blocked-up shadows in dark areas and blown-out highlights in bright areas, making it an image closer to the scenery seen by the user with the naked eye. The image synthesis unit 122c also generates a thumbnail image corresponding to the generated HDR synthesis image.

[0065] Alternatively, the first image processing unit 122a and the second image processing unit 122b may convert the high-resolution image signals into low-resolution image signals, and the image synthesis unit 122c may generate a low-resolution HDR synthesized image. Then, the process proceeds to step S115.

[0066] In step S115, the HDR composite image generated in step S114, the captured images before composition, the video, the thumbnail images, etc. are stored in the storage unit 110. Specifically, the image composition unit 122c outputs the generated HDR composite image (A0+A1) and the thumbnail images to the storage 110, and the storage 110 stores the input HDR image composition (A0+A1) and the thumbnail images. Furthermore, the first image processing unit 122a and the second image processing unit 122b output the captured images before composition, the video, and the thumbnail images to the storage 110, and the storage 110 stores the input captured images before composition, the video, and the thumbnail images. When these processes are performed, the process of HDR composite image generation step S40 is completed, and the process proceeds to step S104. Step S104 will be described later.

[0067] [Slow motion imaging mode] 10 and 11 are diagrams schematically illustrating HDR blending processing according to the first embodiment of the present invention. If the main control unit 101 determines in step S102 of the imaging step S30 that the motion vector is equal to or greater than the first motion threshold, step S123 or step S133 is performed. More specifically, if the main control unit 101 determines that the motion vector is equal to or greater than the first motion threshold and equal to or less than the second motion threshold, step S123 is performed. That is, the main control unit 101 determines that the subject is moving more than in the still image imaging mode but less than in the moving object imaging mode described below, and switches to the micromotion imaging mode.

[0068] In step S123, the main control unit 101 causes the first video input unit 123 and the second video input unit 124 to simultaneously capture an image of the subject with different exposure amounts, and then causes the second video input unit 124 to capture an image of the subject with different exposure amounts. Specifically, as shown in FIG. 8(B), for example, the main control unit 101 sets the exposure amount for capturing an image of the first video input unit 123 to an exposure amount L that prevents a dark subject on the low-illuminance side from being crushed into black. A0In other words, the first video input unit 123 captures an image of the subject with appropriate exposure. Note that appropriate exposure here means capturing an image of a dark subject on the low-illuminance side with sufficient exposure. Furthermore, the main control unit 101 sets the exposure amount in the first image capture by the second video input unit 124 to, for example, an exposure amount L between an exposure amount that causes blackout on the low-illuminance side and an exposure amount that causes whiteout on the high-illuminance side. B0 and the exposure amount in the second image capture by the second video input unit 124 is set to L B0 Less exposure L A1 Set to (L A0 >L B0 >L B1 ).

[0069] Furthermore, because the resolution of second image sensor 124b is lower than that of first image sensor 123b, the area per pixel is larger and the amount of exposure increases, so second video input unit 124 can obtain the same amount of exposure even with a shorter imaging time (faster shutter speed) than first video input unit 123. Therefore, the imaging interval (time difference) in second video input unit 124 can be made shorter than that of first video input unit 123, so that noise caused by slight movement of the subject when HDR combining processing is performed can be kept low.

[0070] The main control unit 101 outputs information relating to the determined exposure amount as exposure information to the image input unit 120. In the image input unit 120, the first image input unit 123 determines the exposure amount L A0 The second image input unit 124 captures an image of the subject with an exposure amount L B0 The subject is photographed with exposure L B1 Capture the subject with.

[0071] The first image sensor 123b is exposed to light with an exposure amount L A0 The generated electrical signal (A0) is output to the first image processing unit 122a of the video signal processing unit 122.

[0072] The second image sensor 124b detects an exposure amount L as shown in FIG. 10(A), for example, every time it captures an image of a subject. B0 Electrical signal (B0), exposure amount L B1 The generated electrical signal (B0, B1) is output to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to the HDR composite image generating step S40.

[0073] In the HDR composite image generating step S40, first, step S124 is performed. In step S124, the resolution of the captured image of the electrical signal generated in step S123 is converted. Specifically, the main control unit 101 causes the video signal processing unit 122 to convert the resolution of the image signal generated by the first video input unit 123 to the second resolution, and to convert the resolution of the image signal generated by the second video input unit 124 to the first resolution.

[0074] Specifically, the main control unit 101 instructs the first image processing unit 122a to convert the resolution of the electrical signal (A0) generated by the first imaging element 123b from the first resolution to the second resolution, and instructs the second image processing unit 122b to convert the resolution of the electrical signal (B0) generated by the second imaging element 124b from the second resolution to the first resolution. More specifically, the first image processing unit 122a performs resolution conversion on the electrical signal (A0) of the first resolution to generate an electrical signal (A0d) of the second resolution as shown in FIG. 10B. The second image processing unit 122b performs resolution conversion on the electrical signal (B0) of the second resolution, for example, by performing interpolation processing or the like to generate an electrical signal (B0u) of the first resolution as shown in FIG. 10A. Then, the process proceeds to step S125.

[0075] In step S125, an HDR composite image of a first resolution and an HDR composite image of a second resolution are generated based on the multiple image signals (A0, B0, B1, A0d, B0u) generated in step S124. Specifically, the first image processing unit 122a converts each of the electrical signals (A0) of the first resolution and the electrical signal (A0d) of the second resolution into digital image data having a gradation width of a predetermined number of bits. The first image processing unit 122a then performs image processing on each of the digital image data to generate each of intermediate image data suitable for HDR image synthesis. The first image processing unit 122a then outputs each of the intermediate image data to the image synthesis unit 122c. The first image processing unit 122a also generates each of the captured images before synthesis based on the electrical signals (A0, A0d). The first image processing unit 122a also generates thumbnail images corresponding to each of the captured images before synthesis.

[0076] The second image processing unit 122b converts each of the electrical signals (B0, B1) of the second resolution and the electrical signal (B0u) of the first resolution into digital image data having a gradation width of a predetermined number of bits. The second image processing unit 122b then performs image processing on each piece of digital image data to generate each piece of intermediate image data suitable for HDR image synthesis. The second image processing unit 122b then outputs each piece of intermediate image data to the image synthesis unit 122c. The second image processing unit 122b also generates each captured image before synthesis based on the electrical signals (B0, B1, B0u). The second image processing unit 122b also generates thumbnail images corresponding to each of the generated captured images before synthesis.

[0077] The image synthesis unit 122c generates an HDR synthesis image of a first resolution and an HDR synthesis image of a second resolution by weighting the intermediate image data output from the first image processing unit 122a and the intermediate image data output from the second image processing unit 122b.

[0078] Specifically, the image synthesis unit 122c generates an HDR synthesized image (A0+B0u) of a first resolution, for example, as shown in FIG. 10A, based on the intermediate image data of the electrical signal (A0) and the intermediate image data of the electrical signal (B0u). The image synthesis unit 122c generates an HDR synthesized image (A0d+B0) of a second resolution, for example, as shown in FIG. 10B, based on the intermediate image data of the electrical signal (A0d) and the intermediate image data of the electrical signal (B0). The image synthesis unit 122c generates an HDR synthesized image (B0+B1) of a second resolution, for example, as shown in FIG. 11A, based on the intermediate image data of the electrical signal (B0) and the intermediate image data of the electrical signal (B1). The image synthesis unit 122c generates an HDR composite image (A0d+B0+B1) of a second resolution, for example, as shown in Fig. 11(B), based on the intermediate image data of the electrical signal (A0d) and the intermediate image data of the electrical signals (B0, B1). The image synthesis unit 122c also generates a thumbnail image corresponding to the generated HDR composite image.

[0079] When the user later views the HDR composite image, the user may select the image that is visually judged to be the best from these images and treat it as the image resulting from the final HDR blending process. Then, the process proceeds to step S126.

[0080] In step S126, the HDR composite image generated in step S125, the captured images before composition, thumbnail images, etc. are stored in the storage unit 110. Specifically, the image composition unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), (B0+B1), and (A0d+B0+B1) and thumbnail images to the storage 110, and the storage 110 stores the input HDR image composites (A0+B0u), (A0d+B0), (B0+B1), and (A0d+B0+B1) and thumbnail images. In addition, the first image processing unit 122a and the second image processing unit 122b output the captured images before composition and thumbnail images to the storage 110, and the storage 110 stores the input captured images before composition and thumbnail images. When these processes are performed, the process of HDR composite image generation step S40 is completed, and the process proceeds to step S104. Step S104 will be described later.

[0081] [Motion capture mode] If the main control unit 101 determines in step S102 of the imaging step S30 that the motion vector exceeds the second motion threshold, step S133 is executed. That is, the main control unit 101 determines that the motion of the subject is greater than that in the micromotion imaging mode, and switches to the moving object imaging mode.

[0082] In step S133, the same process as in step S123 is performed. That is, in step S123, the main control unit 101 causes the first image input unit 123 and the second image input unit 124 to simultaneously capture images of the subject with different exposure amounts. The first image input unit 123 uses the exposure amount L A0 The second image input unit 124 captures an image of the subject at a first resolution, for example, as shown in FIGS. 10(A) and 10(B), and generates an electrical signal (A0) for the captured image at a first resolution. B010(A) and (B), an electrical signal (B0) for the captured image at the second resolution is generated. Note that in the moving object imaging mode, the second video input unit 124 does not capture images of the object with a time lag, as in the micromotion imaging mode. This is because the movement of the object is large, and capturing images with a time lag would result in noticeable object shake in the generated HDR composite image.

[0083] The first image sensor 123b has an exposure amount L A0 The second image sensor 124b outputs the electric signal (A0) to the first image processor 122a. B0 The electric signal (B0) is output to the second image processing unit 122b, and the process then proceeds to the HDR composite image generating step S40.

[0084] In the HDR composite image generating step S40, first, step S134 is performed. In step S134, the same processing as step S124 described above is performed. That is, in step S134, the first image processing unit 122a performs resolution conversion on the electrical signal (A0) of the first resolution to generate, for example, an electrical signal (A0d) of the second resolution as shown in FIG. 10(B). The second image processing unit 122b performs resolution conversion on the electrical signal (B0) of the second resolution to generate an electrical signal (B0u) of the first resolution as shown in FIG. 10(A). Then, the process proceeds to step S135.

[0085] In step S135, substantially the same processing as in step S125 described above is performed. That is, in step S135, an HDR composite image of a first resolution and an HDR composite image of a second resolution are generated based on the multiple image signals (A0, B0, A0d, B0u) generated in step S134. Specifically, the image synthesis unit 122c generates the HDR composite image of the first resolution (A0+B0u) shown in FIG. 10(B) and the HDR composite image of the second resolution (A0d+B0) shown in FIG. 10(A). In addition, the first image processing unit 122a and the second image processing unit 122b generate captured images before synthesis and thumbnail images corresponding to the captured images before synthesis, respectively.

[0086] When the user later views the HDR composite image, the user may select the image that is visually judged to be the best from these images and treat it as the image resulting from the final HDR blending process. Then, the process proceeds to step S126.

[0087] In step S126, the HDR composite image generated in step S135, the captured images before composition, thumbnail images, etc. are stored in the storage unit 110. Specifically, the image composition unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), and thumbnail images to the storage 110, and the storage 110 stores the input HDR image compositions (A0+B0u), (A0d+B0), and thumbnail images. In addition, the first image processing unit 122a and the second image processing unit 122b output the captured images before composition and thumbnail images to the storage 110, and the storage 110 stores the input captured images before composition and thumbnail images. When these processes are performed, the process of HDR composite image generation step S40 is completed, and the process proceeds to step S104.

[0088] In step S104, an image or the like to be displayed on the display unit 121 is selected. Specifically, an image or the like to be displayed on the display unit 121 is selected from the images or the like (HDR composite image, captured image before composition, moving image) generated in each step S114, S125, S135 of each imaging mode and stored in the storage unit 110.

[0089] For example, storage unit 110 outputs information (e.g., thumbnail images) relating to stored images, etc. to image output unit 122d. Image output unit 122d outputs the input thumbnail images to display unit 121. Display unit 121 displays the input thumbnail images. The user selects an image, etc. to be displayed on display unit 121 from the displayed thumbnail images. Then, the process proceeds to step S105.

[0090] In step S105, an image or the like corresponding to the selected thumbnail image is displayed. Specifically, the image output unit 122d outputs, for example, an image or the like corresponding to the thumbnail image selected by the user from the storage unit 110. The image output unit 122d outputs the read image or the like to the display unit 121. The display unit 121 displays the input image or the like.

[0091] The user may arbitrarily select the images etc. to be displayed on the display unit 121, or may register the display priority in advance in the imaging device 100 so that the images etc. are displayed in order of priority.

[0092] Furthermore, images etc. displayed on display unit 121 may be enlarged or reduced as appropriate in accordance with the resolution of display unit 121. At this time, multiple images etc. may be displayed at the same size, or multiple images etc. may be displayed at different sizes in accordance with the resolution of the selected image etc.

[0093] It should be noted that the imaging device 100 according to this embodiment is not limited to the configuration shown in Fig. 2. For example, the imaging device 100 may not include the communication processing unit 150 or the sensor unit 160, and may include various functions such as a digital television broadcast receiving function and an electronic money payment function.

[0094] According to this embodiment, based on the motion vector of the subject, the main control unit 101 causes the video input unit 120 to capture images of the subject multiple times with different exposure amounts, and causes the video signal processing unit to generate an HDR composite image of the subject based on multiple image signals with different exposure amounts.

[0095] According to this configuration, an appropriate imaging mode can be selected according to the movement of the subject, and a high-quality HDR composite image can be generated that is adapted to the imaging environment, such as the movement of the subject and camera shake.

[0096] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is less than the first motion threshold, it switches to a still image capturing mode, causes the first video input unit 123 having a first image sensor 123b with a high first resolution to continuously capture images of the subject with different exposure amounts, and causes the video signal processing unit 122 to generate an HDR composite image with the first resolution based on multiple image signals (A0, A1) with different exposure amounts.

[0097] With this configuration, since the subject hardly moves in still image capture mode, even if the subject is captured with a time lag, noise caused by subject movement in the HDR composite image is suppressed. This allows the subject to be captured continuously using only high-resolution image sensors, resulting in the generation of a high-quality HDR composite image.

[0098] Furthermore, with this configuration, an electrical signal (A0) generated by capturing a dark subject on the low-light side with sufficient exposure is combined with an electrical signal (A1) generated by capturing a bright subject on the high-light side with reduced exposure, thereby generating a high-quality HDR composite image with an expanded dynamic range.

[0099] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the first motion threshold, it switches to micromotion imaging mode or moving object imaging mode, causes the first video input unit 123 and the second video input unit 124 to simultaneously capture images of the subject with different exposure amounts, causes the image signal processing unit 122 to convert image signals of the first resolution into image signals of the second resolution, and convert image signals of the second resolution into image signals of the first resolution, and generates an HDR composite image of the first resolution and an HDR composite image of the second resolution.

[0100] With this configuration, even if the subject is moving, an HDR composite image is generated based on multiple image signals captured simultaneously, thereby reducing the effects of noise caused by subject movement or camera shake, and generating a high-quality HDR composite image with an expanded dynamic range.

[0101] Furthermore, with this configuration, a plurality of HDR images with different resolutions are generated, so that an HDR image suited to the user's purpose is provided, and an imaging device 100 that is easy to use is provided.

[0102] Furthermore, according to this embodiment, when the control unit 101 determines that the motion vector is greater than or equal to the first motion threshold and less than or equal to the second motion threshold, it switches to micromotion imaging mode, causes the first video input unit 123 and the second video input unit 124 to simultaneously image the subject with different exposure amounts, and then causes the second video input unit 124 to image the subject with different exposure amounts.

[0103] According to this configuration, HDR blending processing is performed based on at least three types of image signals with different exposure amounts, so that a higher quality HDR blended image is generated.

[0104] Furthermore, according to this embodiment, the resolution of second imaging element 124b is lower than the resolution of first imaging element 123b.

[0105] According to this configuration, second video input unit 124 can obtain the same amount of exposure even with a shorter imaging time than first video input unit 123. As a result, even if second video input unit 124 images the subject with a time lag, the occurrence of noise due to the movement of the subject is suppressed, and therefore the occurrence of noise due to the movement of the subject in the HDR composite image is suppressed.

[0106] Furthermore, with this configuration, the cost of the second imaging element 124 can be reduced, and therefore the manufacturing cost of the imaging device 100 can be reduced.

[0107] Furthermore, according to the present embodiment, the display unit 121 displays thumbnail images such as HDR composite images, and when a thumbnail image is selected, the display unit 121 displays the HDR composite image corresponding to the selected thumbnail image.

[0108] According to this configuration, the user can easily identify the HDR composite image etc. stored in the storage unit 110, and therefore the burden on the user when selecting the HDR image etc. to be displayed on the display unit 121 is reduced.

[0109] Furthermore, according to this embodiment, the process includes a video input step S10 in which video input unit 120 images a subject and generates an image signal of the subject, a motion information detection step S20 in which main control unit 101 detects motion information of the subject based on the image signal, an imaging step S30 in which main control unit 101 causes video input unit 120 to image the subject multiple times with different exposure amounts based on the motion information, and an HDR composite image generation step S40 in which video signal processing unit 122 generates an HDR composite image of the subject based on the multiple image signals with different exposure amounts.

[0110] According to this configuration, an appropriate imaging mode can be selected according to the movement of the subject to capture the subject, thereby providing an imaging method that can generate a high-quality HDR composite image that is suited to the imaging environment.

[0111] Furthermore, according to this embodiment, main control unit 101, which is a computer, executes a video input step S10 in which video input unit 120 images a subject and generates an image signal of the subject; a motion information detection step S20 in which motion information of the subject is detected based on the image signal; an imaging step S30 in which the video input unit images the subject multiple times with different exposure amounts based on the motion information; and an HDR composite image generation step S40 in which video signal processing unit 122 generates an HDR composite image of the subject based on multiple image signals with different exposure amounts.

[0112] According to this configuration, an appropriate imaging mode can be selected according to the movement of the subject to capture the subject, thereby providing an imaging program that can generate a high-quality HDR composite image that is suited to the imaging environment.

[0113] (Embodiment 2) Next, a second embodiment will be described. In this embodiment, a case where an imaging mode is switched based on the movement of a subject and the amount of light will be described. Note that, in the following, detailed descriptions of parts that overlap with the first embodiment may be omitted as appropriate.

[0114] 12 and 13 are block diagrams showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention. As shown in FIG.

[0115] As shown in Figs. 12 and 13, the video input unit 220 includes a second video input unit 224, a resolution conversion unit 228, etc. The second video input unit 224 includes a second image pickup element 224b. The second image pickup element 224b is an image pickup element with a first resolution (high resolution) like the first image pickup element 123b. The first image pickup element 123b and the second image pickup element 224b are configured so that the resolutions thereof are appropriately converted by the resolution conversion unit 228.

[0116] The light quantity sensor 165 measures the quantity of light around the imaging device 200 and near the subject. The light quantity sensor 165 then outputs the measured light quantity information to the exposure control unit 126 and the main control unit 101.

[0117] The exposure control unit 126 sets the exposure amount when the first video input unit 123 and the second video input unit 224d capture images, based on, for example, light amount information output from the light amount sensor 165 and instructions from the main control unit 101.

[0118] The main control unit 101 switches the imaging mode based on the amount of light measured by the light amount sensor 165 and the detection result of the motion vector of the subject. The main control unit 101 also causes the resolution conversion unit 228 to convert the resolution of the first imaging element 123b and the second imaging element 224b from the first resolution (high resolution) to the second resolution (low resolution) by grouping multiple pixels. The main control unit 101 also causes the resolution conversion unit 228 to ungroup multiple pixels, thereby converting the resolution of the first imaging element 123b and the second imaging element 224b from the second resolution (low resolution) to the first resolution (high resolution).

[0119] For example, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the first imaging element 123b and the second imaging element 224b according to the imaging mode. The resolution conversion unit 228 converts the resolution of the first imaging element 123b and the second imaging element 224b based on the resolution conversion information. In the imaging element converted to a lower resolution, the grouped pixels are treated as a single pixel. As a result, the area of ​​one pixel after grouping is larger than that of one pixel before grouping, and the amount of exposure increases, so the exposure time (shutter speed) to obtain the same amount of exposure is shortened.

[0120] The light quantity sensor 165 measures the quantity of light from a high-light area to a low-light area of ​​the subject, for example. 12 and 13 show an example in which the light quantity sensor 165 is provided independently, but the present invention is not limited to such a configuration. For example, the first image sensor 123b or the second image sensor 224b may also have the function of the light quantity sensor 165.

[0121] Next, an imaging method according to this embodiment will be described. Figures 14 and 15 are flowcharts illustrating an imaging method according to embodiment 2 of the present invention. This embodiment also performs the video input step S10, motion information detection step S20, imaging step S30, and HDR composite image generation step S40 shown in Figure 5. The video input step S10 and motion information detection step S20 are the same as those in embodiment 1 described above.

[0122] In the imaging step S30, step S202 is first performed. In step S202, the motion of the subject is determined based on the motion information. Specifically, the main control unit 101 determines the motion of the subject by comparing the motion vector detected in the motion information detection step S20 with a third motion threshold. More specifically, the main control unit 101 reads the third motion threshold stored in the storage unit 110 into the memory 104 and compares the motion vector with the third motion threshold. The third motion threshold is set as appropriate, for example, through experiments or the like.

[0123] [Still image capture mode] If the main control unit 101 determines in step S202 that the motion vector is less than the third motion threshold, step S214 is performed. That is, the main control unit 101 determines that the subject is barely moving, and switches to a still image capture mode.

[0124] In step S214, the same process as in step S113 in the first embodiment is performed. That is, first video input unit 123 successively captures an image of the subject with different exposure amounts. In the first capture, first video input unit 123 uses, for example, exposure amount L A0 The subject is captured with the L A0 Less exposure L A1 Take a picture with (L A0 >L A1 9, for example, and outputs the generated electrical signals (A0, A1) to the first image processing unit 122a. Note that although the case where two types of image signals with different exposure amounts are combined and processed will be described here, for example, three or more types of image signals may be combined and processed.

[0125] On the other hand, the second video input unit 224 may capture a moving image (Bm) of the subject, for example, or may capture an image of the subject with different depths of field. In addition, since the second image sensor 224b of the second video input unit 224 also has high resolution, it may capture an image of the subject with different exposure amounts (for example, L A1 The second image sensor 224b may capture an image of the subject simultaneously with the first image input unit 123. A0 Electrical signal (A0), exposure amount L A1 The second image sensor 224b generates an electric signal (A1) based on the image signal A0 and outputs the generated electric signal (A0, A1) to the first image processing unit 122a. The second image sensor 224b outputs an image signal related to a moving image or the like to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to an HDR composite image generation step S40.

[0126] In the HDR composite image generating step S40, step S215 is performed. In step S215, the same processing as step S114 in the first embodiment is performed. That is, the image synthesis unit 122c generates an HDR composite image (A0+A1) with a first resolution based on multiple image signals (A0, A1) with different exposure amounts generated in step S214. The image synthesis unit 122c may also generate the HDR composite image based on electrical signals generated by the first imaging element 123b and the second imaging element. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate captured images and moving images before synthesis, and thumbnail images corresponding to these images.

[0127] Alternatively, the first image processing unit 122a and the second image processing unit 122b may convert high-resolution image signals into low-resolution image signals, and the image synthesis unit 122c may generate a low-resolution HDR synthesized image. Then, the process proceeds to step S216.

[0128] In step S216, the same process as in step S115 in embodiment 1 is performed. That is, the HDR composite image generated in step S215, the captured images before composition, the moving images, the thumbnail images, and the like are stored in the storage unit 110.

[0129] [Motion capture mode] 16 is a diagram schematically illustrating HDR blending processing according to the second embodiment of the present invention. In step S202 of the imaging step S30, if the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold, it determines that the subject is moving significantly and switches to moving object imaging mode. Once the mode is switched to moving object imaging mode, the process proceeds to step S203.

[0130] In step S203, the imaging mode is switched based on the amount of light measured by the light amount sensor 165. Specifically, the light amount sensor 165 measures the amount of light around the imaging device 200 and outputs information about the measured amount of light to the main control unit 101 as light amount information. Based on the input light amount information, the main control unit 101 compares the measured amount of light with a first light amount threshold and a second light amount threshold that is greater than the first light amount threshold. More specifically, the main control unit 101 reads the first light amount threshold and the second light amount threshold stored in the storage unit 110 into the memory 104 and compares the amount of light with the first light amount threshold and the second light amount threshold. The first light amount threshold and the second light amount threshold are set as appropriate, for example, through experiments, etc.

[0131] <High-light moving subject imaging mode> If the main control unit 101 determines that the light amount exceeds the second light amount threshold, the process proceeds to step S224. That is, the main control unit 101 determines that the light amount is high, and switches to the high-light-amount moving object imaging mode.

[0132] In step S224, the main control unit 101 causes the first image input unit 123 and the second image input unit 224 to simultaneously capture images of the subject with different exposure amounts. At this time, the main control unit 101 sets the resolution of the first image sensor 123b and the second image sensor 224b to the first resolution, which is a high resolution.

[0133] For example, if the resolution of the first imaging element 123b or the second imaging element 224b is set to a low resolution, the main control unit 101 outputs resolution conversion information for converting the resolution of the corresponding imaging element to the resolution conversion unit 228. The resolution conversion unit 228 converts the resolution of the corresponding imaging element from a low resolution to a high resolution based on the input resolution conversion information.

[0134] When the resolution of the image sensor is set in this way, the first image input unit 123 outputs the exposure amount L A0 The second video input unit 124 captures an image of a subject with a high resolution, for example, as shown in FIG. 16, and generates an electrical signal (A0) for the captured image. A0 Less exposure L B0 Shoot the subject with (L A0 >L B0 ), for example, an electrical signal (B0) for a high-resolution captured image as shown in FIG. 16 is generated.

[0135] The first image sensor 123b has an exposure amount L A0 The second image sensor 124b outputs the electric signal (A0) to the first image processor 122a. B0 The electric signal (B0) is output to the second image processing unit 122b, and the process then proceeds to the HDR composite image generating step S40.

[0136] In the HDR composite image generating step S40, step S225 is performed. In step S225, the image synthesis unit 122c generates a high-resolution HDR composite image (A0+B0) as shown in FIG. 16, for example, based on the multiple image signals (A0, B0) generated in step S224. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate captured images and moving images before synthesis, and thumbnail images corresponding to these images.

[0137] Alternatively, the first image processing unit 122a and the second image processing unit 122b may convert high-resolution image signals into low-resolution image signals, and the image synthesis unit 122c may generate a low-resolution HDR synthesized image. Then, the process proceeds to step S237.

[0138] In step S237, the same process as in step S126 in embodiment 1 is performed. That is, the HDR composite image generated in step S225, the captured images before composition, the moving images, the thumbnail images, etc. are stored in the storage unit 110. Then, the process proceeds to step S204. Step S204 will be described later.

[0139] <Medium-light moving subject imaging mode> 17 is a diagram schematically illustrating HDR blending processing according to Embodiment 2 of the present invention. If the main control unit 101 determines that the light amount is equal to or greater than the first light amount threshold and equal to or less than the second light amount threshold, the process proceeds to step S234. That is, the main control unit 101 determines that the light amount is less than in the high-light-intensity moving object imaging mode, and switches to the medium-light-intensity moving object imaging mode.

[0140] In step S234, main control unit 101 causes first image input unit 123 and second image input unit 224 to simultaneously capture images of the subject with different exposure amounts. At this time, main control unit 101 causes resolution conversion unit 228 to convert the resolution of first image sensor 123b to the first resolution and convert the resolution of second image sensor 224b to the second resolution.

[0141] For example, if the resolution of the first imaging element 123b is set to a low resolution, the main control unit 101 outputs resolution conversion information for converting the resolution of the first imaging element 123b to the resolution conversion unit 228. The resolution conversion unit 228 converts the resolution of the first imaging element 123b from a low resolution to a high resolution based on the input resolution conversion information. Also, if the resolution of the second imaging element 224b is set to a high resolution, the main control unit 101 outputs resolution conversion information for converting the resolution of the second imaging element 224b to the resolution conversion unit 228. The resolution conversion unit 228 converts the resolution of the second imaging element 224b from a high resolution to a low resolution based on the input resolution conversion information.

[0142] When the resolution of the image sensor is set in this way, the first image input unit 123 outputs the exposure amount L A0 The second video input unit 124 captures an image of a subject with a high resolution, for example, as shown in FIGS. 17(A) and 17(B), and generates an electrical signal (A0) for the captured image. A0 Less exposure L b0 Shoot the subject with (L A0 >L b0 17(A) and (B), an electrical signal (b0) is generated for a low-resolution captured image. Note that, since the resolution of the second video input unit 224 is smaller than that of the first video input unit 123, it is possible to appropriately set the exposure amount to expand the dynamic range of the HDR blended image.

[0143] The first image sensor 123b has an exposure amount L A0 The second image sensor 124b outputs the electric signal (A0) to the first image processor 122a. b0 The electric signal (b0) is output to the second image processing unit 122b, and the process then proceeds to the HDR composite image generation step S40.

[0144] In the HDR composite image generating step S40, first, step S235 is performed. In step S235, for example, the same processing as step S124 in the first embodiment is performed. That is, in step S235, the resolution of the electrical signal generated in step S234 is converted. Specifically, the first image processing unit 122a performs resolution conversion on the high-resolution electrical signal (A0) to generate, for example, a low-resolution electrical signal (A0d) as shown in FIG. 17(B). The second image processing unit 122b performs resolution conversion on the low-resolution electrical signal (b0) to generate, for example, a high-resolution electrical signal (b0u) as shown in FIG. 17(A). Then, the process proceeds to step S236.

[0145] In step S236, the image synthesis unit 122c generates a high-resolution HDR synthesized image (A0+b0u) as shown in FIG. 17(A), for example, or a low-resolution HDR synthesized image (A0d+b0) as shown in FIG. 17(B), for example, based on the multiple electrical signals (A0, b0, A0d, b0u) generated in step S235. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR synthesized image. The first image processing unit 122a and the second image processing unit 122b generate captured images and moving images before synthesis, and thumbnail images corresponding to these images. Then, the process proceeds to step S237.

[0146] In step S237, the HDR composite image generated in step S236, the captured images before composite, the moving images, the thumbnail images, etc. are stored in the storage unit 110. Then, the process proceeds to step S204, which will be described later.

[0147] <Low-light moving subject imaging mode> 18 is a diagram schematically illustrating HDR blending processing according to Embodiment 2 of the present invention. If the main control unit 101 determines that the light amount is less than the first light amount threshold, the process proceeds to step S244. That is, the main control unit 101 determines that the light amount is lower than in the medium-light amount imaging mode, and switches to the low-light amount moving object imaging mode.

[0148] In step S224, main control unit 101 causes first image input unit 123 and second image input unit 224 to simultaneously capture images of the subject with different exposure amounts. At this time, main control unit 101 causes resolution conversion unit 228 to convert the resolutions of first image pickup element 123b and second image pickup element 224b to a second resolution which is a lower resolution.

[0149] For example, if the resolution of the first imaging element 123b or the second imaging element 224b is set to high resolution, the main control unit 101 outputs resolution conversion information for converting the resolution of the corresponding imaging element to the resolution conversion unit 228. The resolution conversion unit 228 converts the resolution of the corresponding imaging element from high resolution to low resolution based on the input resolution conversion information.

[0150] When the resolution of the image sensor is set in this way, the first image input unit 123 outputs the exposure amount L a0 The second video input unit 124 captures an image of a subject with a low resolution and generates an electrical signal (a0) for the captured image with a low resolution, for example, as shown in FIG. a0 Less exposure L b0 and generates an electrical signal (b0) for the low-resolution captured image, for example, as shown in Fig. 18. Note that the resolutions of the first video input unit 123 and the second video input unit 224 are set to low resolutions, so that the exposure amount can be appropriately set to expand the dynamic range of the HDR blended image.

[0151] The first image sensor 123b has an exposure amount L a0 The second image sensor 124b outputs the electric signal (a0) to the first image processor 122a. b0 The electric signal (b0) is output to the second image processing unit 122b, and the process then proceeds to the HDR composite image generation step S40.

[0152] In the HDR composite image generating step S40, step S225 is performed. In step S225, the image synthesis unit 122c generates a low-resolution HDR composite image (a0+b0) as shown in FIG. 18, for example, based on the multiple image signals (a0, b0) generated in step S224. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate captured images and moving images before synthesis, and thumbnail images corresponding to these images. Then, the process proceeds to step S237.

[0153] In step S237, the HDR composite image generated in step S225, the captured images before composition, the moving images, the thumbnail images, and the like are stored in the storage unit 110.

[0154] Alternatively, the first image processing unit 122a and the second image processing unit 122b may convert the low-resolution image signals into high-resolution image signals, and the image synthesis unit 122c may generate a high-resolution HDR synthesized image. Then, the process proceeds to step S204.

[0155] In step S204, the same process as in step S104 in the first embodiment is carried out, and an image or the like to be displayed on the display unit 121 is selected. Then, the process proceeds to step S205.

[0156] In step S205, the same process as in step S105 in the first embodiment is carried out, and the image selected by the user in step S204 is displayed on the display unit 121.

[0157] According to this embodiment, a light quantity sensor 165 for measuring the quantity of light is provided, and the main control unit 101 causes the image input unit 220 to capture an image of the subject multiple times with different exposure amounts based on the motion vector and the quantity of light.

[0158] According to this configuration, a more appropriate imaging mode can be selected according to the movement of the subject and the amount of ambient light, and a high-quality HDR composite image suited to the imaging environment can be generated.

[0159] Furthermore, according to this embodiment, the main control unit 101 compares the motion vector with a third motion threshold, and if it determines that the motion information is less than the third motion threshold, switches to a still image capturing mode, causes the first video input unit 123 to continuously capture images of the subject with different exposure amounts, and causes the video signal processing unit 122 to generate an HDR composite image with a first resolution.

[0160] With this configuration, since the subject hardly moves, noise caused by subject movement can be suppressed even when the subject is imaged with a time lag. This allows a high-quality HDR composite image adapted to the imaging environment to be generated using only the first video input unit 123. This also makes it possible to use the second video input unit 224 for video capture, etc., providing an easy-to-use imaging device 200.

[0161] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold and the light intensity exceeds the second light intensity threshold, it switches to a high-light moving object imaging mode, causes the first video input unit 123 and the second video input unit 224 to simultaneously capture images of the subject with different exposure amounts, and causes the video signal processing unit 122 to generate a high-resolution HDR composite image.

[0162] According to this configuration, it is possible to suppress the occurrence of noise due to the movement of the subject, and therefore a high-quality, high-resolution HDR composite image (A0+B0) is generated in which the movement of the subject is large.

[0163] Furthermore, according to the present embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold and that the light intensity is equal to or greater than the first light intensity threshold and equal to or less than the second light intensity threshold, the main control unit 101 switches to a medium-light intensity moving object imaging mode, causes the resolution conversion unit 228 to convert the resolution of the second image sensor 224 to the second resolution, and causes the first video input unit 123 and the second video input unit 224 to simultaneously capture images of the subject at different exposure levels. Then, the main control unit 101 converts the resolution of the image signal generated by the first video input unit 123 to the second resolution, converts the resolution of the image signal generated by the second video input unit 224 to the first resolution, and causes the video signal processing unit 122 to generate an HDR composite image at the first resolution and an HDR composite image at the second resolution.

[0164] With this configuration, the resolution of the second image sensor 224b is converted to a lower resolution, which increases the area of ​​the pixels after the resolution conversion, thereby improving the sensitivity of each pixel. This shortens the imaging time (exposure time) by the second video input unit 224, so even in low-light conditions, noise caused by subject movement is suppressed, and a high-quality HDR composite image (A0+b0u, A0d+b0) suited to the imaging environment is generated.

[0165] Furthermore, with this configuration, a plurality of HDR composite images with different resolutions are generated, so that an HDR image suited to the user's purpose is provided, and an imaging device 200 that is easy to use is provided.

[0166] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is equal to or greater than the third motion threshold and the light intensity is less than the first light intensity threshold, it switches to a low-light moving object imaging mode, causes the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b to the second resolution, causes the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure amounts, and causes the video signal processing unit 122 to generate an HDR composite image at the second resolution.

[0167] With this configuration, in first image sensor 123b and second image sensor 224b, the resolution is converted to a lower resolution, which increases the area of ​​the pixels after resolution conversion, thereby improving the sensitivity of each pixel. This shortens the imaging time of both first video input unit 123 and second video input unit 224, so even in situations with even less light, noise caused by subject movement is suppressed, and a high-quality HDR composite image (a0+b0) suited to the imaging environment is generated.

[0168] Furthermore, according to this embodiment, the main control unit 101 causes the resolution conversion unit 228 to convert the resolution of the first imaging element 123b and the second imaging element 224b from high resolution to low resolution by grouping multiple pixels, and causes the resolution conversion unit 228 to ungroup multiple pixels by converting the resolution of the first imaging element 123b and the second imaging element 224b from low resolution to high resolution.

[0169] This configuration eliminates the need to prepare an image sensor for each resolution, thereby reducing the size of the image pickup device and suppressing increases in the manufacturing costs of the image pickup device.

[0170] (Other embodiments) In the first and second embodiments described above, the main control unit 101 detects motion information (e.g., a motion vector) of a subject based on an image signal generated by the first image sensor 123b or the second image sensor 124b (224b). However, the motion information of the subject may also be detected based on, for example, the focal length of the video input unit 120. Specifically, when the focal length of the video input unit 120 is short, the effect of camera shake is small, and when the focal length is long, the effect of camera shake is large. That is, when the focal length is short, the main control unit 101 detects motion information indicating that the subject's motion is small, and when the focal length is long, the main control unit 101 detects motion information indicating that the subject's motion is large. Therefore, the main control unit 101 compares the focal length of the video input unit 120 with a first focal length threshold and a second focal length threshold that is greater than the first focal length threshold. If the main control unit 101 determines that the focal length of the video input unit 120 is less than the first focal length threshold, the main control unit 101 switches to a still image capture mode. Furthermore, when main control unit 101 determines that the focal length of video input unit 120 is equal to or greater than the first focal length threshold and equal to or less than the second focal length, it switches to micromotion imaging mode. Furthermore, when main control unit 101 determines that the focal length of video input unit 120 exceeds the second focal length threshold, it switches to moving object imaging mode. In this way, even when the imaging mode is switched based on the focal length of video input unit 120, the above-mentioned effects can be obtained.

[0171] Furthermore, in the first and second embodiments, we have described a case where an object is captured using two cameras, the first video input unit 123 and the second video input unit 124 (224), and an HDR composite image is generated. However, for example, an object may be captured using three or more cameras and an HDR composite image may be generated.

[0172] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the techniques of the present invention are not limited to these embodiments. Furthermore, numerical values ​​and the like appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different values ​​are used.

[0173] The functions of the present invention described above may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or in software by a computer such as a microprocessor unit interpreting and executing a program that realizes each function, or may be realized by a combination of hardware and software.

[0174] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0175] 100...imaging device, 101...main control unit, 104a...basic operation execution unit, 104b...camera function execution unit, 110a...basic operation program storage area, 110b...camera function program storage area, 120...video input unit, 123...first video input unit, 123b...first imaging element, 124...second video input unit, 124b...second imaging element, 165...light intensity sensor, 224...second video input unit, 224b...second imaging element, 228...resolution conversion unit

Claims

[Claim 1] a video input unit that captures an image of a subject and generates an image signal of the subject; a video signal processing unit that generates a captured image of the subject based on the image signal; a control unit that causes the video input unit to generate image signals of the subject multiple times with different exposure amounts based on a focal length, and causes the video signal processing unit to generate an HDR composite image of the subject based on the multiple image signals with different exposure amounts; Equipped with the video input unit includes a first video input unit that generates an image signal with a first resolution, and a second video input unit that generates an image signal with a second resolution different from that of the first video input unit; The control unit a first imaging mode that compares the focal length with a first threshold, and when it is determined that the focal length is less than the first threshold, causes the first video input unit to generate image signals of the subject multiple times with different exposure amounts, causes the video signal processing unit to generate captured images of the first resolution based on the image signals generated by the first video input unit, and generates the HDR composite image using the captured images of the first resolution; a second imaging mode that compares the focal length with a first threshold, and when it is determined that the focal length is equal to or greater than the first threshold, causes the first video input unit and the second video input unit to generate image signals of the subject with different exposure amounts, causes the video signal processing unit to generate a captured image at the first resolution from the image signal at the first resolution generated by the first video input unit, and generates a captured image at the second resolution from the image signal at the second resolution generated by the second video input unit, and further converts the captured image at the second resolution to the first resolution, or converts the captured image at the first resolution to the second resolution, and generates the HDR composite image using the captured image at the first resolution and the captured image converted to the first resolution, or generates the HDR composite image using the captured image converted to the second resolution and the captured image at the second resolution; Equipped with Imaging device.

Images