Imaging device

Abstract

To generate a high dynamic range (HDR) synthetic image with high quality compatible with imaging environment.SOLUTION: An imaging apparatus includes: a video input unit for taking an image of an object to generate an image signal of the object; a video signal processing unit for generating a taken image of the object on a basis of the image signal; and a control unit configured to detect motion information of the object on the basis of the image signal, cause the video input unit to take an image of the object on a basis of the motion information multiple times so as to differentiate an exposure amount thereof; and cause the video signal processing unit to generate an HDR synthetic image of the object on the basis of a plurality of image signals whose exposure amounts are different from each other.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] This invention relates to an imaging device, an imaging control method, and an imaging program, and more particularly to an imaging device, an imaging control method, and an imaging program that generate an HDR (High Dynamic Range) composite image by combining images captured by changing the exposure levels of multiple cameras. [Background technology]

[0002] Digital cameras, which convert the image of a subject into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) and record the converted electrical signal in memory, have become remarkably widespread. Furthermore, information terminal devices such as mobile phones, smartphones, and tablet devices equipped with digital cameras have also become widely popular.

[0003] Image sensors used in digital cameras and other devices have a much narrower dynamic range compared to film. Depending on the imaging conditions, this can result in so-called "black clipping" or "highlight clipping," significantly degrading image quality. To overcome this drawback, HDR (High Dynamic Range) synthesis, which captures multiple images with varying exposures and then combines them, is gaining attention. This function extracts dark areas from the image with longer exposure and bright areas from the image with shorter exposure, thereby achieving a wide dynamic range while suppressing high and low highlight clipping.

[0004] For example, Patent Document 1 discloses an image generation device that has two image sensors, one of which captures a high-resolution image with a low exposure time, and the other which captures a low-resolution image with a high exposure time, thereby creating an HDR composite image with reduced noise due to camera shake and the like. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2007-336561 [Overview of the project]

Problems to be Solved by the Invention

[0006] However, the technique described in Patent Document 1 aims to reduce noise such as camera shake by shortening the exposure time when imaging an image with a large exposure amount, but does not consider an HDR synthesis method for obtaining a more preferable image quality according to imaging environments such as the movement of the subject, brightness, and camera shake. Also, an effective usage method of two imaging elements is not mentioned.

[0007] Therefore, an object of the present invention is to provide an imaging device, an imaging control method, and a program for generating a high-quality HDR composite image corresponding to an imaging environment.

Means for Solving the Problems

[0008] Among the inventions disclosed in the present application, an outline of typical ones will be briefly described as follows.

[0009] An imaging device according to a typical embodiment of the present invention includes a video input unit that images a subject and generates an image signal of the subject, a video signal processing unit that generates an imaged image of the subject based on the image signal, and a control unit that detects movement information of the subject based on the image signal and, based on the movement information, causes the video input unit to image the subject a plurality of times with different exposure amounts and causes the video signal processing unit to generate an HDR composite image of the subject based on a plurality of image signals with different exposure amounts.

Effects of the Invention

[0010] Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

[0011] That is, according to a typical embodiment of the present invention, it is possible to provide an imaging device, an imaging control method, and a program for generating a high-quality HDR composite image corresponding to an imaging environment.

Brief Description of the Drawings

[0012] [Figure 1] This is an external view showing an example of an imaging device 100 according to Embodiment 1 of the present invention. [Figure 2] This is a block diagram showing an example of the configuration of an imaging device according to Embodiment 1 of the present invention. [Figure 3] This is a block diagram showing an example of the configuration of an imaging device according to Embodiment 1 of the present invention. [Figure 4] This is a block diagram showing an example of the configuration of the video signal processing unit and video input unit according to Embodiment 1 of the present invention. [Figure 5] This is a flowchart illustrating the imaging method according to Embodiment 1 of the present invention. [Figure 6] This flowchart shows an example of processing in the motion information detection step, imaging step, and captured image generation step according to Embodiment 1 of the present invention. [Figure 7] This flowchart shows an example of processing in the motion information detection step, imaging step, and captured image generation step according to Embodiment 1 of the present invention. [Figure 8] This figure shows a timing chart for imaging according to Embodiment 1 of the present invention. [Figure 9] This figure schematically shows the HDR synthesis process according to Embodiment 1 of the present invention. [Figure 10] This figure schematically shows the HDR synthesis process according to Embodiment 1 of the present invention. [Figure 11] This figure schematically shows the HDR synthesis process according to Embodiment 1 of the present invention. [Figure 12] This is a block diagram showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention. [Figure 13] This is a block diagram showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention. [Figure 14] This is a flowchart illustrating the imaging method according to Embodiment 2 of the present invention. [Figure 15] This is a flowchart illustrating the imaging method according to Embodiment 2 of the present invention. [Figure 16]This figure schematically shows the HDR synthesis process according to Embodiment 2 of the present invention. [Figure 17] This figure schematically shows the HDR synthesis process according to Embodiment 2 of the present invention. [Figure 18] This figure schematically shows the HDR synthesis process according to Embodiment 2 of the present invention. [Modes for carrying out the invention]

[0013] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The embodiments described below are merely examples for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is not limited to the embodiments described below. Furthermore, some of the embodiments described later may be combined as appropriate. (Embodiment 1) Figure 1 is a diagram showing an example of the external appearance of an imaging device according to Embodiment 1 of the present invention, where Figure 1(A) is a top view of the imaging device and Figure 1(B) is a rear view of the imaging device 100. Figure 2 is a block diagram showing an example of the configuration of an imaging device according to Embodiment 1 of the present invention. Figure 3 is a diagram showing an example of the software configuration of an imaging device according to Embodiment 1 of the present invention.

[0014] Figure 1 illustrates an imaging device mounted on a smartphone, but the imaging device may also be mounted on other information processing devices such as mobile phones, tablet devices, PDAs (Personal Digital Assistants), and notebook PCs (Personal Computers). Furthermore, the imaging device may consist of a digital still camera.

[0015] As shown in Figures 1 and 2, the imaging device 100 includes a main control unit (control unit) 101, a system bus 102, a memory unit 104, a storage unit 110, a video processing unit 120, an audio processing unit 130, an operation unit 140, a communication processing unit 150, a sensor unit 160, an expansion interface unit 170, a display unit 121, and the like.

[0016] The system bus 102 is a data communication path that connects each component of the imaging device 100. Data input and output are performed between the main control unit 101 and each part of the imaging device 100 via the system bus 102.

[0017] The storage unit 110 is composed of non-volatile memory such as flash ROM (Read Only Memory), SSD (Solid State Drive), or HDD (Hard Disc Drive), and stores information even when power is not supplied to the imaging device 100.

[0018] The storage unit 110 includes, for example, as shown in Figure 3, a basic operation program storage area 110a for storing a basic operation program that executes the basic operations of the imaging device 100, a camera function program storage area 110b for storing a camera function program that realizes the camera function, a voice recognition program storage area 110c for storing a voice recognition program that realizes the voice recognition function, a timing notification program storage area 110d for storing a timing notification program that realizes the timing notification function, an other program storage area 110e for storing other programs, and a various information / data storage area 110f for storing various information such as the operation settings of the imaging device 100 and user information.

[0019] Furthermore, the storage unit 110 stores captured images, moving images, HDR composite images (described later), and their thumbnail images, etc., captured by the imaging device 100, for example, in the various information / data storage area 110f. The storage unit 110 also stores new application programs downloaded from an application server on the internet, for example, in the other program storage area 110e.

[0020] Furthermore, the storage unit 110 stores a first motion threshold, which serves as a criterion for determining the movement of the subject (described later), and a second motion threshold that is greater than the first motion threshold. These motion thresholds are set as appropriate, for example, through experiments.

[0021] In the memory unit 104, various programs stored in each part of the storage unit 110 are unpacked. When various programs are executed by the main control unit 101, various execution units that realize the functions of each program are configured in the memory unit 104. For example, when the basic operation program stored in the basic operation program storage area 110a is unpacked in the memory unit 104, the memory unit 104 is configured with a basic operation execution unit 104a that realizes the basic operation of the imaging device 100. Similarly, when the camera function program stored in the camera function program storage area 110b is unpacked in the memory unit 104, the memory unit 104 is configured with a camera function execution unit 104b that realizes the camera function. Furthermore, when the voice recognition function program stored in the voice recognition function program storage area 110c is unpacked in the memory unit 104, the memory unit 104 is configured with a voice recognition function execution unit 104c that realizes the voice recognition function. Furthermore, when the timing notification function program stored in the timing notification function program storage area 110d is loaded into the memory unit 104, the memory unit 104 is configured with a timing notification function execution unit 104d that realizes the timing notification function. The memory unit 104 also has a temporary storage area 104e, etc., where data is temporarily stored as needed.

[0022] Furthermore, the basic operation execution unit 104a, camera function execution unit 104b, voice recognition function execution unit 104c, timing notification function execution unit 104d, and other units may be composed of hardware having the same functions as these units. Also, the memory unit 104 may be configured as an integral part of the main control unit 101.

[0023] As shown in Figure 1(A), the display unit 121 is provided on the surface 100a of the imaging device 100, which is equipped with a third video input unit 125. The display unit 121 is a display device such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, and displays HDR composite images, uncomposited captured images, moving images, thumbnail images, etc., which are processed by the video signal processing unit 122 and described later.

[0024] The audio processing unit 130 consists of an audio output unit 131, an audio signal processing unit 132, and an audio input unit 133. The audio output unit 131 is a speaker and is located on the periphery of the display unit 121 on the surface 100a of the imaging device 100, as shown in Figure 1(A). The audio output unit 131 emits sound based on the audio signal processed by the audio signal processing unit 132. The audio input unit 133 is a microphone and is located on the side of the imaging device 100, on the side opposite to the audio output unit 131 relative to the display unit 121, as shown in Figures 1(A) and (B). The audio input unit 133 receives audio input from the user of the imaging device 100 and converts the input audio into an audio signal. The audio input unit 133 outputs the input audio signal to the audio signal processing unit 132. The audio input unit 133 may be configured separately from the imaging device 100, in which case the audio input unit 133 and the imaging device 100 may be connected by wired or wireless communication.

[0025] The operation unit 140 is an instruction input unit that inputs operation instructions to the imaging device 100. In this embodiment, for example, as shown in Figures 1(A) and (B), it consists of a touch panel 140a arranged on top of the display unit 121, operation keys 140b provided on the side of the imaging device 100, and operation keys 140c provided on the surface 100a of the imaging device 100 near the display unit 121. However, it is not necessary to have all of these, and it is sufficient to have any one of them. Also, the touch panel 140a may be integrated with the display unit 121. Furthermore, the operation unit 140 may consist of a keyboard or the like (not shown) connected to the expansion interface unit 170, which will be described later. Also, the operation unit 140 may consist of a separate information terminal device connected via wired communication or wireless communication.

[0026] The communication processing unit 150 is composed of, for example, a LAN (Local Area Network) communication unit 151, a mobile telephone network communication unit 152, and a proximity wireless communication unit 153. The LAN communication unit 151 transmits and receives data via wireless communication, for example, via an access point for wireless communication on the Internet. The mobile telephone network communication unit 152 is connected to a base station of the mobile telephone network and performs telephone communication (calls) and data transmission and reception via wireless communication through the base station. The proximity wireless communication unit 153 transmits and receives data with a reader / writer that supports proximity wireless communication. The LAN communication unit 151, the mobile telephone network communication unit 152, and the proximity wireless communication unit 153 are equipped with various devices such as coding circuits, decoding circuits, and antennas, which are not shown in the diagram. The communication processing unit 150 may also be equipped with an infrared communication unit for performing infrared communication.

[0027] The sensor unit 160 is a group of sensors that detect the state of the imaging device 100. In this embodiment, the sensor unit 160 consists of, for example, a GPS (Global Positioning System) receiver 161, an acceleration sensor 162, a gyro sensor 163, a geomagnetic sensor 164, a light intensity sensor 165, and a proximity sensor 166. The sensor unit 160 may also include other sensors.

[0028] The GPS receiver 161 receives GPS signals transmitted from multiple satellites using GPS. The GPS signals received by the GPS receiver 161 are output to, for example, the main control unit 101, where the main control unit 101 detects the position of the imaging device 100 based on the GPS signals.

[0029] The acceleration sensor 162 measures the magnitude and direction of the acceleration (e.g., gravitational acceleration) acting on the imaging device 100. The magnitude and direction of the acceleration measured by the acceleration sensor unit 133 are output as acceleration information to the main control unit 101, and the main control unit 101 detects the acceleration acting on the imaging device 100 based on the acceleration information.

[0030] The gyro sensor 163 measures the angular velocity of the imaging device 100, which is generated when the user moves the imaging device 100, for example. The angular velocity measured by the gyro sensor 163 is output to the main control unit 101 as angular velocity information. The main control unit 101 detects the angular velocity of the imaging device 100 based on the angular velocity information.

[0031] The geomagnetic sensor 164 measures the magnitude and direction of the geomagnetic field acting on the imaging device 100. The measured values ​​of the magnitude and direction of the geomagnetic field measured by the geomagnetic sensor 164 are output to the main control unit 101 as geomagnetic information. The main control unit 101 detects the geomagnetic field acting on the imaging device 100 based on the geomagnetic information.

[0032] The light intensity sensor 165 measures the brightness around the imaging device 100. For example, when imaging a subject, the light intensity sensor 165 measures the amount of light around the subject. The light intensity measurement values ​​from the light intensity sensor 165 are output to the main control unit 101 as light intensity information. The main control unit 101 detects the amount of light around the imaging device 100 based on the light intensity information.

[0033] The proximity sensor 166 measures the proximity of the imaging device 100 to objects in its vicinity. For example, the proximity sensor 166 measures the distance and direction of objects in the vicinity of the imaging device 100. The proximity measurement values ​​detected by the proximity sensor 166 are output to the main control unit 101 as proximity information. The main control unit 101 detects the proximity of objects in the vicinity of the imaging device 100 based on the proximity information.

[0034] The expansion interface unit 170 is a group of interfaces for extending the functionality of the imaging device 100. The expansion interface unit 170 includes, for example, a video / audio interface 171, a USB (Universal Serial Bus) interface 172, and a memory interface 173.

[0035] The video / audio interface 171 is connected to an external video / audio output device and accepts video / audio signals output from the video / audio output device. The video / audio interface 171 also outputs video / audio signals to a video / audio input device. The USB interface 172 is connected to a USB device such as a keyboard and performs information input and output with the USB device. The memory interface 173 is connected to a memory medium such as a memory card and performs data input and output with the memory medium.

[0036] Next, the video input unit 120 and the video signal processing unit 122 will be described. Figure 4 is a block diagram showing an example of the configuration of the video input unit and the video signal processing unit according to Embodiment 1 of the present invention. Note that the third video input unit 125 is omitted in Figure 4.

[0037] The video input unit 120 includes, for example, a first video input unit 123, a second video input unit 124, a third video input unit 125, and an exposure control unit 126, as shown in Figures 2 and 4. The first video input unit 123 and the second video input unit 124 are arranged side by side on the back surface 100b of the imaging device 100, as shown in Figure 1(B). The third video input unit 125 is located near the display unit 121 on the front surface 100a of the imaging device 100, as shown in Figure 1(A).

[0038] In Figure 1(B), the first video input unit 123 and the second video input unit 124 are located on the back surface 100b of the imaging device 100, but they may also be located on the front surface 100a of the imaging device 100. Furthermore, the first video input unit 123 and the second video input unit 124 may be integrated into a single unit.

[0039] The first video input unit 123 includes, for example, a first imaging optical system 123a and a first image sensor 123b, as shown in Figure 4. The second video input unit 124 includes, for example, a second imaging optical system 124a and a second image sensor 124b, as shown in Figure 4.

[0040] The first imaging optical system 123a and the second imaging optical system 124a are composed of, for example, multiple lenses that collect incident light from the subject, an aperture (iris diaphragm, etc.), a mechanical or electronic shutter, etc. The first image sensor 123b and the second image sensor 124b are composed of, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0041] The first video input unit 123 and the second video input unit 124 capture an image of a subject and generate an image signal of the subject. Specifically, in each pixel of the first image sensor 123b, the light from the first imaging optical system 123a is stored as an electric charge, thereby converting the input light into an electrical signal. Similarly, in each pixel of the second image sensor 124b, the light from the second imaging optical system 124a is stored as an electric charge, thereby converting the input light into an electrical signal. In this way, the first image sensor 123 and the second image sensor 124 generate electrical signals as image signals.

[0042] In this embodiment, the first image sensor 123b is an image sensor with a first resolution (high resolution), and the second image sensor 124b is an image sensor with a second resolution (low resolution) that is lower than the first image sensor 123b.

[0043] The exposure control unit 126 determines the aperture and shutter speed of the first imaging optical system 123a and the second imaging optical system 124a based on instructions from the main control unit 101, and controls the amount of exposure input to the first image sensor 123b and the second image sensor 124b.

[0044] The video signal processing unit 122 generates an image of the subject based on the image signal. The video signal processing unit 122 includes, for example, a first image processing unit 122a, a second image processing unit 122b, an image synthesis unit 122c, and an image output unit 124d, as shown in Figure 4.

[0045] The first image processing unit 122a is connected to the first image sensor 123b and converts the electrical signal (image signal) output from the first image sensor 123b into digital image data with a predetermined bit depth. The first image processing unit 122a then performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The first image processing unit 122a outputs the generated intermediate image data to the image synthesis unit 122c. The first image processing unit 122a also outputs the image captured before synthesis to the storage unit 110.

[0046] The second image processing unit 122b is connected to the second image sensor 124b and converts the electrical signal (image signal) output from the second image sensor 124b into digital image data with a predetermined bit depth. The second image processing unit 122b then performs image processing on the digital image data to generate intermediate image data suitable for HDR image synthesis. The second image processing unit 122b outputs the generated intermediate image data to the image synthesis unit 122c. The second image processing unit 122b also outputs the image captured before synthesis to the storage unit 110.

[0047] The image synthesis unit 122c synthesizes the intermediate image data input from the first image processing unit 122a and the second image processing unit 122b to generate an HDR composite image as the captured image. The image synthesis unit 122c outputs the generated HDR composite image to the storage unit 110.

[0048] The image output unit 122d outputs the HDR composite image generated by the image synthesis unit 122c, the captured image before synthesis, the video, and their thumbnail images to the display unit 121 for display.

[0049] The flash unit 129 is provided adjacent to the first video input unit 123 and the second video input unit 124 on the back surface 100b of the imaging device 100, as shown in Figure 1(B). The flash unit 129 illuminates the subject with flash light when the first video input unit 123 and the second video input unit 124 are taking images.

[0050] The main control unit (control unit) 101 is composed of a computer such as a microprocessor unit. The main control unit 101 realizes the functions of each program and operates each component of the imaging device 100 by executing each execution unit 100a to 100d configured in the memory unit 104.

[0051] The main control unit 101 detects motion information of the subject based on the image signal generated by the video input unit 120. Based on the motion information, the main control unit 101 instructs the video input unit 120 to image the subject multiple times with different exposure levels. Specifically, the main control unit 101 compares the motion information with a first motion threshold and a second motion threshold, and based on the result, switches to the respective imaging modes described later to image the subject. The main control unit 101 also generates exposure information such as exposure level, shutter speed, and aperture for the first video input unit 123 and the second video input unit 124 based on the image signal or the captured image. The main control unit 101 outputs this information to the video input unit 120. The main control unit 101 also instructs the video signal processing unit 122 to generate an HDR composite image of the subject based on multiple image signals with different exposure levels.

[0052] Next, the imaging method according to this embodiment will be described. In this embodiment, for example, various programs such as the camera function program 110b shown in Figure 3 are loaded into the memory unit 104, and the main control unit 101 executes each execution unit such as the camera function execution unit 104b, thereby performing operations related to imaging.

[0053] Figure 5 is a flowchart illustrating the imaging method according to Embodiment 1 of the present invention. As shown in Figure 5, when imaging is performed, the following steps are carried out: video input step S10, motion information detection step S20, imaging step S30, and HDR composite image generation step S40.

[0054] When an imaging command is issued from the operation unit 140, the operation related to imaging the subject is started. In the video input step S10, the video input unit 120 images the subject and generates an image signal of the subject. The first video input unit 123 and the second video input unit 124 image the subject and generate electrical signals as image signals in the first image sensor 123b and the second image sensor 124b. The first image sensor 123b outputs the generated electrical signal to the first image processing unit 122a of the video signal processing unit 122, and the second image sensor 124b outputs the generated electrical signal to the second image processing unit 122b of the video signal processing unit 122. Here, the case in which both the first video input unit 123 and the second video input unit 124 image the subject is described, but it is also possible for only one of the video input units to image the subject. Then, the process moves to the motion information detection step S20.

[0055] Figures 6 and 7 are flowcharts showing an example of processing in the motion information detection step, imaging step, and image generation step according to Embodiment 1 of the present invention. In the motion information detection step S20, the main control unit 101 detects motion information of the subject based on the image signal. Specifically, the first image processing unit 122a of the video signal processing unit 122 outputs the electrical signal output from the first image sensor 123b to the main control unit 101. The second image processing unit 122b of the video signal processing unit 122 outputs the electrical signal output from the second image sensor 124b to the main control unit 101. The first image processing unit 122a and the second image processing unit 122b may generate an image based on the electrical signal and output the generated image to the main control unit 101.

[0056] The main control unit 101 detects motion information of the subject (e.g., motion vector) based on these input electrical signals (image signals). The main control unit 101 detects the motion vector of the subject by using a well-known block matching method or the like for these image signals. Alternatively, the main control unit 101 may detect motion information of the subject based on the input captured image. Then, the process proceeds to the imaging step S30.

[0057] In the imaging step S30, the video input unit 120 captures images of the subject multiple times with different exposure amounts based on the motion information detected in the motion information detection step S20. In the imaging step S30, step S102 is performed first. In step S102, the movement of the subject is determined based on the motion information. Specifically, the main control unit 101 determines the movement of the subject by comparing the motion vector detected in the motion information detection step S20 with a first motion threshold and a second motion threshold. More specifically, the main control unit 101 reads the first motion threshold and the second motion threshold stored in the storage unit 110 into the memory 104 and compares the motion vector with the first motion threshold and the second motion threshold.

[0058] [Still image capture mode] In step S102, if the main control unit 101 determines that the motion vector is less than the first motion threshold, step S113 is performed. That is, the main control unit 101 determines that the subject is hardly moving and switches to still image capture mode.

[0059] Figure 8 is a diagram showing a timing chart for imaging according to Embodiment 1 of the present invention. Figure 9 is a diagram schematically showing the HDR synthesis process according to Embodiment 1 of the present invention. In step S113, the first video input unit 123 continuously images the subject with different exposure amounts, for example, as shown in Figure 8(A). Specifically, the main control unit 101 determines the exposure amount (e.g., shutter speed, aperture value, etc.) for each image based on the electrical signal (image signal) output from the first image sensor 123b or the second image sensor 124b. The main control unit 101 determines the exposure amount for the first image, for example, as shown in Figure 8(A), to an exposure amount L such that the dark parts on the low-light side do not become completely black. A0 Set the exposure amount for the second image to L, for example, so that the bright areas on the high-light side do not become overexposed. A0 Lower exposure L A1 Set to (L A0 >L A1)。The main control unit 101 outputs information regarding the determined exposure amount as exposure information to the video input unit 120. In the video input unit 120, based on the input exposure information, the first video input unit 123 captures the subject with an exposure amount L A0 for the first time and captures the subject with an exposure amount L A1 for the second time.

[0060] Each time the first image sensor 123b captures the subject, it generates an electrical signal regarding a captured image as shown in, for example, FIG. 9. Each time the first image sensor 123b captures the subject, it generates an electrical signal (A0) with an exposure amount L A0 and an electrical signal (A1) with an exposure amount L A1 and outputs the generated electrical signals (A0, A1) to the first image processing unit 122a of the video signal processing unit 122. Here, the case of capturing the subject twice with different exposure amounts is taken as an example, but the subject may be captured three or more times, for example.

[0061] On the other hand, the second video input unit 124 captures, for example, a moving image (Bm) of the subject. Also, the second video input unit 124 may capture the subject with different depths of the field of view by, for example, changing the aperture value. The second image sensor 124b outputs an image signal regarding a moving image or the like to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to the HDR composite image generation step S40.

[0062] In the HDR composite image generation step S40, step S114 is performed. In step S114, an HDR composite image of a first resolution is generated based on multiple image signals (A0, A1) with different exposure levels generated in step S113. Specifically, the first image processing unit 122a of the video signal processing unit 122 converts each of the electrical signals (A0, A1) input from the first image sensor 123b into digital image data with a predetermined bit depth. The first image processing unit 122a then performs image processing on each digital image data to generate intermediate image data suitable for HDR image synthesis. The first image processing unit 122a then outputs each intermediate image data to the image synthesis unit 122c. The first image processing unit 122a also generates each of the pre-synthesis captured images based on the electrical signals (A0, A1). The first image processing unit 122a also generates thumbnail images corresponding to each of the pre-synthesis captured images.

[0063] The second image processing unit 122b generates a moving image or an image with a different depth of field based on the image signal input from the second video input unit 124. The second image processing unit 122b also generates thumbnail images corresponding to the generated moving image and image.

[0064] The image synthesis unit 122c generates a first resolution (high resolution) HDR composite image (captured image) (A0+A1), for example, as shown in Figure 9, by weighted synthesis of the intermediate image data input from the first image processing unit 122a. The HDR composite image (A0+A1) thus generated eliminates black crushing in dark areas and overexposure in bright areas, resulting in an image that closely resembles what the user sees with their own eyes. The image synthesis unit 122c also generates a thumbnail image corresponding to the generated HDR composite image.

[0065] The first image processing unit 122a and the second image processing unit 122b may convert the high-resolution image signal into a low-resolution image signal, and the image synthesis unit 122c may generate a low-resolution HDR composite image. Then, the process proceeds to step S115.

[0066] In step S115, the HDR composite image generated in step S114, the captured image before synthesis, the video, the thumbnail image, etc., are stored in the storage unit 110. Specifically, the image synthesis unit 122c outputs the generated HDR composite image (A0+A1) and thumbnail image to the storage unit 110, and the storage unit 110 stores the input HDR image composite (A0+A1) and thumbnail image. In addition, the first image processing unit 122a and the second image processing unit 122b output the captured image before synthesis, the video, and the thumbnail image to the storage unit 110, and the storage unit 110 stores the input captured image before synthesis, the video, and the thumbnail image. Once these processes are performed, the HDR composite image generation step S40 is completed, and the process moves to step S104. Step S104 will be described later.

[0067] [Micro-motion imaging mode] Figures 10 and 11 schematically show the HDR synthesis process according to Embodiment 1 of the present invention. In step S102 of the imaging step S30, if the main control unit 101 determines that the motion vector is greater than or equal to the first motion threshold, step S123 or step S133 is performed. More specifically, if the main control unit 101 determines that the motion vector is greater than or equal to the first motion threshold and less than or equal to the second motion threshold, step S123 is performed. In other words, the main control unit 101 determines that the subject is moving more than in the still image imaging mode, but less than in the motion imaging mode described later, and switches to the micro-motion imaging mode.

[0068] In step S123, the main control unit 101 instructs the first video input unit 123 and the second video input unit 124 to simultaneously image the subject with different exposure levels, and then instructs the second video input unit 124 to image the subject with different exposure levels. Specifically, as shown in Figure 8(B), for example, the main control unit 101 sets the exposure level for imaging with the first video input unit 123 to an exposure level L such that dark subjects on the low-light side do not become completely black. A0The first video input unit 123 captures the subject with appropriate exposure. Here, appropriate exposure means capturing a dark subject on the low-light side with sufficient exposure. The main control unit 101 sets the exposure amount for the first capture by the second video input unit 124 to an exposure amount L between, for example, an exposure amount that causes the low-light side to be underexposed and an exposure amount that causes the high-light side to be overexposed. B0 Set to L in the second image capture by the second video input unit 124, for example, so that bright areas do not become overexposed. B0 Lower exposure L A1 Set to (L A0 >L B0 >L B1 ).

[0069] Furthermore, since the resolution of the second image sensor 124b is lower than that of the first image sensor 123b, the area per pixel is larger and the exposure increases. As a result, the second video input unit 124 can obtain the same exposure even with a shorter imaging time (faster shutter speed) than the first video input unit 123. Therefore, the imaging interval (time difference) in the second video input unit 124 can be shorter than that of the first video input unit 123, which helps to suppress the generation of noise caused by slight movements of the subject when performing HDR synthesis processing.

[0070] The main control unit 101 outputs information regarding the determined exposure amount as exposure information to the video input unit 120. Based on the input exposure information, the first video input unit 123 in the video input unit 120 determines the exposure amount L A0 The subject is imaged, and the second video input unit 124 first sets the exposure amount L B0 The subject is imaged, and the second time exposure amount L B1 The subject is captured using this method.

[0071] The first image sensor 123b has an exposure amount L as shown in Figure 10(A), for example. A0 An electrical signal (A0) is generated, and the generated electrical signal (A0) is output to the first image processing unit 122a of the video signal processing unit 122.

[0072] The second image sensor 124b, each time it captures an image of a subject, adjusts the exposure amount L as shown in Figure 10(A), for example. B0 Electrical signal (B0), exposure amount L B1 The electrical signal (B1) is generated, and the generated electrical signals (B0, B1) are output to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to the HDR composite image generation step S40.

[0073] In the HDR composite image generation step S40, step S124 is performed first. In step S124, the resolution of the captured image of the electrical signal generated in step S123 is converted. Specifically, the main control unit 101 instructs the video signal processing unit 122 to convert the resolution of the image signal generated by the first video input unit 123 to the second resolution, and to convert the resolution of the image signal generated by the second video input unit 124 to the first resolution.

[0074] More specifically, the main control unit 101 instructs the first image processing unit 122a to convert the resolution of the electrical signal (A0) generated by the first image sensor 123b from the first resolution to the second resolution, and instructs the second image processing unit 122b to convert the resolution of the electrical signal (B0) generated by the second image sensor 124b from the second resolution to the first resolution. More specifically, the first image processing unit 122a performs a resolution conversion on the electrical signal (A0) with the first resolution to generate the electrical signal (A0d) with the second resolution shown in Figure 10(B). The second image processing unit 122b performs a resolution conversion on the electrical signal (B0) with the second resolution and, by performing processing such as interpolation, generates the electrical signal (B0u) with the first resolution shown in Figure 10(A). Then, the process proceeds to step S125.

[0075] In step S125, an HDR composite image of a first resolution and an HDR composite image of a second resolution are generated based on the multiple image signals (A0, B0, B1, A0d, B0u) generated in step S124. Specifically, the first image processing unit 122a converts the electrical signal (A0) of the first resolution and the electrical signal (A0d) of the second resolution into digital image data with a predetermined bit depth. The first image processing unit 122a then performs image processing on each digital image data to generate intermediate image data suitable for HDR image synthesis. The first image processing unit 122a then outputs each intermediate image data to the image synthesis unit 122c. The first image processing unit 122a also generates each of the captured images before synthesis based on the electrical signals (A0, A0d). The first image processing unit 122a also generates thumbnail images corresponding to each of the captured images before synthesis.

[0076] The second image processing unit 122b converts the electrical signals of the second resolution (B0, B1) and the electrical signal of the first resolution (B0u) into digital image data with a predetermined bit depth. The second image processing unit 122b then performs image processing on each digital image data to generate intermediate image data suitable for HDR image synthesis. The second image processing unit 122b then outputs each intermediate image data to the image synthesis unit 122c. The second image processing unit 122b also generates each pre-synthesis image based on the electrical signals (B0, B1, B0u). The second image processing unit 122b also generates thumbnail images corresponding to each of the generated pre-synthesis images.

[0077] The image synthesis unit 122c generates an HDR composite image of a first resolution and an HDR composite image of a second resolution by weighted synthesis of the intermediate image data output from the first image processing unit 122a and the intermediate image data output from the second image processing unit 122b.

[0078] Specifically, the image synthesis unit 122c generates an HDR composite image (A0+B0u) of a first resolution, for example, as shown in Figure 10(A), based on the intermediate image data of electrical signal (A0) and the intermediate image data of electrical signal (B0u). The image synthesis unit 122c also generates an HDR composite image (A0d+B0) of a second resolution, for example, as shown in Figure 10(B), based on the intermediate image data of electrical signal (A0d) and the intermediate image data of electrical signal (B0). The image synthesis unit 122c also generates an HDR composite image (B0+B1) of a second resolution, for example, as shown in Figure 11(A), based on the intermediate image data of electrical signal (B0) and the intermediate image data of electrical signal (B1). Furthermore, the image synthesis unit 122c generates a second-resolution HDR composite image (A0d+B0+B1), for example, as shown in Figure 11(B), based on the intermediate image data of the electrical signal (A0d) and the intermediate image data of the electrical signals (B0, B1). The image synthesis unit 122c also generates a thumbnail image corresponding to the generated HDR composite image.

[0079] Furthermore, when the user later views the HDR composite images, they may select the image they deem best by visual inspection and treat it as the final processed image after HDR compositing. Then, the process proceeds to step S126.

[0080] In step S126, the HDR composite image generated in step S125, the captured image before synthesis, and thumbnail images are stored in the storage unit 110. Specifically, the image synthesis unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), (B0+B1), (A0d+B0+B1), and thumbnail images to the storage unit 110, and the storage unit 110 stores the input HDR image composites (A0+B0u), (A0d+B0), (B0+B1), (A0d+B0+B1), and thumbnail images. In addition, the first image processing unit 122a and the second image processing unit 122b output the captured image before synthesis and thumbnail images to the storage unit 110, and the storage unit 110 stores the input captured image before synthesis and thumbnail images. Once these processes are performed, the HDR composite image generation step S40 is completed, and the process moves to step S104. Step S104 will be discussed later.

[0081] [Motion imaging mode] In step S102 of imaging step S30, if the main control unit 101 determines that the motion vector exceeds the second motion threshold, step S133 is performed. That is, the main control unit 101 determines that the subject is moving more than in the micro-motion imaging mode and switches to the motion imaging mode.

[0082] In step S133, the same process as in step S123 is performed. That is, in step S123, the main control unit 101 instructs the first video input unit 123 and the second video input unit 124 to simultaneously image the subject with different exposure amounts. The first video input unit 123 sets the exposure amount L A0 The camera captures an image of the subject and generates an electrical signal (A0) for the captured image of a first resolution, such as shown in Figures 10(A) and (B). The second video input unit 124 controls the exposure amount L B0The system captures the subject and generates an electrical signal (B0) for the captured image of a second resolution, such as shown in Figures 10(A) and (B). In motion imaging mode, unlike in micro-motion imaging mode, the second video input unit 124 does not capture the subject with a time delay. This is because, due to the large movement of the subject, capturing the image with a time delay would result in noticeable subject blur in the generated HDR composite image.

[0083] The first image sensor 123b has an exposure amount L A0 The electrical signal (A0) is output to the first image processing unit 122a. The second image sensor 124b controls the exposure amount L B0 The electrical signal (B0) is output to the second image processing unit 122b. Then, the process proceeds to the HDR composite image generation step S40.

[0084] In the HDR composite image generation step S40, step S134 is performed first. In step S134, the same processing as in step S124 described above is performed. That is, in step S134, the first image processing unit 122a performs a resolution conversion on the electrical signal (A0) of the first resolution to generate the electrical signal (A0d) of the second resolution, for example, as shown in Figure 10(B). The second image processing unit 122b performs a resolution conversion on the electrical signal (B0) of the second resolution to generate the electrical signal (B0u) of the first resolution, as shown in Figure 10(A). Then, the process proceeds to step S135.

[0085] In step S135, processing is carried out that is almost the same as in step S125 described above. That is, in step S135, an HDR composite image of a first resolution and an HDR composite image of a second resolution are generated based on the multiple image signals (A0, B0, A0d, B0u) generated in step S134. Specifically, the image synthesis unit 122c generates the HDR composite image of a first resolution (A0+B0u) shown in Figure 10(B) and the HDR composite image of a second resolution (A0d+B0) shown in Figure 10(A). In addition, the first image processing unit 122a and the second image processing unit 122b generate the captured image before synthesis and thumbnail images corresponding to the captured image before synthesis, respectively.

[0086] Furthermore, when the user later views the HDR composite images, they may select the image they deem best by visual inspection and treat it as the final processed image after HDR compositing. Then, the process proceeds to step S126.

[0087] In step S126, the HDR composite image generated in step S135, the image captured before synthesis, and thumbnail images are stored in the storage unit 110. Specifically, the image synthesis unit 122c outputs the generated HDR composite images (A0+B0u), (A0d+B0), and thumbnail images to the storage unit 110, and the storage unit 110 stores the input HDR composite images (A0+B0u), (A0d+B0), and thumbnail images. In addition, the first image processing unit 122a and the second image processing unit 122b output the image captured before synthesis and thumbnail images to the storage unit 110, and the storage unit 110 stores the input image captured before synthesis and thumbnail images. Once these processes are performed, the HDR composite image generation step S40 is completed, and the process moves to step S104.

[0088] In step S104, an image to be displayed on the display unit 121 is selected. Specifically, an image to be displayed on the display unit 121 is selected from the images (HDR composite image, captured image before synthesis, video) generated in each of the imaging mode steps S114, S125, and S135 and stored in the storage unit 110.

[0089] For example, the storage unit 110 outputs information about the stored images (e.g., thumbnail images) to the image output unit 122d. The image output unit 122d outputs the input thumbnail images to the display unit 121. The display unit 121 displays the input thumbnail images. The user selects the image to be displayed on the display unit 121 from the displayed thumbnail images. Then, the process proceeds to step S105.

[0090] In step S105, the image corresponding to the selected thumbnail image is displayed. Specifically, the image output unit 122d outputs, for example, the image corresponding to the thumbnail image selected by the user from the storage unit 110. The image output unit 122d outputs the read image to the display unit 121. The display unit 121 displays the input image.

[0091] The user may arbitrarily select the images to be displayed on the display unit 121, or they may register a display priority in the imaging device 100 in advance, so that the images are displayed sequentially according to this priority.

[0092] Furthermore, the images displayed on the display unit 121 may be enlarged or reduced as appropriate to match the resolution of the display unit 121. In this case, multiple images may be displayed at the same size, or multiple images may be displayed at different sizes to match the resolution of the selected image.

[0093] It should be noted that the imaging device 100 according to this embodiment is not limited to the configuration shown in Figure 2. For example, the imaging device 100 does not have to include, for example, a communication processing unit 150 or a sensor unit 160, and it may also include various functions such as a digital television broadcasting reception function or an electronic money payment function.

[0094] According to this embodiment, based on the motion vector of the subject, the main control unit 101 causes the video input unit 120 to capture images of the subject multiple times with different exposure levels, and causes the video signal processing unit to generate an HDR composite image of the subject based on multiple image signals with different exposure levels.

[0095] This configuration allows for the selection of an appropriate imaging mode according to the subject's movement, enabling the generation of high-quality HDR composite images that are responsive to imaging conditions such as subject movement and camera shake.

[0096] Furthermore, according to this embodiment, if the main control unit 101 determines that the motion vector is less than the first motion threshold, it switches to still image acquisition mode and causes the first video input unit 123, which has a first image sensor 123b with high resolution and a first resolution, to continuously capture images of the subject with different exposure amounts, and causes the video signal processing unit 122 to generate an HDR composite image of the first resolution based on a plurality of image signals (A0, A1) with different exposure amounts.

[0097] With this configuration, since the subject hardly moves in still image capture mode, even if the subject is captured with a time delay, the generation of noise due to the subject's movement in the HDR composite image is suppressed. As a result, the subject can be continuously captured using only the high-resolution image sensor, thus generating a high-quality HDR composite image.

[0098] Furthermore, this configuration combines an electrical signal (A0) generated by imaging a dark subject in low light with sufficient exposure and an electrical signal (A1) generated by imaging a bright subject in high light with reduced exposure, thereby producing a high-quality HDR composite image with an expanded dynamic range.

[0099] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is greater than or equal to a first motion threshold, it switches to micro-motion imaging mode or motion imaging mode, instructs the first video input unit 123 and the second video input unit 124 to simultaneously image the subject with different exposure amounts, and instructs the image signal processing unit 122 to convert the image signal of the first resolution to an image signal of the second resolution, and convert the image signal of the second resolution to an image signal of the first resolution, thereby generating an HDR composite image of the first resolution and an HDR composite image of the second resolution.

[0100] With this configuration, even when the subject is moving, an HDR composite image is generated based on multiple image signals captured simultaneously. This suppresses the effects of noise caused by subject movement and camera shake, resulting in a high-quality HDR composite image with an expanded dynamic range.

[0101] Furthermore, this configuration generates multiple HDR images with different resolutions, providing the user with an HDR image suitable for their specific needs and resulting in a user-friendly imaging device 100.

[0102] Furthermore, according to this embodiment, if the control unit 101 determines that the motion vector is above a first motion threshold and below a second motion threshold, it switches to micro-motion imaging mode and instructs the first video input unit 123 and the second video input unit 124 to simultaneously image the subject with different exposure levels, and then instructs the second video input unit 124 to image the subject with different exposure levels.

[0103] With this configuration, HDR synthesis processing is performed based on at least three different image signals with varying exposure levels, resulting in the generation of higher-quality HDR composite images.

[0104] Furthermore, according to this embodiment, the resolution of the second image sensor 124b is lower than the resolution of the first image sensor 123b.

[0105] With this configuration, the second video input unit 124 can obtain the same exposure amount even with a shorter imaging time than the first video input unit 123. As a result, even if the second video input unit 124 images the subject with a time delay, the generation of noise due to the subject's movement is suppressed, thus reducing the generation of noise due to the subject's movement in the HDR composite image.

[0106] Furthermore, this configuration reduces the cost of the second image sensor 124, thereby lowering the manufacturing cost of the imaging device 100.

[0107] Furthermore, according to this embodiment, the display unit 121 displays thumbnail images such as HDR composite images, and when a thumbnail image is selected, it displays the HDR composite image corresponding to the selected thumbnail image.

[0108] This configuration allows users to easily identify HDR composite images and the like stored in the storage unit 110, thereby reducing the burden on users when selecting HDR images and the like to display on the display unit 121.

[0109] Furthermore, according to this embodiment, the system includes a video input step S10 in which the video input unit 120 captures an image of a subject and generates an image signal of the subject; a motion information detection step S20 in which the main control unit 101 detects motion information of the subject based on the image signal; an imaging step S30 in which the main control unit 101 causes the video input unit 120 to capture the subject multiple times with different exposure amounts based on the motion information; and an HDR composite image generation step S40 in which the video signal processing unit 122 generates an HDR composite image of the subject based on multiple image signals with different exposure amounts.

[0110] This configuration allows for the selection of an appropriate imaging mode according to the subject's movement, thereby providing an imaging method that can generate high-quality HDR composite images adapted to the imaging environment.

[0111] Furthermore, according to this embodiment, the main control unit 101, which is a computer, is instructed to execute the following steps: a video input step S10 in which the video input unit 120 captures an image of a subject and generates an image signal of the subject; a motion information detection step S20 in which motion information of the subject is detected based on the image signal; an imaging step S30 in which the video input unit captures an image of the subject multiple times with different exposure amounts based on the motion information; and an HDR composite image generation step S40 in which the video signal processing unit 122 generates an HDR composite image of the subject based on multiple image signals with different exposure amounts.

[0112] This configuration allows for the selection of an appropriate imaging mode according to the subject's movement, enabling the creation of an imaging program capable of generating high-quality HDR composite images adapted to the imaging environment.

[0113] (Embodiment 2) Next, Embodiment 2 will be described. In this embodiment, the case in which the imaging mode is switched based on the movement of the subject and the amount of light will be described. Note that detailed explanations of parts that overlap with Embodiment 1 described above may be omitted as appropriate.

[0114] Figures 12 and 13 are block diagrams showing an example of the configuration of an imaging device according to Embodiment 2 of the present invention. The imaging device 200 includes a video input unit 220, etc., as shown in Figure 12.

[0115] As shown in Figures 12 and 13, the video input unit 220 includes a second video input unit 224, a resolution conversion unit 228, and the like. The second video input unit 224 includes a second image sensor 224b. The second image sensor 224b is an image sensor with a first resolution (high resolution), similar to the first image sensor 123b. The first image sensor 123b and the second image sensor 224b are configured so that their resolutions are appropriately converted by the resolution conversion unit 228.

[0116] The light intensity sensor 165 measures the amount of light around the imaging device 200 and near the subject. The light intensity sensor 165 then outputs the measured light intensity information to the exposure control unit 126 and the main control unit 101.

[0117] The exposure control unit 126 sets the exposure amount for the first video input unit 123 and the second video input unit 224d when taking images, based, for example, on light intensity information output from the light intensity sensor 165 and instructions from the main control unit 101.

[0118] The main control unit 101 switches the imaging mode based on the light intensity measured by the light intensity sensor 165 and the detection result of the motion vector of the subject. The main control unit 101 also instructs the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b from the first resolution (high resolution) to the second resolution (low resolution) by grouping multiple pixels. The main control unit 101 also instructs the resolution conversion unit 228 to ungroup multiple pixels, thereby converting the resolution of the first image sensor 123b and the second image sensor 224b from the second resolution (low resolution) to the first resolution (high resolution).

[0119] For example, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b according to the imaging mode. Based on the resolution conversion information, the resolution conversion unit 228 converts the resolution of the first image sensor 123b and the second image sensor 224b. In the image sensor converted to a lower resolution, multiple grouped pixels are treated as a single pixel. As a result, the area of ​​one pixel after grouping increases compared to one pixel before grouping, and the exposure amount increases, so the exposure time (shutter speed) required to obtain the same exposure amount is shortened.

[0120] The light intensity sensor 165 measures, for example, the amount of light from a high-light region to a low-light region of the subject. Figures 12 and 13 illustrate an example where the light intensity sensor 165 is provided independently, but the configuration is not limited to this, and for example, the first image sensor 123b or the second image sensor 224b may also have the function of the light intensity sensor 165.

[0121] Next, the imaging method according to this embodiment will be described. Figures 14 and 15 are flowcharts relating to the imaging method of Embodiment 2 of the present invention. In this embodiment as well, the video input step S10, motion information detection step S20, imaging step S30, and HDR composite image generation step S40 shown in Figure 5 are performed. The video input step S10 and motion information detection step S20 are the same as in Embodiment 1 described above.

[0122] In the imaging step S30, step S202 is performed first. In step S202, the movement of the subject is determined based on motion information. Specifically, the main control unit 101 determines the movement of the subject by comparing the motion vector detected in the motion information detection step S20 with a third motion threshold. More specifically, the main control unit 101 reads the third motion threshold stored in the storage unit 110 into the memory 104 and compares the motion vector with the third motion threshold. The third motion threshold is set as appropriate, for example, through experiments.

[0123] [Still image capture mode] In step S202, if the main control unit 101 determines that the motion vector is less than the third motion threshold, step S214 is performed. That is, the main control unit 101 determines that the subject is hardly moving and switches to still image capture mode.

[0124] In step S214, the same process as in step S113 in Embodiment 1 is performed. That is, the first video input unit 123 continuously images the subject with different exposure amounts. In the first image, the first video input unit 123 uses, for example, exposure amount L A0 The subject is imaged with the first lens, and in the second image, L A0 Lower exposure L A1 Image is taken with (L A0 >L A1 The first image sensor 123b generates electrical signals (A0, A1) for a high-resolution image, as shown in Figure 9, and outputs the generated electrical signals (A0, A1) to the first image processing unit 122a. Here, we describe the case where two types of image signals with different exposure levels are combined, but for example, three or more types of image signals may be combined.

[0125] On the other hand, the second video input unit 224 may, for example, capture a video (Bm) of the subject, or it may capture the subject with different depths of field. Also, since the second image sensor 224b of the second video input unit 224 is high resolution, for example, the exposure amount may be different (for example, L A1 (etc.) The subject may be imaged simultaneously with the first video input unit 123. The second image sensor 224b has an exposure amount L A0 Electrical signal (A0), exposure amount L A1 The first image processing unit 122a generates an electrical signal (A1) and outputs the generated electrical signals (A0, A1). The second image sensor 224b outputs an image signal related to video, etc., to the second image processing unit 122b of the video signal processing unit 122. Then, the process proceeds to the HDR composite image generation step S40.

[0126] In the HDR composite image generation step S40, step S215 is performed. In step S215, the same processing as in step S114 in Embodiment 1 is performed. That is, based on the multiple image signals (A0, A1) with different exposure amounts generated in step S214, the image synthesis unit 122c generates an HDR composite image (A0+A1) of a first resolution. Alternatively, the image synthesis unit 122c may generate an HDR composite image based on electrical signals generated by the first image sensor 123b and the second image sensor. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate the captured image, the moving image, and the thumbnail images corresponding to these images.

[0127] The first image processing unit 122a and the second image processing unit 122b may convert the high-resolution image signal to a low-resolution image signal, and the image synthesis unit 122c may generate a low-resolution HDR composite image. Then, the process proceeds to step S216.

[0128] In step S216, the same processing as in step S115 in Embodiment 1 is performed. That is, the HDR composite image generated in step S215, the captured image before synthesis, the video, the thumbnail image, etc. are stored in the storage unit 110.

[0129] [Motion imaging mode] Figure 16 is a schematic diagram showing the HDR synthesis process according to Embodiment 2 of the present invention. In step S202 of the imaging step S30, if the main control unit 101 determines that the motion vector is greater than or equal to the third motion threshold, it determines that the subject is moving significantly and switches to motion imaging mode. Once the system switches to motion imaging mode, it proceeds to step S203.

[0130] In step S203, the imaging mode is switched based on the light intensity measured by the light intensity sensor 165. Specifically, the light intensity sensor 165 measures the light intensity around the imaging device 200 and outputs information about the measured light intensity as light intensity information to the main control unit 101. Based on the input light intensity information, the main control unit 101 compares the measured light intensity with a first light intensity threshold and a second light intensity threshold that is greater than the first light intensity threshold. More specifically, the main control unit 101 reads the first light intensity threshold and the second light intensity threshold stored in the storage unit 110 into the memory 104 and compares the light intensity with the first light intensity threshold and the second light intensity threshold. The first light intensity threshold and the second light intensity threshold are set as appropriate, for example, through experiments.

[0131] <High-light dynamic subject imaging mode> If the main control unit 101 determines that the light intensity exceeds the second light intensity threshold, the system proceeds to step S224. That is, the main control unit 101 determines that the light intensity is high and switches to the high-light intensity motion imaging mode.

[0132] In step S224, the main control unit 101 instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure levels. At this time, the main control unit 101 sets the resolution of the first image sensor 123b and the second image sensor 224b to a first resolution, which is high resolution.

[0133] For example, if the resolution of the first image sensor 123b or the second image sensor 224b is set to a low resolution, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the corresponding image sensor. Based on the input resolution conversion information, the resolution conversion unit 228 converts the resolution of the corresponding image sensor from low resolution to high resolution.

[0134] Once the resolution of the image sensor is set in this way, the first video input unit 123 sets the exposure amount L A0 The camera captures an image of the subject and generates an electrical signal (A0) for the high-resolution image, such as the one shown in Figure 16. The second video input unit 124 is L A0 Lower exposure L B0 The subject is imaged (L A0 >L B0 For example, an electrical signal (B0) is generated for a high-resolution image, such as the one shown in Figure 16.

[0135] The first image sensor 123b has an exposure amount L A0 The electrical signal (A0) is output to the first image processing unit 122a. The second image sensor 124b controls the exposure amount L B0 The electrical signal (B0) is output to the second image processing unit 122b. Then, the process proceeds to the HDR composite image generation step S40.

[0136] In the HDR composite image generation step S40, step S225 is performed. In step S225, based on the multiple image signals (A0, B0) generated in step S224, the image synthesis unit 122c generates a high-resolution HDR composite image (A0+B0), for example, as shown in Figure 16. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate the captured image, the video, and the thumbnail images corresponding to these images before synthesis.

[0137] The first image processing unit 122a and the second image processing unit 122b may convert the high-resolution image signal into a low-resolution image signal, and the image synthesis unit 122c may generate a low-resolution HDR composite image. Then, the process proceeds to step S237.

[0138] In step S237, the same processing as in step S126 in Embodiment 1 is performed. That is, the HDR composite image generated in step S225, the captured image before synthesis, the video, the thumbnail image, etc. are stored in the storage unit 110. Then, the process proceeds to step S204. Step S204 will be described later.

[0139] <Medium Light Intensity Motion Imaging Mode> Figure 17 is a schematic diagram showing the HDR synthesis process according to Embodiment 2 of the present invention. When the main control unit 101 determines that the light intensity is above the first light intensity threshold and below the second light intensity threshold, it proceeds to step S234. That is, the main control unit 101 determines that the light intensity is less than in the high-light intensity motion imaging mode and switches to the medium-light intensity motion imaging mode.

[0140] In step S234, the main control unit 101 instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure levels. At this time, the main control unit 101 instructs the resolution conversion unit 228 to convert the resolution of the first image sensor 123b to the first resolution and the resolution of the second image sensor 224b to the second resolution.

[0141] For example, if the resolution of the first image sensor 123b is set to a low resolution, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the first image sensor 123b. Based on the input resolution conversion information, the resolution conversion unit 228 converts the resolution of the first image sensor 123b from low resolution to high resolution. Also, if the resolution of the second image sensor 224b is set to a high resolution, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the second image sensor 224b. Based on the input resolution conversion information, the resolution conversion unit 228 converts the resolution of the second image sensor 224b from high resolution to low resolution.

[0142] Once the resolution of the image sensor is set in this way, the first video input unit 123 sets the exposure amount L A0 The camera captures an image of the subject and generates an electrical signal (A0) for the high-resolution captured image, such as those shown in Figures 17(A) and (B). The second video input unit 124 is L A0 Lower exposure L b0 The subject is imaged (L A0 >L b0 ), for example, it generates an electrical signal (b0) for low-resolution captured images as shown in Figures 17(A) and (B). Since the resolution of the second video input unit 224 is lower than that of the first video input unit 123, the exposure amount can be appropriately set to expand the dynamic range in the HDR composite image.

[0143] The first image sensor 123b has an exposure amount L A0 The electrical signal (A0) is output to the first image processing unit 122a. The second image sensor 124b controls the exposure amount L b0 The electrical signal (b0) is output to the second image processing unit 122b. Then, the process proceeds to the HDR composite image generation step S40.

[0144] In the HDR composite image generation step S40, step S235 is performed first. In step S235, for example, the same processing as in step S124 in Embodiment 1 is performed. That is, in step S235, the resolution of the electrical signal generated in step S234 is converted. Specifically, the first image processing unit 122a performs a resolution conversion on the high-resolution electrical signal (A0) to generate a low-resolution electrical signal (A0d), for example, as shown in Figure 17(B). The second image processing unit 122b performs a resolution conversion on the low-resolution electrical signal (b0) to generate a high-resolution electrical signal (b0u), for example, as shown in Figure 17(A). Then, the process proceeds to step S236.

[0145] In step S236, the image synthesis unit 122c generates a high-resolution HDR composite image (A0+b0u), for example, as shown in Figure 17(A), and a low-resolution HDR composite image (A0d+b0), for example, as shown in Figure 17(B), based on the multiple electrical signals (A0, b0, A0d, b0u) generated in step S235. The image synthesis unit 122c also generates thumbnail images corresponding to the HDR composite images. The first image processing unit 122a and the second image processing unit 122b generate the captured images, moving images, and thumbnail images corresponding to these images before synthesis. Then, the process proceeds to step S237.

[0146] In step S237, the HDR composite image generated in step S236, the captured image before synthesis, the video, the thumbnail image, etc., are stored in the storage unit 110. Then, the process proceeds to step S204. Step S204 will be described later.

[0147] <Low-light motion imaging mode> Figure 18 is a schematic diagram showing the HDR synthesis process according to Embodiment 2 of the present invention. When the main control unit 101 determines that the light intensity is less than the first light intensity threshold, it proceeds to step S244. That is, the main control unit 101 determines that the light intensity is lower than in the medium light intensity imaging mode and switches to the low light intensity motion imaging mode.

[0148] In step S224, the main control unit 101 instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure levels. At this time, the main control unit 101 instructs the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b to a lower resolution, the second resolution.

[0149] For example, if the resolution of the first image sensor 123b or the second image sensor 224b is set to high resolution, the main control unit 101 outputs resolution conversion information to the resolution conversion unit 228 to convert the resolution of the corresponding image sensor. Based on the input resolution conversion information, the resolution conversion unit 228 converts the resolution of the corresponding image sensor from high resolution to low resolution.

[0150] Once the resolution of the image sensor is set in this way, the first video input unit 123 sets the exposure amount L a0 The camera captures an image of the subject and generates an electrical signal (a0) for the low-resolution captured image, for example, as shown in Figure 18. The second video input unit 124 is L a0 Lower exposure L b0 The camera captures an image of the subject and generates an electrical signal (b0) for the low-resolution captured image, as shown in Figure 18. Since the resolutions of the first video input unit 123 and the second video input unit 224 are set to low resolution, the exposure amount can be appropriately set to expand the dynamic range in the HDR composite image.

[0151] The first image sensor 123b has an exposure amount L a0 The electrical signal (a0) is output to the first image processing unit 122a. The second image sensor 124b controls the exposure amount L b0 The electrical signal (b0) is output to the second image processing unit 122b. Then, the process proceeds to the HDR composite image generation step S40.

[0152] In the HDR composite image generation step S40, step S225 is performed. In step S225, based on the multiple image signals (a0, b0) generated in step S224, the image synthesis unit 122c generates a low-resolution HDR composite image (a0+b0), for example, as shown in Figure 18. The image synthesis unit 122c also generates a thumbnail image corresponding to the HDR composite image. The first image processing unit 122a and the second image processing unit 122b generate the captured image, the video, and the thumbnail images corresponding to these images before synthesis. Then, the process proceeds to step S237.

[0153] In step S237, the HDR composite image generated in step S225, the captured image before synthesis, the video, the thumbnail image, etc., are stored in the storage unit 110.

[0154] The first image processing unit 122a and the second image processing unit 122b may convert the low-resolution image signal into a high-resolution image signal, and the image synthesis unit 122c may generate a high-resolution HDR composite image. Then, the process proceeds to step S204.

[0155] In step S204, the same process as in step S104 in Embodiment 1 is performed to select an image or the like to be displayed on the display unit 121. Then, the process proceeds to step S205.

[0156] In step S205, the same process as in step S105 in Embodiment 1 is performed, and the image selected by the user in step S204 is displayed on the display unit 121.

[0157] According to this embodiment, a light intensity sensor 165 is provided to measure the amount of light, and the main control unit 101 causes the video input unit 220 to take multiple images of the subject with different exposure amounts based on the motion vector and the amount of light.

[0158] This configuration allows for the selection of a more appropriate imaging mode based on the subject's movement and ambient light levels, resulting in the generation of high-quality HDR composite images adapted to the imaging environment.

[0159] Furthermore, according to this embodiment, the main control unit 101 compares the motion vector with a third motion threshold, and if it determines that the motion information is less than the third motion threshold, it switches to still image capture mode, instructs the first video input unit 123 to continuously capture images of the subject with different exposure amounts, and instructs the video signal processing unit 122 to generate an HDR composite image of a first resolution.

[0160] With this configuration, since the subject hardly moves, even if the subject is imaged with a time delay, the generation of noise due to the subject's movement can be suppressed. As a result, a high-quality HDR composite image adapted to the imaging environment can be generated using only the first video input unit 123. Furthermore, this makes it possible to use the second video input unit 224 for video capture, etc., providing a user-friendly imaging device 200.

[0161] Furthermore, according to this embodiment, if the main control unit 101 determines that the motion vector is greater than or equal to the third motion threshold and the light intensity exceeds the second light intensity threshold, it switches to high-light intensity motion imaging mode, instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure levels, and instructs the video signal processing unit 122 to generate a high-resolution HDR composite image.

[0162] This configuration suppresses noise caused by subject movement, resulting in the generation of high-quality, high-resolution HDR composite images (A0+B0) even with significant subject movement.

[0163] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is greater than or equal to the third motion threshold, the light intensity is greater than or equal to the first light intensity threshold, and less than or equal to the second light intensity threshold, it switches to the medium light intensity motion imaging mode, instructs the resolution conversion unit 228 to convert the resolution of the second image sensor 224 to the second resolution, and instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure amounts.Then, the main control unit 101 converts the resolution of the image signal generated by the first video input unit 123 to the second resolution, converts the resolution of the image signal generated by the second video input unit 224 to the first resolution, and instructs the video signal processing unit 122 to generate an HDR composite image of the first resolution and an HDR composite image of the second resolution.

[0164] With this configuration, the resolution of the second image sensor 224b is converted to a lower resolution, which increases the area of ​​the pixels after the resolution conversion, thus improving the sensitivity of each pixel. As a result, the imaging time (exposure time) by the second video input unit 224 is shortened, so even in low light conditions, the generation of noise due to the movement of the subject is suppressed, and high-quality HDR composite images (A0+b0u, A0d+b0) adapted to the imaging environment are generated.

[0165] Furthermore, this configuration generates multiple HDR composite images with different resolutions, providing the user with an HDR image suitable for their specific needs and resulting in a user-friendly imaging device 200.

[0166] Furthermore, according to this embodiment, when the main control unit 101 determines that the motion vector is greater than or equal to the third motion threshold and the light intensity is less than the first light intensity threshold, it switches to a low-light motion imaging mode, instructs the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b to the second resolution, instructs the first video input unit 123 and the second video input unit 224 to simultaneously image the subject with different exposure levels, and instructs the video signal processing unit 122 to generate an HDR composite image of the second resolution.

[0167] With this configuration, the resolution of the first image sensor 123b and the second image sensor 224b is converted to a lower resolution, which increases the area of ​​the pixels after the resolution conversion, thus improving the sensitivity of each pixel. As a result, the imaging time for both the first video input unit 123 and the second video input unit 224 is shortened, so even in situations with even less light, the generation of noise due to the movement of the subject is suppressed, and a high-quality HDR composite image (a0+b0) adapted to the imaging environment is generated.

[0168] Furthermore, according to this embodiment, the main control unit 101 instructs the resolution conversion unit 228 to convert the resolution of the first image sensor 123b and the second image sensor 224b from high resolution to low resolution by grouping multiple pixels, and then instructs the resolution conversion unit 228 to ungroup the multiple pixels, thereby converting the resolution of the first image sensor 123b and the second image sensor 224b from low resolution to high resolution.

[0169] This configuration eliminates the need to prepare separate image sensors for each resolution, thus allowing for a smaller imaging device. Furthermore, this helps to reduce the increase in manufacturing costs for the imaging device.

[0170] (Other embodiments) In embodiments 1 and 2 described above, the main control unit 101 detects motion information of the subject (e.g., motion vector) based on the image signal generated by the first image sensor 123b or the second image sensor 124b (224b). However, motion information of the subject may also be detected based on, for example, the focal length of the video input unit 120. Specifically, when the focal length of the video input unit 120 is short, the effect of camera shake is small, and when the focal length is long, the effect of camera shake is large. That is, when the focal length is short, the main control unit 101 detects motion information that the subject is moving slowly, and when the focal length is long, it detects motion information that the subject is moving slowly. Therefore, the main control unit 101 compares the focal length of the video input unit 120 with a first focal length threshold and a second focal length threshold that is greater than the first focal length threshold. Then, if the main control unit 101 determines that the focal length of the video input unit 120 is less than the first focal length threshold, it switches to still image capture mode. Furthermore, the main control unit 101 switches to the micro-motion imaging mode if it determines that the focal length of the video input unit 120 is above the first focal length threshold and below the second focal length threshold. Also, if the main control unit 101 determines that the focal length of the video input unit 120 exceeds the second focal length threshold, it switches to the motion imaging mode. In this way, the above-described effects can be obtained by switching the imaging mode based on the focal length of the video input unit 120.

[0171] Furthermore, while embodiments 1 and 2 described a case where a subject is captured by two cameras, the first video input unit 123 and the second video input unit 124 (224), and an HDR composite image is generated, it is also possible to capture a subject using three or more cameras and generate an HDR composite image.

[0172] Although embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to these embodiments. Furthermore, the numerical values ​​and other figures appearing in the text and figures are merely examples, and using different values ​​will not impair the effects of the present invention.

[0173] The functions of the present invention described above may be implemented in hardware, for example, by designing them with an integrated circuit, or they may be implemented in software by a computer such as a microprocessor unit interpreting and executing a program that implements each of these functions, or they may be implemented by using both hardware and software in combination.

[0174] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0175] 100...Imaging device, 101...Main control unit, 104a...Basic operation execution unit, 104b...Camera function execution unit, 110a...Basic operation program storage area, 110b...Camera function program storage area, 120...Video input unit, 123...First video input unit, 123b...First image sensor, 124...Second video input unit, 124b...Second image sensor, 165...Light intensity sensor, 224...Second video input unit, 224b...Second image sensor, 228...Resolution conversion unit

Claims

1. A video input unit that captures an image of a subject and generates an image signal of the subject, A video signal processing unit that generates an image of the subject based on the aforementioned image signal, A control unit that generates multiple image signals of the subject with different exposure amounts to the video input unit based on the focal length, and generates an HDR composite image of the subject based on the multiple image signals with different exposure amounts to the video signal processing unit, A storage unit for storing the HDR composite image and a thumbnail image corresponding to the HDR composite image, Equipped with, The aforementioned video input unit is A first video input unit that generates an image signal of a first resolution, The system includes a second video input unit that generates an image signal with a second resolution different from that of the first video input unit, The control unit, A first imaging mode is provided, in which the first video input unit generates multiple image signals of the subject with different exposure amounts, the video signal processing unit generates an image of a first resolution based on the image signals generated by the first video input unit, and the HDR composite image is generated using the image of the first resolution. A second imaging mode is provided, in which the first and second video input units generate image signals of the subject with different exposure amounts, the video signal processing unit generates an image of the first resolution from the image signal of the first resolution generated by the first video input unit, generates an image of the second resolution from the image signal of the second resolution generated by the second video input unit, and further converts the image of the second resolution to the first resolution, or converts the image of the first resolution to the second resolution, and generates the HDR composite image using the image of the first resolution and the image converted to the first resolution, or generates the HDR composite image using the image converted to the second resolution and the image of the second resolution. Equipped with, The control unit controls the switching between the first imaging mode and the second imaging mode based on the result of comparing the focal length with a focal length threshold. Imaging device.

2. In the imaging apparatus according to claim 1, The resolution of the first image sensor in the first video input unit is different from the resolution of the second image sensor in the second video input unit. Imaging device.

3. In the imaging device according to claim 2, The resolution of the second image sensor is lower than the resolution of the first image sensor. Imaging device.

4. In the imaging apparatus according to claim 1, It includes a display unit that displays the aforementioned HDR composite image, The control unit displays the HDR composite image on the display unit at different sizes based on the resolution of the HDR composite image. Imaging device.

5. In the imaging apparatus according to claim 1, Equipped with a GPS receiver, The control unit stores the location information detected based on the GPS signal acquired by the GPS receiver and the HDR composite image of the first resolution or the HDR composite image of the second resolution in the storage unit. Imaging device.

6. A video input unit that captures an image of a subject and generates an image signal of the subject, A video signal processing unit that generates an image of the subject based on the aforementioned image signal, A control unit that, based on the focal length, causes the video input unit to generate multiple image signals of the subject with different exposure levels, and causes the video signal processing unit to generate an HDR composite image of the subject based on the multiple image signals with different exposure levels, A storage unit for storing the HDR composite image and a thumbnail image corresponding to the HDR composite image, Equipped with, The aforementioned video input unit is A first video input unit that generates an image signal of a first resolution, A second video input unit that generates an image signal with a second resolution different from that of the first video input unit, Equipped with, The control unit, A first imaging mode is provided, in which the first video input unit generates multiple image signals of the subject with different exposure amounts, the video signal processing unit generates an image of a first resolution based on the image signals generated by the first video input unit, and the HDR composite image is generated using the image of the first resolution. A second imaging mode is provided, in which the second video input unit generates multiple image signals of the subject with different exposure amounts, the video signal processing unit generates an image of the second resolution based on the image signals generated by the second video input unit, and the HDR composite image is generated using the image of the second resolution. Equipped with, The control unit controls the switching between the first imaging mode and the second imaging mode based on the result of comparing the focal length with a focal length threshold. Imaging device.

7. In the imaging device according to claim 6, The resolution of the first image sensor in the first video input unit is different from the resolution of the second image sensor in the second video input unit. Imaging device.

8. In the imaging device according to claim 7, The resolution of the second image sensor is lower than the resolution of the first image sensor. Imaging device.

9. In the imaging device according to claim 6, It includes a display unit that displays the aforementioned HDR composite image, The control unit displays the HDR composite image on the display unit at different sizes based on the resolution of the HDR composite image. Imaging device.

10. In the imaging device according to claim 6, Equipped with a GPS receiver, The control unit stores the location information detected based on the GPS signal acquired by the GPS receiver and the HDR composite image of the first resolution or the HDR composite image of the second resolution in the storage unit. Imaging device.

Images