Imaging device

Abstract

To provide an imaging apparatus capable of facilitating photographing of a composite image even in a situation where camera shake occurs.SOLUTION: An imaging apparatus includes an imaging element (110) that captures an image of a subject and generates image data, a control unit (140) that controls an imaging operation for generating image data representing a composite image obtained by synthesizing a plurality of captured images on the basis of the image data of an imaging result multiple times by the imaging element, a blur detection unit (184) that detects a blur state of the imaging apparatus, and a display (120) that displays information. The control unit controls the display unit to display blur state information including a plurality of blurring states detected by the blur detection unit when the imaging element takes multiple images in a composite image shooting operation.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an imaging device having a function of capturing and synthesizing a plurality of images. [Background technology]

[0002] Patent Document 1 discloses an imaging device that obtains a high-definition image by synthesizing a plurality of sets of image signals obtained while performing pixel shifting. In the imaging device of Patent Document 1, an image stabilization actuator that drives a lens is controlled by a command for displacing an image for pixel shifting. This imaging device judges the amount of image blur during image synthesis processing, and if the amount of image blur is equal to or greater than a predetermined value, displays a warning message on the display unit indicating that image blur is large. When the warning message is displayed, image synthesis by pixel shifting is not performed, and the original image is recorded. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-274281 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging device that can easily capture a composite image even in a situation where camera shake occurs. [Means for solving the problem]

[0005] The imaging device according to the present disclosure includes an imaging element that captures an image of a subject and generates image data, a control unit that controls a shooting operation that generates image data showing a composite image obtained by combining the multiple captured images based on image data of the multiple captured images by the imaging element, a blur detection unit that detects a blur state of the imaging device, and a display unit that displays information. The control unit controls the display unit to display blur state information including multiple blur states detected by the blur detection unit during multiple captures by the imaging element in the shooting operation of the composite image. Effect of the Invention

[0006] According to the imaging device of the present disclosure, it is possible to easily capture a composite image even in a situation where camera shake occurs. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing a configuration of a digital camera according to a first embodiment of the present disclosure. [Diagram 2] Block diagram showing the configuration of a BIS processing unit in a digital camera [Diagram 3] A flowchart illustrating a handheld high-resolution shooting operation in a digital camera [Figure 4] FIG. 1 is a diagram for explaining a camera shake status screen in the digital camera according to the first embodiment; [Diagram 5] 1 is a flowchart illustrating a shooting and display process of a digital camera according to a first embodiment; [Figure 6] FIG. 1 is a diagram for explaining the shooting and display process of the digital camera according to the first embodiment; [Figure 7] FIG. 13 is a diagram for explaining a modified example of the shooting and display processing of the digital camera according to the first embodiment; [Figure 8] 1 is a flowchart illustrating a high-resolution synthesis process of a digital camera according to a first embodiment; [Figure 9] A diagram explaining the positional relationship between images to be synthesized in high-resolution synthesis processing [Figure 10] FIG. 1 is a diagram for explaining image synthesis in high-resolution synthesis processing; [Figure 11] 11 is a flowchart illustrating a shooting and display process of a digital camera according to a second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of a camera shake status screen of a digital camera according to a second embodiment; [Figure 13] FIG. 13 shows a first modified example of a camera shake status screen of a digital camera. [Figure 14] FIG. 2 shows a second modified example of the image stabilization status screen of the digital camera. [Figure 15] FIG. 3 shows a third modified example of the image stabilization status screen of the digital camera. [Figure 16] FIG. 4 shows a fourth modified example of the image stabilization status screen of the digital camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings as appropriate. However, in the detailed description, unnecessary parts of the description of the prior art and substantially the same configuration may be omitted. This is for the purpose of simplifying the description. In addition, the following description and the accompanying drawings are disclosed so that a person skilled in the art can fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0009] (Embodiment 1) In the first embodiment, a digital camera with interchangeable lenses having an image blur correction function will be described as an example of an imaging apparatus.

[0010] 1. Configuration 1 is a block diagram showing the configuration of a digital camera 1 according to the first embodiment. Digital camera 1 is made up of a camera body 100 and an interchangeable lens 200 that is detachable from the camera body 100. In the following description, the function of correcting blur by moving an image sensor in the camera body 100 is referred to as a "Body Image Stabilizer (BIS) function." Additionally, the function of correcting blur by moving a correction lens in the interchangeable lens 200 is referred to as an "Optical Image Stabilizer (OIS) function."

[0011] 1-1.Camera body The camera body 100 (an example of an imaging device) includes an image sensor 110, a liquid crystal monitor 120, an operation unit 130, a camera control unit 140, a body mount 150, and a card slot 170.

[0012] The camera control unit 140 controls the operation of the entire digital camera 1 by controlling components such as the image sensor 110 in response to instructions from the release button. The camera control unit 140 transmits a vertical synchronization signal to the timing generator 112. In parallel with this, the camera control unit 140 generates an exposure synchronization signal. The camera control unit 140 periodically transmits the generated exposure synchronization signal to the lens control unit 240 via the body mount 150 and the lens mount 250. The camera control unit 140 uses the DRAM 141 as a working memory during control operations and image processing operations.

[0013] The image sensor 110 is an example of an imaging element that captures an image of a subject incident via the interchangeable lens 200 and generates image data. The image sensor 110 is, for example, a CCD, a CMOS image sensor, or an NMOS image sensor. The generated image data is digitized by an AD converter 111. The digitized image data is subjected to predetermined image processing by the camera control unit 140. The predetermined image processing is, for example, gamma correction processing, white balance correction processing, scratch correction processing, YC conversion processing, electronic zoom processing, and JPEG compression processing.

[0014] The image sensor 110 operates at a timing controlled by a timing generator 112. The image sensor 110 generates still or moving images for recording, or through images (i.e., live view images). The through images are mainly moving images, and are displayed on the liquid crystal monitor 120 so that the user can decide the composition for capturing a still image.

[0015] The liquid crystal monitor 120 displays various information such as images such as through images and menu screens. The liquid crystal monitor 120 is an example of a display unit in this embodiment. Instead of the liquid crystal monitor, other types of display devices, for example, an organic EL display device, may be used.

[0016] The operation unit 130 includes various operation members such as a release button for instructing the start of shooting, a mode dial for setting a shooting mode, a power switch, etc. The operation unit 130 also includes a touch panel arranged so as to be superimposed on the liquid crystal monitor 120.

[0017] The card slot 170 is capable of receiving a memory card 171, and controls the memory card 171 under control of the camera control unit 140. The digital camera 1 is capable of storing image data in the memory card 171 and reading image data from the memory card 171.

[0018] The body mount 150 can be mechanically and electrically connected to the lens mount 250 of the interchangeable lens 200. The body mount 150 is an example of a communication unit on the camera body 100 side capable of transmitting and receiving data to and from the interchangeable lens 200 via the lens mount 250. The body mount 150 transmits an exposure synchronization signal received from the camera control unit 140 to the lens control unit 240 via the lens mount 250. In addition, the body mount 150 transmits other control signals received from the camera control unit 140 to the lens control unit 240 via the lens mount 250. In addition, the body mount 150 transmits signals received from the lens control unit 240 to the camera control unit 140 via the lens mount 250.

[0019] Moreover, as components for realizing the BIS function, camera body 100 further includes a gyro sensor 184 (shake detection unit) that detects shake of camera body 100, and a BIS processing unit 183 that controls shake correction processing based on the detection result of gyro sensor 184. Furthermore, camera body 100 includes a sensor driving unit 181 that moves image sensor 110, and a position sensor 182 that detects the position of image sensor 110.

[0020] The sensor driving unit 181 can be realized by, for example, a magnet and a flat coil. The sensor driving unit 181 may include other motors, actuators, etc. The position sensor 182 is a sensor that detects the position of the image sensor 110 in a plane perpendicular to the optical axis of the optical system. The position sensor 182 can be realized by, for example, a magnet and a Hall element.

[0021] The BIS processing unit 183 controls the sensor driving unit 181 based on the signal from the gyro sensor 184 and the signal from the position sensor 182 to shift the image sensor 110 within a plane perpendicular to the optical axis so as to offset the shake of the camera body 100.

[0022] 1-2.Interchangeable lenses The interchangeable lens 200 includes an optical system, a lens control unit 240, and a lens mount 250. The optical system includes a zoom lens 210, an OIS (Optical Image Stabilizer) lens 220, a focus lens 230, and an aperture 260.

[0023] The zoom lens 210 is a lens for changing the magnification of a subject image formed by the optical system. The zoom lens 210 is composed of one or more lenses. The zoom lens 210 is driven by a zoom driving unit 211. The zoom driving unit 211 includes a zoom ring that can be operated by the user. Alternatively, the zoom driving unit 211 may include a zoom lever and an actuator or a motor. The zoom driving unit 211 moves the zoom lens 210 along the optical axis direction of the optical system in response to an operation by the user.

[0024] The focus lens 230 is a lens for changing the focus state of a subject image formed on the image sensor 110 by the optical system. The focus lens 230 is composed of one or more lenses. The focus lens 230 is driven by a focus driver 233.

[0025] The focus driver 233 includes an actuator or a motor, and moves the focus lens 230 along the optical axis of the optical system under the control of the lens controller 240. The focus driver 233 can be realized by a DC motor, a stepping motor, a servo motor, an ultrasonic motor, or the like.

[0026] The OIS lens 220 is an example of a lens for correcting blur of a subject image formed by the optical system of the interchangeable lens 200 in the OIS function, that is, a correction lens. The OIS lens 220 reduces blur of the subject image on the image sensor 110 by moving in a direction that offsets blur of the digital camera 1. The OIS lens 220 is composed of one or more lenses. The OIS lens 220 is driven by an OIS driver 221.

[0027] The OIS driving unit 221 shifts the OIS lens 220 in a plane perpendicular to the optical axis of the optical system under the control of the OIS processing unit 223. The OIS driving unit 221 can be realized, for example, by a magnet and a flat coil. The position sensor 222 is a sensor that detects the position of the OIS lens 220 in a plane perpendicular to the optical axis of the optical system. The position sensor 222 can be realized, for example, by a magnet and a Hall element. The OIS processing unit 223 controls the OIS driving unit 221 based on the output of the position sensor 222 and the output of a gyro sensor 224 (shake detection unit).

[0028] The lens mount 250 is an example of a communication section on the interchangeable lens 200 side that is capable of transmitting and receiving data to and from the camera body 100 via the body mount 150 .

[0029] The diaphragm 260 adjusts the amount of light incident on the image sensor 110. The diaphragm 260 is driven by an diaphragm driving unit 262, and the size of the opening is controlled. The diaphragm driving unit 262 includes a motor or an actuator.

[0030] The gyro sensor 184 or 224 detects shake (vibration) in, for example, one or more of the yaw direction, pitch direction, and roll direction based on the angle change per unit time of the digital camera 1, i.e., the angular velocity. The gyro sensor 184 or 224 outputs an angular velocity signal indicating the amount of shake (angular velocity) detected to the BIS processing unit 183 or OIS processing unit 223. The angular velocity signal output by the gyro sensor 184 or 224 may include a wide range of frequency components due to camera shake, mechanical noise, etc. Instead of the gyro sensor, another sensor capable of detecting shake of the digital camera 1 may be used.

[0031] The camera control unit 140 and the lens control unit 240 may be configured with a hardwired electronic circuit, or may be configured with a microcomputer using a program, etc. For example, the camera control unit 140 and the lens control unit 240 can be realized by various processors such as a CPU, an MPU, a GPU, a DSU, an FPGA, or an ASIC.

[0032] 1-3.BIS Processing Section 2, the configuration of the BIS processing unit 183 in the camera body 100 will be described. The BIS processing unit 183 includes an ADC (analog / digital conversion unit) / LPF (low pass filter) 405, an HPF (high pass filter) 406, a phase compensation unit 407, an integrator 408, and a PID control unit 410.

[0033] The ADC / LPF 405 converts the angular velocity signal from the gyro sensor 184 from analog to digital format. Furthermore, the ADC / LPF 405 cuts off high frequency components of the angular velocity signal converted into digital format in order to eliminate noise and extract only the shake of the digital camera 1. The frequency of the photographer's hand shake is a low frequency of about 1 to 10 Hz, and the cutoff frequency of the LPF is set taking this into consideration. If noise is not a problem, the LPF function can be omitted.

[0034] The HPF 406 blocks a predetermined low-frequency component contained in the signal received from the ADC / LPF 405 in order to block the drift component.

[0035] The phase compensation unit 407 compensates for the phase delay of the signal received from the HPF 406, the phase delay being caused by the sensor driving unit 181 and the like.

[0036] The integrator 408 integrates the signal indicating the angular velocity of the shake (vibration) input from the phase compensation unit 407 to generate a signal indicating the angle of the shake (vibration) (hereinafter referred to as a "shake detection signal"). The shake detection signal from the integrator 408 is input to the PID control unit 410.

[0037] The PID control unit 410 generates a drive signal for shifting the image sensor 110 based on the output from the position sensor 182 and the output from the integrator 408, and outputs the drive signal to the sensor drive unit 181. The sensor drive unit 181 drives the image sensor 110 based on the drive signal.

[0038] The BIS processing unit 183 is configured to be able to perform data communication with the camera control unit 140. For example, the BIS processing unit 183 starts / ends the image stabilization operation in response to a control signal from the camera control unit 140. In addition, the BIS processing unit 183 transmits various pieces of information related to the image stabilization operation to the camera control unit 140.

[0039] The OIS processing unit 223 can be configured to drive the OIS driving unit 221 instead of the sensor driving unit 181 in the same configuration as the BIS processing unit 183 described above. The OIS processing unit 223 also operates using the detection result of the gyro sensor 224 in the interchangeable lens 200 instead of the gyro sensor 184 in the camera body 100, for example.

[0040] 2. Operation The operation of the digital camera 1 configured as above will now be described.

[0041] Digital camera 1 of this embodiment has an operation mode in which it takes images multiple times while held by a user, and synthesizes the multiple captured images to generate a high-resolution composite image (hereinafter referred to as "handheld high-resolution shooting mode"). Digital camera 1 can be set to the handheld high-resolution shooting mode by a user operation of operation unit 130, such as a setting menu.

[0042] In the handheld high-resolution shooting mode, a composite image is generated from multiple captured images with half-pixel position shifts, utilizing camera shake that occurs while the user is holding digital camera 1 and capturing multiple images. If camera shake is excessive, the angle of view of the composite image may be reduced, and image composition may fail. In this embodiment, digital camera 1 visualizes the state of camera shake to the user while capturing multiple images in handheld high-resolution shooting mode, making it easier for the user to suppress camera shake. The operation of digital camera 1 in this embodiment will be described in detail below.

[0043] 2-1. Handheld high-resolution shooting operation The operation of the handheld high-resolution shooting mode in digital camera 1 of this embodiment will be described with reference to FIGS.

[0044] 3 is a flowchart illustrating a handheld high-resolution shooting operation in digital camera 1. The handheld high-resolution shooting operation is an example of a shooting operation for a composite image in this embodiment. The process shown in the flowchart in FIG. 3 is started, for example, when a live view screen showing a live view image is displayed on digital camera 1, and is executed by camera control unit 140.

[0045] First, the camera control unit 140 accepts input of an instruction to start image capture (S1) in response to, for example, a user operation on the operation unit 130. The user can input a capture instruction by, for example, pressing a release button on the operation unit 130.

[0046] When a shooting instruction is input (YES in S1), the camera control unit 140 executes a shooting display process (S2) in which the state of camera shake is displayed on the liquid crystal monitor 120 while capturing images multiple times to generate one composite image. An example of the display in step S2 is shown in FIG.

[0047] 4 shows an example of a screen for visualizing the camera shake state during multiple captures in the capture display process (S2) of this embodiment, i.e., a camera shake state screen. While viewing the camera shake state screen, the user holds the digital camera 1 in a held position until multiple captures in step S2 are completed. The capture display process (S2) will be described in detail later.

[0048] Next, the camera control unit 140 performs a high-resolution synthesis process (S3) to generate image data showing a composite image based on image data of the multiple captured images obtained as a result of the capture display process (S2). In the high-resolution synthesis process (S3), high-resolution image data in which the number of pixels of the composite image is greater than that of each of the original captured images is generated based on data of portions in which the same subject image is captured between multiple captured images having a positional shift of, for example, about 1 / 2 pixel. The details of the high-resolution synthesis process (S3) will be described later.

[0049] The camera control unit 140 records image data of a composite image obtained by, for example, the high-resolution composition process (S3) in the memory card 171, and ends the process shown in this flowchart.

[0050] According to the above-described handheld high-resolution shooting operation, the digital camera 1 performs a high-resolution synthesis process (S3) based on a plurality of captured images that are taken with a relative shift due to camera shake during the shooting and display process (S2) to generate a high-resolution composite image.

[0051] In the high-resolution synthesis process (S3), the required positional deviation between the multiple captured images is considered to be small enough to occur unintentionally when the user is holding the digital camera 1. On the other hand, if the positional deviation between the multiple captured images becomes too large due to excessive camera shake, the angle of view of the synthesized image will be reduced.

[0052] Therefore, the digital camera 1 of this embodiment displays a camera shake status screen (S2) as shown in FIG. 4, for example, during image capture before synthesis, to assist the user in suppressing camera shake.

[0053] 4, the camera shake state screen includes a preview image 20, a scope section 30 superimposed on the preview image 20, and a number of camera shake pointers 40 plotted and displayed. The preview image 20 is, for example, a live view image immediately before the start of the shooting and display process (S2). The scope section 30 is an example of a reference area indicating the allowable range of camera shake during operation in the handheld high-resolution shooting mode.

[0054] Each camera shake pointer 40 indicates the camera shake state of one captured image in the image capturing and displaying process (S2), and is plotted according to the amount of camera shake in each capture. In the example of Fig. 4, each camera shake pointer 40 displays a number indicating the order in which the image was captured. The shape of the camera shake pointer 40 is, for example, a circle.

[0055] For example, the user can recognize the state of camera shake in the captured image according to the position where the camera shake pointer 40 is displayed on the camera shake state screen relative to the scope unit 30. In the example of Fig. 4, the scope unit 30 includes two areas 31 and 32 arranged concentrically.

[0056] In the scope section 30 illustrated in Fig. 4, the inner area 31 indicates, for example, an allowable range of camera shake amount that can ensure the image quality (or angle of view) of the composite image, and is set, for example, according to the number of pixels of the margin in the high-resolution synthesis process (S3). The outer area 32 indicates, for example, an allowable range of camera shake amount that can perform the high-resolution synthesis process (S3) while reducing the image quality of the composite image, and is set, for example, according to a predetermined number of pixels (allowable value) that is greater than the margin. In this example, the scope section 30 displays a crosshair in the area 31 as a reference for the direction of camera shake.

[0057] 2-2. Shooting and display processing The shooting and display process (S2 in FIG. 3) in the handheld high-resolution shooting operation of this embodiment will be described in detail with reference to FIGS.

[0058] Fig. 5 is a flowchart illustrating the shooting display process of the digital camera 1 according to this embodiment. Fig. 6 is a diagram for explaining the shooting display process of this embodiment. The process shown in the flowchart of Fig. 5 is started when a shooting instruction is input in handheld high-resolution shooting mode (YES in S1 of Fig. 3), for example.

[0059] First, the camera control unit 140 controls the liquid crystal monitor 120 to transition from, for example, a live view screen to a camera shake state screen (S10). For example, the camera control unit 140 superimposes and displays the scope unit 30 on the preview image 20 based on the live view image immediately before step S2 as the initial screen of the camera shake state screen (see FIG. 6(A)). For example, at the time of step S10, the camera shake pointer 40 is not particularly displayed.

[0060] Furthermore, the camera control unit 140 starts, for example, the image stabilization operation by the BIS processing unit 183 (S11). For example, the BIS processing unit 183 (FIG. 2) receives an input of a signal from the gyro sensor 184 and starts generating a blur detection signal by the integrator 408 or the like. In the BIS processing unit 183, the PID control unit 410 controls the sensor driving unit 181 in response to the generated blur detection signal, and sequentially shifts the position of the image sensor 110.

[0061] Next, the camera control unit 140 starts exposure for one image capture in the image sensor 110 (S12). The image sensor 110 is exposed to incident light for a preset exposure period. During the exposure period, the image stabilization operation by the BIS processing unit 183 is performed every moment.

[0062] At this time, the camera control unit 140 acquires, for example, from the BIS processing unit 183, camera shake information corresponding to the camera shake state per imaging (S13). The camera shake information indicates the amount of camera shake remaining after subtracting the amount of camera shake corrected by the camera shake correction operation of the BIS processing unit 183 from the amount of camera shake at the start timing of the exposure period, for example.

[0063] For example, in the BIS processing unit 183, the amount of camera shake before correction is calculated from the integral of the angular velocity detected by the gyro sensor 184. Also, the amount of camera shake correction that has been performed is obtained from the displacement of the image sensor 110 detected by the position sensor 182 or the drive amount of the sensor drive unit 181 by the PID control unit 410. The camera control unit 140 acquires the calculation result obtained by subtracting the amount of camera shake correction from the amount of camera shake before correction as camera shake information (S13).

[0064] Next, the camera control unit 140 plots and displays a camera shake pointer 40 as information indicating the camera shake state of one image capture on the camera shake status screen being displayed on the liquid crystal monitor 120 based on, for example, the acquired camera shake information (S14). An example of the display in step S14 is shown in Fig. 6(A). Here, the information displayed is camera shake information, but the amount of camera shake may also be used, and subsequent processing may be performed using the amount of camera shake.

[0065] 6A shows an example of the camera shake state screen display in step S14 for the first time. For example, when the camera control unit 140 acquires camera shake information for the first image capture (S13), it displays the camera shake pointer 40 at a reference position such as the center of the scope unit 30 on the camera shake state screen (S14). At this time, the preview image 20 may be shifted by an amount of camera shake indicated by the acquired camera shake information.

[0066] For example, when the camera control unit 140 completes exposure for one image capture (S15), it determines whether the number of times image capture has been performed reaches a predetermined number (S16). The predetermined number is preset to a number equal to or greater than the number of captured images to be subjected to image synthesis in the high-resolution synthesis process (S3), and is set to, for example, 16 times.

[0067] If the number of times of imaging has not reached the predetermined number (NO in S16), the camera control unit 140 controls, for example, the BIS processing unit 183 to continue the image stabilization operation (S17), and executes the processes from step S12 onwards again. By repeating steps S12 to S15, imaging is performed multiple times and the plot display on the image stabilization status screen is updated sequentially (S14). An example display of step S14 in the second imaging is shown in Fig. 6(B).

[0068] Fig. 6(B) illustrates an example of a camera shake state screen updated from the state of Fig. 6(A). For example, in steps S12 to S15 for the second time, the camera control unit 140 plots a new camera shake pointer 40 at a position shifted by the amount of camera shake indicated by the camera shake information acquired in step S13, based on the position of the first camera shake pointer 40 on the camera shake state screen (S14). At this time, for example, the preview image 20 etc. is fixed without being linked to the amount of camera shake. The user can understand that camera shake has occurred by the amount that the position where the second camera shake pointer 40 is plotted is shifted from the first camera shake pointer 40.

[0069] Similarly, in each subsequent imaging, the camera control unit 140 successively updates and displays the camera shake state screen so that a new camera shake pointer 40 is plotted at a position shifted by the amount of camera shake from the reference camera shake pointer 40 (S14). As a result, a plot of the camera shake pointer 40 that visualizes the camera shake state at each imaging is obtained on the camera shake state screen (see FIG. 4).

[0070] When the number of times of imaging reaches a predetermined number (YES in S16), the camera control unit 140 returns the screen display of the liquid crystal monitor 120 from the camera shake state screen to the live view screen, for example, and ends the imaging and display process (S2 in FIG. 3). After that, the camera control unit 140 executes a high-resolution synthesis process (S3) based on the image data of the imaging result by the imaging and display process.

[0071] According to the above-described shooting display process (S2), during handheld high-resolution shooting operation, while a predetermined number of images are captured to generate a composite image, a camera shake pointer 40 indicating the camera shake condition at each image capture is successively plotted and displayed on the camera shake condition screen (S14).

[0072] Fig. 6(B) illustrates an example in which the amount of camera shake during the second image capture exceeds the margin corresponding to the area 31 inside the scope unit 30. By visually checking the plot of the camera shake pointer 40 on the camera shake state screen, the user can easily understand the camera shake state in which the most recently captured image is shifted from the first captured image. Fig. 4 illustrates an example in which the user, who has seen the camera shake pointer 40 in Fig. 6(B), subsequently consciously suppresses camera shake so that the camera shake pointer 40 falls within the area 31.

[0073] As described above, even if a captured image has significant camera shake from the first capture, the user can visually confirm the camera shake state using the camera shake pointer 40 and easily grasp the direction for improving the camera shake during subsequent capture. In this way, the camera shake state screen in the shooting display process (S2) of this embodiment makes it easy for the user to reduce the relative camera shake between multiple captured images, and as a result, the image quality of the composite image can be improved.

[0074] In such a camera shake state screen, for example, it is conceivable to display all of the camera shake states that change from moment to moment during each exposure period in multiple captures as the locus of the camera shake pointer 40. However, in this case, there is a concern that the display of the camera shake state screen will become excessively complicated, making it more difficult for the user to grasp the camera shake state. In contrast, according to the shooting display process (S2) of this embodiment, the camera shake pointer 40 is intermittently displayed for each capture (see FIG. 4), making it easier for the user to grasp the camera shake state when multiple captures are performed, such as in a handheld high-resolution shooting mode.

[0075] In the above step S13, an example has been described in which the camera shake information is acquired based on the start timing of the exposure period. However, the camera shake information is not limited to this, and may be based on the middle or end timing of the exposure period, or various average values ​​of the camera shake amount at multiple timings during the exposure period may be used.

[0076] In the above step S14, for example, a message regarding the camera shake state may be displayed on the camera shake state screen in addition to the plot display of the camera shake pointer 40. Such a modified example will be described with reference to FIG.

[0077] Fig. 7 shows an example of the display of the camera shake status screen when the amount of camera shake exceeds the allowable value corresponding to the area 32 outside the scope unit 30 during imaging after Fig. 6(B). The camera control unit 140 of this modified example may determine whether the amount of camera shake indicated by the camera shake information acquired in step S13 exceeds the allowable value. If the amount of camera shake exceeds the allowable value, the camera control unit 140 of this modified example may display, in step S14, together with the plot of the camera shake pointer 40, a message 21 warning that image synthesis may fail due to excessive camera shake, as exemplified in Fig. 7.

[0078] By displaying such a message, the user can easily understand the state of camera shake when capturing multiple images for image synthesis. Also, similar to the example of Fig. 7, when the inner area 31 of the scope section 30 is exceeded, the digital camera 1 may display a message such as a warning.

[0079] 2-3. High-resolution composition processing The high-resolution synthesis process (S3 in FIG. 3) in this embodiment will be described in detail with reference to FIGS.

[0080] Fig. 8 is a flowchart illustrating the high-resolution composition process of the digital camera 1 according to this embodiment. The process shown in the flowchart of Fig. 8 is started in a state where the shooting display process (S2 in Fig. 3) is completed during handheld high-resolution shooting operation, for example.

[0081] First, the camera control unit 140 selects, for example, eight captured images to be combined from among a plurality of captured images (for example, 16 captured images) obtained based on the captured image results of the capture and display process (S2 in FIG. 3) (S21). The process of step S21 will be described with reference to FIG.

[0082] Fig. 9 is a diagram explaining the positional relationship (P1 to P8) between images to be synthesized in the high-resolution synthesis process. Fig. 9 illustrates an example of the arrangement of pixels 5r, 5g, and 5b of each color in the image data of a captured image. The red pixels 5r, green pixels 5g, and blue pixels 5b are arranged in a two-dimensional array of a Bayer pattern, and have a period 2W that is twice the pixel pitch W.

[0083] 9 illustrates a position P1 (i.e., a reference position) in a captured image (i.e., a reference image) that is used as a reference among the captured images to be synthesized, and positions P2 to P8 (i.e., corresponding positions) that correspond to the reference position P1 in other captured images to be synthesized. The reference position P1 and each of the corresponding positions P2 to P8 correspond to each other when the same part of the subject image is captured in each captured image.

[0084] 9, the reference position P1 and the corresponding positions P2 to P8 are mutually shifted in units of 1 / 2 the pixel pitch W. Here, for example, when the above-mentioned part of the subject image is captured at positions P2a, P2b, and P2c that are an integer multiple of the period 2W of the Bayer array away from one corresponding position P2, it can be identified as the case of the corresponding position P2 by processing the image data that shifts the captured images by the period 2W.

[0085] Based on the above positional relationships (P1 to P8), the camera control unit 140 selects, for example, the first captured image as a reference image and selects the captured images that are closest to the corresponding positions P2 to P8 with respect to the reference position P1 (S21). In the process of step S21, the identification based on the period 2W of the Bayer array described above is applied.

[0086] For example, in step S21, the camera control unit 140 compares the reference image with each of the other captured images for each appropriately divided region, detects a positional shift of each captured image relative to the reference image, and calculates a remainder positional shift by dividing the detected positional shift by the Bayer period 2W. The camera control unit 140 selects, as the composition target, each captured image whose calculated positional shift is closest to the positional shift between the reference position P1 and each of the corresponding positions P2 to P8.

[0087] Next, the camera control unit 140 judges whether or not the positional deviation between the image data of the selected captured images is within a preset tolerance range (S22). In step S22, the positional deviation including the shift for every period 2W is judged, instead of the remainder positional deviation of the period 2W used in step S21.

[0088] When the camera control unit 140 determines that the positional deviation between the selected captured images is within the range of the allowable value (YES in S22), it executes a process of synthesizing the selected eight captured images (S23). The process of step S22 will be described with reference to FIG.

[0089] FIG. 10 is a diagram for explaining image synthesis in high-resolution synthesis processing. FIG. 10(A) illustrates image data 50 of a captured image to be synthesized. FIG. 10(B) illustrates synthesized data 6r, 6g, and 6b for each color based on the image data 50 of FIG. 10(A). FIG. 10(C) illustrates complementary data 6ra and 6ba for the synthesized data 6r and 6b of FIG. 10(B). FIG. 10(D) illustrates image data 60 of a synthesized image based on the data 6ra, 6g, and 6ba of FIG. 10(B) and (C).

[0090] For example, in step S23, the camera control unit 140 first generates composite data 6r, 6g, and 6b for each color from the image data 50 to be synthesized, as shown in Figures 10(A) and (B). The composite data 6r, 6g, and 6b for each color are generated by arranging the pixel values ​​of the red pixels 5r, green pixels 5g, and blue pixels 5b (see Figure 9) in each image data 50 to be synthesized by color according to the above-mentioned positional relationships (P1 to P8). At this time, if there are pixel values ​​at the same position, for example, an average value is used.

[0091] The composite data 6r, 6g, and 6b of each color obtained as described above are arranged in a staggered pattern, for example, as shown in Fig. 10(B). Therefore, the green composite data 6g matches the Bayer array, but the red and blue composite data 6r and 6b are shifted from the Bayer array. Therefore, the camera control unit 140 performs an interpolation operation on pixel values ​​at positions corresponding to the Bayer array for the red and blue composite data 6r and 6b, for example, based on adjacent pixel values, to generate red and blue complementary data 6ra and 6ba, as shown in Fig. 10(C).

[0092] 10B to 10D, the camera control unit 140 generates image data 60 representing a composite image of the Bayer array by combining the red and blue complementary data 6ra, 6ba and the green composite data 6g (S23). In step S23, the camera control unit 140 may output the Bayer array image data 60 in RAW format as a processing result, or may convert the Bayer array image data 60 into image data in JPEG format or the like as appropriate.

[0093] After generating image data of the composite image (S23), the camera control unit 140 ends the process shown in this flowchart.

[0094] On the other hand, if the camera control unit 140 determines that the positional deviation between the selected captured images is not within the allowable range (NO in S22), the camera control unit 140 does not execute the image synthesis process (S23). In this case, the camera control unit 140 causes the liquid crystal monitor 120 to display an error message indicating that the high-resolution synthesis process has failed (S24), and ends the process of this flow.

[0095] According to the above high-resolution synthesis process (S3 in FIG. 3), if the positional deviation between the multiple captured images is within the allowable range (YES in S22), a composite image can be generated. Also, the smaller the camera shake in the multiple captured images, the higher the quality of the composite image that can be obtained.

[0096] In the above step S21, an example has been described in which image data to be synthesized is selected from the shooting results of the shooting and displaying process (S2). The process of step S21 is not particularly limited to the above, and for example, image data to be synthesized may be generated from image data of the shooting results of the shooting and displaying process (S2). For example, the camera control unit 140 may perform an interpolation process using a weighted average or the like between the multiple captured images in the shooting and displaying process (S2) according to the positional relationship of the above-mentioned synthesis targets (P1 to P8 in FIG. 9) to generate image data to be synthesized according to each corresponding position P2 to P8.

[0097] In step S21, the camera control unit 140 may appropriately process the image data so as to shift or rotate the captured image in accordance with the detected positional deviation from the reference image. The reference image is not necessarily limited to the first captured image, and may be selected from the second captured image and thereafter, or may be generated by an interpolation process or the like. In addition, in the process of step S21, a captured image with a positional deviation within a tolerance range may be preferentially used as appropriate.

[0098] 3. Summary As described above, the digital camera 1, which is an example of an imaging device in this embodiment, includes the image sensor 110, which is an example of an imaging element, the camera control unit 140, which is an example of a control unit, the gyro sensor 184, which is an example of a blur detection unit, and the liquid crystal monitor 120, which is an example of a display unit. The image sensor 110 captures an image of a subject and generates image data. The camera control unit 140 controls a photographing operation to generate image data showing a composite image obtained by combining a plurality of captured images based on image data of the results of multiple captures by the imaging element. The gyro sensor 184 detects the state of blur of the digital camera 1. The liquid crystal monitor 120 displays information. In a photographing operation of a composite image such as a handheld high-resolution photographing operation, the camera control unit 140 controls the liquid crystal monitor 120 to display a camera shake state screen as an example of blur state information including a plurality of blur states detected by the gyro sensor 184 when multiple images are captured by the image sensor 110 (see FIG. 4).

[0099] According to the digital camera 1 described above, the camera shake state when capturing images multiple times is visualized for the user on the camera shake state screen, making it easier to capture a composite image even in a situation where camera shake occurs.

[0100] In this embodiment, the camera control unit 140 controls the liquid crystal monitor 120 to separately display the blurring state at each of the multiple image captures on the camera shake state screen (S12 to S16, see FIG. 6). This makes it easier for the user to understand the blurring state at each image capture on the camera shake state screen. This makes it easier for the user to suppress camera shake and to capture a composite image.

[0101] In this embodiment, the camera shake status screen includes a camera shake pointer 40, which is an example of a pointer that is arranged according to the state of camera shake at each image capture among multiple images. The camera shake pointer 40 allows the user to easily grasp the state of camera shake at each image capture through a simple display, making it easier to capture a composite image.

[0102] In this embodiment, the camera shake state screen further includes a scope section 30 which is an example of a reference area showing a reference for the camera shake state. The camera control section 140 causes the liquid crystal monitor 120 to display the camera shake state screen so that the camera shake pointer 40 is plotted in the scope section 30 according to the camera shake state at each of a plurality of image captures (S12 to S16, see FIG. 6). In this way, the camera shake pointer 40 which is plotted and displayed sequentially during the composite image capture operation makes it easy for the user to grasp the camera shake state during the capture operation being performed.

[0103] It should be noted that the blurring state during each imaging session that is to be displayed as described above does not have to be the blurring state during each imaging operation of the composite image, and some of the imaging sessions out of all imaging sessions may be omitted, or a predetermined number of imaging sessions may be combined.

[0104] In this embodiment, the camera shake state screen includes a preview image 20 captured before the composite image shooting operation. The camera control unit 140 may control the liquid crystal monitor 120 to move the preview image 20 according to the camera shake state, for example, at the time of the first image capture in the composite image shooting operation. The movement of the preview image 20 also allows the user to understand the camera shake state.

[0105] In this embodiment, the camera control unit 140 synthesizes a plurality of captured images having mutual positional deviations based on image data of the captured images taken multiple times, and generates image data showing the composite image (S3). This makes it possible to obtain a composite image with higher resolution than a single captured image, such as a high-resolution composite image. When obtaining a plurality of captured images having such appropriate positional deviations, a camera shake state screen can be shown to the user, making it easier to obtain an appropriate camera shake.

[0106] In this embodiment, the digital camera 1 includes a BIS processing unit 183, which is an example of an image blur correction unit, and a sensor driving unit 181. The camera control unit 140 acquires camera shake information indicating the blur state after correction by the image blur correction unit, and causes the liquid crystal monitor 120 to display a camera shake state screen according to the acquired blur state (S13 to S14). The camera shake state screen is also useful when using such a BIS function.

[0107] (Embodiment 2) Hereinafter, a second embodiment of the present disclosure will be described with reference to Figs. 11 and 12. In the first embodiment, a digital camera 1 that displays the camera shake state during multiple image captures on the camera shake state screen has been described. In the second embodiment, a digital camera 1 that erases the display of the camera shake state during past image captures as appropriate will be described.

[0108] Hereinafter, the description of the configuration and operation similar to those of the digital camera 1 according to the first embodiment will be omitted as appropriate, and the digital camera 1 according to this embodiment will be described.

[0109] Fig. 11 is a flowchart illustrating the shooting and display processing of the digital camera 1 according to the second embodiment. In the digital camera 1 of this embodiment, the camera control unit 140 performs display control (S18) to gradually erase the previous camera shake pointer 40 displayed in step S14, in addition to performing steps S10 to S17 of the shooting and display processing (Fig. 5) of the first embodiment. The display control of step S18 will be described with reference to Fig. 12.

[0110] Fig. 12 illustrates an example of a camera shake status screen of a digital camera according to embodiment 2. Fig. 12 shows an example of the camera shake status screen display at the time of the fifth image capture in the image capture display process of this embodiment. In this example, under the control of the camera control unit 140, the liquid crystal monitor 120 displays the fifth and first camera shake pointers 40 on the camera shake status screen (S14), while not displaying the second to fourth camera shake pointers 40 that have been previously displayed (S18).

[0111] For example, when displaying a new camera shake pointer 40 (S14) during each image capture (S12 to S18) in the image capture display process, the camera control unit 140 sequentially erases the plot display of the previous camera shake pointer 40, except for the first camera shake pointer 40 (S18). Such display control in step S18 simplifies the plot display of the camera shake pointer 40 on the camera shake status screen, making it easier for the user to understand the current camera shake status. In this case, by leaving the display of the first camera shake pointer 40, it is easier for the user to understand how much the current camera shake status has changed since the first image capture.

[0112] The display control in step S18 is not limited to the above, and may be various display controls that gradually erase past displays of the camera shake pointer 40. For example, the camera control unit 140 may maintain the plot display of the previous camera shake pointer 40, or erase a predetermined number of past plot displays of the camera shake pointer 40 from the previous time (S18). The camera control unit 140 may also perform various fade-out display controls that reduce the degree of highlighting, such as the brightness or saturation of the plot display, or increase the transparency of the plot display, as the camera shake pointer 40 becomes more past. Also, the display of the first camera shake pointer 40 does not necessarily have to be left, and may be erased as appropriate.

[0113] As described above, in this embodiment, the camera control unit 140 controls the liquid crystal monitor 120 during the photographing operation of the composite image so as to gradually erase past blur conditions from the blur conditions included in the displayed camera shake condition screen (S18). This simplifies the display of the camera shake condition screen, making it easier for the user to understand the camera shake condition.

[0114] (Other embodiments) As described above, the first and second embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are appropriately performed. In addition, it is also possible to combine the components described in the first and second embodiments to form new embodiments. Therefore, other embodiments will be exemplified below.

[0115] In the above-mentioned first and second embodiments, the camera shake state screen is exemplified, but the camera shake state screen is not limited to this. Modifications of the camera shake state screen will be described with reference to Figs.

[0116] Fig. 13 shows a first modified example of the camera shake state screen of the digital camera 1. In each of the above embodiments, a camera shake state screen including a preview image 20 has been exemplified. In this embodiment, the camera shake state screen does not have to include the preview image 20, for example, as shown in Fig. 13. In this way, even with a camera shake state screen that does not display the preview image 20, the display of the camera shake pointer 40 is updated sequentially during multiple image captures, making it easier for the user to grasp the camera shake state, as in each of the above embodiments.

[0117] FIG. 14 shows a second modified example of the camera shake status screen of the digital camera 1. In this embodiment, the camera shake status screen may display a camera shake pointer 41 moving according to the camera shake state during image capture. Also, the camera shake status screen of this embodiment may display a trajectory 45 of the camera shake pointer 41 moving during one image capture, for example. For example, the camera control unit 140 sequentially acquires camera shake information during each image capture (S12 to S15), changes the display position of the camera shake pointer 40 in accordance with the change in the camera shake information, and draws the trajectory 45. For example, each time one image capture is completed, the camera control unit 140 plots the camera shake pointer 40 and erases the trajectory 45 for the completed image capture.

[0118] As described above, in this embodiment, the camera control unit 140 may move the camera shake pointer 40 on the scope unit 30 of the camera shake status screen according to the camera shake status during one image capture in the composite image capture operation. The display of such a camera shake status screen also makes it easier for the user to grasp the camera shake status.

[0119] 15 shows a third modified example of the camera shake status screen of the digital camera 1. In the camera shake status screen of this embodiment, an arrow 46 or the like may be displayed between a plurality of camera shake pointers 40 plotted corresponding to a plurality of image captures. For example, when plotting a new camera shake pointer 40 (S14), the camera control unit 140 may display an arrow 46 pointing from the previously plotted camera shake pointer 40 to the current camera shake pointer 40.

[0120] 16(A) and (B) show a fourth modified example of the camera shake state screen of the digital camera 1. In the camera shake state screen of this embodiment, the preview image 20 may be moved sequentially during multiple image captures, as illustrated in FIG. 16(A) and (B). For example, the camera control unit 140 may move the preview image 20 on the camera shake state screen by the amount of camera shake for each image capture, based on the camera shake information (S13) acquired during each image capture. At this time, the plot display (S14) of the new camera shake pointer 40 is performed at a predetermined position such as the center of the screen, and the camera shake pointer 40 and the scope unit 30 that were previously plotted may be moved together with the preview image 20.

[0121] In each of the above embodiments, an example of the scope section 30 on the camera shake state screen has been described, but the scope section 30 is not particularly limited to this. The scope section 30 is not limited to the two concentric regions 31, 32, but may include three or more regions, or may be one region. Furthermore, each of the regions 31, 32 of the scope section 30 is not particularly limited to a circle, but may be a rectangle or various other shapes. Furthermore, the scope section 30 may display various directional references, not particularly limited to crosshairs, or may not particularly display such directional references.

[0122] In each of the above embodiments, an example of the camera shake pointer 40 on the camera shake state screen has been described, but the camera shake pointer 40 is not particularly limited to this. For example, in this embodiment, the number of the camera shake pointer 40 does not need to be displayed. In addition, the shape of the camera shake pointer 40 is not particularly limited to a circular shape, and may be various shapes. For example, the direction in which the camera shake pointer 40 is plotted may be set according to the camera shake in the roll direction.

[0123] In the above-mentioned embodiments, the handheld high-resolution shooting operation has been described as an example of the shooting operation of the composite image, but the shooting operation of the composite image is not particularly limited to this. For example, the shooting operation of the composite image of this embodiment may be a so-called real resolution shooting operation in which the captured images having a positional shift of one pixel unit are the objects of synthesis. For example, the captured images corresponding to the corresponding positions P3, P4, and P5, which are in a positional relationship shifted by pixel pitch W units from the reference position P1 in FIG. 9, and the reference image may be the objects of synthesis. In addition, the image synthesis process at this time may include pixel values ​​of three colors in each pixel in the composite image instead of increasing the number of pixels as in the high-resolution synthesis process (S3). Even in such a case, a high-quality composite image can be obtained by making it easier to suppress camera shake by the camera shake state screen in the shooting display process (S2) as in the case of the high-resolution synthesis process.

[0124] In addition, in this embodiment, the photographing operation of the composite image may be performed with the photographed image having no positional deviation as the composite target. For example, in the photographing operation for image composition such as depth composition or HDR (high dynamic range) composition, the camera shake state screen of the photographing display process (S2) described above may be applied.

[0125] In addition, in each of the above embodiments, an example in which the BIS processing unit 183 performs the image stabilization operation during the composite image shooting operation has been described, but the present invention is not limited to this. For example, the BIS processing unit 183 may not perform the image stabilization operation during the composite image shooting operation. In addition to or instead of the image stabilization operation, the BIS processing unit 183 may perform an operation to shift pixels by a predetermined value, such as 1 / 2 pixel units, in step S17. In addition to or instead of the operation of the BIS processing unit 183, the OIS processing unit 223 may operate. In this case, the camera control unit 140 may acquire image stabilization information through data communication with the OIS processing unit 223.

[0126] In addition, in each of the above-described embodiments, the liquid crystal monitor 120 has been illustrated as an example of a display unit. In this embodiment, the display unit is not limited to the above, and may be various monitors other than a liquid crystal monitor, or may be a viewfinder such as an EVF or various other display devices.

[0127] In the above embodiments, a digital camera with interchangeable lenses has been described as an example of an imaging device, but the imaging device of the present embodiment may be a digital camera that is not particularly an interchangeable lens type. The concept of the present disclosure is also applicable not only to digital cameras, but also to movie cameras, and to electronic devices with various imaging functions such as camera-equipped mobile phones, smartphones, and PCs.

[0128] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed description have been provided for this purpose.

[0129] Therefore, among the components described in the attached drawings and the detailed description, not only are there components essential for solving the problem, but there may also be components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that such non-essential components are described in the attached drawings or the detailed description should not be interpreted as immediately indicating that such non-essential components are essential.

[0130] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0131] The concept of the present disclosure can be applied to electronic devices having an imaging function that includes a function for capturing a composite image (imaging devices such as digital cameras and camcorders, mobile phones, smartphones, etc.). [Explanation of symbols]

[0132] 1. Digital camera 100 Camera body 110 Image Sensor 120 LCD monitor 140 Camera control unit 183 BIS Processing Department 184 Gyro sensor

Claims

1. an image sensor that captures an image of a subject and generates image data; a control unit that controls a photographing operation to generate image data representing a composite image obtained by combining a plurality of photographed images based on image data of the photographed results obtained by the imaging element; A shake detection unit that detects a shake state of the own device; A display unit for displaying information, The control unit controls the display unit to display blur state information including a plurality of blur states detected by the blur detection unit during the plurality of imaging operations by the image sensor in the photographing operation of the composite image, and causes the blur state at each imaging operation among the plurality of imaging operations in the photographing operation of the composite image to be displayed separately in the blur state information. Imaging device.

2. The control unit controls the display unit to display, in the blur state information, discrete pointers arranged one by one according to a blur state during an exposure period at the time of each of the multiple image captures. The imaging device according to claim 1 .

3. The blur state information includes a pointer that is arranged according to a blur state at the time of each of the plurality of image capturing operations. The imaging device according to claim 1 .

4. The blur state information further includes a reference area indicating a reference of the blur state, The control unit causes the display unit to display the blur state information so as to position the pointer in the reference area according to a blur state at the time of each of the plurality of imaging operations. The imaging device according to claim 3 .

5. the blur state information includes a preview image captured before the composite image is captured, The control unit controls the display unit to move the preview image in accordance with the blur state during the composite image capturing operation. The imaging device according to claim 1 .

6. The control unit controls the display unit so that, during the photographing operation of the composite image, past blur states are gradually erased from the displayed blur state information. The imaging device according to claim 1 .

7. The control unit combines a plurality of captured images having a mutual positional shift based on image data of the plurality of captured images, and generates image data representing the combined image. The imaging device according to claim 1 .

8. an image blur correction unit that performs image blur correction by driving the image sensor based on a detection result of the blur detection unit; The control unit acquires a blur state after correction by the image blur correction unit, and causes the display unit to display the blur state information according to the acquired blur state. The imaging device according to claim 1 .

Images