Imaging device

Abstract

To provide an imaging device capable of performing suitable imaging. [Solution] The imaging device comprises an imaging element that captures an image of a subject and outputs image data, a display unit that displays an image generated based on the image data output from the imaging element, and an image processing unit that generates a composite image based on multiple images captured while changing the imaging direction, and the display unit displays the composite image generated by the image processing unit along with the image generated based on the image data output from the imaging element, and displays a warning message to the user based on the direction in which the imaging direction is changed.

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] Conventionally, a plurality of images are captured while panning a digital camera or the like, and a panoramic image is generated by sequentially connecting these images. For example, Patent Document 1 discloses a technique for performing panoramic synthesis processing using frame images of a moving image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, when the amount of movement due to panning is too large and the images cannot be connected to each other, the images have to be recaptured from the beginning.

[0005] An object of the present embodiment is to provide an imaging device capable of performing suitable imaging.

Means for Solving the Problems

[0006] One aspect of the imaging device exemplifying the present invention is an imaging device including: an imaging element that captures a subject and outputs image data; a display unit that displays an image generated based on the image data output from the imaging element; and an image processing unit that generates a composite image based on a plurality of images captured while changing an imaging direction. The display unit displays a direction image indicating a direction in which the imaging device is moved together with an image generated based on the image data output from the imaging element and a composite image generated by the image processing unit. When the imaging device is moved in a direction different from the direction indicated by the direction image,Instead of the direction image, an object image that can detect the amount of movement and is connected to the composite image is imaged at a position where it can be imaged. Displays information indicating the direction.

Advantages of the Invention

[0007] According to the present embodiment, an imaging device capable of performing suitable imaging can be provided.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0009] FIG. 1 is a block diagram showing the configuration of a digital camera according to an embodiment of the present invention.

[0010] The digital camera of the present embodiment includes an imaging optical system 11, an imaging element 12, an AFE 13, a CPU 14, a memory 15, a monitor 16, a recording interface (I / F) 17, an operation member 19, a release button 20, a touch panel 21, and a bus. The CPU 14, the memory 15, the monitor 16, and the recording I / F 17 are respectively connected via the bus so as to be able to transmit information. Further, the operation member 19, the release button 20, and the touch panel 21 are respectively connected to the CPU 14. Note that the digital camera of the present embodiment has a panorama mode for generating one panorama image from a plurality of images obtained by dividing the subject field for imaging as one function of the shooting mode.

[0011] The imaging optical system 11 is composed of a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, in FIG. 1, the imaging optical system 11 is illustrated as a single lens.

[0012] The image sensor 12 is a device that images the subject field imaged by the light beam that has passed through the imaging optical system 11. The output of this image sensor 12 is input to the AFE 13. Note that the image sensor 12 of this embodiment may be a sequential scanning type solid-state image sensor (such as a CCD) or an XY address type solid-state image sensor (such as a CMOS).

[0013] In addition, a plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. Color filters of red (R), green (G), and blue (B) are arranged on each light receiving element of the image sensor 12 according to a known Bayer array. Therefore, each light receiving element of the image sensor 12 outputs an image signal corresponding to each color by color separation with the color filter. Thereby, the image sensor 12 can acquire a color image.

[0014] Here, in the shooting mode of the digital camera, the image sensor 12 images a recorded image (this image) in response to a full press operation of the release button 20. Also, the image sensor 12 in the shooting mode images a low-resolution image (through image) for composition confirmation at a predetermined time interval even during shooting standby. The data of this through image is output from the image sensor 12 by subsampled reading. The data of the through image is used for image display on the monitor 16 and various arithmetic processes by the CPU 14. Also, when the shooting mode is the video mode or the panorama mode, it is assumed that the image sensor 12 images a video at a predetermined frame rate (such as 30 fps or 60 fps) as this image.

[0015] AFE13 is an analog front-end circuit that performs analog signal processing on the output of the imaging device 12. This AFE13 performs correlated double sampling, adjustment of the gain of the image signal, and A / D conversion of the image signal. The output of AFE13 is connected to the CPU14. In this embodiment, the imaging device 12 and AFE13 constitute an imaging unit.

[0016] CPU14 is a processor that comprehensively controls the operation of the digital camera. By executing a control program, CPU14 performs known autofocus (AF) control, automatic exposure (AE) calculation, auto white balance calculation, etc. based on the data of the through-image by the phase difference detection method or the contrast detection method. In addition, CPU14 also performs display processing of the setting screen, the captured main image, and the through-image. Furthermore, CPU14 operates as an image processing unit 22 that performs image processing on the digital image signal output from AFE13 by executing an image processing program stored in the memory 15.

[0017] The image processing unit 22 performs various image processes (for example, color interpolation processing, tone conversion processing, edge enhancement processing, white balance adjustment, etc.) on the digital image signal output from AFE13. Also, as will be described later, when the shooting mode is set to the above panoramic mode, the image processing unit 22 in this embodiment stitches together the frames (target images) captured as a video in chronological order based on known panoramic synthesis processing, and performs panoramic synthesis to generate a panoramic image (synthesized image).

[0018] The memory 15 stores the image data captured by the digital camera, the control program and the image processing program executed by the CPU14, etc. A non-volatile semiconductor memory or the like can be used for the memory 15.

[0019] The monitor 16 is a monitor such as a liquid crystal monitor, and displays various setting screens and images in response to instructions from the CPU14.

[0020] The recording I / F 17 is formed with a connector for connecting the storage medium 18. Then, the recording I / F 17 executes data writing / reading with respect to the storage medium 18 connected to the connector. The above storage medium 18 is composed of a hard disk, a memory card incorporating a semiconductor memory, etc. Note that in FIG. 1, a memory card is shown as an example of the storage medium 18.

[0021] The operation member 19 is composed of, for example, a command dial, a cross-shaped cursor key, a determination button, etc. Then, the operation member 19 receives various inputs of the digital camera from the user. For example, the operation member 19 is used for switching the operation mode of the digital camera, input operations on the setting screen, etc.

[0022] The release button 20 receives from the user an instruction input for starting the AF operation before imaging by a half-press operation and an instruction input for starting the imaging operation by a full-press operation.

[0023] The touch panel 21 detects the position of a stylus (or fingertip, etc.) that has touched the panel surface, and receives an instruction input from the user by outputting the detected position information to the CPU 14. Note that the configuration of the touch panel 21 may be either a capacitive type or a pressure-sensitive type. The touch panel 21 of the present embodiment is composed of a transparent panel having the same shape as the monitor 16, and is laminated and arranged on the entire surface of the monitor 16. The user visually recognizes the GUI-form buttons displayed on the monitor 16, and inputs instructions to the CPU 14 via the touch panel 21.

[0024] Next, with reference to the flowchart of FIG. 2, the operation of image processing by the digital camera of the present embodiment will be described. In the following, it is assumed that the shooting mode has been preset to the panorama mode by the user via the operation member 19.

[0025] When the user presses the power button of the operation member 19, the CPU 14 turns on the digital camera and initializes each part. When in the imaging standby state in the shooting mode, the CPU 14 starts imaging the through-image with the imaging device 12, and for example, displays the through-image over the entire display range 30 of the monitor 16 (Fig. 3a). In response to the imaging instruction of the full press operation of the release button 20 by the user, the CPU 14 starts video imaging of the subject in the panorama mode with the imaging device 12. At the same time, the user starts the pan operation of the digital camera. In this embodiment, although the user is described as panning the digital camera in the horizontal direction, the same applies to the case of the vertical direction and the description thereof is omitted.

[0026] Also, when starting the video imaging in the panorama mode, the CPU 14 switches the display of the monitor 16 from the display shown in Fig. 3a to the display shown in Fig. 3b. That is, as shown in Fig. 3b, the CPU 14 reduces and displays the frame of the video captured by video imaging in real time in the display area 40 arranged on the right side of the display range 30 of the monitor 16 instead of the through-image. On the other hand, the CPU 14 displays a part of the panorama image sequentially generated by the panorama synthesis process described later in real time in the display area 50 arranged on the left side of the display range 30. However, as shown in Fig. 3b, since there is no panorama image subjected to the panorama synthesis process at the start of video imaging, nothing is displayed in the display area 50.

[0027] Step S101: The CPU 14 reads the first frame output from the imaging device 12 via the AEF 13 and sets it as the composite image serving as the reference for the panorama synthesis process. Then, the CPU 14 reads the second frame and sets it as the target image to be joined to the composite image.

[0028] Step S102: The image processing unit 22 extracts image feature amounts such as the edge amount and color information of the target image and the composite image by known image processing. Based on the extracted image feature amounts, the image processing unit 22 determines whether the movement amount and the pan direction of the target image with respect to the composite image can be detected by correlation processing between the target image and the composite image, etc., due to the pan operation of the user's digital camera. If the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S103 (YES side), and if the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S106 (NO side).

[0029] Step S103: The image processing unit 22 performs panorama synthesis processing to connect the image area newly captured by the target image to the composite image based on the movement amount and the pan direction detected in step S102.

[0030] Step S104: The image processing unit 22 simultaneously displays the obtained pan direction on the monitor 16 with the arrow 60, and at the same time, the composite image panoramically synthesized in step S103 is displayed in real time while sliding in the direction opposite to the pan direction in the display area 50 (Fig. 4a). Here, Fig. 4a shows, as an example, the display state of the monitor 16 immediately after the start of the panorama synthesis processing.

[0031] In addition, in the case where the target image is displaced vertically or obliquely with respect to the composite image, when the CPU 14 displays the composite image in which the target image is panoramically synthesized, it is preferable to trim and display the image area that protrudes from the display area 50. Alternatively, the CPU 14 may display the composite image in a reduced size so as to also display the protruding image area.

[0032] Step S105: The CPU 14 determines whether it has received an instruction to end the imaging of the full-press operation of the release button 20 again by the user. When the CPU 14 receives the imaging end instruction, it ends the panoramic mode video imaging. The image processing unit 22 generates a composite image as a panoramic image file and records it in the memory 15 or the storage medium 18. The CPU 14 ends a series of processes. In this embodiment, it is preferable that the CPU 14 trims the vertical size of the composite image with the vertical width of the image of the first frame, for example, to generate a panoramic image file. Also, when the user pans the digital camera in the vertical direction, it is preferable that the CPU 14 trims the horizontal size of the composite image with the horizontal width of the image of the first frame, for example.

[0033] On the other hand, when the CPU 14 has not received the imaging end instruction, it shifts to step S101 (NO side) and reads the next captured frame as the target image. The CPU 14 performs the processes of steps S101 to S104 on the target image.

[0034] Step S106: When the image processing unit 22 cannot detect the amount of movement and the pan direction of the target image with respect to the composite image, such as when the speed of the panning operation of the digital camera by the user is too fast and the amount of movement is too large, or when the posture of the digital camera changes too much, it notifies the user of the detection failure. That is, the image processing unit 22 displays a warning display as shown in FIG. 4b on the monitor 16 to notify the user. For example, in order to image a target image whose amount of movement is detectable and is connected to the composite image, the image processing unit 22 displays and instructs the direction in which the digital camera should be panned to the user with an arrow 70 instead of an arrow 60. Also, the image processing unit 22 may display a warning such as a symbol 71 or text 72 on the monitor 16 together with the arrow 70. Alternatively, as another example of the warning display, the image processing unit 22 may indicate the area where the target image should be connected in the composite image as a shaded area 73.

[0035] Note that when the user stops the digital camera without panning it for a certain period of time, or pans it in a direction different from the arrow 60, etc., it is preferable for the CPU 14 to display the above warning to the user.

[0036] Step S107: The image processing unit 22 temporarily stores the image feature amount of the composite image extracted in step S102, for example, in an internal memory (not shown) of the CPU 14. Note that the image feature amount of the composite image to be stored may be only the image feature amount of an image area having the same size as the target image or about 70 to 80% of the size, starting from the edge of the composite image where the target image is to be joined. Thereby, the memory capacity can be reduced and the circuit scale can be miniaturized.

[0037] Step S108: While causing the user to refer to the frame of the display area 40, based on the warning display, the image processing unit 22 pans in the direction of the arrow 70, for example, and reads the next frame (for example, the third frame) captured by the imaging device 12 as a new target image.

[0038] Step S109: The image processing unit 22 extracts the image feature amount of the target image read in step S108. Based on the extracted image feature amount of the target image and the image feature amount of the composite image stored in the internal memory (not shown), the image processing unit 22 determines whether the movement amount and the pan direction of the target image with respect to the composite processing can be detected. When the movement amount and the pan direction can be detected, the image processing unit 22 proceeds to step S110 (YES side). On the other hand, when the movement amount and the pan direction cannot be detected, the image processing unit 22 proceeds to step S108 (NO side). The image processing unit 22 performs the processing of steps S108 to S109 until the movement amount and the pan direction of the target image with respect to the composite processing are detected.

[0039] Step S110: The image processing unit 22 cancels the warning display shown in FIG. 4b and switches to the display shown in FIG. 4c. The image processing unit 22 proceeds to step S103 and resumes the panorama synthesis process of connecting the target image newly read in step S108 to the composite image.

[0040] As described above, in this embodiment, even if the digital camera pans significantly with respect to the composite image, by displaying the video and the composite image that has been panoramically synthesized so far in real time, it is possible to easily determine how far the panorama synthesis has been performed and where the panorama synthesis has failed, and it is possible to easily return the imaging direction of the digital camera to the point where the panorama synthesis has failed. As a result, it is possible to generate a panorama image with high accuracy without having to perform imaging in the panorama mode again from the beginning. 《Supplementary Matters of the Embodiment》 (1) In the above embodiment, an example in which the processing of the image processing unit 22 is realized software-wise in the CPU 14 has been described, but the processing may be realized hardware-wise using an ASIC.

[0041] (2) In the above embodiment, the image processing unit 22 detects the movement amount and the pan direction of the target image based on the image feature amounts of the entire image regions of the target image and the composite image, but the present invention is not limited to this. For example, a glue margin portion of a predetermined width is set around the target image and the composite image, respectively, and the image processing unit 22 may detect the movement amount and the pan direction with respect to the composite process of the target image based on the image feature amounts within the image regions other than the glue margin portions.

[0042] Also, if the digital camera is equipped with an acceleration sensor, an electronic gyro, etc., the image processing unit 22 may connect the target image to the composite image based on the movement amount and the pan direction detected by the image processing unit 22 and the movement amount and the pan direction detected by the acceleration sensor, etc. This enables more accurate panorama synthesis.

[0043] (3) In the above embodiment, when the user starts panning the digital camera, the CPU 14 displays the video frame in real time in the right display area 40 and slides the synthesized image subjected to panorama synthesis processing in the left display area 50 within the display range 30 of the monitor 16. However, the present invention is not limited to this. For example, as shown in FIG. 5a, the CPU 14 may display the video frame in real time in the upper display area 40' above the center of the monitor 16 and display the synthesized image in real time in the lower display area 50' according to the progress of the panorama synthesis processing. FIG. 5b shows an example of a warning display when the detection of the movement amount and the pan direction of the target image with respect to the synthesized image fails.

[0044] Before starting imaging in the panorama mode, the CPU 14 may receive from the user a designation of an imaging range for panorama imaging such as 180 degrees or 360 degrees via the operation member 19 or the like, and set the size of the display area 50' according to the imaging range. Thereby, it is possible for the user to easily recognize up to which range the imaging has been completed. Further, when the imaging is completed up to the set imaging range, the CPU 14 may automatically end the imaging in the panorama mode without the release button 20 being fully pressed again.

[0045] Furthermore, while the panorama synthesis processing is being performed smoothly, the CPU 14 may perform full-screen display of the video frame in the display range 30 shown in FIG. 3a, and may switch to the warning display shown in FIG. 4b or FIG. 5b when the detection of the movement amount and the pan direction of the target image with respect to the synthesis processing fails.

[0046] Also, FIG. 6 shows an example of the display on the monitor 16 when the user pans the digital camera in the vertical direction to image a tower or a high-rise building.

[0047] (4) In the above embodiment, in step S105, when the user presses the release button 20 again fully, the CPU 14 is set to end the imaging in the panorama mode. However, the present invention is not limited to this. For example, at any time during the imaging in the panorama mode, the user can fully press the release button 20, and it is preferable that the CPU 14 can accept the interrupt for the end of the imaging.

[0048] (5) In the above embodiment, it is assumed that video imaging is performed and panorama synthesis is carried out. However, the present invention is not limited to this, and it can also be applied to the case of continuous shooting or imaging still images one by one for panorama synthesis.

[0049] Through the above detailed description, the characteristic points and advantages of the embodiment will become clear. This is intended to mean that the scope of the claims extends to the characteristic points and advantages of the above-described embodiment without departing from its spirit and scope of rights. Also, those having ordinary knowledge in the relevant technical field should be able to easily conceive of any improvements and modifications, and there is no intention to limit the scope of the inventive embodiments to those described above. It is also possible to use appropriate improvements and equivalents included in the scope disclosed in the embodiment.

Description of Reference Numerals

[0050] 11... Imaging optical system, 12... Image sensor, 13... AFE, 14... CPU, 15... Memory, 16... Monitor, 17... Recording I / F, 18... Storage medium, 19... Operating member, 20... Release button, 21... Touch panel, 22... Image processing unit

Claims

1. An imaging device, an image sensor that captures a subject and outputs image data, a display unit that displays an image generated based on the image data output from the image sensor, and an image processing unit that generates a composite image based on a plurality of the images generated by imaging while changing the imaging direction, wherein the display unit displays a direction image indicating a direction in which the imaging device is moved, together with an image generated based on the image data output from the image sensor and the composite image generated by the image processing unit; when the imaging device is moved in a direction different from the direction indicated by the direction image, instead of the direction image, information indicating a direction of a position where a target image whose movement amount can be detected and that can be connected to the composite image can be imaged is displayed. An imaging device.

2. The imaging device according to claim 1, wherein the information indicating the direction of a position where a target image whose movement amount can be detected and that can be connected to the composite image can be imaged is a display indicating a direction in which a user moves the imaging device.

3. The imaging device according to claim 1, wherein the information indicating the direction of a position where a target image whose movement amount can be detected and that can be connected to the composite image can be imaged is a display indicating an area in the composite image where the image captured by the image sensor should be connected.

4. The imaging device according to any one of claims 1 to 3, wherein the information indicating the direction of a position where a target image whose movement amount can be detected and that can be connected to the composite image can be imaged is a display by text.

5. The imaging device according to any one of claims 1 to 3, wherein the information indicating the direction of a position where a target image whose movement amount can be detected and that can be connected to the composite image can be imaged is a display by symbol.

Images