Imaging device, imaging method, program, and recording medium

Abstract

To provide an imaging apparatus that changes the interval of imaging in continuous imaging and the width for cut-out and composition of images according to the distance to a subject, and thereby can reduce shift in composition of the subject located frontward (at a closer range) without reducing the image quality of the entire images.SOLUTION: An imaging apparatus has: determination means that determines an imaging interval; imaging means that picks up a plurality of images at the imaging interval; acquisition means that acquires information on the distance to a subject from the plurality of images picked up by the imaging means; and composition means that composes the plurality of images. The determination means determines the imaging interval based on the distance information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an imaging device, and more particularly to an imaging device capable of capturing panoramic images. [Background technology]

[0002] A method has been proposed in the past in which still images are captured continuously while panning an imaging device, and the continuously captured still images are then combined to generate a panoramic image (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-28764 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a panoramic image generated by combining still images captured by panning imaging, a composite shift may occur at the joining portion of images of a subject in the foreground (closer) due to the influence of parallax.

[0005] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an imaging device that can reduce composite misalignment of a foreground (closest) subject in a panoramic image due to parallax. [Means for solving the problem]

[0006] Image pickup device according to the present invention One of a determination means for determining an imaging interval; During panning an imaging means for capturing a plurality of images at the imaging intervals; an acquisition means for acquiring distance information of a subject from the plurality of images captured by the imaging means; and a synthesis means for synthesizing the plurality of images; a notification means for notifying the user of the panning speed; and the determining means determines the distance information. is equal to or greater than a predetermined first distance, the imaging interval is determined to be a first imaging interval, and when the distance information is less than the first predetermined distance, the imaging interval is determined to be a second imaging interval shorter than the first imaging interval, and the notification means performs the notification when the distance information is less than a predetermined second distance.It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an imaging device that can reduce composite misalignment in a foreground (closest) subject without degrading the image quality of the entire image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] 10A and 10B are diagrams illustrating panning directions and image clipping regions according to an embodiment of the present invention. [Figure 3] 3 is a diagram for explaining the flow of processing for panoramic imaging and the correspondence relationship between image data in the first embodiment of the present invention. FIG. [Figure 4] 4 is a flowchart illustrating the operation of panoramic imaging in the first embodiment of the present invention. [Figure 5] 10A and 10B are diagrams for explaining the flow of processing for panoramic imaging, the correspondence relationship with image data, and message display timing in the second embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating the operation of panoramic imaging in the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0010] In the following description, the imaging device is a digital (still) camera, but the imaging device also includes electronic devices with imaging capabilities, such as movie cameras, smartphones with cameras, tablet computers with cameras, in-vehicle cameras, and network cameras.

[0011] (First embodiment) FIG. 1 is a block diagram illustrating the configuration of an imaging device according to this embodiment.

[0012] As shown in FIG. 1, the imaging device of this embodiment is mainly composed of a digital camera 100 and an interchangeable lens unit 300 that is detachable from the digital camera 100.

[0013] In the digital camera 100, the shutter 20 controls the time (exposure time) that an optical image is incident on the imaging unit 22, which will be described later. The imaging unit 22 includes an imaging element configured with a CCD, CMOS element, or the like that converts the optical image into an electrical signal. The imaging unit 22 functions as an imaging means that captures an image of a subject.

[0014] The imaging unit 22 also incorporates an A / D conversion processing function and an AF evaluation value detection unit 23. The AF evaluation value detection unit 23 calculates an AF evaluation value based on contrast information obtained from the digital image signal and a phase difference obtained from the parallax image, and outputs the calculated AF evaluation value to the system control unit 50.

[0015] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the image data output from the imaging unit 22 or image data from the memory control unit 15 .

[0016] Furthermore, the image processing unit 24 can acquire distance information of the subject based on the AF evaluation value. That is, the distance to the subject can be acquired by detecting the phase difference between the two input parallax images, and distance information from the imaging unit to the subject can be acquired for each pixel. A sensor such as a TOF (Time of Flight) sensor may be used to acquire the distance information.

[0017] Furthermore, the image processing unit 24 performs predetermined calculations using the captured image data to obtain exposure control information, and the system control unit 50 performs exposure control and focus adjustment control based on the distance information and exposure control information, thereby performing TTL (through-the-lens) AE (auto exposure) processing, EF (auto flash) processing, etc.

[0018] The image processing unit 24 also performs AF (autofocus) processing based on distance information, using the output of the AF evaluation value detection unit 23 provided in the imaging unit 22. The image processing unit 24 also performs TTL-type AWB (auto white balance) processing by performing predetermined arithmetic processing using the captured image data. The image processing unit 24 also performs processing to combine multiple images obtained by capturing images multiple times, thereby generating a panoramic image.

[0019] The detection unit 26 includes a gyro sensor and other sensors, and acquires angular velocity information, attitude information, and the like of the digital camera 100. The angular velocity information includes information on the angular velocity and angular acceleration when panning image capture is performed by the digital camera 100. The attitude information includes information such as the tilt of the digital camera 100 with respect to the horizontal direction.

[0020] The output data of the imaging unit 22 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data acquired and A / D converted by the imaging unit 22, as well as image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 also serves as a memory for displaying images (video memory).

[0021] The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies it to a display unit 28 such as an LCD.

[0022] The digital signal that is first A / D converted by the imaging unit 22 and stored in the memory 32 is converted to analog by the D / A converter 13, and then sequentially transferred to the display unit 28 to display a through image (live view image), so that the display unit functions as an electronic viewfinder.

[0023] The nonvolatile memory 52 is an electrically erasable and recordable memory, such as a flash memory. The nonvolatile memory 52 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are computer programs for executing various flowcharts described later in this embodiment.

[0024] The system control unit 50 has a built-in CPU as a computer, and functions as a control unit that controls the entire digital camera 100. By executing the computer program recorded in the nonvolatile memory 52 described above, each process in the flowchart of this embodiment, which will be described later, is realized.

[0025] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0026] The memory 56 is a system memory using RAM or the like, and constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 52, etc. are developed in the memory 56.

[0027] The mode switch 60, shutter button 61 (including a first shutter switch 62 and a second shutter switch 63), operation unit 70, and touch panel operation unit 72 function as operation means for inputting various operation instructions to the system control unit 50.

[0028] The mode switch 60 switches the operation mode of the system control unit 50 to one of a still image capturing mode, a moving image capturing mode, a playback mode, and the like.

[0029] The still image capture mode includes an auto capture mode, an auto scene determination mode, a manual mode, various scene modes that set the capture mode for each capture scene, a program AE mode, a custom mode, a panoramic capture mode, etc. The mode switch 60 allows direct switching to one of these modes included in the still image capture mode.

[0030] Alternatively, after switching to the still image capture mode with the mode switch 60, another operating member may be used to switch to one of these modes included in the still image capture mode. Similarly, the video capture mode may include multiple modes.

[0031] The first shutter switch 62 is turned on when the shutter button 61 provided on the digital camera 100 is pressed halfway (instruction to prepare for image capture) and generates a first shutter switch signal SW1.

[0032] The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (auto flash dimming) processing.

[0033] The second shutter switch 63 is turned on when the shutter button 61 is fully pressed (image capture instruction) and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of still image capture processing operations, reads signals from the image capture unit 22, writes image data to the recording medium 200, etc.

[0034] The various operating members of the operating unit 70 and the touch panel operating unit 72 are assigned appropriate functions for each situation by selecting and operating various functional icons displayed on the display unit 28. A mouse or the like may be connectable to the digital camera 100, and the operating unit 70 includes such a mouse or the like.

[0035] The function buttons include, for example, an end button, a back button, an image forward button, a jump button, a narrow down button, an attribute change button, a video capture button, etc. For example, when the menu button is pressed, a pull-down menu screen in which various settings can be made is displayed on the display unit 28.

[0036] The user can make various settings by selecting menus on the menu screen displayed on the display unit 28 using the four-way buttons (up, down, left, right) and the SET button.

[0037] The power supply control unit 41 is made up of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery capacity.

[0038] Furthermore, the power supply control unit 41 controls the DC-DC converter based on the detection result and instructions from the system control unit 50, and supplies the necessary voltage to each unit including the recording medium 200 for the necessary period.

[0039] The power supply unit 40 is made up of a primary battery, a secondary battery, an AC adapter, etc. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is made up of a semiconductor memory, a magnetic disk, etc.

[0040] Connector 80 is a contact point for electrically connecting the camera body of digital camera 100 to lens unit 300. Connector 80 communicates control signals, status signals, data signals, etc. between digital camera 100 and lens unit 300, and also has the function of supplying power to lens unit 300.

[0041] Next, the configuration of the lens unit 300 will be described.

[0042] In the lens unit 300, the imaging lens 301 is made up of multiple lenses, including a focus lens that adjusts the focus by moving in the optical axis direction and a zoom lens that performs zooming. Also, the diaphragm 302 adjusts the amount of light entering the imaging unit 22 of the digital camera 100 by controlling the diaphragm opening.

[0043] The lens control unit 303 has a built-in CPU as a computer, and controls the entire lens unit 300 in response to control signals from the system control unit 50 of the digital camera 100, as well as driving and controlling the focus lens, zoom lens, and aperture 302 of the imaging lens 301. It also has memory for storing operational constants and variables, computer programs, etc. It also has non-volatile memory for storing identification information such as a number unique to the lens unit 300, management information, functional information such as the maximum aperture value, minimum aperture value, focal length, and amount of aberration such as distortion, current and past setting values, etc.

[0044] The connector 304 is a contact point for electrically connecting the lens unit 300 to the camera body of the digital camera 100. The connector 304 communicates control signals, status signals, data signals, etc. between the digital camera 100 and the lens unit 300, and also has the function of receiving power supply voltage or current from the digital camera 100.

[0045] Next, the method for capturing a panning panoramic image and the method for combining the images will be described in detail with reference to Figs. 2 and 3. The images captured by panning are combined by cutting out a portion of each image. Fig. 2 is a diagram for explaining the panning direction and the image cutout area in this embodiment. Fig. 3 is a diagram for explaining the correspondence between the process flow of panoramic imaging and image data in this embodiment.

[0046] First, the cropping area (image area) of captured image data will be explained using Figure 2. In Figure 2, the widely spaced hatched areas indicate the cropping area of ​​the image data, the narrowly spaced hatched areas indicate the distance information evaluation area used to determine the imaging interval until the next image capture, and the solid black arrows indicate the panning direction. If the panning speed is constant, a wide imaging interval also means a long time between capturing two adjacent images. Figure 2(a) shows the effective image area of ​​the imaging unit 22, where W is the number of effective pixels in the horizontal direction and H is the number of effective pixels in the vertical direction. Figure 2(b) shows the cropping area of ​​the image data, where Wcrop is the horizontal cropping area width (number of pixels) and Hcrop is the vertical cropping area width (number of pixels). Figure 2(c) shows the cropping area and distance information evaluation area of ​​the image when panning to the right, (d) to the left, (e) to the top, and (f) to the bottom. When panning right or left, image data is cropped horizontally such that W>Wcrop, and image data is cropped vertically such that H=Hcrop. When panning up or down, image data is cropped horizontally such that W=Wcrop, and image data is cropped vertically such that H>Hcrop. In this embodiment, the imaging interval for continuous image capture is switched according to the distance of the subject included in the distance information evaluation area of ​​the image, and the width of the crop area for the captured image data is switched according to the imaging interval at the time of image capture. Specifically, when panning horizontally, the width of the crop area for the image is set to Wcrop1>Wcrop2, and when the imaging interval for continuous image capture is a first imaging interval that is long, the image is cropped using the wide width Wcrop1. When the imaging interval for continuous image capture is a second imaging interval that is short, the image is cropped using the narrow width Wcrop2. Furthermore, when panning in the vertical direction, the width of the cropping area of ​​the image is set to Hcrop1>Hcrop2, and when the continuous imaging interval is a first imaging interval that is long, the image is cropped to the wide width Hcrop1. When the continuous imaging interval is a second imaging interval that is short, the image is cropped to the narrow width Hcrop2.

[0047] The distance information evaluation area is set to include the area in the panning direction and outside the image cutout area. That is, when panning right as shown in FIG. 2(c), the distance information evaluation area includes the right side of the image cutout area. When panning left as shown in FIG. 2(d), the distance information evaluation area includes the left side of the image cutout area. When panning upward as shown in FIG. 2(e), the distance information evaluation area includes the top side of the image cutout area. When panning in the direction shown in FIG. 2(f), the distance information evaluation area includes the bottom side of the image cutout area. By setting the distance information evaluation area according to the panning direction in this way, the imaging interval until the next imaging can be determined based on the distance information of the subject included in the cutout area of ​​the image data for the next imaging. By changing the imaging interval according to the distance information of the subject in this way, in areas where a foreground (closer) subject is present, which is susceptible to the influence of parallax, images captured at a short imaging interval are cut out and combined into narrower images. In areas where a foreground (closer) subject is not present, images captured at a long imaging interval are cut out and combined into wider images. By doing so, it is possible to reduce the composite shift in the foreground (closest) subject without degrading the image quality of the entire image.

[0048] Next, the image synthesis process will be described. The system control unit 50 reads from the memory 32 image data that has been captured, clipped, and saved. Next, the system control unit 50 detects misalignment between the captured images. The detection of misalignment is performed by dividing the image into small blocks of any size, calculating corresponding points for each small block that minimize the luminance SAD (Sum of Absolute Difference), and calculating a motion vector. Alternatively, angular velocity information, position information, or orientation information detected by the detection unit 26 may be used. Next, the system control unit 50 calculates mapping parameters from the calculated motion vectors to correct for misalignment, and performs weighted addition or the like on overlapping portions between the images to synthesize them.

[0049] 3 is a diagram illustrating the correspondence between the processing flow and image data in this embodiment. In FIG. 3, the dotted hatched area is a schematic representation of the area of ​​the subject that is closer (closer) than a predetermined distance. The broadly hatched area indicates a cutout area of ​​the image data, and the narrowly hatched area indicates a distance information evaluation area for determining the imaging interval until the next imaging.

[0050] Figure 3(a) shows the location where the second shutter switch SW2 is pressed. The user sets the field of view at one end of the panoramic image to be synthesized. Figures 3(b) to 3(h) schematically show the state in which continuous image capturing is performed while panning the digital camera 100 toward the other end of the panoramic image to be synthesized.

[0051] In Figure 3(a), there is no subject (area hatched with dots) closer than a certain distance to the distance information evaluation area (area hatched with narrow diagonal lines), so the imaging interval between Figures 3(a) and (b) is set to a long imaging interval (N msec). In addition, the captured image (1) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0052] In Figure 3(b), the distance information evaluation area (the area hatched with narrow diagonal lines) contains a subject (the area hatched with dots) that is closer than a certain distance (closer), so the imaging interval between Figures 3(b) and (c) is set to a short imaging interval (M msec). However, the imaging interval between Figures 3(a) and (b) when image (2) was captured is long (N msec), so the captured image (2) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0053] In Figures 3(c) to (e), the distance information evaluation area (the narrow diagonal hatched area) contains a subject (the dotted hatched area) that is closer than a certain distance. Therefore, the imaging intervals between Figures 3(c) and 3(d) and between Figures 3(d) and 3(e) are short (Mmsec). In addition, captured images (3) to (5) are cropped with a narrow width Wcrop2 and used for panoramic composition.

[0054] In Figure 3(f), there is no subject (area hatched with dots) closer than a certain distance in the distance information evaluation area (area hatched with narrow diagonal lines), so the imaging interval between (f) and (g) is set to a long imaging interval (N msec). However, the imaging interval between Figure 3(e) and (f) when image (6) was captured is short (M msec), so the captured image (6) is cropped with a narrow width Wcrop2 and used for panorama composition.

[0055] In Figure 3(g), since there is no subject (area hatched with dots) closer than a certain distance to the distance information evaluation area (area hatched with narrow diagonal lines), the imaging interval between (g) and (h) is set to a long imaging interval (N msec). In addition, the captured image (7) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0056] Figure 3(h) shows the state where the digital camera 100 has panned to the set angle and continuous image capture has ended. In Figure 3(h), the image capture interval between Figures 3(g) and 3(h) when image (8) was captured is long (N msec), so the captured image (8) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0057] FIG. 3(i) shows a panoramic image synthesized from a plurality of captured images (still images).

[0058] In this way, images of areas where foreground (close) subjects exist that are susceptible to the effects of parallax are captured at narrow imaging intervals, and images are cut out and combined over a narrow width. On the other hand, images of areas where foreground (close) subjects do not exist are captured at wide imaging intervals, and images are cut out and combined over a wide width. In this way, it is possible to reduce composite misalignment in foreground (close) subjects without degrading the image quality of the entire image.

[0059] FIG. 4 is a flowchart illustrating the flow of processing in this embodiment.

[0060] 4 is executed when the user selects the panoramic imaging mode with the mode selector switch 60. The processing of each step shown in Fig. 4 is realized by the system control unit 50 loading a predetermined program read from the nonvolatile memory 56 into the system memory 52 and controlling the operation and processing of each unit constituting the digital camera 100.

[0061] In step S101, the system control unit 50 determines whether or not the panning condition setting button has been pressed by operating the operation unit 70. The panning condition setting button is one of the function buttons implemented by the operation unit 70. If the panning condition setting button has been pressed (YES in S101), the system control unit 50 proceeds to step S102, and if the panning condition setting button has not been pressed (NO in S101), the system control unit 50 proceeds to step S103.

[0062] In step S102, the system control unit 50 accepts a change in imaging conditions (such as panning direction) for panning imaging by the user, stores the accepted conditions in the nonvolatile memory 56, and then proceeds to step S103. Note that if the panning condition setting button has not been pressed and panning imaging is being performed for the first time, the default setting conditions are applied.

[0063] In step S103, the system control unit 50 determines whether the first shutter switch 62 of the shutter button 61 is turned on (whether the first shutter switch signal SW1 is generated). If the first shutter switch 62 is turned on (YES in S103), the system control unit 50 proceeds to step S104, and if the first shutter switch 62 is not turned on (NO in S103), the system control unit 50 returns to step S101.

[0064] In step S104, the system control unit 50 performs AE processing and AF processing (autofocus processing). In the following step S105, the system control unit 50 determines the imaging conditions (exposure, imaging sensitivity, WB, etc.) based on the AE processing result of step S104. Note that in the subsequent step S108, imaging is performed with the imaging lens 103 fixed to the lens position when the subject was focused in the AF processing of step S104.

[0065] Next, in step S106, the system control unit 50 determines whether the second shutter switch 63 is turned on (whether the second shutter switch signal SW2 is generated). If the second shutter switch 63 is not turned on (NO in S106), the system control unit 50 proceeds to step S107, and if the second shutter switch 63 is turned on (YES in S106), the system control unit 50 proceeds to step S108.

[0066] In step S107, the system control unit 50 determines whether the first shutter switch 62 remains on. If the first shutter switch 62 remains on (YES in S107), the system control unit 50 returns to step S106 and repeats this process until the second shutter switch 63 is turned on in step S106. If the first shutter switch 62 is off (NO in S107), the system control unit 50 returns to step S101.

[0067] In step S108, the system control unit 50 controls each unit of the digital camera 100 to perform imaging processing (capturing one image). At this time, parallax images A and B are acquired in one imaging (exposure) simultaneously with the captured image and stored in the memory 32. When capturing an image for the second time or later, the imaging interval from the previous imaging is set to either the first imaging interval or a second imaging interval shorter than the first imaging interval.

[0068] In step S109, the system control unit 50 determines whether the imaging in step S108 was performed at the first imaging interval or at a second imaging interval that is shorter than the first imaging interval. If the first image is captured, the system control unit 50 determines that the imaging was performed at the first imaging interval. If the imaging in step S108 was performed at the first imaging interval (YES in S109), the system control unit 50 proceeds to step S110, and if the imaging was performed at the second imaging interval that is shorter than the first imaging interval (NO in S109), the system control unit 50 proceeds to step S111.

[0069] In step S110, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a strip shape with a first width according to the panning direction as an area to be used for panoramic synthesis.

[0070] In step S111, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a strip shape with a second width narrower than the first width in accordance with the panning direction as an area to be used for panoramic synthesis.

[0071] Then, in step S112, the system control unit 50 synthesizes a panoramic image by sequentially combining the images cut out into strips in step S110 or step S111. At this time, each time imaging is performed, the panoramic image is synthesized by adding strip-shaped images cut out from images obtained in the current imaging to the edges of the synthesized images obtained in the previous imaging. Image data of the panoramic image synthesized in this way is saved in memory 32. In the case of the first image, step S112 is skipped.

[0072] In step S113, the system control unit 50 uses the parallax images A and B captured in step S108 to acquire distance information of the subject in the image processing unit 24. Specifically, the system control unit 50 acquires the distance to the subject by detecting the phase difference between the two input parallax images, and acquires distance information from the imaging unit to the subject for each pixel. As described in FIG. 2, the distance information of the subject in this case is acquired from a distance information evaluation area that is set to include the area in the panning direction and outside the cutout area of ​​the image.

[0073] In step S114, the system control unit 50 determines whether or not a subject closer than a predetermined distance (closer) exists within the distance information evaluation area, based on the distance information acquired in step S113. If a subject closer than the predetermined distance (closer) does not exist (NO in S114), the system control unit 50 proceeds to step S115, and if a subject closer than the predetermined distance (closer) exists (YES in S114), the system control unit 50 proceeds to step S116.

[0074] In step S115, the system control unit 50 sets the imaging interval until the next imaging to the first imaging interval.

[0075] In step S116, the system control unit 50 sets the imaging interval until the next imaging to a second imaging interval that is shorter than the first imaging interval.

[0076] Next, in step S117, the system control unit 50 detects the angle of the panning direction of the digital camera 100 (hereinafter referred to as the "camera angle").

[0077] In step S118, the system control unit 50 determines whether the camera angle detected in step S117 has reached the panning end angle. If the camera angle has reached the panning end angle (YES in step S118), the system control unit 50 ends the processing of this flow, and if the camera angle has not reached the panning end angle (NO in step S118), the processing proceeds to step S119. Note that if the determination in step S118 is YES, the desired panoramic image has been obtained, and the image is recorded on the recording medium 200, and the processing ends.

[0078] In step S119, the system control unit 50 determines whether panning has ended based on the camera angle detected in step S117. Specifically, the system control unit 50 compares the angle of the digital camera 100 at the time of capturing an image with the angle of the digital camera 100 at the time of capturing the previous frame, and determines whether the angle has increased in the panning direction. If the angle has not increased in the panning direction, it is determined that panning has ended.

[0079] If the system control unit 50 determines that panning has ended (YES in S119), the processing of this flow ends. Note that if the determination in step S119 is YES, it means that panoramic image capture has been interrupted midway, and the images synthesized up to that point are recorded on the recording medium 200, and the processing ends. If the system control unit 50 determines that panning has not ended (NO in S119), the processing returns to step S108. As a result, the processing of steps S108 to S118 is repeated until image capture is completed, thereby performing continuous image capture while panning.

[0080] Furthermore, the method of determining the end of panning image capture described in step S118 is merely one example, and various modifications are applicable. For example, the system control unit 50 may end panning image capture when a preset number of images have been captured. Alternatively, panning image capture may be performed while the second shutter switch 63 is pressed, and then ended when the second shutter switch 63 is released.

[0081] In the above description, the composition is performed simultaneously with the panning image capture, but this is not limiting, and the composition of the panoramic image may start after all images have been captured by the panning image capture.

[0082] In the above description, the image clipping and compositing are completed by the digital camera 100, but the image clipping and compositing may be performed by an image processing device other than the digital camera 100. For example, the images captured by the digital camera 100 may be output to a computer, and a panoramic image may be generated by the computer.

[0083] In the above description, the system control unit 50 determines whether to use the first imaging interval or the second imaging interval based on a predetermined distance, but this is not limiting. The system control unit 50 may also determine the imaging interval based on distance information about the subject. In other words, the function that indicates the relationship between the distance information about the subject and the imaging interval is a continuous function.

[0084] As described above, images of an area where a foreground (close) subject exists, which is susceptible to the influence of parallax, are captured at a narrow imaging interval, and images are cut out and combined over a narrow width. On the other hand, images of an area where a foreground (close) subject does not exist are captured at a wide imaging interval, and images are cut out and combined over a wide width. In this way, it is possible to reduce composite misalignment in foreground (close) subjects without degrading the image quality of the entire image.

[0085] (Second embodiment) Next, a second embodiment will be described.

[0086] In the first embodiment, when a subject closer than a predetermined distance (closer) is not present within the distance information evaluation area of ​​the image, which is set to include the area in the panning direction and outside the image cropping area, the imaging interval is set to a long first imaging interval. Furthermore, when a subject closer than a predetermined distance (closer) is present within the distance information evaluation area of ​​the image, the imaging interval is set to a second imaging interval shorter than the first imaging interval. Images captured at the first imaging interval are cropped with a first width, which is wider, and images captured at the second imaging interval are cropped with a second width, which is narrower than the first width. In this way, panoramic synthesis is performed using images cropped with a narrow width in areas where foreground (closer) subjects, which are susceptible to the effects of parallax, are present, and with a wide width in areas where foreground (closer) subjects are not present.

[0087] In contrast, in the second embodiment, if a subject closer than a predetermined distance (closer) exists within the distance information evaluation area of ​​the image, a message is displayed informing the user to slow down the panning speed, and the image is cut out and composited in a narrow width. In this way, in areas where a subject closer than a predetermined distance (closer) exists, which is susceptible to the effects of parallax, a user is prompted to pan slowly. At the same time, the image is cut out and composited in a narrow width, and in areas where no subject closer than a predetermined distance (closer) exists, the image is cut out and composited in a wide width. In this way, it is possible to reduce composite misalignment of subjects closer than a predetermined distance (closer) without degrading the image quality of the entire image.

[0088] Figure 5 is a diagram illustrating the processing flow of this embodiment, the correspondence with image data, and the message display timing. In Figure 5, the dotted hatched area is a schematic representation of the area of ​​the subject that is closer than a predetermined distance (closer). The wide hatched area indicates the cutout area of ​​the image data, and the narrow hatched area indicates the distance information evaluation area for determining the imaging interval until the next imaging.

[0089] Figure 5(a) shows the location where the second shutter switch SW2 is pressed. The user sets the field of view at one end of the panoramic image to be synthesized. Figures 3(b) to 3(h) schematically show the state in which continuous image capture is being performed while panning the digital camera 100 toward the other end of the panoramic image to be synthesized. In (a), no subject (area hatched with dots) closer than a predetermined distance exists in the distance information evaluation area (area hatched with narrow diagonal lines). To capture the first image, a wide crop width Wcrop1 is created from the captured image (1) and used for panoramic synthesis.

[0090] 5(b), because a subject (area hatched with dots) closer than a predetermined distance (closer) exists in the distance information evaluation area (area hatched with narrow diagonal lines), a message informing the user to pan slowly is displayed on the display unit 28 as shown in (P). However, because no message was displayed when image (2) was captured, the captured image (2) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0091] 5(c) to 5(e), a subject (area hatched with dots) exists in the distance information evaluation area (area hatched with narrow diagonal lines) that is closer than a predetermined distance (closer). Therefore, a message such as (P) is displayed on the display unit 28 to advise the user to pan slowly. In addition, captured images (3) to (5) are cropped with a narrow width Wcrop2 and used for panoramic composition.

[0092] 5(f), since there is no subject (area hatched with dots) closer than a predetermined distance (closer) in the distance information evaluation area (area hatched with narrow diagonal lines), the message informing the user to pan slowly is not displayed on the display unit 28 as shown in (Q). However, since the message was displayed when image (6) was captured, the captured image (6) is cropped with a narrow width Wcrop2 and used for panoramic composition.

[0093] 5(g), since there is no subject (area hatched with dots) closer than a predetermined distance (closer) in the distance information evaluation area (area hatched with narrow diagonal lines), the message instructing the user to pan slowly is not displayed on the display unit 28, as shown in (Q). Also, a wide crop of the captured image (7) is performed with a width Wcrop1 and used for panoramic composition.

[0094] 5(h) shows the state where the digital camera 100 has panned to the set angle and continuous image capture has ended. In FIG. 5(h), since no message was displayed when image (8) was captured, the captured image (8) is cropped with a wide width Wcrop1 and used for panoramic composition.

[0095] FIG. 5(i) shows a panoramic image synthesized from a plurality of captured images (still images).

[0096] In this way, when capturing an image of an area where a foreground (close) subject is present and is susceptible to the effects of parallax, a message informing the user to pan slowly is displayed, and the image is cut out and combined in a narrow width. Also, when capturing an image of an area where a foreground (close) subject is not present, the message informing the user to pan slowly is not displayed, and the image is cut out and combined in a wide width. In this way, it is possible to reduce composite misalignment in foreground (close) subjects without degrading the image quality of the entire image.

[0097] FIG. 6 is a flowchart illustrating the flow of processing in this embodiment.

[0098] Hereinafter, the description of the same processes as those in the first embodiment will be omitted.

[0099] When the user selects the panoramic imaging mode with the mode switch 60, the processing of FIG. 6 is executed.

[0100] The processing of each step in Figure 6 is realized by the system control unit 50 loading a predetermined program read from the non-volatile memory 52 onto the system memory 56 and controlling the operation and processing of each part that makes up the digital camera 100.

[0101] Steps S201 to S208 are the same as steps S101 to S108 in the first embodiment, and therefore a description thereof will be omitted.

[0102] In step S209, the system control unit 50 determines whether the set image cropping width is the first cropping width or a second cropping width that is narrower than the first cropping width. The image cropping width is set in step S216 or step S218, which will be described later. For the first image capture, the first cropping width is set. If the set image cropping width is the first cropping width (YES in S209), the system control unit 50 proceeds to step S210, and if the set image cropping width is the second cropping width that is narrower than the first cropping width (NO in S209), the system control unit 50 proceeds to step S211.

[0103] In step S210, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a strip shape with a first width according to the panning direction as an area to be used for panoramic synthesis.

[0104] In step S211, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a strip shape with a second width narrower than the first width in accordance with the panning direction as an area to be used for panoramic synthesis.

[0105] Then, in step S212, the system control unit 50 sequentially combines the images cut out into strips in step S210 or step S211 to create a panoramic image. Each time image capture is performed, a strip-shaped image cut out from the image captured in the current capture is added to the edge of the combined image obtained in the previous capture, thereby creating a panoramic image. Image data of the panoramic image thus combined is stored in memory 32.

[0106] In step S213, the system control unit 50 uses the parallax images A and B captured in step S208 to acquire distance information of the subject in the image processing unit 24. Specifically, the system control unit 50 acquires the distance to the subject by detecting the phase difference between the two input parallax images, and acquires distance information from the imaging unit to the subject for each pixel. As described in FIG. 2, the distance information of the subject in this case is acquired from a distance information evaluation area that is set to include the area in the panning direction and outside the cutout area of ​​the image.

[0107] In step S214, the system control unit 50 determines whether or not a subject closer than a predetermined distance (closer) exists within the distance information evaluation area, based on the distance information acquired in step S213. If a subject closer than the predetermined distance (closer) does not exist (NO in S214), the system control unit 50 proceeds to step S215, and if a subject closer than the predetermined distance (closer) exists (YES in S214), the system control unit 50 proceeds to step S217.

[0108] In step S215, if a message indicating slow panning is displayed on the display unit 28, the system control unit 50 erases the message, and then in step S216 sets the cutout width for the next image capture to the first cutout width.

[0109] In step S217, the system control unit 50 displays a message on the display unit 28 to indicate that panning will be performed slowly, and then in step S218 sets the cutout width for the next image capture to a second cutout width that is narrower than the first cutout width.

[0110] Next, in step S219, the system control unit 50 detects the angle of the panning direction of the digital camera 100 (hereinafter referred to as the "camera angle").

[0111] In step S220, the system control unit 50 determines whether the camera angle detected in step S219 has reached the panning end angle. If the camera angle has reached the panning end angle (YES in step S220), the system control unit 50 ends the processing of this flow, and if the camera angle has not reached the panning end angle (NO in step S220), the processing proceeds to step S221. Note that if the determination in step S220 is YES, the desired panoramic image has been obtained, and the image is recorded on the recording medium 200, and the processing ends.

[0112] In step S221, the system control unit 50 determines whether panning has ended based on the camera angle detected in step S219. Specifically, the system control unit 50 compares the angle of the digital camera 100 at the time of capturing an image with the angle of the digital camera 100 at the time of capturing the previous frame, and determines whether the angle has increased in the panning direction. If the angle has not increased in the panning direction, it is determined that panning has ended.

[0113] If the system control unit 50 determines that panning has ended (YES in S221), the processing of this flow ends. Note that if the determination in step S221 is YES, it means that panoramic imaging has been interrupted midway, and the images synthesized up to that point are recorded on the recording medium 200, and the processing ends. If the system control unit 50 determines that panning has not ended (NO in S221), the processing returns to step S208. As a result, the processing of steps S208 to S220 is repeated until imaging is completed, thereby performing continuous imaging while panning.

[0114] As in the first embodiment, the above-described methods for determining the end of imaging and the method for determining the width to be cut out are merely examples, and various modifications can be applied.

[0115] As described above, within the area where foreground (closer) subjects exist, which is susceptible to the effects of parallax, the portion where foreground (closer) subjects exist is cut out and combined from an image synthesized with a narrower width, and the portion where foreground (closer) subjects do not exist is cut out and combined from an image with a wider width. This makes it possible to reduce the misalignment of foreground (closer) subjects without reducing the image quality of the entire image.

[0116] (Other embodiments) Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0117] The above embodiment has been described based on implementation in a digital camera, but is not limited to a digital camera. For example, the present invention may be implemented in a mobile device with a built-in image sensor, or in a network camera capable of capturing images.

[0118] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and run the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0119] 13 D / A converter 15 Memory control unit 18 Recording Media I / F 20 Shutter 22 Imaging unit 23 AF evaluation value detection section 24 Image processing section 26 Detector 28 Display section 32 memory 40 Power supply section 41 Power supply control unit 50 System control section 52 Non-volatile memory 53 System Timer 56 system memory 60 Mode Switch 61 Shutter button 62 First shutter switch 63 Second shutter switch 70 Operation section 72 Touch panel operation unit 80 Connector 100 digital cameras 200 Recording Media 300 Lens Unit 301 Imaging lens 302 Aperture 303 Lens control unit 304 Connector

Claims

1. a determination means for determining an imaging interval; an imaging means for capturing a plurality of images at the imaging intervals during panning; an acquisition means for acquiring distance information of a subject from the plurality of images captured by the imaging means; a synthesis means for synthesizing the plurality of images; a notification means for notifying the user of the panning speed; and the determination means determines the imaging interval to be a first imaging interval when the distance information is equal to or greater than a predetermined first distance, and determines the imaging interval to be a second imaging interval shorter than the first imaging interval when the distance information is less than the first predetermined distance; The imaging device is characterized in that the notification means issues the notification when the distance information is less than a predetermined second distance.

2. The imaging device described in Claim 1, characterized in that the notification means issues the notification so as to slow down the panning speed.

3. 2. The imaging device according to claim 1, wherein the determining unit determines the imaging interval based on the distance information of the area set by the obtaining unit based on the panning direction of the image.

4. the combining means uses a partial area cut out from each of the plurality of images for the combining; The imaging device according to claim 1, characterized in that the determination means determines the imaging interval for at least a portion of the plurality of images based on the distance information of an area that is located in the panning direction relative to the portion of the image cut out by the synthesis means.

5. The imaging device according to claim 4, characterized in that the determination means determines, for at least a portion of the plurality of images, an imaging interval between the image and the image of the next frame of the image based on the distance information of an area in the image that is in the panning direction relative to the portion of the image cut out by the synthesis means.

6. 6. The imaging device according to claim 1, wherein the combining means cuts out a portion of the image from the plurality of images, performs the combining using the portion of the image, and determines the size of the portion of the image cut out from the plurality of images based on the distance information.

7. 3. The imaging device according to claim 1, wherein the notification unit issues the notification based on the distance information of the area set by the acquisition unit based on the panning direction of the image.

8. the combining means uses a partial area cut out from each of the plurality of images for the combining; The imaging device described in claim 1 or 2, characterized in that the notification means issues the notification for at least a portion of the multiple images based on the distance information of an area that is in the panning direction relative to the portion of the image cut out by the synthesis means.

9. a determination step of determining an imaging interval; an imaging step of capturing a plurality of images at the imaging intervals during panning; an acquisition step of acquiring distance information of a subject from the plurality of images captured in the imaging step; a synthesis step of synthesizing the plurality of images; a notification step of notifying the user of the panning speed, In the determining step, when the distance information is equal to or greater than a predetermined first distance, the imaging interval is determined to be a first imaging interval, and when the distance information is less than the first predetermined distance, the imaging interval is determined to be a second imaging interval shorter than the first imaging interval, The imaging method, wherein the notification step performs the notification when the distance information is less than a predetermined second distance.

10. A program for causing a computer to operate an image processing method, a determination step of determining an imaging interval; an imaging step of capturing a plurality of images at the imaging intervals during panning; an acquisition step of acquiring distance information of a subject from the plurality of images captured in the imaging step; a synthesis step of synthesizing the plurality of images; a notification step of notifying the user of the panning speed; In the determining step, when the distance information is equal to or greater than a predetermined first distance, the imaging interval is determined to be a first imaging interval, and when the distance information is less than the first predetermined distance, the imaging interval is determined to be a second imaging interval shorter than the first imaging interval, The program, wherein the notification step causes the notification to be made when the distance information is less than a predetermined second distance.

11. A determination means for determining an imaging interval; an imaging means for capturing a plurality of images at the imaging interval; an acquisition means for acquiring distance information of a subject from the plurality of images captured by the imaging means; a synthesis means for synthesizing the plurality of images; and the determination means determines the imaging interval to be a first imaging interval when the distance information is equal to or greater than a predetermined first distance, and determines the imaging interval to be a second imaging interval shorter than the first imaging interval when the distance information is less than the first predetermined distance; The imaging device is characterized in that the synthesis means cuts out a portion of the image from the plurality of images, performs the synthesis using the portion of the image, and determines the size of the portion of the image cut out from the plurality of images based on the distance information.

12. The imaging device according to claim 11, wherein the imaging means captures the plurality of images by panning.

13. The imaging device described in Claim 12, characterized in that the determination means determines the imaging interval based on the distance information of the area set by the acquisition means based on the panning direction of the image.

14. A determination step of determining an imaging interval; an imaging step of capturing a plurality of images at the imaging interval; an acquisition step of acquiring distance information of a subject from the plurality of images captured in the imaging step; a synthesis step of synthesizing the plurality of images, In the determining step, when the distance information is equal to or greater than a predetermined first distance, the imaging interval is determined to be a first imaging interval, and when the distance information is less than the first predetermined distance, the imaging interval is determined to be a second imaging interval shorter than the first imaging interval, an imaging method characterized in that, in the combining step, partial areas are cut out from the plurality of images, and the combining is performed using the partial areas, and the size of the partial areas cut out from the plurality of images is determined based on the distance information.

15. A program for causing a computer to operate an image processing method, a determination step of determining an imaging interval; an imaging step of capturing a plurality of images at the imaging interval; an acquisition step of acquiring distance information of a subject from the plurality of images captured in the imaging step; a synthesis step of synthesizing the plurality of images, In the determining step, when the distance information is equal to or greater than a predetermined first distance, the imaging interval is determined to be a first imaging interval, and when the distance information is less than the first predetermined distance, the imaging interval is determined to be a second imaging interval shorter than the first imaging interval, In the synthesis step, a partial area is cut out from the plurality of images, the synthesis is performed using the partial area, and the size of the partial area cut out from the plurality of images is determined based on the distance information.

16. A computer-readable storage medium storing the program described in claim 10 or 15.

Images