Imaging apparatus, imaging method, program, and storage medium

Abstract

To provide an imaging apparatus that can reduce the frequency of failure in panoramic composition due to the fact that the feature quantity of positioning cannot be extracted because of the difference in luminance between images.SOLUTION: An imaging apparatus has: imaging means that picks up a plurality of images; determination means that determines a main subject area; luminance information acquisition means that acquires luminance information of the plurality of images and the main subject area; and composition means that performs composition of the plurality of images. The composition means changes an area used for composition of the plurality of images on the basis of the luminance information.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging device, and more particularly to an imaging device capable of capturing a panoramic image. [Background technology]

[0002] Conventionally, a method has been proposed in which still images are continuously captured while panning an imaging device, and a panoramic image is generated by combining the continuously captured still images (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-28764 A Summary of the Invention [Problem to be solved by the invention]

[0004] In an imaging device that generates a panoramic image by continuously taking still images while panning the imaging device and synthesizing the taken still images as described in Patent Document 1, synthesis may fail if the alignment feature cannot be extracted. If the brightness of the images to be synthesized is extremely low or high, it is considered that synthesis will often fail because the alignment feature cannot be extracted. For example, if a subject with a high brightness such as the sun is used as the main subject and still images are continuously taken while panning after adjusting the exposure, the brightness of the image in the area where the sun is not present will be low, and it is considered that synthesis will fail because the alignment feature cannot be extracted.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its object is to provide an imaging device that can reduce failures in image synthesis in low- or high-brightness areas when taking panoramic photographs while panning a subject scene with large brightness differences. [Means for solving the problem]

[0006] The imaging device of the present invention comprises an imaging means for capturing a plurality of images, a determination means for determining a main subject region, a luminance information acquisition means for acquiring luminance information of each of the plurality of images and the main subject region, and a synthesis means for synthesizing the plurality of images, wherein the synthesis means changes the region used for synthesizing each of the plurality of images based on the luminance information. Effect of the Invention

[0007] According to the present invention, it is possible to provide an imaging device that can reduce image synthesis failures in low-luminance or high-luminance areas when taking panoramic photographs while panning a subject scene with large differences in luminance. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] A diagram showing panning directions and image cropping areas. [Diagram 3] 3 is a diagram showing the flow of processing for panoramic photography and the corresponding relationship with image data in the first embodiment of the present invention. [Figure 4] 5 is a flowchart showing the operation of panoramic shooting in the first embodiment of the present invention. [Diagram 5] 11 is a diagram showing the panning direction, the image cutout area, and the brightness evaluation area. [Figure 6] FIG. 11 is a diagram showing the flow of processing for panoramic photography and the corresponding relationship with image data in the second embodiment of the present invention. [Figure 7] 10 is a flowchart showing the operation of panoramic shooting in the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0010] In the embodiment, the imaging device is a digital (still) camera, but the imaging device may include electronic devices having an imaging function, such as movie cameras, smartphones with cameras, tablet computers with cameras, in-vehicle cameras, and network cameras.

[0011] (First embodiment) FIG. 1 is a block diagram for explaining the configuration of an imaging device according to this embodiment.

[0012] As shown in FIG. 1, the imaging device of this embodiment is mainly composed of a digital camera 100 and a lens unit 300 of an interchangeable lens type that is detachable from the digital camera 100.

[0013] In the digital camera 100, the shutter 20 controls the time (exposure time) that an optical image is incident on the imaging unit 22, which will be described later. The imaging unit 22 includes an imaging element that is configured with a CCD, CMOS element, or the like, that converts the optical image into an electrical signal. The imaging unit 22 functions as an imaging means that captures a subject image.

[0014] The imaging unit 22 also has a built-in A / D conversion processing function and an AF evaluation value detection unit 23. The AF evaluation value detection unit 23 calculates an AF evaluation value based on contrast information obtained from a digital image signal and a phase difference obtained from a parallax image, and outputs the AF evaluation value to the system control unit 50.

[0015] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the image data output from the imaging unit 22 or image data from the memory control unit 15 .

[0016] Furthermore, the image processing unit 24 can obtain distance information of the subject based on the AF evaluation value. That is, the distance to the subject can be obtained by detecting the phase difference from the two input parallax images, and distance information from the imaging unit to the subject can be obtained for each pixel. A sensor such as a TOF (Time of Flight) sensor may be used to obtain the distance information.

[0017] Furthermore, the image processing unit 24 performs a predetermined calculation process using the captured image data to obtain exposure control information, and the system control unit 50 performs exposure control and focus adjustment control based on the distance information and exposure control information, thereby performing TTL (through-the-lens) type AE (automatic exposure) processing, EF (automatic flash light emission) processing, etc.

[0018] The image processing unit 24 also performs AF (autofocus) processing based on distance information, using the output of the AF evaluation value detection unit 23 provided in the imaging unit 22. The image processing unit 24 also performs TTL-type AWB (auto white balance) processing by performing a predetermined calculation process using the captured image data. The image processing unit 24 also performs processing to synthesize multiple images obtained by capturing images multiple times, and generates a panoramic image.

[0019] The detection unit 26 includes a gyro sensor and other sensors, and acquires angular velocity information, attitude information, and the like of the digital camera 100. The angular velocity information includes information on the angular velocity and angular acceleration during panning photography by the digital camera 100. The attitude information includes information such as the inclination of the digital camera 100 with respect to the horizontal direction.

[0020] The output data of the imaging unit 22 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data acquired and A / D converted by the imaging unit 22, and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 also serves as a memory for displaying images (video memory).

[0021] The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies it to a display unit 28 such as an LCD.

[0022] The digital signal that is first A / D converted by the imaging unit 22 and stored in the memory 32 is then converted to analog by the D / A converter 13 and sequentially transferred to the display unit 28 to display a through image (live view image), so that the display unit functions as an electronic viewfinder.

[0023] The non-volatile memory 52 is an electrically erasable and recordable memory, such as a flash memory. Constants, programs, and the like for the operation of the system control unit 50 are stored in the non-volatile memory 52. ​​The programs referred to here are computer programs for executing various flowcharts described later in this embodiment.

[0024] The system control unit 50 has a built-in CPU as a computer, and functions as a control unit that controls the entire digital camera 100. By executing the computer program recorded in the non-volatile memory 52 described above, each process of the flowchart of this embodiment, which will be described later, is realized.

[0025] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0026] The memory 56 is a system memory using a RAM or the like, in which constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 52, etc. are developed.

[0027] The mode switch 60, the shutter button 61 (including a first shutter switch 62 and a second shutter switch 63), the operation unit 70, and the touch panel operation unit 72 function as operation means for inputting various operational instructions to the system control unit 50.

[0028] The mode switch 60 switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, and the like.

[0029] The modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, various scene modes in which the shooting settings are made according to the shooting scene, a program AE mode, a custom mode, a panoramic shooting mode, etc. The mode switch 60 can be used to directly switch to any of these modes included in the still image shooting mode.

[0030] Alternatively, after switching to the still image shooting mode with the mode switch 60, another operating member may be used to switch to any of the modes included in the still image shooting mode. Similarly, the moving image shooting mode may include multiple modes.

[0031] A first shutter switch 62 is turned on when the shutter button 61 provided on the digital camera 100 is pressed halfway (instruction to prepare for shooting) and generates a first shutter switch signal SW1.

[0032] The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (auto flash control) processing.

[0033] The second shutter switch 63 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of still image photographing processing operations, reads signals from the imaging unit 22, writes image data to the recording medium 200, and so on.

[0034] Each operation member of the operation unit 70 and the touch panel operation unit 72 are assigned a function appropriate for each situation by selecting and operating various functional icons displayed on the display unit 28, and act as various function buttons. Note that a mouse or the like may be connectable to the digital camera 100, and the operation unit 70 includes such a mouse or the like.

[0035] The function buttons include, for example, an end button, a back button, an image forward button, a jump button, a narrowing down button, an attribute change button, a video capture button, etc. For example, when the menu button is pressed, a pull-down menu screen in which various settings can be made is displayed on the display unit 28.

[0036] The user can make various settings by selecting a menu on the menu screen displayed on the display unit 28 using the four-way buttons (up, down, left, right) and the SET button.

[0037] The power supply control unit 41 is made up of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery capacity.

[0038] Furthermore, the power supply control unit 41 controls the DC-DC converter based on the detection result and instructions from the system control unit 50, and supplies the necessary voltage to each unit including the recording medium 200 for the necessary period.

[0039] The power supply unit 40 is composed of a primary battery, a secondary battery, an AC adapter, etc. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc.

[0040] The connector 80 is a contact point for electrically connecting the camera body of the digital camera 100 to the lens unit 300. The connector 80 communicates control signals, status signals, data signals, and the like between the digital camera 100 and the lens unit 300, and also has a function of supplying power to the lens unit 300.

[0041] Next, the configuration of the lens unit 300 will be described.

[0042] In the lens unit 300, the imaging lens 301 is composed of multiple lenses including a focus lens that adjusts the focus by moving in the optical axis direction and a zoom lens that performs zooming. In addition, the diaphragm 302 adjusts the amount of light entering the imaging unit 22 of the digital camera 100 by controlling the diaphragm aperture.

[0043] The lens control unit 303 has a built-in CPU as a computer, and controls the entire lens unit 300 in response to control signals from the system control unit 50 of the digital camera 100, and also controls the driving of the focus lens, zoom lens, and aperture 302 of the imaging lens 301. It also has a memory for storing constants and variables for operation, computer programs, etc. It also has a non-volatile memory for storing identification information such as a number unique to the lens unit 300, management information, functional information such as the maximum aperture value, minimum aperture value, focal length, and amount of aberration such as distortion aberration, and various current and past setting values.

[0044] The connector 304 is a contact point for electrically connecting the lens unit 300 to the camera body of the digital camera 100. The connector 304 communicates control signals, status signals, data signals, and the like between the digital camera 100 and the lens unit 300, and also has a function of receiving a power supply voltage or current from the digital camera 100.

[0045] Next, a method for capturing a panning panoramic image and a method for synthesizing the images will be described in detail with reference to FIGS.

[0046] First, the cropping area (image range) of the captured image data will be described with reference to FIG. 2. The hatched area in FIG. 2 indicates the cropping (acquisition) area of ​​the image data, and the black arrow in FIG. 2 indicates the panning direction. FIG. 2(a) shows the effective image area of ​​the imaging unit 22, where W is the number of effective pixels in the horizontal direction, and H is the number of effective pixels in the vertical direction. FIG. 2(b) shows the cropping area of ​​the image data, where Wcrop is the cropping area width (number of pixels) in the horizontal direction, and Hcrop is the cropping area width (number of pixels) in the vertical direction. FIG. 2(c) shows the cropping area of ​​the image when panning in the horizontal direction, where the cropping of the image data in the horizontal direction is W>Wcrop, and the cropping of the image data in the vertical direction is H=Hcrop. FIG. 3(d) shows the cropping area of ​​the image when panning in the vertical direction, where the cropping of the image data in the horizontal direction is W=Wcrop, and the cropping of the image data in the vertical direction is H>Hcrop.

[0047] In this embodiment, the width of the cropping area of ​​the captured image data is switched according to the difference in brightness between the image used for synthesis to crop the image data and the main subject area (the area where AF processing and AE processing are performed by pressing the first shutter switch SW1). Specifically, in horizontal panning, the width of the cropping area of ​​the image is set to Wcrop1>Wcrop2, and if there is no brightness difference of a predetermined value or more, the image is cropped with the wide width Wcrop1. If there is a brightness difference of a predetermined value or more, the image is cropped with the narrow width Wcrop2. In addition, in vertical panning, the width of the cropping area of ​​the image is set to Hcrop1>Hcrop2, and if there is no brightness difference of a predetermined value or more, the image is cropped with the wide width Hcrop1. If there is a brightness difference of a predetermined value or more, the image is cropped with the narrow width Hcrop2.

[0048] Next, the image synthesis process will be described. The system control unit 50 reads from the memory 32 the image data that has been shot, cut out, and saved. Then, the system control unit 50 detects the positional deviation between the shot images. The detection of the positional deviation is performed by dividing the image into small blocks of any size, calculating corresponding points that minimize the SAD (Sum of Absolute Difference) of luminance for each small block, and calculating a motion vector. Not limited to this, angular velocity information, position information, and attitude information detected by the detection unit 26 may also be used. Next, the system control unit 50 calculates mapping parameters from the calculated motion vector to correct the positional deviation, and performs weighted addition on the overlapping parts between the images to perform synthesis.

[0049] Fig. 3 is a diagram for explaining the correspondence between the process flow and image data in this embodiment. In Fig. 3, 401 indicates the main subject region. The dotted hatched region is a region that is a schematic representation of a region with a large luminance difference from the main subject region, and the diagonal line hatched region indicates a cut-out region of the image data.

[0050] Figure 3(a) shows the location where the first shutter switch SW1 is pressed. By operating SW1, AF and AE processing are performed on the area to be the main subject, so that the main subject area is focused on and the exposure is appropriate. Here, the bright sun is captured as the main subject area 401, and the areas hatched with dots are areas with a large difference in brightness from the main subject area, i.e., areas with low brightness.

[0051] Fig. 3(b) shows the location where the second shutter switch SW2 is pressed. The user sets the field of view at one end of the panoramic image to be synthesized. Figs. 3(b) to (m) show schematic diagrams of the state in which continuous shooting is performed while panning the digital camera 100 toward the other end of the panoramic image to be synthesized.

[0052] In Figure 3(b) to (e), the captured images contain many dot-hatched areas. Because there is a large difference in brightness between the captured image and the main subject area, all captured images (1) to (4) are cropped with a narrow width Wcrop2 and used for panorama synthesis.

[0053] In Figure 3(f) to (i), the dotted hatched areas are not included in the captured images. Since the difference in brightness between the captured image and the main subject area is small, a part of the captured images (5) to (8), images (5) and (7), are cropped with a wide width Wcrop1 and used for panorama synthesis.

[0054] In (j) to (m), the captured images also contain many dot-hatched areas. Because there is a large difference in brightness between the captured image and the main subject area, all captured images (9) to (12) are cropped with a narrow width Wcrop2 and used for panorama synthesis.

[0055] Fig. 3(m) shows the state where digital camera 100 has been panned to the set angle and continuous shooting has ended. Fig. 3(n) shows a panoramic image synthesized from multiple captured images (still images).

[0056] In this way, images with a large luminance difference from the main subject region, from which it is difficult to extract alignment features, are cut out in a narrow width, and images with a small luminance difference are cut out in a wide width, and then combined.

[0057] By cutting out the images in such a narrow width and compositing them, it is possible to make the area of ​​the image that is not used for compositing and is used for alignment wider, making it easier to extract alignment features. In this way, it is possible to reduce compositing failures caused by the inability to extract alignment features without degrading the image quality of the entire image.

[0058] FIG. 4 is a flowchart illustrating the flow of processing in this embodiment.

[0059] 4 is executed when the user selects the panoramic shooting mode with the mode change switch 60. The processing of each step shown in FIG. 4 is realized by the system control unit 50 loading a predetermined program read from the non-volatile memory 56 onto the system memory 52 and controlling the operation and processing of each unit constituting the digital camera 100.

[0060] In step S101, the system control unit 50 determines whether or not the panning condition setting button has been pressed by operating the operation unit 70. The panning condition setting button is one of the function buttons realized by the operation unit 70. If the panning condition setting button has been pressed (YES in S101), the system control unit 50 advances the process to step S102, and if the panning condition setting button has not been pressed (NO in S101), the system control unit 50 advances the process to step S103.

[0061] In step S102, the system control unit 50 accepts a change in the shooting conditions (such as the panning direction) for panning shooting made by the user, stores the accepted conditions in the non-volatile memory 56, and then proceeds to step S103. Note that if the panning condition setting button has not been pressed and panning shooting is being performed for the first time, the default setting conditions are applied.

[0062] In step S103, the system control unit 50 determines whether the first shutter switch 62 of the shutter button 61 is turned on (whether the first shutter switch signal SW1 is generated). If the first shutter switch 62 is turned on (YES in S103), the system control unit 50 proceeds to step S104, and if the first shutter switch 62 is not turned on (NO in S103), the system control unit 50 returns to step S101.

[0063] In step S104, the system control unit 50 performs AE processing and AF processing (autofocus processing). In the following step S105, the system control unit 50 determines the shooting conditions (exposure, shooting sensitivity, WB, etc.) based on the AE processing result in step S104. Note that in the following step S109, the shooting lens 103 is fixed to the lens position when the subject is focused in the AF processing in step S104, and shooting is performed under fixed shooting conditions that provide appropriate exposure of the main subject area in the AE processing.

[0064] In step S106, the system control unit 50 calculates the average luminance (average luminance) of the main subject area for which the AE process and the AF process were performed in step S104 (obtaining luminance information). The average luminance of the main subject area calculated here is used when determining the clipping width of the synthesis area used in image synthesis in steps S111 to S114.

[0065] Next, in step S107, the system control unit 50 determines whether the second shutter switch 63 is turned on (whether the second shutter switch signal SW2 is generated). If the second shutter switch 63 is not turned on (NO in S107), the system control unit 50 proceeds to step S108, and if the second shutter switch 63 is turned on (YES in S107), the system control unit 50 proceeds to step S109.

[0066] In step S108, the system control unit 50 determines whether or not the first shutter switch 62 remains on. If the first shutter switch 62 remains on (YES in S108), the system control unit 50 returns to step S107 and repeats the process until the second shutter switch 63 is turned on in step S107. If the first shutter switch 62 is off (NO in S108), the system control unit 50 returns to step S101.

[0067] In step S109, the system control unit 50 controls each unit of the digital camera 100 to execute a photographing process (taking a single image).

[0068] In step S110, the system control unit 50 calculates the average luminance of the images captured in step S109 (obtaining luminance information).

[0069] In step S111, the system control unit 50 compares the average image luminance calculated in step S110 with the average luminance of the main subject region calculated in step S106, and determines whether the luminance difference is smaller than a predetermined threshold value. If the luminance difference is small (YES in S111), the system control unit 50 proceeds to step S112, and if the luminance difference is large (NO in S111), the system control unit 50 proceeds to step S114.

[0070] In step S112, the system control unit 50 determines whether or not the image captured in step S109 is used for panoramic composition. As described with reference to FIG. 3, if the luminance difference from the luminance of the main subject region is smaller than a predetermined threshold, only a part of the image captured in step S109 is used, and the image is cut out and synthesized over a wide width. For example, every other image is used instead of even-numbered images having a small luminance difference from the luminance of the main subject region, so that one out of N images is used for cutting and synthesis (N is an integer). If the image captured in step S109 is used for panoramic composition (YES in S112), the system control unit 50 advances the process to step S113, and if the image is not used for panoramic composition (NO in S112), the system control unit 50 advances the process to step S116.

[0071] In step S113, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a rectangular shape with a first width according to the panning direction as an area to be used for panoramic synthesis.

[0072] In step S114, the system control unit 50 cuts out the captured image data into a strip shape with a second width narrower than the first width in accordance with the panning direction as an area to be used for panoramic synthesis by the image processing unit 24. At this time, if one of the N images captured in step S109 is to be used for cutting and synthesis in step S112, it is preferable that the second clipping width is 1 / N (1 / integer) of the first clipping width.

[0073] Then, in step S115, the system control unit 50 sequentially combines the images cut into strips in step S113 or step S114 to create a panoramic image. At this time, each time shooting is performed, the panoramic image is combined by adding strip-shaped images cut out from the images obtained in the current shooting to the ends of the combined image obtained by the previous shooting. The image data of the panoramic image thus combined is stored in the memory 32.

[0074] Next, in step S116, the system control unit 50 detects the angle of the panning direction of the digital camera 100 (hereinafter referred to as the "camera angle").

[0075] In step S117, the system control unit 50 determines whether the camera angle detected in step S116 is the panning end angle. If the camera angle is the panning end angle (YES in S117), the system control unit 50 ends the processing of this flow, and if the camera angle is not the panning end angle (NO in S117), the processing proceeds to step S118. Note that if the determination in step S117 is YES, the desired panoramic image has been obtained, and the image is recorded on the recording medium 200, and the processing ends.

[0076] In step S118, the system control unit 50 determines whether or not panning has ended based on the camera angle detected in step S116. Specifically, this is done by comparing the angle of the digital camera 100 at the time of shooting with the angle of the digital camera 100 at the time of shooting one frame before, and determining whether or not the angle has increased in the panning direction. If the angle has not increased in the panning direction, it is determined that panning has ended.

[0077] If the system control unit 50 determines that panning has ended (YES in S118), the process of this flow ends. Note that if the determination in step S118 is YES, panoramic shooting is interrupted midway, and the images synthesized up to that point are recorded on the recording medium 200, and the process ends. If the system control unit 50 determines that panning has not ended (NO in S118), the process returns to step S109. As a result, the processes in steps S109 to S117 are repeated until shooting ends, thereby performing continuous shooting while panning.

[0078] As described above, images with a large luminance difference from the luminance of the main subject region, from which it is difficult to extract alignment features, are cut out in a narrow width, and images with a small luminance difference are cut out in a wide width, and then composited. By cropping images in a narrow width and then composited, it is possible to make the image region used for alignment rather than composite wider, making it easier to extract alignment features. In this way, it is possible to reduce composite failures caused by the inability to extract alignment features without degrading the image quality of the entire image.

[0079] Second embodiment Next, a second embodiment will be described.

[0080] In the first embodiment, when the luminance difference between the main subject region and the image is smaller than a predetermined threshold, the image is cut out from a part of the continuously shot images with a wide first width. Also, when the luminance difference between the main subject region and the image is larger than a predetermined threshold, the image is cut out from all the continuously shot images with a second width narrower than the first width. In this way, when the image has a large luminance difference between the main subject region and the image from which the alignment feature is difficult to extract, the image is cut out with a narrow width and synthesized, so that the area of ​​the image used for alignment rather than synthesis can be made wider, making it easier to extract the alignment feature.

[0081] In contrast, in the second embodiment, when the difference in brightness between the brightness of the brightness information evaluation area of ​​the image set to include the area outside the cutout area of ​​the image in the panning proceeding direction and the brightness of the main subject area is smaller than a predetermined threshold, the imaging interval is set to a long first imaging interval. Also, when the difference in brightness between the brightness of the brightness information evaluation area of ​​the image and the brightness of the main subject area is larger than a predetermined threshold, the imaging interval is set to a second imaging interval shorter than the first imaging interval. Then, the image is cut out with a wide first width from the image taken with the first imaging interval, and the image is cut out with a narrow second width from the image taken with the second imaging interval. In this way, an image with a large brightness difference from the brightness of the main subject area from which it is difficult to extract the feature amount of the alignment is cut out with a narrow width, and an image with a small brightness difference from the brightness of the main subject area is cut out with a wide width for panoramic synthesis.

[0082] Next, a method for capturing a panning panoramic image and a method for synthesizing the images will be described in detail with reference to FIGS.

[0083] First, the cropping area (image range) of the captured image data will be described with reference to FIG. 5. In FIG. 5, the hatched area with wide-spaced diagonal lines going up to the right indicates the cropping area of ​​the image data, the hatched area with narrow-spaced diagonal lines going down to the right indicates the brightness information evaluation area for determining the imaging interval until the next image capture, and the black arrow indicates the panning direction. FIG. 5(a) shows the effective image area of ​​the imaging unit 22, where W is the number of effective pixels in the horizontal direction, and H is the number of effective pixels in the vertical direction. FIG. 5(b) shows the cropping area of ​​the image data, where Wcrop is the cropping area width (number of pixels) in the horizontal direction, and Hcrop is the cropping area width (number of pixels) in the vertical direction. FIG. 5(c) shows the cropping area and brightness information evaluation area of ​​the image when panned to the right, (d) to the left, (e) to the top, and (f) to the bottom. When panning to the right or left, the horizontal cropping of image data is W>Wcrop, and the vertical cropping of image data is H=Hcrop. When panning up or down, the horizontal cropping of image data is W=Wcrop, and the vertical cropping of image data is H>Hcrop. In this embodiment, the imaging interval of continuous shooting is switched according to the luminance difference between the average luminance of the luminance information evaluation area of ​​the image and the average luminance of the main subject area, and the width of the cropping area of ​​the captured image data is switched according to the imaging interval at the time of shooting. Specifically, in horizontal panning, the width of the cropping area of ​​the image is Wcrop1>Wcrop2, and when the imaging interval of continuous shooting is the first imaging interval, which is long, cropping is performed with the wide width Wcrop1. When the imaging interval of continuous shooting is the second imaging interval, which is short, cropping is performed with the narrow width Wcrop2. In addition, in vertical panning, the width of the cropping area of ​​the image is set to Hcrop1>Hcrop2, and when the continuous shooting interval is a first interval where the imaging interval is long, the image is cropped to the wide width Hcrop1. When the continuous shooting interval is a second interval where the imaging interval is short, the image is cropped to the narrow width Hcrop2.

[0084] The luminance information evaluation area is set to include the area in the panning progression direction and outside the cutout area of ​​the image. That is, when panning to the right as in (c), the luminance information evaluation area is set to include the right side of the cutout area of ​​the image, and when panning to the left as in (d), the luminance information evaluation area is set to include the left side of the cutout area of ​​the image. Also, when panning upward as in (e), the upper side of the cutout area of ​​the image is included, and when panning in the direction as in (f), the lower side of the cutout area of ​​the image is included. In this way, by setting the luminance information evaluation area according to the panning progression direction, it is possible to determine the imaging interval until the next shooting based on the luminance of the cutout area of ​​the image data in the next shooting. In this way, the imaging interval is changed according to the luminance difference between the luminance of the luminance information evaluation area of ​​the image and the luminance of the main subject area. Then, for an image with a large luminance difference with the luminance of the main subject area from which it is difficult to extract the feature amount of alignment, an image shot at a short imaging interval is cut out with a narrow width and composited. Also, for an image with a small luminance difference with the luminance of the main subject area, an image shot at a long imaging interval is cut out with a wide width and composited. In this way, it is possible to reduce synthesis failures caused by an inability to extract alignment features without degrading the image quality of the entire image.

[0085] Fig. 6 is a diagram for explaining the correspondence between the process flow and image data in this embodiment. In Fig. 6, 401 indicates the main subject region. The dotted hatched region is a region that is a schematic representation of a region with a large luminance difference from the main subject region. In addition, the hatched region with a wide diagonal line slanting upwards to the right indicates a cut-out region of the image data, and the hatched region with a narrow diagonal line slanting downwards to the right indicates a luminance information evaluation region for determining the imaging interval until the next shooting.

[0086] Figure 6(a) shows the location where the first shutter switch SW1 is pressed. By operating SW1, AF and AE processing are performed on the area to be the main subject, so that the main subject area is focused on and the exposure is appropriate. Here, the bright sun is captured as the main subject area 401, and the areas hatched with dots are areas with a large difference in brightness from the main subject area, i.e., areas with low brightness.

[0087] Fig. 6(b) shows the location where the second shutter switch SW2 is pressed. The user sets the field of view at one end of the panoramic image to be synthesized. Fig. 6(b) to (k) show schematic diagrams of the state in which continuous shooting is performed while panning the digital camera 100 toward the other end of the panoramic image to be synthesized.

[0088] In Fig. 6(b) to (d), since there is a large difference in brightness between the brightness information evaluation area and the main subject area, the image capture intervals between (b) and (c), between (c) and (d), and between (d) and (e) are short (Mmsec). In addition, the captured images (1) to (3) are cropped with a narrow width Wcrop2 and used for panorama synthesis.

[0089] In (e), the difference in luminance between the luminance information evaluation area and the main subject area is small, so the imaging interval between (e) and (f) is long (N msec). However, the imaging interval between (d) and (e) when image (4) was taken is short (M msec), so the captured image (4) is cropped with a narrow width Wcrop2 and used for panorama synthesis.

[0090] In (f), the difference in brightness between the brightness information evaluation area and the main subject area is small, so the imaging interval between (f) and (g) is set to a long imaging interval (N msec). In addition, the captured image (5) is cropped with a wide width Wcrop1 and used for panorama synthesis.

[0091] In (g), the difference in luminance between the luminance information evaluation area and the main subject area is large, so the imaging interval between (g) and (h) is short (M msec). However, the imaging interval between (f) and (g) when image (6) is captured is long (N msec), so the captured image (6) is cropped with a wide width Wcrop1 and used for panorama synthesis.

[0092] In (h) to (k), the difference in brightness between the brightness information evaluation area and the main subject area is large, so the imaging intervals between (h)(i), (i)(j), and (j)(k) are short (Mmsec). In addition, the captured images (7) to (10) are cropped with a narrow width Wcrop2 and used for panorama synthesis.

[0093] FIG. 6(k) shows the state where digital camera 100 has been panned to the set angle and continuous shooting has ended.

[0094] FIG. 6(l) shows a panoramic image synthesized from a plurality of captured images (still images).

[0095] In this way, when shooting an area with a large difference in luminance from the main subject area, from which it is difficult to extract alignment features, the area is shot at short shooting intervals, the image is cut out in a narrow width, and then composited. When shooting an area with a small difference in luminance from the main subject area, the area is shot at long shooting intervals, the image is cut out in a wide width, and then composited. In this way, it is possible to reduce composite failures caused by the inability to extract alignment features without degrading the image quality of the entire image.

[0096] 7 is a flowchart for explaining the flow of processing in this embodiment. Hereinafter, explanations of processing that is the same as the processing in the first embodiment will be omitted.

[0097] When the user selects the panoramic photography mode with the mode switch 60, the process of FIG. 7 is executed.

[0098] The processing of each step in FIG. 7 is realized by the system control unit 50 loading a predetermined program read from the non-volatile memory 52 onto the system memory 56 and controlling the operation and processing of each component of the digital camera 100.

[0099] Steps S201 to S209 are the same as steps S101 to S109 in the first embodiment, and therefore the description thereof will be omitted.

[0100] In step S210, the system control unit 50 calculates the average luminance value of a luminance information evaluation region that is set so as to include the region in the panning direction of the image captured in step S209 and outside the cutout region of the image.

[0101] In step S211, the system control unit 50 compares the average luminance of the luminance information evaluation region calculated in step S210 with the average luminance of the main subject region calculated in step S206, and determines whether the luminance difference is smaller than a predetermined threshold value. If the luminance difference is small (YES in S211), the system control unit 50 proceeds to step S212, and if the luminance difference is large (NO in S211), the system control unit 50 proceeds to step S214.

[0102] In step S212, the system control unit 50 sets the imaging interval until the next shooting to the first imaging interval.

[0103] In step S213, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a strip of a first width according to the panning direction as an area to be used for panoramic synthesis.

[0104] In step S214, the system control unit 50 sets the imaging interval until the next shooting to a second imaging interval that is shorter than the first imaging interval.

[0105] In step S215, the system control unit 50 causes the image processing unit 24 to cut out the captured image data into a rectangular shape with a second width narrower than the first width in accordance with the panning direction as an area to be used for panoramic synthesis.

[0106] Then, in step S216, the system control unit 50 sequentially combines the images cut into strips in step S213 or step S215 to create a panoramic image. At this time, each time shooting is performed, the panoramic image is combined by adding strip-shaped images cut out from the images obtained in the current shooting to the ends of the combined image obtained by the previous shooting. The image data of the panoramic image thus combined is stored in the memory 32.

[0107] Steps S217 to S219 are the same as steps S116 to S118 in the first embodiment, and therefore a description thereof will be omitted.

[0108] As described above, when shooting an area with a large luminance difference from the main subject area, from which it is difficult to extract alignment features, the image is shot at a short shooting interval, the image is cut out in a narrow width, and then composited. When shooting an area with a small luminance difference from the main subject area, the image is shot at a long shooting interval, the image is cut out in a wide width, and then composited. In this way, it is possible to reduce composite failures caused by the inability to extract alignment features without degrading the image quality of the entire image.

[0109] (Other embodiments) Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0110] The above embodiment has been described based on the implementation in a digital camera, but the present invention is not limited to a digital camera. For example, the present invention may be implemented in a mobile device with a built-in image sensor, or in a network camera capable of capturing images.

[0111] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and run the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0112] 13 D / A converter 15 Memory control section 18 Recording media I / F 20 Shutter 22 Imaging unit 23 AF evaluation value detection section 24 Image Processing Section 26 Detection unit 28 Display section 32 Memory 40 Power supply section 41 Power supply control unit 50 System control section 52 Non-volatile memory 53 System Timer 56 System Memory 60 Mode Switch 61 Shutter button 62 First shutter switch 63 Second shutter switch 70 Operation section 72 Touch panel operation unit 80 Connectors 100 Digital Camera 200 Recording media 300 Lens Unit 301 Imaging Lens 302 Aperture 303 Lens control unit 304 Connector

Claims

1. An imaging means for capturing a plurality of images; A determining means for determining a main subject region; an acquisition means for acquiring luminance information of each of the plurality of images and the main subject region; A synthesis means for synthesizing the plurality of images, The imaging device according to claim 1, wherein the combining means changes an area used for combining each of the plurality of images based on the luminance information.

2. The image processing apparatus according to claim 1 , wherein the acquiring means acquires information regarding a luminance difference between the plurality of images and the main subject region as the luminance information.

3. 3. The imaging apparatus according to claim 2, wherein the imaging means captures the plurality of images by panning.

4. 4. The image processing device according to claim 3, wherein the synthesis means performs cut-out synthesis with a first width for those of the plurality of images whose average luminance difference from the average luminance of the main subject region is a first value, and performs cut-out synthesis with a second width narrower than the first width for those of the plurality of images whose average luminance difference from the average luminance of the main subject region is a second value greater than the first value.

5. 5. The image pickup apparatus according to claim 4, wherein the synthesizing means sets the second width to a width that is an integer fraction of the first width.

6. The imaging device according to claim 4, characterized in that, when cutting out the multiple images captured by the imaging means at the first width, the synthesis means selects the multiple images every N (N is an integer) and sets the second width to 1 / N of the first width.

7. The imaging device further includes a setting unit for setting an imaging interval of the imaging unit, 4. The imaging device according to claim 3, wherein the determining unit determines the imaging interval based on the luminance information of an evaluation area that is set by the acquiring unit based on the panning direction of the plurality of images.

8. the synthesis means uses a partial area cut out from each of the plurality of images for the synthesis, The imaging device according to claim 7, characterized in that the determination means determines the imaging interval for at least a portion of the plurality of images based on the luminance information of the evaluation area that is located in the panning direction relative to the portion of the image cut out by the synthesis means.

9. the determination means sets the imaging interval to a first imaging interval when a luminance difference between the average luminance of the main subject region and the average luminance of the evaluation region is a first value, and sets the imaging interval to a second imaging interval shorter than the first imaging interval when a luminance difference between the average luminance of the main subject region and the average luminance of the evaluation region is a second value larger than the first value; The imaging device according to claim 8, characterized in that the combining means performs cut-out and compositing with a first width from images continuously photographed at the first imaging interval, and performs cut-out and compositing with a second width narrower than the first width from images continuously photographed at the second imaging interval.

10. 10. The imaging device according to claim 9, wherein the synthesizing means sets the second width to a width that is an integer division of the first width.

11. 10. The imaging device according to claim 9, wherein the setting means sets the second imaging interval to a length that is 1 / N (N is an integer) of the first imaging interval, and the combining means sets the second width to a width that is 1 / N of the first width.

12. An imaging step of capturing a plurality of images; A determining step of determining a main subject region; a luminance information acquisition step of acquiring luminance information of each of the plurality of images and the main subject region; A synthesis step of synthesizing the plurality of images, A method for controlling an imaging apparatus, wherein the combining step changes an area used for combining each of the plurality of images based on the luminance information.

13. A program for causing a computer to function as each of the means of the imaging apparatus according to any one of claims 1 to 11.

14. A computer-readable storage medium storing the program according to claim 13.

Images