Image processing device, imaging device, image processing method, program, recording medium

Abstract

To solve the problem in which a merged image of a subject in the foreground (closer to the subject) may be out of alignment due to parallax when generating a panoramic image by merging still images captured.SOLUTION: An image processing device has synthesizing means for synthesizing by cropping a portion from a plurality of images and acquiring means for obtaining distance information of a subject from at least a portion of the plurality of images. The synthesizing means determines the size of the portion to be cropped from the images with respect to at least a portion of the plurality of images on the basis of the distance information of the image acquired by the acquiring means.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that cuts out and synthesizes a part from a plurality of images.

Background Art

[0002] Conventionally, a method has been proposed to generate a panoramic image by continuously capturing still images while panning an imaging device and synthesizing the continuously captured still images (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in an image processing apparatus that generates a panoramic image by synthesizing captured still images using the method for generating a panoramic image described in Patent Document 1, a synthesis shift may occur at the joint of the images of a subject in the foreground (closer than the nearest point) due to the influence of parallax.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus capable of reducing the synthesis shift in a subject in the foreground (closer than the nearest point) without degrading the image quality of the entire image.

Means for Solving the Problems

[0006] The image processing apparatus according to the present invention comprises a synthesis means for synthesizing regions cut out from each of a plurality of images, and an acquisition means for acquiring distance information of a subject from at least a portion of the plurality of images, wherein the synthesis means, for at least a portion of the plurality of images, synthesizes a region to be cut out from the image based on the distance information of the image acquired by the acquisition means. width of When it is determined that the width of the region to be cut out from the first image having the first distance information is the first width, the second width, which is the width of the region to be cut out from the second image having the second distance information which is closer than the first distance information, is narrower than the first width. This is its defining characteristic. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device that can reduce the image distortion in the foreground (closest) subject without degrading the overall image quality. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of an image processing apparatus in an embodiment of the present invention. [Figure 2] This figure illustrates the panning direction and image cropping region in an embodiment of the present invention. [Figure 3] This diagram illustrates the processing flow of panoramic imaging in the first embodiment and its correspondence with the image data. [Figure 4] This is a flowchart illustrating the operation of panoramic imaging in the first embodiment. [Figure 5] This diagram illustrates the processing flow of panoramic imaging in the second embodiment and its correspondence with the image data. [Figure 6] This is a flowchart illustrating the operation of panoramic imaging in the second embodiment. [Modes for carrying out the invention]

[0009] Preferred embodiments of the present invention will be described below with reference to the attached drawings. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0010] Furthermore, the following description will focus on an example where the image processing device is applied to a digital (still) camera. However, the image processing device can also include electronic devices such as movie cameras, smartphones with cameras, tablet computers with cameras, in-vehicle cameras, and network cameras. In addition, the image processing device in the following embodiments may be a computer capable of processing images captured by another device.

[0011] (First embodiment) Figure 1 is a block diagram showing the configuration of an image processing apparatus in a first embodiment of the present invention.

[0012] As shown in Figure 1, the image processing device of this embodiment mainly consists of a digital camera 100 and a detachable interchangeable lens unit 300 for the digital camera 100.

[0013] In the digital camera 100, the shutter 20 controls the incident time (exposure time) of the optical image to the imaging unit 22, which will be described later. The imaging unit 22 includes an image sensor composed of a CCD or CMOS element, which converts the optical image into an electrical signal. The imaging unit 22 functions as an imaging means for capturing an image of a subject.

[0014] The imaging unit 22 also incorporates an A / D conversion processing function and an AF evaluation value detection unit 23. The AF evaluation value detection unit 23 calculates an AF evaluation value based on contrast information obtained from the digital image signal and phase difference obtained from the parallax image, and outputs it to the system control unit 50.

[0015] The image processing unit 24 performs resizing and color conversion processing, such as predetermined pixel interpolation and reduction, on the image data output from the imaging unit 22 or the image data from the memory control unit 15.

[0016] In addition, the image processing unit 24 can obtain subject distance information based on the AF evaluation value. That is, by detecting the phase difference from the two input parallax images, the distance to the subject can be obtained, and it is possible to obtain the distance information from the imaging unit to the subject for each pixel. For obtaining the distance information, a sensor such as a TOF (Time of Flight) sensor may be used.

[0017] Furthermore, the image processing unit 24 performs predetermined arithmetic processing using the captured image data to obtain exposure control information, and based on this distance information and exposure control information, the system control unit 50 performs exposure control and focus adjustment control. As a result, TTL (Through-the-Lens) type AE (Automatic Exposure) processing, EF (Flash Automatic Light Control) processing, etc. are performed.

[0018] Also, in the image processing unit 24, AF (Auto Focus) processing based on the distance information is performed, and at this time, the output of the AF evaluation value detection unit 23 provided in the imaging unit 22 is used. The image processing unit 24 further performs TTL type AWB (Auto White Balance) processing by performing predetermined arithmetic processing using the captured image data. In addition, the image processing unit 24 performs a process of synthesizing a plurality of images obtained by imaging a plurality of times and also generates a panoramic image.

[0019] The detection unit 26 includes a gyro sensor and other sensors, and obtains the angular velocity information, attitude information, etc. of the digital camera 100. Note that the angular velocity information includes the information of the angular velocity and angular acceleration during panning imaging by the digital camera 100. Also, the attitude information includes information such as the inclination of the digital camera 100 with respect to the horizontal direction.

[0020] The output data of the imaging unit 22 is written directly into the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15. The memory 32 stores the image data acquired and A / D converted by the imaging unit 22, as well as the image data for display on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a moving image for a predetermined time, and audio. Also, the memory 32 doubles as a memory for image display (video memory).

[0021] The D / A converter 13 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28 such as an LCD.

[0022] The digital signal once A / D converted by the imaging unit 22 and accumulated in the memory 32 is analog-converted by the D / A converter 13 and sequentially transferred to the display unit 28 to display a through-image (live view image), whereby the display unit functions as an electronic viewfinder.

[0023] The nonvolatile memory 52 is an electrically erasable and recordable memory, and for example, a flash memory or the like is used. The nonvolatile memory 52 stores constants for the operation of the system control unit 50, programs, and the like. Here, the program refers to a computer program for executing various flowcharts described later in this embodiment.

[0024] The system control unit 50 incorporates a CPU as a computer and functions as control means for controlling the entire digital camera 100. By executing the computer program recorded in the aforementioned nonvolatile memory 52, each process of the flowchart of this embodiment described later is realized.

[0025] The system timer 53 is a timing unit that measures the time used for various controls and the time of the built-in clock.

[0026] 56 is a system memory using RAM or the like, where constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 52, etc., are stored.

[0027] The mode switch 60, shutter button 61 (including the first shutter switch 62 and the second shutter switch 63), operation unit 70, and touch panel operation unit 72 function as operating means for inputting various operation instructions to the system control unit 50.

[0028] The mode switch 60 switches the operating mode of the system control unit 50 to one of the following: still image capture mode, video capture mode, playback mode, etc.

[0029] The modes included in the still image capture mode are Auto mode, Auto scene detection mode, Manual mode, various scene modes that provide different imaging settings for different shooting scenes, Program AE mode, Custom mode, Panorama mode, etc. The mode switch 60 allows you to directly switch to any of these modes included in the still image capture mode.

[0030] Alternatively, the system may be switched to still image capture mode using mode switch 60, and then another operating component may be used to switch to one of the modes included in still image capture mode. Similarly, the video capture mode may also include multiple modes.

[0031] The first shutter switch 62 turns ON during the operation of the shutter button 61 provided on the digital camera 100, specifically when it is half-pressed (indicating preparation for imaging), and generates the first shutter switch signal SW1.

[0032] The first shutter switch signal SW1 initiates operations such as AF (autofocus), AE (automatic exposure), AWB (auto white balance), and EF (automatic flash exposure).

[0033] The second shutter switch 63 turns ON when the shutter button 61 is fully pressed (imaging instruction), generating the second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of still image imaging processes, such as reading signals from the imaging unit 22 and writing image data to the recording medium 200.

[0034] Each operating element of the control unit 70 and the touch panel control unit 72 can be assigned a function as appropriate for each situation by selecting various function icons displayed on the display unit 28, and they function as various function buttons. The digital camera 100 may also be made compatible with a mouse, and the control unit 70 may include such a mouse.

[0035] Function buttons include, for example, an exit button, a back button, an image forward button, a jump button, a filter button, an attribute change button, and a video capture button. For example, when the menu button is pressed, various configurable pull-down menu screens are displayed on the display unit 28.

[0036] Users can make various settings by selecting menus on the menu screen displayed on the display unit 28 using the four directional buttons (up, down, left, and right) or the SET button.

[0037] The power control unit 41 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which block is energized, and detects whether a battery is installed, the type of battery, and the remaining battery level.

[0038] Furthermore, the power control unit 41 controls the DC-DC converter based on its detection results and instructions from the system control unit 50, supplying the necessary voltage to each part, including the recording medium 200, for the required period of time.

[0039] The power supply unit 40 consists of a primary battery, a secondary battery, an AC adapter, etc. The recording medium I / F 18 is an interface with the recording medium 200, such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of semiconductor memory, magnetic disks, etc.

[0040] Connector 80 is a contact for electrically connecting the camera body of the digital camera 100 to the lens unit 300. Connector 80 communicates control signals, status signals, data signals, etc., between the digital camera 100 and the lens unit 300, and also has the function of supplying power to the lens unit 300.

[0041] Next, we will describe the configuration of the lens unit 300.

[0042] In the lens unit 300, the imaging lens 301 is composed of multiple lenses, including a focus lens that adjusts focus by moving along the optical axis and a zoom lens that performs zooming. The aperture 302 adjusts the amount of light reaching the imaging unit 22 of the digital camera 100 by controlling the aperture opening.

[0043] The lens control unit 303 incorporates a CPU as a computer and controls the entire lens unit 300 in response to control signals from the system control unit 50 of the digital camera 100, including the drive control of the focusing lens, zoom lens, and aperture 302 of the imaging lens 301. It also has memory for storing constants, variables, and computer programs for operation. Furthermore, it has non-volatile memory that stores identification information such as a unique number for the lens unit 300, management information, functional information such as the maximum aperture value, minimum aperture value, focal length, and aberration amounts such as distortion, as well as current and past setting values.

[0044] Connector 304 is a contact for electrically connecting the lens unit 300 to the camera body of the digital camera 100. Connector 304 communicates control signals, status signals, data signals, etc., between the digital camera 100 and the lens unit 300, and also has the function of receiving power voltage or current from the digital camera 100.

[0045] Next, the details of the method for capturing panning panoramic images and the method for combining those images will be explained using Figures 2 and 3. Figure 2 is a diagram illustrating the panning direction and the image cropping area in this embodiment. Figure 3 is a diagram illustrating the processing flow of panoramic imaging and its correspondence with image data in this embodiment. Figures 2 and 3 show multiple images obtained from panning imaging performed at a constant speed, with similar spacing between them.

[0046] First, the cropping area (image range) of the image data captured by the imaging unit 22 will be explained using Figure 2. The hatched area in Figure 2 indicates the cropping (acquisition) area of ​​the image data, and the black-filled arrows in Figure 2 indicate the direction of panning. Figure 2(a) shows the effective image area of ​​the imaging unit 22, where W is the number of effective pixels in the horizontal direction and H is the number of effective pixels in the vertical direction. Figure 2(b) shows the cropping area of ​​the image data, where Wcrop is the width (number of pixels) of the cropping area in the horizontal direction and Hcrop is the width (number of pixels) of the cropping area in the vertical direction. Figure 2(c) shows the cropping area of ​​the image when panning in the horizontal direction, where W > Wcrop for horizontal image data cropping and H = Hcrop for vertical image data cropping. Figure 2(d) shows the cropping area of ​​the image when panning vertically, where W = Wcrop for horizontal image data cropping and H > Hcrop for vertical image data cropping.

[0047] In this embodiment, the width of the cropping area of ​​the captured image data is switched according to the distance of the subject included in the image. Specifically, in horizontal panning, the width of the image cropping area is set to Wcrop1 > Wcrop2, and if there are no subjects closer than a predetermined distance, the image is cropped with the wider width Wcrop1. If there are subjects closer than a predetermined distance, the image is cropped with the narrower width Wcrop2. Similarly, in vertical panning, the width of the image cropping area is set to Hcrop1 > Hcrop2, and if there are no subjects closer than a predetermined distance, the image is cropped with the wider width Hcrop1. If there are subjects closer than a predetermined distance, the image is cropped with the narrower width Hcrop2.

[0048] Next, the image synthesis process will be explained. The system control unit 50 reads the captured, cropped, and saved image data from the memory 32. Subsequently, the system control unit 50 detects the positional misalignment between the captured images. Positional misalignment is detected by dividing the image into small blocks of an arbitrary size, calculating the corresponding point where the Sum of Absolute Difference (SAD) of brightness is minimized for each small block, and calculating the motion vector. However, this is not limited to this method; angular velocity information, position information, or attitude information detected by the detection unit 26 may also be used. Subsequently, the system control unit 50 calculates mapping parameters from the calculated motion vector, corrects the positional misalignment, and synthesizes the images by performing weighted addition on the overlapping parts between the images.

[0049] Figure 3 shows the processing flow of this embodiment and its correspondence with image data. In Figure 3, the areas hatched with dots schematically represent the area of ​​the subject closer than a predetermined distance, and the areas hatched with diagonal lines represent the cropped area of ​​the image data.

[0050] Figure 3(a) shows the position where the second shutter switch SW2 is pressed. The user has set the field of view to one end of the panoramic image to be combined. Figures 3(b) to (k) schematically show the state in which continuous imaging is being performed while panning the digital camera 100 toward the other end of the panoramic image to be combined.

[0051] In Figures 3(a) to (d), since there are no subjects (areas hatched with dots) that are closer than a predetermined distance, some of the captured images (1) to (4), specifically images (1) and (3), are cropped with a wide width Wcrop1 and used for panoramic stitching.

[0052] In Figures 3(e) through (h), since there are subjects (areas hatched with dots) that are closer than a predetermined distance, a narrow crop width Wcrop2 is used to crop all the captured images (5) through (8) for panoramic stitching. As will be described later, Wcrop2 may be an integer fraction of Wcrop1.

[0053] In Figures 3(i) through (k), since there are no subjects (areas hatched with dots) that are closer than a predetermined distance, some of the captured images (9) and (11) are cropped with a wide width Wcrop1 and used for panoramic stitching.

[0054] Figure 3(k) shows the state after the digital camera 100 has panned to the set angle and continuous imaging has ended. Figure 3(l) shows a panoramic image created by combining multiple captured images (still images).

[0055] As described above, the area containing a subject in the foreground (closer to the viewer), which is susceptible to parallax, is cropped and composited with a narrow width, while the area without a subject in the foreground (closer to the viewer) is cropped and composited with a wider width. By doing so, it is possible to reduce the compositing misalignment in the foreground (closer to the viewer) without degrading the overall image quality.

[0056] Figure 4 is a flowchart illustrating the processing flow in this embodiment.

[0057] The process shown in Figure 4 is executed when the user selects the panoramic imaging mode using the mode switch 60. Each step in Figure 4 is performed by the system control unit 50, which reads a predetermined program from the non-volatile memory 56 and loads it onto the system memory 52, thereby controlling the operation and processing of each part that makes up the digital camera 100.

[0058] In step S101, the system control unit 50 determines whether the panning condition setting button has been pressed as a result of operating the operation unit 70. The panning condition setting button is one of the function buttons implemented by the operation unit 70. If the panning condition setting button is pressed (YES in S101), the system control unit 50 proceeds to step S102; if the panning condition setting button is not pressed (NO in S101), the system control unit 50 proceeds to step S103.

[0059] In step S102, the system control unit 50 accepts a change in the imaging conditions (such as the direction of panning) for panning imaging made by the user, stores the accepted conditions in the non-volatile memory 56, and then proceeds to step S103. If the panning condition setting button is not pressed and it is the first time panning imaging is performed, the default setting conditions are applied.

[0060] In step S103, the system control unit 50 determines whether the first shutter switch 62 of the shutter button 61 is turned on (whether the first shutter switch signal SW1 is generated). If the first shutter switch 62 is turned on (YES in S103), the system control unit 50 proceeds to step S104; if the first shutter switch 62 is not turned on (NO in S103), the process returns to step S101.

[0061] In step S104, the system control unit 50 performs AE processing and AF processing (autofocus adjustment processing). In the following step S105, the system control unit 50 determines the imaging conditions (exposure, image sensitivity, white balance, etc.) based on the AE processing results in step S104. In the subsequent step S108, imaging is performed with the imaging lens 103 fixed to the lens position at which the subject was focused during the AF processing in step S104.

[0062] Next, in step S106, the system control unit 50 determines whether the second shutter switch 63 is turned on or not (whether the second shutter switch signal SW2 is generated or not). If the second shutter switch 63 is not turned on (NO in S106), the system control unit 50 proceeds to step S107, and if the second shutter switch 63 is turned on (YES in S106), the system control unit 50 proceeds to step S108.

[0063] In step S107, the system control unit 50 determines whether the first shutter switch 62 remains ON or OFF. If the first shutter switch 62 remains ON (YES in S107), the system control unit 50 returns to step S106 and repeats until the second shutter switch 63 is turned ON in step S106. If the first shutter switch 62 is OFF (NO in S107), the system control unit 50 returns to step S101.

[0064] In step S108, the system control unit 50 controls each part of the digital camera 100 to perform imaging processing (capture of one image). At that time, the disparity images A and B are acquired simultaneously with the captured image in a single exposure and stored in the memory 32.

[0065] In step S109, the system control unit 50 uses the disparity images A and B captured in step S108 to obtain distance information of the subject using the image processing unit 24. Specifically, the distance to the subject is obtained by detecting the phase difference from the two input disparity images, and distance information from the imaging unit to the subject is obtained pixel by pixel.

[0066] In step S110, the system control unit 50 determines from the distance information acquired in step S109 whether or not there are subjects closer than a predetermined distance within the area used for panoramic image stitching. If there are no subjects closer than a predetermined distance (NO in S110), the system control unit 50 proceeds to step S111. If there are subjects closer than a predetermined distance (YES in S110), the system control unit 50 proceeds to step S113.

[0067] In step S111, the system control unit 50 determines whether or not to use the images captured in step S108 for panoramic stitching. As explained with reference to Figure 3, if there are no subjects closer than a predetermined distance within the area to be used for panoramic stitching, the system control unit 50 uses some of the images without subjects for stitching and crops the images to create a wide field of view. For example, even-numbered images without subjects closer than a predetermined distance within the area to be used for panoramic stitching are not used, and every other image is used, so that one out of N images is cropped and used for stitching at integer intervals. If the system control unit 50 determines that the images captured in step S108 should be used for panoramic stitching (YES in S111), it proceeds to step S112. If the images should not be used for panoramic stitching (NO in S111), it proceeds to step S112.

[0068] In step S112, the system control unit 50 uses the image processing unit 24 to cut out the captured image data into strips of a first width according to the panning direction, which will be used for panoramic stitching.

[0069] In step S113, the system control unit 50 uses the image processing unit 24 to cut out the captured image data into strips with a second width narrower than the first width, according to the panning direction, as an area to be used for panoramic stitching. In this case, if in step S111 one of the N images captured in step S108 is used for stitching, the second stitching width should be set to 1 / N of the first stitching width.

[0070] Then, in step S114, the system control unit 50 synthesizes a panoramic image by sequentially combining the strip-shaped images cut out in step S112 or step S113. At this time, each time imaging is performed, the panoramic image is synthesized by adding the strip-shaped images cut out from the image obtained in the current imaging to the edges of the synthesized image obtained from the previous imaging. The image data of the panoramic image thus synthesized is stored in the memory 32.

[0071] Note that when processing the first image, step S114 in the flowchart shown in Figure 2 is omitted.

[0072] Next, in step S115, the system control unit 50 detects the angle of the panning direction of the digital camera 100 (hereinafter referred to as "camera angle").

[0073] In step S116, the system control unit 50 determines whether the camera angle detected in step S115 is the panning end angle. If the camera angle is the panning end angle (YES in S116), the system control unit 50 terminates the processing of this flow. If the camera angle is not the panning end angle (NO in S116), the processing proceeds to step S117. If the determination in step S116 is YES, the desired panoramic image has been obtained, and the image is recorded on the recording medium 200 to end the process.

[0074] In step S117, the system control unit 50 determines whether panning has been completed based on the camera angle detected in step S115. Specifically, this is done by comparing the angle of the digital camera 100 at the time of imaging with the angle of the digital camera 100 at the time of imaging one frame prior, and determining whether the angle has increased in the panning direction. If the angle has not increased in the panning direction, it is determined that panning has been completed.

[0075] Furthermore, the method for determining the end of the process described above is merely an example, and various modifications can be applied. For example, the imaging unit 22 may take an image while the second shutter switch 63 is pressed, and the system control unit 50 may terminate panning when the second shutter switch 63 is released. Alternatively, the system control unit 50 may terminate panning when the imaging unit 22 has taken a predetermined number of images.

[0076] If the system control unit 50 determines that panning has been completed (YES in S117), the processing of this flow ends. Note that if the determination in step S117 is YES, it means that panoramic imaging has been interrupted midway, and the images synthesized up to that point are recorded on the recording medium 200 and the process ends. If the system control unit 50 determines that panning has not been completed (NO in S117), the process returns to step S108. As a result, the processing in steps S108 to S116 is repeated until imaging is completed, so that continuous imaging with panning is performed.

[0077] The above description states that the system control unit 50 selects either a first width or a second width depending on whether there is a subject closer than a predetermined distance, but it is not limited to this. For example, the system control unit 50 may continuously change the cropping width based on the subject distance of the subject.

[0078] According to this embodiment, cropping and compositing of an image can be performed with a narrow width in areas where a subject in the foreground (closer) that is susceptible to parallax is present, and with a wider width in areas where a subject in the foreground (closer) is not present. In this way, it is possible to reduce the compositing misalignment in the foreground (closer) subject without degrading the overall image quality.

[0079] (Second embodiment) A second embodiment of the present invention will now be described.

[0080] In the first embodiment, if there are no subjects closer than a predetermined distance within the area used for panoramic image stitching, a wide first width image is cropped from a portion of the captured image. If there are subjects closer than a predetermined distance within the area used for panoramic image stitching, a second width image, narrower than the first width image, is cropped from all images. In this way, panoramic stitching is performed using images cropped with a narrow width in the area where subjects closer to the viewer (who are susceptible to parallax) are present, and with a wide width in the area where no such subjects are present.

[0081] In contrast, in this embodiment, when a subject is located closer than a predetermined distance within the area used for panoramic image stitching, the image is not only cropped with a second width, but also cropped with a wider first width from a portion of the image. Then, the portion of the first width image that does not contain a subject in the foreground (closest) is cropped, and the portion of the image obtained by stitching together the second width images that contain a subject in the foreground (closest) is cropped and then stitched together.

[0082] Figure 5 is a diagram illustrating the correspondence between the panoramic imaging processing flow and image data in this embodiment. In Figure 5, the areas hatched with dots schematically represent the area of ​​the subject closer than a predetermined distance, and the areas hatched with diagonal lines indicate the cropped area of ​​the image data.

[0083] Figure 5(a) shows the position where the second shutter switch SW2 is pressed. The user has set the field of view to one end of the panoramic image to be combined. Figures 3(b) to 3(k) schematically show the state in which continuous imaging is being performed while panning the digital camera 100 toward the other end of the panoramic image to be combined.

[0084] In Figures 5(a) to (d), since there are no subjects (areas hatched with dots) that are closer than a predetermined distance, some of the captured images (1) to (4), specifically images (1) and (3), are cropped with a wide width Wcrop1 and used for panoramic stitching.

[0085] In Figures 5(e) to (h), since there are subjects (areas hatched with dots) that are closer than a predetermined distance, cropping is performed on all images from (5) to (8) using a narrow width Wcrop2. Furthermore, cropping is performed on some of the images (5) and (7) from the captured images (5) to (8) using a wider width Wcrop1. Then, the area containing subjects closer than a predetermined distance from the image cropped and combined from (5) and (6) using Wcrop2, and the area not containing subjects closer than a predetermined distance from the image cropped from (5) using Wcrop1 are cropped and combined for use in panorama stitching. Similarly, the area containing subjects closer than a predetermined distance from the image cropped and combined from (7) and (8) using Wcrop2, and the area not containing subjects closer than a predetermined distance from the image cropped from (7) using Wcrop1 are cropped and combined for use in panorama stitching. Note that here, we are cropping the area cropped with Wcrop1 twice with Wcrop2, but you can increase the number of times you crop with Wcrop2.

[0086] In Figures 5(i) to (k), since there are no subjects (areas hatched with dots) that are closer than a predetermined distance, some of the captured images (9) and (11) are cropped with a wide width Wcrop1 and used for panoramic stitching.

[0087] Figure 5(k) shows the state after the digital camera 100 has panned to the set angle and continuous imaging has ended. Figure 5(l) shows a panoramic image created by combining multiple captured images (still images).

[0088] In this way, within the region where a subject in the foreground (closest) that is susceptible to parallax exists, the part containing the foreground (closest) subject can be extracted and composited from an image created by combining narrow-width images, while the part without the foreground (closest) subject can be extracted and composited from a wider-width image. This makes it possible to reduce the compositing misalignment in the foreground (closest) subject without degrading the overall image quality.

[0089] Figure 6 is a flowchart illustrating the processing flow in this embodiment.

[0090] The following explanation will omit details of processes that are the same as those in the first embodiment.

[0091] The process shown in Figure 6 is executed when the user selects the panoramic imaging mode using the mode switch 60.

[0092] Each step in Figure 6 is performed by the system control unit 50 loading a predetermined program read from the non-volatile memory 52 onto the system memory 56, and controlling the operation and processing of each part that makes up the digital camera 100.

[0093] Steps S201 to S209 are identical to steps S101 to S109 in the first embodiment, so their description is omitted.

[0094] In step S210, the system control unit 50 determines from the distance information acquired in step S209 whether or not there are subjects closer than a predetermined distance within the area used for panoramic image stitching. If there are subjects closer than a predetermined distance (YES in S210), the system control unit 50 proceeds to step S211; if there are no subjects closer than a predetermined distance (NO in S210), the system control unit 50 proceeds to step S217.

[0095] In step S211, the system control unit 50 determines whether or not to crop an image from the image captured in step S208 with a first width and use it for panoramic stitching. As explained with reference to Figure 5, if there is a subject closer than a predetermined distance within the area used for panoramic stitching, a wider image is cropped from a portion of the image captured in step S208 and used for stitching. For example, if there is a subject closer than a predetermined distance within the area used for panoramic stitching, a wider image is cropped from an odd-numbered image and used for stitching, so that at least one image out of N is cropped with a wider width and used for stitching. If the system control unit 50 decides to crop an image from the image captured in step S208 with a first width and use it for panoramic stitching (YES in S211), it proceeds to step S212. If the system control unit 50 decides not to crop an image from the image captured in step S208 with a first width and use it for panoramic stitching (NO in S211), it proceeds to step S213.

[0096] In step S212, the system control unit 50 uses the image processing unit 24 to cut the captured image data into strips of a first width according to the panning direction, as an area to be used for panoramic stitching, and stores them in the memory 32.

[0097] In step S213, the system control unit 50 uses the image processing unit 24 to crop the captured image data into strips with a second width narrower than the first width according to the panning direction, as an area to be used for panoramic stitching, and stores them in the memory 32. In this case, if in step S211 one of the N images captured in step S108 is cropped with a wider width for use in stitching, the second cropping width should be set to 1 / N of the first cropping width.

[0098] In step S214, the system control unit 50 determines whether the image cropped with the second width in step S213 can be combined. As explained with reference to Figure 5, if there is a subject closer than a predetermined distance within the area used for panoramic image stitching, the narrow-width images cropped from the image captured in step S208 are combined and used for stitching with the image cropped with a wider width. For example, a narrow-width image is cropped from an image in which a subject closer than a predetermined distance exists within the area used for panoramic stitching, stored in memory 32, and once a predetermined number of narrow-width images have been accumulated, the narrow-width images are combined.

[0099] If the image cropped with the second width in step S213 is suitable for synthesis (YES in S214), the system control unit 50 proceeds to step S215. If the image cropped with the second width in step S213 is not suitable for synthesis (NO in S214), the system control unit 50 proceeds to step S220.

[0100] In step S215, the system control unit 50 uses the image processing unit 24 to combine the second width images that were cropped in step S213 and stored in the memory 32, and stores them in the memory 32.

[0101] In step S216, the system control unit 50 further combines the composite image obtained by combining the first width image cropped in step S212 and the second width image stored in the memory 32. At this time, from the first width image cropped in step S212, the portion where no subjects closer than a predetermined distance exist is cropped. Also, from the composite image obtained by combining the second width image in step S215, the portion where no subjects closer than a predetermined distance exist is cropped, and by combining these, a strip-shaped composite image to be used for panoramic stitching is generated.

[0102] In step S217, the system control unit 50 determines whether or not to use the image captured in step S208 for panoramic stitching. As explained with reference to Figure 5, if there are no subjects closer than a predetermined distance within the area to be used for panoramic stitching, only a portion of the image captured in step S208 is used, and only the widest image is cropped and stitched together. For example, even-numbered images of images in which there are no subjects closer than a predetermined distance within the area to be used for panoramic stitching are not used, and every other image is used. If the system control unit 50 decides to use the image captured in step S208 for panoramic stitching (YES in S217), it proceeds to step S218; if the image is not to be used for panoramic stitching (NO in S217), it proceeds to step S220.

[0103] In step S218, the system control unit 50 uses the image processing unit 24 to cut out the captured image data into strips of a first width according to the panning direction, which will be used for panoramic stitching.

[0104] Then, in step S219, the system control unit 50 synthesizes a panoramic image by sequentially combining the strip-shaped images synthesized in step S216 or the strip-shaped images cut out in step S218. At this time, each time imaging is performed, the panoramic image is synthesized by adding the strip-shaped images cut out from the image obtained in the current imaging to the edges of the synthesized image obtained from the previous imaging. The image data of the panoramic image thus synthesized is stored in the memory 32.

[0105] Next, in step S220, the system control unit 50 detects the angle of the panning direction of the digital camera 100 (hereinafter referred to as "camera angle").

[0106] In step S221, the system control unit 50 determines whether the camera angle detected in step S220 is the panning end angle. If the camera angle is the panning end angle (YES in S221), the system control unit 50 terminates the processing of this flow; if the camera angle is not the panning end angle (NO in S221), the processing proceeds to step S222. If the determination in step S221 is YES, the desired panoramic image has been obtained, and the image is recorded on the recording medium 200 to end the process.

[0107] In step S222, the system control unit 50 determines whether panning has been completed based on the camera angle detected in step S220. Specifically, this is done by comparing the angle of the digital camera 100 at the time of imaging with the angle of the digital camera 100 at the time of imaging one frame prior, and determining whether the angle has increased in the panning direction. If the angle has not increased in the panning direction, it is determined that panning has been completed.

[0108] If the system control unit 50 determines that panning has been completed (YES in S222), the processing of this flow ends. Note that if the determination in step S222 is YES, panoramic imaging is interrupted midway, and the images synthesized up to that point are recorded on the recording medium 200 and the process ends. If the system control unit 50 determines that panning has not been completed (NO in S222), the process returns to step S208. As a result, the processing in steps S208 to S221 is repeated until imaging is completed, allowing for continuous imaging while panning.

[0109] Similar to the first embodiment, the method for determining the end of imaging and the method for determining the cropping width described above are merely examples, and various modifications can be applied.

[0110] According to this embodiment, within the region where a subject in the foreground (closest) that is susceptible to parallax exists, the portion containing the foreground (closest) subject can be extracted and composited from an image created by combining narrow-width images, while the portion without the foreground (closest) subject can be extracted and composited from a wider-width image. In this way, it is possible to reduce the compositing misalignment in the foreground (closest) subject without degrading the overall image quality.

[0111] (Other embodiments) Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0112] The above embodiments have been described based on implementation with a digital camera, but are not limited to digital cameras. For example, they may be implemented with a portable device that has a built-in image sensor, or with a network camera that can capture images.

[0113] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and operate the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0114] 13 D / A Converters 15 Memory Control Unit 18 Recording medium I / F 20 shutters 22 Imaging Department 23 AF evaluation value detection unit 24 Image Processing Unit 26 Detection unit 28 Display section 32 memory 40 Power supply section 41 Power supply control unit 50 System Control Unit 52 Non-volatile memory 53 System Timer 56 System Memory 60 Mode Switch 61 Shutter button 62 First shutter switch 63 Second shutter switch 70 Operation section 72 Touch panel operation unit 80 connectors 100 Digital Cameras 200 recording media 300 Lens Unit 301 Imaging Lens 302 aperture 303 Lens control unit 304 connector

Claims

1. A means for combining regions extracted from each of multiple images, The system includes an acquisition means for acquiring distance information of a subject from at least a portion of the plurality of images, The synthesis means determines the width of the region to be cut out from at least some of the images among the plurality of images, based on the distance information of the image acquired by the acquisition means. An image processing apparatus characterized in that, when the width of the region to be cut out from a first image having first distance information is a first width, the second width, which is the width of the region to be cut out from a second image having second distance information that is closer than the first distance information, is narrower than the first width.

2. The image processing apparatus according to claim 1, wherein the synthesis means performs synthesis using a further cropped region for an image among the plurality of images in which different distance information exists within the region used for synthesis.

3. The image processing apparatus according to claim 2, characterized in that the synthesis means is such that, among the plurality of images, the number of times the first cropping is performed on a first portion having a third distance information within the region of an image in which different distance information exists within the region used for synthesis is less than the number of times the second cropping is performed on a second portion having a fourth distance information that is closer than the third distance information.

4. The synthesis means crops a third image from among the plurality of images that has distance information that is further than a predetermined distance by a third width, The image processing apparatus according to any one of claims 1 to 3, characterized in that it crops a fourth image having distance information that is closer than the predetermined distance with a fourth width narrower than the third width.

5. The image processing apparatus according to claim 4, characterized in that the fourth width is an integer fraction of the third width.

6. An imaging means for capturing multiple images, A means for combining regions extracted from each of multiple images, The system includes an acquisition means for acquiring distance information of a subject from at least a portion of the plurality of images, The synthesis means determines the width of the region to be cut out from at least some of the images among the plurality of images, based on the distance information of the image acquired by the acquisition means. The synthesis means crops a region of the first image with a first width from among the plurality of images, where the distance information is farther than a predetermined distance. An imaging device characterized by cropping a region of a second image with distance information that is closer than the predetermined distance, using a second width that is narrower than the first width.

7. The imaging device according to claim 6, characterized in that the plurality of images are images captured by panning the imaging device.

8. The imaging apparatus according to claim 7, characterized in that the panning is performed at a constant speed and the intervals between the plurality of images are the same.

9. The imaging apparatus according to any one of claims 6 to 8, characterized in that the synthesis means is used for synthesis at integer intervals from the first images.

10. The imaging apparatus according to any one of claims 6 to 8, characterized in that the second width is an integer fraction of the first width.

11. The synthesis means uses every N images from the first image for the synthesis, The imaging apparatus according to any one of claims 6 to 8, characterized in that the second width is 1 / N of the first width.

12. A compositing step that combines regions extracted from each of multiple images, The process includes an acquisition step of acquiring distance information of a subject from at least a portion of the plurality of images, In the synthesis step, for at least some of the multiple images, the width of the region to be cut out from the image is determined based on the distance information of the image acquired in the acquisition step. An image processing method characterized in that, when the width of the region to be cut out from a first image having first distance information is a first width, the second width, which is the width of the region to be cut out from a second image having second distance information that is closer than the first distance information, is narrower than the first width.

13. A program that causes a computer to perform an image processing method, A compositing step that combines regions extracted from each of multiple images, The procedure involves an acquisition step of obtaining distance information of a subject from at least a portion of the aforementioned plurality of images, In the synthesis step, for at least some of the multiple images, the width of the region to be cut out from the image is determined based on the distance information of the image acquired in the acquisition step. A program characterized in that, when the width of the region to be cut out from a first image having first distance information is a first width, the second width, which is the width of the region to be cut out from a second image having second distance information that is closer than the first distance information, is narrower than the first width.

14. A storage medium that is readable by a computer and stores the program described in claim 13.

Images