Imaging apparatus and method for controlling imaging apparatus

Abstract

To provide an imaging apparatus that can perform image composition while preventing a change of a focus plane in a screen even when image shift occurs during photographing.SOLUTION: An imaging apparatus has: imaging means that picks up a plurality of images; acquisition means that acquires distance information in the images; selection means that selects feature points from the images; transformation means that performs projective transformation on the images; and composition means that combines the images. The selection means selects, by using the distance information, the feature points from an area within the depth of field of a main subject in a first image of the plurality of images. The transformation means performs projective transformation on a second image picked up after the first image, so as to correspond to the movement of the feature points in the first image. The composition means combines the first image with the second image on which the projective transformation is performed.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a method for controlling the imaging apparatus. [Background technology]

[0002] 2. Description of the Related Art Conventionally, in imaging devices such as digital cameras, an image synthesis method is known in which continuously captured images are synthesized in order to suppress camera shake in a captured image.

[0003] For example, Patent Document 1 discloses a technology in which a tilt angle at each capture is calculated from the displacement and movement of image shift between multiple captured images obtained by multiple captures, and keystone correction is performed based on the tilt angle, thereby correcting image shift and synthesizing the images.

[0004] In Patent Document 2, for multiple captured images, the area in the image is divided into distance groups based on subject distance information obtained by a distance information acquisition means and the calculated displacement and movement amount of image misalignment between images, and geometric deformation is performed using a conversion coefficient for each distance group. This discloses a technology for correcting image misalignment according to the distance of each subject in the captured images and synthesizing the images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-220745 A [Patent Document 2] Patent No. 6415228 Summary of the Invention [Problem to be solved by the invention]

[0006] In the conventional technology disclosed in Patent Document 1, image misalignment between images is corrected by coordinate transformation using keystone correction to synthesize the images. However, there is no specific description of the selection method or criteria for the reference point of the coordinates during projective transformation such as keystone correction.

[0007] The amount of movement due to tilt within the screen varies depending on the subject distance from the photographer. If points with different subject distances are selected as reference points for projective transformation, projective transformation using reference points with different distances may result in parts of the image with a different amount of movement from the original. In this case, the image corrected by projective transformation may look unnatural compared to the original image.

[0008] In addition, in Patent Document 2, a projective transformation (geometric transformation) is performed by changing a transformation coefficient for an area divided by distance group, and image synthesis is performed by positioning the image using a projective transformation (geometric transformation) suitable for each subject of a plurality of distance groups in the image. However, by performing synthesis processing using images transformed by distance group, blurring that is not dependent on the subject distance on the image occurs, and there is a possibility that the image will look unnatural and different from normal photography.

[0009] In view of the above-mentioned problems, the present invention aims to provide an imaging device that enables image synthesis with suppressed change in the focal plane within the screen when synthesizing images by taking multiple shots, even when images are misaligned during shooting due to tilt or the like. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, an imaging device according to the present invention comprises an imaging means for capturing a plurality of images, an acquisition means for acquiring distance information within the images, a selection means for selecting feature points from within the images, a transformation means for performing projective transformation on the images, and a synthesis means for synthesizing the images, wherein the selection means uses the distance information to select feature points from an area within a depth of field of a main subject in a first image of the plurality of images, the transformation means performs projective transformation on a second image captured after the first image so as to correspond to movement of the feature points in the first image, and the synthesis means synthesizes the first image with the second image after the projective transformation. Effect of the Invention

[0011] According to the present invention, it is possible to provide an imaging device which, when taking multiple shots to synthesize images, enables image synthesis with reduced change in the focal plane within the screen in the composite image, by image correction using projective transformation based on a reference point within the depth of field, even when image misalignment occurs during shooting due to tilt or the like. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an imaging apparatus 100 according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a conceptual diagram of a unit pixel 200 according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a partial block diagram of an image signal processing unit 106 according to the embodiment of the present invention. [Figure 4] 2 is a flow chart of an embodiment of the present invention. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of a first captured image according to an embodiment of the present invention. [Figure 6] FIG. 11 is an explanatory diagram showing an example of a second captured image according to an embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing an example of selected feature points in a first captured image according to the embodiment of the present invention. [Figure 8] FIG. 13 is a diagram showing an example of selecting a destination of a feature point in a second captured image according to an embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing an example of feature points selected from a first captured image using conventional technology. [Figure 10] FIG. 13 is a diagram showing an example of selecting a destination of a feature point in a second captured image according to the conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] In the embodiment, the imaging device is a digital (still) camera, but the imaging device may include electronic devices having an imaging function, such as movie cameras, smartphones with cameras, and tablet computers with cameras.

[0015] (First embodiment) FIG. 1 is a system block diagram of the entire imaging apparatus according to this embodiment.

[0016] 1, lens 101 is a lens that forms an optical image of a subject on image sensor 105, and zoom control, focus control, aperture control, etc. are performed by lens driver 102. Note that while lens 101 in FIG. 1 is shown as a single lens for convenience, it may be composed of multiple lenses.

[0017] A shutter 103 is a mechanical shutter that is controlled by a shutter driver 104 and controls the incidence time (exposure time) of an optical image on an image sensor 105 described later.

[0018] The image sensor 105 is an image sensor for capturing an optical image of a subject formed via an optical system constituted by the lens 101 etc., and fetching it as an image signal, and may be constituted by, for example, a CCD or CMOS sensor.

[0019] In this embodiment, the image sensor 105 has a plurality of unit pixels 200, which are divided pixels consisting of one microlens (described later) and a photodiode (PD) divided into multiple parts, and the plurality of unit pixels 200 are arranged in a matrix.

[0020] An image signal processing unit 106 performs digital gain processing for increasing gain, various corrections, data compression, etc., on the image signal output from the image sensor 105. It also performs various types of image processing such as various processes for acquiring information on the distance to the subject, and projection transformation and position alignment processing related to image synthesis.

[0021] The memory unit 107 temporarily stores image data and various other data.

[0022] The system control unit 108 is a control circuit such as a microcomputer equipped with a ROM and RAM (not shown), and controls the operation of each block of the imaging device. The system control unit 108 reads out the operation program of each block stored in the ROM, expands it in the RAM, and executes it to control the operation of each block.

[0023] The I / F unit 109 is an interface for recording or reading data on a recording medium 110, and the recording medium 110 is a removable recording medium such as a flash memory for recording or reading image data.

[0024] The display unit 111 displays various types of information, viewfinder images during shooting, shot images, GUI (Graphical User Interface) images for interactive operations, and the like.

[0025] 2 is a schematic diagram showing a unit pixel 200 of the image sensor 105. A pair of divided PDs (photodiodes) 201A and PD201B are arranged relative to a microlens 202. Light incident from pupil region 203A and pupil region 203B on exit pupil 203 is imaged on PD201A and PD201B, respectively, and output as image signals of each PD. Note that hereinafter, the image signal obtained from PD201A is referred to as signal A, and the image signal obtained from PD201B is referred to as signal B.

[0026] The image signal output from each PD obtained here is used to detect the image shift amount (pupil division phase difference) by known correlation calculations, etc., making it possible to measure distances based on the image plane phase difference. In addition, the unit pixel output obtained by adding up the image signal outputs from both PDs can also be used as a normal captured image output.

[0027] FIG. 3 is a block diagram showing an example of a main configuration of the image signal processing unit 106 related to this embodiment.

[0028] Distance information acquisition unit 301 is a distance information acquisition unit that performs known correlation calculations and phase difference detection from A signal and B signal, which are image signals of the pupil-divided PD of image sensor 105, and acquires distance information at the time of shooting in conjunction with the lens focal length, focus position, aperture value, etc. Regarding distance information, distance information is acquired corresponding to each of the regions obtained by dividing the image into a predetermined number of regions, and distance information to the main subject, such as a main subject region set by the photographer (user) or a main subject region set by known technology such as image recognition, is acquired.

[0029] At this time, the depth of field of the main subject etc. during shooting is also calculated by a known calculation using the permissible circle of confusion diameter based on the unit pixel size etc. of the image sensor 105, the lens aperture value etc. The obtained distance information and depth of field information are temporarily stored in the memory unit 107.

[0030] In the image synthesis shooting mode, the image detection unit 302 detects the presence or absence of a subject for each of the regions obtained by dividing the image into a predetermined number of regions using a method such as edge extraction in the image of a contour line, etc., for the first captured image serving as the alignment reference. Information on the subject in each of the regions divided into the predetermined number by the image detection unit 302 is temporarily stored in the memory unit 107.

[0031] In order to perform projective transformation on the first or subsequent captured image of a series of multiple images, the feature point selection unit 303 selects and sets feature points that serve as the basis for the projective transformation from the first captured image based on distance information from the distance information unit 301 and subject information from the image detection unit 302.

[0032] In this embodiment, feature points are selected from among the divided regions from regions where the subject is recognized by the image and where distance information within the depth of field of the main subject is obtained, and 16 points are selected in the region surrounding the main subject. The regions and coordinates of the selected 16 points are temporarily stored in memory unit 107. In this example, 16 feature points are temporarily selected, but this number is not limited, and multiple points that are possible within the shooting processing time, etc. may be selected.

[0033] The image matching unit 304 detects the movement destination of the feature points set by the feature point selection unit 303 for the first and subsequent images of a multiple shot shoot using a known technique such as template matching, and calculates a motion vector.

[0034] The projective transformation unit 305 performs a known projective transformation on the second or subsequent captured image of the multiple shots, based on the destination of the detected feature points. If the feature points selected by the feature point selection unit 303 are within a predetermined depth of field, i.e., within the depth of field of the main subject, and there are less than three feature points, the projective transformation may only involve scaling, rotation, and translation. In this example, a case has been described where the number of feature points selected within the predetermined depth of field is less than a predetermined number (less than three), but the number is not limited to this.

[0035] The synthesis processing unit 306 synthesizes the first captured image and the projectively transformed captured image. The synthesis process involves addition and averaging processes using multiple captured images corresponding to the types of image synthesis shooting modes set in the imaging device.

[0036] The captured image processing unit 307 generates a normal captured image by adding together the A signal and the B signal of the image signals of each PD.

[0037] Next, the selection of feature points and the correction operation by projective transformation in image synthesis will be described.

[0038] Fig. 5 shows an example of the first captured image, and Fig. 6 shows an example of the second captured image, with a person as the main subject and trees and buildings in the image. The second image shows an image that was affected by tilt when shooting. The dashed lines in the screen indicate the divided areas for obtaining distance measurement information and subject information, and in this example, are divided vertically and horizontally into 12 x 10. The number of divisions is not limited to this number.

[0039] 7 and 8 show the selection of feature points and the correction operation of projective transformation in this embodiment, and FIGS. 9 and 10 show an example of the prior art for comparison with this embodiment.

[0040] Fig. 7 shows that four feature points are selected from the first captured image by feature point selection unit 303. The black dots in the figure represent feature points, and the trees on the left and right sides within the depth of field of the main subject human are selected as feature points, and projective transformation and image synthesis processing are performed. If the area in which the divided regions of the screen are located is represented as (x, y), and the area from the top left is (1,1), its immediate right is (1,2), and the area below is (1,2), then (1,3), (1,11), (9,3), and (9,11) in the figure are the areas selected as feature points.

[0041] Fig. 8 shows that the image matching unit 304 has selected a destination point for the feature point in the second captured image. The white dots in the figure represent the destination points for the feature point, and the black dots represent points with the same coordinates as the feature point selected in the first captured image. Note that for the area (1,3)(1,11), the feature point and the destination point for the feature point overlap, so black dots are not displayed. Coordinate conversion and alignment are performed by projective conversion of vectors that represent the amount of movement from the white dots set in the tree on the right side of the screen shown in Fig. 8 to the black dots.

[0042] In Fig. 8, projective transformation is performed based on feature points set on the trees within the depth of field of the main subject. Therefore, the amount of movement of the main subject and the image within the depth of field due to projective transformation does not differ, and a corrected image can be obtained in which the change in focus after projective transformation is suppressed.

[0043] Next, problems that arise when a conventional method different from that of this embodiment is used will be described with reference to FIGS.

[0044] Fig. 9 shows that four feature points are selected from the first captured image by the feature point selection unit 303. As described above, Fig. 9 shows an example of a conventional method that is different from the present embodiment shown in Figs.

[0045] The black dots in Figure 9 represent feature points, and indicate that the areas (1,3), (1,11), (9,4), and (9,11) were selected as feature points because they are at different distances, i.e., the tree on the left side of the screen, which is not included in the depth of field, and the building on the right side of the screen.

[0046] Fig. 10 shows that the feature point selected in Fig. 9 has been selected as a destination point for the second captured image by image matching unit 304, and the white dots represent the destination points of the feature points. Note that for the area (1,3)(1,11), black dots are not displayed because the feature point and the destination point of the feature point overlap.

[0047] In Fig. 10, vector transformation is performed using projective transformation to convert the amount of movement from the white dots on the building on the right side of the screen to the black dots. By using a building located farther away from the photographer as a part of the reference point, the amount of movement of the building in the depth direction of the screen is smaller than the amount of movement of projective transformation using feature points within the depth of field shown in Fig. 8. As a result, the amount of movement is too small for the tree in the foreground, which is within the depth of field. This may result in an image corrected using projective transformation that has a change in focus compared to the original image, making it look unnatural.

[0048] Therefore, instead of selecting feature points at different distances, feature points that are within the depth of field described using Figures 7 and 8 are selected and then projective transformation is performed, thereby making it possible to obtain a corrected image in which change in focus after projective transformation is suppressed.

[0049] Next, the processing operation in the image synthesis mode will be described with reference to the flowchart shown in Fig. 4. This flowchart is executed by the system control unit 108 reading and executing a predetermined program and controlling each unit of the imaging device. First, in response to the setting to the image synthesis mode and the start of shooting being performed by an operation unit (not shown), image synthesis mode shooting is started.

[0050] In this embodiment, an image synthesis mode in which multiple shots with short exposure times are taken and then added together to produce a long exposure shot will be described. For convenience, image synthesis using two shots will be described here. In the image synthesis mode, focus control is performed prior to the synthesis shot, and synthesis shooting using multiple shots is started from a focus state on the main subject intended by the photographer.

[0051] First, in S401, the image capturing device captures the first image of a plurality of images. Under control of the system control unit 108, the image sensor 105 outputs image signals so that A and B signals are output for each pupil division pixel of each divided PD.

[0052] Next, in S402, distance information acquisition unit 301 performs correlation calculation and phase difference detection using the A signal and B signal of the pupil division image signal for each divided PD to acquire distance information. Calculation and detection processing are performed for each region obtained by dividing the image into a predetermined number of regions, and distance information for each of the predetermined number of divided regions and the depth of field for each region including the main subject are acquired.

[0053] In S403, the captured image processing unit 307 sums the pair of A and B signals as unit pixel signals to generate a normal captured image signal, and then the image detection unit 302 detects the edges of the subject from the image. Note that, since the main subject is in focus using a distance measurement method based on the phase difference of the pupil division, sufficient contrast is obtained to perform edge detection. The image that has been converted into the normal captured image signal is also temporarily stored in the memory unit 107.

[0054] In S404, the feature point selection unit 303 sets feature points using the depth of field information of the distance measurement information acquired from the distance information acquisition unit 301 in S402 and the edge information of the subject acquired by the image detection unit 302 in S403.

[0055] In S405, the second of the multiple shots is taken by the imaging device. The subsequent image shots, including the second, are taken under the same shooting conditions such as the aperture value and shooting time, but the image sensor 105 outputs a normal image signal obtained by adding together the A and B signals of the pupil-divided image signals.

[0056] Here, by using the normal image summed by the image sensor 105, it is possible to improve the signal-to-noise ratio of the captured image and to shorten the overall processing time for composite shooting by eliminating the need for summation processing time by the captured image processing unit. Note that, as with the first shot, the captured image may be output as an image for each pupil division pixel and processed in the same manner.

[0057] In S406, the image matching unit 304 detects a point in the second captured image that is the destination of the feature point set in S403. As a detection method, for example, it is possible to perform matching of the area of ​​the feature point selected from the first captured image with the same area in the two captured images and select the coordinates of the destination of the feature point, but is not limited to this.

[0058] In S407, the feature point selection unit 303 sets feature points for the second captured image as reference points for performing projective transformation. For the coordinates of the destinations of each of the feature points selected in S406, four points in total are selected from those set as the destinations of the feature points, one point from each direction close to the four corners of the image, that are highly correlated. In addition, the selection may be performed by calculating the standard deviation of the output signal in each region and selecting the output signal with the largest standard deviation value, thereby selecting the output signal with the largest contrast difference. In addition, the output signal whose distance from the four corners of the image falls within a predetermined range may be preferentially selected.

[0059] In S408, the projective transformation unit 305 performs projective transformation on the second captured image. Specifically, using the four coordinates of the destinations of the feature points set in S407, projective transformation is performed on the second captured image, converting it into a position-corrected captured image.

[0060] In S409, the synthesis processing unit 306 performs addition processing on the first normally captured image in S403 and the second captured image that has been projectively transformed and positionally corrected in S408, to generate a synthesized image.

[0061] According to the embodiment described above, when taking multiple shots to synthesize images, even if images are misaligned during shooting due to tilt or the like, image synthesis can be performed with reduced changes in the focal plane on the screen when the composite image is created by image correction using projective transformation based on a reference point within the depth of field.

[0062] In this embodiment, the image sensor having pupil division pixels is configured to acquire distance information, but the present invention is not limited to this. For example, a sensor capable of acquiring distance information separately, such as a TOF (Time Of Flight) distance sensor, may be mounted, and the distance information from the sensor and the captured image may be combined to perform projective transformation and image synthesis. In addition, although an example in which projective transformation is calculated from image misalignment has been described, the present invention is not limited to this. For example, the camera may be mounted with an acceleration sensor or an angular velocity sensor, and projective transformation may be performed based on the detection information of these sensors.

[0063] (Other embodiments) Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0064] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and run the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0065] 100 Imaging device 106 Image signal processing unit 200 unit pixels 301 Distance information acquisition section 302 Image detection unit 303 Feature Point Selection Unit 304 Image Matching Unit 305 Projection transformation section 306 Synthesis Treatment Unit 307 Image Processing Department

Claims

1. An imaging means for capturing a plurality of images; An acquisition means for acquiring distance information within the image; A selection means for selecting feature points from within the image; A transformation means for performing a projective transformation on the image; A synthesis means for synthesizing images, the selection means uses the distance information to select feature points from an area within a depth of field of a main subject in a first image of the plurality of images; the transformation means performs projective transformation on a second image captured after the first image so as to correspond to the movement of the feature point in the first image; The synthesizing means synthesizes the first image and the second image obtained by performing the projective transformation.

1. An imaging device comprising:

2. 2. The imaging apparatus according to claim 1, wherein the first image is an image captured first among a plurality of images used for the composition by the composition means.

3. 2. The image processing apparatus according to claim 1, wherein said selection means selects said feature points based on a standard deviation of an output signal in each region of said image.

4. 2. The imaging apparatus according to claim 1, wherein the selection means selects the feature points by giving priority to those within a predetermined range of distances from the four corners of the image.

5. 2. The imaging apparatus according to claim 1, wherein the transformation means performs the projection transformations of enlargement / reduction, rotation, and translation when the number of the feature points selected by the selection means is less than a predetermined number.

6. The imaging apparatus according to claim 1 , wherein the main subject is set by a user's selection or by image recognition.

7. An imaging step of capturing a plurality of images; acquiring distance information within the image; a selection step of selecting feature points within the image; a transformation step of performing a projective transformation on the image; A synthesis step of synthesizing the images, In the selecting step, a feature point is selected from a region within a depth of field of a main subject in a first image of the plurality of images using the distance information; In the transformation step, a projective transformation is performed on a second image captured after the first image so as to correspond to the movement of the feature point in the first image; a combining step of combining the first image and the second image obtained by performing the projective transformation,

8. A program for causing a computer to execute each step of the method for controlling an image sensor according to claim 7.

9. A computer-readable storage medium storing the program according to claim 8.

Images