Subject tracking device, imaging device, subject tracking method, and program

Abstract

To provide a technique for highly accurately tracking a subject between photographed images.SOLUTION: A subject tracking device comprises: acquisition means which acquires a first histogram in a plurality of defocus amounts corresponding to a plurality of range finding points of a first photographed image, a first subject class of the first histogram to which the defocus amount in the first photographed image of the subject being the tracking object belongs, and a second histogram of the second photographed image captured subsequently to the first photographed image; calculation means which calculates the feature amount in the first type of the first subject class in the first histogram and the feature amount in the plurality of first types of the first classes in the second histogram; detection means which detects the specific first class on the basis of the feature amount in the first type of the first subject class from the first classes of the second histogram; and specification means which specifies the second subject class of the second histogram to which the defocus amount in the second photographed image of the subject being the tracking object belongs on the basis of the specific first class.SELECTED DRAWING: Figure 8B

Description

[Technical Field]

[0001] The present invention relates to a subject tracking device, an imaging device, a subject tracking method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for obtaining a frequency distribution (histogram) of subject distances (defocus amounts), selecting multiple subject distances based on the peaks of appearance frequency, and acquiring multiple captured images focused at each subject distance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-232181 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing continuous shooting using servo AF, users generally want to capture multiple images that are focused on a specific subject and are consecutive in the time direction. However, the technology described in Patent Document 1 does not allow for the capture of multiple images that are focused on a specific subject and are consecutive in the time direction.

[0005] The present invention has been made in view of the above circumstances, and has as its object to provide a technique that enables tracking of a subject between captured images (between frames) with high accuracy. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention provides an acquisition unit that acquires a first histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a first captured image, a first object class that is a class of the first histogram to which the defocus amounts of the object to be tracked in the first captured image belong, and a second histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a second captured image captured after the first captured image; and a first type feature amount of the first object class in the first histogram and a second histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a second captured image captured after the first captured image. 、 Provided is a subject tracking device comprising: a calculation means for calculating a plurality of first type feature amounts of a first plurality of classes of the second histogram; a detection means for detecting a specific first class from the first plurality of classes of the second histogram based on the first type feature amounts of the first subject class; and an identification means for identifying a second subject class, which is a class of the second histogram to which a defocus amount in the second captured image of the subject to be tracked belongs, based on the specific first class. [Effects of the Invention]

[0007] According to the present invention, it is possible to track a subject between captured images (between frames) with high accuracy.

[0008] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the configuration of an imaging device 10 equipped with an imaging control device. [Figure 2] 3 is a schematic diagram of an array of imaging pixels (and focus detection pixels) of the image sensor 122. [Figure 3] (a) A plan view of one pixel 200G of the image sensor 122 shown in Figure 2, viewed from the light receiving surface side (+z side) of the image sensor 122, (b) A cross-sectional view of the aa cross section of Figure 3(a) viewed from the -y side. [Figure 4] 4 is a schematic explanatory diagram showing the correspondence between the pixel structure and pupil division of the embodiment shown in FIG. 3. [Figure 5] FIG. 3 is a schematic diagram showing the correspondence between the image sensor 122 and pupil division. [Figure 6] 10 is a diagram showing a schematic relationship between the defocus amount based on the first focus detection signal and the second focus detection signal and the image shift amount between the first focus detection signal and the second focus detection signal. FIG. [Figure 7A] 1 is a flowchart of AF operation. [Figure 7B] 1 is a flowchart of AF operation. [Figure 8A] 10 is a conceptual diagram of the subject and histograms in frames 1 to 4. FIG. [Figure 8B] 10 is a conceptual diagram of matching of feature amounts of context and matching of feature amounts of shape in frames 1 to 4. FIG. [Figure 9] FIG. 10(a) is a diagram showing the evaluation value of the shape for the entire histogram of each frame, and FIG. 10(b) is a diagram showing the evaluation value of the shape change of each frame. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] (Configuration of imaging device 10) FIG. 1 is a block diagram showing the configuration of an imaging device 10 equipped with a subject tracking device. In the example of FIG. 1, the imaging device 10 is a single-lens reflex digital camera with interchangeable lenses. The imaging device 10 takes the form of a camera system having a lens unit 100 (interchangeable lens) and a camera body 120. The lens unit 100 is detachably attached to the camera body 120 via a mount M shown by a dotted line in FIG. 1. However, this embodiment is not limited to the configuration shown in FIG. 1 and can also be applied to an imaging device (digital camera) in which the lens unit (imaging optical system) and the camera body are integrated. Furthermore, this embodiment is not limited to digital cameras and can also be applied to other imaging devices such as video cameras.

[0012] Lens unit 100 has an optical system consisting of a first lens group 101, an aperture 102, a second lens group 103, a focus lens group (hereinafter simply referred to as "focus lens") 104, and a drive / control system. Thus, lens unit 100 is a photographing lens (image pickup optical system) that includes focus lens 104 and forms a subject image.

[0013] The first lens group 101 is disposed at the tip of the lens unit 100 and is held so as to be able to move back and forth in the optical axis direction OA. The diaphragm 102 adjusts the amount of light during shooting by adjusting its aperture diameter, and also functions as a shutter for adjusting the exposure time during still image shooting. The diaphragm 102 and the second lens group 103 are movable together in the optical axis direction OA, and a zoom function is achieved in conjunction with the forward and backward movement of the first lens group 101. The focus lens 104 is movable in the optical axis direction OA, and the subject distance (focusing distance) at which the lens unit 100 focuses changes depending on its position. Controlling the position of the focus lens 104 in the optical axis direction OA enables focus adjustment (focus control) to adjust the focusing distance of the lens unit 100.

[0014] The drive / control system includes a zoom actuator 111, an aperture actuator 112, a focus actuator 113, a zoom drive circuit 114, an aperture drive circuit 115, a focus drive circuit 116, a lens MPU 117, and a lens memory 118. The zoom drive circuit 114 uses the zoom actuator 111 to drive the first lens group 101 and the second lens group 103 in the optical axis direction OA, thereby controlling the angle of view of the optical system of the lens unit 100 (performing a zoom operation). The aperture drive circuit 115 uses the aperture actuator 112 to drive the aperture 102, thereby controlling the aperture diameter and opening / closing operation of the aperture 102. The focus drive circuit 116 uses the focus actuator 113 to drive the focus lens 104 in the optical axis direction OA, thereby controlling the focal length of the optical system of the lens unit 100 (performing focus control). The focus drive circuit 116 also functions as a position detector that detects the current position (lens position) of the focus lens 104 using the focus actuator 113.

[0015] The lens MPU 117 (processor) performs all calculations and controls related to the lens unit 100 and controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116. The lens MPU 117 is also connected to the camera MPU 125 via a mount M to exchange commands and data. For example, the lens MPU 117 detects the position of the focus lens 104 and notifies the camera MPU 125 of lens position information in response to a request. This lens position information includes information such as the position of the focus lens 104 in the optical axis direction OA, the position and diameter of the exit pupil in the optical axis direction OA when the optical system is not moving, and the position and diameter of the lens frame that limits the light beam from the exit pupil in the optical axis direction OA. The lens MPU 117 also controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116 in response to a request from the camera MPU 125. The lens memory 118 stores optical information required for automatic focus adjustment (AF control). The camera MPU 125 controls the operation of the lens unit 100 by executing programs stored in, for example, an internal nonvolatile memory or the lens memory 118 .

[0016] The camera body 120 has an optical low-pass filter 121, an image sensor 122, and a drive / control system. The optical low-pass filter 121 and the image sensor 122 function as an imaging section that photoelectrically converts an object image (optical image) formed via the lens unit 100 and outputs image data. In this embodiment, the image sensor 122 photoelectrically converts an object image formed via the imaging optical system and outputs an imaging signal and a focus detection signal as image data. In this embodiment, the first lens group 101, the aperture 102, the second lens group 103, the focus lens 104, and the optical low-pass filter 121 constitute an imaging optical system.

[0017] The optical low-pass filter 121 reduces false colors and moiré in captured images. The image sensor 122 is composed of a CMOS image sensor and its peripheral circuits, and is arranged with m pixels in the horizontal direction and n pixels in the vertical direction (m and n are integers of 2 or greater). The image sensor 122 of this embodiment also serves as a focus detection element, has a pupil-splitting function, and has pupil-splitting pixels that are capable of phase-difference detection focus detection (phase-difference AF) using image data (image signals). The image processing circuit 124 generates data for phase-difference AF and image data for display, recording, and subject detection based on the image data output from the image sensor 122.

[0018] The drive / control system includes an image sensor drive circuit 123, an image processing circuit 124, a camera MPU 125, a display 126, a group of operation switches (operation SW) 127, a memory 128, a phase-difference AF unit 129 (image plane phase-difference focus detection unit, control unit), an AE unit 130 (control unit), a white balance adjustment unit 131 (control unit), and an object detection unit 132 (detection unit). The image sensor drive circuit 123 controls the operation of the image sensor 122 and performs A / D conversion on the image signal (image data) output from the image sensor 122 and transmits the converted image signal to the camera MPU 125. The image processing circuit 124 performs typical image processing performed in digital cameras, such as gamma conversion, color interpolation, and compression encoding, on the image signal output from the image sensor 122. The image processing circuit 124 also generates signals for phase-difference AF, AE, white balance adjustment, and object detection. In this embodiment, a signal for phase difference AF, a signal for AE, a signal for white balance adjustment, and a signal for subject detection are generated, but for example, the signal for AE, the signal for white balance adjustment, and the signal for subject detection may be generated as a common signal. Also, the combination of common signals is not limited to this.

[0019] The camera MPU 125 (processor, control device) performs all calculations and controls related to the camera body 120. That is, the camera MPU 125 controls the image sensor drive circuit 123, image processing circuit 124, display 126, operation switch group 127, memory 128, phase difference AF unit 129, AE unit 130, white balance adjustment unit 131, and subject detection unit 132. The camera MPU 125 is connected to the lens MPU 117 via a signal line of the mount M, and exchanges commands and data with the lens MPU 117. The camera MPU 125 issues requests to the lens MPU 117 to acquire the lens position and to drive the lens by a predetermined drive amount, and also issues requests from the lens MPU 117 to acquire optical information specific to the lens unit 100.

[0020] The camera MPU 125 incorporates a ROM 125a that stores a program for controlling the operation of the camera body 120, a RAM 125b (camera memory) that stores variables, and an EEPROM 125c that stores various parameters. The camera MPU 125 also executes focus detection processing based on the program stored in the ROM 125a. In the focus detection processing, a known correlation calculation process is executed using a pair of image signals obtained by photoelectrically converting optical images formed by light beams that have passed through different pupil regions (pupil partial regions) of the imaging optical system.

[0021] The display 126 is composed of an LCD or the like, and displays information about the shooting mode of the imaging device 10, a preview image before shooting and a confirmation image after shooting, an in-focus state display image during focus detection, etc. The operation switch group 127 is composed of a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, etc. The memory 128 (storage unit) is a removable flash memory that records shot images.

[0022] The phase-difference AF unit 129 performs focus detection processing using a phase-difference detection method based on image signals (signals for phase-difference AF) of focus detection image data obtained from the image sensor 122 and the image processing circuit 124. More specifically, the image processing circuit 124 generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system as focus detection data, and the phase-difference AF unit 129 detects the amount of focus deviation based on the amount of deviation between the pair of image data. As described above, the phase-difference AF unit 129 of this embodiment performs phase-difference AF (image-surface phase-difference AF) based on the output of the image sensor 122 without using a dedicated AF sensor. In this embodiment, the phase-difference AF unit 129 includes an acquisition unit 129a and a calculation unit 129b. The operation of each of these units will be described later.

[0023] At least a part of phase difference AF unit 129 (a part of acquisition unit 129a or calculation unit 129b) may be provided in camera MPU 125. Details of the operation of phase difference AF unit 129 will be described later. Phase difference AF unit 129 functions as a focus control unit that controls the position of focus lens 104 using the focus detection result.

[0024] The AE unit 130 performs exposure adjustment processing to optimize the shooting conditions by performing photometry based on AE signals obtained from the image sensor 122 and the image processing circuit 124. Specifically, it performs photometry based on the AE signals and calculates the exposure amount at the currently set aperture value, shutter speed, and ISO sensitivity. From the difference between the calculated exposure amount and a predetermined appropriate exposure amount, it calculates the appropriate aperture value, shutter speed, and ISO sensitivity to be set during shooting and sets these as the shooting conditions, thereby performing exposure adjustment processing. The AE unit 130 calculates the exposure conditions during shooting using the photometry results and functions as an exposure adjustment unit that controls the aperture value, shutter speed, and ISO sensitivity of the aperture 102.

[0025] The white balance adjustment unit 131 performs white balance adjustment processing based on a signal for white balance adjustment obtained from the image sensor 122 and the image processing circuit 124. Specifically, the white balance adjustment processing is performed by calculating the white balance of the signal for white balance adjustment and adjusting the weight of colors based on the difference from a predetermined appropriate white balance.

[0026] The subject detection unit 132 performs subject detection processing based on a subject detection signal generated by the image processing circuit 124. The subject detection processing detects the type and state of the subject (detection attribute), and the position and size of the subject (detection area). Details of the operation of the subject detection unit 132 will be described later.

[0027] In this way, the imaging device 10 of this embodiment can perform a combination of phase-difference AF, photometry (exposure adjustment), white balance adjustment, and subject detection, and can select the position (image height range) for performing phase-difference AF, photometry, and white balance adjustment depending on the results of subject detection.

[0028] (Configuration of image sensor 122) FIG. 2 is a schematic diagram of the array of imaging pixels (and focus detection pixels) of the image sensor 122. FIG. 2 shows the pixel (imaging pixel) array of the two-dimensional CMOS sensor (image sensor 122) of this embodiment in an area of ​​4 columns by 4 rows, and the focus detection pixel array in an area of ​​8 columns by 4 rows. In the first embodiment, the 2-column by 2-row pixel group 200 shown in FIG. 2 has a pixel 200R having a spectral sensitivity of R (red) at the upper left, pixels 200G having a spectral sensitivity of G (green) at the upper right and lower left, and a pixel 200B having a spectral sensitivity of B (blue) at the lower right. Furthermore, each pixel is composed of a first focus detection pixel 201 and a second focus detection pixel 202 arranged in a 2-column by 1-row array.

[0029] 2 are arranged on a surface, making it possible to acquire a captured image (focus detection signal). In this embodiment, the pixel period P is 4 μm, the number of pixels N is 5,575 columns horizontally by 3,725 rows vertically = approximately 20.75 million pixels, the column-direction period PAF of the focus detection pixels is 2 μm, and the number of focus detection pixels NAF is 11,150 columns horizontally by 3,725 rows vertically = approximately 41.5 million pixels.

[0030] Figure 3(a) is a plan view of one pixel 200G of the imaging element 122 shown in Figure 2, viewed from the light receiving surface side (+z side) of the imaging element 122, and Figure 3(b) is a cross-sectional view of the aa section of Figure 3(a) viewed from the -y side.

[0031] 3, in the pixel 200G of this embodiment, a microlens 305 for collecting incident light is formed on the light-receiving side of each pixel, and a photoelectric conversion unit 301 and a photoelectric conversion unit 302 are formed that are divided into N-H divisions (two divisions) in the x direction and N-V divisions (one division) in the y direction. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively.

[0032] The photoelectric conversion units 301 and 302 may be pin structure photodiodes in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer, or may be pn junction photodiodes in which the intrinsic layer is omitted as necessary. In each pixel, a color filter 306 is formed between the microlens 305 and the photoelectric conversion units 301 and 302. Furthermore, as necessary, the spectral transmittance of the color filter 306 may be changed for each subpixel, or the color filter 306 may be omitted.

[0033] Light incident on pixel 200G shown in FIG. 3 is collected by microlens 305, dispersed by color filter 306, and then received by photoelectric conversion unit 301 and photoelectric conversion unit 302. In photoelectric conversion unit 301 and photoelectric conversion unit 302, electron-hole pairs are generated according to the amount of received light, and after being separated by a depletion layer, the negatively charged electrons are accumulated in an n-type layer (not shown). Meanwhile, the holes are discharged to the outside of image sensor 122 through a p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layers (not shown) of photoelectric conversion unit 301 and photoelectric conversion unit 302 are transferred to a capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0034] Fig. 4 is a schematic diagram illustrating the correspondence between the pixel structure of this embodiment shown in Fig. 3 and pupil division. Fig. 4 shows a cross-sectional view of the aa cross section of the pixel structure of this embodiment shown in Fig. 3(a) as viewed from the +y side, and the pupil plane (pupil distance Ds) of the image sensor 122. In Fig. 4, the x-axis and y-axis of the cross-sectional view are reversed with respect to Fig. 3 in order to correspond to the coordinate axes of the pupil plane of the image sensor 122.

[0035] 4, the first partial pupil region 501 of the first focus detection pixel 201 is generally conjugate with the light receiving surface of the photoelectric conversion unit 301, whose center of gravity is decentered in the -x direction, via a microlens, and represents the pupil region that can receive light at the first focus detection pixel 201. The first partial pupil region 501 of the first focus detection pixel 201 has its center of gravity decentered on the +X side on the pupil plane. In FIG. 4, the second partial pupil region 502 of the second focus detection pixel 202 is generally conjugate with the light receiving surface of the photoelectric conversion unit 302, whose center of gravity is decentered in the +x direction, via a microlens, and represents the pupil region that can receive light at the second focus detection pixel 202. The second partial pupil region 502 of the second focus detection pixel 202 has its center of gravity decentered on the -X side on the pupil plane. In addition, in FIG. 4, pupil region 500 is the pupil region that can receive light in the entire pixel 200G when all of the photoelectric conversion units 301 and 302 (first focus detection pixel 201 and second focus detection pixel 202) are combined.

[0036] Image plane phase-difference AF is affected by diffraction because it uses microlenses on the image sensor to divide the pupil. In Figure 4, the pupil distance to the pupil plane of the image sensor is several tens of mm, while the diameter of the microlens is several microns. As a result, the aperture value of the microlens is several tens of thousands, causing diffraction blur on the level of several tens of mm. As a result, the image on the light-receiving surface of the photoelectric conversion unit does not become a clear pupil region or pupil subregion, but rather becomes a light-receiving sensitivity characteristic (incident angle distribution of light-receiving rate).

[0037] 5 is a schematic diagram showing the correspondence between the image sensor 122 and pupil division. The image sensor 122 is disposed on an imaging plane 600. Light beams that pass through different pupil partial regions, the first pupil partial region 501 and the second pupil partial region 502, are incident on each pixel of the image sensor 122 at different angles and are received by the first focus detection pixel 201 and the second focus detection pixel 202, which are divided into 2×1 regions. In this embodiment, the pupil region is divided into two in the horizontal direction. If necessary, the pupil may also be divided vertically.

[0038] The image sensor 122 of this embodiment has an array of imaging pixels, each having a first focus detection pixel 201 and a second focus detection pixel 202. The first focus detection pixel 201 receives a light beam that passes through a first pupil partial region 501 of the imaging optical system. The second focus detection pixel 202 receives a light beam that passes through a second pupil partial region 502 of the imaging optical system that is different from the first pupil partial region 501. The imaging pixel receives a light beam that passes through a pupil region that is the combined first pupil partial region 501 and second pupil partial region 502 of the imaging optical system.

[0039] In the image sensor 122 of this embodiment, each imaging pixel is made up of a first focus detection pixel 201 and a second focus detection pixel 202. If necessary, the imaging pixel, the first focus detection pixel 201, and the second focus detection pixel 202 may be configured as separate pixels, and the first focus detection pixel 201 and the second focus detection pixel 202 may be partially arranged in a portion of the imaging pixel array.

[0040] In this embodiment, focus detection is performed by collecting light reception signals from the first focus detection pixels 201 of each pixel of the image sensor 122 to generate a first focus signal, and collecting light reception signals from the second focus detection pixels 202 of each pixel to generate a second focus signal. Furthermore, an image pickup signal (captured image) with a resolution of N effective pixels is generated by adding the signals from the first focus detection pixels 201 and second focus detection pixels 202 for each pixel of the image sensor 122. The method of generating each signal is not limited to the methods described above, and for example, the second focus detection signal may be generated from the difference between the image pickup signal and the first focus signal.

[0041] (Relationship between defocus amount and image shift amount) The relationship between the defocus amount and the image shift amount based on the first focus detection signal and the second focus detection signal acquired by the image sensor 122 of this embodiment will be described below.

[0042] 6 is a schematic diagram illustrating the relationship between the defocus amount based on the first focus detection signal and the second focus detection signal and the image shift amount between the first focus detection signal and the second focus detection signal. The image sensor 122 is disposed on an imaging plane 600. As in FIGS. 4 and 5, the pupil plane of the image sensor 122 is divided into a first pupil partial region 501 and a second pupil partial region 502.

[0043] The defocus amount d is defined as the distance from the subject's imaging position to the imaging plane, with magnitude |d|, and a front-focus state in which the subject's imaging position is closer to the subject than the imaging plane is defined as a negative sign (d<0). A back-focus state in which the subject's imaging position is on the opposite side of the subject than the imaging plane is defined as a positive sign (d>0). In a focused state in which the subject's imaging position is on the imaging plane (focus position), d=0. In FIG. 6, subject 601 shows an example of a focused state (d=0), and subject 602 shows an example of a front-focus state (d<0). The front-focus state (d<0) and the back-focus state (d>0) are combined to form a defocused state (|d|>0).

[0044] In a front-focus state (d<0), a light beam from the subject 602 that passes through the first pupil partial region 501 (second pupil partial region 502) is first focused and then spreads to a width Γ1 (Γ2) around the center of gravity G1 (G2) of the light beam, forming a blurred image on the imaging surface 600. The blurred image is received by the first focus detection pixels 201 (second focus detection pixels 202) that constitute the pixels arranged on the image sensor 122, and a first focus detection signal (second focus detection signal) is generated. Therefore, the first focus detection signal (second focus detection signal) is recorded as a subject image in which the subject 602 is blurred to a width Γ1 (Γ2) at the center of gravity G1 (G2) on the imaging surface 600. The blur width Γ1 (Γ2) of the subject image increases roughly proportionally as the magnitude of the defocus amount d, |d|, increases. Similarly, the magnitude |p| of the image shift amount p (= the difference G1-G2 in the center of gravity positions of the light beams) of the subject image between the first focus detection signal and the second focus detection signal also increases roughly proportionally as the magnitude |d| of the defocus amount d increases. The same is true in the back-focus state (d>0), although the direction of the image shift of the subject image between the first focus detection signal and the second focus detection signal is opposite to that in the front-focus state.

[0045] As the magnitude of the defocus amount of the first focus detection signal and the second focus detection signal, or the image capture signal obtained by adding the first focus detection signal and the second focus detection signal, increases, the magnitude of the image shift amount between the first focus detection signal and the second focus detection signal also increases. Therefore, in this embodiment, the phase difference AF section 129 utilizes the relationship in which the magnitude of the image shift amount between the first focus detection signal and the second focus detection signal increases as the magnitude of the defocus amount of the image capture signal increases, and converts the image shift amount into a detected defocus amount using a conversion coefficient calculated based on the base length.

[0046] (AF operation flow) 7A and 7B are flowcharts of the AF operation. This AF operation is performed for each frame by the camera MPU 125. For this AF operation, the camera MPU 125 cooperates with the lens MPU 117 as necessary.

[0047] 8A and 8B are conceptual diagrams of matching of subject, histogram, and front-to-back feature amounts, and matching of shape feature amounts, in frames 1 to 4. From frame 1 to frame 4, subject 801 (the subject to be tracked) moves closer (toward the front), while subject 802 remains stationary. In frame 1, image capture device 10 performs a focusing operation on subject 801 based on a displayed AF frame 800. From frame 2 onward, image capture device 10 tracks the position of subject 801 using a histogram and performs a focusing operation on subject 801. However, as will be described later, for frame 4, there is a large change in the histogram between the current frame (frame 4) and the previous frame, so subject tracking using the histogram is not performed.

[0048] The processing for each frame from frame 1 to frame 4 will be described according to the flowcharts of FIGS. 7A and 7B.

[0049] (1) Frame 1 In S701, the camera MPU 125 sets the entire angle of view as the defocus amount calculation area. Note that the defocus amount calculation area does not have to be the entire angle of view. For example, the camera MPU 125 may set a part of the angle of view as the defocus amount calculation area based on the size of the display AF frame 800, the detected subject area, or the like.

[0050] In S702, the camera MPU 125 calculates the defocus amount for each of the multiple distance measurement points in the defocus amount calculation area set in S701.

[0051] In S703, the camera MPU 125 calculates a histogram based on the defocus amount for each ranging point calculated in S702. In the histogram of FIG. 8A, the horizontal axis represents the defocus amount. The left side of the horizontal axis represents infinity, and the right side represents close range. The vertical axis represents the frequency of defocus amounts belonging to each class. The camera MPU 125 sets the bin interval (width of each class) and range of the histogram based on at least one of the subject distance (distance in the depth direction of the subject), the subject size, and the aperture value at the time of shooting. The aperture value at the time of shooting can be used to calculate the subject depth size on the image plane by converting the value. By setting the bin interval according to the subject depth size based on the aperture value, the resolution for representing the subject on the histogram can be appropriately set regardless of the conditions. The method for setting the bin interval and range of the histogram is not limited to this. For example, the camera MPU 125 may calculate the histogram using a predetermined bin interval and range.

[0052] In S704, the camera MPU 125 determines whether the frame to be processed is frame 1. Since frame 1 is the first frame, the camera MPU 125 advances the processing step to S715.

[0053] In S716, the camera MPU 125 performs normal focus point selection. Normal focus point selection here refers to, but is not limited to, selecting the focus point at the center of the displayed AF frame 800. For example, the camera MPU 125 may select the focus point that indicates the closest defocus amount within the displayed AF frame 800, the closest focus point within the area near the center of the angle of view, or a focus point close to the focus point selected previously.

[0054] In S717, the camera MPU 125 identifies the histogram class to which the ranging point selected in S716 belongs as the object class of frame 1. Here, it is assumed that class 841 of the histogram of frame 1 is identified as the object class of frame 1.

[0055] In S718, the camera MPU 125 drives the focus lens 104 based on the defocus amount of the range-finding point selected in S716 (performs drive control of the focus lens 104).

[0056] With the above processing, the AF operation for frame 1 is completed.

[0057] (2) Frame 2 Next, a description will be given of frame 2. The processing of S701 to S703 is omitted here because it is the same as that of frame 1. In S704, since frame 2 is not the first frame, the camera MPU 125 advances the processing step to S705.

[0058] In S705, the camera MPU 125 acquires a histogram for frame 1, which is the previous frame, and a histogram for frame 2, which is the current frame. In each histogram, the peak on the left corresponds to subject 801, and the peak on the right corresponds to subject 802. In frame 2, subject 801 is closer to the close side than in frame 1, and its size within the angle of view is also larger, so the peak on the left corresponding to subject 801 has moved to the close side (right side) and is larger. Class 841 of the histogram for frame 1 is the subject class identified in frame 1.

[0059] In S706, the camera MPU 125 determines whether there is a large change (shape change) in the histogram between the previous frame and the current frame. If there is a large change in the histogram, the object class cannot be identified by histogram matching (comparison process), so in this step the camera MPU 125 determines whether to perform matching process.

[0060] Figure 9 shows the shape evaluation value for the entire histogram of each frame and the shape change evaluation value for each frame. Figure 9(a) shows the shape evaluation value for the entire histogram of each frame, where the shape evaluation value is the sum of squares of the differences between adjacent bins of the histogram of each frame. However, the shape evaluation value is not limited to the sum of squares of the differences between adjacent bins, and may be, for example, the sum of absolute values ​​of the differences between adjacent bins. Figure 9(b) shows the shape change evaluation value for each frame, where the shape change evaluation value is the absolute difference between the shape evaluation values ​​of the current frame and the previous frame.

[0061] A threshold value 901 (second threshold value) shown in FIG. 9(b) is a threshold value for determining whether the shape change is large. If the evaluation value of the shape change is larger than the threshold value 901, the camera MPU 125 determines that the shape change is large and does not perform matching processing. The evaluation value of the shape change for frame 2 corresponds to the leftmost frame in FIG. 9(b) and is smaller than the threshold value 901. Therefore, the camera MPU 125 determines that the shape change is not large and proceeds to processing step S707 for performing matching processing.

[0062] In S707, the camera MPU 125 calculates the feature amounts of the object class of the previous frame, frame 1. Here, two types of feature amounts (front-to-back feature amounts and shape feature amounts) are calculated.

[0063] The feature quantity (second type feature quantity) of the front-to-back relationship of the subject class (class 841) in frame 1 is the sum of the frequencies of each class in the range on the near side of class 841 in the histogram, and is front-to-back relationship feature quantity 821 in FIG. 8B. However, the configuration of the feature quantity of the front-to-back relationship is not limited to this, and for example, the feature quantity of the front-to-back relationship may be the sum of the frequencies of each class in the range on the infinity side of class 841 in the histogram. Because the sum of the entire area of ​​the histogram is the number of ranging points, the front-to-back relationship between the subject and other obstacles, etc. can be determined by calculating the sum of the frequencies for either the near side or the infinity side.

[0064] The shape feature (first type feature) of the object class (class 841) of frame 1 is the sum of the frequency of each class in the range of class 841±x1 in the histogram. ±x1 is a value related to the depth size of the object, and the depth distribution of the object can be determined by the sum of the frequency in the range of class 841±x1. The camera MPU 125 determines the value of x1 based on, for example, at least one of the object distance (distance in the depth direction of the object), the size of the object, and the aperture value at the time of shooting. The aperture value at the time of shooting can be used to calculate the depth size of the object on the image plane by converting the value. The object distance and the size of the object are also related to the depth size of the object. Note that the configuration of the shape feature is not limited to this. For example, the shape feature may be the sum of absolute differences between the frequency of the object class (class 841) and each class in the range of class 841±x1.

[0065] In S708, the camera MPU 125 calculates the feature amounts of each class for the current frame, frame 2. Here, as in S707, two types of feature amounts (front-to-back feature amounts and shape feature amounts) are calculated. However, while in S707 the feature amounts were calculated only for the object class, in S708 the feature amounts are calculated for each class.

[0066] In S709, the camera MPU 125 performs a matching process on the context feature amounts of the previous frame (frame 1) and the current frame (frame 2) calculated in S707 and S708. In FIG. 8B, context feature amount 821 indicates the context feature amount for the object class of frame 1, and context feature amount 812 indicates the context feature amount for each class of frame 2. In this step, the camera MPU 125 identifies the class (class 832 in the example of FIG. 8B) in context feature amount 812 of frame 2 that matches (has the closest value to) context feature amount 821 of frame 1. In this step, the destination of the specific subject is identified by utilizing the fact that subjects existing before and after the specific subject do not change significantly between frames.

[0067] In S710, the camera MPU 125 performs matching processing on the shape features of the previous frame (frame 1) and the current frame (frame 2) calculated in S707 and S708. In Fig. 8B, correlation value 862 indicates the absolute difference between the shape features of the object class in frame 1 and the shape features of each class in frame 2, and the smaller the value, the higher the degree of matching.

[0068] In S711, the camera MPU 125 selects object class candidates for the current frame (frame 2). Specifically, the camera MPU 125 selects object class candidates based on the correlation value 862 from within the range of class 832±x2 (first plurality of classes) that is near class 832 identified in S709. By limiting the selection range of object class candidates to the range of class 832±x2, similar shapes at distant locations can be excluded. The value of x2 is, for example, predetermined and stored in ROM 125a. A specific value can be used as x2. Next, the camera MPU 125 calculates a correlation threshold 852 (first threshold) as a value x3 times the minimum value of the correlation value 862, taking into account the possibility that multiple similar shapes exist within the selection range. The value of x3 is, for example, predetermined and stored in ROM 125a. The camera MPU 125 then selects classes that exhibit correlation values ​​smaller than the correlation threshold 852 within the selection range of correlation values ​​862 as object class candidates. In frame 2, class 842 is the only class that exhibits a correlation value smaller than correlation threshold 852 within the selected range of correlation value 862, so camera MPU 125 selects class 842 as the subject class candidate.

[0069] If reducing the processing load is prioritized over improving the accuracy of identifying object classes, the camera MPU 125 may omit the process of limiting the selection range of object class candidates based on the matching process of contextual features. In this case, the camera MPU 125 selects object class candidates from the entire range of the histogram (the first plurality of classes when the selection range is not limited) based on the correlation value 862 and the correlation threshold 852.

[0070] Furthermore, camera MPU 125 may omit the process of selecting object class candidates based on correlation threshold 852. In this case, camera MPU 125 may detect the class corresponding to the smallest value of correlation value 862 in the selected range (or the entire range) (i.e., the class having shape features closest to the shape features of the object class of the previous frame), and specify the detected class as the object class of the current frame.

[0071] The camera MPU 125 may also reverse the use of the context feature and the use of the shape feature. That is, the camera MPU 125 may limit the selection range based on the matching of the shape feature, and select subject class candidates based on the matching of the context feature in the selection range.

[0072] Alternatively, the camera MPU 125 may select candidate object classes based on matching of contextual features, without using shape features. In this case, class 832 shown in FIG. 8B (or multiple classes near class 832) is selected as the candidate object class.

[0073] In S712, the camera MPU 125 determines whether there are multiple object class candidates. Since there is one object class candidate in frame 2, the camera MPU 125 advances the process to processing step S714.

[0074] In S714, the camera MPU 125 identifies the object class of the current frame. Since the only object class candidate in frame 2 is class 842, the camera MPU 125 identifies class 842 as the object class of the current frame.

[0075] In S715, the camera MPU 125 selects one ranging point from among the ranging points that belong to the object class identified in S714.

[0076] In S718, the camera MPU 125 drives the focus lens 104 based on the defocus amount of the range-finding point selected in S715.

[0077] With the above processing, the AF operation for frame 2 is completed.

[0078] Although the above description has been given of a configuration in which a ranging point is selected in S715 and drive control of the focus lens 104 is performed based on the defocus amount of the selected ranging point in S718, this embodiment is not limited to this configuration. For example, a configuration may be adopted in which drive control of the focus lens 104 is performed based on the representative defocus amount, average defocus amount, or closest defocus amount of the subject class identified in S714.

[0079] (3) Frame 3 Next, frame 3 will be described. The processes of S701 to S710 are similar to those of frame 2, and therefore will not be described here. However, for frame 3, "previous frame" corresponds to frame 2, and "current frame" corresponds to frame 3. Therefore, with regard to the context feature, in S707, context feature 822 shown in FIG. 8B is calculated as the context feature of the object class of the previous frame (frame 2). In addition, in S708, context feature 813 shown in FIG. 8B is calculated as the context feature of each class of the current frame (frame 3). In addition, in S709, class 833 shown in FIG. 8B is identified by matching processing. Similarly, for shape feature, processing is performed for the previous frame (frame 2) and the current frame (frame 3) in S707, S708, and S710.

[0080] In S711, the camera MPU 125 selects object class candidates for the current frame (frame 3). As in the case of frame 2, the camera MPU 125 selects object class candidates based on correlation value 863 from within the range of class 833±x2, which is near class 833 identified in S709. Next, as in the case of frame 2, the camera MPU 125 calculates correlation threshold 853 (first threshold) as a value x3 times the minimum value of correlation value 863. Then, the camera MPU 125 selects, as object class candidates, classes that exhibit correlation values ​​smaller than correlation threshold 852 within the selected range of correlation value 863. In frame 3, the classes that exhibit correlation values ​​smaller than correlation threshold 853 within the selected range of correlation value 863 are class 843a (first class) and class 843b (second class), so the camera MPU 125 selects class 843a and class 843b as object class candidates.

[0081] In S712, the camera MPU 125 determines whether there are multiple object class candidates. Since there are two object class candidates in frame 3, the camera MPU 125 advances the processing step to S713.

[0082] In S713, the camera MPU 125 predicts the defocus amount of the current frame based on the defocus amount of the subject in the previous frame (or in multiple past frames), thereby obtaining the predicted defocus amount.

[0083] In S714, the camera MPU 125 identifies the object class candidate (e.g., class 843a) corresponding to the defocus amount closest to the predicted defocus amount obtained in S713 as the object class of the current frame from among the object class candidates (class 843a and class 843b) selected in S711.

[0084] The processing in steps S715 and S718 is the same as that in frame 2, and therefore a description thereof will be omitted. With the above processing, the AF operation in frame 3 is completed.

[0085] (4) Frame 4 Next, frame 4 will be described. The processes of S701 to S705 are the same as those of frame 2, so the description will be omitted. However, for frame 4, the "previous frame" corresponds to frame 3, and the "current frame" corresponds to frame 4.

[0086] In S706, the camera MPU 125 determines whether there is a large change in the shape of the histogram between the previous frame and the current frame. If there is a large change in the shape of the histogram, the object class cannot be identified by histogram matching processing, so in this step the camera MPU 125 determines whether to perform matching processing.

[0087] As described above for frame 2, threshold value 901 shown in FIG. 9(b) is a threshold value for determining whether the shape change is large. If the evaluation value of the shape change is larger than threshold value 901, camera MPU 125 determines that the shape change is large and does not perform matching processing. The evaluation value of the shape change for frame 4 corresponds to the rightmost frame in FIG. 9(b) and is larger than threshold value 901. Therefore, camera MPU 125 determines that the shape change is large and proceeds to processing step S716. That is, for frame 4, matching processing is not performed, and object tracking using a histogram (identification of the object class of the current frame based on the histogram matching processing shown in S714) is not performed.

[0088] In S716, the camera MPU 125 performs normal selection of a distance measurement point. Here, normal selection of a distance measurement point means selection of a distance measurement point close to the previously selected distance measurement point.

[0089] In S717, the camera MPU 125 identifies the histogram class to which the range-finding point selected in S716 belongs as the object class of frame 4.

[0090] In S718, the camera MPU 125 drives the focus lens 104 based on the defocus amount of the range-finding point selected in S716.

[0091] With the above processing, the AF operation for frame 4 is completed.

[0092] As described above, according to the first embodiment, the imaging device 10 acquires a first histogram of multiple defocus amounts corresponding to multiple ranging points in a first captured image (e.g., frame 1 in FIG. 8A). The imaging device 10 also acquires a first object class (e.g., class 841 in FIG. 8A), which is a first histogram class to which the defocus amounts of the tracked object in the first captured image belong. The imaging device 10 also acquires a second histogram of multiple defocus amounts corresponding to multiple ranging points in a second captured image (e.g., frame 2 in FIG. 8A), which is captured after the first captured image. Next, the imaging device 10 calculates a first type of feature (e.g., shape feature) of the first object class in the first histogram. The imaging device 10 also calculates a plurality of first type feature (e.g., shape feature) of a first plurality of classes in the second histogram (e.g., each class in the entire range of the second histogram). Thereafter, the imaging device 10 detects a first class (e.g., class 842 in FIG. 8B ) having a first type feature amount that is closest to the first type feature amount of the first object class from among a first plurality of classes in the second histogram (e.g., each class in the entire range of the second histogram). Then, based on the first class, the imaging device 10 identifies a second object class (e.g., class 842 in FIG. 8B ) that is the class of the second histogram to which the defocus amount of the object to be tracked in the second captured image belongs. This makes it possible to track the object between captured images (frames) with high accuracy.

[0093] The first class does not necessarily have to be a class having a first-type feature quantity closest to the first-type feature quantity of the first object class. For example, the image capture device 10 may detect a specific first class from among the first plurality of classes of the second histogram (e.g., each class in the entire range of the second histogram) based on the first-type feature quantity of the first object class.

[0094] Furthermore, as shown in FIG. 8B as the range of class 832±x2, the image capture device 10 may limit the detection range of the first class to a specific range of the second histogram. In this case, the image capture device 10 can limit the detection range of the first class based on a matching process between a second type feature (e.g., a context feature) of the first object class and a plurality of second type feature (e.g., a context feature) of a second plurality of classes of the second histogram (e.g., each class in the entire range of the second histogram). This can further improve the accuracy of object detection.

[0095] In this embodiment, the defocus amount is used as distance information relating to the position (distance) of the subject in the depth direction. However, distance information can be applied to various embodiments. That is, the distance information may directly represent the distance value of the subject in the depth direction in the image, or may indirectly represent information corresponding to the distance value. Specifically, the distance information may be the amount of image shift or the subject distance, which is the distance to the subject.

[0096] That is, the acquisition unit may acquire at least one of an image shift map, a defocus map, or a subject distance map corresponding to distribution information of the image shift amount, the defocus amount, or the subject distance, and calculate (acquire) a corresponding histogram.

[0097] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0098] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0099] 10...imaging device, 100...lens unit, 120...camera body, 122...imaging element, 125...camera MPU, 129...phase difference AF section, 130...AE section, 131...white balance adjustment section, 132...subject detection section

Claims

1. an acquisition means for acquiring a first histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a first captured image, a first object class which is a class of the first histogram to which the defocus amounts of the object to be tracked in the first captured image belong, and a second histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a second captured image captured after the first captured image; a calculation means for calculating a first type feature of the first object class in the first histogram and a plurality of first type feature of a first plurality of classes of the second histogram in the second histogram; a detection means for detecting a specific first class from the first plurality of classes in the second histogram based on the feature amount of the first type of the first object class; an identification means for identifying a second object class, which is a class of the second histogram to which a defocus amount of the object to be tracked in the second captured image belongs, based on the identified first class; A subject tracking device comprising:

2. The detecting means detects, as a first class, a class having a feature amount of the first type that is closest to the feature amount of the first type of the first object class, from among the first plurality of classes in the second histogram.

2. The subject tracking device according to claim 1.

3. The identification means identifies the first class as the second object class.

3. The subject tracking device according to claim 2.

4. and when a difference between a second class having a feature amount of the first type closest to the feature amount of the first type of the first class among the first plurality of classes in the second histogram and the feature amount of the first type of the second class is greater than a first threshold, the specifying unit specifies the first class as the second object class; If the difference between the feature amount of the first type of the second class and the feature amount of the first type of the first class is smaller than the first threshold, the specifying unit specifies the first class or the second class as the second object class based on a predetermined criterion.

3. The subject tracking device according to claim 2.

5. a prediction unit configured to predict a defocus amount in the second captured image of the subject to be tracked based on a defocus amount in the first captured image of the subject to be tracked; The predetermined criterion is configured such that one of the first class and the second class, which corresponds to a defocus amount close to the predicted defocus amount, is identified as the second object class.

5. The subject tracking device according to claim 4.

6. The first type of individual feature represents a shape feature in a first range including a class corresponding to the individual feature, of a histogram including the corresponding class.

6. The subject tracking device according to claim 1,

7. The first type of individual feature is a sum of the frequency of each class within a first range including the corresponding class of a histogram including the class corresponding to the individual feature.

6. The subject tracking device according to claim 1,

8. The size of the first range corresponding to the feature amount of the first type of the first object class is based on at least one of a depth direction distance of the object to be tracked in the first captured image, a size of the object to be tracked in the first captured image, and an aperture value at the time of capturing the first captured image.

8. The subject tracking device according to claim 6 or 7.

9. The bin interval of the first histogram is based on at least one of a depth direction distance of the subject to be tracked in the first captured image, a size of the subject to be tracked in the first captured image, and an aperture value at the time of capturing the first captured image.

9. The subject tracking device according to claim 1,

10. the calculation means calculates a second type feature of the first object class in the first histogram and a plurality of second type feature of a second plurality of classes of the second histogram in the second histogram; The subject tracking device further includes a selection means for selecting, from the second plurality of classes of the second histogram, a plurality of classes within a second range including a class having a feature amount of the second type closest to the feature amount of the second type of the first subject class, as the first plurality of classes of the second histogram.

10. The subject tracking device according to claim 1.

11. The second type of individual feature amount is the sum of the frequencies of each class in a range on the closer side or the infinity side of the corresponding class in a histogram including the class corresponding to the individual feature amount.

11. The subject tracking device according to claim 10.

12. a determining unit for determining whether a change in shape between the first histogram and the second histogram is smaller than a second threshold; The identification means identifies the second object class when the shape change is smaller than the second threshold value.

12. The subject tracking device according to claim 1.

13. a control unit that controls the driving of a focus lens based on a defocus amount that belongs to the second object class among the plurality of defocus amounts corresponding to the plurality of distance measuring points of the second captured image; 13. The subject tracking device according to claim 1.

14. an acquisition unit that acquires first and second captured images and depth direction distance information for a plurality of regions of the first and second captured images; an identification unit that identifies a subject to be tracked in the first and second captured images based on distance information of the first and second captured images, the specifying means specifies an area of ​​the subject to be tracked in the second captured image that corresponds to an area of ​​the subject to be tracked in the first captured image by comparing histograms of distance information in the first and second captured images.

15. 15. The object tracking device according to claim 14, wherein the specifying means specifies the class of the object to be tracked in the second captured image from a result of calculating a correlation between a feature amount of a class of a histogram of the first captured image to which distance information of the object to be tracked in the first captured image belongs and a feature amount of each class of a histogram of the second captured image.

16. 16. The object tracking device according to claim 15, wherein the specifying means specifies an area in the second captured image corresponding to the class specified as the class of the object to be tracked, as the area of ​​the object to be tracked in the second captured image.

17. An object tracking device according to any one of claims 1 to 16; an imaging means for generating individual captured images; An imaging device comprising:

18. An object tracking method executed by an object tracking device, an acquisition step of acquiring a first histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a first captured image, a first object class which is a class of the first histogram to which the defocus amounts of the object to be tracked in the first captured image belong, and a second histogram of a plurality of defocus amounts corresponding to a plurality of distance measurement points of a second captured image captured after the first captured image; a calculation step of calculating a first type feature of the first object class in the first histogram and a plurality of first type feature of a first plurality of classes of the second histogram in the second histogram; a detecting step of detecting a specific first class from the first plurality of classes in the second histogram based on the feature amount of the first type of the first object class; a specifying step of specifying a second object class, which is a class of the second histogram to which a defocus amount of the object to be tracked in the second captured image belongs, based on the specified first class; A subject tracking method comprising:

19. A program for causing a computer to function as each of the means of the subject tracking device according to any one of claims 1 to 16.

Images