Object Tracking Device and Control Method Thereof

Abstract

To highly accurately track a subject in real time even when the subject is constituted of a plurality of motions.SOLUTION: A subject tracking device comprises: motion vector calculation means for calculating a motion vector between images; feature amount area determination means for determining a feature amount area from an image: weight map creation means for creating a weight map that is weight distribution of each area where the motion vector is calculated, based on the motion vector and the feature amount area; and subject tracking means for tracking a subject based on the weight map.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a subject tracking device having a function of detecting and tracking a subject in a captured image and a control method thereof.

Background Art

[0002] In a digital camera, a template is created for the feature amount of a specific region of an image from the obtained image data, and the subject is tracked by performing template matching, and the subject is photographed with the focus, brightness, and color adjusted to a suitable state. Such a method is known.

[0003] In Patent Document 1, a region having a feature amount used for tracking is used as a template, and even when there is a change in the region to be tracked, by weighting the template, tracking can be performed with high accuracy.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, this method requires prior knowledge of parts where many changes occur. For example, it is difficult to accurately track a subject composed of multiple movements such as a running person, where the changing parts cannot be grasped in advance.

[0006] In view of the above problems, an object of the present invention is to provide a subject tracking device and a control method thereof that can perform high-precision tracking even for a subject composed of multiple movements.

Means for Solving the Problems

[0007] In order to solve the above problems, the subject tracking device of the present invention uses the region of the subject detected from the image as a feature amount region Registration feature amount region Registration means, drawing motion vectors of each pixel between images the first motion vector which is to calculate the first motion vector calculation means, a second motion vector calculation means for calculating a second motion vector which is the motion vector of the entire subject from the first motion vector and the feature amount region, and a third motion vector calculation means for calculating a third motion vector which is the motion vector of a local region of the subject from the first motion vector and the second motion vector, the the first reliability evaluation means for evaluating the reliability of the motion vector, the third motion vector of Based on the above and the reliability, of the feature amount region weight map creation means for creating a weight map, the feature amount region and and subject tracking means for tracking the subject based on the weight map.

[0008] Also, the control method of the subject tracking device of the present invention uses the region of the subject detected from the image as a feature amount region Registration feature amount region Registration step, drawing motion vectors of each pixel between images the first motion vector which is to calculate the first motion vector calculation step, a second motion vector calculation step for calculating a second motion vector which is the motion vector of the entire subject from the first motion vector and the feature amount region, and a third motion vector calculation step for calculating a third motion vector which is the motion vector of a local region of the subject from the first motion vector and the second motion vector, the the first reliability evaluation step for evaluating the reliability of the motion vector, the third motion vector of Based on the above and the reliability, of the feature amount region weight map creation step for creating a weight map, the feature amount region and and subject tracking step for tracking the subject based on the weight map.

Effect of the Invention

[0009] According to the present invention, it is possible to perform highly accurate subject tracking even when the subject is composed of a plurality of motions.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the attached images. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the attached images, the same or similarly configured components are given the same reference numerals, and duplicate explanations are omitted.

[0012] [First Embodiment] (Configuration of Imaging Device) FIG. 1 is a block diagram showing a functional configuration example of an imaging device 100, which is an example of a subject tracking device in the present embodiment. The imaging device 100 can capture and record moving images and still images. Each functional block in the imaging device 100 is communicably connected to each other via a bus 160. The operation of the imaging device 100 is realized by the CPU 151 (Central Processing Unit) executing a program to control each functional block.

[0013] The imaging device 100 of the present embodiment has a subject tracking unit 161 that detects a specific subject from the captured image and continuously tracks the detected subject in a plurality of images. The subject tracking unit 161 includes a subject detection function for detecting the position and size of the subject to be imaged, a subject tracking function for tracking by continuously searching for an area similar to the detected area, and a motion vector detection function for obtaining a motion vector between images. Details of the configuration and operation of the subject tracking unit 161 will be described later.

[0014] The photographing lens 101 (lens unit) includes a fixed single-group lens 102, a zoom lens 111, a diaphragm 103, a fixed three-group lens 121, a focus lens 131, a zoom motor 112, a diaphragm motor 104, and a focus motor 132. The fixed single-group lens 102, the zoom lens 111, the diaphragm 103, the fixed three-group lens 121, and the focus lens 131 constitute an imaging optical system. For the sake of convenience, the lenses 102, 111, 121, and 131 are illustrated as a single lens, but each may be composed of a plurality of lenses. Further, the photographing lens 101 may be configured as an interchangeable lens detachable from the imaging device 100.

[0015] The diaphragm control unit 105 controls the operation of the diaphragm motor 104 that drives the diaphragm 103, and changes the aperture diameter of the diaphragm 103. The zoom control unit 113 controls the operation of the zoom motor 112 that drives the zoom lens 111, and changes the focal length (angle of view) of the photographing lens 101.

[0016] The focus control unit 133 calculates the defocus amount and defocus direction of the photographing lens 101 based on the phase difference between a pair of focus detection signals (A image and B image) obtained from the imaging element 141. Then, the focus control unit 133 converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132. Based on this drive amount and drive direction, the focus control unit 133 controls the operation of the focus motor 132, and drives the focus lens 131 to control the focus state of the photographing lens 101. In this way, the focus control unit 133 performs automatic focus detection (AF) using the phase difference detection method. Note that the focus control unit 133 may execute AF using the contrast detection method based on the contrast evaluation value obtained from the image signal obtained from the imaging element 141.

[0017] The subject image formed on the imaging plane of the imaging element 141 by the photographing lens 101 is converted into an electrical signal (image signal) by the photoelectric conversion elements each of the plurality of pixels arranged in the imaging element 141. In the present embodiment, on the imaging element 141, pixels of m in the horizontal direction and n (n and m are plural) in the vertical direction are arranged in a matrix, and two photoelectric conversion elements (photoelectric conversion regions) are provided for each pixel. The signal reading from the imaging element 141 is controlled by the imaging control unit 143 according to an instruction from the CPU 151.

[0018] The image signal read from the imaging element 141 is supplied to the imaging signal processing unit 142. The imaging signal processing unit 142 applies signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing to the image signal and outputs it to the imaging control unit 143. The imaging control unit 143 accumulates the image signal received from the imaging signal processing unit 142 in the RAM (Random Access Memory) 154.

[0019] The change acquisition unit 162 is constituted by, for example, position and attitude sensors such as a gyro, an acceleration sensor, and an electronic compass, and measures the position and attitude change of the imaging device with respect to the photographing scene. The acquired position and attitude change is stored in the RAM 154 and referred to from the subject tracking unit 161.

[0020] The image processing unit 152 applies predetermined image processing to the image data accumulated in the RAM 154. The image processing applied by the image processing unit 152 includes so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing, as well as signal format conversion processing, scaling processing, etc., but is not limited thereto. Also, information regarding the subject luminance for use in automatic exposure control (AE) can be generated by the image processing unit 152. Information regarding a specific subject region is supplied from the subject tracking unit 161 and may be used, for example, in white balance adjustment processing. In the case of performing AF by the contrast detection method, the image processing unit 152 may generate an AF evaluation value. The image processing unit 152 stores the processed image data in the RAM 154.

[0021] When recording the image data stored in the RAM 154, the CPU 151 generates a data file according to the recording format, for example, by adding a predetermined header to the image processing data. At this time, the CPU 151 encodes the image data in the compression / decompression unit 153 to compress the amount of information as necessary. The CPU 151 records the generated data file in a recording medium 157 such as a memory card.

[0022] Also, when displaying the image data stored in the RAM 154, the CPU 151 scales the image data in the image processing unit 152 so as to fit the display size in the display unit 150, and then writes it into the area (VRAM area) used as the video memory in the RAM 154. The display unit 150 reads the image data for display from the VRAM area of the RAM 154 and displays it on a display device such as an LCD or an organic EL display.

[0023] The imaging device 100 according to the present embodiment causes the display unit 150 to function as an electronic viewfinder (EVF) by immediately displaying the captured moving image on the display unit 150 during moving image shooting (shooting standby state or during moving image recording). The moving image and its frame image displayed when the display unit 150 functions as an EVF are called a live view image or a through image. Also, when the imaging device 100 performs still image shooting, the imaging device 100 displays the immediately preceding still image on the display unit 150 for a certain period of time so that the user can confirm the shooting result. These display operations are also realized under the control of the CPU 151.

[0024] The operation unit 156 includes switches, buttons, keys, touch panels, etc. for the user to input instructions to the imaging device 100. The input through the operation unit 156 is detected by the CPU 151 through the bus 160, and the CPU 151 controls each unit to realize an operation according to the input.

[0025] The CPU 151 has one or more programmable processors such as a CPU or an MPU, and controls each part by loading a program stored in the storage unit 155 into the RAM 154 and executing it, thereby realizing the functions of the imaging device 100. The CPU 151 also executes AE processing to automatically determine exposure conditions (shutter speed or integration time, aperture value, sensitivity) based on the subject luminance information. The subject luminance information can be obtained, for example, from the image processing unit 152. The CPU 151 can also determine the exposure conditions based on a specific subject area such as a person's face.

[0026] During video shooting, the CPU 151 fixes the aperture and controls the exposure with the electronic shutter speed (integration time) and the gain magnitude. The CPU 151 notifies the imaging control unit 143 of the determined integration sensitivity and the gain magnitude. The imaging control unit 143 controls the operation of the image sensor 141 so that shooting is performed according to the notified exposure conditions.

[0027] The result of the subject tracking unit 161 can be used, for example, for automatically setting the focus detection area. As a result, a tracking AF function for a specific subject area can be realized. Also, AE processing can be performed based on the luminance information of the focus detection area, or image processing (such as gamma correction processing or white balance adjustment processing) can be performed based on the pixel values of the focus detection area. Additionally, image blur correction can also be performed using the motion vectors between images calculated by the subject tracking unit 161. Specifically, the image processing unit 152 refers to the motion vectors between images calculated by the subject tracking unit 161 and calculates the component due to the blur of the imaging device. It is possible to correct the image blur by driving the lens in the imaging lens 101 so as to correct the blur. Also, although not shown, it is possible to provide a driving element to the image sensor 141 and control the position of the image sensor 141 so as to correct the blur of the imaging device.

[0028] Note that the CPU 151 may superimpose and display an indicator (for example, a rectangular frame surrounding the area) representing the position of the current subject area on the display image.

[0029] The battery 159 is managed by the power management unit 158 and supplies power to the entire imaging device 100. The storage unit 155 stores programs executed by the CPU 151, setting values necessary for program execution, GUI data, user setting values, and the like. For example, when a transition from the power-off state to the power-on state is instructed by an operation of the operation unit 156, the program stored in the storage unit 155 is read into a part of the RAM 154, and the CPU 151 executes the program.

[0030] (Configuration of the subject tracking unit) FIG. 2 is a block diagram showing a functional configuration example of the subject tracking unit 161.

[0031] The subject detection unit 201 sequentially receives image signals from the image processing unit 152 in time series, detects a specific subject of the imaging target included in each image, and specifies a subject area including the detected subject. The subject detection unit 201 outputs information such as the position information of the subject area in the image and the reliability of the detection accuracy as the subject detection result.

[0032] The feature amount registration unit 202 registers the image data of the subject area detected by the subject detection unit 201 as a feature amount area.

[0033] The motion vector calculation unit 203 calculates a motion vector between images from the sequentially supplied images.

[0034] The subject motion vector calculation unit 204 calculates the overall motion vector of the feature amount area registered by the feature amount registration unit 202.

[0035] The local area motion vector calculation unit 205 of the subject calculates the local area motion vector of the feature amount area registered by the feature amount registration unit 202.

[0036] The reliability calculation unit 206 calculates the reliability of the motion vector calculated by the motion vector calculation unit 203.

[0037] The weight map creation unit 207 creates a weight map of the registered feature amount region using the motion vector of the local region of the feature amount region calculated by the motion vector calculation unit 205 of the local region of the subject and the reliability calculated by the reliability calculation unit 206.

[0038] In the tracking unit 208, using the weight map created by the weight map creation unit 207, a region with a high degree of similarity to the feature amount region registered by the feature amount registration unit 202 is searched for as the subject region from the sequentially supplied images. The search result includes information such as the subject region in the image, reliability, and the motion vector of the subject, and is used by various processing blocks such as the CPU 151.

[0039] (Processing flow of the imaging device) With reference to the flowchart of FIG. 3, the video shooting operation of the imaging device 100 according to the present embodiment, which involves subject detection, subject tracking, and motion vector detection processing for detecting motion vectors between images, will be described. Each flow of this flowchart is executed by each unit under the instruction of the CPU 151 or the CPU 151. The video shooting operation is executed during shooting standby or video recording. Note that there are differences in details such as the resolution of the images (frames) handled during shooting standby and video recording. On the other hand, since the content of the processing related to subject detection, subject tracking, and motion vector detection processing for detecting motion vectors of the subject and the background is basically the same, it will be described below without particular distinction.

[0040] In S301, the CPU 151 determines whether the power of the imaging device 100 is ON. If it is determined that it is not ON, the process ends. If it is determined that it is ON, the process proceeds to S302.

[0041] In S302, the CPU 151 controls each unit, executes imaging processing for one frame, and proceeds to S303. Here, a pair of parallax images and an imaging image for one screen are generated and stored in the RAM 154.

[0042] In S303, the CPU 151 causes the subject tracking unit 161 to execute subject detection, subject tracking, and motion vector detection processing for detecting a motion vector between images. Details of the processing will be described later. The position, size, and motion vector of the subject area are notified from the subject tracking unit 161 to the CPU 151 and stored in the RAM 154. The CPU 151 sets a focus detection area based on the notified subject area.

[0043] In S304, the CPU 151 causes the focus control unit 133 to execute focus detection processing. The focus control unit 133 combines a plurality of A signals obtained from a plurality of pixels arranged in the same row among the plurality of pixels included in the focus detection area in one pair of parallax images to form an A image (signal), and combines a plurality of B signals to form a B image (signal). Then, the focus control unit 133 calculates the correlation amount between the A image and the B image while shifting the relative position between the A image and the B image, and obtains the relative position at which the similarity between the A image and the B image is the highest as the phase difference (shift amount) between the A image and the B image. Further, the focus control unit 133 converts the phase difference into a defocus amount and a defocus direction.

[0044] In S305, the focus control unit 133 drives the focus motor 132 according to the lens drive amount and the drive direction corresponding to the defocus amount and the defocus direction obtained in S304, and moves the focus lens 131. When the lens drive process is completed, the process returns to S301.

[0045] Thereafter, until it is determined in S301 that the power switch is not ON, the processes of S302 to S305 are repeatedly executed. As a result, the subject area is searched for a plurality of time-series images, and the subject tracking function is realized. Note that in FIG. 3, the subject tracking process is assumed to be executed for each frame, but it may be executed every several frames for the purpose of reducing the processing load and power consumption.

[0046] (Flow of subject tracking process) Referring to the flowchart of FIG. 4, the processing flow of the subject tracking unit 161 will be described. Each flow of this flowchart is executed by each unit under the instruction of the CPU 151 or the CPU 151.

[0047] First, in S401, an input image is supplied from the imaging control unit 143 to the subject tracking unit 161.

[0048] In S402, the subject detection unit 201 sets a plurality of evaluation regions with different center positions and sizes for the image input from the imaging control unit 143, and detects the subject from each evaluation region. Any known method may be used for the subject detection method. For example, it may be automatically detected using the feature extraction process of a specific subject by CNN (Convolutinal Neutral Networks), or the signal of the touch operation from the operation unit 156 may be input and specified by the user.

[0049] In S403, the feature amount registration unit 202 registers the subject region detected by the subject detection unit 201 or the subject region of the previous frame detected by the tracking unit 20 8 described later and stored in the RAM 154 as a feature amount region. In this embodiment, the registration method of the feature amount region is to adopt the result of the subject detection unit 201 in the first frame, and in subsequent frames, the result of the tracking unit 20 8 of the previous frame is adopted.

[0050] In S404, the motion vector calculation unit 203 calculates the motion vector (the first motion vector) of each pixel using the image sequentially supplied in S401 and the image of the current frame and the image one frame before. Any known method may be used for the first motion vector calculation method. In this embodiment, the LucasKanade method is used. Let the luminance Y(x, y, t) of the coordinate (x, y) in the frame at time t, and the luminance of the pixel after movement in the frame after Δt be Y(x + Δx, y + Δy, t + Δt), and by solving Equation 1, Δx and Δy are calculated as the motion vectors of each pixel. [Equation 1] Y(x,y,t)=Y(x + Δx,y + Δy,t + Δt)

[0052] In S405, the subject motion vector calculation unit 204 that calculates the motion vector (the second motion vector) of the entire subject calculates one motion vector of the subject from the first motion vector calculated in S404 and the feature amount region registered in S403. The second motion vector calculation method performs histogram processing on the plurality of vectors calculated in S404 within the feature amount region, and calculates one motion vector with the largest number of bins as the motion vector of the subject.

[0053] In S406, the motion vector calculation unit 205 of the local region of the subject that calculates the motion vector (the third vector) of the local region of the subject calculates the motion vector of the local region of the feature amount region registered in S403 as the third motion vector from the first motion vector calculated in S404 and the second motion vector calculated in S405. The third motion vector calculation method calculates the difference value between the first motion vector calculated in S404 and the second motion vector calculated in S405 as the third motion vector.

[0054] In S407, the reliability calculation unit 206 calculates the reliability in the first motion vector calculated in S404. In the Lucas Kanade method, since the reliability can be obtained simultaneously with the first motion vector by solving Equation 1, this can be used. Other methods such as performing edge extraction processing, determining a low contrast region, and lowering the reliability of the low contrast region, or performing occlusion determination and lowering the reliability of the occlusion region are also acceptable, and the method is not limited.

[0055] In S408, the weight map creation unit 207 creates a weight map of the feature amount region registered in S403 based on the third motion vector calculated in S406 and the reliability calculated in S407. Fig. 5 shows an overview of weight map creation using motion vectors. When Fig. 5(a) is the image of the previous frame and Fig. 5(b) is the current frame, the motion vector as shown in Fig. 5(c) is calculated by S404.

[0056] Fig. 5(d) is the reliability map calculated by S407. The black region of 501 indicates where each pixel has moved from the position in the image of the previous frame to the position in the image of the current frame of Movement of each pixel determined that Result as and shows the region evaluated to have high reliability. On the other hand, the white region of 502 indicates where each pixel has moved from the position in the image of the previous frame to the position in the image of the current frame of Movement of each pixel determined that Result as and shows the region evaluated to have low reliability.

[0057] When the feature amount region registered by S403 is the one shown in FIG. 5(e), FIG. 5(f) becomes a reliability map, a first motion vector, and a weight map created from the second motion vector. At this time, the magnitude of the motion vector to be used is, in S406, a third motion vector obtained by canceling the second motion vector calculated in S405 from the first motion vector calculated in S404. The black region of 503 is a region with high reliability and few third motion vectors, and the weight is set to 1. The hatched region of 504 is a region with high reliability and large third motion vectors, and the weight is set to 0.5. The white region of 505 is a region with low reliability regardless of the third motion vector, and the weight is set to 0. In this embodiment, the magnitude of the third motion vector is represented by a binary value of large and small, but it may be represented by a multi-value. As the third motion vector increases, the weight approaches 0, and as the third motion vector decreases, the weight approaches 1. Similarly, in this embodiment, the reliability is represented by a binary value of high and low, but it may be represented by a multi-value. As the reliability increases, the weight approaches 1, and as the reliability decreases, the weight approaches 0. That is, the weight map is a map showing the distribution of weights for each region where the motion vector is calculated.

[0058] In S409, in the trailing part 208, the subject is tracked from the sequentially supplied images using the feature amount region registered by S403 and the weight map created in S408. Details of the subject tracking process will be described later.

[0059] (Details of the Subject Tracking Process) With reference to FIG. 6, the tracking process in S408 of FIG. 4 will be described.

[0060] Using the feature amount region registered in step S403, search for the subject region. The search result becomes the output information of the trailing part 208. In this embodiment, a search method by template matching using the feature amount region as a template is applied and will be described with reference to FIG. 6. Template matching is a technique of setting a pixel pattern as a template and searching for the region with the highest similarity to the template within an image. As the similarity between the template and the image region, a correlation amount such as the sum of absolute differences between corresponding pixels can be used.

[0061] FIG. 6(a) schematically shows a template 601 and its configuration example 602. When performing template matching, the pixel pattern used as the template is set in advance as the feature amount region 604. Here, the template 601 has a size of horizontal pixel number W and vertical pixel number H, and using the luminance values of the pixels included in the template 601, pattern matching is performed in the trailing part 208 to perform pattern matching.

[0062] The feature amount T(i, j) of the template 601 used for pattern matching can be expressed by the following formula 2 when the coordinates within the template 601 are represented in a coordinate system as shown in FIG. 6(a). [Formula 2] T(i,j)={T(0,0),T(1,0),···,T(W-1,H-1)}

[0063] FIG. 6(b) shows an example of the search region 603 of the subject region and its configuration 605. The search region 603 is the range in which pattern matching is performed within the image and may be the whole or a part of the image. The coordinates within the search region 603 are represented as (x, y). The region 604 has the same size as the template 601 (horizontal pixel number W, vertical pixel number H) and is the object for calculating the similarity to the template 601. The trailing part 208 calculates the similarity between the luminance values of the pixels included in the region 604 and the luminance values included in the template 601.

[0064] Therefore, the feature amount S(i, j) of the region 604 used for pattern matching can be expressed by the following formula 3 when the coordinates in the template 601 are represented in the coordinate system as shown in FIG. 6(b). [Formula 3] S(i,j)={S(0,0),S(1,0),···,S(W-1,H-1)}

[0065] FIG. 6(c) schematically shows the weight map 606 and its configuration example 607. Here, the region 607 has the same size as the template 601 (horizontal pixel number W, vertical pixel number H), and is the target to which weights are applied with respect to the similarity between the template 601 and the search region 603. Therefore, the weight A(i, j) of the weight map 606 used for pattern matching can be expressed by the following formula 4 when the coordinates in the template 601 are represented in the coordinate system as shown in FIG. 6(c). [Formula 4] A(i,j)={A(0,0),A(1,0),···,A(W-1,H-1)}

[0066] Regarding the similarity between the template 601 and the region 604, the evaluation value obtained by weighting with the weight map 606 is defined as V(x, y), and the sum of absolute differences (SAD) value shown in the following formula 5 is calculated.

[0067] [Number]

[0068] Here, V(x, y) represents the evaluation value at the coordinates (x, y) of the upper left vertex of the region 604. The trailing part 208 calculates the evaluation value V(x, y) at each position while shifting the region 604 one pixel at a time from the upper left to the right direction of the search region 603, and then, when x = (X - 1)(W - 1) is reached, setting X = 0 and shifting one pixel at a time in the downward direction. The coordinates (x, y) at which the calculated evaluation value V(x, y) indicates the minimum value represent the position of the region 604 having the pixel pattern most similar to the template 601. The region 604 where the evaluation value V(x, y) indicates the minimum value is detected as the subject region existing within the search region. Note that when the reliability of the search result is low (for example, when the minimum value of the evaluation value V(x, y) exceeds the threshold), it may be determined that the subject region has not been found.

[0069] Here, an example of using the feature amount of the luminance value for pattern matching has been shown, but a feature amount having a plurality of values (for example, lightness, hue, saturation) may also be used. Also, an example of using SAD as the evaluation value of similarity has been shown, but other evaluation values, for example, normalized cross-correlation (NCC) or ZNCC, may also be used.

[0070] (Effect) As described above, according to this embodiment, by weighting the feature amount region based on the motion vector, even when the subject has a plurality of motions, the performance of subject tracking can be improved.

[0071] (Other Embodiments) The present invention can also be realized by supplying a program that realizes the functions of the above-described embodiment to a system or device via a network or a storage medium, and causing one or more processors in the computer of the system or device to read and execute the program.

Explanation of Reference Numerals

[0072] 100 Imaging device 101 Lens unit 102 Fixed first lens group 103 Diaphragm 104 Aperture motor 105 Aperture control unit 111 Zoom lens 112 Zoom motor 113 Zoom control unit 121 Fixed three-group lens 131 Focus lens 132 Focus motor 133 Focus control unit 141 Image sensor 142 Image signal processing unit 143 Imaging control unit 150 Monitor display 151 CPU 152 Image processing unit 153 Compression / decompression unit 154 RAM 155 Flash memory 156 Operation switch 157 Image recording medium 158 Power management unit 159 Battery 160 Bus 161 Subject tracking unit 162 Change acquisition unit

Claims

1. Feature amount region registration means for registering the region of the subject detected from the image as a feature amount region; First motion vector calculation means for calculating a first motion vector which is a motion vector of each pixel between images; Second motion vector calculation means for calculating a second motion vector which is a motion vector of the entire subject from the first motion vector and the feature amount region; Third motion vector calculation means for calculating a third motion vector which is a motion vector of a local region of the subject from the first motion vector and the second motion vector; Reliability evaluation means for evaluating the reliability of the first motion vector; Weight map creation means for creating a weight map of the feature amount region based on the third motion vector and the reliability; Subject tracking means for tracking the subject based on the feature amount region and the weight map, wherein the subject tracking device is characterized by comprising the same.

2. The subject tracking device according to claim 1, wherein the second motion vector is calculated as the bin with the maximum number by performing histogram processing on the first motion vector within the feature amount region.

3. The subject tracking device according to claim 1, wherein the weight map creation means lowers the weight of the weight map corresponding to the local region when the third motion vector is a second value larger than a first value as compared with when the third motion vector is the first value.

4. The subject tracking device according to any one of claims 1 to 3, wherein the weight map creation means lowers the weight of the weight map corresponding to a region where the reliability is a second value lower than a first value as compared with a region where the reliability is the first value.

5. An image pickup device comprising an image pickup element for obtaining a picked-up image of a subject image formed through an imaging optical system; and the subject tracking device according to any one of claims 1 to 4.

6. The image pickup device according to claim 5, wherein image blur correction is performed by moving a lens provided in the imaging optical system with reference to the first motion vector.

7. The image pickup device according to claim 5 or 6, wherein image blur correction is performed by controlling the position of the image pickup element with reference to the first motion vector.

8. A feature amount region registration step of registering the region of the subject detected from the image as a feature amount region; A first motion vector calculation step of calculating a first motion vector which is a motion vector of each pixel between images; A second motion vector calculation step of calculating a second motion vector which is a motion vector of the entire subject from the first motion vector and the feature amount region; A third motion vector calculation step of calculating a third motion vector which is a motion vector of a local region of the subject from the first motion vector and the second motion vector; A reliability evaluation step of evaluating the reliability of the first motion vector; A weight map creation step of creating a weight map of the feature amount region based on the third motion vector and the reliability; A subject tracking step of tracking the subject based on the feature amount region and the weight map, wherein the control method of the subject tracking device is characterized by having the above steps.

9. A program for causing a computer to function as each means of the subject tracking device according to any one of Claims 1 to 4.

10. A computer-readable storage medium storing a program for causing a computer to function as each means of the subject tracking device according to any one of Claims 1 to 4.

Images