Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus capable of properly updating a tracking area even when the tracking area is designated at a position deviated from an intended object area.SOLUTION: The image processing apparatus includes a setting unit configured to set a tracking area in a first image, a generation unit configured to generate a template used for template matching based on the set tracking area, and a detection unit configured to detect a first area similar to the template in a second image by applying the template matching using the template generated by the generation unit to the second image. The setting unit compares an evaluation value of a first area detected in the second image by the detection unit with an evaluation value of a second area of the second image corresponding in position to the tracking area of the first image, and sets one of the first area and the second area as the tracking area of the second image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a technique for tracking a subject. [Background technology]

[0002] There is a known subject tracking technology that sequentially searches for a region (subject region) in which a specific subject appears in a plurality of images captured in time series. Template matching is also known as a method for searching for a subject region (see Patent Document 1). Template matching is a method for searching for a region in an image to be searched that has the highest similarity to an image registered as a template. An index of similarity between an image region of the same size as the template can be calculated using various methods. For example, the sum of absolute differences between corresponding pixels can be calculated as an index of similarity; in this case, a smaller sum indicates a higher similarity. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-134438 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, the camera may be configured so that the user can specify a subject to track by specifying a position in a live view display image. In this case, the user can specify the desired position by touching the touch display showing the live view or by moving a pointer such as a cursor using a combination of key and button operations.

[0005] However, since the position is specified while holding the imaging device, the specified position may deviate from the area of ​​the intended subject. In this case, the tracking process may be performed using an area not intended by the user as a template, and the intended subject may not be tracked.

[0006] One of the objectives of the present invention is to at least alleviate the problems of the conventional technology and to provide an image processing device and an image processing method that can appropriately update a tracking area even when the tracking area is specified at a position that is shifted from the area of ​​the intended subject. [Means for solving the problem]

[0007] The above-mentioned object can be achieved by an image processing device for tracking a subject, comprising: a setting means for setting a tracking area in a first image; a generation means for generating a template to be used for template matching based on the set tracking area; and a detection means for detecting a first area in the second image that is similar to the template by applying template matching using the template generated by the generation means to the second image, wherein the setting means compares an evaluation value of the first area detected in the second image by the detection means with an evaluation value of a second area in the second image that corresponds in position to the tracking area in the first image, and sets one of the first area and the second area as the tracking area of ​​the second image. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method that can appropriately update a tracking area even if the tracking area is specified at a position that is shifted from the area of ​​the intended subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device as an example of an image processing device according to an embodiment; [Figure 2]1 is a flowchart illustrating a template stabilization process according to an embodiment of the present invention; [Figure 3] Flowchart showing the details of the tracking area setting process in Figure 2 [Figure 4] FIG. 10 is a schematic diagram illustrating the effect of the tracking area setting process according to the embodiment; [Figure 5] FIG. 10 is a schematic diagram illustrating the effect of the first predetermined time T1 in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiments, the present invention will be described with respect to an imaging device such as a digital camera or digital video camera. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device capable of handling image data. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] (Configuration of imaging device) An example of the configuration of an image capture device 100 as an example of an image processing device according to an embodiment of the present invention will be described with reference to Fig. 1. Here, it is assumed that the lens unit 101 of the image capture device 100 is not replaceable, but the present invention can also be implemented in an image capture device with an interchangeable lens.

[0013] Lens unit 101 has fixed lenses 102 and 121, movable lenses, zoom lens 111 and focus lens 131, and aperture 103. Note that each lens shown as a single lens in the figure may be made up of multiple lenses.

[0014] The aperture 103 also functions as a shutter. The aperture diameter and opening / closing operation of the aperture 103 are controlled by an aperture control unit 105 driving an aperture motor 104 (AM) under the control of a CPU 151.

[0015] The zoom lens 111 moves along the optical axis of the lens unit 101 to change the focal length (angle of view) of the lens unit 101. The position of the zoom lens 111 is controlled by the zoom control unit 113 driving the zoom motor 112 (ZM) under the control of the CPU 151.

[0016] Focus lens 131 moves along the optical axis of lens unit 101 to change the focal distance of lens unit 101. The position of focus lens 131 is controlled by focus control unit 133 driving focus motor 132 (FM) under the control of CPU 151. The drive direction and drive amount of focus lens 131 are determined by CPU 151 according to the defocus amount calculated by defocus calculation unit 163.

[0017] The CPU 151 (main control unit) is one or more processors, and for example, reads one or more programs stored in the ROM 155 into the RAM 154 and executes them to control the functional blocks connected to the bus 160 and realize the functions of the imaging device 100. Note that at least some of the functions realized by the functional blocks connected to the bus 160 may be implemented by the CPU 151 executing the programs.

[0018] The lens unit 101 forms an optical image of a subject on the imaging surface of the image sensor 141. The image sensor 141 may be, for example, a CCD image sensor or a CMOS image sensor having a color filter. The image sensor 141 has a plurality of pixels, each having a photoelectric conversion unit, arranged, for example, in a matrix, and converts the optical image of the subject into an analog image signal by the plurality of pixels. The image sensor 141 is provided with a circuit for controlling the operation of the pixels. The analog image signal read out from the image sensor 141 is supplied to a signal processing unit 142.

[0019] The signal processing unit 142 applies processes such as noise removal, defective pixel correction, and A / D conversion to the analog image signal to generate a digital image signal in RAW format (RAW image data). The signal processing unit 142 outputs the RAW image data to the imaging control unit 143.

[0020] The imaging control unit 143 stores the RAW image data in the RAM 154. The imaging control unit 143 also controls the operation of the imaging element 141 under the control of the CPU 151.

[0021] The image processing unit 152 applies predetermined image processing to the RAW image data stored in the RAM 154 to generate signals and image data, and acquire and / or generate various types of information. The image processing unit 152 may be, for example, a dedicated hardware circuit such as an ASIC designed to achieve a specific function, or may be configured to achieve a specific function by a programmable processor such as a DSP executing software.

[0022] The image processing applied by the image processing unit 152 includes color interpolation, correction, data processing, evaluation value calculation, special effect processing, and the like. Color interpolation, also known as demosaicing, is a process of interpolating the values ​​of color components that cannot be obtained for each pixel at the time of capture from the values ​​of surrounding pixels. Correction processes include white balance adjustment, gradation correction (gamma processing), correction for optical aberrations and peripheral shading of the lens unit 101, and color correction. Data processing includes synthesis, scaling, and data file header information generation. Evaluation value calculation processes include generation of signals and evaluation values ​​used for autofocus detection (AF) and calculation of evaluation values ​​used for automatic exposure control (AE). Special effect processing includes blurring, color tone modification, relighting, and the like. The image processing unit 152 can also apply image processing using the detection results of the object detection unit 162, which will be described later. For example, the image processing unit 152 can execute pattern matching in subject tracking processing (calculation processing of a value (amount of correlation) indicating the degree of correlation between image regions) by using the detection result by the object detection unit 162. Note that these are examples of image processing that the image processing unit 152 can apply, and do not limit the image processing that the image processing unit 152 applies.

[0023] Among the image processing described above, the color interpolation processing and correction processing are also called RAW image data development processing. The image processing unit 152 applies image processing including the color interpolation processing and correction processing to the RAW image data, and generates, for example, display image data to be displayed on the display 150 and recording image data to be recorded in the recording unit 157, and stores them in the RAM 154.

[0024] The CPU 151 uses the evaluation value generated by the image processing unit 152 to determine the shooting conditions (aperture value, shutter speed (exposure time), and shooting sensitivity) of the imaging device 100. The CPU 151 controls the aperture control unit 105 according to the determined aperture value and shutter speed. The CPU 151 also controls the imaging control unit 143 according to the determined exposure time and shooting sensitivity.

[0025] The codec 153 encodes data and decodes encoded data. The codec 153 can support multiple encoding methods. The codec 153 encodes recording image data and RAW image data stored in the RAM 154. The codec 153 also decodes encoded data that has been read from the recording unit 157 or received from an external device and stored in the RAM 154.

[0026] The RAM 154 is a so-called main memory, and is used to store programs and data required for executing the programs, and to temporarily store image data, etc. A part of the RAM 154 is also used as a VRAM.

[0027] The ROM 155 is an electrically rewritable non-volatile memory. The ROM 155 stores programs and constants executed by the CPU 151, various setting values ​​of the image capture device 100, GUI data, etc. When the image capture device 100 transitions from a power-off state to a power-on state, the programs stored in the ROM 155 are read into the RAM 154 and executed by the CPU 151.

[0028] The display 150 is, for example, a liquid crystal display (LCD). By displaying moving images being captured in real time on the display 150, the display 150 can function as an electronic viewfinder (EVF). The display 150 also displays GUI screens such as menu screens, recorded images, and information such as the status and setting values ​​of the imaging device 100.

[0029] The object detection unit 162 applies a predetermined subject detection process to image data (for example, image data for display) stored in RAM 154, and detects an area (subject area) determined to contain a predetermined subject. In this embodiment, the object detection unit 162 is capable of applying a plurality of subject detection processes with different accuracy and processing time. In the following, it is assumed that the object detection unit 162 is capable of applying a first subject detection process and a second subject detection process that has lower detection accuracy than the first subject detection process but shorter processing time, but it may also be capable of applying three or more types of subject detection process.

[0030] As an example, the first subject detection process is a process for detecting a characteristic region using Haar-Like features, and the second subject detection process is a process for detecting a characteristic region based on color distribution. Furthermore, each subject detection process is assumed to have learned about the subject to be detected. When detecting a characteristic region based on color distribution, if the color distribution of a target region and its surrounding regions differs by a predetermined amount or more, the target region can be detected as a characteristic region. The second subject detection process makes it easier to extract the boundary between the subject region and the background, but has a higher probability of erroneously detecting the background as a subject region than the first subject detection process.

[0031] The first subject detection process and the second subject detection process may each be performed by applying different parameters for each type of subject to be detected. For example, the first subject detection process and the second subject detection process may be performed using parameters for multiple types of objects that can be main subjects, such as human or animal faces, automobiles, airplanes, trains, birds, and flowers.

[0032] Operation unit 156 is a general term for input devices provided for the user to give instructions to imaging device 100. Input devices include buttons, keys, dials, touch panels, and the like. If display 150 is a touch display, display 150 also functions as operation unit 156. Functions are statically or dynamically assigned to the input devices that make up operation unit 156. When CPU 151 detects an operation of an input device, it executes an operation corresponding to the detected operation.

[0033] The defocus calculation unit 163 calculates the defocus amount of the focus detection area by a phase difference detection method using a signal pair obtained from a dedicated focus sensor or a signal pair generated from image data by the image processing unit 152. The focus detection area in which the lens unit 101 is focused within the shooting range is set by the user or the CPU 151.

[0034] The CPU 151 controls the focus control unit 133 based on the defocus amount calculated by the defocus calculation unit 163. As a result, the focus lens 131 is driven by the FM 132 to a position according to the defocus amount, and the lens unit 101 focuses on the focus detection area.

[0035] The battery 159 is, for example, a secondary battery attached to the image capture device 100. The battery 159 is managed by a power management unit 158 ​​and supplies power to the entire image capture device 100.

[0036] The position and orientation detection unit 161 is a position and orientation sensor such as a gyro, acceleration sensor, or electronic compass, and outputs values ​​representing the orientation and movement of the image capture device 100 at a predetermined cycle. The output values ​​of the position and orientation detection unit 161 are stored in the RAM 154.

[0037] (Template stabilization) Next, an example of template stabilization processing performed at the beginning of subject tracking processing in this embodiment will be described with reference to the flowchart shown in FIG. 2. The subject tracking processing is performed in response to a tracking area being specified by the user via the operation unit 156, for example, while the imaging device 100 is capturing a video. There are no limitations on the method for specifying the tracking area, and any method of specifying a range or position using a touch operation or an input device on a live view image displayed on the display 150 may be used. For example, the user can specify the tracking area by tapping the live view image to specify the position of the subject to be tracked, or by framing the live view image so that the subject to be tracked is positioned in the center of the live view image and pressing a predetermined button on the operation unit 156.

[0038] The template stabilization process is a process that, for example, allows the user to specify a position that is different from the intended subject as the position of the tracking area, or to set an appropriate tracking area during a period when framing may not be determined.

[0039] In S200, the CPU 151 initializes a variable t representing the elapsed time of the tracking process to 0 and starts measuring the elapsed time using a timer. Alternatively, the CPU 151 may obtain the current time from an internal clock and store it in the RAM 154 as the start time of the tracking process.

[0040] In S201, the CPU 151 determines whether the size of the specified tracking area is equal to or smaller than a predetermined size, and if it is determined that the size is equal to or smaller than the predetermined size, executes S202, and if it is not determined that the size is equal to or smaller than the predetermined size, executes template stabilization processing. Note that if the user specifies the position of the tracking area and the size of the tracking area is set by the imaging device 100 (CPU 151), S201 is skipped.

[0041] If the size of the specified tracking area is small, there is a high possibility that the tracking area has been designated as being shifted from the subject area, so template stabilization processing is performed. On the other hand, if the size of the tracking area is not small, there is a low possibility that the tracking area has been designated as being shifted from the subject area, so template stabilization processing is not performed, and normal subject tracking processing is performed using the specified tracking area as a template.

[0042] In S202, the CPU 151 captures one frame of a moving image through the imaging control unit 143. As a result, RAW image data for one frame is stored in the RAM 154. The image processing unit 152 generates display image data from the RAW image data and stores the display image data in the RAM 154.

[0043] In S203, the image processing unit 152 as a detection means performs template matching processing on the display image data stored in RAM 154, using the tracking area specified by the user or the updated tracking area as a template. This corresponds to a search process for the subject area in the current frame. Template matching detects an area in the current frame that is similar to the template. Note that since there is no template to use for the first frame after starting the subject tracking processing, S203 is skipped.

[0044] In S204, the image processing unit 152, which serves as a setting unit, sets a tracking area for the image data of the current frame. For the first frame after starting the subject tracking process, a tracking area specified by the user or a rectangular area of ​​a predetermined size centered on a position (coordinates) specified by the user is set as the tracking area. The setting process of the tracking area for the second frame and thereafter will be described in detail later.

[0045] In S205, the image processing unit 152, which serves as a generating unit, stores data of the tracking area set in S204 as a template in the RAM 154, among the display image data of the current frame stored in the RAM 154. In this embodiment, the template is updated for each frame. However, if predetermined conditions are met, such as if the reliability of the tracking area set in S204 is low or if the frame rate is high, the current template may be maintained without updating the template in S205.

[0046] In S206, object detection unit 162 applies the first subject detection process to the display image data stored in RAM 154. Object detection unit 162 stores in RAM 154 the total number of detected subject regions, the position, size, reliability of each individual subject region, and the like as processing results.

[0047] In S207, object detection unit 162 applies the second subject detection process to the display image data stored in RAM 154. Object detection unit 162 stores in RAM 154 the total number of detected subject regions, the position, size, reliability of each subject region, and the like as processing results.

[0048] In S208 to S212, CPU 151 determines which of the results of the first subject detection process and the second subject detection process to adopt. In S208, CPU 151 refers to RAM 154 and determines whether or not a subject area whose distance from the currently set tracking area is equal to or less than a predetermined value has been detected in the first subject detection process. If CPU 151 determines that a subject area whose distance from the currently set tracking area is equal to or less than a predetermined value has been detected in the first subject detection process, CPU 151 executes S212, and if not, executes S209.

[0049] In S209, the CPU 151 determines whether the time elapsed since the start of the subject tracking process is less than a first predetermined time T1, and if it is determined that the time elapsed since the start of the subject tracking process is less than the first predetermined time T1, the CPU 151 ends the processing of the current frame and executes S202. On the other hand, if it is not determined that the time elapsed since the start of the subject tracking process is less than the first predetermined time T1, the CPU 151 executes S210.

[0050] In S210, CPU 151 refers to RAM 154 and determines whether or not a subject area whose distance from the currently set tracking area is less than or equal to a predetermined value has been detected in the second subject detection process. If CPU 151 determines that a subject area whose distance from the currently set tracking area is less than or equal to the predetermined value has been detected in the second subject detection process, CPU 151 executes S212, and if not, executes S211.

[0051] In S211, the CPU 151 determines whether the time elapsed since the start of the subject tracking process is less than a second predetermined time T2 (>T1), and if it is determined that the time elapsed is less than the second predetermined time T2, it ends the processing of the current frame and executes S202. On the other hand, if the CPU 151 does not determine that the time elapsed since the start of the subject tracking process is less than the second predetermined time T2, it ends the template stabilization process. Thereafter, the subject tracking process continues by template matching using the tracking area at the time when the second predetermined time T2 has elapsed as the template.

[0052] In S212, CPU 151 updates the setting of the tracking area based on the detection result of the first subject detection process or the second subject detection process. For example, CPU 151 sets the subject area whose distance from the currently set tracking area is equal to or shorter than a predetermined value as the new tracking area. Furthermore, image processing unit 152 updates the template in accordance with the updated tracking area. Note that since the tracking area updated in S212 is based on the detected subject area, its size and shape do not need to be constant. Alternatively, a rectangular area of ​​the same size as that set in S204 may be set as the updated tracking area, centered on the center or center of gravity coordinates of the detected subject area.

[0053] When S212 is executed, the template stabilization process ends, and thereafter, the subject tracking process continues by template matching using the tracking area updated in S212 as the template.

[0054] (About tracking area settings) The tracking area setting process in S204 will be described in further detail with reference to the flowchart shown in FIG.

[0055] In S300, the image processing unit 152 acquires from the RAM 154 information on the area (candidate area) in the current frame that is most similar to the template, detected by template matching in S203. Then, the image processing unit 152 extracts a rectangular area of ​​a predetermined size that includes the candidate area from the current frame, and sets it as the first area. The first area may be, for example, a rectangular area centered on the center or centroid coordinates of the candidate area, a rectangular area that includes the most candidate areas, or a rectangular area that includes the most candidate areas and whose center is closest to the center or centroid coordinates of the candidate area, but is not limited to these.

[0056] In S301, the image processing unit 152 extracts an area of ​​the current frame that corresponds to the previously set (updated) tracking area, and sets it as the second area. Here, the tracking area set (updated) in S205 or S212 in the processing of the previous frame is the previously set (updated) tracking area. If the tracking area has been updated in S212, the image processing unit 152 extracts a rectangular area of ​​a predetermined size centered on the center or centroid coordinates of the tracking area from the current frame, and sets it as the second area. The first area and the second area are rectangular areas of the same size.

[0057] The first area and the second area will be approximately the same area unless the user changes the shooting range by panning the camera or the like between the previous frame and the current frame.

[0058] In S302, image processing unit 152 calculates an evaluation value for each of the first and second regions acquired in S300 and S301. The evaluation value calculated here may be any evaluation value that represents the likelihood that the region is a subject (other than the background) of the image. The evaluation value represents the likelihood that the region contains some subject that is not the background, but the calculation process of the evaluation value is a simpler arithmetic process than the process of detecting a specific subject, such as the subject detection process performed by object detection unit 162.

[0059] In this embodiment, as an example, the contrast value of the region is calculated as the evaluation value. The contrast value is the sum of absolute differences between values ​​of pairs of adjacent pixels in the horizontal direction within the region. The larger the contrast value, the more likely the image within the region is to be the subject. Note that the evaluation value may also be calculated as the sum of absolute values ​​of specific band components (e.g., high-frequency components) extracted by applying filter processing to the region, or any of known feature amounts or movement amounts.

[0060] In S303, image processing unit 152 compares the evaluation value of the first region calculated in S302 with the evaluation value of the second region. If the evaluation value of the second region is greater, image processing unit 152 executes S304, and if the evaluation value of the second region is equal to or less than the evaluation value of the first region, image processing unit 152 executes S305. Note that S304 may be executed if the evaluation value of the second region is greater than the evaluation value of the first region and the difference in evaluation values ​​is equal to or greater than a predetermined value, and S305 may be executed in other cases.

[0061] In S304, the image processing unit 152 sets the second area as the tracking area, and ends the tracking area setting process. In S305, the image processing unit 152 sets the first area as the tracking area, and ends the tracking area setting process. For the tracking area set in this way, a template is generated in S205. In the template generation process in S205, the magnitude of the evaluation value calculated in S302 can be used as the reliability of the tracking area.

[0062] FIG. 4 is a diagram schematically illustrating the effect of the tracking area setting process described with reference to FIG. 3. Here, FIGS. 4(a) and 4(b) illustrate a case where the same tracking area is specified for the same scene even though the user intended to track different objects. Specifically, tracking areas 410 and 510 are specified at the same positions for the same scene in FIGS. 4(a) and 4(b), but the user intended to track an automobile in FIG. 4(a) and a plant in FIG. 4(b). Note that, for ease of explanation and understanding, it is assumed here that the automobile is stationary or that its movement between frames is negligible. It is also assumed that the tracking area is not updated based on the first object detection process and the second object detection process in S208 to S212.

[0063] In the case of Figure 4(a), in the first frame, the tracking area 410 is specified at a position shifted from the center of the intended tracking target (car), and the tracking area 410 contains only a small amount of the intended tracking target, but contains many other subjects (plants). On the other hand, in the case of FIG. 4(b), in the first frame, tracking region 510 is specified at a position that includes the intended tracking target (plant), and tracking region 510 includes almost no other subjects (cars).

[0064] In the case of FIG. 4(a), in the processing of the first frame, a tracking area 410 is generated as a template in S205. Here, assume that the user moves the imaging device 100 between the first and second frames, changing (framing) the shooting range in a direction (to the right in the figure) so that the intended tracking target (car) is in the center of the tracking area 410.

[0065] In the processing of the second frame, template matching in S203 detects region 421 as the region in the current frame that is most similar to the template. In S300, region 421 is extracted from the current frame as the first region. Meanwhile, in S301, region 422, which is in the same position as designated tracking region 410, is extracted from the current frame as the second region.

[0066] Then, in S302, evaluation values ​​are calculated for the first region (region 421) and the second region (region 422). Since the evaluation value of the second region (region 422) is greater than the evaluation value of the first region (region 421), the second region (region 422) is set as the tracking region in S304. Therefore, in S205, the area 422 is generated as a template.

[0067] As framing continues, the shooting range changes further in the third frame. In the processing of the third frame, template matching in S203 detects region 431 as the region in the current frame that is most similar to the template. In S300, region 431 is extracted from the current frame as the first region. Meanwhile, in S301, region 432, which is in the same position as region 422 in the previous frame, is extracted from the current frame as the second region.

[0068] Then, in S302, evaluation values ​​are calculated for the first region (region 431) and the second region (region 432). Since the evaluation value of the first region (region 431) is greater than the evaluation value of the second region (region 432), the first region (region 431) is set as the tracking region in S305. Therefore, in S205, the area 431 is generated as a template.

[0069] In this manner, in this embodiment, of the area extracted as the previous template and the area detected in the current frame as having the highest similarity to the template, the area that is most likely to be the subject is set as the tracking area. Therefore, even if a position that is slightly deviated from the intended subject is designated as the tracking area, the user can change the shooting range in the direction of the intended subject, so that the intended subject can be tracked.

[0070] 4(b), an appropriate tracking area 510 is set for the subject (plant) intended to be tracked in the first frame. In the processing of the first frame, the tracking area 510 is generated as a template in S205. Here, the shooting range is not changed between the first and second frames and remains substantially the same.

[0071] In the processing of the second frame, template matching in S203 detects region 521 as the region in the current frame that is most similar to the template. In S300, region 521 is extracted from the current frame as the first region. Meanwhile, in S301, region 522, which is in the same position as specified tracking region 510, is extracted from the current frame as the second region.

[0072] Then, in S302, evaluation values ​​are calculated for the first region (region 421) and the second region (region 422). Because the first region (region 421) and the second region (region 422) are almost identical, the evaluation value of the first region (region 421) and the evaluation value of the second region (region 422) are equal or the difference therebetween is small. Therefore, in S305, the first region (region 421) is set as the tracking region. Therefore, in S205, the area 421 is generated as a template.

[0073] The shooting range is substantially the same in the third frame. In the processing of the third frame, template matching in S203 detects region 531 as the region in the current frame that is most similar to the template. In S300, region 531 is extracted from the current frame as the first region. Meanwhile, in S301, region 532, which is in the same position as region 522 in the previous frame, is extracted from the current frame as the second region.

[0074] In the third frame, the first region (region 421) and the second region (region 422) are almost the same, so the evaluation value of the first region (region 421) and the evaluation value of the second region (region 422) are equal or the difference between them is small. Therefore, in S305, the first region (region 421) is set as the tracking region.

[0075] Next, the processes of S208 to S212 will be further explained. As described above, the first subject detection process can detect a subject area more accurately than the second subject detection process, but it has a higher computational load than the second subject detection process. Therefore, the first subject detection process takes longer to obtain a detection result than the second subject detection process.

[0076] If a subject area whose distance from the current tracking area is less than or equal to a threshold is detected in the first subject detection process, the detection result of the first subject detection process should be used preferentially. However, if a subject area whose distance from the current tracking area is less than or equal to a threshold is not detected in the first subject detection process, the detection result of the second subject detection process is used. In this embodiment, a first predetermined time T1 is set as the upper limit of the time to wait for detection of a subject area by the first subject detection process. Until the first predetermined time T1 has elapsed, the tracking area is determined by the tracking area setting process described above.

[0077] If the first subject detection process detects a subject area whose distance from the current tracking area is less than or equal to a threshold before the first predetermined time T1 has elapsed, the tracking area is updated in S212 based on the highly accurate detection result, which is expected to improve tracking accuracy from the next frame onwards.

[0078] On the other hand, if the first subject detection process does not detect a subject area whose distance from the current tracking area is less than or equal to the threshold value even after the first predetermined time T1 has elapsed, the detection result of the second subject detection process is used. Even in this case, the tracking area is appropriately updated by the above-mentioned tracking area setting process before the first predetermined time T1 has elapsed. If the second subject detection process detects a subject area whose distance from the current tracking area is less than or equal to the threshold value before the second predetermined time T2 has elapsed, S212 is executed.

[0079] In S212, the tracking area is updated based on the subject areas detected in the second subject detection process whose distance from the current tracking area is less than a threshold (e.g., overlapping), thereby suppressing the effects of erroneous detection by the second subject detection process.

[0080] (Method of setting the first predetermined time T1) The first predetermined time T1 can be determined appropriately taking into consideration the frame rate, etc., but for example, the first predetermined time T1 may be made longer in a state where it is difficult to accurately specify the tracking area than in other cases. This is because, even if the tracking area is specified deviated from the intended subject, it is desirable that the tracking area be set to the intended subject by the tracking area setting process before the first predetermined time T1 has elapsed and the detection result of the second subject detection process is used.

[0081] For example, if the focal length of the lens unit 101 is long, the shooting range becomes less stable, and the image displayed in live view becomes more likely to move. It is not easy to specify the intended position for an image whose display position is unstable, and there is a high possibility that a position different from the intended position will be specified. Therefore, when the focal length of the lens unit 101 when the tracking area is specified is equal to or greater than a threshold (telephoto side), the CPU 151 may make the first predetermined time T1 longer than when the focal length is less than the threshold. Alternatively, the CPU 151 may make the first predetermined time T1 longer the greater the focal length of the lens unit 101 when the tracking area is specified.

[0082] Furthermore, the first predetermined time T1 may be determined taking into consideration the movement of the image capture device 100 when the tracking area is specified. For example, if the image capture device 100 is moving when the tracking area is specified, there is a high possibility that the specified tracking area is misaligned with the intended subject. Therefore, if the CPU 151 determines that the image capture device 100 was moving when the tracking area was specified, it can set the first predetermined time T1 to be longer than if it was not determined that the image capture device 100 was moving. Based on the output signal of the position and orientation detection unit 161, the CPU 151 can determine that the image capture device 100 is moving if the amount of change per unit time in the magnitude of movement in either the yaw or pitch direction of the image capture device 100 is equal to or greater than a predetermined threshold.

[0083] Alternatively, the first predetermined time T1 may be varied depending on whether or not the moving object region is located near the outside of the tracking region. When the moving object region is located near the outside of the tracking region, the CPU 151 determines that there is a high possibility that the specified tracking region and the intended subject are misaligned, and sets the first predetermined time T1 to be longer than when the moving object region is not located near the outside of the tracking region. Note that the moving object region can be detected by any known method, and for example, the method described in JP 2020-95673 A can be used.

[0084] Alternatively, CPU 151 may determine that the first predetermined time T1 has elapsed when it is determined that the tracking area has stabilized. For example, CPU 151 can determine that the tracking area has stabilized when the percentage of times the first area has been set as the tracking area in multiple tracking area setting processes executed within the most recent predetermined time period or in a predetermined number of tracking area setting processes executed most recently is equal to or greater than a threshold value.

[0085] The first region is set as the tracking region when the template set in the previous frame is considered appropriate. Therefore, for example, if the percentage of times the first region was set as the tracking region in the most recent multiple tracking region setting processes is equal to or greater than a threshold (e.g., 80% or greater), it is highly likely that the tracking region has been continuously set to an appropriate subject. Therefore, even if the detection result of the second subject detection process is used, it is considered that accuracy is guaranteed, and it can be determined that the first predetermined time T1 has elapsed.

[0086] (Method of setting the second predetermined time T2) In this embodiment, if no subject area whose distance from the current tracking area is less than or equal to the threshold is detected during the second predetermined time T2 in either the first subject detection process or the second subject detection process, the template stabilization process is terminated and the process transitions to normal subject tracking process.

[0087] Therefore, the second predetermined time T2 is set to end, for example, after the tracking area has been stably set. Basically, the second predetermined time T2 can be set using the same concept as the first predetermined time T1. Most simply, the second predetermined time T2 can be set to twice the first predetermined time T1. Alternatively, the second predetermined time T2 may be set to the sum of the time required to detect a subject area by the second subject detection process when a subject area exists and the first predetermined time T1.

[0088] FIG. 5 shows a schematic diagram of the influence of the first predetermined time T1 on the template stabilization process in a situation similar to that shown in FIG. 4(a).

[0089] Fig. 5(a) shows a case where the first predetermined time T1 is not actually set (first predetermined time T1 = 0). Fig. 5(b) shows a case where the first predetermined time T1 is long enough for the tracking area to be stabilized in the first area by the tracking area setting in S204. Fig. 5(c) shows a case where the first predetermined time T1 is the same as in Fig. 5(b), but a subject area whose distance from the current tracking area is less than or equal to a threshold is not detected in the first subject detection process even after the first predetermined time T1 has elapsed.

[0090] In the following description, the subject area detected by the first subject detection process and the second subject detection process is the subject area whose distance from the current tracking area is equal to or less than a threshold. Reference numeral 601 denotes the subject area detected by the first subject detection process, and 602 denotes the subject area detected by the second subject detection process.

[0091] FIG. 5(a) shows a state where, at elapsed time t=0, that is, after the first predetermined time T1 has elapsed, no subject area is detected in the first subject detection process, but a subject area is detected in the second subject detection process.

[0092] In this case, a tracking area based on the subject area 602 detected by the second subject detection process at the elapsed time t = 0 is set in S212. Then, from the next frame, subject tracking process is performed using the tracking area based on the subject area 602 extracted from the current frame as a template.

[0093] In the first frame, the tracking area is set at a position different from the intended subject, and the tracking area includes an unintended subject. In this case, the subject area detected in the second subject detection process, whose distance from the current tracking area is equal to or less than the threshold, is the area of ​​the unintended subject. Furthermore, since the first predetermined time T1 is not actually provided, the unintended subject continues to be tracked without the user's change of the shooting range affecting the setting of the tracking area.

[0094] In Fig. 5(b), the first predetermined time T1 is set to T1>0 in the state of Fig. 5(a). Therefore, the tracking area setting process in S204 is repeatedly executed until a subject area is detected in the first subject detection process or the elapsed time reaches T1. Then, until the elapsed time reaches T1, the subject area 602 detected in the second subject detection process is not taken into consideration in setting the tracking area.

[0095] Between the first and second frames, the user pans the imaging device 100 toward the car, and the car is included in the second region 422 of the second frame. As a result, in the tracking region setting process for the second frame (S204), the evaluation value of the second region becomes larger than that of the first region, and the second region 422 is set as the tracking region for the next frame. At this point, the first subject detection process has detected a subject region 601 that overlaps with the tracking region 422, so the process moves from S208 to S212.

[0096] In S212, tracking area 422 is updated based on the result of the first object detection process. Here, an example is shown in which object area 601 detected in the first object detection process is set as the tracking area for the next frame. The template stabilization process ends before the first predetermined time T1 has elapsed, and from the third frame onwards, object tracking process is executed using object area 601 as a template.

[0097] By setting the first predetermined time T1, even if the tracking area is specified at a position shifted from the intended subject, the probability that the intended subject can be tracked can be increased by the user panning the imaging device 100 in the direction of the intended subject.

[0098] FIG. 5(c) shows a case where a first predetermined time T1 similar to that in FIG. 5(b) is set, and no subject area is detected by the first subject detection process once the first predetermined time T1 has elapsed. In the first frame, a subject area is detected in the second subject detection process, but because the first predetermined time T1 has not yet elapsed, the detection result of the second subject detection process is not taken into consideration when setting the tracking area.

[0099] Between the first and second frames, the user pans the imaging device 100 toward the car, and the car is included in the second region 422 of the second frame, but the plants are no longer included. In the tracking region setting process for the second frame (S204), the evaluation value of the second region is greater than that of the first region, and the second region 422 is set as the tracking region for the next frame.

[0100] As a result, the subject area 602 detected as the result of the second subject detection process changes from the plant area to the car area. However, because the first predetermined time T1 has not yet elapsed, the process proceeds from S209 to S202, and the detection result of the second subject detection process is not taken into consideration.

[0101] After that, the first region and the second region remain substantially the same until the first predetermined time T1 has elapsed. In the tracking region setting process shown in Fig. 3, the first region continues to be set as the tracking region during this time, but is substantially the same as the second region 422 in the second frame.

[0102] For the m-th frame after the first predetermined time T1 has elapsed without a subject area being detected in the first subject detection process, the process proceeds from S209 to S210, and the detection result of the second subject detection process is taken into consideration. Then, the process proceeds from S210 to S212, and a tracking area is set based on the subject area 602 that overlaps with the set tracking area (first area 421).

[0103] Here, an example is shown in which the subject area 602 detected in the second subject detection process is set as the tracking area for the next frame (the (m+1)th frame). The template stabilization process ends before the second predetermined time T2 has elapsed, and from the (m+1)th frame onwards, subject tracking process is executed using the subject area 602 as a template.

[0104] By setting the first predetermined time T1, even if the tracking area is specified at a position shifted from the intended subject, the probability that the intended subject can be tracked can be increased by the user panning the imaging device 100 in the direction of the intended subject.

[0105] Therefore, even when the tracking area is set based on the detection result of the second subject detection process, it is possible to reduce the possibility that erroneous detection in the second subject detection process will affect the setting of the tracking area.

[0106] According to this embodiment, the tracking area for the current frame is updated to the area that is determined to be more likely to be the subject based on the evaluation value, out of the area detected by pattern matching and the area corresponding to the tracking area for the previous frame. Therefore, even if the user designates the tracking area in a position that is shifted from the intended subject, the intended subject can be tracked by moving the shooting range in the direction of the intended subject, which is easy to use.

[0107] In addition, by updating the tracking area using the detection results of the subject detection process, it is possible to further improve the accuracy of tracking the subject. Furthermore, by using multiple subject detection processes with different accuracies, it is possible to prioritize the result of the subject detection process with higher accuracy, while still being able to set a tracking area based on the subject detection result even when the result of the subject detection process with higher accuracy cannot be used.

[0108] Furthermore, by using the result of the subject detection process with lower accuracy after setting the tracking area based on the evaluation value for a predetermined period of time, the possibility of using an incorrect subject detection result can be reduced.

[0109] (Other embodiments) In the above-described embodiment, a configuration has been described in which the detection result of the second subject detection process is not used until the first predetermined time T1 has elapsed. However, a configuration in which the detection result of the second subject detection process can be used before the first predetermined time T1 has elapsed may also be used. For example, an acceptance level may be set to determine whether or not to accept the subject detection result, and before the first predetermined time T1 has elapsed, the acceptance level of the detection result of the second subject detection process may be set low and the acceptance level of the detection result of the first subject detection process may be set high. In this case, before the first predetermined time T1 has elapsed, the detection result of either the first subject detection process or the second subject detection process may be used with a probability according to the acceptance level.

[0110] In the above-described embodiment, the template stabilization process is performed at the start of the subject tracking process, and after the stabilization process is completed, the previous subject tracking process is performed. However, the template stabilization process may be performed not only at the start of the subject tracking process, but also during the subject tracking process. For example, if the focus detection area is set at a position different from the tracking area, the template stabilization process may be performed for the tracking area that includes the focus detection area.

[0111] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0112] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0113] 100...imaging device, 101...lens unit, 141...imaging element, 151...CPU, 152...image processing unit, 154...RAM, 155...ROM

Claims

1. a setting means for setting a tracking area in the first image; a generation means for generating a template to be used in template matching based on the set tracking area; a detection unit that detects a first region in the second image that is similar to the template by applying template matching using the template generated by the generation unit to the second image, the image processing device performing subject tracking, an image processing device characterized in that the setting means compares the evaluation value of the first area detected in the second image by the detection means with the evaluation value of a second area of ​​the second image whose position corresponds to the tracking area of ​​the first image, and sets one of the first area and the second area as the tracking area of ​​the second image.

2. the evaluation value of the first region and the evaluation value of the second region are evaluation values ​​that indicate the likelihood of the image being a subject; 2. The image processing apparatus according to claim 1, wherein the setting unit sets one of the first area and the second area having a higher evaluation value as the tracking area of ​​the second image.

3. 2. The image processing device according to claim 1, wherein the evaluation value of the first region and the evaluation value of the second region are one or more of a contrast value, a specific band component, a feature amount, and a motion amount of the region.

4. The image processing device further includes a subject detection means for detecting a subject area in which a predetermined subject is captured in the image, 4. The image processing device according to claim 1, wherein the setting means sets the tracking area of ​​the second image based on the subject when a subject area is detected in the first image.

5. The image processing device according to claim 4, characterized in that the setting means sets the tracking area of ​​the second image based on a subject area detected in the first image whose distance from the tracking area of ​​the second image set as one of the first area and the second area is equal to or less than a predetermined value.

6. 6. The image processing device according to claim 4, wherein the setting of the tracking area of ​​the second image, the generation of the template, and the detection of the first area are repeatedly performed until the subject area is detected by the subject detection means or a predetermined time has elapsed.

7. the subject detection means detects a subject area by a first subject detection process and a second subject detection process having lower detection accuracy than the first subject detection process; the setting means uses the detection result of the first object detection process in preference to the detection result of the second object detection process.

7. The image processing device according to claim 4, wherein the image processing device is a computer.

8. 8. The image processing device according to claim 7, wherein the setting means does not use the processing result of the second subject detection processing until a first predetermined time has elapsed.

9. The image processing device according to claim 8, characterized in that the image processing device is an imaging device, and when the focal length of a lens unit of the imaging device when the tracking area of ​​the second image is initially set is equal to or greater than a threshold, the first predetermined time is made longer than when the focal length is less than the threshold.

10. 9. The image processing device according to claim 8, wherein if the image processing device is moving when the tracking area of ​​the second image is initially set, the first predetermined time is made longer than if the image processing device is not moving.

11. 9. The image processing device according to claim 8, wherein when a moving object area exists near the tracking area of ​​the second image, the first predetermined time is made longer than when no moving object area exists near the tracking area of ​​the second image.

12. 9. The image processing device according to claim 8, wherein the setting means determines that the first predetermined time has elapsed if a proportion of the first area set as the tracking area of ​​the second image is equal to or greater than a threshold value.

13. An image processing method for object tracking executed by an apparatus, comprising: a setting step of setting a tracking area in the first image; a generation step of generating a template to be used for template matching based on the set tracking area; a detection step of detecting a first region in the second image that is similar to the template by applying template matching using the template generated in the generation step to the second image, an image processing method characterized in that, in the setting step, an evaluation value of the first area detected in the second image in the detection step is compared with an evaluation value of a second area of ​​the second image whose position corresponds to that of the tracking area of ​​the first image, and one of the first area and the second area is set as the tracking area of ​​the second image.

14. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 12.

Images