Image processing device, image processing method, and imaging device

Abstract

To provide an image processing device, an image processing method, and an imaging device that achieve the improved display of indicators for feature regions.SOLUTION: An image processing device provides a tracking function for a feature region to be tracked. When overlap between the feature region to be tracked and other feature regions, a change in the position or size of the feature region to be tracked, orientation of the feature region to be tracked, or the movement of a device that has captured a plurality of images satisfies a predetermined condition, the image processing device generates display image data on which indexes indicating feature regions other than the feature region to be tracked are further superimposed. On the other hand, if the condition is not satisfied, the image processing device generates display image data in which the indexes are not superimposed on feature regions other than the feature region to be tracked.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and more particularly to an improvement in a function that utilizes detection of a characteristic region. [Background technology]

[0002] Conventionally, image processing devices such as imaging devices detect characteristic regions such as faces and heads from an image and use them, for example, to set a focus detection area (Patent Document 1). Patent Document 1 discloses an imaging device that automatically selects a main subject from detected candidates and performs autofocus to focus on the selected main subject. It also discloses superimposing a frame-shaped indicator on the main subject region and candidate regions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-51280 A (0049-0050, 0113, Fig. 2, Fig. 10(a)) Summary of the Invention [Problem to be solved by the invention]

[0004] If markers are displayed for all detected feature regions as in Patent Document 1, when many feature regions are detected, the displayed markers may make it difficult to see the subject, which may hinder photography. Furthermore, in Patent Document 1, when there is a possibility that the automatically selected main subject region will change, the user cannot know in advance which candidate region is likely to become the new main subject region.

[0005] In one aspect, the present invention provides an image processing device, an image processing method, and an imaging device that realize the display of an index for a feature region, which can alleviate at least one of the problems of the conventional techniques. [Means for solving the problem]

[0006] The above-mentioned object can be achieved by an image processing device comprising: a tracking means for detecting the position of a characteristic region of a tracking target in a plurality of images acquired in time series; and a generation means for generating display image data on which an index indicating the characteristic region of the tracking target is superimposed, based on data from the plurality of images; wherein the generation means generates display image data on which an index indicating a characteristic region other than the characteristic region of the tracking target is further superimposed, if a predetermined condition is met by the overlap of the characteristic region of the tracking target with other characteristic regions, a change in the position or size of the characteristic region of the tracking target, the orientation of the characteristic region of the tracking target, or the movement of the device that captured the plurality of images; and if the condition is not met, generates display image data on which no index is superimposed on characteristic regions other than the characteristic region of the tracking target. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device, an image processing method, and an imaging device that realize an improved display of an index for a feature region. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to an embodiment; [Figure 2] FIG. 1 is a diagram showing an example of the configuration of an image sensor according to an embodiment; [Figure 3] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 4] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 5] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 6] Flowchart regarding the operation of the digital camera according to the embodiment [Figure 7] FIG. 10 is a diagram showing an example of displaying an index for a feature region in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following embodiments, the present invention will be described with reference to a digital camera as an example of image processing. However, imaging functionality is not essential for the present invention, and the present invention can be implemented in any electronic device capable of detecting feature regions. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] <Camera configuration> FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing device according to this embodiment. Lens unit 10 forms an optical image of a subject on the imaging plane of image sensor 14. Aperture 12 with a shutter function is provided between lens unit 10 and image sensor 14. Lens unit 10 has a focus lens for adjusting the focal length and a zoom lens for adjusting the angle of view. The focus lens is driven by focus control unit 42, and the zoom lens is driven by zoom control unit 44, both under the control of system control unit 50.

[0012] The image sensor 14 may be, for example, a known CCD or CMOS color image sensor with a primary-color Bayer array of color filters. The image sensor 14 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained.

[0013] The pixel array of the image sensor 14 includes dedicated pixels (focus detection pixels) for generating signals for focus detection. FIG. 2 shows a portion of the pixel array of the image sensor 14. The pixel array includes image capture pixels 250 whose photoelectric conversion regions are not shaded and focus detection pixels 251 whose photoelectric conversion regions are partially shaded. R, Gr, Gb, and B indicate the colors of the color filters provided in each pixel. In the example shown in FIG. 2, the focus detection pixel 251 is located at the position of a B pixel in a 2×2 pixel array, which is a repeating unit of color filters in a primary color Bayer array. Note that instead of shading a portion of the photoelectric conversion region, pixels having a configuration in which the photoelectric conversion region is divided may be disposed as focus detection pixels. Furthermore, pixels having a configuration in which the photoelectric conversion region is divided may be provided over a wide area (e.g., the entire surface) and may be configured to double as image capture pixels by arranging color filters in a primary color Bayer array.

[0014] Of the focus detection pixels arranged in the focus detection area, a signal group (image A) obtained from focus detection pixels whose right halves of their photoelectric conversion areas are shielded from light, and a signal group (image B) obtained from focus detection pixels whose left halves of their photoelectric conversion areas are shielded from light, form a pair of focus detection signals. The defocus amount is obtained from the phase difference between the pair of focus detection signals.

[0015] The A / D converter 16 converts the analog image signal output by the image sensor 14 into a digital image signal (image data) and outputs it.

[0016] The timing generation circuit 18 supplies clock signals and control signals to the image sensor 14 and the A / D converter 16. The operation of the timing generation circuit 18 is controlled by the memory control circuit 22 and the system control circuit 50.

[0017] The image processing circuit 20 applies predetermined image processing to image data output by the A / D converter 16 or the memory control unit 22 to generate signals and image data according to the intended use, and acquire and / or generate various types of information. The image processing circuit 20 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing circuit 20 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function. The image processing circuit 20 outputs the acquired or generated information and data to the system control unit 50, the memory control unit 22, or the like according to the intended use.

[0018] The image processing applied by the image processing circuit 20 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the lens unit 10 (image restoration), correction of the effects of peripheral shading of the lens unit 10, color correction, and the like. The detection process may include detecting and tracking characteristic regions (for example, human and / or animal faces, heads, torsos, and specific organs (eyes, pupils, noses, mouths, etc.)), person recognition processing, and the like. The data processing may include processes such as area cutting (trimming), image synthesis, scaling, header information generation (data file generation), etc. The image data may be encoded and decoded by the image processing circuit 20 instead of by the compression / decompression unit 32. The generation of image data for display or image data for recording is also included in the data processing. The evaluation value calculation process may include processes such as generating signals and evaluation values ​​used for autofocus detection (AF), generating evaluation values ​​used for automatic exposure control (AE), etc. The generation of the pair of focus detection pixel signals described above is also performed in the evaluation value calculation process. Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are examples of processes that the image processing circuit 20 can apply, and do not limit the processes that the image processing circuit 20 can apply.

[0019] The image processing circuit 20 outputs the position and size of the characteristic region, the detection reliability, etc. as the detection processing result. The detection of the characteristic region in the image processing circuit 20 can be realized by any known method. For example, the face region can be detected by pattern matching using data representing the contour shape of a face that is stored in advance in the image processing circuit 20 as a template. Alternatively, the image processing circuit 20 may determine the degree of match with the template as the reliability, and only regions where the degree of match is above a threshold may be considered as face regions.

[0020] Furthermore, in order to increase the number of types of faces that can be detected and to improve detection accuracy, multiple types of templates related to the contours of a face may be prepared, and a face region may be detected based on the results of template matching using each of the templates. Alternatively, a face region may be detected based on the results of template matching using a contour template and a template of a type other than the contour. For example, a part of the shape of a face may be used as a template. Furthermore, in order to detect face regions of different sizes, templates of different sizes may be generated and used by enlarging and / or reducing the size.

[0021] Furthermore, when detecting facial organs (eyes, pupils, nose, mouth, etc.), the image processing circuit 20 applies pattern matching using pre-prepared templates relating to the shapes of organs to the detected face area. Alternatively, the image processing circuit 20 may determine the degree of match with the template as reliability, and may regard only areas where the degree of match is equal to or greater than a threshold as organ areas.

[0022] Feature regions may be detected using methods other than template matching. For example, feature regions can be detected using machine learning (deep learning, etc.). For example, the image processing circuit 20 uses a circuit or program that realizes a convolutional neural network (CNN), and a feature region detector can be generated by applying a trained model prepared for each type of feature region to the CNN. The trained model (parameter set) can be prepared in advance in, for example, the non-volatile memory 31. By switching the trained model used for the same image data, multiple types of subject regions can be detected.

[0023] For example, a trained model for detecting the pupils, faces, and body parts of dogs and cats, a trained model for detecting the pupils, faces, and body parts of birds, and a trained model for detecting vehicles such as trains and automobiles are prepared in non-volatile memory 31. By selecting one of the three trained models and applying it to image processing circuit 20, system control circuit 50 can cause image processing circuit 20 to detect feature regions corresponding to the selected trained model.

[0024] By sequentially applying the three trained models to the image processing circuit 20 and performing the detection process three times on the same image data, it is possible to detect all types of subject areas corresponding to the three trained models.

[0025] Here, examples have been described in which faces and organs are detected by pattern matching, and the eyes, faces, body parts, or vehicles of dogs, cats, and birds are detected by using a trained model. However, the types of feature regions to be detected are not limited to these. Furthermore, the combinations of the types of feature regions to be detected and the detection methods are not limited to these.

[0026] Next, we will explain the tracking process performed by the image processing circuit 20. The tracking process is a process of tracking a specific area in multiple images captured in time series (for example, multiple frames of a video, or multiple frames of still images captured continuously). The tracking process can be achieved by repeating, for each frame, pattern matching using the specific area as a template and updating the template using the area detected by pattern matching.

[0027] The image processing circuit 20 calculates a correlation value while changing the position of the template for the target frame, and detects the area with the highest correlation with the template. The correlation value may be, for example, the sum of absolute differences in luminance values ​​of pixels at corresponding positions. Note that instead of luminance values, the correlation value may be calculated using differences in color component values ​​or the degree of histogram agreement. There are no particular limitations on the method for searching for the area most similar to the template, and any other known method may be used.

[0028] The system control unit 50 is, for example, one or more processors (CPUs) capable of executing programs. The system control unit 50 loads programs stored in the nonvolatile memory 31 into the memory 30 and executes them, thereby controlling the operations of the functional blocks that make up the digital camera 100 and realizing the functions of the digital camera 100.

[0029] As part of its operation, the system controller 50 performs automatic exposure control (AE) and automatic focus detection (AF) based on the evaluation value generated by the image processing circuit 20. Specifically, the system controller 50 determines exposure conditions (aperture value, shutter speed, and sensitivity) based on the evaluation value so that the focus detection area is properly exposed. Based on the exposure conditions, the system controller 50 drives the aperture 12 via the exposure controller 40 and controls the operation of the image sensor 14. Furthermore, the system controller 50 calculates a defocus amount based on the phase difference between a pair of focus detection pixel signals generated by the signal processing circuit 20. Based on the defocus amount, the system controller 50 drives the focus lens of the lens unit 10 via the focus controller 42, thereby focusing the lens unit 10 on the focus detection area. Note that the system controller 50 may also perform autofocus based on the contrast evaluation value.

[0030] In this embodiment, the system control unit 50 sets the focus detection area for generating an evaluation value for AF based on the subject area detected by the image processing circuit 20. For example, by setting the subject area or a partial area of ​​the focus detection area as the focus detection area, autofocus can be achieved so that the lens unit 10 focuses on the subject area. If the image processing circuit 20 detects multiple subject areas, the system control unit 50 selects one of them as the main subject area and uses it to set the focus detection area. There are no particular limitations on the method for selecting the main subject area from multiple subject areas. The user may specify the main subject area, or the system control unit 50 may select it based on one or more of the position, size, and reliability of each subject area.

[0031] The nonvolatile memory 31 is electrically rewritable and stores programs executed by the system control unit 50, various setting values ​​for the digital camera 100, GUI data, etc. The trained models and feature data used by the image processing circuit 20 can also be stored in the nonvolatile memory 31. The memory 30 is used when the system control unit 50 executes programs and to temporarily store image data. In addition, a portion of the memory 30 is used as video memory for the image display unit 28.

[0032] The memory control circuit 22 controls access to the memory 30 by the A / D converter 16 and the image processing circuit 20. The memory control circuit 22 also controls the operation of the timing generation circuit .

[0033] The image display unit 28 is a display device that displays images based on display image data written to the video memory area of ​​the memory 30. By immediately displaying captured video on the image display unit 28, the image display unit 28 functions as an electronic viewfinder (EVF). The processing for causing the image display unit 28 to function as an EVF is called live view processing, and the display image data used in live view processing is called live view image data. In this embodiment, the image processing unit 28 is a touch display.

[0034] The compression / decompression unit 32 encodes the image data for recording generated by the image processing circuit 20, and decodes encoded image data read from the recording unit 202. There are no particular restrictions on the encoding method, but it is common to use an encoding method conforming to the JPEG standard for still images and the MPEG standard for moving images.

[0035] The exposure control unit 40 drives the diaphragm 12 under the control of the system control unit 50 . The flash 48 is an auxiliary light source. Whether the flash 48 is turned on or off is determined by the system control unit 50 based on the settings of the digital camera 100 and an evaluation value generated by the image processing circuit 20. The operation of the flash 48 is controlled by the system control unit 50.

[0036] The mode dial 60 sets one of the function modes of the digital camera 100, such as power off, automatic shooting mode, shooting mode, panoramic shooting mode, video shooting mode, playback mode, and PC connection mode.

[0037] The shutter switch 62 is a switch for capturing still images, and has SW1, which turns ON when pressed halfway, and SW2, which turns ON when pressed all the way. The system control unit 50 recognizes SW1 being ON as a capture preparation instruction and SW2 being ON as a capture instruction. The system control unit 50 executes capture preparation instructions such as AF and AE in response to the capture preparation instruction. The system control unit 50 also executes a series of operations from still image capture to recording in response to the capture instruction. The recording image data generated by the image processing circuit 20 is encoded by the compression / decompression unit 32 as necessary, and then recorded in the form of an image data file on the recording medium 200 via the I / F 90.

[0038] The display changeover switch 66 switches ON and OFF the image display unit 28. By turning off the image display unit 28 when it is not in use, such as when using the optical viewfinder 104, it is possible to reduce power consumption.

[0039] When the zoom switch 72 is operated, the system control unit 50 drives the zoom lens via the zoom control unit 44 to change the angle of view of the lens unit 10. Whether the angle of view is to be widened or narrowed is determined according to the operation of the zoom switch 72.

[0040] Operation unit 70 is a general term for input devices provided on digital camera 100 that have not been described above. These include a menu button, a set button, direction keys, a video recording button, etc. If image display unit 28 is a touch display, the touch panel of image display unit 28 is included in operation unit 70.

[0041] A power supply 86 supplies power to the digital camera 100. The power supply 86 may be a battery installed in the digital camera 100, or an external power supply such as an AC adapter.

[0042] Interface 90 is a communication interface with recording medium 200. Recording medium 200 is connected to connector 92. Recording medium 200 is, for example, a memory card. Recording medium 200 has a recording unit 202 that stores data, and an interface 204 for communicating with an external device (here, digital camera 100) to which connector 206 of recording medium 200 is connected.

[0043] The communication unit 110 communicates with external devices via a connector or an antenna 112. The communication unit 110 complies with one or more wired and / or wireless communication standards, and has hardware corresponding to the standards it complies with.

[0044] The gyro sensor 115 outputs a signal corresponding to the movement (angular velocity) of the digital camera 100 to the system control unit 50. The system control unit 50 integrates the signal output by the gyro sensor and detects the movement of the digital camera 100 as a combination of angular changes around the axes. The system control unit 50 can use the detected movement of the digital camera 100 for, for example, image stabilization or detecting a panning operation.

[0045] <Still image shooting processing> The still image capturing process for the digital camera 100 to capture and record a still image will be described using the flowchart shown in FIG. In S301, the system control unit 50 determines whether the operation mode of the digital camera 100 is set to still image shooting mode, and if it is determined that it is set to still image shooting mode, executes S302. If it is not determined that it is set to still image shooting mode, operations related to shooting and recording still images are not performed, and the system control unit 50 ends the still image shooting process.

[0046] In S302, the system control unit 50 determines whether preparation for live view processing is complete, and if it is determined that preparation is complete, executes S303, and if not, executes S302 repeatedly. The system control unit 50 can determine that preparation for live view processing is complete when the timing generation circuit 18 is able to supply clock signals and control signals for live view display to the image sensor 14 and the A / D converter 16.

[0047] In S303, the system control unit 50 starts live view processing, and then executes S304. Live view processing is a series of processes for causing the image display unit 28 to function as an EVF. Specifically, the system control unit 50 continuously performs the following processes: capturing a moving image with the image sensor 14, generating display image data with the image processing circuit 20, storing the data in the memory 30, and displaying the data on the image display unit 28. After starting live view processing in S303, the system control unit 50 continuously performs live view processing while executing the processes from S304 onwards until the still image shooting mode ends.

[0048] The image processing circuit 20 also performs subject detection processing on the image data generated by the live view processing, and outputs the results to the system control unit 50. The subject detection processing may be performed on all frames of the captured video, or may be performed every predetermined number of frames. The frequency with which the subject detection processing is performed may be dynamically determined by the system control unit 50, taking into account the video capture or display frame rate, the processing capacity of the image processing circuit 20, and the like.

[0049] In S304, the system control unit 50 executes a main subject determination process, determines a main subject region from the subject regions detected by the image processing circuit 20, and executes S305. The main subject determination process will be described in detail later.

[0050] In S305, the system control unit 50 determines whether or not the main subject region has been determined in the main subject determination process, and if it is determined that the main subject region has been determined, executes S306, and if not, executes S310.

[0051] In S306, the system control unit 50 instructs the image processing circuit 20 to superimpose an index indicating the main subject region along with information about the main subject region. The image processing circuit 20 generates image data for live view display on which an image of the index indicating the main subject region is superimposed, and stores the image data in the memory 30. The index may be, for example, at least a part of the outline of a rectangle circumscribing the main subject region. Here, the entire outline, i.e., a frame-shaped index, is superimposed and displayed. However, an index of any size and shape that indicates the main subject region may be used.

[0052] 7(a) is a diagram showing an example of the display of an index relative to a main subject region. In this example, the main subject is the person on the right, and a rectangular frame-shaped index 701 encompassing the face region is displayed. On the other hand, an index 702 that is visually different from index 701 is superimposed on the region of a subject (sub-subject) other than the main subject detected in the same image. For convenience, the index displayed relative to the sub-subject region will be referred to as a sub-frame below.

[0053] 3, in S307, the system control unit 50 executes sub-frame display processing, which is index display processing for the sub-subject region, and then executes S308. Details of the sub-frame display processing will be described later.

[0054] In S308, the system control unit 50 determines whether or not a sub-frame was displayed in the sub-frame display process of S307, and if it is determined that a sub-frame was displayed, it executes S309, and if not, it executes S310.

[0055] In S309, the system control unit 50 executes a main subject switching process, and then executes S310. The main subject switching process is a process for switching the main subject in response to a user instruction. The main subject switching process will be described in detail later.

[0056] In S310, the system control unit 50 determines whether to end the shooting mode. The system control unit 50 can determine to end the shooting mode, for example, when the operation mode of the digital camera 100 is switched from still image shooting mode to another mode (e.g., playback mode) or when an instruction to turn off the power is received. If it is determined that the shooting mode should be ended, the system control unit 50 ends the still image shooting operation and executes processing according to the state. On the other hand, if it is not determined that the shooting mode should be ended, the system control unit 50 executes S311.

[0057] In S311, the system control unit 50 determines whether SW1 of the shutter switch 62 is ON or not, and if it is determined to be ON, executes S312, and if not, executes S317.

[0058] In S312, the system control unit 50 determines whether the main subject region has been determined, similarly to S305, and if it is determined that the main subject region has been determined, executes S313, and if not, executes S314.

[0059] In S313, the system control unit 50 sets a focus detection area based on the main subject area and performs AF processing. Specifically, the system control unit 50 acquires a focus detection signal for the focus detection area from the image processing circuit 20 and calculates a defocus amount. The system control unit 50 then drives the focus lens based on the defocus amount via the focus control unit 42 so that the lens unit 10 focuses on the focus detection area. The system control unit 50 also acquires an AE evaluation value for the focus detection area from the image processing circuit 20 and performs AE processing to determine the exposure conditions. The system control unit 50 then executes S315.

[0060] In S314, the system control unit 50 acquires focus detection signals from the image processing circuit 20 for a plurality of predetermined regions and calculates the defocus amount. Then, the system control unit 50 drives the focus lens via the focus control unit 42 so that the lens unit 10 focuses on the region with the shortest subject distance based on the defocus amount. The system control unit 50 then executes S315. The region with the shortest subject distance within the shooting range may be determined based on a defocus map that can be generated using focus detection pixels of the image sensor 14, for example.

[0061] In S315, the system control unit 50 determines whether SW2 of the shutter switch 62 is ON or not, and if it is determined to be ON, executes S316, and if not, executes S317.

[0062] In S316, the system control unit 50 executes still image shooting and recording processing. The system control unit 50 controls the aperture 12 and the image sensor 14 based on the exposure conditions most recently determined in the shooting preparation instruction executed while SW1 is ON, and executes still image shooting. The image processing circuit 20 then generates still image data for recording and stores it in the memory 30. The system control unit 50 encodes the still image data using the compression / decompression unit 32 as necessary, and then records the image data file on the recording medium 200 via the I / F 90. Note that live view processing may be suspended during still image shooting processing. When shooting processing ends, the system control unit 50 executes S317.

[0063] In S317, the system control unit 50 determines whether the main subject region has been determined, similarly to S305, and if it is determined that the main subject region has been determined, executes S318, and if not, executes S304.

[0064] In S318, the system control unit 50 causes the image processing circuit 20 to perform tracking processing of the main subject region. The image processing circuit 20 searches for a region most similar to the main subject region determined in the most recent main subject determination processing in a frame (e.g., the next frame) captured after the frame in which the most recent main subject determination processing was performed. The image processing circuit 20 then notifies the system control unit 50 of the position, size, search reliability, and other information of the searched region as a result of the tracking processing. In tracking processing for a frame in which subject detection processing was performed, the search range may be limited to the detected subject region. Note that if the image processing circuit 20 does not find a region whose search reliability satisfies a predetermined threshold, it notifies the system control unit 50 that the search processing has failed.

[0065] When the tracking process is started, the image processing circuit 20 continues the tracking process with the area detected in the most recently executed tracking process as the main subject area until the next execution of the main subject determination process, thereby tracking the area of ​​the same subject.

[0066] In S319, the system control unit 50 determines whether the tracking process was successful based on the result of the tracking process notified by the image processing circuit 20. For example, the system control unit 50 determines that the tracking process was successful if the reliability of the search result satisfies a predetermined threshold. If the system control unit 50 determines that the tracking process was successful, it executes S306, and if not, it executes S304. In this way, if the image processing circuit 20 fails the tracking process, it executes a main subject determination process to update the main subject area that the image processing circuit 20 tracks. Note that the main subject determination process may be executed not when the tracking process fails once, but when the tracking process fails a predetermined number of times in succession.

[0067] <Main subject determination process> Next, the main subject determination process in S304 will be described in detail with reference to the flowchart in FIG. In S401, the system control unit 50 determines whether or not a human subject area has been detected based on the results of the subject detection process by the image processing circuit 20. Here, as an example, it is assumed that the subjects of the subject detection process are human subjects, animal subjects, and vehicle subjects. If it is determined that a human subject area has been detected, the system control unit 50 executes S402, and if it is not determined that a human subject area has been detected, the system control unit 50 executes S405.

[0068] In S402, the system control unit 50 determines whether or not multiple human subject regions have been detected, and if it is determined that multiple human subject regions have been detected, it executes S403, and if not, it executes S404.

[0069] In S403, the system control unit 50 determines a main subject region from the multiple detected regions. As an example, the main subject is determined based on the position and size of a face region. The system control unit 50 can, for example, convert the position and size into weights (coefficients) and determine the main subject based on the value obtained by multiplying them. For example, the system control unit 50 can preset the position weight to be larger the shorter the distance (e.g., center-to-center distance) between the focus detection region and the subject region, and the size weight to be larger the larger the size of the subject region. In this case, the system control unit 50 determines the subject region with the largest product of the position and size weights as the main subject region, and ends the main subject determination process.

[0070] Note that the main subject region may be determined by other methods, such as using a weight that increases as the face orientation approaches the front, or using a weight that increases as the reliability of detection increases.

[0071] S404 is executed when there is one detected human subject area, so the system control unit 50 determines the detected human subject area as the main subject area and ends the main subject determination process.

[0072] In S405, the system control unit 50 determines whether or not an area of ​​an animal subject has been detected, and if it is determined that it has been detected, executes S406, and if it is not determined that it has been detected, executes S411. In S406, the system control unit 50 determines whether or not a region of a vehicle object has been detected, and if it is determined that it has been detected, executes S407, and if it is not determined that it has been detected, executes S408.

[0073] In S407, the system control unit 50 determines the main subject based on the position and size of each subject region, and the main subject determination process ends. As with the case of human subjects, the main subject can be determined by converting the position and size into weights. However, because both animal subjects and vehicle subjects are detected, the size is normalized before being converted into a weight. Specifically, the size is normalized so that the sizes of the animal subject and vehicle subject regions that are the same distance from the digital camera 100 in the optical axis direction are equal. For example, size can be normalized by applying a coefficient statistically calculated in advance using multiple samples to at least one of the animal subject regions and the vehicle subject regions. Note that the position and size weights may be different from those used for the human subject regions.

[0074] In S408, the system control unit 50 determines whether or not multiple animal subject regions have been detected, and if it is determined that they have been detected, it executes S409, and if not, it executes S410. In S409, the system control unit 50 determines the main subject based on the position and size of each subject area, and ends the main subject determination process. As with the case of human subjects, the main subject can be determined by converting the position and size into weights. S410 is executed when there is one detected animal subject area, so the system control unit 50 determines the detected animal subject area as the main subject area and ends the main subject determination process.

[0075] In S411, the system control unit 50 determines whether or not a region of a vehicle object has been detected, and if it is determined that it has been detected, executes S412, and if it is not determined that it has been detected, executes S415. In S412, the system control unit 50 determines whether or not a plurality of vehicle subject regions have been detected, and if it is determined that they have been detected, it executes S413, and if not, it executes S414.

[0076] In S413, the system control unit 50 determines the main subject based on the position and size of each subject area, and ends the main subject determination process. As with the case of human subjects, the main subject can be determined by converting the position and size into weights. S414 is executed when there is one detected vehicle subject area, so the system control unit 50 determines the detected vehicle subject area as the main subject area and ends the main subject determination process.

[0077] In S415, the system control unit 50 determines that there is no main subject area, and ends the main subject determination process.

[0078] <Sub-frame display determination> Next, details of the sub-frame display process in S307 will be described using the flowchart in Fig. 5. In the sub-frame display process, it is determined whether or not to display an index (sub-frame) for the sub-main subject region, and if it is determined that an index is to be displayed, the index is displayed in the sub-subject region to be displayed.

[0079] In S501, the system control unit 50 determines whether the main subject region overlaps with the sub-subject region. For example, the system control unit 50 can determine that the main subject region overlaps with the sub-subject region if the proportion of the main subject region that overlaps with one or more other sub-subject regions exceeds a threshold. Alternatively, the system control unit 50 can determine that the main subject region overlaps with the sub-subject region if the sub-subject region overlaps with a specific partial region of the main subject region (e.g., a face region in a person subject region) within a certain distance from the center of the main subject region. These are merely examples, and the determination may be made based on other conditions. If the system control unit 50 determines that the main subject region overlaps with the sub-subject region, it executes S509; if not, it executes S502.

[0080] In S509, the system control unit 50 determines that the main subject region is likely to be lost, and after setting a flag, executes S510. The state in which the main subject region is likely to be lost can be said to be a state in which it is highly likely that tracking processing for the current main subject region will no longer be possible in the near future.

[0081] In S502, the system control unit 50 determines whether the main subject region is moving, and if it is determined that the main subject region is moving, executes S503, and if it is not, executes S508. The system control unit 50 can determine whether the main subject region is moving based on, for example, a change over time in the position of the main subject region obtained as a result of the tracking process, but may use other methods to make the determination.

[0082] In S503, the system control unit 50 determines whether the main subject region is moving in a direction away from the image capture range. For example, the system control unit 50 can determine that the main subject region is moving in a direction away from the image capture range if the shortest distance between the center of the main subject region and the four sides of the image decreases over time. Alternatively, the system control unit 50 may estimate the time it will take for a portion of the main subject region to move outside the image capture range based on the amount of movement of the main subject region, and determine that the main subject region is moving in a direction away from the image capture range if the estimated time is within a predetermined time. Of course, other methods may also be used for this determination. If the system control unit 50 determines that the main subject region is moving in a direction away from the image capture range, it executes S505; if not, it executes S504.

[0083] In S504, system control unit 50 determines whether digital camera 100 is being panned in the direction of movement of the main subject region, and if it is determined that a panning operation is being performed, executes S505, and if not, executes S509. If the angular difference between the direction of movement of digital camera 100 based on the signal output by gyro sensor 115 and the direction of movement of the main subject region is less than a threshold, system control unit 50 determines that digital camera 100 is being panned in the direction of movement of the main subject region. Note that the movement of digital camera 100 may be determined using other methods, such as determining it based on a motion vector between frames.

[0084] If the panning direction is different from the direction of movement of the main subject area, or if panning is not being performed, it is considered that there is a relatively high possibility that the main subject will move out of the shooting range in the near future and that tracking processing for the current main subject area will fail. Therefore, in S509, system control unit 50 determines that the main subject area is likely to be lost, sets a flag, for example, and then executes S510.

[0085] In S505, the system control unit 50 determines whether there is an object (obstruction) in the direction of movement of the main subject that may obscure the main subject. For example, the system control unit 50 estimates the change over time in the position of the main subject within the screen and the distance from the digital camera 100 from the direction of movement and speed of movement of the main subject area. The system control unit 50 then determines whether there is an object in front of the movement path of the main subject based on a defocus map that can be generated using focus detection pixels of the image sensor 14, for example. If such an object is present, the system control unit 50 can determine that there is an obstruction in the direction of movement of the main subject. If the system control unit 50 determines that there is an obstruction in the direction of movement of the main subject, it executes S509; otherwise, it executes S506.

[0086] If it is determined that there is an obstruction in the direction of the subject's movement, it is considered that there is a relatively high possibility that tracking processing for the current main subject area will fail in the near future. Therefore, the system control unit 50 determines in S509 that the main subject area is likely to be lost, sets a flag, for example, and then executes S510.

[0087] In S506, system control unit 50 determines whether the size of the main subject area has decreased over time, and if it is determined that it has decreased, executes S509, and if not, executes S507. This determination may be made based on whether the distance between the main subject and digital camera 100 has increased over time.

[0088] If the size of the main subject region is decreasing over time, or if the main subject is moving away from digital camera 100, it is considered that there is a relatively high possibility that tracking processing for the current main subject region will fail in the near future. Therefore, system control unit 50 determines in S509 that the state is such that the main subject region is likely to be lost, and after setting a flag, for example, executes S510.

[0089] In S507, the system control unit 50 determines whether the main subject is facing backward. If it is determined that the main subject is facing backward, it executes S509; if not, it executes S508. Here, it is assumed that the main subject is a type of subject, such as a person or an animal, for which the accuracy or reliability of the tracking process significantly decreases when the main subject is facing backward. For subjects whose tracking accuracy or reliability does not change significantly depending on their orientation, such as a vehicle subject, S507 may be skipped or the subject may be considered not facing backward. Whether the main subject is facing backward can be determined using a deep learning model that can determine the orientation of the subject, such as the orientation of a person or animal's face. The image processing circuit 20 can determine the orientation of the subject based on the trained model and notify the system control unit 50 of the determination result. Note that instead of determining whether the main subject is facing backward, it may also be determined whether the orientation decreases the tracking accuracy or reliability.

[0090] If the main subject is facing backward, it is considered that there is a relatively high possibility that tracking processing for the current main subject area will fail in the near future. Therefore, in S509, the system control unit 50 determines that the main subject area is likely to be lost, sets a flag, for example, and then executes S510.

[0091] In S508, the system control unit 50 determines that the state is not such that the main subject area is likely to be lost, and, for example, if a flag is set, it is cleared, and the process goes to S510.

[0092] In S510, the system control unit 50 determines whether or not the state is such that the main subject region is likely to be lost, for example, based on the state of a flag, and if it is determined that the state is such that the main subject region is likely to be lost, it executes S511, and if it is not determined that the state is such that the main subject region is likely to be lost, it executes S514. Here, if the flag is set (for example, 1), the system control unit 50 determines that the state is such that the main subject region is likely to be lost.

[0093] In S511, the system control unit 50 determines whether or not a sub-subject region has been detected by the image processing circuit 20, and if it is determined that a sub-subject region has been detected, the system control unit 50 executes S512, and if not, the system control unit 50 executes S514.

[0094] In S512, the system control unit 50 determines the sub-subject region with the highest priority. For example, the system control unit 50 can determine the sub-subject region with the highest priority by executing the main subject determination process described with reference to FIG. 4 on the sub-subject region.

[0095] In S513, the system control unit 50 instructs the image processing circuit 20 to display a sub-frame together with information about the sub-subject region determined in S512. The image processing circuit 20 generates image data for live view display in which the sub-frame is superimposed on the sub-subject region whose information was acquired from the system control unit 50, and ends the sub-frame display process. FIG. 7(a) shows an example of the display of a sub-frame 702. The indicator 701 displayed in the main subject region and the sub-frame 702 have different colors, patterns (for example, solid lines or dotted lines), etc. so that they can be visually distinguished.

[0096] On the other hand, in S514, the system control unit 50 determines not to display a sub-frame, and ends the sub-frame display process.

[0097] In this way, in this embodiment, even if a sub-subject region is detected, an index is displayed only in the main subject region as long as the main subject region is being stably tracked, allowing the user to concentrate on photographing the main subject.

[0098] Furthermore, if it is determined that tracking of the main subject region is likely to fail in the near future, an indicator is added to one sub-subject region that is likely to become the new main subject region in place of the current main subject region. Therefore, by displaying an indicator on the sub-subject region during shooting, the user can know in advance that tracking of the current main subject region may soon end. Furthermore, the user can also know in advance which of the sub-subject regions is likely to become the next main subject region. For example, if the sub-subject region that is likely to become the next main subject region is not the one intended, the user can consider whether to select a different sub-subject region as the new main subject region.

[0099] <Subject switching processing> Next, the main subject switching process in S309 will be described in detail with reference to the flowchart in Fig. 6. The main subject switching process is a process for changing the main subject area in response to a user instruction.

[0100] In S601, the system control unit 50 determines whether a touchdown operation is being performed on the touch panel of the image display unit 28. A touchdown operation is a state in which contact is sensed on the touch panel. If the system control unit 50 determines that a touchdown operation is being performed, it executes S602; otherwise, it executes S605. Note that the system control unit 50 may also determine that a touchdown operation is being performed when the touchdown operation has continued for a certain period of time.

[0101] In S602, the system control unit 50 instructs the image processing circuit 20 to generate image data for live view display in which sub-frames are displayed in all selectable sub-subject regions. FIG. 7(b) shows an example in which sub-frames 802 to 804 are displayed in all sub-subject regions. Note that, because the index 801 is displayed in the main subject region, in S602, an index is displayed for all subject regions detected by the image processing circuit 20. Note that it is not necessary to display an index for detected sub-subject regions whose reliability is equal to or less than a threshold value so that they are not selected.

[0102] In S603, the system control unit 50 determines whether a touch-up has been detected. Touch-up is a state in which a touch (touch-down) that had been sensed on the touch panel is no longer sensed. If it is determined that a touch-up has been detected, the system control unit 50 executes S604; if it is not determined that a touch-up has been detected, the system control unit 50 repeatedly executes S603.

[0103] In S604, the system control unit 50 detects the position where the touch-up was detected (the position where the touch-up was detected immediately before the touch-up was detected). Then, the system control unit 50 sets the subject area including the detected position as the main subject area designated by the user, and then executes S615.

[0104] 7(b), if a touch-up is detected inside indices 801 to 804, the system control unit 50 sets the subject area corresponding to the indices including the position where the touch-up was detected as the main subject area. Also, if a touch-up is detected in the torso or limb area of ​​a human or animal subject, the system control unit 50 may set the corresponding face area as the main subject area.

[0105] If no touch-up is detected within the index, the system control unit 50 may set a subject area in the image whose distance from the position where the touch-up was detected is equal to or less than a threshold as the user-specified main subject area. Alternatively, the system control unit 50 may set a subject area determined based on the subject distance to the position where the touch-up was detected or the color of the pixel at the position where the touch-up was detected as the main subject area.

[0106] In this way, when a touch-down operation is detected, indicators are displayed in all selectable subject areas. This allows the user to grasp the detected subject areas and the subject areas that can be selected as the main subject area. Furthermore, when a new main subject area is selected, the display of indicators for areas other than the main subject area is terminated, so the display of indicators does not interfere with subsequent shooting.

[0107] In S605, the system control unit 50 determines whether an operation of one of the multiple operation members of the operation unit 70, each corresponding to one of the multiple directions, has been detected, and if it is determined that an operation has been detected, it executes S606, and if it is not determined that an operation has been detected, it executes S610. Here, it is assumed that the multiple directions are left and right, and the multiple operation members are the left direction key or the right direction key.

[0108] In S606, the system control unit 50 determines whether or not a sub-subject area exists in the direction corresponding to the directional key whose operation was detected in S605, based on the current main subject area. If it is determined that a sub-subject area exists in the direction corresponding to the operated directional key, the system control unit 50 executes S607, and if not, executes S615.

[0109] In S607, the system control unit 50 sets a sub-subject area that exists in the direction corresponding to the operated direction key as a new main subject area, and executes S608. Note that if there are multiple sub-subject areas in the direction corresponding to the operated direction key, the system control unit 50 sets the sub-subject area that is closest to the current main subject area as the new main subject area.

[0110] In S608, the system control unit 50 instructs the image processing circuit 20 to display an index in the newly set main subject region and to display a sub-frame in the sub-subject region that is closest in the left-right direction to the newly set main subject region. In response to the instruction, the image processing circuit 20 changes the index to be superimposed on the image data for live view display.

[0111] 7(c) shows an example of the live view display in S608. An index 901 is displayed in the new main subject area, and sub-frames 902 and 903 are displayed in the sub-subject areas closest to it in the left and right directions. Indicators 912 and 931 are also displayed to the left and right of index 901, indicating that the main subject area can be switched by operating the left and right arrow keys.

[0112] In S609, the system control unit 50 determines whether the switching operation of the main subject area has ended. If it is determined that the switching operation of the main subject area has ended, the system control unit 50 executes S615. If it is not determined that the switching operation of the main subject area has ended, the system control unit 50 can determine that the switching operation of the main subject area has ended, for example, when the right direction key and the left direction key have not been operated for a certain period of time. Alternatively, the system control unit 50 can determine that the switching operation of the main subject area has ended, for example, when the system control unit 50 detects the operation of a specific input device other than the right direction key and the left direction key.

[0113] When the main subject region is changed by operating the directional keys, a sub-frame is displayed in each direction in the subject region that will be set as the main subject region the next time the directional keys are operated. This allows the user to know, for each directional key, which subject region will be set as the main subject region the next time the directional keys are operated. Furthermore, when it is determined that selection of a new main subject region has been completed, the display of indices for regions other than the main subject region is terminated, so that the display of indices does not interfere with subsequent shooting.

[0114] In S610, the system control unit 50 determines whether or not the subject switching button included in the operation unit 70 has been operated, and if it is determined that it has been operated, executes S611, and if not, executes S615. The subject switching button is an operating member that switches the feature area of ​​the tracking target each time it is operated.

[0115] In S611, the system control unit 50 determines whether or not a sub-subject region has been detected by the image processing circuit 20, and if it is determined that a sub-subject region has been detected, the system control unit 50 executes S612, and if it is not determined that a sub-subject region has been detected, the system control unit 50 executes S615.

[0116] In S612, the system control unit 50 sets the sub-subject region with the highest priority as the new main subject region. In S613, the system control unit 50 instructs the image processing circuit to display indices on the new main subject region and the sub-subject region with the highest priority among the sub-subject regions after the change of the main subject region, and then executes S610. The image processing circuit 20 generates image data for live view display in which indices are superimposed on the new main subject region and the sub-subject region with the highest priority.

[0117] Each time the subject switching button is operated, the sub-subject region with the highest priority at that time is set as the new main subject region. Furthermore, when the main subject region is changed, a sub-frame is displayed around the sub-subject region with the highest priority at that time, which will be selected the next time the subject switching button is operated. This allows the user to know which subject region will be set as the main subject region the next time the subject switching button is operated. Furthermore, once it is determined that selection of a new main subject region has been completed, the display of indices for regions other than the main subject region is terminated, so that the display of indices does not interfere with subsequent shooting.

[0118] In this way, when an operation to change the main subject region is detected, the method of displaying indices for the subject region is changed depending on the type of operation, thereby making it possible to appropriately support the user in selecting a new main subject region. Furthermore, when it is determined that the change of the main subject region is complete, the display of indices for regions other than the main subject region is terminated, so that the display of indices does not interfere with subsequent shooting.

[0119] As described above, according to this embodiment, in an image processing device having a subject tracking function, when the tracking process is stable, an index is displayed only for the main subject region of the tracking target. Therefore, even when a large number of subject regions are detected, there is no risk that the display of the index will reduce the visibility of the subject. On the other hand, when conditions are met that consider there to be a relatively high possibility that the tracking process for the current main subject region will fail in the near future, the display of an index for a sub-subject region that is likely to become the next main subject region is added. When the display of the index for the sub-subject region begins, the user can understand that there is a high possibility that the tracking process for the current main subject region will not be able to be completed in the near future, and the subject region that is likely to become the next main subject region.

[0120] Furthermore, when the main subject region is switched by a user instruction, the method of displaying the marker in the sub-subject region varies depending on the type of operation for switching. Specifically, in the case of an operation that allows one subject region to be selected from all subject regions with a single operation (e.g., a touch operation), sub-frames are displayed in all subject regions. This allows the user to easily grasp the selectable subject regions. In addition, in the case of an operation that selects one of the sub-subject regions adjacent to the current main subject region in response to an operation, the marker is displayed only in the sub-subject region that can be selected by the operation. Furthermore, in the case of an operation that causes the image processing device to automatically select a new main subject region, the marker is displayed only in the sub-subject region that will be selected next by the operation. This allows the user to appropriately grasp the sub-subject region that can be selected by the operation. Furthermore, when it is determined that the selection of a new main subject region has been completed, the display of the marker in the sub-subject region is terminated, thereby preventing the visibility of the subject from being reduced by the display of the marker in the sub-subject region.

[0121] (Other embodiments) For example, when a sub-frame is displayed in S308 in FIG. 3, the main subject processing is executed in S309, but the main subject processing in S309 may be executed regardless of the determination in S308. Although the display of indicators for live view display has been described, the same can be applied to the display of indicators for recorded video data or video data input in real time from an external device.

[0122] In the above-described embodiment, when multiple types of subject regions are detected, the sub-subject region with the highest priority can be determined from subject regions of the same type as the current main subject region. Alternatively, the sub-subject region with the highest priority may be determined from all subject regions other than the current main subject region, without taking the type of subject into consideration. Furthermore, the user may be able to select whether or not to take the type of the current main subject region into consideration when determining the sub-subject region with the highest priority.

[0123] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0124] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) a tracking means for detecting the position of a characteristic region of a tracking target in a plurality of images acquired in time series; a generating means for generating display image data on which an index indicating a characteristic region of the tracking target is superimposed based on data of the plurality of images, The generating means an image processing device that generates display image data on which an index indicating a feature area other than the characteristic area of ​​the tracking target is further superimposed when a predetermined condition is met by an overlap of the characteristic area of ​​the tracking target with another characteristic area, a change in the position or size of the characteristic area of ​​the tracking target, an orientation of the characteristic area of ​​the tracking target, or a movement of a device that captured the plurality of images; and that generates display image data on which no index is superimposed on a feature area other than the characteristic area of ​​the tracking target when the condition is not met. (Item 2) 2. The image processing device according to item 1, wherein the predetermined condition is a condition that is considered to be highly likely to make it impossible to detect the characteristic region of the tracking target. (Item 3) 3. The image processing device according to item 1 or 2, wherein the feature area in which the indicator is displayed when the predetermined condition is satisfied is the feature area that is most likely to become the feature area of ​​a new tracking target when the feature area of ​​the tracking target can no longer be detected. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the feature area for which the indicator is displayed when the predetermined condition is satisfied is, among feature areas of the same type as the feature area of ​​the tracking target, the feature area that is most likely to become the feature area of ​​a new tracking target when the feature area of ​​the tracking target can no longer be detected. (Item 5) a detection means for detecting a feature region from the image; a selection means for selecting a feature region of the tracking target from the feature regions detected by the detection means; 5. The image processing device according to any one of items 1 to 4, further comprising: (Item 6) 6. The image processing device according to item 5, wherein the selection means selects the feature region of the tracking target from the feature regions detected by the detection means in response to an operation on an operation means provided in the image processing device. (Item 7) Item 6. The image processing device according to item 6, characterized in that when an operation on a touch display serving as the operating means is detected, the generating means generates display image data in which indicators are superimposed for all selectable feature areas among the feature areas detected by the detecting means. (Item 8) When an operation on the touch display serving as the operation means is no longer detected, the selection means selects the feature region of the tracking target according to a position where the operation is no longer detected; 8. The image processing device according to item 7, wherein the generating means generates display image data in which no index is superimposed on any of the feature regions detected by the detecting means other than the feature region selected by the selecting means. (Item 9) Item 6. The image processing device according to item 6, characterized in that when an operation of one of a plurality of operating members, each corresponding to one of a plurality of directions, is detected, the selection means selects, from the feature areas detected by the detection means, the feature area that is closest to the feature area of ​​the tracking target in a direction corresponding to one of the plurality of operating members as the feature area of ​​the tracking target. (Item 10) 10. The image processing device according to item 9, wherein the generating means generates display image data on which an index is superimposed for a feature region that is closest in the plurality of directions to the feature region of the tracking target selected by the selecting means. (Item 11) Item 7. The image processing device according to item 6, characterized in that when an operation on an operating member that switches the feature area of ​​the tracking target each time it is operated is detected, the selection means selects, from the feature areas detected by the detection means, the feature area with the highest priority other than the feature area of ​​the current tracking target as the feature area of ​​the tracking target. (Item 12) Item 12. The image processing device according to item 11, wherein the generating means generates display image data on which an index is superimposed for a feature region that will be selected the next time the operating member is operated. (Item 13) 13. The image processing device according to any one of items 9 to 12, wherein, when it is determined that selection of the feature area of ​​the tracking target in response to the operation of the operating member has been completed, the generation means generates display image data in which no index is superimposed on any of the feature areas detected by the detection means other than the feature area selected by the selection means. (Item 14) an imaging means for capturing a plurality of images in time series; 14. The image processing device according to any one of items 1 to 13, which uses the plurality of images captured by the imaging means; An imaging device comprising: (Item 15) An image processing method performed by an apparatus, comprising: a tracking step of detecting the position of a characteristic region of a tracking target in a plurality of images acquired in time series; a generating step of generating display image data on which an index indicating a characteristic region of the tracking target is superimposed based on data of the plurality of images, The generating step includes: generating display image data on which an index indicating a feature area other than the feature area of ​​the tracking target is further superimposed when a predetermined condition is satisfied by an overlap between the feature area of ​​the tracking target and another feature area, a change in the position or size of the feature area of ​​the tracking target, an orientation of the feature area of ​​the tracking target, or a movement of a device that captured the plurality of images; If the condition is not satisfied, generating display image data in which no index is superimposed on a feature region other than the feature region of the tracking target. An image processing method comprising: (Item 16) 14. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 13.

[0125] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0126] 10... lens unit, 20... image processing circuit, 28... image display unit, 50... system control unit, 70... operation unit, 100... digital camera

Claims

1. a tracking means for detecting the position of a characteristic region of a tracking target in a plurality of images acquired in time series; a generating means for generating display image data on which an index indicating a characteristic region of the tracking target is superimposed based on data of the plurality of images, The generating means an image processing device that generates display image data on which an index indicating a feature area other than the characteristic area of ​​the tracking target is further superimposed when a predetermined condition is met by an overlap of the characteristic area of ​​the tracking target with another characteristic area, a change in the position or size of the characteristic area of ​​the tracking target, an orientation of the characteristic area of ​​the tracking target, or a movement of a device that captured the plurality of images; and that generates display image data on which no index is superimposed on a feature area other than the characteristic area of ​​the tracking target when the condition is not met.

2. The image processing device according to claim 1 , wherein the predetermined condition is a condition that is considered to be highly likely to make it impossible to detect the characteristic region of the tracking target.

3. 2. The image processing device according to claim 1, wherein the characteristic area for which the indicator is displayed when the predetermined condition is satisfied is the characteristic area that is most likely to become the characteristic area of ​​a new tracking target when the characteristic area of ​​the tracking target can no longer be detected.

4. 2. The image processing device according to claim 1, wherein the characteristic area for which the indicator is displayed when the predetermined condition is satisfied is the characteristic area that is the most likely to become the characteristic area of ​​a new tracking target when the characteristic area of ​​the tracking target cannot be detected, among the characteristic areas of the same type as the characteristic area of ​​the tracking target.

5. a detection means for detecting a feature region from the image; a selection means for selecting a feature region of the tracking target from the feature regions detected by the detection means; 2. The image processing apparatus according to claim 1, further comprising:

6. The image processing device according to claim 5 , wherein the selection means selects the feature region of the tracking target from the feature regions detected by the detection means in response to an operation on an operation means provided in the image processing device.

7. 7. The image processing device according to claim 6, wherein when an operation on a touch display serving as the operation means is detected, the generation means generates display image data in which indicators are superimposed on all selectable feature areas among the feature areas detected by the detection means.

8. When an operation on the touch display serving as the operation means is no longer detected, the selection means selects the feature region of the tracking target according to a position where the operation is no longer detected; 8. The image processing device according to claim 7, wherein the generating means generates display image data in which no index is superimposed on any of the characteristic regions detected by the detecting means other than the characteristic region selected by the selecting means.

9. 7. The image processing device according to claim 6, characterized in that when an operation of one of a plurality of operating members, each corresponding to one of a plurality of directions, is detected, the selection means selects, from the feature areas detected by the detection means, the feature area that is closest to the feature area of ​​the tracking target in a direction corresponding to one of the plurality of operating members as the feature area of ​​the tracking target.

10. 10. The image processing device according to claim 9, wherein the generating means generates display image data on which an index is superimposed for a feature region that is closest in the plurality of directions to the feature region of the tracking target selected by the selecting means.

11. 10. The image processing device according to claim 9, wherein, when it is determined that selection of the feature areas of the tracking target in response to operation of the plurality of operating members has been completed, the generation means generates display image data in which no indicator is superimposed on any of the feature areas detected by the detection means other than the feature area selected by the selection means.

12. 7. The image processing device according to claim 6, wherein when an operation on an operating member that switches the feature area of ​​the tracking target each time it is operated is detected, the selection means selects, from the feature areas detected by the detection means, the feature area with the highest priority other than the feature area of ​​the current tracking target as the feature area of ​​the tracking target.

13. 13. The image processing apparatus according to claim 12, wherein the generating means generates display image data on which an index is superimposed for a feature region that will be selected when the operating member is next operated.

14. 13. The image processing device according to claim 12, wherein, when it is determined that selection of the feature area of ​​the tracking target in response to operation of the operating member has been completed, the generation means generates display image data in which no index is superimposed on any of the feature areas detected by the detection means other than the feature area selected by the selection means.

15. an imaging means for capturing a plurality of images in time series; an image processing device according to any one of claims 1 to 14, which uses the plurality of images captured by the imaging means; An imaging device comprising:

16. An image processing method performed by an apparatus, comprising: a tracking step of detecting the position of a characteristic region of a tracking target in a plurality of images acquired in time series; a generating step of generating display image data on which an index indicating a characteristic region of the tracking target is superimposed based on data of the plurality of images, The generating step includes: generating display image data on which an index indicating a feature area other than the feature area of ​​the tracking target is further superimposed when a predetermined condition is satisfied by an overlap between the feature area of ​​the tracking target and another feature area, a change in the position or size of the feature area of ​​the tracking target, an orientation of the feature area of ​​the tracking target, or a movement of a device that captured the plurality of images; If the condition is not satisfied, generating display image data in which no index is superimposed on a feature region other than the feature region of the tracking target. An image processing method comprising:

17. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 14.

Images